www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - More radical ideas about gc and reference counting

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter and I have had a long chat in which we figured our current 
offering of abstractions could be improved. Here are some thoughts. 
There's a lot of work ahead of us on that and I wanted to make sure 
we're getting full community buy-in and backup.

First off, we're considering eliminating destructor calls from within 
the GC entirely. It makes for a faster and better GC, but the real 
reason here is that destructors are philosophically bankrupt in a GC 
environment. I think there's no need to argue that in this community. 
The GC never guarantees calling destructors even today, so this decision 
would be just a point in the definition space (albeit an extreme one).

That means classes that need cleanup (either directly or by having 
fields that are structs with destructors) would need to garner that by 
other means, such as reference counting or manual. We're considering 
deprecating ~this() for classes in the future.

Also, we're considering a revamp of built-in slices, as follows. Slices 
of types without destructors stay as they are.

Slices T[] of structs with destructors shall be silently lowered into 
RCSlice!T, defined inside object.d. That type would occupy THREE words, 
one of which being a pointer to a reference count. That type would 
redefine all slice primitives to update the reference count accordingly.

RCSlice!T will not convert implicitly to void[]. Explicit cast(void[]) 
will be allowed, and will ignore the reference count (so if a void[] 
extracted from a T[] via a cast outlives all slices, dangling pointers 
will ensue).

I foresee any number of theoretical and practical issues with this 
approach. Let's discuss some of them here.


Thanks,

Andrei
Apr 30 2014
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:

 Walter and I have had a long chat in which we figured our 
 current offering of abstractions could be improved. Here are 
 some thoughts. There's a lot of work ahead of us on that and I 
 wanted to make sure we're getting full community buy-in and 
 backup.

 First off, we're considering eliminating destructor calls from 
 within the GC entirely. It makes for a faster and better GC, 
 but the real reason here is that destructors are 
 philosophically bankrupt in a GC environment. I think there's 
 no need to argue that in this community. The GC never 
 guarantees calling destructors even today, so this decision 
 would be just a point in the definition space (albeit an 
 extreme one).


An extreme one indeed, it would break a lot of my code. Every D 
project I wrote that does networking manages memory using a class 
that resides on the managed heap, but holds the actual wrapped 
data in the unmanaged heap. It has a number of advantages over 
other approaches, and I've posted it to the announce group, but 
the idea didn't seem to catch on.

https://github.com/CyberShadow/ae/blob/master/sys/data.d

I could migrate the concept to use reference counting (it was 
implemented in D1, before reference counting was possible), but 
that's my situation regarding breaking existing code.
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:31 PM, Vladimir Panteleev wrote:

 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote:

 Walter and I have had a long chat in which we figured our current
 offering of abstractions could be improved. Here are some thoughts.
 There's a lot of work ahead of us on that and I wanted to make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within
 the GC entirely. It makes for a faster and better GC, but the real
 reason here is that destructors are philosophically bankrupt in a GC
 environment. I think there's no need to argue that in this community.
 The GC never guarantees calling destructors even today, so this
 decision would be just a point in the definition space (albeit an
 extreme one).


 An extreme one indeed, it would break a lot of my code. Every D project
 I wrote that does networking manages memory using a class that resides
 on the managed heap, but holds the actual wrapped data in the unmanaged
 heap.


So should I take it those classes all have destructors? -- Andrei
Apr 30 2014
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 30 April 2014 at 20:45:57 UTC, Andrei Alexandrescu 
wrote:

 An extreme one indeed, it would break a lot of my code. Every 
 D project
 I wrote that does networking manages memory using a class that 
 resides
 on the managed heap, but holds the actual wrapped data in the 
 unmanaged
 heap.


 So should I take it those classes all have destructors? --


One class, many instances (one for every chunk of data sent or 
received).
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:57 PM, Vladimir Panteleev wrote:

 On Wednesday, 30 April 2014 at 20:45:57 UTC, Andrei Alexandrescu wrote:

 An extreme one indeed, it would break a lot of my code. Every D project
 I wrote that does networking manages memory using a class that resides
 on the managed heap, but holds the actual wrapped data in the unmanaged
 heap.


 So should I take it those classes all have destructors? --


 One class, many instances (one for every chunk of data sent or received).


The question is, how comfortable are you with today's reality that some 
of those instances may be never destroyed? -- Andrei
Apr 30 2014
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 30 April 2014 at 20:59:34 UTC, Andrei Alexandrescu 
wrote:

 On 4/30/14, 1:57 PM, Vladimir Panteleev wrote:

 On Wednesday, 30 April 2014 at 20:45:57 UTC, Andrei 
 Alexandrescu wrote:

 An extreme one indeed, it would break a lot of my code. 
 Every D project
 I wrote that does networking manages memory using a class 
 that resides
 on the managed heap, but holds the actual wrapped data in 
 the unmanaged
 heap.


 So should I take it those classes all have destructors? --


 One class, many instances (one for every chunk of data sent or 
 received).


 The question is, how comfortable are you with today's reality 
 that some of those instances may be never destroyed? -- Andrei


I'm fine with it. The proxy objects are purposefully as small as 
possible, to make false references (and thus, memory leaks) 
unlikely, even on 32 bits. If I was concerned with it, I'd have 
switched to reference counting as soon as it was possible.
Apr 30 2014
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2014 10:45 PM, Andrei Alexandrescu wrote:

 An extreme one indeed, it would break a lot of my code. Every D project
 I wrote that does networking manages memory using a class that resides
 on the managed heap, but holds the actual wrapped data in the unmanaged
 heap.


 So should I take it those classes all have destructors? -- Andrei


(Yes, those destructors free the unmanaged memory.)
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:57 PM, Timon Gehr wrote:

 On 04/30/2014 10:45 PM, Andrei Alexandrescu wrote:

 An extreme one indeed, it would break a lot of my code. Every D project
 I wrote that does networking manages memory using a class that resides
 on the managed heap, but holds the actual wrapped data in the unmanaged
 heap.


 So should I take it those classes all have destructors? -- Andrei


 (Yes, those destructors free the unmanaged memory.)


Thanks... that would need to change :o). -- Andrei
Apr 30 2014
parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 14:00:31 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 On 4/30/14, 1:57 PM, Timon Gehr wrote:

 On 04/30/2014 10:45 PM, Andrei Alexandrescu wrote:

 An extreme one indeed, it would break a lot of my code. Every D
 project I wrote that does networking manages memory using a class
 that resides on the managed heap, but holds the actual wrapped
 data in the unmanaged heap.


 So should I take it those classes all have destructors? -- Andrei


 (Yes, those destructors free the unmanaged memory.)


 
 Thanks... that would need to change :o). -- Andrei


And it doesn't even work now, because it's not guaranteed that
finalizers get run. And IIRC, based on some of the discussions at dconf
last year, dealing with the GC and unloading shared libraries would
probably make the situation even worse.

But not being able to rely on finalizers running does put us in a bit
of a pickle, because it basically means that any case where you need a
finalizer, you should probably be using reference counting rather than
the GC. That would tend to mean that either classes are going to need
to be wrapped in a struct that reference-counts them and/or they're
going to need to be allocated with a custom allocator rather than the
GC.


created at the beginning of the using block, and it's dispose method is
called when that block is exited (it may also be collected then, but I
don't remember). I don't think that that's quite what we want, since
there are plenty of cases where you want to pass a class around, so
some kind of reference counting would be better, but we probably should
consider having some kind of standard function similar to dispose so
that there's a standard method to call when a class needs to be cleaned
up. And that could tie into a struct in Phobos that we would have to do
the reference counting by wrapping the class.

- Jonathan M Davis
May 01 2014
prev sibling next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:

 We're considering deprecating ~this() for classes in the future.

 [...]

 Slices T[] of structs with destructors shall be silently 
 lowered into RCSlice!T, defined inside object.d


I'm sure instead of outright deprecating destructor for class, we 
could just store them in a RCClass!C (which would be 2 words big).

If you can get the scheme working with slices of structs with 
destructors, I don't see why the same scheme couldn't work with 
Class'es too.
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:33 PM, monarch_dodra wrote:

 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote:

 We're considering deprecating ~this() for classes in the future.

 [...]

 Slices T[] of structs with destructors shall be silently lowered into
 RCSlice!T, defined inside object.d


 I'm sure instead of outright deprecating destructor for class, we could
 just store them in a RCClass!C (which would be 2 words big).


That's an interesting idea, but the whole notion of converting to Object 
etc. makes the concept difficult (unless we define another root etc).

Andrei
Apr 30 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:

 Walter and I have had a long chat in which we figured our 
 current offering of abstractions could be improved. Here are 
 some thoughts. There's a lot of work ahead of us on that and I 
 wanted to make sure we're getting full community buy-in and 
 backup.


Still no consideration for isolated.


 First off, we're considering eliminating destructor calls from 
 within the GC entirely. It makes for a faster and better GC, 
 but the real reason here is that destructors are 
 philosophically bankrupt in a GC environment. I think there's 
 no need to argue that in this community. The GC never 
 guarantees calling destructors even today, so this decision 
 would be just a point in the definition space (albeit an 
 extreme one).

 That means classes that need cleanup (either directly or by 
 having fields that are structs with destructors) would need to 
 garner that by other means, such as reference counting or 
 manual. We're considering deprecating ~this() for classes in 
 the future.


Not sure how I feel about that. This has been proposed as a 
solution to some GC issue at last DConf and has been dismissed. 
Your post is unclear on why this decision has changed. There must 
be some new information or data that inform this decision.

Or is it random ?

Also, what about cycle in RC things ? Also, RC is good on top of 
GC, so you can collect loop, especially if we are going to RC 
automagically. That is a major issue.

Finally, immutable is sharable accross thread. That mean, even if 
we bypass the type system, that RC must be atomic for immutable. 
As they convert automatically for co,st, that mean that all const 
code will be full of atomic increment/decrement. They are bad for 
the CPU, and cannot be optimized away.

That mean we are back to having a const and a mutable version of 
functions for performance reasons.


 Also, we're considering a revamp of built-in slices, as 
 follows. Slices of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently 
 lowered into RCSlice!T, defined inside object.d. That type 
 would occupy THREE words, one of which being a pointer to a 
 reference count. That type would redefine all slice primitives 
 to update the reference count accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit 
 cast(void[]) will be allowed, and will ignore the reference 
 count (so if a void[] extracted from a T[] via a cast outlives 
 all slices, dangling pointers will ensue).


This won't work. array have a lot of magic that cannot be created 
as library. ie:

RCSlice!(const T) has nothing to do with const(RCSlice!T) as far 
as the compiler is concerned.

This is why we still don't have any library that provides 
collection properly. One of the best is probably dcollection, and 
it has no solution for that problem.

I'd like to see some core issue like that one before changing 
everything on top of it. When the whole thing is built on goo, 
readjusting the top to make it point upward is doomed to create 
the Pisa tower. Fine for tourists, but not very practical nor 
useful.
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:46 PM, deadalnix wrote:

 Still no consideration for isolated.


https://github.com/D-Programming-Language/dmd/pull/3452 may be of 
interest to you. -- Andrei
Apr 30 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 20:49:30 UTC, Andrei Alexandrescu 
wrote:

 On 4/30/14, 1:46 PM, deadalnix wrote:

 Still no consideration for isolated.


 https://github.com/D-Programming-Language/dmd/pull/3452 may be 
 of interest to you. -- Andrei


I already commented on past similar proposal. These are piling up 
more and more hacks to achieve roughly the same thing, badly. 
This makes it impossible to provide a 3rd party implementation 
that is compliant, is guaranteed to ends up in a nightmarish 
explosion of special cases, or to be vastly underpowered.
Apr 30 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 1:57 PM, deadalnix wrote:

 I already commented on past similar proposal. These are piling up more and more
 hacks to achieve roughly the same thing, badly. This makes it impossible to
 provide a 3rd party implementation that is compliant, is guaranteed to ends up
 in a nightmarish explosion of special cases, or to be vastly underpowered.


Actually, I think inference of the ability to implicitly and safely convert
from 
mutable to immutable or shared to be very powerful. I also think it is superior 
than requiring the user to add ever more annotations, which some research (and 
my experience) shows that users are reluctant to do.

I don't buy that it is hackish, etc., especially without more of a rationale as 
to why.

A link to your previous comment would be useful.
Apr 30 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 1 May 2014 at 01:20:37 UTC, Walter Bright wrote:

 On 4/30/2014 1:57 PM, deadalnix wrote:

 I already commented on past similar proposal. These are piling 
 up more and more
 hacks to achieve roughly the same thing, badly. This makes it 
 impossible to
 provide a 3rd party implementation that is compliant, is 
 guaranteed to ends up
 in a nightmarish explosion of special cases, or to be vastly 
 underpowered.


 Actually, I think inference of the ability to implicitly and 
 safely convert from mutable to immutable or shared to be very 
 powerful. I also think it is superior than requiring the user 
 to add ever more annotations, which some research (and my 
 experience) shows that users are reluctant to do.

 I don't buy that it is hackish, etc., especially without more 
 of a rationale as to why.

 A link to your previous comment would be useful.


I've written several proposal regarding this. Please at least 
read them as what you just wrote only proves you are not well 
informed.

Notably, the proposal do not require annotations from the users 
to do the kind of things are are currently being special cased.
Apr 30 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 6:50 PM, deadalnix wrote:

 On Thursday, 1 May 2014 at 01:20:37 UTC, Walter Bright wrote:

 A link to your previous comment would be useful.


 I've written several proposal regarding this. Please at least read them as what
 you just wrote only proves you are not well informed.

 Notably, the proposal do not require annotations from the users to do the kind
 of things are are currently being special cased.


A link would be nice.
Apr 30 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 1 May 2014 at 02:36:23 UTC, Walter Bright wrote:

 On 4/30/2014 6:50 PM, deadalnix wrote:

 On Thursday, 1 May 2014 at 01:20:37 UTC, Walter Bright wrote:

 A link to your previous comment would be useful.


 I've written several proposal regarding this. Please at least 
 read them as what
 you just wrote only proves you are not well informed.

 Notably, the proposal do not require annotations from the 
 users to do the kind
 of things are are currently being special cased.


 A link would be nice.


http://forum.dlang.org/thread/yiwcgyfzfbkzcavuqdwz forum.dlang.org

Just made a new topic with code sample, so the idea can be 
grasped more easily (or so I hope).
May 01 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 Actually, I think inference of the ability to implicitly and 
 safely convert from mutable to immutable or shared to be very 
 powerful. I also think it is superior than requiring the user 
 to add ever more annotations, which some research (and my 
 experience) shows that users are reluctant to do.


I agree the people are reluctant to add annotations, but I think 
it's also depends on the return of investment of the specific 
annotation. I am annotating all my code carefully with "pure", 
but the ROI of all those purity annotations is not great.

And I agree the piling of special cases is a not clean design. 
It's now simpler to ask the compiler if a certain assignment to 
immutable is allowed or not. Because I can't remember the 
supported cases.

So I suggest a more principled and visible approach at the 
problem of ownership and uniqueness.

Bye,
bearophile
Apr 30 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 11:57 PM, bearophile wrote:

 I agree the people are reluctant to add annotations, but I think it's also
 depends on the return of investment of the specific annotation. I am annotating
 all my code carefully with "pure", but the ROI of all those purity annotations
 is not great.


That's why I want to move more towards attribute inference.



 And I agree the piling of special cases is a not clean design. It's now simpler
 to ask the compiler if a certain assignment to immutable is allowed or not.
 Because I can't remember the supported cases.
 So I suggest a more principled and visible approach at the problem of ownership
 and uniqueness.


The "supported cases" are not random special case hacks. They are based on a 
solid principle - if the expression represents a unique reference to an object, 
then it is implicitly convertible to immutable or shared.

For example,

     new int;

Formerly, that created a pointer to a mutable int object. It could not be 
implicitly cast to a pointer to an immutable int. But we know that operator new 
returns the only pointer to what it created, so it should be convertible. This 
knowledge was simply added to the compiler.

I don't see anything unprincipled about that, or hard to understand, or
hackish, 
or whatever.
May 01 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:46 PM, deadalnix wrote:

 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote:

 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that by
 other means, such as reference counting or manual. We're considering
 deprecating ~this() for classes in the future.


 Not sure how I feel about that.


"Ecstatic" is what you're looking for.


 This has been proposed as a solution to
 some GC issue at last DConf and has been dismissed. Your post is unclear
 on why this decision has changed. There must be some new information or
 data that inform this decision.

 Or is it random ?


I don't remember a proposal being made that made slices of structs with 
destructors distinct from other slices.


 Also, what about cycle in RC things?


Resource leaks may happen in programs in any language. I'm prepared to 
allow leaks in certain cases, and I posit here that such cases are rare 
(arrays of objects that embed references to the very array they're in).


 Also, RC is good on top of GC, so
 you can collect loop, especially if we are going to RC automagically.
 That is a major issue.


That stays, and it's useful. I'm talking destructors here.


 Finally, immutable is sharable accross thread. That mean, even if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const code will be
 full of atomic increment/decrement. They are bad for the CPU, and cannot
 be optimized away.


Good point. I see that as a problem, albeit a solvable one.


 That mean we are back to having a const and a mutable version of
 functions for performance reasons.


I don't think that's the case. We could and should be able to only use 
refcounting for truly immutable slices. (Using a bit in the counter to 
indicate sharing comes to mind.)


 RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
 will be allowed, and will ignore the reference count (so if a void[]
 extracted from a T[] via a cast outlives all slices, dangling pointers
 will ensue).


 This won't work. array have a lot of magic that cannot be created as
 library. ie:

 RCSlice!(const T) has nothing to do with const(RCSlice!T) as far as the
 compiler is concerned.


We need to improve the language to allow for such. Did I mention it's 
not going to be easy?


Andrei
Apr 30 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu 
wrote:

 "Ecstatic" is what you're looking for.


Probably...


 This has been proposed as a solution to
 some GC issue at last DConf and has been dismissed. Your post 
 is unclear
 on why this decision has changed. There must be some new 
 information or
 data that inform this decision.

 Or is it random ?


 I don't remember a proposal being made that made slices of 
 structs with destructors distinct from other slices.


So the RC slice thing is what make all that worthwhile ?


 Also, RC is good on top of GC, so
 you can collect loop, especially if we are going to RC 
 automagically.
 That is a major issue.


 That stays, and it's useful. I'm talking destructors here.


Then the RCSlice do not provide any guarantee at all.


 RCSlice!T will not convert implicitly to void[]. Explicit 
 cast(void[])
 will be allowed, and will ignore the reference count (so if a 
 void[]
 extracted from a T[] via a cast outlives all slices, dangling 
 pointers
 will ensue).


 This won't work. array have a lot of magic that cannot be 
 created as
 library. ie:

 RCSlice!(const T) has nothing to do with const(RCSlice!T) as 
 far as the
 compiler is concerned.


 We need to improve the language to allow for such. Did I 
 mention it's not going to be easy?


If I understand what you mentioned, RCSlice is what make to you 
the ditching of destructor worthwhile. The fact that this is not 
doable in current D should be a showstopper for this whole thread 
as all that is discussed here is dependent of what solution we 
choose for this problem (right now we have none).
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:24 PM, deadalnix wrote:

 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu wrote:

 I don't remember a proposal being made that made slices of structs
 with destructors distinct from other slices.


 So the RC slice thing is what make all that worthwhile ?


I can't discuss this as I'm not sure what past discussions it refers.


 Also, RC is good on top of GC, so
 you can collect loop, especially if we are going to RC automagically.
 That is a major issue.


 That stays, and it's useful. I'm talking destructors here.


 Then the RCSlice do not provide any guarantee at all.


The guarantee it provides is once the refcount goes down to 0, elements 
will be destroyed.


 We need to improve the language to allow for such. Did I mention it's
 not going to be easy?


 If I understand what you mentioned, RCSlice is what make to you the
 ditching of destructor worthwhile.


No, they are somewhat loosely related. Destructors are not called for 
arrays of struct _today_, so in that regard we're looking at improving a 
preexisting issue.


 The fact that this is not doable in
 current D should be a showstopper for this whole thread as all that is
 discussed here is dependent of what solution we choose for this problem
 (right now we have none).


This thread is a good place to discuss language improvements that make 
RCSlice possible.


Andrei
Apr 30 2014
prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu 
wrote:

 Finally, immutable is sharable accross thread. That mean, even 
 if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const code 
 will be
 full of atomic increment/decrement. They are bad for the CPU, 
 and cannot
 be optimized away.


 Good point. I see that as a problem, albeit a solvable one.


How? Having lock; instructions implicitly appearing in normal 
looking slice code is unacceptable.
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:47 PM, John Colvin wrote:

 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu wrote:

 Finally, immutable is sharable accross thread. That mean, even if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const code will be
 full of atomic increment/decrement. They are bad for the CPU, and cannot
 be optimized away.


 Good point. I see that as a problem, albeit a solvable one.


 How? Having lock; instructions implicitly appearing in normal looking
 slice code is unacceptable.


I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and 
interlocked refcounts go like 2, 4, 6, ...

Then you do an unprotected read of the refcount. If it's odd, then it's 
impossible to having originated as an interlocked one. So proceed with 
simple increment. If it's even, do an interlocked increment.


Andrei
Apr 30 2014
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 30 April 2014 at 21:51:17 UTC, Andrei Alexandrescu 
wrote:

 On 4/30/14, 2:47 PM, John Colvin wrote:

 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei 
 Alexandrescu wrote:

 Finally, immutable is sharable accross thread. That mean, 
 even if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const 
 code will be
 full of atomic increment/decrement. They are bad for the 
 CPU, and cannot
 be optimized away.


 Good point. I see that as a problem, albeit a solvable one.


 How? Having lock; instructions implicitly appearing in normal 
 looking
 slice code is unacceptable.


 I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, 
 ... and interlocked refcounts go like 2, 4, 6, ...

 Then you do an unprotected read of the refcount. If it's odd, 
 then it's impossible to having originated as an interlocked 
 one. So proceed with simple increment. If it's even, do an 
 interlocked increment.


 Andrei


I don't think I fully understand.

Either all RC changes for a given type need to be atomic or none 
do, and that information is given by the type (everything that is 
immutable/const/shared). I don't see any feasible way of escaping 
this, or any advantage to a runtime convention like the odd/even 
trick above.
Apr 30 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 22:08:23 UTC, John Colvin wrote:

 I don't think I fully understand.

 Either all RC changes for a given type need to be atomic or 
 none do, and that information is given by the type (everything 
 that is immutable/const/shared). I don't see any feasible way 
 of escaping this, or any advantage to a runtime convention like 
 the odd/even trick above.


If you CPU is decent you have some cache coherency protocol in 
place. This won't ensure that thing appears sequentially 
consistent, but you don't care here.

You can proceed as follow in pseudo assembly :

count = load count_addr
need_atomic = count & 0x01
brtr atomic
count = count + 2
store count count_addr
br follow_up

atomic:
atomic_add count_addr 2

follow_up:
// Code after increment goes here

Note that is working as count may not be the correct number in 
case of sharing, but will always have the same parity, so even 
reading the wrong value will make you branch properly and the 
value of count is not used to increment in the atomic block.

I'm not happy with this solution, because:
  - You still have an atomic in there, and the compiler can't 
remove it. This reduce greatly the capability of the compiler to 
optimize. For instance, the compiler cannot optimize away 
redundant pairs of increment/decrement.
  - You have a branch in there. Atomic are expensive, but branch 
as well. Especially since both are storing (and one atomically), 
so it can't be speculated.
  - If we start using that all over the place, the codegen will 
ends up being quite fat. That means less friendly cache behavior.

That odd/even solution surely works, but ultimately do not solve 
the issue: if you want full speed, you'll have to provide both a 
const and a mutable version of the code, which defeat the purpose 
of const. Note that the exact same issue exists with inout.
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:32 PM, deadalnix wrote:

 On Wednesday, 30 April 2014 at 22:08:23 UTC, John Colvin wrote:

 I don't think I fully understand.

 Either all RC changes for a given type need to be atomic or none do,
 and that information is given by the type (everything that is
 immutable/const/shared). I don't see any feasible way of escaping
 this, or any advantage to a runtime convention like the odd/even trick
 above.


 If you CPU is decent you have some cache coherency protocol in place.
 This won't ensure that thing appears sequentially consistent, but you
 don't care here.

 You can proceed as follow in pseudo assembly :

 count = load count_addr
 need_atomic = count & 0x01
 brtr atomic
 count = count + 2
 store count count_addr
 br follow_up

 atomic:
 atomic_add count_addr 2

 follow_up:
 // Code after increment goes here


brtr_atomic won't be needed on x86.


 Note that is working as count may not be the correct number in case of
 sharing, but will always have the same parity, so even reading the wrong
 value will make you branch properly and the value of count is not used
 to increment in the atomic block.

 I'm not happy with this solution, because:
   - You still have an atomic in there, and the compiler can't remove it.
 This reduce greatly the capability of the compiler to optimize. For
 instance, the compiler cannot optimize away redundant pairs of
 increment/decrement.


Only on const slices, architectures more relaxed than x86, and objects 
with destructors.


 That odd/even solution surely works, but ultimately do not solve the
 issue: if you want full speed, you'll have to provide both a const and a
 mutable version of the code, which defeat the purpose of const. Note
 that the exact same issue exists with inout.


I don't think it's all doom and gloom. Defining a destructor for an 
object suggests that the object is willing and able to give up a little 
speed for the sake of automation. I think it all dovetails nicely.


Andrei
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:54 PM, Andrei Alexandrescu wrote:

 On 4/30/14, 3:32 PM, deadalnix wrote:

 On Wednesday, 30 April 2014 at 22:08:23 UTC, John Colvin wrote:

 I don't think I fully understand.

 Either all RC changes for a given type need to be atomic or none do,
 and that information is given by the type (everything that is
 immutable/const/shared). I don't see any feasible way of escaping
 this, or any advantage to a runtime convention like the odd/even trick
 above.


 If you CPU is decent you have some cache coherency protocol in place.
 This won't ensure that thing appears sequentially consistent, but you
 don't care here.

 You can proceed as follow in pseudo assembly :

 count = load count_addr
 need_atomic = count & 0x01
 brtr atomic
 count = count + 2
 store count count_addr
 br follow_up

 atomic:
 atomic_add count_addr 2

 follow_up:
 // Code after increment goes here


 brtr_atomic won't be needed on x86.


Sorry, I misread your code. I meant to say on x86 there's no need to do 
any handshake on the single-threaded case. -- Andrei
Apr 30 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 23:05:29 UTC, Andrei Alexandrescu 
wrote:

 Sorry, I misread your code. I meant to say on x86 there's no 
 need to do any handshake on the single-threaded case. -- Andrei


You don't need it on most arch. Except alpha, I don't know any 
that would require it.
Apr 30 2014
next sibling parent reply "froglegs" <barf barf.com> writes:
  Here is an alternative:

  1. classes are unique by default, they cannot be shared. No GC 
or RF required, and covers 90% of objects.
  2. To share, retype as GC, the requirement being that it does 
not have a destructor.
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 4:19 PM, froglegs wrote:

   Here is an alternative:

   1. classes are unique by default, they cannot be shared. No GC or RF
 required, and covers 90% of objects.


Where does 90% come from? On the contrary, it seems to me unique 
references are rather rare and fleeting. -- Andrei
Apr 30 2014
next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 30 April 2014 at 23:34:33 UTC, Andrei Alexandrescu 
wrote:

 On 4/30/14, 4:19 PM, froglegs wrote:

  Here is an alternative:

  1. classes are unique by default, they cannot be shared. No 
 GC or RF
 required, and covers 90% of objects.


 Where does 90% come from? On the contrary, it seems to me 
 unique references are rather rare and fleeting. -- Andrei


Yeah, 90% doesn't feel right to me. In C++ I try to use 
unique_ptr instead of shared_ptr wherever possible but I find I'm 
often not able to because my references need to be held in 
several places for efficiency. I'd say I'm more like 90% 
reference counted, 10% unique throughout my 300kloc codebase.
Apr 30 2014
prev sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:


 Where does 90% come from? On the contrary, it seems to me 
 unique references are rather rare and fleeting. -- Andrei


I think unique references (linear types) are used rather commonly 
among the reference types of Rust.

Bye,
bearophile
Apr 30 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 4:14 PM, deadalnix wrote:

 On Wednesday, 30 April 2014 at 23:05:29 UTC, Andrei Alexandrescu wrote:

 Sorry, I misread your code. I meant to say on x86 there's no need to
 do any handshake on the single-threaded case. -- Andrei


 You don't need it on most arch. Except alpha, I don't know any that
 would require it.


So much the better. How about ARM? All I know is it's very relaxed. -- 
Andrei
Apr 30 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 23:27:49 UTC, Andrei Alexandrescu 
wrote:

 On 4/30/14, 4:14 PM, deadalnix wrote:

 On Wednesday, 30 April 2014 at 23:05:29 UTC, Andrei 
 Alexandrescu wrote:

 Sorry, I misread your code. I meant to say on x86 there's no 
 need to
 do any handshake on the single-threaded case. -- Andrei


 You don't need it on most arch. Except alpha, I don't know any 
 that
 would require it.


 So much the better. How about ARM? All I know is it's very 
 relaxed. -- Andrei


It is expected that a barrier (dmb on ARM) is inserted when 
sharing information (in std.concurency for instance). So you 
should have some initialized value in there. It can be anything 
though.

That mean that the load can get any value that was there since 
the last sharing done properly. It doesn't matter which one we 
get as they all have the same parity.

This value can't be used in the atomic branch. That is why the 
pseudo-assembly code I posted do not use the counter value in the 
atomic branch.
Apr 30 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:08 PM, John Colvin wrote:

 On Wednesday, 30 April 2014 at 21:51:17 UTC, Andrei Alexandrescu wrote:

 On 4/30/14, 2:47 PM, John Colvin wrote:

 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu wrote:

 Finally, immutable is sharable accross thread. That mean, even if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const code will be
 full of atomic increment/decrement. They are bad for the CPU, and
 cannot
 be optimized away.


 Good point. I see that as a problem, albeit a solvable one.


 How? Having lock; instructions implicitly appearing in normal looking
 slice code is unacceptable.


 I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and
 interlocked refcounts go like 2, 4, 6, ...

 Then you do an unprotected read of the refcount. If it's odd, then
 it's impossible to having originated as an interlocked one. So proceed
 with simple increment. If it's even, do an interlocked increment.


 Andrei


 I don't think I fully understand.

 Either all RC changes for a given type need to be atomic or none do, and
 that information is given by the type (everything that is
 immutable/const/shared). I don't see any feasible way of escaping this,
 or any advantage to a runtime convention like the odd/even trick above.


An object starting as shared or immutable would always need to be 
atomically refcounted. That information is statically known. For those 
we'd initialize the refcount to 2.

An object starting as regular mutable would always be refcounted 
non-atomically. That's also known statically. So that constructor 
initializes the refcount to 1.

Then a const object would dynamically choose the approach to refcounting 
depending on the counter's evenness.


Andrei
Apr 30 2014
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 30 April 2014 at 22:48:28 UTC, Andrei Alexandrescu 
wrote:

 On 4/30/14, 3:08 PM, John Colvin wrote:

 On Wednesday, 30 April 2014 at 21:51:17 UTC, Andrei 
 Alexandrescu wrote:

 On 4/30/14, 2:47 PM, John Colvin wrote:

 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei 
 Alexandrescu wrote:

 Finally, immutable is sharable accross thread. That mean, 
 even if we
 bypass the type system, that RC must be atomic for 
 immutable.
 As they
 convert automatically for co,st, that mean that all const 
 code will be
 full of atomic increment/decrement. They are bad for the 
 CPU, and
 cannot
 be optimized away.


 Good point. I see that as a problem, albeit a solvable one.


 How? Having lock; instructions implicitly appearing in 
 normal looking
 slice code is unacceptable.


 I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, 
 ... and
 interlocked refcounts go like 2, 4, 6, ...

 Then you do an unprotected read of the refcount. If it's odd, 
 then
 it's impossible to having originated as an interlocked one. 
 So proceed
 with simple increment. If it's even, do an interlocked 
 increment.


 Andrei


 I don't think I fully understand.

 Either all RC changes for a given type need to be atomic or 
 none do, and
 that information is given by the type (everything that is
 immutable/const/shared). I don't see any feasible way of 
 escaping this,
 or any advantage to a runtime convention like the odd/even 
 trick above.


 An object starting as shared or immutable would always need to 
 be atomically refcounted. That information is statically known. 
 For those we'd initialize the refcount to 2.

 An object starting as regular mutable would always be 
 refcounted non-atomically. That's also known statically. So 
 that constructor initializes the refcount to 1.

 Then a const object would dynamically choose the approach to 
 refcounting depending on the counter's evenness.


 Andrei


Ah ok, that makes sense.
May 01 2014
prev sibling parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-04-30 21:51:18 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and 
 interlocked refcounts go like 2, 4, 6, ...
 
 Then you do an unprotected read of the refcount. If it's odd, then it's 
 impossible to having originated as an interlocked one. So proceed with 
 simple increment. If it's even, do an interlocked increment.


Nice idea, although I'd add a twist to support polymorphism in class 
hierarchies: add a magic value (say zero) to mean "not 
reference-counted". When you instantiate an object that has a 
destructor, the reference counter is set to 1 or 2 depending on whether 
it's shared or not. If however the object has no destructor and is not 
reference counted, set the counter to the magic value.

Then have the compiler assume all objects are reference counted. At 
runtime only those objects with a non-magic value as the reference 
count will actually increment/decrement the counter.

Finally, let the the compiler understand some situations where objects 
are guarantied to have no destructors a destructor so it can omit 
automatic reference counting code in those cases. One such situation is 
when you have a reference to a final class with no destructor. We could 
also add a  nodestructor attribute to forbid a class and all its 
descendants from having destructors.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca
Apr 30 2014
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2014 10:57 PM, Andrei Alexandrescu wrote:

 RCSlice!(const T) has nothing to do with const(RCSlice!T) as far as the
 compiler is concerned.


 ...


There's always this hack:

struct RCSlice(T){
     static if(!is(T==const))
         RCSlice!(const(StripImmutable!T)) upcast(){
             return cast(typeof(return))this;
         }
     alias upcast this;
     // ...
}


 We need to improve the language to allow for such. Did I mention it's
 not going to be easy?


Well, actually an easy way would be to have the compiler infer safe 
coercions.

(A 'coercion' here is meant to denote a no-op implicit conversion, for 
example, conversion to const.)

It would be enabled with an attribute on the template. Coercions between 
different instantiations of the same templated type would be safe and 
allowed if the memory layout does not change between them and the 
compiler can prove the coercion safe field-wise. There needn't be 
restrictions on non-field members. Types annotated like this would get 
the top-level mutability specifier stripped off if possible, during IFTI 
/ when cast().

In particular, this would work:

class C{}
class D:C{}

 infer_coercions
struct Slice(T){
     size_t length;
     T* ptr;
}

void foo(T)(T arg){ /* ... */ }

void main() safe{
     auto a=[new D,new D];
     auto s=Slice!D(a.length,a.ptr);
     Slice!(const(C)) sc=s; // ok, D* coercible to const(C)*
     const(Slice!C) cs=s; // ok, D* coercible to n-m-ind const(C*)
     sc=cs; // ok, D* is coercible to n-m-ind const(C)*
     // (length works because const(size_t) and size_t can be
     //  freely corced between each other.)

     foo(cs); // actually calls foo!(Slice!(const(C))),

     // ==> no more magic in T[]
}

I.e. slices can be implemented in the library under this proposal.
(The above just forwards the magical behaviour of T* to a user-defined 
type.)

I think this fixes the issue in general, for example, for ranges:

const rng1 = [1,2,3].map!(a=>2*a);
const rng2 = rng1.filter!(a=>!!(a&1)); // ok



Probably this should be slightly generalized, eg:

 infer_coercions
struct S(T){
     T field;
}

void foo(T)(T arg){}

void main(){
     S!(T[]) s;
     foo(s); // foo!(S!(const(T)[]))
}

However, then, whether to do const(S!T) => S!(const(T)) or const(S!T) => 
S!(TailConst!T) should maybe be specified on a per-parameter basis, 
because this is in general not easy to figure out for the compiler.
Apr 30 2014
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/01/2014 12:24 AM, Timon Gehr wrote:

 I think this fixes the issue in general, for example, for ranges:

 const rng1 = [1,2,3].map!(a=>2*a);
 const rng2 = rng1.filter!(a=>!!(a&1)); // ok



 Probably this should be slightly generalized, eg:
 ...


(The generalization is actually needed for the range example to work.)
Apr 30 2014
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/01/2014 12:24 AM, Timon Gehr wrote:

      const(Slice!C) cs=s; // ok, D* coercible to n-m-ind const(C*)
      sc=cs; // ok, D* is coercible to n-m-ind const(C)*


(Ignore the 'n-m-ind', those are accidental leftovers from a half-baked 
version of the post.)
Apr 30 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 22:24:29 UTC, Timon Gehr wrote:

 However, then, whether to do const(S!T) => S!(const(T)) or 
 const(S!T) => S!(TailConst!T) should maybe be specified on a 
 per-parameter basis, because this is in general not easy to 
 figure out for the compiler.


That is the whole problem :D
Apr 30 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/01/2014 12:48 AM, deadalnix wrote:

 On Wednesday, 30 April 2014 at 22:24:29 UTC, Timon Gehr wrote:

 However, then, whether to do const(S!T) => S!(const(T)) or const(S!T)
 => S!(TailConst!T) should maybe be specified on a per-parameter basis,
 because this is in general not easy to figure out for the compiler.


 That is the whole problem :D


Well, actually, just check whether there is a field that would need to 
change type, of the exact type of some parameter. If so, try to do the 
second thing for that parameter, otherwise try to do the first.
May 01 2014
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2014 10:21 PM, Andrei Alexandrescu wrote:

 Walter and I have had a long chat in which we figured our current
 offering of abstractions could be improved. Here are some thoughts.
 There's a lot of work ahead of us on that and I wanted to make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within
 the GC entirely. It makes for a faster and better GC, but the real
 reason here is that destructors are philosophically bankrupt in a GC
 environment. I think there's no need to argue that in this community.
 The GC never guarantees calling destructors even today, so this decision
 would be just a point in the definition space (albeit an extreme one).

 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that by
 other means, such as reference counting or manual. We're considering
 deprecating ~this() for classes in the future.
 ...


struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?


 Also, we're considering a revamp of built-in slices, as follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently lowered into
 RCSlice!T, defined inside object.d. That type would occupy THREE words,
 one of which being a pointer to a reference count. That type would
 redefine all slice primitives to update the reference count accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
 will be allowed, and will ignore the reference count (so if a void[]
 extracted from a T[] via a cast outlives all slices, dangling pointers
 will ensue).

 I foresee any number of theoretical and practical issues with this
 approach. Let's discuss some of them here.
 ...


struct S{
     ~this(){ /* ... */ }
}

class C{
     S[] s;
     this(S[] s){ /* ... */ }
}

void main(){
     new C(buildSs());
     // memory leak
}

Also, cycles.
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:56 PM, Timon Gehr wrote:

 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?


By hand, as I mentioned. -- Andrei
Apr 30 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:

 On 4/30/14, 1:56 PM, Timon Gehr wrote:

 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?


 By hand, as I mentioned. -- Andrei


I meant, is it going to be deprecated too?
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:09 PM, Timon Gehr wrote:

 On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:

 On 4/30/14, 1:56 PM, Timon Gehr wrote:

 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?


 By hand, as I mentioned. -- Andrei


 I meant, is it going to be deprecated too?


No, that will continue to be allowed. -- Andrei
Apr 30 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 4/30/14, 2:09 PM, Timon Gehr wrote:

On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:

On 4/30/14, 1:56 PM, Timon Gehr wrote:

struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?


By hand, as I mentioned. -- Andrei


I meant, is it going to be deprecated too?


 
 No, that will continue to be allowed. -- Andrei


Then what is it going to do? S.~this will never get called?


T

-- 
Never wrestle a pig. You both get covered in mud, and the pig likes it.
Apr 30 2014
next sibling parent reply "sclytrack" <sclytrack fake.com> writes:
On Wednesday, 30 April 2014 at 21:17:23 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 On 4/30/14, 2:09 PM, Timon Gehr wrote:

On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:

On 4/30/14, 1:56 PM, Timon Gehr wrote:

struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?


By hand, as I mentioned. -- Andrei


I meant, is it going to be deprecated too?


 
 No, that will continue to be allowed. -- Andrei


 Then what is it going to do? S.~this will never get called?


It gets called when the gc runs.


 T
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:26 PM, sclytrack wrote:

 It gets called when the gc runs.


No. No calls whatsoever while the gc runs. GC is notoriously bad for 
handling non-memory resources, and I see no reason to continue 
attempting it to do so after so much evidence to the contrary has been, 
um, collected :o).

Andrei
Apr 30 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:15 PM, H. S. Teoh via Digitalmars-d wrote:

 On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 4/30/14, 2:09 PM, Timon Gehr wrote:

 On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:

 On 4/30/14, 1:56 PM, Timon Gehr wrote:

 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?


 By hand, as I mentioned. -- Andrei


 I meant, is it going to be deprecated too?


 No, that will continue to be allowed. -- Andrei


 Then what is it going to do? S.~this will never get called?


That is correct. -- Andrei
Apr 30 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 02:30:05PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 4/30/14, 2:15 PM, H. S. Teoh via Digitalmars-d wrote:

On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

On 4/30/14, 2:09 PM, Timon Gehr wrote:

On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:

On 4/30/14, 1:56 PM, Timon Gehr wrote:

struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?


By hand, as I mentioned. -- Andrei


I meant, is it going to be deprecated too?


No, that will continue to be allowed. -- Andrei


Then what is it going to do? S.~this will never get called?


 
 That is correct. -- Andrei


I don't like the sound of that. I haven't found myself in a place where
I needed to do something like this, but if I had to, I'd be very unhappy
if struct dtors only work when they're not class members. Can we make
them always work, and if necessary prohibit using them as class members?


T

-- 
People tell me I'm stubborn, but I refuse to accept it!
Apr 30 2014
next sibling parent reply "fra" <a b.it> writes:
On Wednesday, 30 April 2014 at 22:49:01 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 On Wed, Apr 30, 2014 at 02:30:05PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 On 4/30/14, 2:15 PM, H. S. Teoh via Digitalmars-d wrote:

On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu 
via Digitalmars-d wrote:

On 4/30/14, 2:09 PM, Timon Gehr wrote:

On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:

On 4/30/14, 1:56 PM, Timon Gehr wrote:

struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?


By hand, as I mentioned. -- Andrei


I meant, is it going to be deprecated too?


No, that will continue to be allowed. -- Andrei


Then what is it going to do? S.~this will never get called?


 
 That is correct. -- Andrei


 I don't like the sound of that. I haven't found myself in a 
 place where
 I needed to do something like this, but if I had to, I'd be 
 very unhappy
 if struct dtors only work when they're not class members. Can 
 we make
 them always work, and if necessary prohibit using them as class 
 members?


 T


Can't do nothing but agree. Unless I'm missing something, no 
destructors means memory leaks every time I use non-class objects 
inside a class object. At least current behaviour makes sure that 
when memory is needed, everything gets cleaned up properly. 
Honestly, this sounds crazy to me.
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:54 PM, fra wrote:

 Can't do nothing but agree. Unless I'm missing something, no destructors
 means memory leaks every time I use non-class objects inside a class
 object.


If said non-class objects have destructors and are not cleaned up via 
other mechanisms.


 At least current behaviour makes sure that when memory is
 needed, everything gets cleaned up properly.


I think this was a false sense of security. There's no "making sure". 
Due to imprecision a destructor may or may not be called. Also, arrays 
of structs _never_ call destructors for their elements, which should 
shock you quite a bit more.


 Honestly, this sounds crazy
 to me.


s/crazy/radical/

Yes, radical. Because we want to uproot some bad weeds here.


Andrei
Apr 30 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote:

 On Wed, Apr 30, 2014 at 02:30:05PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 4/30/14, 2:15 PM, H. S. Teoh via Digitalmars-d wrote:

 On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 4/30/14, 2:09 PM, Timon Gehr wrote:

 On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:

 On 4/30/14, 1:56 PM, Timon Gehr wrote:

 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?


 By hand, as I mentioned. -- Andrei


 I meant, is it going to be deprecated too?


 No, that will continue to be allowed. -- Andrei


 Then what is it going to do? S.~this will never get called?


 That is correct. -- Andrei


 I don't like the sound of that. I haven't found myself in a place where
 I needed to do something like this, but if I had to, I'd be very unhappy
 if struct dtors only work when they're not class members. Can we make
 them always work, and if necessary prohibit using them as class members?


Then we're back to effectively class destructors. I think we've gathered 
quite a bit of evidence there are pernicious issues associated with 
them. -- Andrei
Apr 30 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 03:55:38PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote:


[...]

I don't like the sound of that. I haven't found myself in a place
where I needed to do something like this, but if I had to, I'd be
very unhappy if struct dtors only work when they're not class
members. Can we make them always work, and if necessary prohibit
using them as class members?


 
 Then we're back to effectively class destructors. I think we've
 gathered quite a bit of evidence there are pernicious issues
 associated with them. -- Andrei


[...]

How so? If we prohibit structs with dtors from being class members,
then it could work. Of course, then you'd have to deal with the backlash
from users who inevitably find some use case for it... but at least in
theory it's possible.

But if we're going to go in the direction of lack of dtor guarantees,
then I might suggest we kill off struct dtors as well, since they also
have their own set of pathologies.  Like passing structs with dtors by
value (this includes returning them from a function) causing double
destruction if the dtor isn't written carefully, or putting structs with
dtors into an array or std.container.*, and they stop working as one
might expect.  Wasn't this whole fiasco the whole reason
std.stdio.ByLine was recently rewritten to use ref counting? It relied
on struct dtors before, and look how well that turned out.

Basically, you can't allow something to have dtors, yet have no way to
guarantee it will be called at the expected time. That kind of
under-specification is the source of endless hard-to-trace bugs,
gotchas, unfixable issues, and holes in the type system. If we're going
to have dtors at all, let's do it *right*. Guarantee they always work,
and reject all usages that break this guarantee (like putting a struct
with dtor inside a class, putting it inside a dynamic array in GC
memory, etc.). If we can't guarantee anything at all, then let's not
have dtors at all.  Trying to stay in the gray zone in between does
nothing but cause endless problems down the road. Not to mention
language smells. What's the use of a feature that only sometimes works,
where "sometimes" is un(der)specified, left to implementation, or
depends on undefined behaviour?


T

-- 
In a world without fences, who needs Windows and Gates? -- Christian Surchi
Apr 30 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 4:17 PM, H. S. Teoh via Digitalmars-d wrote:

 How so? If we prohibit structs with dtors from being class members,
 then it could work. Of course, then you'd have to deal with the backlash
 from users who inevitably find some use case for it... but at least in
 theory it's possible.


Oh I see. I'd eliminated that ex officio because (a) it's a frequent 
case so lotsa breakage, and (b) we lack a replacement to recommend. -- 
Andrei
Apr 30 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 4:17 PM, H. S. Teoh via Digitalmars-d wrote:

 But if we're going to go in the direction of lack of dtor guarantees,
 then I might suggest we kill off struct dtors as well, since they also
 have their own set of pathologies.  Like passing structs with dtors by
 value (this includes returning them from a function) causing double
 destruction if the dtor isn't written carefully,


That sounds like a compiler bug.


 or putting structs with
 dtors into an array or std.container.*,


What's wrong with that? std.container.Array should destroy things 
properly, if not that's a library bug.


 and they stop working as one
 might expect.  Wasn't this whole fiasco the whole reason
 std.stdio.ByLine was recently rewritten to use ref counting? It relied
 on struct dtors before, and look how well that turned out.


But ref counting IS destructors.


 Basically, you can't allow something to have dtors, yet have no way to
 guarantee it will be called at the expected time.


I don't think so - it's navigating close enough to the "so let's not use 
cars anymore" fallacy. There are plenty many situations in which dtors 
are working fabulously well, and throwing them away because we can't 
guarantee they'll work absolutely well seems bad decision making to me.


 That kind of
 under-specification is the source of endless hard-to-trace bugs,
 gotchas, unfixable issues, and holes in the type system. If we're going
 to have dtors at all, let's do it *right*. Guarantee they always work,
 and reject all usages that break this guarantee (like putting a struct
 with dtor inside a class, putting it inside a dynamic array in GC
 memory, etc.). If we can't guarantee anything at all, then let's not
 have dtors at all.  Trying to stay in the gray zone in between does
 nothing but cause endless problems down the road. Not to mention
 language smells. What's the use of a feature that only sometimes works,
 where "sometimes" is un(der)specified, left to implementation, or
 depends on undefined behaviour?


A lot of other languages have such imperfections, and nobody raises a brow.


Andrei
Apr 30 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 04:33:23PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 4/30/14, 4:17 PM, H. S. Teoh via Digitalmars-d wrote:


[...]

and they stop working as one might expect.  Wasn't this whole fiasco
the whole reason std.stdio.ByLine was recently rewritten to use ref
counting? It relied on struct dtors before, and look how well that
turned out.


 
 But ref counting IS destructors.


Which will stop working once class dtors are deprecated, yet you still
allow putting File in a class member.



Basically, you can't allow something to have dtors, yet have no way
to guarantee it will be called at the expected time.


 
 I don't think so - it's navigating close enough to the "so let's not
 use cars anymore" fallacy. There are plenty many situations in which
 dtors are working fabulously well, and throwing them away because we
 can't guarantee they'll work absolutely well seems bad decision making
 to me.


No, allowing the user to specify a struct with a dtor, and put that
struct in a class, and then saying "nyah nyah class dtors don't exist
anymore so your struct will never destruct" -- *that* is bad decision
making. At the very least, the compiler should complain loudly and
clearly that structs with dtors should not be put inside a class. Isn't
this what D philosophy is supposed to be? Things should *work* by
default, and unexpected behaviour (dtor never gets called) should only
happen when the user explicitly asks for it.



That kind of under-specification is the source of endless
hard-to-trace bugs, gotchas, unfixable issues, and holes in the type
system. If we're going to have dtors at all, let's do it *right*.
Guarantee they always work, and reject all usages that break this
guarantee (like putting a struct with dtor inside a class, putting it
inside a dynamic array in GC memory, etc.). If we can't guarantee
anything at all, then let's not have dtors at all.  Trying to stay in
the gray zone in between does nothing but cause endless problems down
the road. Not to mention language smells. What's the use of a feature
that only sometimes works, where "sometimes" is un(der)specified,
left to implementation, or depends on undefined behaviour?


 
 A lot of other languages have such imperfections, and nobody raises a
 brow.


[...]

I thought D is here because we want to do better than that?


T

-- 
GEEK = Gatherer of Extremely Enlightening Knowledge
May 01 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 7:17 AM, H. S. Teoh via Digitalmars-d wrote:

 On Wed, Apr 30, 2014 at 04:33:23PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
 No, allowing the user to specify a struct with a dtor, and put that
 struct in a class, and then saying "nyah nyah class dtors don't exist
 anymore so your struct will never destruct" -- *that* is bad decision
 making. At the very least, the compiler should complain loudly and
 clearly that structs with dtors should not be put inside a class. Isn't
 this what D philosophy is supposed to be? Things should *work* by
 default, and unexpected behaviour (dtor never gets called) should only
 happen when the user explicitly asks for it.


Problem is there's a lot of correct code that e.g. closes the File 
members properly etc. All that code would be disable together with the 
risky code.


 That kind of under-specification is the source of endless
 hard-to-trace bugs, gotchas, unfixable issues, and holes in the type
 system. If we're going to have dtors at all, let's do it *right*.
 Guarantee they always work, and reject all usages that break this
 guarantee (like putting a struct with dtor inside a class, putting it
 inside a dynamic array in GC memory, etc.). If we can't guarantee
 anything at all, then let's not have dtors at all.  Trying to stay in
 the gray zone in between does nothing but cause endless problems down
 the road. Not to mention language smells. What's the use of a feature
 that only sometimes works, where "sometimes" is un(der)specified,
 left to implementation, or depends on undefined behaviour?


 A lot of other languages have such imperfections, and nobody raises a
 brow.


 [...]

 I thought D is here because we want to do better than that?


Better != perfection at all costs.


Andrei
May 01 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 4:17 PM, H. S. Teoh via Digitalmars-d wrote:

 If we're going
 to have dtors at all, let's do it *right*. Guarantee they always work,
 and reject all usages that break this guarantee (like putting a struct
 with dtor inside a class,


Seems to work when I try it:

bar.d:
-------------------------
   import core.stdc.stdio;

   struct S { ~this() { printf("S.~this()\n"); } }

   class C { S s; }

   void main()
   {
     C c = new C();
     c = null;
   }
-------------------------
C:\cbx\mars>bar
S.~this()
Apr 30 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 06:30:16PM -0700, Walter Bright via Digitalmars-d wrote:

 On 4/30/2014 4:17 PM, H. S. Teoh via Digitalmars-d wrote:

If we're going
to have dtors at all, let's do it *right*. Guarantee they always work,
and reject all usages that break this guarantee (like putting a struct
with dtor inside a class,


 
 Seems to work when I try it:
 
 bar.d:
 -------------------------
   import core.stdc.stdio;
 
   struct S { ~this() { printf("S.~this()\n"); } }
 
   class C { S s; }
 
   void main()
   {
     C c = new C();
     c = null;
   }
 -------------------------
 C:\cbx\mars>bar
 S.~this()


The proposal was to get rid of class dtors, in which case this code will
no longer work. Is that really the direction we want to move in?


T

-- 
Let's call it an accidental feature. -- Larry Wall
May 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 7:01 AM, H. S. Teoh via Digitalmars-d wrote:

 The proposal was to get rid of class dtors, in which case this code will
 no longer work. Is that really the direction we want to move in?


My point was that the compiler wasn't ignoring destructors for fields.

Who should call the class destructor and when is a different issue.
May 01 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, May 01, 2014 at 02:19:17PM -0700, Walter Bright via Digitalmars-d wrote:

 On 5/1/2014 7:01 AM, H. S. Teoh via Digitalmars-d wrote:

The proposal was to get rid of class dtors, in which case this code will
no longer work. Is that really the direction we want to move in?


 
 My point was that the compiler wasn't ignoring destructors for fields.
 
 Who should call the class destructor and when is a different issue.


I'm confused. Andrei's original proposal was to deprecate class dtors,
with the view to eventually getting rid of them altogether. If indeed
that's going to be the case, then wouldn't that mean that field dtors
won't be invoked either (because otherwise, how would you know when to
invoke a field dtor?)?

Without a clear definition of who is going to clean up those fields and
invoke their dtors, it would seem to me that having fields with dtors in
a class (once class dtors no longer exist, according to Andrei's
proposal) will be a dangerous thing to do, since the very guarantee by
having a dtor in the first place (it will get called to clean up when
the struct goes out of scope) is broken. If so, what's the point of
having a dtor in the first place? It becomes no different from the
manual cleanup that you have to do in C.


T

-- 
Verbing weirds language. -- Calvin (& Hobbes)
May 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 3:02 PM, H. S. Teoh via Digitalmars-d wrote:

 I'm confused. Andrei's original proposal was to deprecate class dtors,
 with the view to eventually getting rid of them altogether. If indeed
 that's going to be the case, then wouldn't that mean that field dtors
 won't be invoked either (because otherwise, how would you know when to
 invoke a field dtor?)?

 Without a clear definition of who is going to clean up those fields and
 invoke their dtors, it would seem to me that having fields with dtors in
 a class (once class dtors no longer exist, according to Andrei's
 proposal) will be a dangerous thing to do, since the very guarantee by
 having a dtor in the first place (it will get called to clean up when
 the struct goes out of scope) is broken. If so, what's the point of
 having a dtor in the first place? It becomes no different from the
 manual cleanup that you have to do in C.


The thing is, GC is a terrible and unreliable method of managing non-memory 
resource lifetimes. Destructors for GC objects are not guaranteed to ever run. 
If you do have struct with a destructor as a field in a class, you've got, at 
minimum, suspicious code and a latent bug.
May 01 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, May 01, 2014 at 03:10:04PM -0700, Walter Bright via Digitalmars-d wrote:

 On 5/1/2014 3:02 PM, H. S. Teoh via Digitalmars-d wrote:

I'm confused. Andrei's original proposal was to deprecate class
dtors, with the view to eventually getting rid of them altogether. If
indeed that's going to be the case, then wouldn't that mean that
field dtors won't be invoked either (because otherwise, how would you
know when to invoke a field dtor?)?

Without a clear definition of who is going to clean up those fields
and invoke their dtors, it would seem to me that having fields with
dtors in a class (once class dtors no longer exist, according to
Andrei's proposal) will be a dangerous thing to do, since the very
guarantee by having a dtor in the first place (it will get called to
clean up when the struct goes out of scope) is broken. If so, what's
the point of having a dtor in the first place? It becomes no
different from the manual cleanup that you have to do in C.


 
 The thing is, GC is a terrible and unreliable method of managing
 non-memory resource lifetimes. Destructors for GC objects are not
 guaranteed to ever run. If you do have struct with a destructor as a
 field in a class, you've got, at minimum, suspicious code and a latent
 bug.


Exactly!!! This is why I said we should ban the use of structs with
dtors as a field in a class. Isn't the D philosophy to be correct by
default, and only allow potentially wrong/error-prone code if the user
explicitly asks for it?  So this kind of error-prone usage should not be
allowed by default.  Was my logic really that hard to follow?


T

-- 
"Hi." "'Lo."
May 01 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 3:22 PM, H. S. Teoh via Digitalmars-d wrote:

 Exactly!!! This is why I said we should ban the use of structs with
 dtors as a field in a class. Isn't the D philosophy to be correct by
 default, and only allow potentially wrong/error-prone code if the user
 explicitly asks for it?  So this kind of error-prone usage should not be
 allowed by default.  Was my logic really that hard to follow?


Yes, we need to do something about it, no, I don't know what at the moment.
May 01 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:22 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 03:10:04PM -0700, Walter Bright via Digitalmars-d
wrote:

 On 5/1/2014 3:02 PM, H. S. Teoh via Digitalmars-d wrote:

 I'm confused. Andrei's original proposal was to deprecate class
 dtors, with the view to eventually getting rid of them altogether. If
 indeed that's going to be the case, then wouldn't that mean that
 field dtors won't be invoked either (because otherwise, how would you
 know when to invoke a field dtor?)?

 Without a clear definition of who is going to clean up those fields
 and invoke their dtors, it would seem to me that having fields with
 dtors in a class (once class dtors no longer exist, according to
 Andrei's proposal) will be a dangerous thing to do, since the very
 guarantee by having a dtor in the first place (it will get called to
 clean up when the struct goes out of scope) is broken. If so, what's
 the point of having a dtor in the first place? It becomes no
 different from the manual cleanup that you have to do in C.


 The thing is, GC is a terrible and unreliable method of managing
 non-memory resource lifetimes. Destructors for GC objects are not
 guaranteed to ever run. If you do have struct with a destructor as a
 field in a class, you've got, at minimum, suspicious code and a latent
 bug.


 Exactly!!! This is why I said we should ban the use of structs with
 dtors as a field in a class.


No.


 Isn't the D philosophy to be correct by
 default, and only allow potentially wrong/error-prone code if the user
 explicitly asks for it?


We can't propose correctness at the price of getting work done.


 So this kind of error-prone usage should not be
 allowed by default.  Was my logic really that hard to follow?


Your logic is flawed. I am sorry.


Andrei
May 01 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 1 May 2014 at 22:23:46 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 On Thu, May 01, 2014 at 03:10:04PM -0700, Walter Bright via 
 Digitalmars-d wrote:

 The thing is, GC is a terrible and unreliable method of 
 managing
 non-memory resource lifetimes. Destructors for GC objects are 
 not
 guaranteed to ever run. If you do have struct with a 
 destructor as a
 field in a class, you've got, at minimum, suspicious code and 
 a latent
 bug.


 Exactly!!! This is why I said we should ban the use of structs 
 with
 dtors as a field in a class.


No, not in a class, but in any GC-managed object. It's 
unfortunate that class currently implies GC.
May 02 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/2/14, 3:09 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Thursday, 1 May 2014 at 22:23:46 UTC, H. S. Teoh via Digitalmars-d
 wrote:

 On Thu, May 01, 2014 at 03:10:04PM -0700, Walter Bright via
 Digitalmars-d wrote:

 The thing is, GC is a terrible and unreliable method of managing
 non-memory resource lifetimes. Destructors for GC objects are not
 guaranteed to ever run. If you do have struct with a destructor as a
 field in a class, you've got, at minimum, suspicious code and a latent
 bug.


 Exactly!!! This is why I said we should ban the use of structs with
 dtors as a field in a class.


 No, not in a class, but in any GC-managed object. It's unfortunate that
 class currently implies GC.


So now it looks like dynamic arrays also can't contain structs with 
destructors :o). -- Andrei
May 02 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 2 May 2014 at 15:06:59 UTC, Andrei Alexandrescu wrote:

 On 5/2/14, 3:09 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Thursday, 1 May 2014 at 22:23:46 UTC, H. S. Teoh via 
 Digitalmars-d
 wrote:

 On Thu, May 01, 2014 at 03:10:04PM -0700, Walter Bright via
 Digitalmars-d wrote:

 The thing is, GC is a terrible and unreliable method of 
 managing
 non-memory resource lifetimes. Destructors for GC objects 
 are not
 guaranteed to ever run. If you do have struct with a 
 destructor as a
 field in a class, you've got, at minimum, suspicious code 
 and a latent
 bug.


 Exactly!!! This is why I said we should ban the use of 
 structs with
 dtors as a field in a class.


 No, not in a class, but in any GC-managed object. It's 
 unfortunate that
 class currently implies GC.


 So now it looks like dynamic arrays also can't contain structs 
 with destructors :o). -- Andrei


Well, that would be the logical consequence...

But my main point was actually this:
Don't disallow destructors on classes just because they are 
classes, but disallow them when they are not guaranteed to be 
called (or calling them is troublesome because of 
multi-threading), i.e. GC.
May 02 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-05-02 17:38, "Marc Schütz" <schuetzm gmx.net>" wrote:


 But my main point was actually this:
 Don't disallow destructors on classes just because they are classes, but
 disallow them when they are not guaranteed to be called (or calling them
 is troublesome because of multi-threading), i.e. GC.


Tango for D1 added a "dispose" method to Object. This method was called 
when "delete" or "scope" was used.

-- 
/Jacob Carlborg
May 02 2014
prev sibling next sibling parent Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5/2/14, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 So now it looks like dynamic arrays also can't contain structs with
 destructors :o). -- Andrei


I suggest tracking these ideas on a wiki page so we don't lose track
of what the latest proposal is (we'll run in circles otherwise).

These threads tend to literally explode with posts from everyone. :)
May 02 2014
prev sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 2 May 2014 at 15:06:59 UTC, Andrei Alexandrescu wrote:

 So now it looks like dynamic arrays also can't contain structs 
 with destructors :o). -- Andrei


Well, that's always been the case, and even worst, since in a 
dynamic array, destructor are guaranteed to *never* be run. 
Furthermore, given the "append causes relocation which 
duplicates", you are almost *guaranteed* to leak your 
destructors. You just can't keep track of the usage of a "naked" 
dynamic array.

This usually comes as a great surprise to users in ".learn". It's 
also the reason why using "File[]" never ends well...
May 02 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, May 02, 2014 at 09:03:15PM +0000, monarch_dodra via Digitalmars-d wrote:

 On Friday, 2 May 2014 at 15:06:59 UTC, Andrei Alexandrescu wrote:

So now it looks like dynamic arrays also can't contain structs with
destructors :o). -- Andrei


 
 Well, that's always been the case, and even worst, since in a dynamic
 array, destructor are guaranteed to *never* be run. Furthermore, given
 the "append causes relocation which duplicates", you are almost
 *guaranteed* to leak your destructors. You just can't keep track of
 the usage of a "naked" dynamic array.
 
 This usually comes as a great surprise to users in ".learn". It's also
 the reason why using "File[]" never ends well...


This is why I'm unhappy with the way this is going. Current behaviour of
structs with dtors is already fragile enough, now we're pulling the rug
out from under classes as well. So that will be yet another case where
dtors won't work as expected.

I'm getting the feeling that dtors were bolted on as an afterthought,
and only works properly for a very narrow spectrum of use cases. Rather
than expand the usable cases, we're proposing to reduce them (by getting
rid of class dtors). I can't see *this* ending well either. :-(


T

-- 
Democracy: The triumph of popularity over principle. -- C.Bond
May 02 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 2 May 2014 at 21:12:12 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 On Fri, May 02, 2014 at 09:03:15PM +0000, monarch_dodra via 
 Digitalmars-d wrote:

 On Friday, 2 May 2014 at 15:06:59 UTC, Andrei Alexandrescu 
 wrote:

So now it looks like dynamic arrays also can't contain 
structs with
destructors :o). -- Andrei


 
 Well, that's always been the case, and even worst, since in a 
 dynamic
 array, destructor are guaranteed to *never* be run. 
 Furthermore, given
 the "append causes relocation which duplicates", you are almost
 *guaranteed* to leak your destructors. You just can't keep 
 track of
 the usage of a "naked" dynamic array.
 
 This usually comes as a great surprise to users in ".learn". 
 It's also
 the reason why using "File[]" never ends well...


 This is why I'm unhappy with the way this is going. Current 
 behaviour of
 structs with dtors is already fragile enough, now we're pulling 
 the rug
 out from under classes as well. So that will be yet another 
 case where
 dtors won't work as expected.

 I'm getting the feeling that dtors were bolted on as an 
 afterthought,
 and only works properly for a very narrow spectrum of use 
 cases. Rather
 than expand the usable cases, we're proposing to reduce them 
 (by getting
 rid of class dtors). I can't see *this* ending well either. :-(


Unfortunately, the rug they are standing is placed over a gaping 
hole in the ground, metaphorically speaking, so it never was a 
very stable situation to begin with. I'm not sure it makes sense 
to put much effort into increasing the likelihood that 
destructors are called, when we then _still_ can't guarantee it.

Better to acknowledge that it simply can't work (if that's the 
case), and try to find a way to avoid these situations. For 
example, we could statically forbid GC managed objects with 
destructors (not classes, but _anything_ GC managed). That means 
that calling new for these types, or allocating a dynamic array 
of them, will be an error. Of course, an escape hatch needs to be 
put in place. "isolated" might be a candidate.
May 04 2014
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 02 May 2014 21:03:15 +0000
monarch_dodra via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On Friday, 2 May 2014 at 15:06:59 UTC, Andrei Alexandrescu wrote:

 So now it looks like dynamic arrays also can't contain structs
 with destructors :o). -- Andrei


 Well, that's always been the case, and even worst, since in a
 dynamic array, destructor are guaranteed to *never* be run.
 Furthermore, given the "append causes relocation which
 duplicates", you are almost *guaranteed* to leak your
 destructors. You just can't keep track of the usage of a "naked"
 dynamic array.

 This usually comes as a great surprise to users in ".learn". It's
 also the reason why using "File[]" never ends well...


Heck, I probably knew that before, but I had completely forgotten. If you'd
asked me yesterday whether struct destructors were run in dynamic arrays,
I'd have said yes. And if someone like me doesn't remember that, would you
expect the average D programmer to? The current situation is just plain
bug-prone.

Honestly, I really think that we need to figure out how to make it so that
struct destructors are guaranteed to be run so long as the memory that
they're in is collected. Without that, having destructors in structs
anywhere other than directly on the stack is pretty much broken.

- Jonathan M Davis
May 02 2014
parent reply "Nick B" <nick.barbalich gmail.com> writes:
[ monarch_dodra wrote ]  Well, that's always been the case, and 
even worst, since in a
dynamic array, destructor are guaranteed to *never* be run.



https://issues.dlang.org/show_bug.cgi?id=2757


Resource Management.  A issue that has been discussed since 2009, 
and still no *GOOD* solution.


Look at these arguements made back then.


email 23 Mar 2009 from the D.d list. Subject : "Re: new D2.0 + 
C++ language".

Sat, 21 Mar 2009 20:16:07 -0600, Rainer Deyke wrote:


 Sergey Gromov wrote:

 I think this is an overstatement.  It's only abstract write 
 buffers
 where GC really doesn't work, like std.stream.BufferedFile. 
  In any
 other resource management case I can think of GC works fine.




 
 OpenGL objects (textures/shader programs/display lists).
 SDL surfaces.
 Hardware sound buffers.
 Mutex locks.
 File handles.
 Any object with a non-trivial destructor.
 Any object that contains or manages one of the above.
 
 Many of the above need to be released in a timely manner. For 
 example,
 it is a serious error to free a SDL surface after closing the 
 SDL video
 subsystem, and closing the SDL video subsystem is the only 
 way to close
 the application window under SDL.  Non-deterministic garbage 
 collection
 cannot work.
 
 Others don't strictly need to be released immediately after 
 use, but
 should still be released as soon as reasonably possible to 
 prevent
 resource hogging.  The GC triggers when the program is low on 
 system
 memory, not when the program is low on texture memory.
 
 By my estimate, in my current project (rewritten in C++ after 
 abandoning
 D due to its poor resource management), about half of the 
 classes manage
 resources (directly or indirectly) that need to be released 
 in a timely
 manner.  The other 50% does not need RAII, but also wouldn't 
 benefit
 from GC in any area other than performance.





The language set up the defaults when these are to run.  The 
programmer has to override the defaults. [Sure this crude, but it 
deterministic]

[comment by dsimcha inm 2009 ] Come to think of it, as simple and 
kludgey sounding as it is, this is an incredibly good idea if you 
have an app that does a lot of sitting around waiting for input, 
etc. and therefore not allocating memory and you want an easy way 
to make sure it releases resources in a reasonable amount of 
time.  This belongs in
an FAQ somewhere.
May 02 2014
parent "Nick B" <nick.barbalich gmail.com> writes:
I forgot to add these comments by walter at the top of my 
previous post:



 [walter Bright wrote ] the thing is, GC is a terrible and 
 unreliable method of managing non-memory


resource lifetimes. Destructors for GC objects are not guaranteed 
to ever run.



 So now it looks like dynamic arrays also can't contain structs 
 with destructors :o). -- Andrei


Nick
May 02 2014
prev sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, May 02, 2014 at 11:44:47PM +0200, Jonathan M Davis via Digitalmars-d
wrote:

 On Fri, 02 May 2014 21:03:15 +0000
 monarch_dodra via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 

 On Friday, 2 May 2014 at 15:06:59 UTC, Andrei Alexandrescu wrote:

 So now it looks like dynamic arrays also can't contain structs
 with destructors :o). -- Andrei


 Well, that's always been the case, and even worst, since in a
 dynamic array, destructor are guaranteed to *never* be run.
 Furthermore, given the "append causes relocation which duplicates",
 you are almost *guaranteed* to leak your destructors. You just can't
 keep track of the usage of a "naked" dynamic array.

 This usually comes as a great surprise to users in ".learn". It's
 also the reason why using "File[]" never ends well...


 
 Heck, I probably knew that before, but I had completely forgotten. If
 you'd asked me yesterday whether struct destructors were run in
 dynamic arrays, I'd have said yes. And if someone like me doesn't
 remember that, would you expect the average D programmer to? The
 current situation is just plain bug-prone.
 
 Honestly, I really think that we need to figure out how to make it so
 that struct destructors are guaranteed to be run so long as the memory
 that they're in is collected. Without that, having destructors in
 structs anywhere other than directly on the stack is pretty much
 broken.


[...]

Thank you, that's what I've been trying to say. Having dtors sometimes
run and sometimes not, is very bug-prone -- if for a seasoned D
programmer, then how much more for an average D programmer? We need some
kind of guarantees. The current situation sux. I might even say we'd
have been better off having no dtors in the first place -- at least then
it's consistent, you know you always have to cleanup. But the current
situation of being neither here nor there, neither always cleaning up
nor never, is not a good place to be in.

I've to say that the more I look at this, the more I don't like this
part of the language. :-/


T

-- 
MASM = Mana Ada Sistem, Man!
May 02 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:10 PM, Walter Bright wrote:

 On 5/1/2014 3:02 PM, H. S. Teoh via Digitalmars-d wrote:

 I'm confused. Andrei's original proposal was to deprecate class dtors,
 with the view to eventually getting rid of them altogether. If indeed
 that's going to be the case, then wouldn't that mean that field dtors
 won't be invoked either (because otherwise, how would you know when to
 invoke a field dtor?)?

 Without a clear definition of who is going to clean up those fields and
 invoke their dtors, it would seem to me that having fields with dtors in
 a class (once class dtors no longer exist, according to Andrei's
 proposal) will be a dangerous thing to do, since the very guarantee by
 having a dtor in the first place (it will get called to clean up when
 the struct goes out of scope) is broken. If so, what's the point of
 having a dtor in the first place? It becomes no different from the
 manual cleanup that you have to do in C.


 The thing is, GC is a terrible and unreliable method of managing
 non-memory resource lifetimes. Destructors for GC objects are not
 guaranteed to ever run.


Agree.


 If you do have struct with a destructor as a
 field in a class, you've got, at minimum, suspicious code and a latent bug.


Disagree. The simple matter here is that there's no "right" solution to 
counter what you deem as suspicious code.

For example, we have a File struct. It is entirely legitimate for 
someone to want to keep a File member variable in a class object. They 
take care of opening and closing it appropriately. Again, this is 100% 
legit code to want to write. Coming to them from the perspective "you 
are doing it wrong and there is no recourse for achieving what you need 
to do" strikes me as odd.


Andrei
May 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 5:43 PM, Andrei Alexandrescu wrote:

 For example, we have a File struct. It is entirely legitimate for someone to
 want to keep a File member variable in a class object. They take care of
opening
 and closing it appropriately. Again, this is 100% legit code to want to write.
 Coming to them from the perspective "you are doing it wrong and there is no
 recourse for achieving what you need to do" strikes me as odd.


I meant from the perspective of expecting the GC to close the file for you. If 
you manually run the destructor, such as from using delete, that wouldn't be a 
problem.
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 6:06 PM, Walter Bright wrote:

 On 5/1/2014 5:43 PM, Andrei Alexandrescu wrote:

 For example, we have a File struct. It is entirely legitimate for
 someone to
 want to keep a File member variable in a class object. They take care
 of opening
 and closing it appropriately. Again, this is 100% legit code to want
 to write.
 Coming to them from the perspective "you are doing it wrong and there
 is no
 recourse for achieving what you need to do" strikes me as odd.


 I meant from the perspective of expecting the GC to close the file for
 you. If you manually run the destructor, such as from using delete, that
 wouldn't be a problem.


Or if you... close the file :o). OK, thanks for clarifying. -- Andrei
May 01 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 6:19 PM, Andrei Alexandrescu wrote:

 Or if you... close the file


Hmm. Never thought of that!
May 01 2014
prev sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 30 April 2014 at 23:19:18 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 On Wed, Apr 30, 2014 at 03:55:38PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote:


 [...]

I don't like the sound of that. I haven't found myself in a 
place
where I needed to do something like this, but if I had to, 
I'd be
very unhappy if struct dtors only work when they're not class
members. Can we make them always work, and if necessary 
prohibit
using them as class members?


 
 Then we're back to effectively class destructors. I think we've
 gathered quite a bit of evidence there are pernicious issues
 associated with them. -- Andrei


 [...]

 How so? If we prohibit structs with dtors from being class 
 members,
 then it could work.


Why would we prohibit structs with destructors from being class 
members? That don't make sense to me.

"A class is allowed to have a destructor" <=> "A class can have 
members with destructors".

So we either kill off class destructor entirelly (which would 
mean no members with destructors) (But that seems like a bad 
idea), or we keep both.

Or did I miss something in the argument?
May 01 2014
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, May 01, 2014 at 10:06:17AM +0000, monarch_dodra via Digitalmars-d wrote:

 On Wednesday, 30 April 2014 at 23:19:18 UTC, H. S. Teoh via Digitalmars-d
 wrote:

On Wed, Apr 30, 2014 at 03:55:38PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:

On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote:


[...]

I don't like the sound of that. I haven't found myself in a >place
where I needed to do something like this, but if I had to, >I'd be
very unhappy if struct dtors only work when they're not class
members. Can we make them always work, and if necessary >prohibit
using them as class members?


Then we're back to effectively class destructors. I think we've
gathered quite a bit of evidence there are pernicious issues
associated with them. -- Andrei


[...]

How so? If we prohibit structs with dtors from being class members,
then it could work.


 
 Why would we prohibit structs with destructors from being class
 members?  That don't make sense to me.


The proposal was to get rid of class dtors at some point. Which will
introduce the problem of what to do when a class member is a struct with
a dtor, since that dtor will never run.



 "A class is allowed to have a destructor" <=> "A class can have
 members with destructors".
 
 So we either kill off class destructor entirelly (which would mean no
 members with destructors) (But that seems like a bad idea), or we keep
 both.
 
 Or did I miss something in the argument?


Yes that's the gist of it. But Andrei seems to be convinced that class
dtors are a bad idea, so to me, that also implies that class members
with dtors are an equally bad (or worse) idea, so therefore they must be
gotten rid of too.


T

-- 
All problems are easy in retrospect.
May 01 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:06 AM, monarch_dodra wrote:

 "A class is allowed to have a destructor" <=> "A class can have members
 with destructors".


No equivalence, but implication. If a class has at least one member with 
a destructor, the compiler might need to generate a destructor for the 
class. -- Andrei
May 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:

 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for the class.


And in fact that's what dmd does.
May 03 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/3/14, 12:40 PM, Walter Bright wrote:

 On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:

 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for the
 class.


 And in fact that's what dmd does.


Which suggests a simple solution for calling destructors for structs and 
arrays:

* Lower new for structs to return:

new S;
->
return &(new class { S member; }).member;

* Lower array construction similarly.

Then voila, the anonymous classes will destroy structs and arrays 
appropriately.


Andrei
May 03 2014
next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 5/3/2014 6:44 PM, Andrei Alexandrescu wrote:

 On 5/3/14, 12:40 PM, Walter Bright wrote:

 On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:

 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for the
 class.


 And in fact that's what dmd does.


 Which suggests a simple solution for calling destructors for structs and
 arrays:

 * Lower new for structs to return:

 new S;
 ->
 return &(new class { S member; }).member;

 * Lower array construction similarly.

 Then voila, the anonymous classes will destroy structs and arrays
 appropriately.


Uhh, but doesn't this completely break as soon as class dtors go away?
May 03 2014
parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, 03 May 2014 22:44:39 -0400
Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On 5/3/2014 6:44 PM, Andrei Alexandrescu wrote:

 On 5/3/14, 12:40 PM, Walter Bright wrote:

 On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:

 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for
 the class.


 And in fact that's what dmd does.


 Which suggests a simple solution for calling destructors for
 structs and arrays:

 * Lower new for structs to return:

 new S;
 ->
 return &(new class { S member; }).member;

 * Lower array construction similarly.

 Then voila, the anonymous classes will destroy structs and arrays
 appropriately.


 Uhh, but doesn't this completely break as soon as class dtors go away?


Based on other comments, I think that Andrei has been convinced that class
destructors can't go away at this point and that there certainly isn't any
consensus that it would even be desirable for them to go away (though what
he'd choose to do if we could break code willy-nilly, I don't know). So,
this particular proposal is presumably done with the idea that class
destructors are here to stay. Rather, it's trying to make it so that the
destructors for structs on the heap get run unlike now - which is a much
better direction to try and go IMHO.

- Jonathan M Davis
May 03 2014
prev sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, 03 May 2014 15:44:03 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 On 5/3/14, 12:40 PM, Walter Bright wrote:

 On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:

 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for
 the class.


 And in fact that's what dmd does.


 Which suggests a simple solution for calling destructors for structs
 and arrays:

 * Lower new for structs to return:

 new S;
 ->
 return &(new class { S member; }).member;

 * Lower array construction similarly.

 Then voila, the anonymous classes will destroy structs and arrays
 appropriately.


This might be a good approach, though I confess that it strikes me as rather
weird to wrap structs in classes like that. It also leaves open the question
of how to deal with structs that are newed directly rather than put in an
array. _Those_ really need to be destroyed properly as well. And unless
you're suggesting that S* effectively become a pointer to a struct within
an anonymous class in all cases, not only would this not work for structs
which were newed up directly on the heap, but I'd be worried about what would
happen when someone did something like &arr[5] with an array of structs. Would
they get a pointer to a class or a struct?

It was my understanding that some of what Rainer Schutze did with his precise
GC involved adding RTInfo which could make it possible to run struct
destructors for structs that were newed up directly on the heap. If that is
indeed the case, then I would think that we could use that for arrays of
structs as well.

- Jonathan M Davis
May 03 2014
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 Apr 2014 16:21:33 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 Walter and I have had a long chat in which we figured our current  
 offering of abstractions could be improved. Here are some thoughts.  
 There's a lot of work ahead of us on that and I wanted to make sure  
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within  
 the GC entirely. It makes for a faster and better GC, but the real  
 reason here is that destructors are philosophically bankrupt in a GC  
 environment. I think there's no need to argue that in this community.  
 The GC never guarantees calling destructors even today, so this decision  
 would be just a point in the definition space (albeit an extreme one).


destructors are for cleaning up non-GC resources. File handles, malloc'd  
memory, etc. I don't see why these need to be eliminated.

What about changing destructors so that they DON'T implicitly call member  
struct dtors? I'd prefer that, since struct dtors executed on the heap I  
agree are an issue. One can always call the dtor in the class dtor if  
necessary. Right now, you have no choice.


 That means classes that need cleanup (either directly or by having  
 fields that are structs with destructors) would need to garner that by  
 other means, such as reference counting or manual. We're considering  
 deprecating ~this() for classes in the future.


So essentially, any class with a dtor needs reference counting? How does  
one clean up a cycle of them?


 Slices T[] of structs with destructors shall be silently lowered into  
 RCSlice!T, defined inside object.d. That type would occupy THREE words,  
 one of which being a pointer to a reference count. That type would  
 redefine all slice primitives to update the reference count accordingly.


This is a possibility, but I think you would need to prove performance is  
not severely affected.


 I foresee any number of theoretical and practical issues with this  
 approach. Let's discuss some of them here.


I'd rather see a RC array type, which specifically supports reference  
counting for opt-in support.

But we are getting WAY ahead of ourselves. A workable RC solution has to  
be proven first. I'd feel much more comfortable making such a change with  
a working, tested solution to look at.

-Steve
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:

 destructors are for cleaning up non-GC resources. File handles, malloc'd
 memory, etc. I don't see why these need to be eliminated.


Virtually all GCs are known to be horrible at managing scarce resources 
(including memory itself). Giving the task of managing such to the GC is 
like making the cross-eyed guy be the shooter in 
https://www.youtube.com/watch?v=sen8Tn8CBA4.

Andrei
Apr 30 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 6:51 PM, Andrei Alexandrescu wrote:

 On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:

 destructors are for cleaning up non-GC resources. File handles, malloc'd
 memory, etc. I don't see why these need to be eliminated.


 Virtually all GCs are known to be horrible at managing scarce resources
 (including memory itself).


Here the paren means "... when memory is scarce". GCs work well only in 
3x+ the memory needed. -- Andrei
Apr 30 2014
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 Apr 2014 21:51:38 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:

 destructors are for cleaning up non-GC resources. File handles, malloc'd
 memory, etc. I don't see why these need to be eliminated.


 Virtually all GCs are known to be horrible at managing scarce resources  
 (including memory itself).


The destructor can be a crutch, but it's not good to leave open resources  
when the user of your code has not cleaned them up manually.

I can see no reason to disallow destruction from cleaning up resources  
that nobody else has managed to clean up.

The largest problem with D destructors comes from trying to clean up D  
objects in destructors of structs, that you never expected to be done via  
the GC.

Cleaning up files and malloc'd memory is not an issue.

-Steve
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 7:20 PM, Steven Schveighoffer wrote:

 On Wed, 30 Apr 2014 21:51:38 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:

 destructors are for cleaning up non-GC resources. File handles, malloc'd
 memory, etc. I don't see why these need to be eliminated.


 Virtually all GCs are known to be horrible at managing scarce
 resources (including memory itself).


 The destructor can be a crutch, but it's not good to leave open
 resources when the user of your code has not cleaned them up manually.


Sounds like defensive programming to me.


 I can see no reason to disallow destruction from cleaning up resources
 that nobody else has managed to clean up.


Make memory management faster and better for everyone. I'm not saying 
it's reason enough, but it's a reason.


 The largest problem with D destructors comes from trying to clean up D
 objects in destructors of structs, that you never expected to be done
 via the GC.

 Cleaning up files and malloc'd memory is not an issue.


Not getting this...


Andrei
Apr 30 2014
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 00:07:42 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 4/30/14, 7:20 PM, Steven Schveighoffer wrote:

 On Wed, 30 Apr 2014 21:51:38 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:

 destructors are for cleaning up non-GC resources. File handles,  
 malloc'd
 memory, etc. I don't see why these need to be eliminated.


 Virtually all GCs are known to be horrible at managing scarce
 resources (including memory itself).


 The destructor can be a crutch, but it's not good to leave open
 resources when the user of your code has not cleaned them up manually.


 Sounds like defensive programming to me.


Not at all. I've written programs that run for weeks at a time without  
ever crashing, who continually manage such resources via the GC.

If the GC cleans up your object, but your object wasn't finalized, you  
would have to make that a hard error. i.e. crash.

Still not talked about that I have seen, is the fact that with RC, we need  
a solution for cycles. If that's the GC then the GC WILL be cleaning up  
objects, even if they are ref counted.


 I can see no reason to disallow destruction from cleaning up resources
 that nobody else has managed to clean up.


 Make memory management faster and better for everyone. I'm not saying  
 it's reason enough, but it's a reason.


It doesn't make it better. The GC still is cleaning up the memory. Having  
it call close(fd) doesn't delay or make worse anything.


 The largest problem with D destructors comes from trying to clean up D
 objects in destructors of structs, that you never expected to be done
 via the GC.

 Cleaning up files and malloc'd memory is not an issue.


 Not getting this...


class C
{
    private int fd = -1;
    private ubyte[] buffer;
    this(string fname) {
       fd = open(fname, O_CREAT);
       buffer = (cast(ubyte*).malloc(100))[0..100];
    }

    ~this() {
       if(fd != -1) { close(fd); fd = -1;}
       if(buffer.ptr) { .free(buffer.ptr); buffer = null}
    }
}

There is no problem with this code. It works correctly in all cases. It's  
not inefficient. It's not cumbersome or awkward or confusing. It's exactly  
what class destructors are for.

However, this *is* a problem:

class C
{
    RCValue s;
}

I have no way to say how s will be destructed, unless I poison it somehow.  
Even still, s.~this() will be called when C.~this() is done. This can be  
run in a separate thread as another object that is calling s.~this() at  
the same time.

Removing ~this() for classes just ADDS complication that we don't need.  
You are trying to fix the problem for the second situation by introducing  
a problem for the first that is unnecessary.

Solutions for the second problem are not predicated on jettisoning class  
dtors. You can, for instance, try and run the dtor for the instance of C  
in the same thread where it is owned. You could, as you say, force C into  
a ref counted mechanism. But this doesn't mean you should prevent C from  
cleaning up its resources!

Please understand that I'm not arguing for the status quo. What I'm  
arguing is that class destructors DO have some valid uses, and removing  
them would introduce new problems. I agree we need to find a solution to  
the ref counting + GC problem.

-Steve
May 01 2014
prev sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 1 May 2014 at 01:04:08 UTC, Steven Schveighoffer 
wrote:

 That means classes that need cleanup (either directly or by 
 having fields that are structs with destructors) would need to 
 garner that by other means, such as reference counting or 
 manual. We're considering deprecating ~this() for classes in 
 the future.


 So essentially, any class with a dtor needs reference counting? 
 How does one clean up a cycle of them?


Yeah, what he said. This has me very worried. Making cycles is 
actually incredibly easy. Any "inteligently" implemented 
"node-based" data structure, implemented with classes, such as a 
linked list, a tree or a graph, is virtually guaranteed to have a 
cycle somewhere...

*Or*

is the idea that reference counting only works for finalization, 
but the memory is still managed by the GC? In that case, the 
destructors would leak, but the memory still be released?

That's still bad, of course, but not "as" bad...
May 01 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:10 AM, monarch_dodra wrote:

 On Thursday, 1 May 2014 at 01:04:08 UTC, Steven Schveighoffer wrote:

 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that
 by other means, such as reference counting or manual. We're
 considering deprecating ~this() for classes in the future.


 So essentially, any class with a dtor needs reference counting? How
 does one clean up a cycle of them?


 Yeah, what he said. This has me very worried. Making cycles is actually
 incredibly easy. Any "inteligently" implemented "node-based" data
 structure, implemented with classes, such as a linked list, a tree or a
 graph, is virtually guaranteed to have a cycle somewhere...


In my experience classes with destructors are simpler (files, 
sockets)... Lists, trees, and graphs tend to be in-memory objects. But 
of course this is just speculation.


 *Or*

 is the idea that reference counting only works for finalization, but the
 memory is still managed by the GC? In that case, the destructors would
 leak, but the memory still be released?

 That's still bad, of course, but not "as" bad...


Yah, that would be the case. Essentially we're talking about making the 
entire GC heap passive.


Andrei
May 01 2014
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:

 That means classes that need cleanup (either directly or by 
 having fields that are structs with destructors) would need to 
 garner that by other means, such as reference counting or 
 manual. We're considering deprecating ~this() for classes in 
 the future.


I can't claim it should be done, but any time I've thought 
something should go in the destructor I was wrong and would end 
up with a close() or similar function.
Apr 30 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:


 We're considering deprecating ~this() for classes in the future.


Such changes could happen in parallel to the (planned?) removal 
of some of the methods of Object.

Bye,
bearophile
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 13:21:33 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 Walter and I have had a long chat in which we figured our current 
 offering of abstractions could be improved. Here are some thoughts. 
 There's a lot of work ahead of us on that and I wanted to make sure 
 we're getting full community buy-in and backup.
 
 First off, we're considering eliminating destructor calls from within 
 the GC entirely. It makes for a faster and better GC, but the real 
 reason here is that destructors are philosophically bankrupt in a GC 
 environment. I think there's no need to argue that in this community. 
 The GC never guarantees calling destructors even today, so this
 decision would be just a point in the definition space (albeit an
 extreme one).


I really don't like the fact that struct destructors are not called by
the GC, and if anything, I'd be inclined to argue for finding a way to
guarantee that they get run rather than guaranteeing that they never
get run. It's just far too easy to have a struct expect that its
destructor will be run and then have issues when it's not run. But it
would be better to define struct destructors as never getting run rather
than having it be undefined as it is now.


 We're considering deprecating ~this() for classes in the future.


While it's not good to rely on finalizers, they're good to have as
backup if the appropriate cleanup function doesn't get called like it's
supposed to. They're not as critical as they'd be in Java, since we
have structs, but I'd be disinclined to remove finalizers from D
without a really good reason.


 Also, we're considering a revamp of built-in slices, as follows.
 Slices of types without destructors stay as they are.
 
 Slices T[] of structs with destructors shall be silently lowered into 
 RCSlice!T, defined inside object.d. That type would occupy THREE
 words, one of which being a pointer to a reference count. That type
 would redefine all slice primitives to update the reference count
 accordingly.
 
 RCSlice!T will not convert implicitly to void[]. Explicit
 cast(void[]) will be allowed, and will ignore the reference count (so
 if a void[] extracted from a T[] via a cast outlives all slices,
 dangling pointers will ensue).
 
 I foresee any number of theoretical and practical issues with this 
 approach. Let's discuss some of them here.


I'm really going to have to think about this one. It's such a radical
change that I really don't know what to think about it. It will be
interesting to see what others have to say about.

- Jonathan M Davis
May 01 2014
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:

 First off, we're considering eliminating destructor calls from 
 within the GC entirely. It makes for a faster and better GC, 
 but the real reason here is that destructors are 
 philosophically bankrupt in a GC environment. I think there's 
 no need to argue that in this community. The GC never 
 guarantees calling destructors even today, so this decision 
 would be just a point in the definition space (albeit an 
 extreme one).

 That means classes that need cleanup (either directly or by 
 having fields that are structs with destructors) would need to 
 garner that by other means, such as reference counting or 
 manual. We're considering deprecating ~this() for classes in 
 the future.


I'm all for it, this will break code that was not correct in the 
first place, and was working by accident.

Also, being called by the GC from any thread prevent many kinds 
of cleanup, such as those with a thread-wise bound context (CUDA, 
OpenGL, etc...).
May 01 2014
prev sibling next sibling parent reply "w0rp" <devw0rp gmail.com> writes:
Removing class destructors would break my DQt library as it 
currently stands, which wraps C++ classes with D classes, Qt uses 
classes for polymorphism, for protected overrides and slots. I 
would have to consider some other means of wrapping C++. It might 
also be confusing for a few people.

So, having said that... I am completely in favour of removing 
class destructors. It's a simple principle. When you have to do 
less work, things can run faster. So I can see removing class 
destructors as a feature leading to reduced pause times. 
Destructors exist only for resource management. Doing resource 
management through GC... is a bad idea.
May 01 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 5:04 AM, w0rp wrote:

 Removing class destructors would break my DQt library as it currently
 stands, which wraps C++ classes with D classes, Qt uses classes for
 polymorphism, for protected overrides and slots. I would have to
 consider some other means of wrapping C++. It might also be confusing
 for a few people.

 So, having said that... I am completely in favour of removing class
 destructors. It's a simple principle. When you have to do less work,
 things can run faster. So I can see removing class destructors as a
 feature leading to reduced pause times. Destructors exist only for
 resource management. Doing resource management through GC... is a bad idea.


I've decided what I'll do. I'll implement both and let the user choose. 
Push policy up, implementation down! -- Andrei
May 01 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, May 01, 2014 at 01:37:28AM -0700, Jonathan M Davis via Digitalmars-d
wrote:

 On Wed, 30 Apr 2014 13:21:33 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:
 


[...]

 First off, we're considering eliminating destructor calls from
 within the GC entirely. It makes for a faster and better GC, but the
 real reason here is that destructors are philosophically bankrupt in
 a GC environment. I think there's no need to argue that in this
 community.  The GC never guarantees calling destructors even today,
 so this decision would be just a point in the definition space
 (albeit an extreme one).


 
 I really don't like the fact that struct destructors are not called by
 the GC, and if anything, I'd be inclined to argue for finding a way to
 guarantee that they get run rather than guaranteeing that they never
 get run. It's just far too easy to have a struct expect that its
 destructor will be run and then have issues when it's not run. But it
 would be better to define struct destructors as never getting run
 rather than having it be undefined as it is now.


+1.



 We're considering deprecating ~this() for classes in the future.


 
 While it's not good to rely on finalizers, they're good to have as
 backup if the appropriate cleanup function doesn't get called like
 it's supposed to. They're not as critical as they'd be in Java, since
 we have structs, but I'd be disinclined to remove finalizers from D
 without a really good reason.


[...]

I'd like to hear an enumeration of those reasons as well.

While in principle I agree with the sentiment to get rid of class dtors,
I'm concerned about the rippling side-effects this will have throughout
the rest of the language. Such as class members that are structs with
dtors, which will mean that the struct dtors will never get called.

Using another poster's argument: if dtors are used for resource
management, then it's a bad idea to mix dtors with GC (i.e. classes). So
in this sense I agree with getting rid of class dtors. But we have to
consider what happens to the case where you stick a struct with a dtor
into a class. I'm inclined to say that we should outright prohibit that,
because again, the dtor (struct dtor this time) is used for resource
management, and therefore shouldn't be mixed with a GC'd resource (i.e.,
classes). I find it unacceptable that the compiler will silently accept
such a usage, and yet have no semantic guarantees (that dtor will run).
This is the kind of behaviour I expect from C/C++, but not from D.  Next
thing you know, somebody is gonna start wrapping structs inside classes
as a way of suppressing the dtor. I don't think that's the kind of thing
the language should allow. I contend that it will do the language a
great deal of good to outright ban such a kind of usage.

(Structs with no dtors, OTOH, can be safely used as class members,
obviously.)


T

-- 
It only takes one twig to burn down a forest.
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 7:30 AM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 01:37:28AM -0700, Jonathan M Davis via Digitalmars-d
wrote:

 While it's not good to rely on finalizers, they're good to have as
 backup if the appropriate cleanup function doesn't get called like
 it's supposed to. They're not as critical as they'd be in Java, since
 we have structs, but I'd be disinclined to remove finalizers from D
 without a really good reason.


 [...]

 I'd like to hear an enumeration of those reasons as well.


1. Most scarce resources must be released eagerly. GC collections occur 
relatively rarely and are triggered by different signals (low on memory).

2. The notion of running the GC when one runs out of file handles or 
sockets is terribly inefficient.

3. Destructors are odd - they will run in a different thread than the 
one that created the object. They also are limited in surprising ways, 
i.e. may not allocate memory or block on other threads directly or 
indirectly.

4. Due to imprecision, there's no actual guarantee any given destructor 
will end up running. Leaving a scarce resources at the whim of a 
best-effort approach is poor design.


 While in principle I agree with the sentiment to get rid of class dtors,
 I'm concerned about the rippling side-effects this will have throughout
 the rest of the language. Such as class members that are structs with
 dtors, which will mean that the struct dtors will never get called.

 Using another poster's argument: if dtors are used for resource
 management, then it's a bad idea to mix dtors with GC (i.e. classes). So
 in this sense I agree with getting rid of class dtors. But we have to
 consider what happens to the case where you stick a struct with a dtor
 into a class. I'm inclined to say that we should outright prohibit that,


That can't happen.


Andrei
May 01 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 1 May 2014 at 15:28:25 UTC, Andrei Alexandrescu 
wrote:

 On 5/1/14, 7:30 AM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 01:37:28AM -0700, Jonathan M Davis via 
 Digitalmars-d wrote:

 While it's not good to rely on finalizers, they're good to 
 have as
 backup if the appropriate cleanup function doesn't get called 
 like
 it's supposed to. They're not as critical as they'd be in 
 Java, since
 we have structs, but I'd be disinclined to remove finalizers 
 from D
 without a really good reason.


 [...]

 I'd like to hear an enumeration of those reasons as well.


 1. Most scarce resources must be released eagerly. GC 
 collections occur relatively rarely and are triggered by 
 different signals (low on memory).

 2. The notion of running the GC when one runs out of file 
 handles or sockets is terribly inefficient.

 3. Destructors are odd - they will run in a different thread 
 than the one that created the object. They also are limited in 
 surprising ways, i.e. may not allocate memory or block on other 
 threads directly or indirectly.

 4. Due to imprecision, there's no actual guarantee any given 
 destructor will end up running. Leaving a scarce resources at 
 the whim of a best-effort approach is poor design.


These are good arguments against destructors for GC managed 
objects, IMO.

But conceptually, this is _not_ the same as classes! As others 
have mentioned, it's possible to created structs with `new`, or 
have them in dynamic arrays, as well as managing class objects 
manually.

Maybe the language should have some way to distinguish between 
GC-managed and manually-managed objects, preferably in the type 
system. Then it could be statically checked whether an object is 
supposed to be GC-managed, and consequentially shouldn't have a 
destructor.

The difference between classes and structs should then be 
reference vs. value semantics, and polymorphic vs static (which 
correlate nicely). It should, however, not imply whether the 
object is managed by the GC or not. Some kind of ownership 
mechanism would be more suited for that.
May 01 2014
next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.05.2014 19:35, schrieb "Marc Schütz" <schuetzm gmx.net>":

 But conceptually, this is _not_ the same as classes! As others have
 mentioned, it's possible to created structs with `new`, or have them in
 dynamic arrays, as well as managing class objects manually.

 Maybe the language should have some way to distinguish between
 GC-managed and manually-managed objects, preferably in the type system.
 Then it could be statically checked whether an object is supposed to be
 GC-managed, and consequentially shouldn't have a destructor.

 The difference between classes and structs should then be reference vs.
 value semantics, and polymorphic vs static (which correlate nicely). It
 should, however, not imply whether the object is managed by the GC or
 not. Some kind of ownership mechanism would be more suited for that.


+1
May 03 2014
prev sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Saturday, 3 May 2014 at 11:12:56 UTC, Michel Fortin wrote:

 On 2014-05-01 17:35:36 +0000, "Marc Schütz" <schuetzm gmx.net> 
 said:


 Maybe the language should have some way to distinguish between 
 GC-managed and manually-managed objects, preferably in the 
 type system. Then it could be statically checked whether an 
 object is supposed to be GC-managed, and consequentially 
 shouldn't have a destructor.


 Or turn the rule on its head: make it so having a destructor 
 makes the heap memory block reference counted. With this adding 
 a destructor always cause deterministic destruction.

 The compiler knows statically whether a struct has a 
 destructor. For a class you need a runtime trick because the 
 root object which can be either. Use a virtual call or a magic 
 value in the reference count field to handle the reference 
 count management. You also need a way to tag a class to be 
 guarantied it has no derived class with a destructor (to 
 provide a static proof for the compiler it can omit ARC code), 
 perhaps  disable ~this().

 Then remains the problem of cycles. It could be a hard error if 
 the destructor is  safe (error thrown when the GC collects it). 
 The destructor could be allowed to run (in any thread) if the 
 destructor is  system or  trusted.

 The interesting thing with this is that the current D semantics 
 are preserved, destructors become deterministic (except in the 
 presence of cycles, which the GC will detect for you), and if 
 you're manipulating pointers to pure memory (memory blocks 
 having no destructor) there's no ARC overhead. And finally, no 
 new pointer attributes; Walter will like this last one.


This is certainly also an interesting idea, but I suspect it is 
bound to fail, simply because it involves ARC. Reference counting 
always makes things so much more complicated... See for example 
the cycles problem you mentioned: If you need a GC for that, you 
cannot guarantee that the objects will be collected, which was 
the reason to introduce ARC in the first place. Then there are 
the problems with shared vs. thread-local RC (including casting 
between the two), and arrays/slices of RC objects. And, of 
course, Walter doesn't like it ;-)



 May 04 2014



 next sibling parent  Michel Fortin <michel.fortin michelf.ca>  writes:


prev sibling  parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 4 May 2014 19:00, via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Saturday, 3 May 2014 at 11:12:56 UTC, Michel Fortin wrote:

 On 2014-05-01 17:35:36 +0000, "Marc Schütz" <schuetzm gmx.net> said:


 Maybe the language should have some way to distinguish between GC-managed
 and manually-managed objects, preferably in the type system. Then it could
 be statically checked whether an object is supposed to be GC-managed, and
 consequentially shouldn't have a destructor.



 Or turn the rule on its head: make it so having a destructor makes the
 heap memory block reference counted. With this adding a destructor always
 cause deterministic destruction.

 The compiler knows statically whether a struct has a destructor. For a
 class you need a runtime trick because the root object which can be either.
 Use a virtual call or a magic value in the reference count field to handle
 the reference count management. You also need a way to tag a class to be
 guarantied it has no derived class with a destructor (to provide a static
 proof for the compiler it can omit ARC code), perhaps  disable ~this().

 Then remains the problem of cycles. It could be a hard error if the
 destructor is  safe (error thrown when the GC collects it). The destructor
 could be allowed to run (in any thread) if the destructor is  system or
  trusted.

 The interesting thing with this is that the current D semantics are
 preserved, destructors become deterministic (except in the presence of
 cycles, which the GC will detect for you), and if you're manipulating
 pointers to pure memory (memory blocks having no destructor) there's no ARC
 overhead. And finally, no new pointer attributes; Walter will like this last
 one.



 This is certainly also an interesting idea, but I suspect it is bound to
 fail, simply because it involves ARC. Reference counting always makes things
 so much more complicated... See for example the cycles problem you
 mentioned: If you need a GC for that, you cannot guarantee that the objects
 will be collected, which was the reason to introduce ARC in the first place.


So specify that improper weak reference attribution may lead to
interference with proper execution of destructors. People generally
understand this, and at least they'd have such a tool to make their
code behave correctly.
Perhaps even have rules that things with destructors create static
errors if they are used in a way that may create circular references
when effective weak attribution is not detected by the compiler (if
such a thing is statically possible?).


 Then there are the problems with shared vs. thread-local RC (including
 casting between the two),


The problem is exactly the same as 'shared' exists already. What makes
it any different?
shared <-> not-shared requires blunt casting, and the same can apply
to shared RC. 'shared' implies RC access must use atomics, and
otherwise not, I don't imagine any distinction in data structure?


 and arrays/slices of RC objects.


Slices need to know their offset (or base pointer), or have an
explicit RC pointer. Either way, I don't see how slices are a
particularly troublesome case.
12byte slices on x32 - needs an extra field, 16 bytes should still be
sufficient on x64 considering that 64bit pointers are only 40-48 bits,
which means there are loads of spare bits in the pointer and in the
slice length field; should be plenty to stash an offset.


 And, of course,
 Walter doesn't like it ;-)


True. But I'm still waiting to see another even theoretically workable solution.



 May 05 2014



prev sibling next sibling parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com>  writes:


 into a class. I'm inclined to say that we should outright 
 prohibit that,


 That can't happen.


Why is that?

/Per



 May 01 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/1/14, 12:52 PM, "Nordlöw" wrote:

 into a class. I'm inclined to say that we should outright prohibit that,


 That can't happen.


 Why is that?


(1) Too much breakage, (2) would disallow a ton of correct code, (3) no 
reasonable alternative to propose. We'd essentially hang our users out 
to dry. -- Andrei



 May 01 2014



 next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 5/1/14, 12:52 PM, "Nordlw" wrote:

into a class. I'm inclined to say that we should outright prohibit that,


That can't happen.


Why is that?


 
 (1) Too much breakage, (2) would disallow a ton of correct code, (3)
 no reasonable alternative to propose. We'd essentially hang our users
 out to dry. -- Andrei


Isn't this what we're already doing by (eventually) getting rid of class
dtors?

(Don't forget emplace'd class instances, which do not live in GC heap.)


T

-- 
Let's call it an accidental feature. -- Larry Wall



 May 01 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 5/1/14, 12:52 PM, "Nordlw" wrote:

 into a class. I'm inclined to say that we should outright prohibit that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct code, (3)
 no reasonable alternative to propose. We'd essentially hang our users
 out to dry. -- Andrei


 Isn't this what we're already doing by (eventually) getting rid of class
 dtors?


Not even close. (1) A lot less breakage, (2) disallowed code was already 
not guaranteed to work, (3) reasonable alternatives exist.

Andrei



 May 01 2014



 next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Thu, May 01, 2014 at 02:29:22PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:

On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

On 5/1/14, 12:52 PM, "Nordlw" wrote:

into a class. I'm inclined to say that we should outright
prohibit that,


That can't happen.


Why is that?


(1) Too much breakage, (2) would disallow a ton of correct code, (3)
no reasonable alternative to propose. We'd essentially hang our
users out to dry. -- Andrei


Isn't this what we're already doing by (eventually) getting rid of
class dtors?


 
 Not even close. (1) A lot less breakage, (2) disallowed code was
 already not guaranteed to work, (3) reasonable alternatives exist.


[...]

Argh, I feel like this discussion is going in circles. Do I really have
to spell everything out just to make my point?

1) Today, a struct with a dtor will have the dtor invoked when the
struct goes out of scope.

2) Today, the language allows you to put such a struct in a class as a
member variable.

3) Today, the struct dtor of the class member will get invoked when the
class dtor is invoked.

4) Andrei is proposing to deprecate class dtors, meaning that at some
point they will no longer exist.

5) When that day comes, class dtors will no longer exist.

6) Since class dtors were the only thing that cleaned up the struct
member variables by invoking their dtors, that means the struct dtor
will *never* get invoked.

Since the struct dtor will *never* get invoked, it makes no sense to
define one in the first place. What's the point of defining a struct
dtor if there is no guarantee it will get invoked? That completely
defeats the purpose of a dtor. Do we really want D's dtors to have the
semantics of "this *might* get called with the struct is destroyed, or
it *might* not?" Then we might as well go back to C and do manual
cleanups.

And if Andrei's proposal to get rid of class dtors goes through, then
the only way to guarantee dtor semantics is to prohibit the use of
structs with dtors as class members. It's either that, or struct dtors
will be left in the limbo of "maybe it will get called, maybe it won't".
I don't see this as an advantageous position at all -- in fact, it's a
regression from the current language, which, as Walter demonstrated,
*does* invoke the struct dtor when the class goes out of scope.


T

-- 
We are in class, we are supposed to be learning, we have a teacher... Is it too
much that I expect him to teach me??? -- RL



 May 01 2014



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/1/14, 3:17 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 02:29:22PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 5/1/14, 12:52 PM, "Nordlw" wrote:

 into a class. I'm inclined to say that we should outright
 prohibit that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct code, (3)
 no reasonable alternative to propose. We'd essentially hang our
 users out to dry. -- Andrei


 Isn't this what we're already doing by (eventually) getting rid of
 class dtors?


 Not even close. (1) A lot less breakage, (2) disallowed code was
 already not guaranteed to work, (3) reasonable alternatives exist.


 [...]

 Argh, I feel like this discussion is going in circles. Do I really have
 to spell everything out just to make my point?

 1) Today, a struct with a dtor will have the dtor invoked when the
 struct goes out of scope.

 2) Today, the language allows you to put such a struct in a class as a
 member variable.

 3) Today, the struct dtor of the class member will get invoked when the
 class dtor is invoked.

 4) Andrei is proposing to deprecate class dtors, meaning that at some
 point they will no longer exist.

 5) When that day comes, class dtors will no longer exist.

 6) Since class dtors were the only thing that cleaned up the struct
 member variables by invoking their dtors, that means the struct dtor
 will *never* get invoked.

 Since the struct dtor will *never* get invoked, it makes no sense to
 define one in the first place.


Here's where the point derails. A struct may be preexisting; the 
decision to define a destructor for it and the decision to use 
polymorphism for an object that needs that structure are most of the 
time distinct.

Andrei



 May 01 2014



 next sibling parent  "deadalnix" <deadalnix gmail.com>  writes:


On Friday, 2 May 2014 at 00:45:42 UTC, Andrei Alexandrescu wrote:

 Here's where the point derails. A struct may be preexisting; 
 the decision to define a destructor for it and the decision to 
 use polymorphism for an object that needs that structure are 
 most of the time distinct.

 Andrei


Sheep eat grass.
Boat float on water.

These are completely distinct. Therefore, there won't be any 
issue if we ditch all the grass in the sea from that boat the 
carry sheeps around.



 May 02 2014



prev sibling  parent  "w0rp" <devw0rp gmail.com>  writes:


On Friday, 2 May 2014 at 00:45:42 UTC, Andrei Alexandrescu wrote:

 Here's where the point derails. A struct may be preexisting; 
 the decision to define a destructor for it and the decision to 
 use polymorphism for an object that needs that structure are 
 most of the time distinct.

 Andrei


I wonder how common the pattern of putting a struct with a 
destructor in a class actually is. It might be a case for 
defining data structures with GC allocation rather than reference 
counting (as in std.container). I suppose a choice of allocator 
will change this quite a lot. That which is allocated with 
reference counting could uniquely hold its container member and 
then call the destructor when it dies, or similar.

I find it kind of a funny thing to put something like a File 
inside of a class. I have always seen the mix of GC and resource 
management as more managing resources like Files in scopes and 
reading data from the resources which turn into objects in memory 
which are garbage collected. Not allocating garbage collected 
objects which contain resources.



 May 03 2014



prev sibling  parent  "ed" <sillymongrel gmail.com>  writes:


On Thursday, 1 May 2014 at 22:18:59 UTC, H. S. Teoh via 
Digitalmars-d wrote:
[snip]

 6) Since class dtors were the only thing that cleaned up the 
 struct
 member variables by invoking their dtors, that means the struct 
 dtor
 will *never* get invoked.


[snip]

I might be mistaken but isn't it the case now that class dtors 
may never get invoked by the GC anyway? In which case having 
class dtors is pointless because one cannot rely on them.

But dtors for structs I thought would still be useful. The dtor 
might not be called by the class dtor as it no longer exists, but 
it is called if I reassign the struct member by value.

C c = new C();

// some code
auto s = SomeStruct(/*with some parms*/);
c.setStruct(s); // the existing c.struct dtor will be called, 
releasing whatever resources it needs


Cheers,
ed



 May 01 2014



prev sibling  parent reply "Paolo Invernizzi" <paolo.invernizzi no.address>  writes:


On Thursday, 1 May 2014 at 21:29:19 UTC, Andrei Alexandrescu 
wrote:

 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 On 5/1/14, 12:52 PM, "Nordlöw" wrote:

 into a class. I'm inclined to say that we should outright 
 prohibit that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct 
 code, (3)
 no reasonable alternative to propose. We'd essentially hang 
 our users
 out to dry. -- Andrei


 Isn't this what we're already doing by (eventually) getting 
 rid of class
 dtors?


 Not even close. (1) A lot less breakage, (2) disallowed code 
 was already not guaranteed to work, (3) reasonable alternatives 
 exist.

 Andrei


I have 165k lines of code to review for that change... I would 
not call it a minor breakage...

/Paolo



 May 02 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/2/14, 1:34 AM, Paolo Invernizzi wrote:

 On Thursday, 1 May 2014 at 21:29:19 UTC, Andrei Alexandrescu wrote:

 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via
 Digitalmars-d wrote:

 On 5/1/14, 12:52 PM, "Nordlöw" wrote:

 into a class. I'm inclined to say that we should outright
 prohibit that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct code, (3)
 no reasonable alternative to propose. We'd essentially hang our users
 out to dry. -- Andrei


 Isn't this what we're already doing by (eventually) getting rid of class
 dtors?


 Not even close. (1) A lot less breakage, (2) disallowed code was
 already not guaranteed to work, (3) reasonable alternatives exist.

 Andrei


 I have 165k lines of code to review for that change... I would not call
 it a minor breakage...


I didn't. I said a lot less that straight out disallowing struct 
members. -- Andrei



 May 02 2014



  parent  "Paolo Invernizzi" <paolo.invernizzi no.address>  writes:


On Friday, 2 May 2014 at 15:03:47 UTC, Andrei Alexandrescu wrote:

 On 5/2/14, 1:34 AM, Paolo Invernizzi wrote:

 On Thursday, 1 May 2014 at 21:29:19 UTC, Andrei Alexandrescu 
 wrote:

 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei 
 Alexandrescu via
 Digitalmars-d wrote:

 On 5/1/14, 12:52 PM, "Nordlöw" wrote:

 into a class. I'm inclined to say that we should outright
 prohibit that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct 
 code, (3)
 no reasonable alternative to propose. We'd essentially hang 
 our users
 out to dry. -- Andrei


 Isn't this what we're already doing by (eventually) getting 
 rid of class
 dtors?


 Not even close. (1) A lot less breakage, (2) disallowed code 
 was
 already not guaranteed to work, (3) reasonable alternatives 
 exist.

 Andrei


 I have 165k lines of code to review for that change... I would 
 not call
 it a minor breakage...


 I didn't. I said a lot less that straight out disallowing 
 struct members. -- Andrei


I would also add one point, just because it is not so obvious: 
I'll be more than happy to review my company code, if the 
proposed solutions about finalisation in class/struct turn out to 
take D one step forward being a better programming language.

/Paolo



 May 03 2014



prev sibling next sibling parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com>  writes:


 (1) Too much breakage, (2) would disallow a ton of correct 
 code, (3) no reasonable alternative to propose. We'd 
 essentially hang our users out to dry. -- Andrei


I misunderstood your usage of the word "can't". I thought it 
symbolized a practical/theoretical limitation, not design 
decision.

So to check if I got this right; class dtors will, with your 
proposal, only be called by the GC when any of its struct members 
has dtors, right?

For this to work the compiler will have to do some kind dtor 
inference of members, members of members etc, right?



 May 01 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/1/14, 2:04 PM, "Nordlöw" wrote:

 So to check if I got this right; class dtors will, with your proposal,
 only be called by the GC when any of its struct members has dtors, right?


No, in this proposal the GC heap would be entirely passive. Given the 
pushback in this thread that prolly won't fly.

Andrei



 May 01 2014



prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Thu, 01 May 2014 16:03:06 -0400, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:


 On 5/1/14, 12:52 PM, "Nordl=C3=B6w" wrote:

 into a class. I'm inclined to say that we should outright prohibit =








 =


 that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct code, (3) n=


o  =


 reasonable alternative to propose. We'd essentially hang our users out=


  =


 to dry. -- Andrei


class C
{
    void finalizer();
    private struct S {
      C owner;
      ~this() {
         owner.finalize();
      }
    }
    private S s;
}

You aren't helping anything.

-Steve



 May 01 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/1/14, 2:21 PM, Steven Schveighoffer wrote:

 On Thu, 01 May 2014 16:03:06 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 5/1/14, 12:52 PM, "Nordlöw" wrote:

 into a class. I'm inclined to say that we should outright prohibit
 that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct code, (3)
 no reasonable alternative to propose. We'd essentially hang our users
 out to dry. -- Andrei


 class C
 {
     void finalizer();
     private struct S {
       C owner;
       ~this() {
          owner.finalize();
       }
     }
     private S s;
 }

 You aren't helping anything.


There's a misunderstanding somewhere. -- Andrei



 May 01 2014



  parent  "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Thu, 01 May 2014 17:33:56 -0400, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:


 On 5/1/14, 2:21 PM, Steven Schveighoffer wrote:

 On Thu, 01 May 2014 16:03:06 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 5/1/14, 12:52 PM, "Nordl=C3=B6w" wrote:

 into a class. I'm inclined to say that we should outright prohibi=












t

 that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct code, (3)=







 no reasonable alternative to propose. We'd essentially hang our user=






s

 out to dry. -- Andrei


 class C
 {
     void finalizer();
     private struct S {
       C owner;
       ~this() {
          owner.finalize();
       }
     }
     private S s;
 }

 You aren't helping anything.


 There's a misunderstanding somewhere. -- Andrei


The above is or is not allowed under the new proposal?

You said class dtors will cease to exist, but that you can have structs =
 =

(with dtors?) embedded in them.

I defined a de-facto destructor for a class, when none is allowed. Unles=
s  =

you are planning on having classes not call the struct dtors?

-Steve



 May 01 2014



prev sibling  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Thursday, 1 May 2014 at 20:03:03 UTC, Andrei Alexandrescu 
wrote:

 On 5/1/14, 12:52 PM, "Nordlöw" wrote:

 into a class. I'm inclined to say that we should outright 
 prohibit that,


 That can't happen.


 Why is that?


 (1) Too much breakage, (2) would disallow a ton of correct 
 code, (3) no reasonable alternative to propose. We'd 
 essentially hang our users out to dry. -- Andrei


(1) is made of turbo lol. A huge amount of binding to C++ lib 
rely on destructor. The breakage in the proposal is already 
massive. For instance, GtkD won't work. And other example have 
been presented.



 May 02 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/2/14, 12:07 AM, deadalnix wrote:

 The breakage in the proposal is already massive. For instance, GtkD
 won't work. And other example have been presented.


Yah, prolly we can't go that far. -- Andrei



 May 02 2014



prev sibling  parent  =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com>  writes:


 into a class. I'm inclined to say that we should outright 
 prohibit that,


 That can't happen.


Why is that?

/Per



 May 01 2014



prev sibling next sibling parent reply "fra" <a b.it>  writes:


On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:

 I think there's no need to argue that in this community. The GC 
 never guarantees calling destructors even today, so this 
 decision would be just a point in the definition space (albeit 
 an extreme one).


I think I (we) need a bit of clarification.
Docs in http://dlang.org/class.html#destructors states that: "The 
garbage collector calls the destructor function when the object 
is deleted."
As far as I understand, this means that destructors are always 
called when an instance is collected. Is this right?
Doesn't this mean that destructors are guaranteed to run for 
unreferenced objects if we force the GC to do a full collect 
cycle?



 May 02 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/2/14, 9:04 AM, fra wrote:

 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote:

 I think there's no need to argue that in this community. The GC never
 guarantees calling destructors even today, so this decision would be
 just a point in the definition space (albeit an extreme one).


 I think I (we) need a bit of clarification.
 Docs in http://dlang.org/class.html#destructors states that: "The
 garbage collector calls the destructor function when the object is
 deleted."
 As far as I understand, this means that destructors are always called
 when an instance is collected. Is this right?
 Doesn't this mean that destructors are guaranteed to run for
 unreferenced objects if we force the GC to do a full collect cycle?


False pointers make it seem like unreferenced objects are in fact 
referenced, so fewer destructors will run than there should. -- Andrei



 May 02 2014



  parent reply "monarch_dodra" <monarchdodra gmail.com>  writes:


On Friday, 2 May 2014 at 16:20:47 UTC, Andrei Alexandrescu wrote:

 On 5/2/14, 9:04 AM, fra wrote:

 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei 
 Alexandrescu wrote:

 I think there's no need to argue that in this community. The 
 GC never
 guarantees calling destructors even today, so this decision 
 would be
 just a point in the definition space (albeit an extreme one).


 I think I (we) need a bit of clarification.
 Docs in http://dlang.org/class.html#destructors states that: 
 "The
 garbage collector calls the destructor function when the 
 object is
 deleted."
 As far as I understand, this means that destructors are always 
 called
 when an instance is collected. Is this right?
 Doesn't this mean that destructors are guaranteed to run for
 unreferenced objects if we force the GC to do a full collect 
 cycle?


 False pointers make it seem like unreferenced objects are in 
 fact referenced, so fewer destructors will run than there 
 should. -- Andrei


Yeah, you have to read the "fine print": "collection implies 
destruction" *but* "no guarantees the collection will actually 
ever happen".



 May 02 2014



 next sibling parent  "deadalnix" <deadalnix gmail.com>  writes:


On Friday, 2 May 2014 at 20:59:46 UTC, monarch_dodra wrote:

 Yeah, you have to read the "fine print": "collection implies 
 destruction" *but* "no guarantees the collection will actually 
 ever happen".


That sound like the right balance.

Also, make construction of object with destructor  system as 
there is no way to ensure destructor won't resurrect the object 
or do some goofy thing with finalized reference it has.



 May 02 2014



prev sibling  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Friday, 2 May 2014 at 20:59:46 UTC, monarch_dodra wrote:

 Yeah, you have to read the "fine print": "collection implies 
 destruction" *but* "no guarantees the collection will actually 
 ever happen".


Which make destructors dangerous constructs. It means you now 
risk getting random bugs-reports after deployment. Acceptable for 
a hobby language, not acceptable for a systems programming 
language.



 May 03 2014



prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:

 Walter and I have had a long chat in which we figured our current
 offering of abstractions could be improved. Here are some thoughts.
 There's a lot of work ahead of us on that and I wanted to make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within
 the GC entirely. It makes for a faster and better GC, but the real
 reason here is that destructors are philosophically bankrupt in a GC
 environment. I think there's no need to argue that in this community.
 The GC never guarantees calling destructors even today, so this decision
 would be just a point in the definition space (albeit an extreme one).

 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that by
 other means, such as reference counting or manual. We're considering
 deprecating ~this() for classes in the future.

 Also, we're considering a revamp of built-in slices, as follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently lowered into
 RCSlice!T, defined inside object.d. That type would occupy THREE words,
 one of which being a pointer to a reference count. That type would
 redefine all slice primitives to update the reference count accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
 will be allowed, and will ignore the reference count (so if a void[]
 extracted from a T[] via a cast outlives all slices, dangling pointers
 will ensue).

 I foresee any number of theoretical and practical issues with this
 approach. Let's discuss some of them here.


 Thanks,

 Andrei


Honestly, that sounds like the entierly wrong apporach to me. Your 
approaching the problem in this way:

"We can not implement a propper GC in D because the language design 
prevents us from doing so. So lets remove destructors to migate the 
issue of false pointers."

While the approach should be.

"The language does not allow to implement a propper GC (anything else 
then dirty mark & sweep), what needs to be changed to allow a 
implementation of a more sophisticated GC."


heavily relies on resource management. So basically every class in there 


finalizer!). Basically the entire codebase feels like manual memory 
management. You have to think about manually destroying every class and 
the entire advantage of having a GC, e.g. not having to think about 
memory management and thus beeing more productive, vanished. It really 


And what if I want unsafe slices of structs with destructors, for 
performance? Maybe I perfectly know that the memory behind the slice 
will outlive the slice, and I don't want the overhead of all the 
reference counthing behind it?

If you actually deprecate ~this, there would be two options for me.
1) Migrate my entire codebase to some user defiend finalizer function 
(which doesn't have compiler support), which would be a lot of work.
2) Quit D. (which is becomeing more and more an option when reading the 
recent news group discussions.)

-- 
Kind Regards
Benjamin Thaut



 May 03 2014



 next sibling parent reply "froglegs" <barf barf.com>  writes:









 May 03 2014



  parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 03.05.2014 10:57, schrieb froglegs:







Which does not work across threads, relies on stack allocations and has 
issues if cleaning a resource implies having a not handled exception on 
the destructor of the RAII class.

I rather make use of "using" and FP resource handling via lambdas.

--
Paulo



 May 03 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/3/14, 1:49 AM, Benjamin Thaut wrote:

 2) Quit D. (which is becomeing more and more an option when reading the
 recent news group discussions.)


The entire idea of starting these discussions is to gather a sense of 
shared vision with the community on the best direction to follow.

I'm trying to steer things so as to gain the maximum benefit from each 
potential breakage. In wake of that, it seems incongruous that first you 
ask for even larger breaking changes to the language, but then consider 
quitting it entirely on account of changes being too disruptive.

Anyhow, just to clarify, it seems like eliminating destructor calls 
during GC is not a viable option. I'll define std.allocator to allow 
users to define such a GC if they so want, without prescribing either 
policy.


Andrei



 May 03 2014



  parent  "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Sat, May 03, 2014 at 11:12:36AM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 Anyhow, just to clarify, it seems like eliminating destructor calls
 during GC is not a viable option. I'll define std.allocator to allow
 users to define such a GC if they so want, without prescribing either
 policy.


[...]

Thank you. ;-)

On that note, it would be very nice if we could clearly define when dtor
calls will / will not happen. As someone else said, to an end-user of D
it is hard to accept that basic language constructs like arrays and
dtors, when used together, fail to function in an expected way (without
understanding what goes on underneath the hood, e.g. why dtors of array
elements may not get called). Ideally, the language should be designed
such that these obvious combinations of language constructs should
either Just Work(tm), or not be allowed, or require explicit annotation
(so that the user knows something unexpected might happen).

But, since we're not in an ideal world, the very least we could do is to
clearly define exactly which combinations of language constructs may
behave in an unexpected way, and document them up front in a prominent
place.

Otherwise, we risk turning off potential users -- I can just imagine a
newbie to D writing something like "File[] files;" and then wondering
why things don't work as expected, and then throwing in the towel and
say "what a lousy language, let me move on to another one".

And on that note, I'd like to say that my previous posts about
prohibiting structs with dtors as class members (if we were to move in
the direction of getting rid of class dtors altogether, which happily
isn't the case anymore) were primarily motivated by the desire to see
more of D obeying the principle of least surprise: two built-in language
constructs, structs with dtors and class members, when combined
together, *should* Just Work -- one expects that dtors will get called
with the struct goes out of scope, by the very definition of a dtor, so
one would expect they will still get cleaned up when they happen to be a
class member. The fact that they don't is a surprise, which then
requires some other way of warning the user that things aren't what they
might be expected to be. Maybe my proposed solutions suck, but the
underlying problem still needs to be addressed. The scope of the problem
is smaller, now that we're no longer killing off class dtors, but
nevertheless something needs to be done about it.

I think this is one area where D could use a lot of improvement. There
are currently a handful of glaring holes where built-in language
constructs interact with each other in unexpected or buggy ways.  While
it may feel like mole-whacking (because of combinatorial explosion as
you add features to the language), I think it's very important to
address, since otherwise it gives new users an impression (not
necessarily well-founded, but first impressions do matter) of sloppy
language design. The seams show through, and it's not very nice. One
example is the interaction of dtors with the GC. Another example is the
interaction of const/immutable with AA's. Another is the interaction of
 disabled with .init and other generic code. It seems that almost every
non-trivial use of these features is like navigating a minefield --
there are so many gotchas, unexpected behaviours, and implementation
bugs, that it's embarrassing. Other areas include shared, though I don't
have first-hand experience of that so I can't speak for it.

While proposing radical changes may be exhilarating, in the long run
that may do more harm than good. What these features need is some TLC
and detailed fine-tuning within the current, already-existing framework.

TL;DR: I'd like to see more attention paid to the details of how
language features interact with each other, and fixing those issues,
rather than inventing more radical new ideas that may or may not solve
the problem, and more likely than not will introduce new problems to add
to our already-long list of issues.


T

-- 
Amateurs built the Ark; professionals built the Titanic.



 May 03 2014



prev sibling next sibling parent reply Caligo via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sat, May 3, 2014 at 3:49 AM, Benjamin Thaut via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 2) Quit D. (which is becomeing more and more an option when reading the
 recent news group discussions.)

 --
 Kind Regards
 Benjamin Thaut


I never thought I would say this, but I have begun to move away from D.  I
know others who are doing the same.  The advantages D has over other
languages are slowly diminishing.  Plus, as it is, D is just as complex and
complicated as C++, if not more.  Couple that with the never-ending bugs
and weak development process, there isn't much confidence in choosing D.

One of the main and crippling issues D has is its development process.  Its
inner circle, mainly Walter, Andrei and a few others, do not seem to have a
good understanding of FOSS development process.  For example, Rust is only
a few years old and much younger than D, but it has a greater number of
contributors, and its rate of contributors seems to be growing faster.  D
has failed at recruitment; I always read people (mainly the inner circle)
mention the everlasting low-hanging fruits.  Well, if the number of
contributors was growing then the number of low-hanging fruits would be
decreasing.  Andrei recently introduced the bounty system, which not only
is an insult to those who contribute to FOSS, but it also goes to show that
he really doesn't understand how and why people contribute to FOSS without
ever asking to be compensated.

Last but not least, currently there are two main ways for new features to
make it into D/Phobos: you either have to belong to the inner circle, or
have to represent some corporation that's doing something with D.  I don't
remember seeing a feature that was added to D/Phobos with some on/off
switch that people could try in the next release, and then send in
feedback.  You're in a much better position to make a decision about a
feature if the users have actually used it and reported feedback.  Ahh, and
don't get me started on Phobos review process; it's a joke, it's bogus and
just pathetic.

If D had a sound development process, I don't think we would be having the
problems that we have today.



 May 03 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/3/14, 8:48 PM, Caligo via Digitalmars-d wrote:

 On Sat, May 3, 2014 at 3:49 AM, Benjamin Thaut via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     2) Quit D. (which is becomeing more and more an option when reading
     the recent news group discussions.)

     --
     Kind Regards
     Benjamin Thaut


 I never thought I would say this, but I have begun to move away from D.
   I know others who are doing the same.  The advantages D has over other
 languages are slowly diminishing.  Plus, as it is, D is just as complex
 and complicated as C++, if not more.  Couple that with the never-ending
 bugs and weak development process, there isn't much confidence in
 choosing D.

 One of the main and crippling issues D has is its development process.
   Its inner circle, mainly Walter, Andrei and a few others, do not seem
 to have a good understanding of FOSS development process.  For example,
 Rust is only a few years old and much younger than D, but it has a
 greater number of contributors, and its rate of contributors seems to be
 growing faster.  D has failed at recruitment; I always read people
 (mainly the inner circle) mention the everlasting low-hanging fruits.
   Well, if the number of contributors was growing then the number of
 low-hanging fruits would be decreasing.  Andrei recently introduced the
 bounty system, which not only is an insult to those who contribute to
 FOSS, but it also goes to show that he really doesn't understand how and
 why people contribute to FOSS without ever asking to be compensated.


Mostly good points, but the bountysource program is an experiment by 
Facebook, not by myself. And (without me trying to speak on Facebook's 
behalf) it would be difficult to argue that Facebook doesn't understand 
FOSS or is out there to insult contributors. We're just experimenting 
with various angles.


 Last but not least, currently there are two main ways for new features
 to make it into D/Phobos: you either have to belong to the inner circle,
 or have to represent some corporation that's doing something with D.  I
 don't remember seeing a feature that was added to D/Phobos with some
 on/off switch that people could try in the next release, and then send
 in feedback.  You're in a much better position to make a decision about
 a feature if the users have actually used it and reported feedback.


The on/off switch may be a nice idea in the abstract but is hardly the 
perfect recipe to good language feature development; otherwise everybody 
would be using it, and there's not overwhelming evidence to that. (I do 
know it's been done a few times, such as the (in)famous "new scoping 
rule of the for statement" for C++ which has been introduced as an 
option by VC++.)

I wonder how you've gotten the perception that one needs to be a member 
of the inner circle mafia to get things into D. Today's major 
contributors to D came from all over, without any preexisting 
relationship to anyone else, and their admission ticket has been getting 
work done. Could you please get into detail on how you view things? (I 
tried to look over your past posts to see a pattern of rejected 
contributions, but didn't find such.)


   Ahh, and don't get me started on Phobos review process; it's a joke,
 it's bogus and just pathetic.


Actually I'd love to get you started so I'd understand your angle 
better. I'm sure we can do a lot better. One good thing Phobos reviews 
have done since we initiated them has been to prevent bad artifacts to 
make it into the library. We'd love to make it better. From what I saw 
witnessing similar processes (C++, Boost, Python, Scala) - they all have 
some sense of awkward self-importance to them upon the first look. I 
think that's the way such things work.


 If D had a sound development process, I don't think we would be having
 the problems that we have today.


I've discussed development process with a number of people who 
participated at such. They mentioned that until you get teams paid to 
work on the respective system (OS, language, framework etc) it all works 
on the basis of people doing things to scratch an itch they have. The 
critical mass is attained when there are enough people to cover a large 
enough itching area :o). If telling people what to work on on their free 
time works, I haven't succeeded at it and don't know anyone who has.


Andrei



 May 03 2014



  parent reply Caligo via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sun, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 Mostly good points, but the bountysource program is an experiment by
 Facebook, not by myself. And (without me trying to speak on Facebook's
 behalf) it would be difficult to argue that Facebook doesn't understand
 FOSS or is out there to insult contributors. We're just experimenting with
 various angles.



If the bounty system was such a great idea, then every FOSS project would
be using it.  Now, hiring full-time engineers to work on a FOSS project,
that's an entirely different issue.  Besides, if someone is trying to
figure out how FOSS teams manage to become successful in regards to
development and all the associated technical and social complexities, then
all they have to do is study one of the million different FOSS projects out
there.  Many well known FOSS contributors have actually documented their
experience and knowledge of managing FOSS projects.



 The on/off switch may be a nice idea in the abstract but is hardly the
 perfect recipe to good language feature development; otherwise everybody
 would be using it, and there's not overwhelming evidence to that. (I do
 know it's been done a few times, such as the (in)famous "new scoping rule
 of the for statement" for C++ which has been introduced as an option by
 VC++.)


No, it's nothing abstract, and it's very practical and useful.  Rust has

 Even Python has __future__, and many others.



 I wonder how you've gotten the perception that one needs to be a member of
 the inner circle mafia to get things into D. Today's major contributors to
 D came from all over, without any preexisting relationship to anyone else,
 and their admission ticket has been getting work done. Could you please get
 into detail on how you view things? (I tried to look over your past posts
 to see a pattern of rejected contributions, but didn't find such.)


I wasn't trying to imply that contributions are rejected if contributors
are not members of a certain group.  I was just trying to say that it's
more likely for a _new_ feature to make it into D/Phobos if they are
proposed by members of the inner circle or someone representing a
corporation, probably because they become more noticeable, or because they
get lost in all the forum noise, I don't know.  I could be wrong, but
that's just how I perceive it.




 Actually I'd love to get you started so I'd understand your angle better.
 I'm sure we can do a lot better. One good thing Phobos reviews have done
 since we initiated them has been to prevent bad artifacts to make it into
 the library. We'd love to make it better. From what I saw witnessing
 similar processes (C++, Boost, Python, Scala) - they all have some sense of
 awkward self-importance to them upon the first look. I think that's the way
 such things work.


Here is an idea:  include new features in DMD/Phobos as soon as they
arrive, and make them part of the official binary release so that the
average D user can try them out.  Make sure they are marked as unstable,
and put a on/off switch on them (something like what Rust/Haskell have; not
a compiler switch).  If the feature receives no implementation bug reports
for X consecutive days AND no design bug reports for Y consecutive days,
then the feature is marked stable and officially becomes part of
DMD/Phobos.  The X and the Y can be decreased as D's number of users
increases over the years.  The whole idea is very much like farming: you
are planting seeds.  As the plants grow, some of them will not survive,
others will be destroyed, and some of them will take years to grow.  In any
case, you harvest the fruits when they are ready.

 Here are good starting values for X and Y:
X = 90 days
Y = 180 days



 May 04 2014



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/4/14, 5:38 PM, Caligo via Digitalmars-d wrote:

 On Sun, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:


     Mostly good points, but the bountysource program is an experiment by
     Facebook, not by myself. And (without me trying to speak on
     Facebook's behalf) it would be difficult to argue that Facebook
     doesn't understand FOSS or is out there to insult contributors.
     We're just experimenting with various angles.


 If the bounty system was such a great idea, then every FOSS project
 would be using it.


As I said: experiment.


 Now, hiring full-time engineers to work on a FOSS
 project, that's an entirely different issue.  Besides, if someone is
 trying to figure out how FOSS teams manage to become successful in
 regards to development and all the associated technical and social
 complexities, then all they have to do is study one of the million
 different FOSS projects out there.  Many well known FOSS contributors
 have actually documented their experience and knowledge of managing FOSS
 projects.


Great, a few representative links would be most welcome.


 Here is an idea:  include new features in DMD/Phobos as soon as they
 arrive, and make them part of the official binary release so that the
 average D user can try them out.  Make sure they are marked as unstable,
 and put a on/off switch on them (something like what Rust/Haskell have;
 not a compiler switch).  If the feature receives no implementation bug
 reports for X consecutive days AND no design bug reports for Y
 consecutive days, then the feature is marked stable and officially
 becomes part of DMD/Phobos.  The X and the Y can be decreased as D's
 number of users increases over the years.  The whole idea is very much
 like farming: you are planting seeds.  As the plants grow, some of them
 will not survive, others will be destroyed, and some of them will take
 years to grow.  In any case, you harvest the fruits when they are ready.

   Here are good starting values for X and Y:
 X = 90 days
 Y = 180 days


This is nice, but on the face of it it's just this: an idea on how other 
people should do things on their free time. I'd have difficulty 
convincing people they should work that way. The kind of ideas that I 
noticed are successful are those that actually carry the work through 
and serve as good examples to follow.


Andrei



 May 04 2014



 next sibling parent reply Caligo via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sun, May 4, 2014 at 11:09 PM, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 Great, a few representative links would be most welcome.


Here is a good starting point (it's a classic):

http://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar


 Here is an idea:  include new features in DMD/Phobos as soon as they

 arrive, and make them part of the official binary release so that the
 average D user can try them out.  Make sure they are marked as unstable,
 and put a on/off switch on them (something like what Rust/Haskell have;
 not a compiler switch).  If the feature receives no implementation bug
 reports for X consecutive days AND no design bug reports for Y
 consecutive days, then the feature is marked stable and officially
 becomes part of DMD/Phobos.  The X and the Y can be decreased as D's
 number of users increases over the years.  The whole idea is very much
 like farming: you are planting seeds.  As the plants grow, some of them
 will not survive, others will be destroyed, and some of them will take
 years to grow.  In any case, you harvest the fruits when they are ready.

   Here are good starting values for X and Y:
 X = 90 days
 Y = 180 days


 This is nice, but on the face of it it's just this: an idea on how other
 people should do things on their free time. I'd have difficulty convincing
 people they should work that way. The kind of ideas that I noticed are
 successful are those that actually carry the work through and serve as good
 examples to follow.


 Andrei


*sigh*



 May 04 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/4/14, 9:58 PM, Caligo via Digitalmars-d wrote:

 On Sun, May 4, 2014 at 11:09 PM, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:


     Great, a few representative links would be most welcome.


 Here is a good starting point (it's a classic):

 http://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar


Of course I've read that, and multiple times! I thought of mentioning it 
but concluded it would be trite. -- Andrei



 May 04 2014



prev sibling  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/4/14, 5:38 PM, Caligo via Digitalmars-d wrote:

 On Sun, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:
 Here is an idea:  include new features in DMD/Phobos as soon as they
 arrive, and make them part of the official binary release so that the
 average D user can try them out.  Make sure they are marked as unstable,
 and put a on/off switch on them (something like what Rust/Haskell have;
 not a compiler switch).  If the feature receives no implementation bug
 reports for X consecutive days AND no design bug reports for Y
 consecutive days, then the feature is marked stable and officially
 becomes part of DMD/Phobos.  The X and the Y can be decreased as D's
 number of users increases over the years.  The whole idea is very much
 like farming: you are planting seeds.  As the plants grow, some of them
 will not survive, others will be destroyed, and some of them will take
 years to grow.  In any case, you harvest the fruits when they are ready.

   Here are good starting values for X and Y:
 X = 90 days
 Y = 180 days



 This is nice, but on the face of it it's just this: an idea on how other
 people should do things on their free time. I'd have difficulty convincing
 people they should work that way. The kind of ideas that I noticed are
 successful are those that actually carry the work through and serve as good
 examples to follow.


There's imperfect but useful pull requests hanging around for years,
extern(Obj-C) for instance, which may be useful as an experimental
feature to many users, even if it's not ready for inclusion in the
official feature list and support.
I suspect it's (experimental) presence would stimulate further
contribution towards D on iOS for instance; it may be an enabler for
other potential contributors.

What about AST macros? It seems to me that this is never going to be
explored and there are competing proposals, but I wonder if there's
room for experimental implementations that anyone in the community can
toy with?

UDA's are super-useful, but they're still lacking the thing to really
set them off, which is the ability to introduce additional boilerplate
code at the site of the attribute.

I reckon there's a good chance that creating a proper platform for
experimental features would also have an advantage for community
building and increase contribution in general. If new contributors can
get in, have some fun, and start trying their ideas while also being
able to share them with the community for feedback without fear
they'll just be shot down and denied after all their work... are they
not more likely to actually make a contribution in the first place?
Once they've made a single contribution of any sort, are they then
more likely to continue making other contributions in the future
(having now taken the time to acclimatise themselves with the
codebase)?

I personally feel the perceived unlikeliness of any experimental
contribution being accepted is a massive deterrence to making compiler
contributions in the first place by anyone other than the most serious
OSS advocates. I have no prior experience with OSS, and it's certainly
a factor that's kept me at arms length.



 May 05 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/5/14, 8:19 PM, Manu via Digitalmars-d wrote:

 On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 This is nice, but on the face of it it's just this: an idea on how other
 people should do things on their free time. I'd have difficulty convincing
 people they should work that way. The kind of ideas that I noticed are
 successful are those that actually carry the work through and serve as good
 examples to follow.


 There's imperfect but useful pull requests hanging around for years,
 extern(Obj-C) for instance, which may be useful as an experimental
 feature to many users, even if it's not ready for inclusion in the
 official feature list and support.
 I suspect it's (experimental) presence would stimulate further
 contribution towards D on iOS for instance; it may be an enabler for
 other potential contributors.


So it would be nice if you reviewed that code.


 What about AST macros? It seems to me that this is never going to be
 explored and there are competing proposals, but I wonder if there's
 room for experimental implementations that anyone in the community can
 toy with?


There would be of course room as long as there's be one or more 
champions for it. Would that be something you'd be interested in?


 UDA's are super-useful, but they're still lacking the thing to really
 set them off, which is the ability to introduce additional boilerplate
 code at the site of the attribute.


Interesting. Have you worked on a related proposal?


 I reckon there's a good chance that creating a proper platform for
 experimental features would also have an advantage for community
 building and increase contribution in general. If new contributors can
 get in, have some fun, and start trying their ideas while also being
 able to share them with the community for feedback without fear
 they'll just be shot down and denied after all their work... are they
 not more likely to actually make a contribution in the first place?


I'd say so, but we'd need initiative and quite a bit of work for such a 
platform. Would you be interested?


 Once they've made a single contribution of any sort, are they then
 more likely to continue making other contributions in the future
 (having now taken the time to acclimatise themselves with the
 codebase)?


I agree - and that applies to you, too.


 I personally feel the perceived unlikeliness of any experimental
 contribution being accepted is a massive deterrence to making compiler
 contributions in the first place by anyone other than the most serious
 OSS advocates.


Contributions make it into the compiler and standard library if and they 
are properly motivated, well done, and reviewed by the core team which 
is literally self-appointed. The key to being on the core team is just 
reviewing contributions. Have you considered looking at submissions that 
are "hanging around for years"?


 I have no prior experience with OSS, and it's certainly
 a factor that's kept me at arms length.


It's as easy as just reviewing stuff. Acta, non verba.


Andrei



 May 05 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 6 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/5/14, 8:19 PM, Manu via Digitalmars-d wrote:

 On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:

 This is nice, but on the face of it it's just this: an idea on how other
 people should do things on their free time. I'd have difficulty
 convincing
 people they should work that way. The kind of ideas that I noticed are
 successful are those that actually carry the work through and serve as
 good
 examples to follow.



 There's imperfect but useful pull requests hanging around for years,
 extern(Obj-C) for instance, which may be useful as an experimental
 feature to many users, even if it's not ready for inclusion in the
 official feature list and support.
 I suspect it's (experimental) presence would stimulate further
 contribution towards D on iOS for instance; it may be an enabler for
 other potential contributors.



 So it would be nice if you reviewed that code.


I don't really know anything about it... and that's not the point.
I'm just suggesting by my prior email that some steps like creating an
experimental space with a lower barrier to entry might encourage
growth in the number of overall contributors, which I think was the
basic flavour of the emails leading up to it.


 What about AST macros? It seems to me that this is never going to be
 explored and there are competing proposals, but I wonder if there's
 room for experimental implementations that anyone in the community can
 toy with?



 There would be of course room as long as there's be one or more champions
 for it. Would that be something you'd be interested in?


I have no horse in that race, but I see it come up all the time, and
it is something I am passively interested in.

There's at least one DIP which received little attention afaict, it's
an example of something that I think would probably manifest into code
in an experimental space, but clearly couldn't be accepted as a
language feature without lots of field time.
In lieu of an experimental space, there will be no action.

It's an interesting example actually. I think lots feel the DIP isn't
really an effective solution, but nobody has the motivation or ideas
to refine it. The DIP author clearly has no motivation to test it
experimentally, but perhaps that's what it needs to progress? The
DIP's shortcomings might be discovered by experimental users in the
field? It's hard to know, but it's an example of the sort of things
that may have a stifling effect on progress and contribution.


 UDA's are super-useful, but they're still lacking the thing to really
 set them off, which is the ability to introduce additional boilerplate
 code at the site of the attribute.



 Interesting. Have you worked on a related proposal?


Not really, I've initiated numerous discussions which always seems to
end at AST macros.
The only other semi-reasonable idea I've had is the concept that
tagging mixin templates as UDA's might be a practical angle, but it
doesn't really make clean sense, creates a syntactic special case and
also doesn't seem powerful enough, so I'm not with any solid proposal
that I can imagine within the current language framework.

There are presently bigger issues that keep me awake at night.


 I reckon there's a good chance that creating a proper platform for
 experimental features would also have an advantage for community
 building and increase contribution in general. If new contributors can
 get in, have some fun, and start trying their ideas while also being
 able to share them with the community for feedback without fear
 they'll just be shot down and denied after all their work... are they
 not more likely to actually make a contribution in the first place?



 I'd say so, but we'd need initiative and quite a bit of work for such a
 platform. Would you be interested?


Well, in phobos, just approve 'exp' which has been raised countless
times. I've got contributions that should be in exp, but instead,
they're in limbo, and I've lost momentum and motivation since their
completion is blocked by other issues, and I'm receiving no feedback
from field testing.
What happened to std.serislisation? There was motion there a year or
so back... I was looking forward to it, and did some minor reviewing
at the time. I wonder if that's an interesting case study? (I haven't
looked)

In the compiler... I may be interested, but I don't have any such
compiler feature in mind to motivate the effort.
I have no idea what an experimental feature platform should look like
in the compiler, and if it were to exist, I have no such feature in
mind to make use of it, but I have raised examples of others that
have.


 Once they've made a single contribution of any sort, are they then
 more likely to continue making other contributions in the future
 (having now taken the time to acclimatise themselves with the
 codebase)?



 I agree - and that applies to you, too.


Sure, but my point... is below.


 I personally feel the perceived unlikeliness of any experimental
 contribution being accepted is a massive deterrence to making compiler
 contributions in the first place by anyone other than the most serious
 OSS advocates.



 Contributions make it into the compiler and standard library if and they are
 properly motivated, well done, and reviewed by the core team which is
 literally self-appointed. The key to being on the core team is just
 reviewing contributions. Have you considered looking at submissions that are
 "hanging around for years"?


Perhaps you misunderstood the point of my post. I've watched people
make solid contributions that haven't gotten through. That is
discouraging to others considering starting their own work, and for
the person who has already put in the effort to continue to do so in
the future.
The Obj-C thing as an example. Granted, it's a huge feature and has
extensive implications. The Authors have said themselves that they
agree it's not 'ready' for inclusion... so, what? It sits and rots?
I think it needs an experimental place to live and have people make
use of it for what it is. If it's blocked by other unpopular issues
that aren't receiving attention, perhaps it's presence will manifest
the appropriate motivation to see those other unpopular issues
resolved at some point?
My point is that successful OSS seems to be about enabling the
lowest-friction contribution, and my feeling right now, is that the
barrier to entry is high. Whether that's true or not I can't really
comment, but it's a perception, and the mental barrier inhibiting the
first step is perhaps the most significant barrier of all.

I can't review the Obj-C patch. 1, it's huge, 2, I don't know anything
about it, other than I'd really like to use D on iOS and that's a
major hurdle. Also, the authors themselves said they recognise it's
not 'ready'. But is it 'experimental'?


 I have no prior experience with OSS, and it's certainly
 a factor that's kept me at arms length.



 It's as easy as just reviewing stuff. Acta, non verba.


I've never felt I have any particular authority to comment on pulls
that I have no experience or vested interest in. (I do occasionally
comment on pull requests that I have some interest or knowledge in)
I've also had some (hopefully useful) commentary in features that did
make it; Win64, UDA's, some traits extensions, and lots of bug reports
and fix confirmations.

I'm plenty vocal and active on things I do feel I know about, but
they're often pretty radical, unpopular, and rarely come even close to
turning into code. I'm pretty certain that nothing left on my short
list that I personally *really* care about will ever get pulled, even
if I did do the work.
There's a perfectly good pull there for not-virtual-by-default. No
amount of beating will get that horse through, despite almost
unanimous community support. That was... extremely discouraging, to
say the least.



 May 05 2014



 next sibling parent reply "Wyatt" <wyatt.epp gmail.com>  writes:


On Tuesday, 6 May 2014 at 06:39:45 UTC, Manu via Digitalmars-d 
wrote:

 The Obj-C thing as an example. Granted, it's a huge feature and
 has extensive implications. The Authors have said themselves
 that they agree it's not 'ready' for inclusion... so, what? It
 sits and rots?  I think it needs an experimental place to live
 and have people make use of it for what it is.


This bit right here reminds me of something the Linux kernel has 
called "staging" [1].  It's basically what you describe: a 
subtree within main source tree for things to be publicly 
available while they finish baking, with the understanding that 
you're going to continue working on it and hopefully get it 
promoted to first-class citizen.


 I'm plenty vocal and active on things I do feel I know about,
 but they're often pretty radical, unpopular, and rarely come
 even close to turning into code. I'm pretty certain that nothing
 left on my short list that I personally *really* care about will
 ever get pulled, even if I did do the work.


I know I and several other people are still interested in 
std.simd... :(

-Wyatt

[1] http://lwn.net/Articles/324279/



 May 06 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 6 May 2014 22:17, Wyatt via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 06:39:45 UTC, Manu via Digitalmars-d wrote:

 The Obj-C thing as an example. Granted, it's a huge feature and
 has extensive implications. The Authors have said themselves
 that they agree it's not 'ready' for inclusion... so, what? It

 sits and rots?  I think it needs an experimental place to live
 and have people make use of it for what it is.



 This bit right here reminds me of something the Linux kernel has called
 "staging" [1].  It's basically what you describe: a subtree within main
 source tree for things to be publicly available while they finish baking,
 with the understanding that you're going to continue working on it and
 hopefully get it promoted to first-class citizen.



 I'm plenty vocal and active on things I do feel I know about,
 but they're often pretty radical, unpopular, and rarely come
 even close to turning into code. I'm pretty certain that nothing
 left on my short list that I personally *really* care about will
 ever get pulled, even if I did do the work.



 I know I and several other people are still interested in std.simd... :(


It hasn't gone away. It's a perfect example of where I got blocked by
some things, and since it wasn't in an accessible location as an
in-progress 'exp' module, it's barely used, and I've never gotten any
feedback. Perhaps the blocks are lifted now (I haven't checked
lately), but clearly I started working on other things and lost
momentum, and I think it's safe to attribute this almost entirely to
the fact it exists in my fork where nobody will find it, rather than
'exp', where people can still report usage experience, feedback, and
keep me on track.



 May 06 2014



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/6/14, 8:43 AM, Manu via Digitalmars-d wrote:

 On 6 May 2014 22:17, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 06:39:45 UTC, Manu via Digitalmars-d wrote:

 The Obj-C thing as an example. Granted, it's a huge feature and
 has extensive implications. The Authors have said themselves
 that they agree it's not 'ready' for inclusion... so, what? It

 sits and rots?  I think it needs an experimental place to live
 and have people make use of it for what it is.



 This bit right here reminds me of something the Linux kernel has called
 "staging" [1].  It's basically what you describe: a subtree within main
 source tree for things to be publicly available while they finish baking,
 with the understanding that you're going to continue working on it and
 hopefully get it promoted to first-class citizen.



 I'm plenty vocal and active on things I do feel I know about,
 but they're often pretty radical, unpopular, and rarely come
 even close to turning into code. I'm pretty certain that nothing
 left on my short list that I personally *really* care about will
 ever get pulled, even if I did do the work.



 I know I and several other people are still interested in std.simd... :(


 It hasn't gone away. It's a perfect example of where I got blocked by
 some things, and since it wasn't in an accessible location as an
 in-progress 'exp' module, it's barely used, and I've never gotten any
 feedback. Perhaps the blocks are lifted now (I haven't checked
 lately), but clearly I started working on other things and lost
 momentum, and I think it's safe to attribute this almost entirely to
 the fact it exists in my fork where nobody will find it, rather than
 'exp', where people can still report usage experience, feedback, and
 keep me on track.


I can't seem to find simd on our dub site 
http://code.dlang.org/search?q=simd. Did you put it there under another 
name?

Andrei



 May 06 2014



  parent reply "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Tuesday, 6 May 2014 at 15:48:59 UTC, Andrei Alexandrescu wrote:

 On 5/6/14, 8:43 AM, Manu via Digitalmars-d wrote:

 On 6 May 2014 22:17, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 06:39:45 UTC, Manu via 
 Digitalmars-d wrote:

 The Obj-C thing as an example. Granted, it's a huge feature 
 and
 has extensive implications. The Authors have said themselves
 that they agree it's not 'ready' for inclusion... so, what? 
 It

 sits and rots?  I think it needs an experimental place to 
 live
 and have people make use of it for what it is.



 This bit right here reminds me of something the Linux kernel 
 has called
 "staging" [1].  It's basically what you describe: a subtree 
 within main
 source tree for things to be publicly available while they 
 finish baking,
 with the understanding that you're going to continue working 
 on it and
 hopefully get it promoted to first-class citizen.



 I'm plenty vocal and active on things I do feel I know about,
 but they're often pretty radical, unpopular, and rarely come
 even close to turning into code. I'm pretty certain that 
 nothing
 left on my short list that I personally *really* care about 
 will
 ever get pulled, even if I did do the work.



 I know I and several other people are still interested in 
 std.simd... :(


 It hasn't gone away. It's a perfect example of where I got 
 blocked by
 some things, and since it wasn't in an accessible location as 
 an
 in-progress 'exp' module, it's barely used, and I've never 
 gotten any
 feedback. Perhaps the blocks are lifted now (I haven't checked
 lately), but clearly I started working on other things and lost
 momentum, and I think it's safe to attribute this almost 
 entirely to
 the fact it exists in my fork where nobody will find it, 
 rather than
 'exp', where people can still report usage experience, 
 feedback, and
 keep me on track.


 I can't seem to find simd on our dub site 
 http://code.dlang.org/search?q=simd. Did you put it there under 
 another name?

 Andrei


I don't think it's there. I would love if it was, it would be a 
great addition to the D ecosystem.



 May 06 2014



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 6 May 2014 at 15:52:10 UTC, John Colvin wrote:

 On Tuesday, 6 May 2014 at 15:48:59 UTC, Andrei Alexandrescu

 I can't seem to find simd on our dub site 
 http://code.dlang.org/search?q=simd. Did you put it there 
 under another name?

 Andrei


 I don't think it's there. I would love if it was, it would be a 
 great addition to the D ecosystem.


Which only confirms my suspicion that those who complain about 
lack of experimental Phobos package are likely to simply not use 
dub.



 May 06 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 7 May 2014 03:32, Dicebot via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 15:52:10 UTC, John Colvin wrote:

 On Tuesday, 6 May 2014 at 15:48:59 UTC, Andrei Alexandrescu

 I can't seem to find simd on our dub site
 http://code.dlang.org/search?q=simd. Did you put it there under another
 name?

 Andrei



 I don't think it's there. I would love if it was, it would be a great
 addition to the D ecosystem.



 Which only confirms my suspicion that those who complain about lack of
 experimental Phobos package are likely to simply not use dub.


As I said before, dub has never even occurred to me. No windows user
is likely to naturally think to use a package manager :/
It's sitting in my phobos fork. Isn't that where everyone keeps their
developments?

If people are pushing dub, then it should really be installed with DMD.



 May 06 2014



 next sibling parent  "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d 
wrote:

 On 7 May 2014 03:32, Dicebot via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 15:52:10 UTC, John Colvin wrote:

 On Tuesday, 6 May 2014 at 15:48:59 UTC, Andrei Alexandrescu

 I can't seem to find simd on our dub site
 http://code.dlang.org/search?q=simd. Did you put it there 
 under another
 name?

 Andrei



 I don't think it's there. I would love if it was, it would be 
 a great
 addition to the D ecosystem.



 Which only confirms my suspicion that those who complain about 
 lack of
 experimental Phobos package are likely to simply not use dub.


 As I said before, dub has never even occurred to me. No windows 
 user
 is likely to naturally think to use a package manager :/


Well, dub and code.dlang.org is a good way for them to start :p


 It's sitting in my phobos fork. Isn't that where everyone keeps 
 their
 developments?


not once they're done enough to be useful. See all of the 
projects: http://code.dlang.org/



 May 06 2014



prev sibling next sibling parent reply "w0rp" <devw0rp gmail.com>  writes:


On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d 
wrote:

 As I said before, dub has never even occurred to me. No windows 
 user
 is likely to naturally think to use a package manager :/
 It's sitting in my phobos fork. Isn't that where everyone keeps 
 their
 developments?

 If people are pushing dub, then it should really be installed 
 with DMD.


DUB should feel pretty natural on any operating system if you are 
a Python (PyPi), Ruby (Gems), JavaScript (NPM), Perl (CPAN), or 
PHP (PEAR) programmer. Although PyPi I know doesn't work as well 
on Windows. (It doesn't explain to you clearly how to install an 
MSVC compiler for PIP and what not.) I think it would be healthy 
to host supplementary modules first with DUB as third party 
modules, and then get them merged into Phobos after they have 
seen some real world use. That's sure to result in a faster 
turnaround time for going from experimentation to real usage.



 May 06 2014



 next sibling parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 06.05.2014 20:10, schrieb w0rp:

 On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d wrote:

 As I said before, dub has never even occurred to me. No windows user
 is likely to naturally think to use a package manager :/
 It's sitting in my phobos fork. Isn't that where everyone keeps their
 developments?

 If people are pushing dub, then it should really be installed with DMD.


 DUB should feel pretty natural on any operating system if you are a
 Python (PyPi), Ruby (Gems), JavaScript (NPM), Perl (CPAN), or PHP (PEAR)
 programmer. Although PyPi I know doesn't work as well on Windows. (It
 doesn't explain to you clearly how to install an MSVC compiler for PIP
 and what not.) I think it would be healthy to host supplementary modules
 first with DUB as third party modules, and then get them merged into
 Phobos after they have seen some real world use. That's sure to result
 in a faster turnaround time for going from experimentation to real usage.


That is the best approach.

--
Paulo



 May 06 2014



prev sibling  parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 7 May 2014 04:10, w0rp via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d wrote:

 As I said before, dub has never even occurred to me. No windows user
 is likely to naturally think to use a package manager :/
 It's sitting in my phobos fork. Isn't that where everyone keeps their
 developments?

 If people are pushing dub, then it should really be installed with DMD.



 DUB should feel pretty natural on any operating system if you are a Python
 (PyPi), Ruby (Gems), JavaScript (NPM), Perl (CPAN), or PHP (PEAR)
 programmer. Although PyPi I know doesn't work as well on Windows. (It
 doesn't explain to you clearly how to install an MSVC compiler for PIP and
 what not.) I think it would be healthy to host supplementary modules first
 with DUB as third party modules, and then get them merged into Phobos after
 they have seen some real world use. That's sure to result in a faster
 turnaround time for going from experimentation to real usage.


I'm not a user of any of those languages or cultures. Web feels like
an alien world to me.

I don't think suggesting that 'I should have known better' is good
defence for something that not obvious or intuitive to end-users.
You can embarrass me, but that doesn't improve the situation for users
who aren't present on this forum.
I'm not unhappy to use DUB to this end, it just never occurred to me.

I suggest it's only practical after these changes though:
1: dub is bundled with DMD
2: The paths for installed dub packages should be either automatic, or
requested/configured during the DMD installer.
3: DMD needs to know where to find DUB packages by default.

I think a massive bonus would be: VisualD/Mono-D integrate a repo
lister with an 'install package' button, and a 'feedback/bug report'
button (linking to the package repo?) for each package. **many
integration, such awesome, wow**
Has anyone done a dub gui already?



 May 06 2014



prev sibling next sibling parent  "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d 
wrote:

 Which only confirms my suspicion that those who complain about 
 lack of
 experimental Phobos package are likely to simply not use dub.


 As I said before, dub has never even occurred to me. No windows 
 user
 is likely to naturally think to use a package manager :/
 It's sitting in my phobos fork. Isn't that where everyone keeps 
 their
 developments?


It is usual way to handle Phobos proposals that are going into 
review queue. If you have a stand-alone module that is ready to 
be experimented with, you make an own repo for it and provide as 
dub package. This makes possible for users to fetch it easily and 
work with latest master version if desired without waiting for 
new DMD release.

I can't comment about Windows user mentality but seems like other 
language package managers are used quite a lot on Windows. 
Suggestions about making it more recognized are welcome.


 If people are pushing dub, then it should really be installed 
 with DMD.


It is the plan right now and I remember getting a conceptual 
agreement from Andrei about it. Problem is there are several 
things that need to be stabilized in it before such distribution 
may happen and Sonke is pretty much only person working on it 
continuously AFAICS. Any effort that is needed to maintain 
"exp.*" is better to be put into getting dub into desired state.

I can volunteer to manage actual dub inclusion process but 
problem is code needs  some work before it (and I have been doing 
close to 0 D programming other than my job lately).



 May 06 2014



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 06/05/14 19:57, Manu via Digitalmars-d wrote:


 As I said before, dub has never even occurred to me. No windows user
 is likely to naturally think to use a package manager :/


Doesn't Windows have NuGet?

-- 
/Jacob Carlborg



 May 06 2014



prev sibling  parent  "ponce" <contact gam3sfrommars.fr>  writes:


On Tuesday, 6 May 2014 at 15:43:22 UTC, Manu via Digitalmars-d 
wrote:

 I think it's safe to attribute this almost entirely to
 the fact it exists in my fork where nobody will find it, rather 
 than
 'exp', where people can still report usage experience, 
 feedback, and
 keep me on track.


As other people said, the dub registry is that 'exp' place.



 May 06 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/5/14, 11:39 PM, Manu via Digitalmars-d wrote:

 On 6 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/5/14, 8:19 PM, Manu via Digitalmars-d wrote:

 On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d


 So it would be nice if you reviewed that code.


 I don't really know anything about it... and that's not the point.
 I'm just suggesting by my prior email that some steps like creating an
 experimental space with a lower barrier to entry might encourage
 growth in the number of overall contributors, which I think was the
 basic flavour of the emails leading up to it.


Oh but that's very much the point. "Creating" is the right word (and the 
gerund form is delicious). In order for the "creating" to happen someone 
would need to do the "creation" - and of course the subsequent related 
work. You frame the matter as something "they" should do, when the 
reality is you're very much a part of "they".

The idea of an experimental/unstable package has been discussed a number 
of times. The conclusion I seem to remember is that currently with dub 
available there's a really simple way to experiment with things in a 
comfortable and decentralized way.

That said, it's not impossible that distributing things in an 
"std.experimental" package along with the official distribution gives 
them an extra umph. We could arrange for that to happen. Is this what 
you think would make the difference?


 What about AST macros? It seems to me that this is never going to be
 explored and there are competing proposals, but I wonder if there's
 room for experimental implementations that anyone in the community can
 toy with?



 There would be of course room as long as there's be one or more champions
 for it. Would that be something you'd be interested in?


 I have no horse in that race, but I see it come up all the time, and
 it is something I am passively interested in.


Same here. I'm too busy working in and on other aspects of D (which I 
believe are more important and more urgent) to have another project 
added to my plate. I'd be in your debt if you could clarify to me that 
AST macros are not something you believe I must work on right now.


 There's at least one DIP which received little attention afaict, it's
 an example of something that I think would probably manifest into code
 in an experimental space, but clearly couldn't be accepted as a
 language feature without lots of field time.
 In lieu of an experimental space, there will be no action.


What makes dub inappropriate as an experimental space?


 It's an interesting example actually. I think lots feel the DIP isn't
 really an effective solution, but nobody has the motivation or ideas
 to refine it. The DIP author clearly has no motivation to test it
 experimentally, but perhaps that's what it needs to progress? The
 DIP's shortcomings might be discovered by experimental users in the
 field? It's hard to know, but it's an example of the sort of things
 that may have a stifling effect on progress and contribution.


I think DIPs are a marked improvement over informal discussions on the 
mailing lists. For them to work well, two ingredients are needed: 
champions and reviewers. In some cases that did happen, in others it didn't.


 UDA's are super-useful, but they're still lacking the thing to really
 set them off, which is the ability to introduce additional boilerplate
 code at the site of the attribute.



 Interesting. Have you worked on a related proposal?


 Not really, I've initiated numerous discussions which always seems to
 end at AST macros.
 The only other semi-reasonable idea I've had is the concept that
 tagging mixin templates as UDA's might be a practical angle, but it
 doesn't really make clean sense, creates a syntactic special case and
 also doesn't seem powerful enough, so I'm not with any solid proposal
 that I can imagine within the current language framework.

 There are presently bigger issues that keep me awake at night.


Same here. So then I think you'd find it reasonable that I asked you to 
clarify these are not things you believe I need to work on.

What I'm trying to get across here is that your and my role and 
opportunities are very much comparable. Anyone's clout within the D 
community is proportional to the work they're doing on D, and even 
Walter's role would be reduced to an honorific one if he ceased spending 
time working on D.

My pedal is sealed to the floor already. I'm tapped out. Walter's too. 
Meaning that anything work you'd find fit to give us to do would need to 
come at other work's expense. In my case, that would mean overseeing 
four D projects at work and allocators on my free time. Walter's last 
piece of work is interfacing with C++ namespaces, which is very 
necessary for adoption. He and I have gotten pretty good at investing 
our time efficiently, but at some point there's only so many hours in a 
day and we need talented people - people like you - to help making 
things happen.


 I reckon there's a good chance that creating a proper platform for
 experimental features would also have an advantage for community
 building and increase contribution in general. If new contributors can
 get in, have some fun, and start trying their ideas while also being
 able to share them with the community for feedback without fear
 they'll just be shot down and denied after all their work... are they
 not more likely to actually make a contribution in the first place?



 I'd say so, but we'd need initiative and quite a bit of work for such a
 platform. Would you be interested?


 Well, in phobos, just approve 'exp' which has been raised countless
 times. I've got contributions that should be in exp, but instead,
 they're in limbo, and I've lost momentum and motivation since their
 completion is blocked by other issues, and I'm receiving no feedback
 from field testing.


I'm fine with adding an "exp" package to phobos if you believe that 
that's what's stopping things from happening. But do you really?


 What happened to std.serislisation? There was motion there a year or
 so back... I was looking forward to it, and did some minor reviewing
 at the time. I wonder if that's an interesting case study? (I haven't
 looked)


I could ask you the very same thing. What happened to std.serialization? 
Do you want me to work on that as well?


 In the compiler... I may be interested, but I don't have any such
 compiler feature in mind to motivate the effort.
 I have no idea what an experimental feature platform should look like
 in the compiler, and if it were to exist, I have no such feature in
 mind to make use of it, but I have raised examples of others that
 have.


Well then someone else would need to do all that work, so you'd need to 
convince them that it's a good idea.


 Once they've made a single contribution of any sort, are they then
 more likely to continue making other contributions in the future
 (having now taken the time to acclimatise themselves with the
 codebase)?



 I agree - and that applies to you, too.


 Sure, but my point... is below.


 I personally feel the perceived unlikeliness of any experimental
 contribution being accepted is a massive deterrence to making compiler
 contributions in the first place by anyone other than the most serious
 OSS advocates.



 Contributions make it into the compiler and standard library if and they are
 properly motivated, well done, and reviewed by the core team which is
 literally self-appointed. The key to being on the core team is just
 reviewing contributions. Have you considered looking at submissions that are
 "hanging around for years"?


 Perhaps you misunderstood the point of my post. I've watched people
 make solid contributions that haven't gotten through. That is
 discouraging to others considering starting their own work, and for
 the person who has already put in the effort to continue to do so in
 the future.


Contributions haven't gotten through because there are not enough people 
to review them. People like you. We have an inflation of contributions, 
or if you wish a deflation of reviews.


 The Obj-C thing as an example. Granted, it's a huge feature and has
 extensive implications. The Authors have said themselves that they
 agree it's not 'ready' for inclusion... so, what? It sits and rots?
 I think it needs an experimental place to live and have people make
 use of it for what it is. If it's blocked by other unpopular issues
 that aren't receiving attention, perhaps it's presence will manifest
 the appropriate motivation to see those other unpopular issues
 resolved at some point?


(I assume you refer to http://wiki.dlang.org/DIP43; couldn't find 
associated pull requests.) I don't know. I am not using Objective C, and 
I think it would be more efficient to have people who already have 
Objective C expertise to author and review the contribution. Would I be 
glad to see good interfacing of D with Objective C? Of course!


 My point is that successful OSS seems to be about enabling the
 lowest-friction contribution, and my feeling right now, is that the
 barrier to entry is high. Whether that's true or not I can't really
 comment, but it's a perception, and the mental barrier inhibiting the
 first step is perhaps the most significant barrier of all.


Forgive me but from where I stand you seem completely wrong. There are 
currently 190 open pull requests in our github projects. The primary 
reason that keeps them from being merged is good peer review from people 
like you and me. So currently the bottleneck is on the receiving side, 
not the initiating side of contributions.

You can't claim incompetence on 190 distinct matters. You are fluent in 
D and a good engineer who could tell good design from bad.


 I can't review the Obj-C patch. 1, it's huge, 2, I don't know anything
 about it, other than I'd really like to use D on iOS and that's a
 major hurdle.


Surely there must be something that should catch your fancy among the 
remaining 189 patches.


 Also, the authors themselves said they recognise it's
 not 'ready'. But is it 'experimental'?


Most likely. Have you tried to build their fork?


 I have no prior experience with OSS, and it's certainly
 a factor that's kept me at arms length.



 It's as easy as just reviewing stuff. Acta, non verba.


 I've never felt I have any particular authority to comment on pulls
 that I have no experience or vested interest in. (I do occasionally
 comment on pull requests that I have some interest or knowledge in)
 I've also had some (hopefully useful) commentary in features that did
 make it; Win64, UDA's, some traits extensions, and lots of bug reports
 and fix confirmations.


That's a great start, thanks. Keep them coming.


 I'm plenty vocal and active on things I do feel I know about, but
 they're often pretty radical, unpopular, and rarely come even close to
 turning into code. I'm pretty certain that nothing left on my short
 list that I personally *really* care about will ever get pulled, even
 if I did do the work.
 There's a perfectly good pull there for not-virtual-by-default. No
 amount of beating will get that horse through, despite almost
 unanimous community support. That was... extremely discouraging, to
 say the least.


What can we do (short of pulling that) to be more encouraging of you to 
do good work?


Andrei



 May 06 2014



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 7 May 2014 01:46, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/5/14, 11:39 PM, Manu via Digitalmars-d wrote:

 On 6 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/5/14, 8:19 PM, Manu via Digitalmars-d wrote:

 On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d


 So it would be nice if you reviewed that code.



 I don't really know anything about it... and that's not the point.
 I'm just suggesting by my prior email that some steps like creating an
 experimental space with a lower barrier to entry might encourage
 growth in the number of overall contributors, which I think was the
 basic flavour of the emails leading up to it.



 Oh but that's very much the point. "Creating" is the right word (and the
 gerund form is delicious). In order for the "creating" to happen someone
 would need to do the "creation" - and of course the subsequent related work.
 You frame the matter as something "they" should do, when the reality is
 you're very much a part of "they".


Throughout this email, you seem to have completely misunderstood my
intent. I haven't asked you to do anything. I originally replied in
support of prior comments that raised yet again the idea of having an
experimental space with significantly relaxed contribution policy.
I was supporting the notion that an agreement to the existence of an
experimental space might be a valuable thing. It requires nothing more
than discussion and agreement that it is actually something the
community thinks is a good idea.


 The idea of an experimental/unstable package has been discussed a number of
 times. The conclusion I seem to remember is that currently with dub
 available there's a really simple way to experiment with things in a
 comfortable and decentralized way.


Is that something that's general knowledge, or a strategy that people
are already using? This is the first time the idea has ever occurred
to me.


 That said, it's not impossible that distributing things in an
 "std.experimental" package along with the official distribution gives them
 an extra umph. We could arrange for that to happen. Is this what you think
 would make the difference?


Yes. This is basically the entire point of my post.


 What about AST macros? It seems to me that this is never going to be
 explored and there are competing proposals, but I wonder if there's
 room for experimental implementations that anyone in the community can
 toy with?




 There would be of course room as long as there's be one or more champions
 for it. Would that be something you'd be interested in?



 I have no horse in that race, but I see it come up all the time, and
 it is something I am passively interested in.



 Same here. I'm too busy working in and on other aspects of D (which I
 believe are more important and more urgent) to have another project added to
 my plate. I'd be in your debt if you could clarify to me that AST macros are
 not something you believe I must work on right now.


I haven't suggested you need to work on anything. Other people with
ideas on the topic have appeared to be interested in working on it,
and it seems they need a place where they can produce the experimental
system and make it available to everyone.
I wonder if that would motivate them to do it.

You can see how there's a distinction between contributors getting a
system to an experimental point and then letting people have at it
while it bakes, and getting it to the point it's perfect and ready for
formal review and acceptance, prior to people really having convenient
access to it to try it out?

I'm not sure how you perceive that I'm saying you need to do something
in this scenario, other than consider some changes in contribution
policy as part of a committee.


 There's at least one DIP which received little attention afaict, it's
 an example of something that I think would probably manifest into code
 in an experimental space, but clearly couldn't be accepted as a
 language feature without lots of field time.
 In lieu of an experimental space, there will be no action.



 What makes dub inappropriate as an experimental space?


I don't know. But at face value, I'd suggest that it's a concept
that's quite foreign to Windows users. It never occurred to me until
right now.
If that's a popular approach, then the the dub package listing
probably needs greater visibility. It well maintained?

dub should almost certainly be bundled with DMD if this is to become a
standard approach.

This only addresses libs though, not experimental compiler features.
That needs an agreed policy for enabling/disabling such features, and
I don't see how that can escape living in the DMD repo?


 It's an interesting example actually. I think lots feel the DIP isn't
 really an effective solution, but nobody has the motivation or ideas
 to refine it. The DIP author clearly has no motivation to test it
 experimentally, but perhaps that's what it needs to progress? The
 DIP's shortcomings might be discovered by experimental users in the
 field? It's hard to know, but it's an example of the sort of things
 that may have a stifling effect on progress and contribution.



 I think DIPs are a marked improvement over informal discussions on the
 mailing lists. For them to work well, two ingredients are needed: champions
 and reviewers. In some cases that did happen, in others it didn't.


Wholesale inclusion is different than getting something in as an
experimental feature though. If there's a proper policy for
experimental features, and there is a much lower standard against
which they are included, then there'll possibly be more people
experimenting. 10x so for compiler features, since most people don't
build their own compiler; unless experimental compiler features are
included, they'll never be tested by end-users.

I'm not suggesting any change to the DIP system, it's fine, and a good
starting point for new development. But I think there's a lot of DIP's
there which may be interesting features, and I wonder if there would
be a difference in perceived friction if they knew there was an
experimental stage available where the community can initially try new
things out, offer useful feedback, and motivating the push to the
goalpost.

Ideally, experimental features should be available to all D users for
maximum possible user testing. Only people on this forum would ever
find someone's personal fork, merge with their local source and
compile themselves a compiler. Most people even here would never do
that.


 UDA's are super-useful, but they're still lacking the thing to really
 set them off, which is the ability to introduce additional boilerplate
 code at the site of the attribute.




 Interesting. Have you worked on a related proposal?



 Not really, I've initiated numerous discussions which always seems to
 end at AST macros.
 The only other semi-reasonable idea I've had is the concept that
 tagging mixin templates as UDA's might be a practical angle, but it
 doesn't really make clean sense, creates a syntactic special case and
 also doesn't seem powerful enough, so I'm not with any solid proposal
 that I can imagine within the current language framework.

 There are presently bigger issues that keep me awake at night.



 Same here. So then I think you'd find it reasonable that I asked you to
 clarify these are not things you believe I need to work on.


I'm not asking you to work on anything.


 What I'm trying to get across here is that your and my role and
 opportunities are very much comparable. Anyone's clout within the D
 community is proportional to the work they're doing on D, and even Walter's
 role would be reduced to an honorific one if he ceased spending time working
 on D.

 My pedal is sealed to the floor already. I'm tapped out. Walter's too.
 Meaning that anything work you'd find fit to give us to do would need to
 come at other work's expense. In my case, that would mean overseeing four D
 projects at work and allocators on my free time. Walter's last piece of work
 is interfacing with C++ namespaces, which is very necessary for adoption. He
 and I have gotten pretty good at investing our time efficiently, but at some
 point there's only so many hours in a day and we need talented people -
 people like you - to help making things happen.


I'm not sure what you think I'm asking you to do.
I'm just suggesting to consider a change in policy wrt experimental
contributions.
My suggestions are:

Allow people to commit to an 'exp' phobos package; reduce the approval
burden for people that want to make experimental modules available.
This is a no-action action. Just a decision that it's a good idea or
not.
I imagine criteria for inclusion as an experimental package might be
something like; as long as a decent amount of people agree that it is
a feature that should eventually be in phobos, and that the initial
API is sensible and reasonably conformant, let it exist in exp while
it bakes, prior to the lengthy formal review process.

Allow people to commit experimental features to the compiler, like
Obj-C, so people can try it. There will need to be some agreement on
how experimental compiler features are to be enabled/disabled in a
standard way, and obviously approval would be subject to reasonable
community approval. Again, I don't think that implies any work on your
part.

The difference is, there's no massive threads and debates about what
something should or shouldn't be, and people can get to work, try
things, make it available, get feedback, etc, prior to a formal review
process.
I'd also suggest that experimental code in phobos or the compiler not
be a required burden for GDC/LDC if it runs into trouble with their
update process. Policy should allow them to stub it out, and refer it
back to the exp author.


 I reckon there's a good chance that creating a proper platform for
 experimental features would also have an advantage for community
 building and increase contribution in general. If new contributors can
 get in, have some fun, and start trying their ideas while also being
 able to share them with the community for feedback without fear
 they'll just be shot down and denied after all their work... are they
 not more likely to actually make a contribution in the first place?




 I'd say so, but we'd need initiative and quite a bit of work for such a
 platform. Would you be interested?



 Well, in phobos, just approve 'exp' which has been raised countless
 times. I've got contributions that should be in exp, but instead,
 they're in limbo, and I've lost momentum and motivation since their
 completion is blocked by other issues, and I'm receiving no feedback
 from field testing.



 I'm fine with adding an "exp" package to phobos if you believe that that's
 what's stopping things from happening. But do you really?



 What happened to std.serislisation? There was motion there a year or
 so back... I was looking forward to it, and did some minor reviewing
 at the time. I wonder if that's an interesting case study? (I haven't
 looked)



 I could ask you the very same thing. What happened to std.serialization? Do
 you want me to work on that as well?


I haven't asked you to work on anything!
I just raised it as a potential example where something might have
gone off the rails because there wasn't an effective experimental
place for it to go while baking.


 In the compiler... I may be interested, but I don't have any such
 compiler feature in mind to motivate the effort.
 I have no idea what an experimental feature platform should look like
 in the compiler, and if it were to exist, I have no such feature in
 mind to make use of it, but I have raised examples of others that
 have.



 Well then someone else would need to do all that work, so you'd need to
 convince them that it's a good idea.


I don't know what work you refer to. Maybe there is work to do to
support this? I don't know. It seems more like a policy decision to
me.


 I personally feel the perceived unlikeliness of any experimental
 contribution being accepted is a massive deterrence to making compiler
 contributions in the first place by anyone other than the most serious
 OSS advocates.




 Contributions make it into the compiler and standard library if and they
 are
 properly motivated, well done, and reviewed by the core team which is
 literally self-appointed. The key to being on the core team is just
 reviewing contributions. Have you considered looking at submissions that
 are
 "hanging around for years"?



 Perhaps you misunderstood the point of my post. I've watched people
 make solid contributions that haven't gotten through. That is
 discouraging to others considering starting their own work, and for
 the person who has already put in the effort to continue to do so in
 the future.



 Contributions haven't gotten through because there are not enough people to
 review them. People like you. We have an inflation of contributions, or if
 you wish a deflation of reviews.


And again, maybe they would have more inertia if they made it 'in' at
an earlier point, as an experimental, and they built a community of
users behind the feature proving it's value, and providing feedback
and motivation?
Experimentals shouldn't require extensive reviews. Basic sanity checks
and code quality compliance are much easier for any uninvested 3rd
party to review/verify.


 The Obj-C thing as an example. Granted, it's a huge feature and has
 extensive implications. The Authors have said themselves that they
 agree it's not 'ready' for inclusion... so, what? It sits and rots?
 I think it needs an experimental place to live and have people make
 use of it for what it is. If it's blocked by other unpopular issues
 that aren't receiving attention, perhaps it's presence will manifest
 the appropriate motivation to see those other unpopular issues
 resolved at some point?



 (I assume you refer to http://wiki.dlang.org/DIP43; couldn't find associated
 pull requests.) I don't know. I am not using Objective C, and I think it
 would be more efficient to have people who already have Objective C
 expertise to author and review the contribution. Would I be glad to see good
 interfacing of D with Objective C? Of course!


But, the question is, should it be accepted as an 'experimental'
(unfinished) feature, that people can access on some flag or whatever?
This particular one has been resurrected multiple times, but always
gets stuck. Would there be more pressure to finish the job if people
were actively using it, despite the fact it's yet incomplete?


 My point is that successful OSS seems to be about enabling the
 lowest-friction contribution, and my feeling right now, is that the
 barrier to entry is high. Whether that's true or not I can't really
 comment, but it's a perception, and the mental barrier inhibiting the
 first step is perhaps the most significant barrier of all.



 Forgive me but from where I stand you seem completely wrong. There are
 currently 190 open pull requests in our github projects. The primary reason
 that keeps them from being merged is good peer review from people like you
 and me. So currently the bottleneck is on the receiving side, not the
 initiating side of contributions.


This is probably true.
I can imagine that I'd feel much more confident and qualified to
review features for inclusion as experimentals, since it should only
require quality compliance and sanity checking, and not deep
understanding of the feature itself, potential interaction with other
features or code that I'm unfamiliar with, etc.


 You can't claim incompetence on 190 distinct matters. You are fluent in D
 and a good engineer who could tell good design from bad.


I'm not even sure what the process it... if I go through and "LGTM" a
bunch of pulls, does someone accept my judgement and click the merge
button?
You can see why I might not feel qualified to do such a thing?

I'll criticise at least 10 pull requests in the morning. I'll be
curious to see what happens.


 I can't review the Obj-C patch. 1, it's huge, 2, I don't know anything
 about it, other than I'd really like to use D on iOS and that's a
 major hurdle.



 Surely there must be something that should catch your fancy among the
 remaining 189 patches.


We'll find out... when I wake up.


 Also, the authors themselves said they recognise it's
 not 'ready'. But is it 'experimental'?



 Most likely. Have you tried to build their fork?


I download DMD from dlang.org. It would be nice if end-users could try
it out under these conventional usage scenarios.
Are they welcome to have the feature enabled on a switch or pragma or
something, and merge into trunk such that it is available to end-users
in the next release?


 I have no prior experience with OSS, and it's certainly
 a factor that's kept me at arms length.




 It's as easy as just reviewing stuff. Acta, non verba.



 I've never felt I have any particular authority to comment on pulls
 that I have no experience or vested interest in. (I do occasionally
 comment on pull requests that I have some interest or knowledge in)
 I've also had some (hopefully useful) commentary in features that did
 make it; Win64, UDA's, some traits extensions, and lots of bug reports
 and fix confirmations.



 That's a great start, thanks. Keep them coming.


 I'm plenty vocal and active on things I do feel I know about, but
 they're often pretty radical, unpopular, and rarely come even close to
 turning into code. I'm pretty certain that nothing left on my short
 list that I personally *really* care about will ever get pulled, even
 if I did do the work.
 There's a perfectly good pull there for not-virtual-by-default. No
 amount of beating will get that horse through, despite almost
 unanimous community support. That was... extremely discouraging, to
 say the least.



 What can we do (short of pulling that) to be more encouraging of you to do
 good work?


Weigh in on policy relating to an experimental feature space in the
compiler, and in phobos?
How should experimental compiler features be enabled? pragma? switches?
How should experimental libraries be presented? phobos 'exp'? dub?
These are just conversations and decisions.

It's this simple; I (and apparently others, it wasn't my idea) think
an experimental space would be good. Do you agree? It's an idea that
has always been rejected in the past.
My logic is, such a thing may theoretically reduce friction and lower
the bar for new development, and maybe stimulate new contributions and
ideally contributors.



 May 06 2014



 next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Tuesday, 6 May 2014 at 17:47:31 UTC, Manu via Digitalmars-d 
wrote:

 Is that something that's general knowledge, or a strategy that 
 people
 are already using? This is the first time the idea has ever 
 occurred
 to me.



 I don't know. But at face value, I'd suggest that it's a concept
 that's quite foreign to Windows users. It never occurred to me 
 until
 right now.


It never occurred to you that people's libraries would be 
published as part of a centralised repository with a tool that 
manages dependencies? It's pretty common-place in a variety of 
languages. (https://rubygems.org/ https://pypi.python.org/pypi 
http://www.cpan.org/ etc...).

With regard to experimental modules, it's good to get them some 
real-world exposure first via dub even if there was some 
"staging" package in phobos.


 If that's a popular approach, then the the dub package listing
 probably needs greater visibility. It well maintained?

 dub should almost certainly be bundled with DMD if this is to 
 become a
 standard approach.


It's been discussed a few times. I think greater stability was 
wanted before bundling.


 This only addresses libs though, not experimental compiler 
 features.
 That needs an agreed policy for enabling/disabling such 
 features, and
 I don't see how that can escape living in the DMD repo?


Good point.



 May 06 2014



  parent reply "Wyatt" <wyatt.epp gmail.com>  writes:


On Tuesday, 6 May 2014 at 18:02:46 UTC, John Colvin wrote:

 It never occurred to you that people's libraries would be 
 published as part of a centralised repository with a tool that 
 manages dependencies?


Hate to be the cynic, but how in the world do you expect people 
to even know about Dub or code.dlang.org in the first place?  I 
don't see it linked from any of the "obvious" places on 
dlang.org, and I can't even find a single _mention_ that we 
apparently have a package manager.  Nothing in the FAQ about 
"Contributing to D".  And as if all that wasn't enough, the 
"Links" page still points to digitalmars.com.

 From a normal user's standpoint, they simply don't exist.


 It's pretty common-place in a variety of languages. 
 (https://rubygems.org/ https://pypi.python.org/pypi 
 http://www.cpan.org/ etc...).


It's rather disingenuous to invoke these three.

Ruby: "Libraries" at the top on the home page links to 
https://www.ruby-lang.org/en/libraries/, which explains gem and 
links to rubygems
Python: Not super easy to see, but "PyPI" _is_ linked in the top 
bar.
Perl: "CPAN" in the top bar links to 
http://www.perl.org/cpan.html, which explains cpan ...and I think 
you can see where this is going.

I love my package manager, but I'm going to have to agree with 
Manu's bewilderment here.

-Wyatt



 May 06 2014



 next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/6/14, 12:18 PM, Wyatt wrote:

 On Tuesday, 6 May 2014 at 18:02:46 UTC, John Colvin wrote:

 It never occurred to you that people's libraries would be published as
 part of a centralised repository with a tool that manages dependencies?


 Hate to be the cynic, but how in the world do you expect people to even
 know about Dub or code.dlang.org in the first place?  I don't see it
 linked from any of the "obvious" places on dlang.org


It's an oversight - code.dlang.org is fairly recent so there are things 
to be still ironed out. Please submit a pull request to 
https://github.com/D-Programming-Language/dlang.org to include 
code.lang.org in the left navigation panel appropriately. -- Andrei



 May 06 2014



prev sibling  parent reply "Meta" <jared771 gmail.com>  writes:


On Tuesday, 6 May 2014 at 19:18:08 UTC, Wyatt wrote:

 On Tuesday, 6 May 2014 at 18:02:46 UTC, John Colvin wrote:

 It never occurred to you that people's libraries would be 
 published as part of a centralised repository with a tool that 
 manages dependencies?


 Hate to be the cynic, but how in the world do you expect people 
 to even know about Dub or code.dlang.org in the first place?  I 
 don't see it linked from any of the "obvious" places on 
 dlang.org, and I can't even find a single _mention_ that we 
 apparently have a package manager.  Nothing in the FAQ about 
 "Contributing to D".  And as if all that wasn't enough, the 
 "Links" page still points to digitalmars.com.

 From a normal user's standpoint, they simply don't exist.


 It's pretty common-place in a variety of languages. 
 (https://rubygems.org/ https://pypi.python.org/pypi 
 http://www.cpan.org/ etc...).


 It's rather disingenuous to invoke these three.

 Ruby: "Libraries" at the top on the home page links to 
 https://www.ruby-lang.org/en/libraries/, which explains gem and 
 links to rubygems
 Python: Not super easy to see, but "PyPI" _is_ linked in the 
 top bar.
 Perl: "CPAN" in the top bar links to 
 http://www.perl.org/cpan.html, which explains cpan ...and I 
 think you can see where this is going.

 I love my package manager, but I'm going to have to agree with 
 Manu's bewilderment here.

 -Wyatt


I believe this is the case. If I did not stay vaguely informed of 
the latest trends with the various popular dynamic languages, 
i.e., Python, Ruby, NodeJS, basically anything popular with the 
Silicon Valley hipsters (no offense to users of those languages), 
I'd have no idea what a language-based package manager is, much 
less why you'd want to use one when the various *nix have it 
built in. If you work in the game industry, then your primary 
language is probably C++ with a bit of Lua on the side. Do those 
languages even *have* package managers? Even if you understand 
how that all works, Dub isn't advertised at all, as you said.



 May 06 2014



 next sibling parent  "Meta" <jared771 gmail.com>  writes:


On that note, 
http://code.dlang.org/?sort=updated&category=library.std_aspirant.



 May 06 2014



prev sibling  parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 6 May 2014 at 22:48:02 UTC, Meta wrote:

 On Tuesday, 6 May 2014 at 19:18:08 UTC, Wyatt wrote:

 On Tuesday, 6 May 2014 at 18:02:46 UTC, John Colvin wrote:

 It never occurred to you that people's libraries would be 
 published as part of a centralised repository with a tool 
 that manages dependencies?


 Hate to be the cynic, but how in the world do you expect 
 people to even know about Dub or code.dlang.org in the first 
 place?  I don't see it linked from any of the "obvious" places 
 on dlang.org, and I can't even find a single _mention_ that we 
 apparently have a package manager.  Nothing in the FAQ about 
 "Contributing to D".  And as if all that wasn't enough, the 
 "Links" page still points to digitalmars.com.

 From a normal user's standpoint, they simply don't exist.


 It's pretty common-place in a variety of languages. 
 (https://rubygems.org/ https://pypi.python.org/pypi 
 http://www.cpan.org/ etc...).


 It's rather disingenuous to invoke these three.

 Ruby: "Libraries" at the top on the home page links to 
 https://www.ruby-lang.org/en/libraries/, which explains gem 
 and links to rubygems
 Python: Not super easy to see, but "PyPI" _is_ linked in the 
 top bar.
 Perl: "CPAN" in the top bar links to 
 http://www.perl.org/cpan.html, which explains cpan ...and I 
 think you can see where this is going.

 I love my package manager, but I'm going to have to agree with 
 Manu's bewilderment here.

 -Wyatt


 I believe this is the case. If I did not stay vaguely informed 
 of the latest trends with the various popular dynamic 
 languages, i.e., Python, Ruby, NodeJS, basically anything 
 popular with the Silicon Valley hipsters (no offense to users 
 of those languages), I'd have no idea what a language-based 
 package manager is, much less why you'd want to use one when 
 the various *nix have it built in. If you work in the game 
 industry, then your primary language is probably C++ with a bit 
 of Lua on the side. Do those languages even *have* package 
 managers? Even if you understand how that all works, Dub isn't 
 advertised at all, as you said.


On Windows NuGet supports all Visual Studio languages, including
C++, since Visual Studio 2010.

For Lua there are LuaRocks and LuaDist.

A *nix package manager is brain dead idea for software
development as it ties the language libraries to the specific OS
one is using.

Good luck getting packages if the author did not consider your
OS. Specially if they are only available in binary format, as it
is standard in the enterprise world.

With a language pakage manager I can produce package XPTO that
will work on all OS, it won't conflict with the system packages,
specially important on servers used for CI of multiple projects.

--
Paulo



 May 06 2014



 next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Wednesday, 7 May 2014 at 06:50:34 UTC, Paulo Pinto wrote:

 A *nix package manager is brain dead idea for software
 development as it ties the language libraries to the specific OS
 one is using.


The difference is that you are more vulnerable by getting 
software from language specific repositories.



 May 07 2014



  parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Wednesday, 7 May 2014 at 07:26:34 UTC, Ola Fosheim Grøstad 
wrote:

 On Wednesday, 7 May 2014 at 06:50:34 UTC, Paulo Pinto wrote:

 A *nix package manager is brain dead idea for software
 development as it ties the language libraries to the specific 
 OS
 one is using.


 The difference is that you are more vulnerable by getting 
 software from language specific repositories.


This is for software development, not end user software.

--
Paulo



 May 07 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Wednesday, 7 May 2014 at 12:47:05 UTC, Paulo Pinto wrote:

 This is for software development, not end user software.


Which makes avoiding trojans or rogue libraries even more 
critical. I avoid sources with low volume, no auditing and 
questionable authentication for commercial development. Basically 
it means downloading from the original author or using a 
distribution like Debian... (or creating a separate account, 
which I have done for D).



 May 07 2014



prev sibling  parent  Marco Leise <Marco.Leise gmx.de>  writes:


Am Wed, 07 May 2014 06:50:33 +0000
schrieb "Paulo Pinto" <pjmlp progtools.org>:


 A *nix package manager is brain dead idea for software
 development as it ties the language libraries to the specific OS
 one is using.


What do you have in mind here? I don't quite get the picture.
Are you opposed to a developer writing a package only for his
pet distribution of Linux and ignoring all others?

The typical packages I know come with some configure script
and offer enough hooks to custom tailer the installation, so
the library/application can work on any distribution.
Most of the Linux software is open source and and their
packages are maintained by the community around the specific
distribution. That doesn't preclude the use of package
managers like Cabal, CPAN, Maven, you-name-it. But for a
systems language integration with the existing C/C++ is of
utmost importance. After all, D compiles to 'regular' native
binaries. Typically when an application is useful, someone
will add it to their favorite distribution as a package
including all the libraries as dependencies.


 Good luck getting packages if the author did not consider your
 OS. Specially if they are only available in binary format, as it
 is standard in the enterprise world.


Then use dub. Oh wait... the packages on code.dlang.org are
open source, too. And at least since the curl debacle we know
that there is not one binary for all *nix systems. I don't
know where you are trying to get with this argument. I think
it has nothing to do with what dub strives for and is worth a
separate topic "Binary D library distribution".


 With a language pakage manager I can produce package XPTO that
 will work on all OS, it won't conflict with the system packages,
 specially important on servers used for CI of multiple projects.
 
 --
 Paulo


What is this XPTO that will magically work on all OS? I never
heard of it, but I'm almost certain it has to do with
languages that compile at most to machine independent byte
code.
And why do you run CI on the live system instead of a chroot
environment, if you are afraid of messing it up? :) I do trust
my package manager to correctly install libraries into a
chroot. It is as simple as prepending an environment variable
override. As a bonus you can then cleanly uninstall/update
libs in the chroot environment with all the sanity checks the
package manager may offer.
A language package manager is a good idea, but there are
certain limits for it you leave the development stage. At that
point the system package manager takes over. Both should be
considered with equal care.

-- 
Marco



 May 11 2014



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/6/2014 10:47 AM, Manu via Digitalmars-d wrote:

 On 7 May 2014 01:46, Andrei Alexandrescu via Digitalmars-d
 I'm not even sure what the process it... if I go through and "LGTM" a
 bunch of pulls, does someone accept my judgement and click the merge
 button?
 You can see why I might not feel qualified to do such a thing?


You don't need to be qualified (although you certainly are) to review PR's. The 
process is anyone can review/comment on them. Non-language-changing PR's can be 
pulled by anyone on "Team DMD". Language changing PR's need to be approved by 
Andrei and I.

"Team DMD" consists of people who have a consistent history of doing solid work 
reviewing PR's.



 I'll criticise at least 10 pull requests in the morning. I'll be
 curious to see what happens.


Great!



 I can't review the Obj-C patch. 1, it's huge, 2, I don't know anything
 about it, other than I'd really like to use D on iOS and that's a
 major hurdle.



 Surely there must be something that should catch your fancy among the
 remaining 189 patches.


 We'll find out... when I wake up.


You don't actually have to understand the entire PR in order to do reviews. You 
can do micro reviews like "needs explanatory comment", "this line is bad
style", 
"this section looks like a buffer overflow", "bad formatting", "don't use 
gets()", etc.



 May 06 2014



  parent  luka8088 <luka8088 owave.net>  writes:


On 6.5.2014. 20:10, Walter Bright wrote:

 On 5/6/2014 10:47 AM, Manu via Digitalmars-d wrote:

 On 7 May 2014 01:46, Andrei Alexandrescu via Digitalmars-d
 I'm not even sure what the process it... if I go through and "LGTM" a
 bunch of pulls, does someone accept my judgement and click the merge
 button?
 You can see why I might not feel qualified to do such a thing?


 
 You don't need to be qualified (although you certainly are) to review
 PR's. The process is anyone can review/comment on them.
 Non-language-changing PR's can be pulled by anyone on "Team DMD".
 Language changing PR's need to be approved by Andrei and I.
 
 "Team DMD" consists of people who have a consistent history of doing
 solid work reviewing PR's.
 


Interesting. This really needs to pointed out on the site.



 May 09 2014



prev sibling  parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


Missed one... >_<

On 7 May 2014 01:46, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/5/14, 11:39 PM, Manu via Digitalmars-d wrote:

 Well, in phobos, just approve 'exp' which has been raised countless
 times. I've got contributions that should be in exp, but instead,
 they're in limbo, and I've lost momentum and motivation since their
 completion is blocked by other issues, and I'm receiving no feedback
 from field testing.



 I'm fine with adding an "exp" package to phobos if you believe that that's
 what's stopping things from happening. But do you really?


Maybe. Surely I've demonstrated that I think it's a definite
possibility, and worth a shot.
But I don't see why this is my question to answer. This requires consensus.

It's been raised and rejected numerous times in the past... why? And
why would those that rejected it previously have changed their mind
right now?
This isn't meant to be a discussion between the 2 of us.



 May 06 2014



prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/5/2014 11:39 PM, Manu via Digitalmars-d wrote:

 There's a perfectly good pull there for not-virtual-by-default. No
 amount of beating will get that horse through, despite almost
 unanimous community support. That was... extremely discouraging, to
 say the least.


If you look, I have authored some pull requests that I think are very good, but 
are twisting in the wind and have for some time. Andrei has some, too. We all
do.

I understand that we'd all like our efforts to find quick approval and karma, 
but it isn't always going to happen, and we need to not let that discourage us.



 May 06 2014



  parent reply "w0rp" <devw0rp gmail.com>  writes:


On Tuesday, 6 May 2014 at 18:00:14 UTC, Walter Bright wrote:

 On 5/5/2014 11:39 PM, Manu via Digitalmars-d wrote:

 There's a perfectly good pull there for 
 not-virtual-by-default. No
 amount of beating will get that horse through, despite almost
 unanimous community support. That was... extremely 
 discouraging, to
 say the least.


 If you look, I have authored some pull requests that I think 
 are very good, but are twisting in the wind and have for some 
 time. Andrei has some, too. We all do.

 I understand that we'd all like our efforts to find quick 
 approval and karma, but it isn't always going to happen, and we 
 need to not let that discourage us.


Generally, championing something you believe in tends to require 
a great deal of personal effort into trying to make it a reality.



 May 06 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/6/2014 11:02 AM, w0rp wrote:

 Generally, championing something you believe in tends to require a great deal
of
 personal effort into trying to make it a reality.


That's right. Sometimes, we're lucky to find a champion step up and carry the 
flag for us, but 98% of the time we've got to champion it ourselves, or it just 
won't happen.



 May 06 2014



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2014-05-06 08:39, Manu via Digitalmars-d wrote:
's at least one DIP which received little attention afaict, it's

 an example of something that I think would probably manifest into code
 in an experimental space, but clearly couldn't be accepted as a
 language feature without lots of field time.
 In lieu of an experimental space, there will be no action.

 It's an interesting example actually. I think lots feel the DIP isn't
 really an effective solution, but nobody has the motivation or ideas
 to refine it. The DIP author clearly has no motivation to test it
 experimentally, but perhaps that's what it needs to progress?


Implementing AST macros is probably quite a big investment in time. I'm 
currently working on other things but I will probably give it a try at 
some point.


 What happened to std.serislisation? There was motion there a year or
 so back... I was looking forward to it, and did some minor reviewing
 at the time. I wonder if that's an interesting case study? (I haven't
 looked)


To be honest, I got board and started to work on D/Objective-C instead, 
which I know you have interest in as well. But when that is done I will 
come back to std.serialization. If not sooner, I have a short attention 
span sometimes ;)


 Perhaps you misunderstood the point of my post. I've watched people
 make solid contributions that haven't gotten through. That is
 discouraging to others considering starting their own work, and for
 the person who has already put in the effort to continue to do so in
 the future.
 The Obj-C thing as an example. Granted, it's a huge feature and has
 extensive implications. The Authors have said themselves that they
 agree it's not 'ready' for inclusion... so, what? It sits and rots?


No, I'm working on it to making it ready. Feature wise I think it's 
complete. Or rather good enough for inclusion. It supports for more 
features than extern(C++) did when it was added.

-- 
/Jacob Carlborg



 May 06 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 7 May 2014 04:55, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 2014-05-06 08:39, Manu via Digitalmars-d wrote:
 's at least one DIP which received little attention afaict, it's

 an example of something that I think would probably manifest into code
 in an experimental space, but clearly couldn't be accepted as a
 language feature without lots of field time.
 In lieu of an experimental space, there will be no action.

 It's an interesting example actually. I think lots feel the DIP isn't
 really an effective solution, but nobody has the motivation or ideas
 to refine it. The DIP author clearly has no motivation to test it
 experimentally, but perhaps that's what it needs to progress?



 Implementing AST macros is probably quite a big investment in time. I'm
 currently working on other things but I will probably give it a try at some
 point.



 What happened to std.serislisation? There was motion there a year or
 so back... I was looking forward to it, and did some minor reviewing
 at the time. I wonder if that's an interesting case study? (I haven't
 looked)



 To be honest, I got board and started to work on D/Objective-C instead,
 which I know you have interest in as well. But when that is done I will come
 back to std.serialization. If not sooner, I have a short attention span
 sometimes ;)


 Perhaps you misunderstood the point of my post. I've watched people
 make solid contributions that haven't gotten through. That is
 discouraging to others considering starting their own work, and for
 the person who has already put in the effort to continue to do so in
 the future.
 The Obj-C thing as an example. Granted, it's a huge feature and has
 extensive implications. The Authors have said themselves that they
 agree it's not 'ready' for inclusion... so, what? It sits and rots?



 No, I'm working on it to making it ready. Feature wise I think it's
 complete. Or rather good enough for inclusion. It supports for more features
 than extern(C++) did when it was added.

 --
 /Jacob Carlborg



Haha, nice! I didn't realise that all my examples for hypothetical
consideration came back to just you! :)

So then, your take on an experimental space in the compiler for
features that are still baking seems especially relevant.
Am I talking shit, or do you think the idea has any value? Would it be
valuable to get users testing these (quite large) developments while
they're still baking?



 May 06 2014



  parent  Jacob Carlborg <doob me.com>  writes:


On 07/05/14 05:36, Manu via Digitalmars-d wrote:


 Haha, nice! I didn't realise that all my examples for hypothetical
 consideration came back to just you! :)

 So then, your take on an experimental space in the compiler for
 features that are still baking seems especially relevant.
 Am I talking shit, or do you think the idea has any value? Would it be
 valuable to get users testing these (quite large) developments while
 they're still baking?


I don't think not so much for D/Objective-C but definitely for AST 
macros and std.serialization. D/Objective-C is pretty straight forward, 
at least for the user facing parts. But it wouldn't hurt to test 
D/Objective-C.

-- 
/Jacob Carlborg



 May 07 2014



prev sibling  parent  "Paolo Invernizzi" <paolo.invernizzi no.address>  writes:


On Monday, 5 May 2014 at 00:44:43 UTC, Caligo via Digitalmars-d 
wrote:

 On Sun, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via 
 Digitalmars-d <
 digitalmars-d puremagic.com> wrote:

 The on/off switch may be a nice idea in the abstract but is 
 hardly the
 perfect recipe to good language feature development; otherwise 
 everybody
 would be using it, and there's not overwhelming evidence to 
 that. (I do
 know it's been done a few times, such as the (in)famous "new 
 scoping rule
 of the for statement" for C++ which has been introduced as an 
 option by
 VC++.)


 No, it's nothing abstract, and it's very practical and useful.  
 Rust has


  Even Python has __future__, and many others.


Well, python __future__ it's not exactly that: it's for 
introducing changes that are impacting the actual codebase...

It's some sort of extreme care for not braking anything out there.

/Paolo



 May 05 2014



prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Sat, May 03, 2014 at 10:48:47PM -0500, Caligo via Digitalmars-d wrote:
[...]

 Last but not least, currently there are two main ways for new features
 to make it into D/Phobos: you either have to belong to the inner
 circle, or have to represent some corporation that's doing something
 with D.


I'm sorry, but this is patently false. I am neither in the inner circle,
nor do I represent any corporation, yet I've had many changes pulled
into Phobos (including brand new code).

I can't say I'm perfectly happy with the D development process either,
but this kind of accusation is bordering on slander, and isn't helping
anything.


T

-- 
Why are you blatanly misspelling "blatant"? -- Branden Robinson



 May 04 2014



 next sibling parent reply "Arlen" <_ gmail.com>  writes:


On Sunday, 4 May 2014 at 22:56:41 UTC, H. S. Teoh via
Digitalmars-d wrote:

 On Sat, May 03, 2014 at 10:48:47PM -0500, Caligo via 
 Digitalmars-d wrote:
 [...]

 Last but not least, currently there are two main ways for new 
 features
 to make it into D/Phobos: you either have to belong to the 
 inner
 circle, or have to represent some corporation that's doing 
 something
 with D.


 I'm sorry, but this is patently false. I am neither in the 
 inner circle,
 nor do I represent any corporation, yet I've had many changes 
 pulled
 into Phobos (including brand new code).

 I can't say I'm perfectly happy with the D development process 
 either,
 but this kind of accusation is bordering on slander, and isn't 
 helping
 anything.


 T


There is a lot of truth in what Caligo has said, but I would word
that part of it differently.

A couple years ago I submitted std.rational, but it didn't go
anywhere.  About a year later I discovered that someone else had
done a similar thing, but it never made it into Phobos either.
Of course, it's not because we didn't belong to some "inner
circle", but I think it has to do with the fact that D has a very
poor development process.  The point being, something as simple
as a Rational library shouldn't take years for it to become part
of Phobos, specially when people are taking the time to do the
work.

--Arlen



 May 04 2014



 next sibling parent  "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Mon, May 05, 2014 at 06:16:34AM +0000, Arlen via Digitalmars-d wrote:

 On Sunday, 4 May 2014 at 22:56:41 UTC, H. S. Teoh via
 Digitalmars-d wrote:

On Sat, May 03, 2014 at 10:48:47PM -0500, Caligo via Digitalmars-d wrote:
[...]

Last but not least, currently there are two main ways for new
features to make it into D/Phobos: you either have to belong to the
inner circle, or have to represent some corporation that's doing
something with D.


I'm sorry, but this is patently false. I am neither in the inner
circle, nor do I represent any corporation, yet I've had many changes
pulled into Phobos (including brand new code).

I can't say I'm perfectly happy with the D development process
either, but this kind of accusation is bordering on slander, and
isn't helping anything.




[...]

 There is a lot of truth in what Caligo has said, but I would word
 that part of it differently.
 
 A couple years ago I submitted std.rational, but it didn't go
 anywhere.  About a year later I discovered that someone else had
 done a similar thing, but it never made it into Phobos either.
 Of course, it's not because we didn't belong to some "inner
 circle", but I think it has to do with the fact that D has a very
 poor development process.  The point being, something as simple
 as a Rational library shouldn't take years for it to become part
 of Phobos, specially when people are taking the time to do the
 work.


[...]

This wording is much more acceptable. ;-) While I think accusations of
an "elite inner circle" are unfounded (and unfair), I do agree with the
sentiment. I think some time ago there was some talk about very old pull
requests that have been stuck at the bottom of the queue for months or
even years, and nobody was looking at them. I don't know what came out
of that talk, though -- apparently not very much. :-(

OTOH, I did find that stubbornness and persistence help. If you keep
pestering everybody about your new contribution, and keep pushing it
even if people seem to ignore/dislike it, keep updating your pull even
if it seems nobody cares, eventually somebody will take notice and do
something about it. Of course, this is not ideal -- open source projects
really should be actively welcoming new contributions, not merely
passively accepting them -- but that's the way it is right now, and I'm
not sure how to change that.  Perhaps stubborn and persistent pestering
of the PTBs until they change?  Might help, you never know. ;-)


T

-- 
Unix was not designed to stop people from doing stupid things, because that
would also stop them from doing clever things. -- Doug Gwyn



 May 05 2014



prev sibling next sibling parent  Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


05-May-2014 10:16, Arlen пишет:

 On Sunday, 4 May 2014 at 22:56:41 UTC, H. S. Teoh via
 Digitalmars-d wrote:

 On Sat, May 03, 2014 at 10:48:47PM -0500, Caligo via Digitalmars-d wrote:
 [...]

 Last but not least, currently there are two main ways for new features
 to make it into D/Phobos: you either have to belong to the inner
 circle, or have to represent some corporation that's doing something
 with D.


 I'm sorry, but this is patently false. I am neither in the inner circle,
 nor do I represent any corporation, yet I've had many changes pulled
 into Phobos (including brand new code).

 I can't say I'm perfectly happy with the D development process either,
 but this kind of accusation is bordering on slander, and isn't helping
 anything.


 T


 There is a lot of truth in what Caligo has said, but I would word
 that part of it differently.

 A couple years ago I submitted std.rational, but it didn't go
 anywhere.  About a year later I discovered that someone else had
 done a similar thing, but it never made it into Phobos either.


The key to getting things done is persistence. Everybody is on their 
spare time, nobody aside from the author would be able to push it through.

The process is not "I submit code and it finds its way into the standard 
library". It's rather getting people to try your stuff first and 
listening to them. Then with enough momentum and feedback one would go 
to review queue. Then start a review if nobody objects, then get into 
pass or postpone cycle, then survive the mess as the pull request goes 
into Phobos proper.

Last but not least the burden of getting something into it is minor 
compared to tending the bugs and maintaining the stuff afterwards.


 Of course, it's not because we didn't belong to some "inner
 circle", but I think it has to do with the fact that D has a very
 poor development process.


What that makes of some other open-source projects, that still traffic 
in patches over email :)


 The point being, something as simple
 as a Rational library shouldn't take years for it to become part
 of Phobos, specially when people are taking the time to do the
 work.


Look at it this way - when something is simpler, it makes it that much 
harder to make the one and true version of it. Everybody knows what it 
is, and tries to put in some of his favorite sauce. The hardest things 
to push into Phobos are one-liners even if it makes a ton of things look 
better, more correct and whatnot.

Anyhow I agree that Phobos development process (the one I know about 
most) is slow and imperfect largely due to the informal nature of 
participation. Some reviews were lively and great, some went in a gloomy 
silence with uncertain results without any good indication of the reason.



 --Arlen



-- 
Dmitry Olshansky



 May 05 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/4/14, 11:16 PM, Arlen wrote:

 A couple years ago I submitted std.rational, but it didn't go
 anywhere.  About a year later I discovered that someone else had
 done a similar thing, but it never made it into Phobos either.
 Of course, it's not because we didn't belong to some "inner
 circle", but I think it has to do with the fact that D has a very
 poor development process.  The point being, something as simple
 as a Rational library shouldn't take years for it to become part
 of Phobos, specially when people are taking the time to do the
 work.


I looked into this (not sure to what extent it's representative of a 
pattern), and probably we could and should fix it. Looks like back in 
2012 you've done the right things (http://goo.gl/kbYQJM) but for 
whatever reason there was not enough response from the community.

Later on, Joseph Rushton Wakeling tried (http://goo.gl/XyQu3D) to put 
std.rational through the review process but things got stuck at 
https://github.com/D-Programming-Language/phobos/pull/1616 with support 
of traits by BigInt.

I think the "needs to support BigInt" argument is not a blocker - we can 
release std.rational to only support built-in integers, and then adjust 
things later to expand support while keeping backward compatibility. I 
do think it's important that BigInt supports appropriate traits to be 
recognized as an integral-like type.

If you, Joseph, or both would want to put std.rational again through the 
review process I think it should get a fair shake. I do agree that a lot 
of persistence is needed.


Andrei



 May 05 2014



  parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Andrei Alexandrescu:


 I think the "needs to support BigInt" argument is not a blocker 
 - we can release std.rational to only support built-in 
 integers, and then adjust things later to expand support while 
 keeping backward compatibility. I do think it's important that 
 BigInt supports appropriate traits to be recognized as an 
 integral-like type.


Bigints support is necessary for usable rationals, but I agree 
this can't block their introduction in Phobos if the API is good 
and adaptable to the successive support of bigints.



 If you, Joseph, or both would want to put std.rational again 
 through the review process I think it should get a fair shake. 
 I do agree that a lot of persistence is needed.


Rationals are rather basic (important) things, so a little of 
persistence is well spent here :-)

Bye,
bearophile



 May 05 2014



  parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Mon, May 05, 2014 at 03:55:12PM +0000, bearophile via Digitalmars-d wrote:

 Andrei Alexandrescu:
 

I think the "needs to support BigInt" argument is not a blocker - we
can release std.rational to only support built-in integers, and then
adjust things later to expand support while keeping backward
compatibility. I do think it's important that BigInt supports
appropriate traits to be recognized as an integral-like type.


 
 Bigints support is necessary for usable rationals, but I agree this
 can't block their introduction in Phobos if the API is good and
 adaptable to the successive support of bigints.


Yeah, rationals without bigints will overflow very easily, causing many
usability problems in user code.



If you, Joseph, or both would want to put std.rational again through
the review process I think it should get a fair shake. I do agree
that a lot of persistence is needed.


 
 Rationals are rather basic (important) things, so a little of
 persistence is well spent here :-)


[...]

I agree, and support pushing std.rational through the queue. So, please
don't give up, we need it get it in somehow. :)


T

-- 
I see that you JS got Bach.



 May 05 2014



  parent reply Marco Leise <Marco.Leise gmx.de>  writes:


Am Mon, 5 May 2014 09:39:30 -0700
schrieb "H. S. Teoh via Digitalmars-d"
<digitalmars-d puremagic.com>:


 On Mon, May 05, 2014 at 03:55:12PM +0000, bearophile via Digitalmars-d wrote:

 Andrei Alexandrescu:
 

I think the "needs to support BigInt" argument is not a blocker - we
can release std.rational to only support built-in integers, and then
adjust things later to expand support while keeping backward
compatibility. I do think it's important that BigInt supports
appropriate traits to be recognized as an integral-like type.


 
 Bigints support is necessary for usable rationals, but I agree this
 can't block their introduction in Phobos if the API is good and
 adaptable to the successive support of bigints.


 
 Yeah, rationals without bigints will overflow very easily, causing many
 usability problems in user code.
 
 

If you, Joseph, or both would want to put std.rational again through
the review process I think it should get a fair shake. I do agree
that a lot of persistence is needed.


 
 Rationals are rather basic (important) things, so a little of
 persistence is well spent here :-)


 [...]
 
 I agree, and support pushing std.rational through the queue. So, please
 don't give up, we need it get it in somehow. :)
 
 
 T


That experimental package idea that was discussed months ago
comes to my mind again. Add that thing as exp.rational and
have people report bugs or shortcomings to the original
author. When it seems to be usable by everyone interested it
can move into Phobos proper after the formal review (that
includes code style checks, unit tests etc. that mere users
don't take as seriously).

As long as there is nothing even semi-official, it is tempting
to write such a module from scratch in a quick&dirty fashion
and ignore existing work.
The experimental package makes it clear that this code is
eventually going to the the official way and home brewed stuff
wont have a future. Something in the standard library is much
less likely to be reinvented. On the other hand, once a module
is in Phobos proper, it is close to impossible to change the
API to accommodate for a new use case. That's why I think the
most focused library testing and development can happen in the
experimental phase of a module. The longer it is, the more
people will have tried it in their projects before formal
review, which would greatly improve informed decisions.
The original std.rationale proposal could have been in active
use now for months!

-- 
Marco



 May 05 2014



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 5 May 2014 at 17:22:58 UTC, Marco Leise wrote:

 Am Mon, 5 May 2014 09:39:30 -0700
 schrieb "H. S. Teoh via Digitalmars-d"
 <digitalmars-d puremagic.com>:


 On Mon, May 05, 2014 at 03:55:12PM +0000, bearophile via 
 Digitalmars-d wrote:

 Andrei Alexandrescu:
 

I think the "needs to support BigInt" argument is not a 
blocker - we
can release std.rational to only support built-in integers, 
and then
adjust things later to expand support while keeping backward
compatibility. I do think it's important that BigInt 
supports
appropriate traits to be recognized as an integral-like 
type.


 
 Bigints support is necessary for usable rationals, but I 
 agree this
 can't block their introduction in Phobos if the API is good 
 and
 adaptable to the successive support of bigints.


 
 Yeah, rationals without bigints will overflow very easily, 
 causing many
 usability problems in user code.
 
 

If you, Joseph, or both would want to put std.rational 
again through
the review process I think it should get a fair shake. I do 
agree
that a lot of persistence is needed.


 
 Rationals are rather basic (important) things, so a little of
 persistence is well spent here :-)


 [...]
 
 I agree, and support pushing std.rational through the queue. 
 So, please
 don't give up, we need it get it in somehow. :)
 
 
 T


 That experimental package idea that was discussed months ago
 comes to my mind again. Add that thing as exp.rational and
 have people report bugs or shortcomings to the original
 author. When it seems to be usable by everyone interested it
 can move into Phobos proper after the formal review (that
 includes code style checks, unit tests etc. that mere users
 don't take as seriously).


And same objections still remain.



 May 05 2014



  parent  Marco Leise <Marco.Leise gmx.de>  writes:


Am Mon, 05 May 2014 17:24:38 +0000
schrieb "Dicebot" <public dicebot.lv>:


 That experimental package idea that was discussed months ago
 comes to my mind again. Add that thing as exp.rational and
 have people report bugs or shortcomings to the original
 author. When it seems to be usable by everyone interested it
 can move into Phobos proper after the formal review (that
 includes code style checks, unit tests etc. that mere users
 don't take as seriously).


 
 And same objections still remain.


Sneaky didn't work this time.

-- 
Marco



 May 05 2014



prev sibling  parent  "HaraldZealot" <harald_zealot tut.by>  writes:


 That experimental package idea that was discussed months ago
 comes to my mind again. Add that thing as exp.rational and
 have people report bugs or shortcomings to the original
 author. When it seems to be usable by everyone interested it
 can move into Phobos proper after the formal review (that
 includes code style checks, unit tests etc. that mere users
 don't take as seriously).

 As long as there is nothing even semi-official, it is tempting
 to write such a module from scratch in a quick&dirty fashion
 and ignore existing work.
 The experimental package makes it clear that this code is
 eventually going to the the official way and home brewed stuff
 wont have a future. Something in the standard library is much
 less likely to be reinvented. On the other hand, once a module
 is in Phobos proper, it is close to impossible to change the
 API to accommodate for a new use case. That's why I think the
 most focused library testing and development can happen in the
 experimental phase of a module. The longer it is, the more
 people will have tried it in their projects before formal
 review, which would greatly improve informed decisions.
 The original std.rationale proposal could have been in active
 use now for months!


+1

The same idea came to my mind yesterday too (that signalize that 
isn't void idea).

And GNU/Linux community has proof that such model works. I mean 
Archlinux as example (but as I think other example exists). There 
are core, extra, community repositories and AUR (arch user 
repository). Today AUR contains 48489 packages and 11814 of which 
are orphan. But even orphanity isn't a sign of useless. 
Possibility to write PKGBUILD and place it in AUR has everybody 
and it is not such difficult. Only small part of these package 
goes to official repository, but this part EXISTS. And in my 
opinion this is the most rapid way for evolution. For example 
archlinux has the best support of D language through abundance of 
linux distributives (http://pastebin.com/tPKWS4ga and this is 
official repositories not in AUR). Oh, I've forgotten to mention 
all sorts of testing repositories.

And D community have all means for evolute such way. We have dub 
and http://code.dlang.org/. It rests to do a little: create 
package (in D sense) which will contain packages that installed 
through dub; and some UI to vote, orphan and adopt package.

There are example as how that I see:

On http://code.dlang.org/ we can found  package "rationals" (for 
example) with mark expstd. Than we install package with dub to 
some environment (it may be fakeroot, peruser import path or even 
perproject directory (the last likes me the least)) and address 
this package as "import expstd.rationals;" in our D projects. 
It's were good when dub or code.dlang.org counts installation. 
When some package counts enough installation and enough history 
(includes bug tracking), it become candidate to include in 
phobos. All trusted users (in therm of Archlinux) receive strong 
signal about and one of them review such package and become its 
auxiliary  comaintainer. No one package can exist in official 
repository without maintainer.

And some words about why "expstd". I think that with such package 
model comes time to add not only phobos in "namespace". There are 
many useful libraries which may not become part of phobos (linear 
algebra for example, or even GUI bindings), but should have some 
standardization and  highway to distribution. I suggest package 
"extra" (in D sense) and "bind" (for deimos and perhaps not only 
deimos packages). Naturally they start their life as "expextra" 
and "expbind". In the future we will need some other package (in 
D sense), but for start that three are enough.

As Andrei said "acta non verba". I promise that I make all, 
considering my life environment, to help and start this project.

P.S. This theme will become independent. And this is yet another 
one argue, that forum forms not ideal to represent news and 
current state of community. Important things sink often in other 
topics.

P.P.S. My apologies for my poor English.



 May 06 2014



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 05/05/14 00:55, H. S. Teoh via Digitalmars-d wrote:


 I'm sorry, but this is patently false. I am neither in the inner circle,
 nor do I represent any corporation, yet I've had many changes pulled
 into Phobos (including brand new code).


I think he's referring to language changes. Things that will require a 
DIP, i.e. not just adding a new __trait.

-- 
/Jacob Carlborg



 May 04 2014



prev sibling  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 3 May 2014 18:49, Benjamin Thaut via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:

 Walter and I have had a long chat in which we figured our current
 offering of abstractions could be improved. Here are some thoughts.
 There's a lot of work ahead of us on that and I wanted to make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within
 the GC entirely. It makes for a faster and better GC, but the real
 reason here is that destructors are philosophically bankrupt in a GC
 environment. I think there's no need to argue that in this community.

 The GC never guarantees calling destructors even today, so this decision
 would be just a point in the definition space (albeit an extreme one).

 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that by
 other means, such as reference counting or manual. We're considering
 deprecating ~this() for classes in the future.

 Also, we're considering a revamp of built-in slices, as follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently lowered into
 RCSlice!T, defined inside object.d. That type would occupy THREE words,
 one of which being a pointer to a reference count. That type would
 redefine all slice primitives to update the reference count accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
 will be allowed, and will ignore the reference count (so if a void[]
 extracted from a T[] via a cast outlives all slices, dangling pointers
 will ensue).

 I foresee any number of theoretical and practical issues with this
 approach. Let's discuss some of them here.


 Thanks,

 Andrei



 Honestly, that sounds like the entierly wrong apporach to me. Your
 approaching the problem in this way:

 "We can not implement a propper GC in D because the language design prevents
 us from doing so. So lets remove destructors to migate the issue of false
 pointers."

 While the approach should be.

 "The language does not allow to implement a propper GC (anything else then
 dirty mark & sweep), what needs to be changed to allow a implementation of a
 more sophisticated GC."


Couldn't agree more.
Abandoning destructors is a disaster.
Without destructors, you effectively have manual memory management, or
rather, manual 'resource' management, which is basically the same
thing, even if you have a GC.
It totally undermines the point of memory management as a foundational
element of the language if most things are to require manual
release/finalisation/destruction or whatever you wanna call it.




 relies on resource management. So basically every class in there inherits


 entire codebase feels like manual memory management. You have to think about
 manually destroying every class and the entire advantage of having a GC,
 e.g. not having to think about memory management and thus beeing more

 really want that for D?


This is interesting to hear someone else say this. I have always found

management in practise too.
I've ranted enough about it already, but I have come to the firm
conclusion that the entire premise of a mark&sweep GC is practically
corrupt. Especially in D.

absolutely parallels your example, I realise that GC's failure extends
into far more cases than just the ones I'm usually representing.

I also maintain that GC isn't future-proof in essence. Computers grow
exponentially, and GC performance inversely tracks the volume of
memory in the system. Anything with an exponential growth curve is
fundamentally not future-proof.
I predict a 2025 Wikipedia entry: "GC was a cute idea that existed for
a few years in the early 2000's while memory ranged in the 100's mb -
few gb's, but quickly became unsustainable as computer technology
advanced".



 And what if I want unsafe slices of structs with destructors, for
 performance? Maybe I perfectly know that the memory behind the slice will
 outlive the slice, and I don't want the overhead of all the reference
 counthing behind it?

 If you actually deprecate ~this, there would be two options for me.
 1) Migrate my entire codebase to some user defiend finalizer function (which
 doesn't have compiler support), which would be a lot of work.


Does ~this() actually work, or just usually work?



 2) Quit D. (which is becomeing more and more an option when reading the
 recent news group discussions.)


I'm starting to fear the same outcome for myself.
I don't have any idea how to reconcile this problem in my working
environment, and very little community sympathy. I'm not seeing real
solutions emerging, and the only one I can imagine that's even
theoretically possible is wildly unpopular (ARC).
For years, I just (naively?) assumed that the GC was immature, and
would improve with time. Never gave it much thought; assumed there
were people much smarter than me with a plan...

I can't imagine any way out of this without significant breaking
changes to the type system. Probably a new pointer type at least.


This thread is starting to convince me that the GC should probably be
repositioned as a convenience library provided *beside* the language,
rather than a foundation of the language. It should be exclusively
opt-in, not opt-out, and upon opting-in, you accept the associated
problems.

broken too hadn't occurred to me until now, but if I had to nominate

it; it's built on a GC, but isn't really compatible with it either.

value in the feature, and a definite tendency to produce unreliability
by users who aren't experts on garbage collection and presume it
should 'just work'.

I support the notion that if the GC isn't removed as a foundational
feature of D, then destructors should probably be removed from D.
That said, I really want my destructors, and would be very upset to
see them go. So... ARC?



 May 05 2014



 next sibling parent reply "HaraldZealot" <harald_zealot tut.by>  writes:


 That said, I really want my destructors, and would be very 
 upset to
 see them go. So... ARC?


Manu, can you direct me what is ARC? This abbreviation is very
misgooglly.



 May 05 2014



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 6 May 2014 13:51, HaraldZealot via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 That said, I really want my destructors, and would be very upset to
 see them go. So... ARC?



 Manu, can you direct me what is ARC? This abbreviation is very
 misgooglly.


Automatic reference counting, and solution used by Apple in Obj-C.
There has been massive debate on the topic already, but it's generally
been dismissed.



 May 05 2014



  parent  Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 06/05/2014 06:05, Manu via Digitalmars-d a écrit :

 On 6 May 2014 13:51, HaraldZealot via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 That said, I really want my destructors, and would be very upset to
 see them go. So... ARC?



 Manu, can you direct me what is ARC? This abbreviation is very
 misgooglly.


 Automatic reference counting, and solution used by Apple in Obj-C.
 There has been massive debate on the topic already, but it's generally
 been dismissed.


As I don't code in Obj-C I can't tell much about that but some of my 
coworkers like that. Me I really love the Memory Analyzer.

Fork?



 May 06 2014



prev sibling next sibling parent reply "HaraldZealot" <harald_zealot tut.by>  writes:


I have to say that all this discussion (more precisely the 
understanding on the side of key developers) make me very upset.

It's good that Andrei agreed with impossibility of the 
harebrained disallowing of the class destructors. But I was very 
surprise, that so thought go to such head, because such solution 
contradicts the D spirit totally. There are many language which 
are very popular and have many dark moments in their design. I 
(and I think not only me) go to the D not for its popularity, but 
for its clarity, power and SANITY (that bases on strong 
guaranties). The strong solutions found on the strong decision 
makes D itself. (Examples of such strong solutions: 
immutabilities, default unshareness, struct and class as distinct 
being). And way that leads us in state where stucts have dtors 
and classes haven't but allow struct with dtor as member and 
people have to call struct dtor manually, isn't D way. Because 
such way relies on programmers discipline, but how Andrei has 
written "If there one thing that decades of computing have taught 
us, it must be that discipline-oriented programming does not 
scale."[TDPL, p. 399].

Our negative filling flood out may be sane from psychologically 
view, but neither sane nor constructive for D future. For solving 
problem it's need its formulate. We have to state that current 
state (lack of structs' dtors call guaranty) is insane, the 
harebrained disallowing of the class destructors is insane too. 
And what is sane? If I properly understand philosophy of D, we 
need semiautomated (not full) resource manager with strong 
guaranty and good performance, and which automated mode covers 
the most part of use-case. It is the target. Garbage collection 
or reference counting or any possible third way is a detail 
therefor task and mean not a target. And one task, that lays on 
the way to target, is minimal rape of D2 language (even if 
solution will be D3), so IMO dtors (perhaps only for structs) 
must survive.

I notice that I view only part of problem, can anybody link or 
describe me completely state and problems of current garbage 
collection and other resource management? It help me in finding 
of existence solution (at least theoretical).

---
Alaksiej Stankievič



 May 05 2014



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).


The major issue with the garbage collector is that it's not guaranteed 
to run a collection. When a collection is run the GC will call the 
destructors for the objects it collects. If there's no guarantee a 
collection is run there can be no guarantee that destructors are called. 
A collection is usually run when allocating new memory and there's not 
enough memory available.

-- 
/Jacob Carlborg



 May 05 2014



 next sibling parent  "HaraldZealot" <harald_zealot tut.by>  writes:


 The major issue with the garbage collector is that it's not 
 guaranteed to run a collection. When a collection is run the GC 
 will call the destructors for the objects it collects. If 
 there's no guarantee a collection is run there can be no 
 guarantee that destructors are called. A collection is usually 
 run when allocating new memory and there's not enough memory 
 available.


So it seems that I have understood more than I've thought.

If really problem rely on this than an idea came to my mind 
yesterday. What about separate destructors call and garbage 
collections mechanism? This idea reflects current D state with 
postblit constructor in structures. Raw memory allocation, 
default initialization (with T.init field) and after all that 
calling constructor (especially postblit). In therms of 
destructors it sound as: after becoming object or structure  
needless some automatic mechanism call cascading destructor, but 
memory still not fried, and GC collects memory in its time. Yes 
it's sounded monstrous, but perhaps when which mechanism will 
done only self work, each of them can be lighter. And you always 
can call some kind of finalizator manually for performance (some 
kind of flag can exist to protect from double call of 
destructors).



 May 06 2014



prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).



 The major issue with the garbage collector is that it's not guaranteed to
 run a collection. When a collection is run the GC will call the destructors
 for the objects it collects. If there's no guarantee a collection is run
 there can be no guarantee that destructors are called. A collection is
 usually run when allocating new memory and there's not enough memory
 available.


I think it's also an important consideration that GC is incompatible
with low-memory and real-time environments.

The trouble is, GC runs more frequently as the amount of available
free memory decreases, and trace&collect takes a long time, long
enough that realtime software typically can't tolerate the pause. If
the pause is super rare (minutes/hours apart), maybe it's acceptable,
but if it's closer to every call to alloc (ie, when there is little/no
free memory), then you realise it's fundamentally incompatible.
Then consider that this is typically the default operating state of
embedded systems; to have no free memory at runtime, and therefore D
is effectively incompatible with a subset of computers...
specifically, games consoles; a gigantic industry, and one of the few
that still absolutely relies on native code where D is particularly
compelling.

If people want to call D a native systems language, this seems like a
serious conflict with that idea. A major subset of the language is
incompatible with realtime or embedded use (surely among the most
compelling use cases remaining for native code these days!). Usually
the argument is that I represent a niche, and therefore I should
accept the restrictions of my environment, and revert to full manual
memory management for my work, while letting everyone else have their
fun.
That is a broken argument, not only because it leaves very little
motivation to leave C++, but also because regardless of whether I
adopt such restrictions, library authors won't, and we don't live in
the 1980's where I can write a few lines of self-contained assembly
and call it a program. To say I can't depend on libraries is
practically absurd. Games are gigantic software projects, and depend
on heaps of libraries.

The GC effectively eliminates libraries from embedded/realtime users,
which I see as an absolute deal-breaker alone, irrespective of a
further bunch more restrictions it places on that subset of users
within their own code :/

I'll be happy with any solution that works, really. But as I see it,
reference counting is the only garbage collection technology I know
which will reliably clean up eagerly as things fall out of scope (also
addressing this issue with destructors as bonus), and as such, it
seems the logical solution. ARC is a demonstrable success in Obj-C,
and works well in realtime and embedded systems. D has type system
improvements which would theoretically allow for significant
improvements over Obj-C.



 May 06 2014



 next sibling parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d 
wrote:

 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link 
 or describe
 me completely state and problems of current garbage 
 collection and other
 resource management? It help me in finding of existence 
 solution (at
 least theoretical).



 The major issue with the garbage collector is that it's not 
 guaranteed to
 run a collection. When a collection is run the GC will call 
 the destructors
 for the objects it collects. If there's no guarantee a 
 collection is run
 there can be no guarantee that destructors are called. A 
 collection is
 usually run when allocating new memory and there's not enough 
 memory
 available.


 I think it's also an important consideration that GC is 
 incompatible
 with low-memory and real-time environments.

 ...


I guess outside the gaming world, embedded and real-time seem to 
be getting lots of Java and .NET love:

https://www.aicas.com/cms/

http://www.is2t.com/products/

http://www.mountaineer.org/netmf-for-stm32/

Just a small sample of the partners providing the said support.


--
Paulo



 May 06 2014



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 6 May 2014 21:39, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d wrote:

 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).




 The major issue with the garbage collector is that it's not guaranteed to
 run a collection. When a collection is run the GC will call the
 destructors
 for the objects it collects. If there's no guarantee a collection is run
 there can be no guarantee that destructors are called. A collection is
 usually run when allocating new memory and there's not enough memory
 available.



 I think it's also an important consideration that GC is incompatible
 with low-memory and real-time environments.

 ...



 I guess outside the gaming world, embedded and real-time seem to be getting
 lots of Java and .NET love:

 https://www.aicas.com/cms/

 http://www.is2t.com/products/

 http://www.mountaineer.org/netmf-for-stm32/

 Just a small sample of the partners providing the said support.


True, much embedded work isn't realtime. There are a lot of purpose
devices where performance is not particularly important, probably
correctness is, hence Java may be popular in this environment, and
 safe D may have a great application here.

In another realm, if we're talking about really small systems
(microcontrollers with memory in the megabytes), the tendency to rely
on libraries is reduced significantly, since you probably can't afford
the memory for the libs. This environment is the one where it's
realistic to say "tag main() with  nogc, and work from there", ie, ban
the GC throughout the whole project.

However, another very popular use for embedded systems IS realtime software.
Games consoles are the obvious one here, but also POS/retail devices,
kiosks, televisions, PVR's/dvd/bluray and other entertainment related
devices, cars and automobile HUD's, music players, advertising
displays, etc. There's no shortage of realtime devices in the embedded
space.
Notably, I didn't say 'phones'. Although I think they do generally
fall into this category, I think they're drifting away. Since they run
full OS stack's, it's typical to have unknown amounts of free memory
for user-space apps and virtual memory managers that can page swap, so
having excess memory overhead probably isn't such a big deal. It's
still a major performance hazard though. Stuttering realtime
applications is never a professional look, and Android suffers
chronically in this department compared to iOS.

I spent my last weekend trying to get a PS4 game (built with Unity;
uses mono as a scripting environment) running at an acceptable frame

collector. We'll probably need to cut content from the game, such that
it leaves plenty of 'excess' resource available to absorb the spikes
it causes when they occur. What a waste of the machine!

realm of single-digit milliseconds.



 May 06 2014



 next sibling parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 6 May 2014 at 12:05:10 UTC, Manu via Digitalmars-d 
wrote:

 On 6 May 2014 21:39, Paulo Pinto via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d 
 wrote:

 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link 
 or describe
 me completely state and problems of current garbage 
 collection and other
 resource management? It help me in finding of existence 
 solution (at
 least theoretical).




 The major issue with the garbage collector is that it's not 
 guaranteed to
 run a collection. When a collection is run the GC will call 
 the
 destructors
 for the objects it collects. If there's no guarantee a 
 collection is run
 there can be no guarantee that destructors are called. A 
 collection is
 usually run when allocating new memory and there's not 
 enough memory
 available.



 I think it's also an important consideration that GC is 
 incompatible
 with low-memory and real-time environments.

 ...



 I guess outside the gaming world, embedded and real-time seem 
 to be getting
 lots of Java and .NET love:

 https://www.aicas.com/cms/

 http://www.is2t.com/products/

 http://www.mountaineer.org/netmf-for-stm32/

 Just a small sample of the partners providing the said support.


 True, much embedded work isn't realtime. There are a lot of 
 purpose
 devices where performance is not particularly important, 
 probably
 correctness is, hence Java may be popular in this environment, 
 and
  safe D may have a great application here.

 In another realm, if we're talking about really small systems
 (microcontrollers with memory in the megabytes), the tendency 
 to rely
 on libraries is reduced significantly, since you probably can't 
 afford
 the memory for the libs. This environment is the one where it's
 realistic to say "tag main() with  nogc, and work from there", 
 ie, ban
 the GC throughout the whole project.

 However, another very popular use for embedded systems IS 
 realtime software.
 Games consoles are the obvious one here, but also POS/retail 
 devices,
 kiosks, televisions, PVR's/dvd/bluray and other entertainment 
 related
 devices, cars and automobile HUD's, music players, advertising
 displays, etc. There's no shortage of realtime devices in the 
 embedded
 space.


I can provide more examples for JVMs in all areas you list.

However those JVMs do cheat a bit, as they provide a fine grain
control over GC and also allow for region based allocations and 
immortal memory.


 Notably, I didn't say 'phones'. Although I think they do 
 generally
 fall into this category, I think they're drifting away. Since 
 they run
 full OS stack's, it's typical to have unknown amounts of free 
 memory
 for user-space apps and virtual memory managers that can page 
 swap, so
 having excess memory overhead probably isn't such a big deal. 
 It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.

 I spent my last weekend trying to get a PS4 game (built with 
 Unity;
 uses mono as a scripting environment) running at an acceptable 
 frame

 garbage
 collector. We'll probably need to cut content from the game, 
 such that
 it leaves plenty of 'excess' resource available to absorb the 
 spikes
 it causes when they occur. What a waste of the machine!

 the
 realm of single-digit milliseconds.


Just as info regarding your experience, please note that both 
Android and Unity have not so good GC implementations.

Dalvik's GC is hardly touched since Android 2.3, hopefully it 
will be improved with the new ART runtime.

Unity's Mono implementation is  a legacy one, based on the old 
Novell Mono implementation as Unity is not willing to pay the 
royalties Xamarin is asking for modern Mono implementations.


As for the gaming world, I don't really have the required 
experience to talk much about it. Just doing the heads up that D 
GC != GC that many of us use.

--
Paulo



 May 06 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 6 May 2014 at 12:21:55 UTC, Paulo Pinto wrote:

 As for the gaming world, I don't really have the required 
 experience to talk much about it. Just doing the heads up that 
 D GC != GC that many of us use.


I very much appreciate the links you provide about different GC 
solutions, and GC can be appropriate for real time situations 
where you don't want to maximize resource utilization or where 
you have predictable downtime to collect.

However the approach taken by appliances (where the application 
reliability is more important than getting the highest possible 
execution speed) or business solutions such as Azul Zing (where 
scalability is more important than hardware cost) is not really 
suitable for system level programming. It is suitable for a wide 
range of applications, that is true.

You can have GC with predictable destructors, but you have to 
design the language and runtime around it. You can have real time 
GC, but you need a dedicated GC and structure your program to 
realize the potential.

I think the only realistic road for D is to imposing/infer 
constraints that makes certain that execution paths is compatible 
with desirable runtime properties and available runtime support 
for that specific thread. There is no silver bullet, but the 
implementation driven design that D seems to be undergoing is 
affecting the resulting language/run-time design in a bad way, 
IMO.



 May 06 2014



prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-05-06 12:04:55 +0000, Manu via Digitalmars-d 
<digitalmars-d puremagic.com> said:


 Notably, I didn't say 'phones'. Although I think they do generally
 fall into this category, I think they're drifting away. Since they run
 full OS stack's, it's typical to have unknown amounts of free memory
 for user-space apps and virtual memory managers that can page swap, so
 having excess memory overhead probably isn't such a big deal. It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.


Note that iOS has no page swap. Apps just get killed if there isn't 
enough memory (after being sent a few low memory signals they can react 
on, clearing caches, etc.). Apps that takes a lot of memory cause other 
apps in the background to be killed (and later restarted when they come 
to the foreground).

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 May 06 2014



 next sibling parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 6 May 2014 22:30, Michel Fortin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 2014-05-06 12:04:55 +0000, Manu via Digitalmars-d
 <digitalmars-d puremagic.com> said:


 Notably, I didn't say 'phones'. Although I think they do generally
 fall into this category, I think they're drifting away. Since they run
 full OS stack's, it's typical to have unknown amounts of free memory
 for user-space apps and virtual memory managers that can page swap, so
 having excess memory overhead probably isn't such a big deal. It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.



 Note that iOS has no page swap. Apps just get killed if there isn't enough
 memory (after being sent a few low memory signals they can react on,
 clearing caches, etc.). Apps that takes a lot of memory cause other apps in
 the background to be killed (and later restarted when they come to the
 foreground).


I'm just saying why those platforms aren't likely to suffer the
no-free-memory environment, and therefore can't really be considered
proper embedded systems like video games consoles.



 May 06 2014



prev sibling  parent  Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 06/05/2014 14:30, Michel Fortin a crit :

 On 2014-05-06 12:04:55 +0000, Manu via Digitalmars-d
 <digitalmars-d puremagic.com> said:


 Notably, I didn't say 'phones'. Although I think they do generally
 fall into this category, I think they're drifting away. Since they run
 full OS stack's, it's typical to have unknown amounts of free memory
 for user-space apps and virtual memory managers that can page swap, so
 having excess memory overhead probably isn't such a big deal. It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.


 Note that iOS has no page swap. Apps just get killed if there isn't
 enough memory (after being sent a few low memory signals they can react
 on, clearing caches, etc.). Apps that takes a lot of memory cause other
 apps in the background to be killed (and later restarted when they come
 to the foreground).


And it's viable because majority of applications can be launch at the 
exact same state (or really close) they was before the kill.



 May 06 2014



prev sibling  parent  Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 06/05/2014 14:04, Manu via Digitalmars-d a écrit :

 On 6 May 2014 21:39, Paulo Pinto via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d wrote:

 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).




 The major issue with the garbage collector is that it's not guaranteed to
 run a collection. When a collection is run the GC will call the
 destructors
 for the objects it collects. If there's no guarantee a collection is run
 there can be no guarantee that destructors are called. A collection is
 usually run when allocating new memory and there's not enough memory
 available.



 I think it's also an important consideration that GC is incompatible
 with low-memory and real-time environments.

 ...



 I guess outside the gaming world, embedded and real-time seem to be getting
 lots of Java and .NET love:

 https://www.aicas.com/cms/

 http://www.is2t.com/products/

 http://www.mountaineer.org/netmf-for-stm32/

 Just a small sample of the partners providing the said support.


 True, much embedded work isn't realtime. There are a lot of purpose
 devices where performance is not particularly important, probably
 correctness is, hence Java may be popular in this environment, and
  safe D may have a great application here.

 In another realm, if we're talking about really small systems
 (microcontrollers with memory in the megabytes), the tendency to rely
 on libraries is reduced significantly, since you probably can't afford
 the memory for the libs. This environment is the one where it's
 realistic to say "tag main() with  nogc, and work from there", ie, ban
 the GC throughout the whole project.

 However, another very popular use for embedded systems IS realtime software.
 Games consoles are the obvious one here, but also POS/retail devices,
 kiosks, televisions, PVR's/dvd/bluray and other entertainment related
 devices, cars and automobile HUD's, music players, advertising
 displays, etc. There's no shortage of realtime devices in the embedded
 space.
 Notably, I didn't say 'phones'. Although I think they do generally
 fall into this category, I think they're drifting away. Since they run
 full OS stack's, it's typical to have unknown amounts of free memory
 for user-space apps and virtual memory managers that can page swap, so
 having excess memory overhead probably isn't such a big deal. It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.


I suspect iOS and Android to have best of swap management. I think that 
almost every time you switch applications, the previous one see his 
memory collected and swapped on the Flash (maybe also compressed).


 I spent my last weekend trying to get a PS4 game (built with Unity;
 uses mono as a scripting environment) running at an acceptable frame

 collector. We'll probably need to cut content from the game, such that
 it leaves plenty of 'excess' resource available to absorb the spikes
 it causes when they occur. What a waste of the machine!

 realm of single-digit milliseconds.





 May 06 2014



prev sibling  parent reply Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 06/05/2014 13:39, Paulo Pinto a écrit :

 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d wrote:

 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and
 other
 resource management? It help me in finding of existence solution (at
 least theoretical).



 The major issue with the garbage collector is that it's not
 guaranteed to
 run a collection. When a collection is run the GC will call the
 destructors
 for the objects it collects. If there's no guarantee a collection is run
 there can be no guarantee that destructors are called. A collection is
 usually run when allocating new memory and there's not enough memory
 available.


 I think it's also an important consideration that GC is incompatible
 with low-memory and real-time environments.

 ...


 I guess outside the gaming world, embedded and real-time seem to be
 getting lots of Java and .NET love:

 https://www.aicas.com/cms/


Is that VM full compatible with Java specifications and standard 
frameworks?

 http://www.is2t.com/products/

 http://www.mountaineer.org/netmf-for-stm32/

 Just a small sample of the partners providing the said support.


 --
 Paulo



Android works well, I love my nexus, it proves to me that it's possible 
to create really smooth applications based completely on Java (not 100% 
of that) but if we compare the Nexus 5 to iPhone 4 :
Memory : 2 GB RAM vs 512 MB RAM
CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8
Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery

And compared to an iPhone 5s
Memory : 2 GB RAM vs 1 GB RAM
CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone
Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery

It's maybe not really significant but the majority of Android devices 
that have acceptable performances have a lot of memory, a quad cores CPU 
and an heavy battery.

So that cool Java can run smoothly but at which price? I think the 
margin of Apple produce is unbelievable.



 May 06 2014



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 7 May 2014 08:07, Xavier Bigand via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Le 06/05/2014 13:39, Paulo Pinto a écrit :


 Android works well, I love my nexus, it proves to me that it's possible to
 create really smooth applications based completely on Java (not 100% of
 that) but if we compare the Nexus 5 to iPhone 4 :
 Memory : 2 GB RAM vs 512 MB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8
 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery

 And compared to an iPhone 5s
 Memory : 2 GB RAM vs 1 GB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone
 Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery

 It's maybe not really significant but the majority of Android devices that
 have acceptable performances have a lot of memory, a quad cores CPU and an
 heavy battery.

 So that cool Java can run smoothly but at which price? I think the margin of
 Apple produce is unbelievable.


Yeah, these are excellent points that I've tried and perhaps failed to
articulate properly in the past.
The amount of 'wasted' resources required to maintain a significant
surplus on Android devices is ridiculous, and that's why I separated
phones from other embedded systems. While phones can, and do, do this,
other embedded systems don't.
To say we need to leave half of the xbox/ps4 resources idle to soak up
intermittency is completely unworkable.

It's always important to remember too that the embedded market is by
far the largest software market in the world.



 May 06 2014



 next sibling parent  "Paulo Pinto" <pjmlp progtools.org>  writes:


On Wednesday, 7 May 2014 at 03:58:38 UTC, Manu via Digitalmars-d 
wrote:

 On 7 May 2014 08:07, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Le 06/05/2014 13:39, Paulo Pinto a écrit :


 Android works well, I love my nexus, it proves to me that it's 
 possible to
 create really smooth applications based completely on Java 
 (not 100% of
 that) but if we compare the Nexus 5 to iPhone 4 :
 Memory : 2 GB RAM vs 512 MB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8
 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery

 And compared to an iPhone 5s
 Memory : 2 GB RAM vs 1 GB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone
 Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery

 It's maybe not really significant but the majority of Android 
 devices that
 have acceptable performances have a lot of memory, a quad 
 cores CPU and an
 heavy battery.

 So that cool Java can run smoothly but at which price? I think 
 the margin of
 Apple produce is unbelievable.


 Yeah, these are excellent points that I've tried and perhaps 
 failed to
 articulate properly in the past.
 The amount of 'wasted' resources required to maintain a 
 significant
 surplus on Android devices is ridiculous, and that's why I 
 separated
 phones from other embedded systems. While phones can, and do, 
 do this,
 other embedded systems don't.
 To say we need to leave half of the xbox/ps4 resources idle to 
 soak up
 intermittency is completely unworkable.

 It's always important to remember too that the embedded market 
 is by
 far the largest software market in the world.


You ignored my link about Aicas products for real time, embedded 
industrial systems.


--
Paulo



 May 06 2014



prev sibling  parent reply Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 07/05/2014 05:58, Manu via Digitalmars-d a écrit :

 On 7 May 2014 08:07, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Le 06/05/2014 13:39, Paulo Pinto a écrit :


 Android works well, I love my nexus, it proves to me that it's possible to
 create really smooth applications based completely on Java (not 100% of
 that) but if we compare the Nexus 5 to iPhone 4 :
 Memory : 2 GB RAM vs 512 MB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8
 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery

 And compared to an iPhone 5s
 Memory : 2 GB RAM vs 1 GB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone
 Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery

 It's maybe not really significant but the majority of Android devices that
 have acceptable performances have a lot of memory, a quad cores CPU and an
 heavy battery.

 So that cool Java can run smoothly but at which price? I think the margin of
 Apple produce is unbelievable.


 Yeah, these are excellent points that I've tried and perhaps failed to
 articulate properly in the past.
 The amount of 'wasted' resources required to maintain a significant
 surplus on Android devices is ridiculous, and that's why I separated
 phones from other embedded systems. While phones can, and do, do this,
 other embedded systems don't.
 To say we need to leave half of the xbox/ps4 resources idle to soak up
 intermittency is completely unworkable.

 It's always important to remember too that the embedded market is by
 far the largest software market in the world.


I started to work on Pocket PC and a year later on Nintendo DS have only 
4Mo of RAM. I sill was at school at this date, and it was hard to deal 
with so few memory, ROM and video memory was very limited. The project I 
worked on was only limited by memory cause of use of precomputed 
pictures (it was an old adventure game).
On this console, game vendors have to pay for the ROM and backup
memories, so we put in place some compressions and data packing logics.

 From this time I always take care of avoiding unnecessary absurd 
allocations.



 May 07 2014



  parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Wednesday, 7 May 2014 at 20:09:07 UTC, Xavier Bigand wrote:

 Le 07/05/2014 05:58, Manu via Digitalmars-d a écrit :

 On 7 May 2014 08:07, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Le 06/05/2014 13:39, Paulo Pinto a écrit :


 Android works well, I love my nexus, it proves to me that 
 it's possible to
 create really smooth applications based completely on Java 
 (not 100% of
 that) but if we compare the Nexus 5 to iPhone 4 :
 Memory : 2 GB RAM vs 512 MB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8
 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery

 And compared to an iPhone 5s
 Memory : 2 GB RAM vs 1 GB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone
 Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery

 It's maybe not really significant but the majority of Android 
 devices that
 have acceptable performances have a lot of memory, a quad 
 cores CPU and an
 heavy battery.

 So that cool Java can run smoothly but at which price? I 
 think the margin of
 Apple produce is unbelievable.


 Yeah, these are excellent points that I've tried and perhaps 
 failed to
 articulate properly in the past.
 The amount of 'wasted' resources required to maintain a 
 significant
 surplus on Android devices is ridiculous, and that's why I 
 separated
 phones from other embedded systems. While phones can, and do, 
 do this,
 other embedded systems don't.
 To say we need to leave half of the xbox/ps4 resources idle to 
 soak up
 intermittency is completely unworkable.

 It's always important to remember too that the embedded market 
 is by
 far the largest software market in the world.


 I started to work on Pocket PC and a year later on Nintendo DS 
 have only 4Mo of RAM. I sill was at school at this date, and it 
 was hard to deal with so few memory, ROM and video memory was 
 very limited. The project I worked on was only limited by 
 memory cause of use of precomputed pictures (it was an old 
 adventure game).
 On this console, game vendors have to pay for the ROM and backup
 memories, so we put in place some compressions and data packing 
 logics.

 From this time I always take care of avoiding unnecessary 
 absurd allocations.


4MB?! That is a world of pleasure.

Try to cram a Z80 application into 48 KB. :)

The main problem nowadays is not automatic memory management, in
whatever form, be it GC, RC, compiler dataflow, dependent types
or whatever.

The problem is how many developers code like the memory was
infinite without pausing a second to think about their data
structures and algorithms.

Just yesterday I have re-written a Java application that in the
application hot path does zero allocations on the code under our
control.

It requires execution analysis tooling, and thinking how to write 
the said code.
That's it.

..
Paulo



 May 07 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 8 May 2014 16:11, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Wednesday, 7 May 2014 at 20:09:07 UTC, Xavier Bigand wrote:

 4MB?! That is a world of pleasure.

 Try to cram a Z80 application into 48 KB. :)


I've never heard of a Z80 program running a tracing GC. (I have used
refcounting on a Z80 though...)


 The main problem nowadays is not automatic memory management, in
 whatever form, be it GC, RC, compiler dataflow, dependent types
 or whatever.

 The problem is how many developers code like the memory was
 infinite without pausing a second to think about their data
 structures and algorithms.


Okay, please don't be patronising. Let's assume the programmers you're
talking to aren't incompetent. This discussion's existence should
prove that your theory is incorrect.

Let's consider your argument though, it sounds EXACTLY like the sort
of programmer that would invent and rely on a tracing GC!
The technology practically implies a presumption of infinite (or
significant excess) memory. Such an excess suggests either the
software isn't making full use of the system it's running on, or the
programmers writing the software are like you say.


 Just yesterday I have re-written a Java application that in the
 application hot path does zero allocations on the code under our
 control.

 It requires execution analysis tooling, and thinking how to write the said
 code.
 That's it.


Congratulations. But I'm not sure how that helps me?

You know my arguments, how does this example of yours address them?
What were the operating and target hardware requirements of your
project? How much memory did you have? How much memory did you use?
What is the frequency of runtime resource loading/swapping? What was
the pause period and the consequence when you did collect?
Let's also bear in mind that Java's GC is worlds ahead of D's.


I am getting very tired of repeating myself and having my points
basically ignored, or dismissed with something like "my project which
doesn't actually share those requirements works fine" (not that I'm
saying you did that just now; I don't know, you need to tell me more
about your project).
I'd really like to establish as fact or fiction whether tracing GC is
_practically_ incompatible with competitive embedded/realtime
environments where pushing the hardware to the limits is a desire.
(Once upon a time, this was a matter of pride for all software
engineers)
If people can prove that I'm making it all up, and "my concerns are
invalid if I just ...", or whether my points are actually true.

It doesn't matter what awesome GC research is out there if it's
incompatible with D and/or small devices that may or may not have a
robust operating system. D is allegedly a systems language, and while
there is no such definition, my own take is that means it shouldn't be
incompatible with, or discourage certain classes of computers or
software by nature, otherwise it becomes a niche language.

Please, prove me wrong. Show me how tracing collection can satisfy the
basic requirements I've raised on countless prior posts, or practical
workarounds that you would find reasonable if you were to consider
working within those restrictions yourself and still remain compelling
enough to adopt D in your corporation in the first place (implying a
massive risk, and cost in retraining all the staff and retooling).

I don't know how to reconcile the problem with the existing GC, and I
am not happy to sacrifice large parts of the language for it. I've
made the argument before that sacrificing large parts of the language
as a 'work-around' is, in essence, sacrificing practically all
libraries. That is a truly absurd notion; to suggest that anybody
should take advice to sacrifice access to libraries is being
unrealistic.


I refer again to my example from last weekend. I was helping my mates
try and finish up their PS4 release. It turns out, the GC is very
erratic, causing them massive performance problems, and they're super
stressed about this.
Naturally, the first thing I did was scolded them for being stupid

search for practical options, I started to realise the gravity of the
situation, and I'm really glad I'm not wearing their shoes!
I've made the argument in the past that this is the single most
dangerous class of issue to encounter; one that emerges as a problem
only at the very end of the project and the issues are fundamental and
distributed, beyond practical control. Massive volumes of code are
committed, and no time or budget exists to revisit and repair the work
that was already signed off.
The only practical solution in this emergency situation is to start
cutting assets (read: eliminate your competitive edge against
competition), and try and get the system resource usage down to that
half-ish of the system, as demonstrated from the prior android vs
iphone comparison.

It comes to this; how can a GC ever work in a memory limited
environment? If the frequency of collection is almost 100% of
allocations, then GC must practically be banned everywhere and all the
associated language features lost. How can I rely on library code to
never allocate?
As I see it, this problem is basically synonymous with the current
issue of lazy/unreliable destruction. They would both be solved by
addressing this fundamental issue. And the only solution I know is to
use RC instead.
Eager release of memory is required to make sure the consecutive alloc
has free memory to allocate, problem solved, and destructors work too!
:)



 May 09 2014



  parent reply "Wyatt" <wyatt.epp gmail.com>  writes:


On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d 
wrote:

 Let's also bear in mind that Java's GC is worlds ahead of D's.


Is Sun/Oracle reference implementation actually any good?


 I am getting very tired of repeating myself and having my points
 basically ignored, or dismissed with something like "my project
 which doesn't actually share those requirements works fine" (not
 that I'm saying you did that just now; I don't know, you need to
 tell me more about your project).  I'd really like to establish
 as fact or fiction whether tracing GC is _practically_
 incompatible with competitive embedded/realtime environments
 where pushing the hardware to the limits is a [requirement].


I've actually become very curious about this, too.  I know that 
our GC isn't good, but I've seen a lot of handwaving.  The 
pattern lately has involved someone talking about ARC being 
insufficient, leading somehow to Manu asserting sufficient GC is 
impossible (why?), and everyone kind of ignoring it after that.

I've been digging into research on the subject while I wait for 
test scripts to run, and my gut feeling is it's definitely 
possible to get GC at least into striking distance, but I'm not 
nearly an expert on this area.

(Some of these are dead clever, though! I just read this one 
today: 
https://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf)


 I don't know how to reconcile the problem with the existing GC,
 and I am not happy to sacrifice large parts of the language for
 it.  I've made the argument before that sacrificing large parts
 of the language as a 'work-around' is, in essence, sacrificing
 practically all libraries. That is a truly absurd notion; to
 suggest that anybody should take advice to sacrifice access to
 libraries is being unrealistic.


This is important, and simply throwing up our collective hands 
and saying to just not use major language features (I believe I 
recall slices were in that list?) really doesn't sit well with me 
either.

But conversely, Manu, something has been bothering me: aren't you 
restricted from using most libraries anyway, even in C++?  
"Decent" or "acceptable" performance isn't anywhere near 
"maximum", so shouldn't any library code that allocates in any 
language be equally suspect?  So from that standpoint, isn't any 
library you use in any language going to _also_ be tuned for 
performance in the hot path?  Maybe I'm barking up the wrong 
tree, but I don't recall seeing this point addressed.

More generally, I feel like we're collectively missing some 
important context:  What are you _doing_ in your 16.6ms 
timeslice?  I know _I'd_ appreciate a real example of what you're 
dealing with without any hyperbole.  What actually _must_ be done 
in that timeframe?  Why must collection run inside that window?  
What must be collected when it runs in that situation?  (Serious 
questions.)

See, in the final-by-default discussions, you clearly explained 
the issues and related them well to concerns that are felt 
broadly, but this... yeah, I don't really have any context for 
this, when D would already be much faster than the thirty years 
of C navel lint (K&R flavour!) that I grapple in my day job.

-Wyatt



 May 09 2014



 next sibling parent reply "Francesco Cattoglio" <francesco.cattoglio gmail.com>  writes:


On Friday, 9 May 2014 at 21:05:18 UTC, Wyatt wrote:

 But conversely, Manu, something has been bothering me: aren't 
 you restricted from using most libraries anyway, even in C++?  
 "Decent" or "acceptable" performance isn't anywhere near 
 "maximum", so shouldn't any library code that allocates in any 
 language be equally suspect?  So from that standpoint, isn't 
 any library you use in any language going to _also_ be tuned 
 for performance in the hot path?  Maybe I'm barking up the 
 wrong tree, but I don't recall seeing this point addressed.

 More generally, I feel like we're collectively missing some 
 important context:  What are you _doing_ in your 16.6ms 
 timeslice?  I know _I'd_ appreciate a real example of what 
 you're dealing with without any hyperbole.  What actually 
 _must_ be done in that timeframe?  Why must collection run 
 inside that window?  What must be collected when it runs in 
 that situation?  (Serious questions.)


I'll try to guess: if you want something running at 60 Frames per 
Second, 16.6ms is the time
you have to do everything between frames. This means that in that 
timeframe
you have to:
-update your game state.
-possibly process all network I/O.
-prepare the rendering pipeline for the next frame.

Updating the game state can imply make computations on lots of 
stuff: physics, animations, creation and deletion of entities and 
particles, AI logic... pick your poison. At every frame you will 
have an handful of objects being destroyed and a few resources 
that might go forgotten. One frame would probably only need very 
little objects collected. But given some times the amount of junk 
can grow out of control easily. Your code will end up stuttering 
at some point (because of random collections at random times), 
and this can be really bad.



 May 09 2014



  parent reply Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 10/05/2014 01:31, Francesco Cattoglio a écrit :

 On Friday, 9 May 2014 at 21:05:18 UTC, Wyatt wrote:

 But conversely, Manu, something has been bothering me: aren't you
 restricted from using most libraries anyway, even in C++? "Decent" or
 "acceptable" performance isn't anywhere near "maximum", so shouldn't
 any library code that allocates in any language be equally suspect?
 So from that standpoint, isn't any library you use in any language
 going to _also_ be tuned for performance in the hot path?  Maybe I'm
 barking up the wrong tree, but I don't recall seeing this point
 addressed.

 More generally, I feel like we're collectively missing some important
 context:  What are you _doing_ in your 16.6ms timeslice?  I know _I'd_
 appreciate a real example of what you're dealing with without any
 hyperbole.  What actually _must_ be done in that timeframe?  Why must
 collection run inside that window?  What must be collected when it
 runs in that situation?  (Serious questions.)


 I'll try to guess: if you want something running at 60 Frames per
 Second, 16.6ms is the time
 you have to do everything between frames. This means that in that timeframe
 you have to:
 -update your game state.
 -possibly process all network I/O.
 -prepare the rendering pipeline for the next frame.

 Updating the game state can imply make computations on lots of stuff:
 physics, animations, creation and deletion of entities and particles, AI
 logic... pick your poison. At every frame you will have an handful of
 objects being destroyed and a few resources that might go forgotten. One
 frame would probably only need very little objects collected. But given
 some times the amount of junk can grow out of control easily. Your code
 will end up stuttering at some point (because of random collections at
 random times), and this can be really bad.


As I know AAA game engine are reputed to do zero allocations during the 
frame computation, but I think it less the case noways cause of the 
dynamism of scene and huge environments that are streamed.

I recently fix a performance issue due to a code design that force 
destruction of walls (I am working on an architecture application) 
before creating them back when the user move them. gprof, show me that 
this point took around 30% of CPU time in a frame, and only 
allocations/destructions was about 5%. This 5% percents contains 
destructions of object in the architecture part and the 3D engine, same 
for construction. Construction also add operation like new geometry 
updload, so I don't think place of new and delete was high without the 
job made by constructors and destructors.
Reserving memory (malloc) isn't really an issue IMO, but operations 
related to constructions and destructions of objects can be expensive.



 May 10 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 10 May 2014 19:07, Xavier Bigand via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Le 10/05/2014 01:31, Francesco Cattoglio a écrit :


 On Friday, 9 May 2014 at 21:05:18 UTC, Wyatt wrote:

 But conversely, Manu, something has been bothering me: aren't you
 restricted from using most libraries anyway, even in C++? "Decent" or
 "acceptable" performance isn't anywhere near "maximum", so shouldn't
 any library code that allocates in any language be equally suspect?
 So from that standpoint, isn't any library you use in any language
 going to _also_ be tuned for performance in the hot path?  Maybe I'm
 barking up the wrong tree, but I don't recall seeing this point
 addressed.

 More generally, I feel like we're collectively missing some important
 context:  What are you _doing_ in your 16.6ms timeslice?  I know _I'd_
 appreciate a real example of what you're dealing with without any
 hyperbole.  What actually _must_ be done in that timeframe?  Why must
 collection run inside that window?  What must be collected when it
 runs in that situation?  (Serious questions.)


 I'll try to guess: if you want something running at 60 Frames per
 Second, 16.6ms is the time
 you have to do everything between frames. This means that in that
 timeframe
 you have to:
 -update your game state.
 -possibly process all network I/O.
 -prepare the rendering pipeline for the next frame.

 Updating the game state can imply make computations on lots of stuff:
 physics, animations, creation and deletion of entities and particles, AI
 logic... pick your poison. At every frame you will have an handful of
 objects being destroyed and a few resources that might go forgotten. One
 frame would probably only need very little objects collected. But given
 some times the amount of junk can grow out of control easily. Your code
 will end up stuttering at some point (because of random collections at
 random times), and this can be really bad.



 As I know AAA game engine are reputed to do zero allocations during the
 frame computation, but I think it less the case noways cause of the dynamism
 of scene and huge environments that are streamed.


Running a game in a zero-alloc environment was a luxury that ended
about 10 years ago, for reasons that you say.
Grand Theft Auto proved that you don't need to have loading screens,
and now it's a basic requirement. We also have a lot more physics,
environmental destruction, and other dynamic behaviour.

It was also something we achieved in the past with extreme complexity,
and the source of many (most?) bugs. We still allocated technically,
but we had to micro-manage every single little detail, mini pools and
regions for everything.
Processors are better now, in the low-frequency code, we can afford to
spend a tiny bit of time using a standard allocation model. The key is
the growing separation between low-frequency and high-frequency code.
On a 33mhz Playstation, there wasn't really much difference between
the worlds. Now there is, and we can afford to allow 'safety' into the
language at the cost of a little memory management. We just can't have
that cost include halting threads all the time for lengthy collect
processes.
I know this, because we already use RC extensively in games anyway;
DirectX uses COM, and most resources use manual RC because we need
things to release eagerly, and for destructors to work properly. I
don't see how ARC would add any significant cost to the manual RC that
is basically standard for many years. It would add simplicity and
safety, which are highly desirable.



 I recently fix a performance issue due to a code design that force
 destruction of walls (I am working on an architecture application) before
 creating them back when the user move them. gprof, show me that this point
 took around 30% of CPU time in a frame, and only allocations/destructions
 was about 5%. This 5% percents contains destructions of object in the
 architecture part and the 3D engine, same for construction. Construction
 also add operation like new geometry updload, so I don't think place of new
 and delete was high without the job made by constructors and destructors.
 Reserving memory (malloc) isn't really an issue IMO, but operations related
 to constructions and destructions of objects can be expensive.


It's better to have eager destructors (which empowered you with the
ability to defer them if you need to), than to not have destructors at
all, which has been the center if this last few days argument.
It's not hard to defer resource destruction to a background thread in
an eager-destruction model.



 May 10 2014



  parent  Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 10/05/2014 15:25, Manu via Digitalmars-d a écrit :

 On 10 May 2014 19:07, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Le 10/05/2014 01:31, Francesco Cattoglio a écrit :


 On Friday, 9 May 2014 at 21:05:18 UTC, Wyatt wrote:

 But conversely, Manu, something has been bothering me: aren't you
 restricted from using most libraries anyway, even in C++? "Decent" or
 "acceptable" performance isn't anywhere near "maximum", so shouldn't
 any library code that allocates in any language be equally suspect?
 So from that standpoint, isn't any library you use in any language
 going to _also_ be tuned for performance in the hot path?  Maybe I'm
 barking up the wrong tree, but I don't recall seeing this point
 addressed.

 More generally, I feel like we're collectively missing some important
 context:  What are you _doing_ in your 16.6ms timeslice?  I know _I'd_
 appreciate a real example of what you're dealing with without any
 hyperbole.  What actually _must_ be done in that timeframe?  Why must
 collection run inside that window?  What must be collected when it
 runs in that situation?  (Serious questions.)


 I'll try to guess: if you want something running at 60 Frames per
 Second, 16.6ms is the time
 you have to do everything between frames. This means that in that
 timeframe
 you have to:
 -update your game state.
 -possibly process all network I/O.
 -prepare the rendering pipeline for the next frame.

 Updating the game state can imply make computations on lots of stuff:
 physics, animations, creation and deletion of entities and particles, AI
 logic... pick your poison. At every frame you will have an handful of
 objects being destroyed and a few resources that might go forgotten. One
 frame would probably only need very little objects collected. But given
 some times the amount of junk can grow out of control easily. Your code
 will end up stuttering at some point (because of random collections at
 random times), and this can be really bad.



 As I know AAA game engine are reputed to do zero allocations during the
 frame computation, but I think it less the case noways cause of the dynamism
 of scene and huge environments that are streamed.


 Running a game in a zero-alloc environment was a luxury that ended
 about 10 years ago, for reasons that you say.
 Grand Theft Auto proved that you don't need to have loading screens,
 and now it's a basic requirement. We also have a lot more physics,
 environmental destruction, and other dynamic behaviour.

 It was also something we achieved in the past with extreme complexity,
 and the source of many (most?) bugs. We still allocated technically,
 but we had to micro-manage every single little detail, mini pools and
 regions for everything.
 Processors are better now, in the low-frequency code, we can afford to
 spend a tiny bit of time using a standard allocation model. The key is
 the growing separation between low-frequency and high-frequency code.
 On a 33mhz Playstation, there wasn't really much difference between
 the worlds. Now there is, and we can afford to allow 'safety' into the
 language at the cost of a little memory management. We just can't have
 that cost include halting threads all the time for lengthy collect
 processes.
 I know this, because we already use RC extensively in games anyway;
 DirectX uses COM, and most resources use manual RC because we need
 things to release eagerly, and for destructors to work properly. I
 don't see how ARC would add any significant cost to the manual RC that
 is basically standard for many years. It would add simplicity and
 safety, which are highly desirable.



 I recently fix a performance issue due to a code design that force
 destruction of walls (I am working on an architecture application) before
 creating them back when the user move them. gprof, show me that this point
 took around 30% of CPU time in a frame, and only allocations/destructions
 was about 5%. This 5% percents contains destructions of object in the
 architecture part and the 3D engine, same for construction. Construction
 also add operation like new geometry updload, so I don't think place of new
 and delete was high without the job made by constructors and destructors.
 Reserving memory (malloc) isn't really an issue IMO, but operations related
 to constructions and destructions of objects can be expensive.


 It's better to have eager destructors (which empowered you with the
 ability to defer them if you need to), than to not have destructors at
 all, which has been the center if this last few days argument.
 It's not hard to defer resource destruction to a background thread in
 an eager-destruction model.


Yep, to fix this performance bottleneck I just took a look of the call 
graph with cost create by gprof, and it reveals to me the easiest points 
need to be modified a cache insertion. I didn't just differs the 
destruction, cause the majority of times walls stay the same with a new 
position, in that case I can directly reuse the same geometry I just 
have to update it. The other case is when a wall is merged to another 
one (when you put 2 rooms one against each other).

One other issue we have, is the usage of containers of std that are 
really slow in debug (from 60fps to 15fps for parts runnings smoothly), 
on Android it's hard to build some part of application with optimization 
and an other one with.



 May 10 2014



prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 10 May 2014 07:05, Wyatt via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:

 I've been digging into research on the subject while I wait for test scripts
 to run, and my gut feeling is it's definitely possible to get GC at least
 into striking distance, but I'm not nearly an expert on this area.

 (Some of these are dead clever, though! I just read this one today:
 https://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf)


Well, that's a nice paper. But I've seen heaps of people paste heaps
of papers, and nobody EVER come along and say "this, we could do this
in D".
I'm happy to be proven wrong, in fact, there's nothing I'd like more.
I'm not an expert on GC (and I don't really want to be), but I've been
trolling this forum for years and the conversation emerges regularly.
As far as I can tell, there is a de facto agreement that any of those
potentially awesome GC's are not actually feasible in the context of
D.

This is perhaps my biggest mistake in my commitment to D; I came along
and identified it as a red-flag on day 1, but I saw there was lots of
discussion and activity on the matter. I assumed I shouldn't worry
about it, it would sort itself out...
Years later, still, nobody seems to have any idea what to do, at
least, such that it would be addressed in a manner acceptable to my
work.
The only option I know that works is Obj-C's solution, as demonstrated
by a very successful embedded RTOS, and compared to competition, runs
silky smooth. Indeed iOS makes it a specific design goal that it
should always feel silky smooth, never stuttery, they consider it a
point of quality, and I completely agree. I don't know what other
horse to back?


 I don't know how to reconcile the problem with the existing GC,
 and I am not happy to sacrifice large parts of the language for
 it.  I've made the argument before that sacrificing large parts
 of the language as a 'work-around' is, in essence, sacrificing
 practically all libraries. That is a truly absurd notion; to
 suggest that anybody should take advice to sacrifice access to
 libraries is being unrealistic.


 This is important, and simply throwing up our collective hands and saying to
 just not use major language features (I believe I recall slices were in that
 list?) really doesn't sit well with me either.

 But conversely, Manu, something has been bothering me: aren't you restricted
 from using most libraries anyway, even in C++?


No, and this is where everyone seems to completely lose the point.
Just because high performance/realtime code has time critical parts,
that tends not to be much of your code. It is small by volume, and
received disproportionate attention by coders. It's fine, forget about
that bit, except for that it needs to be able to run uninterrupted.
_Most_ of your code is ancillary logic and glue, which typically runs
in response to events, and even though it's execution frequency is
super-low, it's still often triggered in realtime thread (just not
very often).
There are also many background threads you employ to do low priority
tasks where the results aren't an immediate requirement.
Some of these tasks include: resource management, loading and
preparing data, communications/networking, processing low-frequency
work; almost all of these tasks make heavy use of 3rd party libraries,
and allocate.

You can't have an allocation stop the world, because it stops the
realtime threads too, at least, under any mythical GC scheme I'm aware
of that's been proposed as a potential option for D?


 "Decent" or "acceptable"
 performance isn't anywhere near "maximum", so shouldn't any library code
 that allocates in any language be equally suspect?  So from that standpoint,
 isn't any library you use in any language going to _also_ be tuned for
 performance in the hot path?  Maybe I'm barking up the wrong tree, but I
 don't recall seeing this point addressed.


A library which is a foundation of a realtime system will employ
realtime practises. Those are not the libraries I'm worried about.
Most libraries that are useful aren't those libraries. They are tool
libs, and they are typically written to be simple and maintainable,
and usually by a PC developer, with no real consideration towards
specific target applications.


 More generally, I feel like we're collectively missing some important
 context:  What are you _doing_ in your 16.6ms timeslice?  I know _I'd_
 appreciate a real example of what you're dealing with without any hyperbole.


It doesn't matter what I'm doing in my 16ms timeslice most of the
time. I'm running background threads, and also triggering occasional
low frequency events in the realtime thread.
Again, most code by volume is logic and glue, it is not typically
serviced intensively like the core realtime systems, and most often,
by the junior programmers...
I appreciate that I haven't successfully articulated the function of
this code, but that is because to describe "what I'm doing" would be
to give you a million lines of code to nit-pick through. Almost
anything you can imagine, is the answer, as long as it's reasonably
short such that it's not worth the synchronisation costs of queueing
it with a parallel job manager or whatever.
This logic and glue needs to have access to all the conveniences of
the language for productivity and maintainability reasons, and
typically, if you execute only one or 2 of these bits of code per
frame, it will have no meaningful impact on performance... unless it
allocates, triggers a collect, and freezes the system. I repeat, the
juniors... D has lots of safety features to save programmers from
themselves, and I don't consider it a workable option, or goal for the
language, to suggest we should abandon them.

ARC overhead would have no meaningful impact on performance, GC may
potentially freeze execution. I am certain I would never notice ARC
overhead on a profiler, and if I did, there are very simple methods to
shift it elsewhere in the few specific circumstances it emerges.


 What actually _must_ be done in that timeframe?  Why must collection run
 inside that window?  What must be collected when it runs in that situation?
 (Serious questions.)


Anything can and does happen in low-frequency event logic. Collection
'must' run in that window in the event an allocation exists, and there
is no free memory, which is the likely scenario.
strings? closures? array initialisations were a problem (i'm not sure
if that was considered a bug and fixed though?). even some should-be
stack allocations are allocated when the compiler thinks it's a
requirement for safety.
string interaction with C libs is good source of allocations, but
there are many.
Or even small transient allocations, temp's to do small amounts of
work which would otherwise be released upon scope exit. Banning that
sort of practise throws a massive spanner in conventional software
engineering practise that everyone is familiar with.


 See, in the final-by-default discussions, you clearly explained the issues
 and related them well to concerns that are felt broadly, but this... yeah, I
 don't really have any context for this, when D would already be much faster
 than the thirty years of C navel lint (K&R flavour!) that I grapple in my
 day job.


I appreciate your argument. I realise it's why I've had so little
effect... I just can't easily articulate specific instances, because
they are basically unknowable until they happen, and they're hidden by
literally millions of loc. Implicit allocations appear all the time,
and if you deliberately ban them, you quickly find all those language
features don't work and your code gets a lot more difficult to write
and maintain. It's a practicality, risk aversion, and maintenance
issue. If you had to tag main() with  nogc, and then write a
millions-of-loc program in a team of 60, D becomes so much less
compelling argument than otherwise. D has 2 major advantages over C++
as I see. 1, meta, and that's great, but you don't sit and write meta
all the time. 2, productivity and correctness, which I find to be the
more compelling case to be made to adopt D. It affects all programmers
all day, every day, and we lose many aspects of the languages offering
if we tag  nogc, which includes libs.



 May 09 2014



 next sibling parent reply Paulo Pinto <pjmlp progtools.org>  writes:


Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:

 On 10 May 2014 07:05, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 ...


 The only option I know that works is Obj-C's solution, as demonstrated
 by a very successful embedded RTOS, and compared to competition, runs
 silky smooth. Indeed iOS makes it a specific design goal that it
 should always feel silky smooth, never stuttery, they consider it a
 point of quality, and I completely agree. I don't know what other
 horse to back?
 ...


The problem when comparing iOS with Android, is that we aren't comparing 
ARC with GC.

We are comparing a full OS, which we don't know how much ARC is actually 
used versus standard malloc/new with another OS, which has a
so-and-so VM implementation, used mostly software rendering until 
version 4.1, and care for low end devices was only done in 4.4.

If we add Windows Phone to the mix, then we have a .NET stack like 
Android (WP7 - GC/JIT) or in its sucessor (WP8)  native code generation 
for .NET (GC) with a COM model for OS APIs (ARC).

Both versions of Windows Phone run smoother that many Android phones, 
even the WP7 ones.

Not saying you are not right, just that we need to look at the whole 
stack when comparing mobile OS, not just GC vs ARC.

--
Paulo



 May 09 2014



 next sibling parent  "Paolo Invernizzi" <paolo.invernizzi no.address>  writes:


On Saturday, 10 May 2014 at 06:53:07 UTC, Paulo Pinto wrote:

 Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:

 On 10 May 2014 07:05, Wyatt via Digitalmars-d


 The problem when comparing iOS with Android, is that we aren't 
 comparing ARC with GC.

 We are comparing a full OS, which we don't know how much ARC is 
 actually used versus standard malloc/new with another OS, which 
 has a
 so-and-so VM implementation, used mostly software rendering 
 until version 4.1, and care for low end devices was only done 
 in 4.4.

 If we add Windows Phone to the mix, then we have a .NET stack 
 like Android (WP7 - GC/JIT) or in its sucessor (WP8)  native 
 code generation for .NET (GC) with a COM model for OS APIs 
 (ARC).

 Both versions of Windows Phone run smoother that many Android 
 phones, even the WP7 ones.

 Not saying you are not right, just that we need to look at the 
 whole stack when comparing mobile OS, not just GC vs ARC.

 --
 Paulo


+1
--
Paolo



 May 10 2014



prev sibling next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 10/05/2014 08:53, Paulo Pinto a écrit :

 Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:

 On 10 May 2014 07:05, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 ...


 The only option I know that works is Obj-C's solution, as demonstrated
 by a very successful embedded RTOS, and compared to competition, runs
 silky smooth. Indeed iOS makes it a specific design goal that it
 should always feel silky smooth, never stuttery, they consider it a
 point of quality, and I completely agree. I don't know what other
 horse to back?
 ...


 The problem when comparing iOS with Android, is that we aren't comparing
 ARC with GC.

 We are comparing a full OS, which we don't know how much ARC is actually
 used versus standard malloc/new with another OS, which has a
 so-and-so VM implementation, used mostly software rendering until
 version 4.1, and care for low end devices was only done in 4.4.

 If we add Windows Phone to the mix, then we have a .NET stack like
 Android (WP7 - GC/JIT) or in its sucessor (WP8)  native code generation
 for .NET (GC) with a COM model for OS APIs (ARC).

 Both versions of Windows Phone run smoother that many Android phones,
 even the WP7 ones.

 Not saying you are not right, just that we need to look at the whole
 stack when comparing mobile OS, not just GC vs ARC.

 --
 Paulo


I don't know well WP8 models, but this one must run smoothly :
http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

Just like Android phones, the battery is huge : 3400mAh
It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
Only RAM seems reasonable : 1Go

And it's maybe easier for Microsoft to do a GC with good performances 
cause they control everything on the OS. There is certainly some 
specifics memory management in the kernel related to the GC. When Google 
"just" put small pieces together based on a linux kernel.

Yesterday I played with my nexus 5 at a game in augmented reality, it 
took 2hr 20 minutes to completely drain the battery (2300 mAh). Sadly 
this game doesn't exist on iOS but IMO an iPhone would do the same with 
a smaller battery (1440 mAh for iPhone 5s).



 May 10 2014



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 10 May 2014 19:43, Xavier Bigand via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 I don't know well WP8 models, but this one must run smoothly :
 http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

 Just like Android phones, the battery is huge : 3400mAh
 It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
 Only RAM seems reasonable : 1Go

 And it's maybe easier for Microsoft to do a GC with good performances cause
 they control everything on the OS. There is certainly some specifics memory
 management in the kernel related to the GC. When Google "just" put small
 pieces together based on a linux kernel.


They don't GC. MS are behind their C++/CX now, which introduced an ARC
pointer type '^'.



 May 10 2014



  parent reply Paulo Pinto <pjmlp progtools.org>  writes:


Am 10.05.2014 15:37, schrieb Manu via Digitalmars-d:

 On 10 May 2014 19:43, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 I don't know well WP8 models, but this one must run smoothly :
 http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

 Just like Android phones, the battery is huge : 3400mAh
 It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
 Only RAM seems reasonable : 1Go

 And it's maybe easier for Microsoft to do a GC with good performances cause
 they control everything on the OS. There is certainly some specifics memory
 management in the kernel related to the GC. When Google "just" put small
 pieces together based on a linux kernel.


 They don't GC. MS are behind their C++/CX now, which introduced an ARC
 pointer type '^'.


.NET Applications on Windows Phone 8 make use of both (GC and COM RC).


--
Paulo



 May 10 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 01:44, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am 10.05.2014 15:37, schrieb Manu via Digitalmars-d:

 On 10 May 2014 19:43, Xavier Bigand via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:

 I don't know well WP8 models, but this one must run smoothly :

 http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

 Just like Android phones, the battery is huge : 3400mAh
 It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
 Only RAM seems reasonable : 1Go

 And it's maybe easier for Microsoft to do a GC with good performances
 cause
 they control everything on the OS. There is certainly some specifics
 memory
 management in the kernel related to the GC. When Google "just" put small
 pieces together based on a linux kernel.



 They don't GC. MS are behind their C++/CX now, which introduced an ARC
 pointer type '^'.


 .NET Applications on Windows Phone 8 make use of both (GC and COM RC).


I don't know any gamedev's who use .NET on WP8. C++/CX was a good step
forward, and addressed one of the primary inhibitors to developer
adoption of WP7.
I don't know a single developer who ever released a WP7 app. WP7 was a
ghost town compared to iOS and Android in terms of developer support.
Lots of people I know are happily supporting WP8 now thanks to C++/CX.



 May 10 2014



  parent  Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 10/05/2014 17:58, Manu via Digitalmars-d a écrit :

 On 11 May 2014 01:44, Paulo Pinto via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Am 10.05.2014 15:37, schrieb Manu via Digitalmars-d:

 On 10 May 2014 19:43, Xavier Bigand via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:

 I don't know well WP8 models, but this one must run smoothly :

 http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

 Just like Android phones, the battery is huge : 3400mAh
 It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
 Only RAM seems reasonable : 1Go

 And it's maybe easier for Microsoft to do a GC with good performances
 cause
 they control everything on the OS. There is certainly some specifics
 memory
 management in the kernel related to the GC. When Google "just" put small
 pieces together based on a linux kernel.



 They don't GC. MS are behind their C++/CX now, which introduced an ARC
 pointer type '^'.


 .NET Applications on Windows Phone 8 make use of both (GC and COM RC).


 I don't know any gamedev's who use .NET on WP8. C++/CX was a good step
 forward, and addressed one of the primary inhibitors to developer
 adoption of WP7.
 I don't know a single developer who ever released a WP7 app. WP7 was a
 ghost town compared to iOS and Android in terms of developer support.
 Lots of people I know are happily supporting WP8 now thanks to C++/CX.


+1
WP7 force all gamedev using c++ to rewrite everything, it's completely 
infeasible. I heard Microsoft propose to paid some studios to port their 
products, but a lot refuse this offer.



 May 10 2014



prev sibling  parent  luckoverthere <luckoverthere gmail.cm>  writes:


On Wednesday, 28 November 2018 at 09:16:14 UTC, VirtualMaria 
wrote:

 On Saturday, 10 May 2014 at 09:43:46 UTC, Xavier Bigand wrote:

 Yesterday I played with my nexus 5 at a game in augmented 
 reality, it took 2hr 20 minutes to completely drain the 
 battery (2300 mAh). Sadly this game doesn't exist on iOS but 
 IMO an iPhone would do the same with a smaller battery (1440 
 mAh for iPhone 5s).


 Luckily, as time passes, technologies are ready to offer us 
 more and more. Now, VR or AR applications are available on 
 Androis, iOS and other platforms and I would say they work 
 similarly. I use it on iOS, but now I think about changing 
 mobile phone and read how AR will work on Android. Multiple 
 resources like 
 https://www.androidauthority.com/best-augmented-reality-apps-and-ar-apps
for-android-584616/ or https://jasoren.com/how-to-create-content-for-a-vr-app/
assure me that it will function properly. I hope so!


Was confused when I saw a thread started by Andrei, then I saw 
the date and now it makes sense it was at a time when the iphone 
5s was considered top of the line.



 Nov 28 2018



prev sibling  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 10 May 2014 16:53, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:

 On 10 May 2014 07:05, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 ...


 The only option I know that works is Obj-C's solution, as demonstrated
 by a very successful embedded RTOS, and compared to competition, runs
 silky smooth. Indeed iOS makes it a specific design goal that it
 should always feel silky smooth, never stuttery, they consider it a
 point of quality, and I completely agree. I don't know what other
 horse to back?
 ...



 The problem when comparing iOS with Android, is that we aren't comparing ARC
 with GC.

 We are comparing a full OS, which we don't know how much ARC is actually
 used versus standard malloc/new with another OS, which has a
 so-and-so VM implementation, used mostly software rendering until version
 4.1, and care for low end devices was only done in 4.4.

 If we add Windows Phone to the mix, then we have a .NET stack like Android
 (WP7 - GC/JIT) or in its sucessor (WP8)  native code generation for .NET
 (GC) with a COM model for OS APIs (ARC).

 Both versions of Windows Phone run smoother that many Android phones, even
 the WP7 ones.

 Not saying you are not right, just that we need to look at the whole stack
 when comparing mobile OS, not just GC vs ARC.


Yes, granted. I shouldn't make that sort of comparison. What I am
really trying to do with such comments, is to insist that it is used
as a foundation for a very successful RTOS. To say "it's no good. ARC
can't work. it can never be used as a foundation for a language. but,
cycles!" and whatever, is kind of ignoring the evidence.
It's interesting too that you bring Windows phone into the mix,
because all the WP8 developers I know use microsoft's C++/CX, which
has the new '^' ARC pointer type (as used intensively by the OS).
Why did MS abandon GC? It looks like a similar sort of back-peddling
from their commitment to GC as did Apple, when they started making a
serious play for the embedded market.


software. I have lots of colleagues who rant about it regularly, and
my experience from 2 weekends back that I shared earlier in this
thread is a scary slap in the face for me.



 May 10 2014



  parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 10.05.2014 15:28, schrieb Manu via Digitalmars-d:

 On 10 May 2014 16:53, Paulo Pinto via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:

 On 10 May 2014 07:05, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 ...


 The only option I know that works is Obj-C's solution, as demonstrated
 by a very successful embedded RTOS, and compared to competition, runs
 silky smooth. Indeed iOS makes it a specific design goal that it
 should always feel silky smooth, never stuttery, they consider it a
 point of quality, and I completely agree. I don't know what other
 horse to back?
 ...



 The problem when comparing iOS with Android, is that we aren't comparing ARC
 with GC.

 We are comparing a full OS, which we don't know how much ARC is actually
 used versus standard malloc/new with another OS, which has a
 so-and-so VM implementation, used mostly software rendering until version
 4.1, and care for low end devices was only done in 4.4.

 If we add Windows Phone to the mix, then we have a .NET stack like Android
 (WP7 - GC/JIT) or in its sucessor (WP8)  native code generation for .NET
 (GC) with a COM model for OS APIs (ARC).

 Both versions of Windows Phone run smoother that many Android phones, even
 the WP7 ones.

 Not saying you are not right, just that we need to look at the whole stack
 when comparing mobile OS, not just GC vs ARC.


 Yes, granted. I shouldn't make that sort of comparison. What I am
 really trying to do with such comments, is to insist that it is used
 as a foundation for a very successful RTOS. To say "it's no good. ARC
 can't work. it can never be used as a foundation for a language. but,
 cycles!" and whatever, is kind of ignoring the evidence.
 It's interesting too that you bring Windows phone into the mix,
 because all the WP8 developers I know use microsoft's C++/CX, which
 has the new '^' ARC pointer type (as used intensively by the OS).
 Why did MS abandon GC? It looks like a similar sort of back-peddling
 from their commitment to GC as did Apple, when they started making a
 serious play for the embedded market.


They did not. A Windows Phone 8 application, if coded in .NET still uses 
a GC for the .NET world.

http://blogs.msdn.com/b/abhinaba/archive/2012/11/08/windows-phone-8-evolution-of-the-runtime-and-application-compatibility.aspx

ARC is used in C++/CX due to the COM nature of WinRT.

After all the Windows Runtime is nothing more than COM with
IInspectable as new derived interface from IUnknown.

Which was actually the original design of .NET, COM+ VOS, before they 
decided to go MSIL.

http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-components-postattachments/00-10-32-72-38/Ext_2D00_VOS.pdf


If anything, the native side has won the current internal politics war 
and they are steering the boat back to ahead of time compilation.




 software. I have lots of colleagues who rant about it regularly, and
 my experience from 2 weekends back that I shared earlier in this
 thread is a scary slap in the face for me.


Where those experiences with Unity/Mono or proper .NET?

Unity still uses Mono 2.6 from 2009, when Mono was under SuSE control. 
Entity to whom Unity has an agreement.

http://unity3d.com/unity/faq#section-1828
==> What version of Mono is used for 4.x? Can I use all .NET features?

A new generational garbage collector was introduced in Mono with version 
2.8 (2010) and improved in consequent releases.

http://www.mono-project.com/Release_Notes_Mono_2.8

It became the default GC in the latest version, currently 3.2.

--
Paulo



 May 10 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.


This is very, very, very wrong. -- Andrei



 May 10 2014



 next sibling parent reply "Francesco Cattoglio" <francesco.cattoglio gmail.com>  writes:


On Saturday, 10 May 2014 at 07:08:04 UTC, Andrei Alexandrescu 
wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.


 This is very, very, very wrong. -- Andrei


I've seen this discussion ("it's almost performance-free", "it's 
a performance killer") so many times, I can't even say who has 
the burden of proof anymore.



 May 10 2014



  parent reply "w0rp" <devw0rp gmail.com>  writes:


On Saturday, 10 May 2014 at 07:42:05 UTC, Francesco Cattoglio 
wrote:

 On Saturday, 10 May 2014 at 07:08:04 UTC, Andrei Alexandrescu 
 wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.


 This is very, very, very wrong. -- Andrei


 I've seen this discussion ("it's almost performance-free", 
 "it's a performance killer") so many times, I can't even say 
 who has the burden of proof anymore.


I wish that someone would take the time and implement ARC in D. 
That's the only way to prove anything. If you implement it and 
you can provide clear evidence for its advantages, then that just 
ends all discussions.



 May 10 2014



  parent  "Nick B" <nick.barbalich gmail.com>  writes:


On Saturday, 10 May 2014 at 08:18:30 UTC, w0rp wrote:


 I've seen this discussion ("it's almost performance-free", 
 "it's a performance killer") so many times, I can't even say 
 who has the burden of proof anymore.


 I wish that someone would take the time and implement ARC in D. 
 That's the only way to prove anything. If you implement it and 
 you can provide clear evidence for its advantages, then that 
 just ends all discussions.


How hard would this be, exactly ?

Perhaps then, and only then, could you make a apples to apples 
comparison ?

Nick



 May 10 2014



prev sibling  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.



 This is very, very, very wrong. -- Andrei


Is there any evidence to support that assertion?
I have many colleagues who work on phones. The patterns I see are that
the iOS guys who use Obj-C never complain about it, and the GC guys
are often struggling with it.
The bias in gamedev is strongly towards iOS too, so that should be a
more populous group to draw samples from.



 May 10 2014



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via Digitalmars-d 
wrote:

 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.



 This is very, very, very wrong. -- Andrei


 Is there any evidence to support that assertion?
 I have many colleagues who work on phones. The patterns I see 
 are that
 the iOS guys who use Obj-C never complain about it, and the GC 
 guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should 
 be a
 more populous group to draw samples from.


You conflate freezing and performance. ARC easily can be slower 
but is has benefit of slowdown distributed more evenly thus not 
obviously notable.



 May 10 2014



  parent reply "Peter Alexander" <peter.alexander.au gmail.com>  writes:


On Saturday, 10 May 2014 at 13:50:05 UTC, Dicebot wrote:

 On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via 
 Digitalmars-d wrote:

 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.



 This is very, very, very wrong. -- Andrei


 Is there any evidence to support that assertion?
 I have many colleagues who work on phones. The patterns I see 
 are that
 the iOS guys who use Obj-C never complain about it, and the GC 
 guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that 
 should be a
 more populous group to draw samples from.


 You conflate freezing and performance. ARC easily can be slower 
 but is has benefit of slowdown distributed more evenly thus not 
 obviously notable.


It should be noted that reference counting can also cause large 
stalls when a root object is freed, causing a cascade of 
decrements-to-zero, which would have been handled better by a GC.



 May 10 2014



  parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 00:23, Peter Alexander via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Saturday, 10 May 2014 at 13:50:05 UTC, Dicebot wrote:

 On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via Digitalmars-d wrote:


 It should be noted that reference counting can also cause large stalls when
 a root object is freed, causing a cascade of decrements-to-zero, which would
 have been handled better by a GC.


That problem is sooo easy to deal with by contrast. And the programmer
maintains control.



 May 10 2014



prev sibling next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com>  writes:


On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via Digitalmars-d 
wrote:

 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.



 This is very, very, very wrong. -- Andrei


 Is there any evidence to support that assertion?


It is well known that reference counting can perform poorly. The 
updates damage the cache and atomic reference counts will stall 
pipelines.

Of course, there are things that can be done to mitigate the 
impact, but the same is true for GC.

Just Google for reference counting performance.



 I have many colleagues who work on phones. The patterns I see 
 are that
 the iOS guys who use Obj-C never complain about it, and the GC 
 guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should 
 be a
 more populous group to draw samples from.


Of course. In game dev a large frame time spike is worse, and 
more obvious/noticeable than a consistent and mostly predictable 
performance hit.



 May 10 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 00:20, Peter Alexander via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via Digitalmars-d wrote:

 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.




 This is very, very, very wrong. -- Andrei



 Is there any evidence to support that assertion?



 It is well known that reference counting can perform poorly. The updates
 damage the cache and atomic reference counts will stall pipelines.

 Of course, there are things that can be done to mitigate the impact, but the
 same is true for GC.

 Just Google for reference counting performance.


I agree that acceptable performance for ARC requires compiler support.
What doesn't? Acceptable performance in general requires extensive
compiler support (in the form of optimisation). Why should this be
considered any differently?

D is thread-local by default. That's a blessing for RC, the
requirement for atomics will be minimal.
D also has pure, which helps a lot. Hopefully one day we'll have
'scope' fleshed out to make proper escaping guarantees in non-pure
functions too, that would really help.

There are strategies to reduce impact on the dcache.

It just seems like a much more approachable problem with actual
options available than the mythical awesome GC.


 I have many colleagues who work on phones. The patterns I see are that
 the iOS guys who use Obj-C never complain about it, and the GC guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should be a
 more populous group to draw samples from.



 Of course. In game dev a large frame time spike is worse, and more
 obvious/noticeable than a consistent and mostly predictable performance hit.


I can't think of many situations where that wouldn't be the case. What
sort of software is it not an issue to experience intermittent
freezing?
Is it shown that cumulative RC fiddling costs significantly exceed a
GC collect pass? Recall too that D has significant opportunity to
improve on ARC as implemented by other languages, and D's GC is much
worse than most, so if you considered the same competitive difference
in the context of D, how would it look then?

It's funny actually, D offers many advantages towards ARC that other
languages don't enjoy... but by contrast, D barely supports the notion
of a GC it seems, or surely we'd have had more people experimenting
with cool advanced implementations by now?



 May 10 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2014 8:54 AM, Manu via Digitalmars-d wrote:

 I can't think of many situations where that wouldn't be the case. What
 sort of software is it not an issue to experience intermittent
 freezing?


Batch programs, for example a compiler.

Essentially any program that is not interactive and does not have hard realtime 
requirements, which is (possibly) most programs.


 Recall too that D has significant opportunity to
 improve on ARC as implemented by other languages,


Compiler improvements to improve on ARC are the same technology as used to 
improve GC.

You're essentially arguing that one is easy pickings and the other is 
impractically difficult, but they're actually about the same level.



 May 10 2014



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 08:40, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/10/2014 8:54 AM, Manu via Digitalmars-d wrote:

 I can't think of many situations where that wouldn't be the case. What
 sort of software is it not an issue to experience intermittent
 freezing?



 Batch programs, for example a compiler.


Indeed, that was the only one that came to my mind. GC might be better
for shell apps because it is unlikely that a collect will ever occur,
it'll usually terminate before it gets to it, but there's certainly no
hard-dependency on GC in this use case.
This is truly a niche usage case though, and yet, it also has no
particular incompatibility with any choice of technology, only a
slight preference towards GC for the reason that you never bother
freeing memory in DMD.

If you're prepared to label the largest entertainment industry on the
planet a niche, then this sort of work is a hyper-niche... and
regardless, not actually incompatible with any particular option.

I'd like to consider this form of software for a moment though.
Assuming the GC was a library offered beside the language, DDMD for
instance would be able to make particularly easy and convenient use of
it.
Shell apps by comparison _are_ generally small and self-contained.
It's far more reasonable to suggest that a shell app can adopt
specific and custom allocation patterns when the code is relatively
small, function is specific, the life is short, the probability of
running into incompatible 3rd party libraries is almost zero by
comparison.
How many libs does DMD link?

This thought has actually left me slightly concerned, that as much as
Andrei is accusing me of unreasonably arguing my "staunch niche
position that ignores factual realities" (a somewhat offensive and
dismissive notion), that there might be some silent, but significant
(perhaps even unconscious) element of the same in reverse.



 Essentially any program that is not interactive and does not have hard
 realtime requirements, which is (possibly) most programs.


I'm hard pressed to think of any software I see people using every day
which isn't realtime in some sense. The majority of relevant software
today is on embedded devices, and visceral touch-based UI is key to
their impression of quality.



 Recall too that D has significant opportunity to
 improve on ARC as implemented by other languages,



 Compiler improvements to improve on ARC are the same technology as used to
 improve GC.

 You're essentially arguing that one is easy pickings and the other is
 impractically difficult, but they're actually about the same level.


Are they? This is the first I've heard such a claim. If that's the
case, then that why is there an argument? That work just needs to be
done, and we can experiment with both targets...
I can fairly easily visualise the path to ARC, but by all prior
reports I've heard, awesome GC is practically incompatible with D for
various reasons. There were discussions about different advanced
options by the experts at dconf last year, which seemed to hit various
impasses.
And I'm still not even sure if 'awesome GC' will solve my problems?
Will it? No matter how awesome it is, it seems conceptually
incompatible with my environment.
I keep saying, I'm more than happy to be proved wrong (really!), but
until someone can, then it's unfair to dismiss my arguments so
easily... and I just don't think it's that unreasonable. ARC is an
extremely successful technology, particularly in the
compiled/native/systems language space (OC, C++/CX, Rust). Is there
actually any evidence of significant GC success in this space?
Successes all seem to be controlled VM based languages like Java and

native worlds. There must be good reason for that apparent separation
in trends?



 May 10 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2014 8:58 PM, Manu via Digitalmars-d wrote:

 This is truly a niche usage case though,


Come on! Like about 80% of the programs on any linux box? Like the OCR program
I 
run? A payroll processing program? Any scientific numerical analysis program? 
Engineering programs?



 If you're prepared to label the largest entertainment industry on the
 planet a niche,


I'm not doing that.



 How many libs does DMD link?


We've gone over this before. You were concerned that the libraries you linked 
with were incompetently written, and implied that if ARC was pervasive, they 
would be competently written. I can guarantee you, however, that ARC leaves 
plenty of opportunity for incompetence :-)



 I'm hard pressed to think of any software I see people using every day
 which isn't realtime in some sense.


Programs you "see" by definition have a user interface. But an awful lot of 
programs are not seen, but that doesn't mean they aren't there and aren't 
running. See my list above.



 Compiler improvements to improve on ARC are the same technology as used to
 improve GC.

 You're essentially arguing that one is easy pickings and the other is
 impractically difficult, but they're actually about the same level.


 Are they? This is the first I've heard such a claim. If that's the
 case,


Yes.



 then that why is there an argument?


Because if this was an easy problem, it would have been solved. In particular, 
if the ARC overhead was easily removed by simple compiler enhancements, why 
hasn't ARC taken the world by storm? It's not like ARC was invented yesterday.



 That work just needs to be done,


That's a massive understatement. This is PhD research topic material, not 
something I can churn out in a week or two if only I had a more positive 
attitude :-)


 but by all prior
 reports I've heard, awesome GC is practically incompatible with D for
 various reasons.


There is no such thing as a GC which would satisfy your requirements.


 No matter how awesome it is, it seems conceptually
 incompatible with my environment.


My point!


 I keep saying, I'm more than happy to be proved wrong (really!), but
 until someone can, then it's unfair to dismiss my arguments so
 easily...


I believe I have answered your arguments, not dismissed them.


 and I just don't think it's that unreasonable. ARC is an
 extremely successful technology, particularly in the
 compiled/native/systems language space (OC, C++/CX,


What was dismissed is the reality pointed out many times that those systems 
resolve the perf problems of ARC by providing numerous means of manually 
escaping it, with the resulting desecration of soundness guarantees.



 Rust).


Rust is not an extremely successful technology. It's barely even been
implemented.



 Is there
 actually any evidence of significant GC success in this space?
 Successes all seem to be controlled VM based languages like Java and

 native worlds. There must be good reason for that apparent separation
 in trends?


There are many techniques for mitigating GC problems in D, techniques that are 

 nogc to guarantee the GC pause troll isn't going to pop up unexpectedly. There 
are a bunch of other techniques, too.

ARC simply is not a magic, no problem solution one can use without careful 
thought in large, complex systems. (Of course, neither is GC nor any other 
memory management scheme.)



 May 10 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 14:57, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/10/2014 8:58 PM, Manu via Digitalmars-d wrote:

 This is truly a niche usage case though,



 Come on! Like about 80% of the programs on any linux box? Like the OCR
 program I run? A payroll processing program? Any scientific numerical
 analysis program? Engineering programs?


Linux programs aren't at risk of being ported from C any time soon.
And If the GC is a library, shell apps probably represent the set
where it is the most convenient to make use of a GC lib.
Accounting software is UI based software. It's just of a sort that
people aren't likely to notice or complain if it stutters
occasionally.
Engineering programs are typically realtime in some way. Most
productivity software of that sort (CAD, ART/design, etc) is highly
interactive.

Tools like OCR as a shell app, refer to the short-lived shell app
point. OCR as a feature of an art package (like photoshop), again,
highly interactive productivity software.



 How many libs does DMD link?



 We've gone over this before. You were concerned that the libraries you
 linked with were incompetently written, and implied that if ARC was
 pervasive, they would be competently written. I can guarantee you, however,
 that ARC leaves plenty of opportunity for incompetence :-)


It's not about incompetence, it's about incompatibility.
If I can't tolerate a collect, not only do I sacrifice extensive and
productive parts of the language, this means a library which I have no
control over is banned from the GC outright. That is entirely
unrealistic.

I can approach a performance hazard which manifests locally, as with
ARC. I have no way to address the sort that manifests in random
places, at random times, for reasons that are outside of my control,
interferes with the entire application, and increases in frequency as
the free memory decreases (read: as I get closer to shipping day).



 I'm hard pressed to think of any software I see people using every day
 which isn't realtime in some sense.



 Programs you "see" by definition have a user interface. But an awful lot of
 programs are not seen, but that doesn't mean they aren't there and aren't
 running. See my list above.


Sure, and of the subset of those which run in a time-critical
environment, they are the best candidates to make use of GC as a lib.
As I see, most of those apps these days are more likely to be written
in a language like python, so it's hard to make any raw performance
argument about that category of software in general. It would be an
extremely small subset where it is a consideration.
Conversely, all realtime/UI based/user-facing software cares about
performance and stuttering.



 then that why is there an argument?



 Because if this was an easy problem, it would have been solved. In
 particular, if the ARC overhead was easily removed by simple compiler
 enhancements, why hasn't ARC taken the world by storm? It's not like ARC was
 invented yesterday.


And as far as I can tell, it has, at least in this
(native/compiled/systems) space. O-C, C++/CX, Rust... where are the
counter-examples?

compared to the other native languages which do truly exist on the
same playing field.



 That work just needs to be done,



 That's a massive understatement. This is PhD research topic material, not
 something I can churn out in a week or two if only I had a more positive
 attitude :-)


That's okay, I am just looking for direction. I want to see a
commitment to a path I can get behind. Not a path that, however many
years from now, I still have good reason to believe it won't satisfy
my requirements.
All my energy in the meantime would be a waste in that case.



 but by all prior
 reports I've heard, awesome GC is practically incompatible with D for
 various reasons.



 There is no such thing as a GC which would satisfy your requirements.


Then... the argument is finished.
You have a choice. You may choose to explore the more inclusive
technology (which also solves some other outstanding language
problems, like destructors), or you confirm D is married to GC, and I
consider my options and/or future involvement in that context.



 No matter how awesome it is, it seems conceptually
 incompatible with my environment.



 My point!


My point too!



 and I just don't think it's that unreasonable. ARC is an
 extremely successful technology, particularly in the
 compiled/native/systems language space (OC, C++/CX,



 What was dismissed is the reality pointed out many times that those systems
 resolve the perf problems of ARC by providing numerous means of manually
 escaping it, with the resulting desecration of soundness guarantees.


You've said that, but I don't think there's any hard evidence of that.
It is an option, of course, which is extremely valuable to have, but I
don't see any evidence that it is a hard requirement.
Andrei's paper which asserted that modern ARC fell within 10% of "the
fastest GC" didn't make that claim with the caveat that "extensive
unsafe escaping was required to produce these results".



 Rust).



 Rust is not an extremely successful technology. It's barely even been
 implemented.


Maybe... but they probably had extensive arguments on the same issues,
and their findings should surely be included among the others. It's
definitely modern, and I'm sure it was considered from a modern point
of view, which I think is meaningful.



 Is there
 actually any evidence of significant GC success in this space?
 Successes all seem to be controlled VM based languages like Java and

 native worlds. There must be good reason for that apparent separation
 in trends?



 There are many techniques for mitigating GC problems in D, techniques that

 use  nogc to guarantee the GC pause troll isn't going to pop up
 unexpectedly. There are a bunch of other techniques, too.


RC is no good without compiler support. One second you're arguing
precisely this case that useful RC requires extensive compiler support
to be competitive (I completely agree), and then you flip about and I
hear the "use RefCounted!" argument again (which also has no influence
on libraries I depend on).

I've argued before that  nogc has no practical effect, unless you tag
it on main(), and then D is as good as if it didn't have memory
management at all. Non-C libraries are practically eliminated. Sounds
realistic in a shell app perhaps, but not in a major software package.
D's appeal depends largely on it's implicit memory management, and
convenience/correctness oriented constructs that it enables.

Both these suggestions only have any effect over my local code, which
ignores the library problem again.



 ARC simply is not a magic, no problem solution one can use without careful
 thought in large, complex systems. (Of course, neither is GC nor any other
 memory management scheme.)


I have never claimed it's magic. It's **workable**. GC is apparently
not, as you admitted a few paragraphs above.
The key difference is that ARC cost is localised, which presents many
options. GC cost is unpredictable, and gets progressively worse as
environments become more and more like mine. Short of banning memory
management program-wide (absurd, it's 2014), or having such an excess
(waste) of available resources that I'm sabotaging competitive
distinction, theres not really workable options.

If it's true that ARC falls within 10% of the best GC's, surely it
must be considered a serious option, especially considering we've
started talking about ideas like "maybe we should make things with
destructors lower to use ARC"?

Performance, it turns out, is apparently much more similar than I had
imagined, which would lead me to factor that out as a significant
consideration. Which is the more _inclusive_ option?
And unless D is capable of 'worlds fastest GC', then ARC would
apparently be a speed improvement over the current offering too.



 May 10 2014



 next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/10/14, 11:27 PM, Manu via Digitalmars-d wrote:

 On 11 May 2014 14:57, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/10/2014 8:58 PM, Manu via Digitalmars-d wrote:

 This is truly a niche usage case though,



 Come on! Like about 80% of the programs on any linux box? Like the OCR
 program I run? A payroll processing program? Any scientific numerical
 analysis program? Engineering programs?


 Linux programs aren't at risk of being ported from C any time soon.


Is this seriously being aired as an argument? -- Andrei



 May 11 2014



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/10/2014 11:27 PM, Manu via Digitalmars-d wrote:

 Because if this was an easy problem, it would have been solved. In
 particular, if the ARC overhead was easily removed by simple compiler
 enhancements, why hasn't ARC taken the world by storm? It's not like ARC was
 invented yesterday.


 And as far as I can tell, it has, at least in this
 (native/compiled/systems) space. O-C, C++/CX, Rust... where are the
 counter-examples?


Again, O-C and C++/CX ARC are not memory safe because in order to make it 
perform they provide unsafe escapes from it. Neither even attempts pervasive
ARC.

Rust is simply not an example of proven technology.

We cannot even discuss this if we cannot agree on basic, objective facts.


 It's **workable**.


Nobody has demonstrated that pervasive ARC is both performant and memory safe.

Have you ever written some code using RC in O-C or C++/CX, and disassembled it 
to see what it looks like? Do you realize that every decrement must happen 
inside an exception handler? Now imagine that for all code that deals with
pointers?

------------- A Comment on Rust ------------------

This is based on my very incomplete knowledge of Rust, i.e. just reading a few 
online documents on it. If I'm wrong, please correct me.

Rust's designers apparently are well aware of the performance cost of pervasive 
ARC. Hence, they've added the notion of a "borrowed" pointer, which is an
escape 
from ARC. The borrowed pointer is made memory safe by:

1. Introducing restrictions on what can be done with a borrowed pointer so the 
compiler can determine its lifetime. I do not know the extent of these
restrictions.

2. Introducing an annotation to distinguish a borrowed pointer from an ARC 
pointer. If you don't use the annotation, you get pervasive ARC with all the 
poor performance that entails.

Implicit in borrowed pointers is Rust did not solve the problem of having the 
compiler eliminate unnecessary inc/dec.


My experience with pointer annotations to improve performance is pretty 
compelling - almost nobody adds those annotations. They get their code to work 
with the default, and never get around to annotating it. This, of course, 
provided me with a large opportunity to kick ass in the performance dept. 
because I would use them, but that didn't help when I had to use other peoples' 
code.

People who have added annotations to Java have seen the same result. They can't 
get regular programmers to use them.

The annotations have their downsides even if you make the effort to use them. 
Since they are a different type from ARC pointers, you cannot have a data 
structure, say a tree, that contains both (without having a tag to say which
one 
it is). They do not mix. A function taking one type of pointer cannot be called 
with the other type.

Worse, these effects are transitive, making a function hierarchy rather
inflexible.

Are these valid concerns with Rust? I haven't written any Rust code, and I 
haven't heard of a whole lot of Rust code being written. So I don't know. We'll
see.



 May 11 2014



 next sibling parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-05-11 08:29:13 +0000, Walter Bright <newshound2 digitalmars.com> said:


 Again, O-C and C++/CX ARC are not memory safe because in order to make 
 it perform they provide unsafe escapes from it.


But D could provide memory-safe escapes. If we keep the current GC to 
collect cycles, we could also allow raw pointers managed by the GC 
alongside ARC.

Let's say we have two kinds of pointers: rc+gc pointers (the default) 
and gc_only pointers (on demand). When assigning from a rc+gc pointer 
to a gc_only pointer, the compiler emits code that disables destruction 
via the reference counting. This makes the GC solely responsible for 
destructing and deallocating that memory block. You can still assign 
the pointer to a rc+gc pointer later on, but the reference count is no 
longer reliable which is why RC-based destruction has been disabled.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 May 11 2014



 next sibling parent reply "w0rp" <devw0rp gmail.com>  writes:


On Sunday, 11 May 2014 at 12:52:29 UTC, Michel Fortin wrote:

 On 2014-05-11 08:29:13 +0000, Walter Bright 
 <newshound2 digitalmars.com> said:


 Again, O-C and C++/CX ARC are not memory safe because in order 
 to make it perform they provide unsafe escapes from it.


 But D could provide memory-safe escapes. If we keep the current 
 GC to collect cycles, we could also allow raw pointers managed 
 by the GC alongside ARC.

 Let's say we have two kinds of pointers: rc+gc pointers (the 
 default) and gc_only pointers (on demand). When assigning from 
 a rc+gc pointer to a gc_only pointer, the compiler emits code 
 that disables destruction via the reference counting. This 
 makes the GC solely responsible for destructing and 
 deallocating that memory block. You can still assign the 
 pointer to a rc+gc pointer later on, but the reference count is 
 no longer reliable which is why RC-based destruction has been 
 disabled.


You know, this doesn't sound that far off from what Python does, 
unless I'm completely wrong about it. I believe that Python uses 
reference counting and uses GC to collect cycles in a way like 
you have described. I'm not sure how efficient it is. Python 
people don't tend to talk about speed that much.



 May 11 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 7:55 AM, w0rp wrote:

 Python people don't tend to talk about speed that much.


:-)



 May 11 2014



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 5:52 AM, Michel Fortin wrote:

 On 2014-05-11 08:29:13 +0000, Walter Bright <newshound2 digitalmars.com> said:


 Again, O-C and C++/CX ARC are not memory safe because in order to make it
 perform they provide unsafe escapes from it.


 But D could provide memory-safe escapes. If we keep the current GC to collect
 cycles, we could also allow raw pointers managed by the GC alongside ARC.

 Let's say we have two kinds of pointers: rc+gc pointers (the default) and
 gc_only pointers (on demand). When assigning from a rc+gc pointer to a gc_only
 pointer, the compiler emits code that disables destruction via the reference
 counting. This makes the GC solely responsible for destructing and deallocating
 that memory block. You can still assign the pointer to a rc+gc pointer later
on,
 but the reference count is no longer reliable which is why RC-based destruction
 has been disabled.


Yes, you can make it memory safe by introducing another pointer type, as Rust 
does. But see my comments about this scheme in the message you replied to.

(Yes, I understand your proposal is a variation on that scheme.)



 May 11 2014



  parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-05-11 19:37:30 +0000, Walter Bright <newshound2 digitalmars.com> said:


 On 5/11/2014 5:52 AM, Michel Fortin wrote:

 On 2014-05-11 08:29:13 +0000, Walter Bright <newshound2 digitalmars.com> said:
 

 Again, O-C and C++/CX ARC are not memory safe because in order to make it
 perform they provide unsafe escapes from it.


 
 But D could provide memory-safe escapes. If we keep the current GC to collect
 cycles, we could also allow raw pointers managed by the GC alongside ARC.
 
 Let's say we have two kinds of pointers: rc+gc pointers (the default) and
 gc_only pointers (on demand). When assigning from a rc+gc pointer to a gc_only
 pointer, the compiler emits code that disables destruction via the reference
 counting. This makes the GC solely responsible for destructing and deallocating
 that memory block. You can still assign the pointer to a rc+gc pointer 
 later on,
 but the reference count is no longer reliable which is why RC-based destruction
 has been disabled.


 
 Yes, you can make it memory safe by introducing another pointer type, 
 as Rust does. But see my comments about this scheme in the message you 
 replied to.
 
 (Yes, I understand your proposal is a variation on that scheme.)


It is a variation on that scheme, but with one significant difference: 
those two pointer types can mix and there's no restriction on 
assignments from one type to the other. There's therefore no transitive 
effect and no complicated ownership rule to understand.

This obviously does not address all your concerns with ARC (which I'll 
admit most are valid), but this "ARC isn't memory-safe" argument has to 
stop. It does not make sense. One doesn't need to sacrifice memory 
safety to use ARC, neither is that sacrifice necessary for having 
islands of non-ARC code. That's what I was trying to point out in my 
previous post.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 May 11 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 1:43 PM, Michel Fortin wrote:

 It is a variation on that scheme, but with one significant difference: those
two
 pointer types can mix and there's no restriction on assignments from one type
to
 the other. There's therefore no transitive effect and no complicated ownership
 rule to understand.

 This obviously does not address all your concerns with ARC (which I'll admit
 most are valid), but this "ARC isn't memory-safe" argument has to stop. It does
 not make sense. One doesn't need to sacrifice memory safety to use ARC, neither
 is that sacrifice necessary for having islands of non-ARC code. That's what I
 was trying to point out in my previous post.


As long as C++/CX and O-C are brought out here as proven, successful examples 
for D to emulate here, and there is no acknowledgement that they are not 
remotely memory safe, I need to continue to point this out.

Rust is memory safe, but is an unproven design.

Your proposal still relies on a GC to provide the memory safety, and has no 
inherent protection against GC pauses. Your idea has a lot of merit, but it is
a 
hybrid ARC/GC system.



 May 11 2014



 next sibling parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-05-11 21:41:10 +0000, Walter Bright <newshound2 digitalmars.com> said:


 Your proposal still relies on a GC to provide the memory safety, and 
 has no inherent protection against GC pauses. Your idea has a lot of 
 merit, but it is a hybrid ARC/GC system.


If you thread carefully you can disable GC collections at runtime and 
not suffer GC pauses. If you have no cycles and no gc_only pointers, 
then you won't run out of memory.

I think if this thread has proven something, it's that people need to 
be able to choose their memory management policy when the default is 
unsatisfactory. I'm trying to find a way to do that, a way to disable 
one side or the other if it is poisonous to your particular 
application. It is a hybrid system I'm suggesting, no doubt. It'd also 
be an interesting experiment, if someone wants to take it.



 As long as C++/CX and O-C are brought out here as proven, successful 
 examples for D to emulate here, and there is no acknowledgement that 
 they are not remotely memory safe, I need to continue to point this out.


You should not say that ARC is not safe then, you should say instead 
that ARC in those languages has to be supplemented with unsafe code to 
be fast enough. That statement I can agree with. Taking the shortcut 
saying simply "ARC is unsafe" is misleading.


-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 May 11 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 8:44 PM, Michel Fortin wrote:

 As long as C++/CX and O-C are brought out here as proven, successful examples
 for D to emulate here, and there is no acknowledgement that they are not
 remotely memory safe, I need to continue to point this out.


 You should not say that ARC is not safe then, you should say instead that ARC
in
 those languages has to be supplemented with unsafe code to be fast enough.


And I did, probably at least a dozen times in this thread, including quotes you 
replied to specifically: "Again, O-C and C++/CX ARC are not memory safe because 
in order to make it perform they provide unsafe escapes from it."


I don't know how I could be clearer.


In the spirit of being clear, I am opposed to any replacement for the GC that:

1. makes it impossible to meet or exceed C++ performance levels
2. is not memory safe
3. implies shipping multiple builds of Phobos, each using a different memory 
management scheme

I am not opposed to ARC as a supplement, even if it is  system only, in 
particular one that can interact with O-C.



 May 11 2014



prev sibling  parent reply Marco Leise <Marco.Leise gmx.de>  writes:


Am Sun, 11 May 2014 14:41:10 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:


 Your proposal still relies on a GC to provide the memory safety,
 [=E2=80=A6] it is a hybrid ARC/GC system.


But I thought ARC cannot be designed without GC to resolve
cycles. Or is your comment pure rethoric?

--=20
Marco



 May 11 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 9:47 PM, Marco Leise wrote:

 Am Sun, 11 May 2014 14:41:10 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:


 Your proposal still relies on a GC to provide the memory safety,
 […] it is a hybrid ARC/GC system.


 But I thought ARC cannot be designed without GC to resolve
 cycles.


It can be, there are various schemes to deal with that, including "don't create 
cycles". GC is just one of them.

http://en.wikipedia.org/wiki/Reference_counting#Dealing_with_reference_cycles



 May 11 2014



  parent reply Marco Leise <Marco.Leise gmx.de>  writes:


Am Sun, 11 May 2014 22:11:28 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:


 But I thought ARC cannot be designed without GC to resolve
 cycles.


=20
 It can be, there are various schemes to deal with that, including "don't =


create=20

 cycles". GC is just one of them.
=20
 http://en.wikipedia.org/wiki/Reference_counting#Dealing_with_reference_cy=


cles

Yes that article mentions:
a) "avoid creating them"
b) "explicitly forbid reference cycles"
c) "Judicious use of weak references"
d) "manually track that data structure's lifetime"
e) "tracing garbage collector"
f) adding to a root list all objects whose reference
   count is decremented to a non-zero value and periodically
   searching all objects reachable from those roots.

To pick up your statement again: =C2=BBYour proposal still relies
on a GC to provide the memory safety, [=E2=80=A6] it is a hybrid
ARC/GC system.=C2=AB

a) and b) let's assume never creating cycles is not a feasible
          option in a systems programming language
c) and d) don't provide said memory safety
e) and f) ARE tracing garbage collectors

ergo: =C2=BBBut I thought ARC cannot be designed without GC to
resolve cycles.=C2=AB

Your were arguing against Michel Fortin's proposal on the
surface, when your requirement cannot even be fulfilled
theoretically it seems. Which could mean that you don't like
the idea of replacing D's GC with an ARC solution.

=C2=BBThis is the best horse I could find for the price. It is
 pretty fast and ...=C2=AB
=C2=BBNo, it still has four legs.=C2=AB

--=20
Marco



 May 12 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 3:18 AM, Marco Leise wrote:

 Your were arguing against Michel Fortin's proposal on the
 surface, when your requirement cannot even be fulfilled
 theoretically it seems.


Lots of people use ARC without a GC.



 Which could mean that you don't like
 the idea of replacing D's GC with an ARC solution.


I don't like the idea of replacing D's GC with ARC. But for different reasons.



 May 12 2014



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 05/11/2014 10:29 AM, Walter Bright wrote:

 ...

 ------------- A Comment on Rust ------------------

 This is based on my very incomplete knowledge of Rust, i.e. just reading
 a few online documents on it. If I'm wrong, please correct me.
 ...


Well, region-based memory management is not new, and Rust's approach is 
natural, even more so when given some background.

Have you ever read up on type systems? Eg:

http://www.cis.upenn.edu/~bcpierce/tapl/
http://www.cis.upenn.edu/~bcpierce/attapl/


 Rust's designers apparently are well aware of the performance cost of
 pervasive ARC. Hence, they've added the notion of a "borrowed" pointer,
 which is an escape from ARC.


It is not an escape from ARC per se. It is a way to write type safe code 
which is not dependent on the allocation strategy of the processed data. 
(One can e.g. safely borrow mutable data as immutable and the type 
system ensures that during the time of the borrow, the data does not 
mutate.)


 The borrowed pointer is made memory safe by:

 1. Introducing restrictions on what can be done with a borrowed pointer
 so the compiler can determine its lifetime. I do not know the extent of
 these restrictions.
 ...


The type system tracks lifetimes across function and data structure 
boundaries. The main restriction is that such a pointer cannot be 
escaped. (But borrowed pointers may still be stored in data structures.)


 2. Introducing an annotation to distinguish a borrowed pointer from an
 ARC pointer. If you don't use the annotation, you get pervasive ARC with
 all the poor performance that entails.
 ...


No, this is completely inaccurate: Both choices are explicit, and 
reference counting is just one of the possible memory management 
schemes. (Hence borrowed pointers are used unless there is a reason not to.)


 Implicit in borrowed pointers is Rust did not solve the problem of
 having the compiler eliminate unnecessary inc/dec.
 ...


Avoiding inc/dec is not what justifies borrowed pointers.


 My experience with pointer annotations to improve performance is pretty
 compelling - almost nobody adds those annotations. They get their code
 to work with the default, and never get around to annotating it.


The 'default' way to pass by reference is by borrowed pointer.


 ...
 They do not mix. A function taking one type of
 pointer cannot be called with the other type.
 ...


A function taking a borrowed pointer can be called with a reference 
counted pointer. Abstracting over allocation strategy is the point of 
borrowing.


 Worse, these effects are transitive, making a function hierarchy rather
 inflexible.

 Are these valid concerns with Rust?


I don't think they are.



 May 11 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 8:54 AM, Timon Gehr wrote:

 It is not an escape from ARC per se. It is a way to write type safe code which
 is not dependent on the allocation strategy of the processed data. (One can
e.g.
 safely borrow mutable data as immutable and the type system ensures that during
 the time of the borrow, the data does not mutate.)


That's clearly an additional benefit of the borrowed pointer notion. But have 
you examined generated Rust code for the cost of inc/dec? I haven't, but I
don't 
see any way they could avoid this (very expensive) cost without borrowed
pointers.



 The type system tracks lifetimes across function and data structure boundaries.
 The main restriction is that such a pointer cannot be escaped. (But borrowed
 pointers may still be stored in data structures.)


The idea must be that the borrowed pointer lifetime cannot exceed the lifetime 
of the lender.

The thing is, if the compiler is capable of figuring out these lifetimes by 
examining the code, then there would be no need for the programmer to specify a 
pointer as borrowed - the compiler could infer it. Hence, there have got to be 
significant restrictions to the point where the compiler could figure out the
rest.



 2. Introducing an annotation to distinguish a borrowed pointer from an
 ARC pointer. If you don't use the annotation, you get pervasive ARC with
 all the poor performance that entails.
 ...


 No, this is completely inaccurate: Both choices are explicit,


Yes, one is & and the other is  . This does not change my point. Back in the
bad 
old DOS days, there was near* and *. The choices were explicit, but the results 
were there anyway - people overwhelmingly chose the more general *, performance 
be damned.



 and reference
 counting is just one of the possible memory management schemes. (Hence borrowed
 pointers are used unless there is a reason not to.)


Yes, I can see it supports other schemes, but I predict that RC will dominate, 
and other schemes will be about as pervasive as using Boehm's GC with C++.



 Implicit in borrowed pointers is Rust did not solve the problem of
 having the compiler eliminate unnecessary inc/dec.


 Avoiding inc/dec is not what justifies borrowed pointers.


I find that very hard to believe. It implies that there is little cost to 
inc/dec, or the compiler is clever enough to eliminate the bulk of the inc/dec 
pairs.



 My experience with pointer annotations to improve performance is pretty
 compelling - almost nobody adds those annotations. They get their code
 to work with the default, and never get around to annotating it.


 The 'default' way to pass by reference is by borrowed pointer.


Time will tell how well having the most restrictive pointer type be the default 
works out.



 They do not mix. A function taking one type of
 pointer cannot be called with the other type.


 A function taking a borrowed pointer can be called with a reference counted
 pointer. Abstracting over allocation strategy is the point of borrowing.


Right, and a function in DOS taking a * would accept a near*. But the other way 
did not work, and so people wrote their functions using *, performance was 
thrown under the bus.



 Are these valid concerns with Rust?


 I don't think they are.


Perhaps you're right. But my experience with DOS programming is programmers 
preferred convenience and reusable functions, and hence used plain *
everywhere. 
And like I said, this provided a huge competitive advantage for me.

The great boon with 32 bit code was the elimination of special pointer types, 
now it was easy to write fast, reusable functions, and the effort involved in 
creating efficient data structures dropped by probably half.

It seems clear that the decisions of borrow/managed are going to pervade Rust 
code. How well is that going to work with complex programs and data structures? 
How well with average programmers? How well is it going to work when some leaf 
has to change from borrowed to managed, and the ripple effects of that change?

These are all unknowns, not proven successful technology. My experience with a 
similar system (16 bit C) does not make it look promising, as far as
performance 
goes.



 May 11 2014



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 05/11/2014 10:05 PM, Walter Bright wrote:

 On 5/11/2014 8:54 AM, Timon Gehr wrote:

 It is not an escape from ARC per se. It is a way to write type safe
 code which
 is not dependent on the allocation strategy of the processed data.
 (One can e.g.
 safely borrow mutable data as immutable and the type system ensures
 that during
 the time of the borrow, the data does not mutate.)


 That's clearly an additional benefit of the borrowed pointer notion. But
 have you examined generated Rust code for the cost of inc/dec? I
 haven't, but I don't see any way they could avoid this (very expensive)
 cost without borrowed pointers.
 ...


Sure, but performance is the additional benefit.


 The type system tracks lifetimes across function and data structure
 boundaries.
 The main restriction is that such a pointer cannot be escaped. (But
 borrowed
 pointers may still be stored in data structures.)


 The idea must be that the borrowed pointer lifetime cannot exceed the
 lifetime of the lender.

 The thing is, if the compiler is capable of figuring out these lifetimes
 by examining the code,


There are explicit lifetime annotations in function signatures.


 then there would be no need for the programmer to
 specify a pointer as borrowed - the compiler could infer it.


Not modularly. (Yes, global lifetime analysis exists, but this is not 
what they are doing.)


 Hence, there have got to be significant restrictions to the point where the
 compiler could figure out the rest.
 ...


It is simply not true that type systems are inherently restricted to 
checking trivial properties. They can be made as strong as mathematical 
logic without much fuss.


 2. Introducing an annotation to distinguish a borrowed pointer from an
 ARC pointer. If you don't use the annotation, you get pervasive ARC with
 all the poor performance that entails.
 ...


 No, this is completely inaccurate: Both choices are explicit,


 Yes, one is & and the other is  .


No, actually currently one is & and the other is RC<T> AFAIK.


 This does not change my point.


In my opinion it actually does, to the point of rendering it invalid.


 Back in
 the bad old DOS days, there was near* and *. The choices were explicit,


Explicit among near* and *. I.e. between adding 'near' and leaving it 
out. Also, if you add 'near' it will not be compatible with pointers 
that do not have it.


 but the results were there anyway - people overwhelmingly chose the more
 general *, performance be damned.
 ...


RC<T> is not more general. It cannot refer to stack-allocated data, for 
instance.


 and reference
 counting is just one of the possible memory management schemes. (Hence
 borrowed
 pointers are used unless there is a reason not to.)


 Yes, I can see it supports other schemes, but I predict that RC will
 dominate, and other schemes will be about as pervasive as using Boehm's
 GC with C++.
 ...


The point is, borrowing does not depend on the allocation scheme, as 
long as it is safe.


 Implicit in borrowed pointers is Rust did not solve the problem of
 having the compiler eliminate unnecessary inc/dec.


 Avoiding inc/dec is not what justifies borrowed pointers.


 I find that very hard to believe. It implies that there is little cost
 to inc/dec, or the compiler is clever enough to eliminate the bulk of
 the inc/dec pairs.
 ...


Sure, borrowing is very lightweight, but ultimately what is most 
important is that it solves the problem of multiple incompatible pointer 
types and makes the type system more expressive as well.


 My experience with pointer annotations to improve performance is pretty
 compelling - almost nobody adds those annotations. They get their code
 to work with the default, and never get around to annotating it.


 The 'default' way to pass by reference is by borrowed pointer.


 Time will tell how well having the most restrictive pointer type be the
 default works out.
 ...


A function that uses none of the specific pointer capabilities is more 
general, so what other choice of 'default' makes sense?


 They do not mix. A function taking one type of
 pointer cannot be called with the other type.


 A function taking a borrowed pointer can be called with a reference
 counted
 pointer. Abstracting over allocation strategy is the point of borrowing.


 Right, and a function in DOS taking a * would accept a near*. But the
 other way did not work, and so people wrote their functions using *,
 performance was thrown under the bus.



 Are these valid concerns with Rust?


 I don't think they are.


 Perhaps you're right. But my experience with DOS programming is
 programmers preferred convenience and reusable functions, and hence used
 plain * everywhere. And like I said, this provided a huge competitive
 advantage for me.
 ...


Convenience and reusable functions means using borrowed pointers 
whenever possible.


 The great boon with 32 bit code was the elimination of special pointer
 types, now it was easy to write fast, reusable functions, and the effort
 involved in creating efficient data structures dropped by probably half.

 It seems clear that the decisions of borrow/managed are going to pervade
 Rust code.


But they are often obvious.


 How well is that going to work with complex programs and data
 structures? How well with average programmers? How well is it going to
 work when some leaf has to change from borrowed to managed, and the
 ripple effects of that change?

 These are all unknowns, not proven successful technology.


They are using Rust to write a safe and performant web browser while 
developing the language.


 My experience with a similar system (16 bit C) does not make it look
 promising, as far as performance goes.


Borrowed pointers are not even superficially similar to near*. They are 
compatible with everything else, because they can store data that was 
borrowed from anywhere else.



 May 11 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 1:59 PM, Timon Gehr wrote:

 On 05/11/2014 10:05 PM, Walter Bright wrote:

 That's clearly an additional benefit of the borrowed pointer notion. But
 have you examined generated Rust code for the cost of inc/dec? I
 haven't, but I don't see any way they could avoid this (very expensive)
 cost without borrowed pointers.


 Sure, but performance is the additional benefit.


One constant theme in this thread, one I find baffling, is the regular
dismissal 
of the performance implications of inc/dec. Borrowed pointers are not necessary 
to support raw pointers - this can be (and is in some systems) supported by 
simply wrapping the raw pointer with a dummy reference count.

The reason for borrowed pointers is performance. Rust would be non-viable 
without them.

I strongly suggest writing a snippet in [[insert your favorite proven
technology 
RC language here]] and disassembling the result, and have a look at what
inc/dec 
entails.



 The thing is, if the compiler is capable of figuring out these lifetimes
 by examining the code,


 There are explicit lifetime annotations in function signatures.


Yes, because the compiler cannot figure it out itself, so the programmer has to 
annotate.



 It is simply not true that type systems are inherently restricted to checking
 trivial properties. They can be made as strong as mathematical logic without
 much fuss.


Again, Rust would not need borrowed pointers nor the annotations for them if 
this knowledge could be deduced by the compiler. Heck, if the compiler can 
deduce lifetimes accurately, you can get rid of GC and RC, and just have the 
compiler insert malloc/free in the right spots.

Note that there is a Java version that does this partway, sometimes it will 
replace a GC object with a stack allocated one if it is successful in deducing 
that the object lifetime does not exceed the lifetime of the function.



 Yes, one is & and the other is  .


 No, actually currently one is & and the other is RC<T> AFAIK.


Then Rust changed again. The document I read on borrowed pointers was likely
out 
of date, though it had no date on it.



 RC<T> is not more general. It cannot refer to stack-allocated data, for
instance.


So there is no general pointer type that has an unbounded lifetime?



 Sure, borrowing is very lightweight, but ultimately what is most important is
 that it solves the problem of multiple incompatible pointer types and makes the
 type system more expressive as well.


Adding more pointer types makes a type system more expressive, by definition.



 A function that uses none of the specific pointer capabilities is more general,
 so what other choice of 'default' makes sense?


A function that doesn't have restrictions on what can be done with the pointers 
passed to it. Borrowed pointers have restrictions on their usage - this is 
explicitly stated in the Rust documentation.



 Convenience and reusable functions means using borrowed pointers whenever
possible.


Of course. And writing fast code means making your functions fast whenever
possible!



 It seems clear that the decisions of borrow/managed are going to pervade
 Rust code.


 But they are often obvious.


I've written a lot of ref counted code in the past, enough to know that such is 
a pretty optimistic statement. Dealing with was a significant and ongoing drain 
on my time, especially when the programs and data structures got more complex.



 They are using Rust to write a safe and performant web browser while developing
 the language.


Sure. But that browser hasn't been released yet. Consider that I've written
safe 
and performant code in D, but others tell me I am unique and that I cannot 
expect average programmers to get it right.

I repeatedly point out to Manu that he can write performant code in D that does 
not suffer from GC stalls, and he repeatedly replies that he has to work with 
average programmers who are not capable of doing this.

So while I have no doubt that the Mozilla team may be very effective at using 
Rust and making it shine, that may not be transferable to the larger community.



 Borrowed pointers are not even superficially similar to near*. They are
 compatible with everything else, because they can store data that was borrowed
 from anywhere else.


As long as those pointers don't escape. Am I right in that one cannot store a 
borrowed pointer into a global data structure? The similarity is that there are 
one way conversions from one to the other, and one of the types is more
general. 
I infer from your other statements about Rust that it doesn't actually have a 
general pointer type.



 May 11 2014



 next sibling parent reply Marco Leise <Marco.Leise gmx.de>  writes:


Am Sun, 11 May 2014 17:50:25 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:


 As long as those pointers don't escape. Am I right in that one cannot sto=


re a=20

 borrowed pointer into a global data structure?


Right, and that's the point and entirely positive-to-do=E2=84=A2.
Your general purpose function does not know how the memory was
allocated, that it receives a pointer to. In particular it
must not assume that it is safe to keep a reference to it as
there are several memory management schemes that are
incompatible with that, like reference counting or stack
allocations.

Expanding on these two, Rust can now safely use _more_
allocation schemes with functions that take borrowed pointers
than is safely possible in D!

RC pointers:
You cannot pass them as raw pointers in D. In Rust they can
be passed as borrowed.

Stack pointers:
Not allowed in D in  safe code and inherently unsafe in
 system code. Again this is safe to do in Rust due to
borrowing.


 The similarity is that there are=20
 one way conversions from one to the other, and one of the types is more g=


eneral.=20

 I infer from your other statements about Rust that it doesn't actually ha=


ve a=20

 general pointer type.


Yes it does:
http://static.rust-lang.org/doc/0.10/guide-unsafe.html#raw-pointers

But the design principle in Rust is to only have them in
 system code (speaking in D terms), in particular to interface
with C.

Turning the argument back to D and assuming you wrote a
function that takes a raw pointer because you plan to store it
in a global variable. How do you make sure you get a pointer
to something with infinite life-time? Let me answer this: You
either use GC pointers exclusively or you rely on the
convention that the function takes ownership of the memory.
The former is impractical and the latter cannot be statically
enforced.

Borrowed pointers add an  safe way to deal with the situation
in all contexts where you don't need to store a reference.
But if you _do_ need that capability: ask explicitly for GC
pointers as they can guarantee unlimited life-time.
If that's still too restrictive mark it  system and use raw
pointers (in Rust: unsafe keyword).

Finally, this is not Rust vs. D, because D has had borrowed
pointer function arguments since ages as well - maybe even
longer than Rust. The semantics of "in/scope" were just never
fully implemented. Once this is done we can also write:

 safe void main()
{
	auto stack =3D 42;
	foo(&stack);
}

 safe void foo(scope int*);

--=20
Marco



 May 11 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 10:57 PM, Marco Leise wrote:

 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:


 As long as those pointers don't escape. Am I right in that one cannot store a
 borrowed pointer into a global data structure?


 Right, and that's the point and entirely positive-to-do™.


This means that a global data structure in Rust has to decide what memory 
allocation scheme its contents must use, and cannot (without tagging) mix
memory 
allocation schemes.

For example, let's say a compiler has internally a single hash table of
strings. 
With a GC, those strings can be statically allocated, or on the GC heap, or 
anything with a lifetime longer than the table's. But I don't see how this
could 
work in Rust.



 May 12 2014



 next sibling parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Walter Bright:


 But I don't see how this could work in Rust.


Ask it to competent Rust developers/programmers.

Bye,
bearophile



 May 12 2014



prev sibling next sibling parent  Marco Leise <Marco.Leise gmx.de>  writes:


Am Mon, 12 May 2014 01:54:58 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:


 On 5/11/2014 10:57 PM, Marco Leise wrote:

 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:


 As long as those pointers don't escape. Am I right in that one cannot =






store a

 borrowed pointer into a global data structure?


 Right, and that's the point and entirely positive-to-do=E2=84=A2.


=20
 This means that a global data structure in Rust has to decide what memory=


=20

 allocation scheme its contents must use, and cannot (without tagging) mix=


 memory=20

 allocation schemes.
=20
 For example, let's say a compiler has internally a single hash table of s=


trings.=20

 With a GC, those strings can be statically allocated, or on the GC heap, =


or=20

 anything with a lifetime longer than the table's. But I don't see how thi=


s could=20

 work in Rust.


:( Good question. I have no idea.

--=20
Marco



 May 12 2014



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 05/12/2014 10:54 AM, Walter Bright wrote:

 On 5/11/2014 10:57 PM, Marco Leise wrote:

 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:


 As long as those pointers don't escape. Am I right in that one cannot
 store a
 borrowed pointer into a global data structure?


 Right, and that's the point and entirely positive-to-do™.


 This means that a global data structure in Rust has to decide what
 memory allocation scheme its contents must use,


Global variables are banned in Rust code outside of unsafe blocks.


 and cannot (without tagging) mix memory allocation schemes.
 ...


Tagging won't help with all memory allocation schemes.


 For example, let's say a compiler has internally a single hash table of
 strings. With a GC, those strings can be statically allocated, or on the
 GC heap, or anything with a lifetime longer than the table's.
 But I don't see how this could work in Rust.


It's possible if you don't make the table global.
(OTOH in D this is not going to work at all.)



 May 12 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 5:15 AM, Timon Gehr wrote:

 On 05/12/2014 10:54 AM, Walter Bright wrote:

 On 5/11/2014 10:57 PM, Marco Leise wrote:

 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:


 As long as those pointers don't escape. Am I right in that one cannot
 store a
 borrowed pointer into a global data structure?


 Right, and that's the point and entirely positive-to-do™.


 This means that a global data structure in Rust has to decide what
 memory allocation scheme its contents must use,


 Global variables are banned in Rust code outside of unsafe blocks.


Global can also mean assigning through a reference passed as a parameter.



 May 12 2014



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 05/12/2014 06:37 PM, Walter Bright wrote:

 On 5/12/2014 5:15 AM, Timon Gehr wrote:

 On 05/12/2014 10:54 AM, Walter Bright wrote:

 On 5/11/2014 10:57 PM, Marco Leise wrote:

 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:


 As long as those pointers don't escape. Am I right in that one cannot
 store a
 borrowed pointer into a global data structure?


 Right, and that's the point and entirely positive-to-do™.


 This means that a global data structure in Rust has to decide what
 memory allocation scheme its contents must use,


 Global variables are banned in Rust code outside of unsafe blocks.


 Global can also mean assigning through a reference passed as a parameter.


Do you mean the table is not actually global but passed by parameter, or 
that the global table is accessed in unsafe code and then passed by 
parameter or something else?



 May 12 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 12:36 PM, Timon Gehr wrote:

 Do you mean the table is not actually global but passed by parameter,


Yes.

But note that the distinction between the two is often blurry. Under the hood
on 
some systems, global data is accessed via the equivalent of a hidden parameter.



 May 12 2014



prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 12 May 2014 10:50, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 They are using Rust to write a safe and performant web browser while
 developing
 the language.



 Sure. But that browser hasn't been released yet. Consider that I've written
 safe and performant code in D, but others tell me I am unique and that I
 cannot expect average programmers to get it right.

 I repeatedly point out to Manu that he can write performant code in D that
 does not suffer from GC stalls, and he repeatedly replies that he has to
 work with average programmers who are not capable of doing this.


What? You've never offered me a practical solution.

You tell me I have to sacrifice any dependency on libraries
(ridiculous), and all the modern conveniences and safety of automatic
memory management to do it!
Indeed, average programmers are a real-life practical problem, and
it's not just them, I also appreciate the convenience offered
personally. I only have one life, I *really* appreciate saving time on
mundane and otherwise inconsequential tasks.
Tell me some other reasons why I would be attracted to D? Take that
all away, and what's the point? Automating some boilerplate is nice,
but it's not the motivating reason for a wholesale adoption.

You haven't told me how I can use the GC (or whatever memory
management scheme, I really don't care) in the low frequency code
(again, read: almost all code ever), and not have it interfere with
the high frequency code.
This is the fundamental problem with the GC. I can't use it
***ANYWHERE***, including any libraries I link. You can't isolate a
GC, it's effects are not localised. If you could, maybe it'd be more
workable... but even if it were properly concurrent and didn't halt
the realtime threads when it collected, it's still totally impractical
for any background thread to freeze for 10s-100s of ms while it runs a
collect because I received a network packet which needs to be sent
somewhere for processing or whatever.

What do I do?



 So while I have no doubt that the Mozilla team may be very effective at
 using Rust and making it shine, that may not be transferable to the larger
 community.





 May 12 2014



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 12 May 2014 at 07:12:29 UTC, Manu via Digitalmars-d 
wrote:

 You haven't told me how I can use the GC (or whatever memory
 management scheme, I really don't care) in the low frequency 
 code
 (again, read: almost all code ever), and not have it interfere 
 with
 the high frequency code.


You will like Don's talk this year ;)
Game dev code in not just any low level code though.



 May 12 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 12 May 2014 17:24, Dicebot via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 May 2014 at 07:12:29 UTC, Manu via Digitalmars-d wrote:

 You haven't told me how I can use the GC (or whatever memory
 management scheme, I really don't care) in the low frequency code
 (again, read: almost all code ever), and not have it interfere with
 the high frequency code.



 You will like Don's talk this year ;)


I'm super-disappointed I can't make it this year! We were evicted from
our house, have to move, and I can't bail for a week and leave that
all on my mrs while she kicks along the fulltime job :(


 Game dev code in not just any low level code though.


It's not just low-level code. It's also very high level code. Gamedev
is a unique (and extremely interesting) field which combines and
tightly integrates almost every imaginable field of software
engineering. I can't think of any that are left out, or any other
industries that also share this property.
I argue, if you can make a modern high-end game in a language, it can
do anything.



 May 12 2014



  parent  "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Mon, 12 May 2014 03:39:12 -0400, Manu via Digitalmars-d  
<digitalmars-d puremagic.com> wrote:


 On 12 May 2014 17:24, Dicebot via Digitalmars-d



 You will like Don's talk this year ;)


 I'm super-disappointed I can't make it this year!


?!! http://dconf.org/2014/talks/evans.html


 We were evicted from
 our house, have to move, and I can't bail for a week and leave that
 all on my mrs while she kicks along the fulltime job :(


Oh that sucks...

-Steve



 May 12 2014



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:

 What? You've never offered me a practical solution.


I have, you've just rejected them.



 What do I do?


1. you can simply do C++ style memory management. shared_ptr<>, etc.

2. you can have the non-pausible code running in a thread that is not
registered 
with the gc, so the gc won't pause it. This requires that this thread not 
allocate gc memory, but it can use gc memory allocated by other threads, as
long 
as those other threads retain a root to it.

3. D allows you to create and use any memory management scheme you want. You
are 
simply not locked into GC. For example, I rewrote my Empire game into D and it 
did not do any allocation at all - no GC, not even malloc. I know that you'll 
need to do allocation, I'm just pointing out that GC allocations and pauses are 
hardly inevitable.

4. for my part, I have implemented  nogc so you can track down gc usage in
code. 
I have also been working towards refactoring Phobos to eliminate unnecessary GC 
allocations and provide alternatives that do not allocate GC memory. 
Unfortunately, these PR's just sit there.

5. you can divide your app into multiple processes that communicate via 
interprocess communication. One of them pausing will not pause the others. You 
can even do things like turn off the GC collections in those processes, and
when 
they run out of memory just kill them and restart them. (This is not an absurd 
idea, I've heard of people doing that effectively.)

6. If you call C++ libs, they won't be allocating memory with the D GC. D code 
can call C++ code. If you run those C++ libs in separate threads, they won't
get 
paused, either (see (2)).

7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory
with 
malloc/free, and (ironically) only using GC for persistent data structures that 
should never be free'd. Then, I just turned off GC collections, because they'd 
never free anything anyway.

8. you can disable and enable collections, and you can cause collections to be 
run at times when nothing is happening (like when the user has not input 
anything for a while).


The point is, the fact that D has 'new' that allocates GC memory simply does
not 
mean you are obliged to use it. The GC is not going to pause your program if
you 
don't allocate with it. Nor will it ever run a collection at uncontrollable, 
random, asynchronous times.



 May 12 2014



 next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:

 On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:

 What? You've never offered me a practical solution.


 I have, you've just rejected them.



 What do I do?


 1. you can simply do C++ style memory management. shared_ptr<>, 
 etc.

 2. you can have the non-pausible code running in a thread that 
 is not registered with the gc, so the gc won't pause it. This 
 requires that this thread not allocate gc memory, but it can 
 use gc memory allocated by other threads, as long as those 
 other threads retain a root to it.

 3. D allows you to create and use any memory management scheme 
 you want. You are simply not locked into GC. For example, I 
 rewrote my Empire game into D and it did not do any allocation 
 at all - no GC, not even malloc. I know that you'll need to do 
 allocation, I'm just pointing out that GC allocations and 
 pauses are hardly inevitable.

 4. for my part, I have implemented  nogc so you can track down 
 gc usage in code. I have also been working towards refactoring 
 Phobos to eliminate unnecessary GC allocations and provide 
 alternatives that do not allocate GC memory. Unfortunately, 
 these PR's just sit there.

 5. you can divide your app into multiple processes that 
 communicate via interprocess communication. One of them pausing 
 will not pause the others. You can even do things like turn off 
 the GC collections in those processes, and when they run out of 
 memory just kill them and restart them. (This is not an absurd 
 idea, I've heard of people doing that effectively.)

 6. If you call C++ libs, they won't be allocating memory with 
 the D GC. D code can call C++ code. If you run those C++ libs 
 in separate threads, they won't get paused, either (see (2)).

 7. The Warp program I wrote avoids GC pauses by allocating 
 ephemeral memory with malloc/free, and (ironically) only using 
 GC for persistent data structures that should never be free'd. 
 Then, I just turned off GC collections, because they'd never 
 free anything anyway.

 8. you can disable and enable collections, and you can cause 
 collections to be run at times when nothing is happening (like 
 when the user has not input anything for a while).


 The point is, the fact that D has 'new' that allocates GC 
 memory simply does not mean you are obliged to use it. The GC 
 is not going to pause your program if you don't allocate with 
 it. Nor will it ever run a collection at uncontrollable, 
 random, asynchronous times.


The only solutions to the libraries problem that I can see here 
require drastic separation of calls to said libraries from any 
even vaguely time critical code. This is quite restrictive.

Yes, calling badly optimised libraries from a hot loop is a bad 
idea anyway, but the GC changes this from

"well it might take a little more time than usual, but we can 
spare a few nano-seconds and it'll show up easily in the profiler"

to

"it might, sometimes, cause the GC to run a full collection on 
our 3.96 / 4.00 GB heap with an associated half-second pause."

And here we go again, "I can't use that library, it's memory 
management scheme is incompatible with my needs, I'll have to 
rewrite it myself..."



 May 12 2014



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote:

 On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:
 The only solutions to the libraries problem that I can see here 
 require drastic separation of calls to said libraries from any 
 even vaguely time critical code. This is quite restrictive.


I think this is more of library writing culture problem than 
engineering problem. High quality library shouldn't rely on any 
internal allocations at all, deferring this decision to user 
code. Otherwise you will eventually have problems, GC or not.



 May 12 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 2:12 AM, Dicebot wrote:

 I think this is more of library writing culture problem than engineering
 problem. High quality library shouldn't rely on any internal allocations at
all,
 deferring this decision to user code. Otherwise you will eventually have
 problems, GC or not.


Consider my PR:

  https://github.com/D-Programming-Language/phobos/pull/2149

This is exactly what it does - it 'pushes' the decisions about allocating
memory 
up out of the library to the user. I suspect a great deal of storage allocation 
can be removed from Phobos with this technique, without sacrificing
performance, 
flexibility, or memory safety. (In fact, it improves on performance and 
flexibility!)


I also agree with your larger point that if you are relying on an unknown 
library for time critical code, and that library was not designed with time 
criticality guarantees in mind, you're going to have nothing but trouble. 
Regardless of GC or RC.



 May 12 2014



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 12 May 2014 at 17:03:18 UTC, Walter Bright wrote:

 On 5/12/2014 2:12 AM, Dicebot wrote:

 I think this is more of library writing culture problem than 
 engineering
 problem. High quality library shouldn't rely on any internal 
 allocations at all,
 deferring this decision to user code. Otherwise you will 
 eventually have
 problems, GC or not.


 Consider my PR:

  https://github.com/D-Programming-Language/phobos/pull/2149

 This is exactly what it does - it 'pushes' the decisions about 
 allocating memory up out of the library to the user. I suspect 
 a great deal of storage allocation can be removed from Phobos 
 with this technique, without sacrificing performance, 
 flexibility, or memory safety. (In fact, it improves on 
 performance and flexibility!)


We have already had discussion where I did state that current 
 nogc implementation is not robust enough and failed to explain 
the use case for weaker  nogc clearly. Conclusion was that we 
should return to this topic after Don's DConf talk ;)



 May 12 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 10:07 AM, Dicebot wrote:

 We have already had discussion where I did state that current  nogc
 implementation is not robust enough and failed to explain the use case for
 weaker  nogc clearly. Conclusion was that we should return to this topic after
 Don's DConf talk ;)


Sure - next week!



 May 12 2014



prev sibling  parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote:

 On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:

 On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:

 What? You've never offered me a practical solution.


 I have, you've just rejected them.



 What do I do?


 1. you can simply do C++ style memory management. 
 shared_ptr<>, etc.

 2. you can have the non-pausible code running in a thread that 
 is not registered with the gc, so the gc won't pause it. This 
 requires that this thread not allocate gc memory, but it can 
 use gc memory allocated by other threads, as long as those 
 other threads retain a root to it.

 3. D allows you to create and use any memory management scheme 
 you want. You are simply not locked into GC. For example, I 
 rewrote my Empire game into D and it did not do any allocation 
 at all - no GC, not even malloc. I know that you'll need to do 
 allocation, I'm just pointing out that GC allocations and 
 pauses are hardly inevitable.

 4. for my part, I have implemented  nogc so you can track down 
 gc usage in code. I have also been working towards refactoring 
 Phobos to eliminate unnecessary GC allocations and provide 
 alternatives that do not allocate GC memory. Unfortunately, 
 these PR's just sit there.

 5. you can divide your app into multiple processes that 
 communicate via interprocess communication. One of them 
 pausing will not pause the others. You can even do things like 
 turn off the GC collections in those processes, and when they 
 run out of memory just kill them and restart them. (This is 
 not an absurd idea, I've heard of people doing that 
 effectively.)

 6. If you call C++ libs, they won't be allocating memory with 
 the D GC. D code can call C++ code. If you run those C++ libs 
 in separate threads, they won't get paused, either (see (2)).

 7. The Warp program I wrote avoids GC pauses by allocating 
 ephemeral memory with malloc/free, and (ironically) only using 
 GC for persistent data structures that should never be free'd. 
 Then, I just turned off GC collections, because they'd never 
 free anything anyway.

 8. you can disable and enable collections, and you can cause 
 collections to be run at times when nothing is happening (like 
 when the user has not input anything for a while).


 The point is, the fact that D has 'new' that allocates GC 
 memory simply does not mean you are obliged to use it. The GC 
 is not going to pause your program if you don't allocate with 
 it. Nor will it ever run a collection at uncontrollable, 
 random, asynchronous times.


 The only solutions to the libraries problem that I can see here 
 require drastic separation of calls to said libraries from any 
 even vaguely time critical code. This is quite restrictive.

 Yes, calling badly optimised libraries from a hot loop is a bad 
 idea anyway, but the GC changes this from

 "well it might take a little more time than usual, but we can 
 spare a few nano-seconds and it'll show up easily in the 
 profiler"

 to

 "it might, sometimes, cause the GC to run a full collection on 
 our 3.96 / 4.00 GB heap with an associated half-second pause."

 And here we go again, "I can't use that library, it's memory 
 management scheme is incompatible with my needs, I'll have to 
 rewrite it myself..."


A badly placed malloc() in library code can also trigger OS
virtualization mechanisms and make processes being swapped out to
disk, with the respective overhead in disk access and time spent
on kernel code.

So it is just not the "we can spare a few nano-seconds".

--
Paulo



 May 12 2014



  parent reply Marco Leise <Marco.Leise gmx.de>  writes:


Am Mon, 12 May 2014 09:32:58 +0000
schrieb "Paulo Pinto" <pjmlp progtools.org>:


 On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote:

 On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:

 On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:

 What? You've never offered me a practical solution.


 I have, you've just rejected them.



 What do I do?


 1. you can simply do C++ style memory management. 
 shared_ptr<>, etc.

 2. you can have the non-pausible code running in a thread that 
 is not registered with the gc, so the gc won't pause it. This 
 requires that this thread not allocate gc memory, but it can 
 use gc memory allocated by other threads, as long as those 
 other threads retain a root to it.

 3. D allows you to create and use any memory management scheme 
 you want. You are simply not locked into GC. For example, I 
 rewrote my Empire game into D and it did not do any allocation 
 at all - no GC, not even malloc. I know that you'll need to do 
 allocation, I'm just pointing out that GC allocations and 
 pauses are hardly inevitable.

 4. for my part, I have implemented  nogc so you can track down 
 gc usage in code. I have also been working towards refactoring 
 Phobos to eliminate unnecessary GC allocations and provide 
 alternatives that do not allocate GC memory. Unfortunately, 
 these PR's just sit there.

 5. you can divide your app into multiple processes that 
 communicate via interprocess communication. One of them 
 pausing will not pause the others. You can even do things like 
 turn off the GC collections in those processes, and when they 
 run out of memory just kill them and restart them. (This is 
 not an absurd idea, I've heard of people doing that 
 effectively.)

 6. If you call C++ libs, they won't be allocating memory with 
 the D GC. D code can call C++ code. If you run those C++ libs 
 in separate threads, they won't get paused, either (see (2)).

 7. The Warp program I wrote avoids GC pauses by allocating 
 ephemeral memory with malloc/free, and (ironically) only using 
 GC for persistent data structures that should never be free'd. 
 Then, I just turned off GC collections, because they'd never 
 free anything anyway.

 8. you can disable and enable collections, and you can cause 
 collections to be run at times when nothing is happening (like 
 when the user has not input anything for a while).


 The point is, the fact that D has 'new' that allocates GC 
 memory simply does not mean you are obliged to use it. The GC 
 is not going to pause your program if you don't allocate with 
 it. Nor will it ever run a collection at uncontrollable, 
 random, asynchronous times.


 The only solutions to the libraries problem that I can see here 
 require drastic separation of calls to said libraries from any 
 even vaguely time critical code. This is quite restrictive.

 Yes, calling badly optimised libraries from a hot loop is a bad 
 idea anyway, but the GC changes this from

 "well it might take a little more time than usual, but we can 
 spare a few nano-seconds and it'll show up easily in the 
 profiler"

 to

 "it might, sometimes, cause the GC to run a full collection on 
 our 3.96 / 4.00 GB heap with an associated half-second pause."

 And here we go again, "I can't use that library, it's memory 
 management scheme is incompatible with my needs, I'll have to 
 rewrite it myself..."


 
 A badly placed malloc() in library code can also trigger OS
 virtualization mechanisms and make processes being swapped out to
 disk, with the respective overhead in disk access and time spent
 on kernel code.
 
 So it is just not the "we can spare a few nano-seconds".
 
 --
 Paulo


Yes, it could easily extend to a longer wait. I think we all
know programs that hang while the system is swapping out.
Don't let it get to that! A PC game would typically reduce
caches or texture resolutions before running out of RAM.

Linux has a threshold of free pages it tries to keep available
at any time to satisfy occasional small allocations.
http://www.science.unitn.it/~fiorella/guidelinux/tlk/node39.html

All-in-all malloc is less likely to cause long pauses. It just
allocates and doesn't ask itself if there might be dead memory
to salvage to satisfy a request.

Time will tell if all "well written" D libraries will be  nogc
to move the question of allocations to the user.

-- 
Marco



 May 12 2014



  parent  "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Monday, 12 May 2014 at 10:51:33 UTC, Marco Leise wrote:


 Time will tell if all "well written" D libraries will be  nogc
 to move the question of allocations to the user.


If there was such a thing as "weakly &nogc" then we would could 
do this

//some library function
void foo(OR, IR)(OR o, IR i)  weak-nogc
{
     //take things from i and put them in o
}

allocations would be possible during the execution of foo, but 
*only* in the implementations of OR and IR, which means that the 
developer gets the control and guarantees they need, but doesn't 
have to explicitly pre-allocate (which might not even be 
possible).

I don't see how it would work with UFCS though...



 May 12 2014



prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 12 May 2014 18:45, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:

 What? You've never offered me a practical solution.



 I have, you've just rejected them.



 What do I do?



 1. you can simply do C++ style memory management. shared_ptr<>, etc.


I already have C++. I don't want another one.


 2. you can have the non-pausible code running in a thread that is not
 registered with the gc, so the gc won't pause it. This requires that this
 thread not allocate gc memory, but it can use gc memory allocated by other
 threads, as long as those other threads retain a root to it.


It still sounds the same as manual memory management though in
practise, like you say, the other thread must maintain a root to it,
which means I need to manually retain it somehow, and when the worker
thread finishes with it, it needs to send a signal or something back
to say it's done so it can be released... it sounds more inconvenient
than direct manual memory management in practise.
Sounds slow too. Dec-ing a ref is certainly faster than inter-thread
communication.

This also makes library calls into effective RPC's if I can't call
into them from the active threads.

How long is a collect liable to take in the event the GC threads need
to collect? Am I likely to lose my service threads for 100s of
milliseconds at a time?

I'll think on it, but I don't think there's anything practically
applicable here, and it really sounds like it creates a lot more
trouble and complexity than it addresses.


 3. D allows you to create and use any memory management scheme you want. You
 are simply not locked into GC. For example, I rewrote my Empire game into D
 and it did not do any allocation at all - no GC, not even malloc. I know
 that you'll need to do allocation, I'm just pointing out that GC allocations
 and pauses are hardly inevitable.


C++ lets me create any memory management scheme I like by the same argument.
I lose all the parts of the language that implicitly depend on the GC,
and 3rd party libs (that don't care about me and my project).
Why isn't it a reasonable argument to say that not having access to
libraries is completely unrealistic? You can't write modern software
without extensive access to libraries. Period.

I've said before, I don't want to be a second class citizen with
access to only a subset of the language.


 4. for my part, I have implemented  nogc so you can track down gc usage in
 code. I have also been working towards refactoring Phobos to eliminate
 unnecessary GC allocations and provide alternatives that do not allocate GC
 memory. Unfortunately, these PR's just sit there.


The effort is appreciated, but it was never a solution. I said  nogc
was the exact wrong approach to my situation right from the start, and
I predicted that would be used as an argument the moment it appeared.
Tracking down GC usage isn't helpful when it leads you to a lib call
that you can't change. And again, eliminating useful and productive
parts of the language is not a goal we should be shooting for.

I'll find it useful in the high-performance realtime bits; ie, the
bits that I typically disassemble and scrutinise after every compile.
But that's not what we're discussing here.
I'm happy with D for my realtime code, I have the low-level tools I
need to make the real-time code run fast.  nogc is a little bonus that
will allow to guarantee no sneaky allocations are finding their way
into the fast code, and that might save a little time, but I never
really saw that as a significant problem in the first place.

What we're talking about is productivity, convenience and safety in
the non-realtime code. The vast majority of code, that programmers
spend most of their days working on.


Consider it this way... why do you have all these features in D that
cause implicit allocation if you don't feel they're useful and
important parts of the language?
Assuming you do feel they're important parts of the language, why do
you feel it's okay to tell me I don't deserve access to them?
Surely I'm *exactly* the target market for D...? High-pressure,
intensive production environments, still depending exclusively on
native code, with code teams often in the realm of 50-100, containing
many juniors, aggressive schedules which can't afford to waste
engineering hours... this is a code environment that's prone to MANY
bugs, and countless wasted hours as a consequence.
Convenience and safety are important to me... I don't know what you
think I'm interested in D for if you think I should be happy to
abandon a whole chunk of the language, just because I have a couple of
realtime threads :/



 5. you can divide your app into multiple processes that communicate via
 interprocess communication. One of them pausing will not pause the others.
 You can even do things like turn off the GC collections in those processes,
 and when they run out of memory just kill them and restart them. (This is
 not an absurd idea, I've heard of people doing that effectively.)


Most of the platforms I work on barely have operating systems.


 6. If you call C++ libs, they won't be allocating memory with the D GC. D
 code can call C++ code. If you run those C++ libs in separate threads, they
 won't get paused, either (see (2)).


Whether this is practical or not thoroughly depends on the lib.
Maybe this concept can be applicable in some small places, but it's
not a salvation. I don't think this sufficient addresses the problems.
None of the problems are actually going away, they're just moved
somewhere else


 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory
 with malloc/free, and (ironically) only using GC for persistent data
 structures that should never be free'd. Then, I just turned off GC
 collections, because they'd never free anything anyway.


That idea is obviously not applicable in my environment. Resource
usage is dynamic and fluid.


 8. you can disable and enable collections, and you can cause collections to
 be run at times when nothing is happening (like when the user has not input
 anything for a while).


If I disable collections, then I just crash when I receive that network packet?
I'm back at manual memory management in practise.

I also don't think it's reasonable to assume there will just be 'times
when nothing is happening'. That's not how games work.
Games are often really fast paced, and even if they're not,
significant stuttering in the animation is usually considered a
non-ship-able bug.

https://www.youtube.com/watch?v=rqjOXR9QnMo
https://www.youtube.com/watch?v=giiZMktZrNI
https://www.youtube.com/watch?v=LoPC_ibBJiQ
Where would you manually issue collects?


 The point is, the fact that D has 'new' that allocates GC memory simply does
 not mean you are obliged to use it.


D also has ~, closures, dynamic arrays, even array literals. There are
various things that create implicit GC allocations. And library
calls...


 The GC is not going to pause your
 program if you don't allocate with it. Nor will it ever run a collection at
 uncontrollable, random, asynchronous times.


Those claims come with massive dependency on very specific
restrictions, like abandoning part of the language and moving library
calls to separate threads and accessing them via RPC or something like
that.


None of your suggestions sound practical, or like they'd result in any
less effort or complexity than manual management in the first place
which everyone is already accustomed to. I'm almost certainly
sacrificing safety in every case.
You can't then go on to say you gave me plenty of options, but I
rejected them, when none of them were really options.

I wonder if you have a good conception of the scope/scale of the
software we write. It's not comparable to Empire, or a linker, or a
compiler, or a web server, or many things at all really. Games are
some of the biggest, broadest software projects there are, very
tightly integrated, with some of the most stringent operating
requirements. They're also growing steadily... it's harder and harder
to manage the scope without helpful language tools; this is why you
see so many gamedevs in the independent space flirting with 'modern'

push the platform (indy/casual games), this is sometimes okay, but
there are plenty of cases where it has been a complete disaster as the
scope has grown towards a more traditional 'big game'. My mates game I
helped them with from last weekend is 'mid-scoped', but it's grown to
saturate the PS4, and the GC is causing them a nightmare... right at
the end of the project when trying to finalise the build for shipping,
precisely as I've always predicted.

to do a lot more work in a lot less time with a lot fewer people. But
it's clearly not really compatible with the workload, and I think the
future of the industry needs to do a lot better.

I've said before, we are an industry in desperate need of salvation,
it's LONG overdue, and I want something that actually works well for
us, not a crappy set of compromises because the language has a
fundamental incompatibility with my industry :/ ... It doesn't have to
be that way.



 May 12 2014



 next sibling parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Manu:


 we are an industry in desperate need of salvation,
 it's LONG overdue, and I want something that actually works 
 well for us, not a crappy set of compromises because the
 language has a fundamental incompatibility with my industry :/


Perhaps the game industry has to start the creation of a language 
designed for its needs, like the scientific people have done 
(Julia), the browser ones (Rust), the Web ones have done, etc. 
With lot of work in less than ten years you can have an usable 
language.

Bye,
bearophile



 May 12 2014



 next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 13 May 2014 02:16, bearophile via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Manu:



 we are an industry in desperate need of salvation,
 it's LONG overdue, and I want something that actually works well for us,
 not a crappy set of compromises because the
 language has a fundamental incompatibility with my industry :/



 Perhaps the game industry has to start the creation of a language designed
 for its needs, like the scientific people have done (Julia), the browser
 ones (Rust), the Web ones have done, etc. With lot of work in less than ten
 years you can have an usable language.


But D is *so close*... and I like it! >_<

I have to say that this discussion has certainly left me somewhat
intrigued by Rust though.
I've never given it a fair go because I find the syntax so distasteful
and deterring.
I wonder if there's a market for a rust fork that re-skin's the language ;)



 May 12 2014



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d 
wrote:

 But D is *so close*... and I like it! >_<

 I have to say that this discussion has certainly left me 
 somewhat
 intrigued by Rust though.
 I've never given it a fair go because I find the syntax so 
 distasteful
 and deterring.
 I wonder if there's a market for a rust fork that re-skin's the 
 language ;)


Right now D has practical benefit of being more stable and 
library rich. But switching to Rust eventually does seem tempting 
as I find foundations of their type system much closer to my 
beliefs about "good coding practices".

It lacks any good static reflection though. And this stuff is 
damn addictive when you try it of D caliber.



 May 12 2014



 next sibling parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 13 May 2014 03:14, Dicebot via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d wrote:

 But D is *so close*... and I like it! >_<

 I have to say that this discussion has certainly left me somewhat
 intrigued by Rust though.
 I've never given it a fair go because I find the syntax so distasteful
 and deterring.
 I wonder if there's a market for a rust fork that re-skin's the language
 ;)



 Right now D has practical benefit of being more stable and library rich. But
 switching to Rust eventually does seem tempting as I find foundations of
 their type system much closer to my beliefs about "good coding practices".

 It lacks any good static reflection though. And this stuff is damn addictive
 when you try it of D caliber.


They have a lot more work to do.
There doesn't seem to be a useful windows compiler for a start... >_<



 May 12 2014



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2014-05-12 19:14, Dicebot wrote:


 It lacks any good static reflection though. And this stuff is damn
 addictive when you try it of D caliber.


It has macros, that basically requires great support for static 
reflection to be usable.

-- 
/Jacob Carlborg



 May 12 2014



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 12 May 2014 at 19:32:49 UTC, Jacob Carlborg wrote:

 On 2014-05-12 19:14, Dicebot wrote:


 It lacks any good static reflection though. And this stuff is 
 damn
 addictive when you try it of D caliber.


 It has macros, that basically requires great support for static 
 reflection to be usable.


Judging by 
http://static.rust-lang.org/doc/0.6/tutorial-macros.html those 
are not full-blown AST macros like ones you have been proposing, 
more like hygienic version of C macros.



 May 13 2014



 next sibling parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 05/13/2014 03:56 PM, Dicebot wrote:

 On Monday, 12 May 2014 at 19:32:49 UTC, Jacob Carlborg wrote:

 On 2014-05-12 19:14, Dicebot wrote:


 It lacks any good static reflection though. And this stuff is damn
 addictive when you try it of D caliber.


 It has macros, that basically requires great support for static
 reflection to be usable.


 Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been proposing,
 more like hygienic version of C macros.


https://github.com/huonw/brainfuck_macro/blob/master/lib.rs



 May 13 2014



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2014-05-13 15:56, Dicebot wrote:


 Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been proposing,
 more like hygienic version of C macros.


Hmm, I haven't looked at Rust macros that much.

-- 
/Jacob Carlborg



 May 13 2014



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 05/13/2014 09:07 PM, Jacob Carlborg wrote:

 On 2014-05-13 15:56, Dicebot wrote:


 Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been proposing,
 more like hygienic version of C macros.


 Hmm, I haven't looked at Rust macros that much.


Again, the following is an example of Rust macros in action. A bf 
program is compiled to Rust code at compile time.

https://github.com/huonw/brainfuck_macro/blob/master/lib.rs

Compile-time computations create an AST which is then spliced. Seems 
full-blown enough to me and not at all like C macros.



 May 13 2014



 next sibling parent  "Tommi" <tommitissari hotmail.com>  writes:


On Tuesday, 13 May 2014 at 21:16:55 UTC, Timon Gehr wrote:

 On 05/13/2014 09:07 PM, Jacob Carlborg wrote:

 On 2014-05-13 15:56, Dicebot wrote:


 Judging by 
 http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been 
 proposing,
 more like hygienic version of C macros.


 Hmm, I haven't looked at Rust macros that much.


 Again, the following is an example of Rust macros in action. A 
 bf program is compiled to Rust code at compile time.

 https://github.com/huonw/brainfuck_macro/blob/master/lib.rs

 Compile-time computations create an AST which is then spliced. 
 Seems full-blown enough to me and not at all like C macros.


I really don't know, but understanding is that there are two 
types of macros in Rust. There's the simple "hygienic macro" one, 
which has some documentation: 
http://static.rust-lang.org/doc/0.10/guide-macros.html. Then 
there's the other, newer kind of macros (they may be called 
"procedural macros"... or not) and those are basically 
undocumented at this point. My understanding is that the newer 
kind of macros can do literally anything, like order you a pizza 
at compile-time.



 May 13 2014



prev sibling  parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 13 May 2014 at 21:16:55 UTC, Timon Gehr wrote:

 On 05/13/2014 09:07 PM, Jacob Carlborg wrote:

 On 2014-05-13 15:56, Dicebot wrote:


 Judging by 
 http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been 
 proposing,
 more like hygienic version of C macros.


 Hmm, I haven't looked at Rust macros that much.


 Again, the following is an example of Rust macros in action. A 
 bf program is compiled to Rust code at compile time.

 https://github.com/huonw/brainfuck_macro/blob/master/lib.rs

 Compile-time computations create an AST which is then spliced. 
 Seems full-blown enough to me and not at all like C macros.


Comments look promising but unfortunately I can't understand a 
single line of actual code :) Mine rust-fu is too weak.



 May 14 2014



  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


14-May-2014 17:16, Dicebot пишет:

 On Tuesday, 13 May 2014 at 21:16:55 UTC, Timon Gehr wrote:

 On 05/13/2014 09:07 PM, Jacob Carlborg wrote:

 On 2014-05-13 15:56, Dicebot wrote:


 Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been proposing,
 more like hygienic version of C macros.


 Hmm, I haven't looked at Rust macros that much.


 Again, the following is an example of Rust macros in action. A bf
 program is compiled to Rust code at compile time.

 https://github.com/huonw/brainfuck_macro/blob/master/lib.rs

 Compile-time computations create an AST which is then spliced. Seems
 full-blown enough to me and not at all like C macros.


 Comments look promising but unfortunately I can't understand a single
 line of actual code :) Mine rust-fu is too weak.


This is curious:
http://burntsushi.net/rustdoc/regex/

-- 
Dmitry Olshansky



 May 14 2014



  parent reply Jacob Carlborg <doob me.com>  writes:


On 14/05/14 23:47, Dmitry Olshansky wrote:


 This is curious:
 http://burntsushi.net/rustdoc/regex/


It seems they have compile time regular expressions too.

-- 
/Jacob Carlborg



 May 14 2014



  parent  "Dicebot" <public dicebot.lv>  writes:


On Thursday, 15 May 2014 at 06:30:40 UTC, Jacob Carlborg wrote:

 On 14/05/14 23:47, Dmitry Olshansky wrote:


 This is curious:
 http://burntsushi.net/rustdoc/regex/


 It seems they have compile time regular expressions too.


Yeah looks like my knowledge about their macro system is very 
outdated. Looks promising.



 May 15 2014



prev sibling  parent reply Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 12/05/2014 19:14, Dicebot a écrit :

 On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d wrote:

 But D is *so close*... and I like it! >_<

 I have to say that this discussion has certainly left me somewhat
 intrigued by Rust though.
 I've never given it a fair go because I find the syntax so distasteful
 and deterring.
 I wonder if there's a market for a rust fork that re-skin's the
 language ;)


 Right now D has practical benefit of being more stable and library rich.
 But switching to Rust eventually does seem tempting as I find
 foundations of their type system much closer to my beliefs about "good
 coding practices".

 It lacks any good static reflection though. And this stuff is damn
 addictive when you try it of D caliber.


All compile time things of D are marvelous.
This with the compile time and the language less error prone make me want D.
I am not sure I need safety so much. It's nice but not mandatory for any 
of my projects. The only one which has to be safe is DQuick.



 May 12 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 2:28 PM, Xavier Bigand wrote:

 All compile time things of D are marvelous.
 This with the compile time and the language less error prone make me want D.
 I am not sure I need safety so much. It's nice but not mandatory for any of my
 projects. The only one which has to be safe is DQuick.


Safety becomes a big concern when you're developing code as part of a team.



 May 12 2014



prev sibling  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Monday, 12 May 2014 at 16:16:06 UTC, bearophile wrote:

 Perhaps the game industry has to start the creation of a 
 language designed for its needs, like the scientific people 
 have done (Julia), the browser ones (Rust), the Web ones have 
 done, etc. With lot of work in less than ten years you can have 
 an usable language.


I don't think games are unique or special. Most games are even in 
the "easy" space by having mostly static data. Meaning the amount 
of unexpected dynamic data is pretty low.  Games also have the 
luxury of redefining the requirements spec to match available 
technology. The games industry does however have its own culture 
and paradigms and fashions… With subcultures.

However, most interactive applications will suffer from the same 
issues if you increase the load so that they run out of headroom. 
Even unix commands like find and grep have latency requirements 
if the interaction is to be pleasant. By good fortune "find" and 
"grep" haven't changed their interface for 40+ years, so they 
were designed for low performance CPUs. That does not mean that 
you cannot design a better "find"-like application today that 
will run into runtime related usability issues if you freeze the 
world.

At the end of the day, a system level language should support key 
strategies used for writing performant system level code in a 
reliable manner. It should also not lock you to a specific 
runtime that you couldn't easily write yourself. It should also 
not lock you to a specific model of how to structure your code 
(like monitors). I am not even sure it should provide OS 
abstractions, because that is not really system level 
programming. That is unixy (Posix) programming. A system level 
programming language should be free of OS and modelling related 
legacy.



 May 12 2014



prev sibling  parent reply "Kapps" <opantm2+spam gmail.com>  writes:


On Monday, 12 May 2014 at 16:03:28 UTC, Manu via Digitalmars-d
wrote:

 How long is a collect liable to take in the event the GC 
 threads need
 to collect? Am I likely to lose my service threads for 100s of
 milliseconds at a time?

 I'll think on it, but I don't think there's anything practically
 applicable here, and it really sounds like it creates a lot more
 trouble and complexity than it addresses.



Your concerns stem not as much from the speed concern of the GC,
but from the freeze-the-world aspect of it. Would a concurrent
collector not solve these issues? As
http://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx#concurrent_garbage_collection
explains a little bit, the actual time your threads spend frozen
should be little (but I admit I don't know exactly how little),
and so long as you don't allocate too much during the collection
itself (which you say you don't), you should be able to keep
running your code during the collection. If it's not possible to
implement concurrent collection in D (and it's already been shown
it is possible), then I'd agree that ARC is very important. But
depending on how little the stop-the-world time from a concurrent
GC can get, perhaps this could work around some issues that
you're desiring ARC for. A generational collector could help in
theory with your high memory usage situations. I doubt you
allocate a gigabyte each frame, so the actual generation 0
content should be fairly low. Much of your memory usage should be
allocations that will not be freed for long periods of time,
while the per-frame and other short allocations should be fast to
collect as there aren't many of them.

Depending on how tunable the GC is, I feel like it should be
possible to get away with a GC even for soft real-time programs
like games. The problem is it's hard to tell until we get a
proper concurrent collector in D2, just like it's hard to tell
how significant the impact of ARC is until we get an optimized
implementation of it in the compiler. Neither of these is simple.
I do quite like the idea of ARC, it's just something that someone
would have to actually implement (well) in order to see how much
of an impact it really has in D. For the truly low frequency
situations, you could get away with a library type for ARC as
well, and as you mentioned, for high frequency you would get
around ARC regardless.



 May 12 2014



 next sibling parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 13 May 2014 04:07, Kapps via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 May 2014 at 16:03:28 UTC, Manu via Digitalmars-d
 wrote:

 How long is a collect liable to take in the event the GC threads need

 to collect? Am I likely to lose my service threads for 100s of
 milliseconds at a time?

 I'll think on it, but I don't think there's anything practically
 applicable here, and it really sounds like it creates a lot more
 trouble and complexity than it addresses.




 Your concerns stem not as much from the speed concern of the GC,
 but from the freeze-the-world aspect of it. Would a concurrent
 collector not solve these issues?


I originally thought it would... but the more I think on it, I don't
think it would make an awful lot of difference in practise.
If the stalls were 'short' (like 1-5ms on the background threads,
500µs-1ms on the realtime threads), then maybe it would be workable,
but I don't know that it would be even close to that?

Also, I think it would be very difficult to implement on a machine
without virtual memory, or much of an operating system in general?

The problem remains that with no free memory, frequency of collection
becomes so high, that it's extremely unlikely full collection so often
would be better than ARC.


 As http://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx#concurrent_garbage_collection
 explains a little bit, the actual time your threads spend frozen
 should be little (but I admit I don't know exactly how little),
 and so long as you don't allocate too much during the collection
 itself (which you say you don't), you should be able to keep
 running your code during the collection. If it's not possible to
 implement concurrent collection in D (and it's already been shown
 it is possible), then I'd agree that ARC is very important. But
 depending on how little the stop-the-world time from a concurrent
 GC can get, perhaps this could work around some issues that
 you're desiring ARC for. A generational collector could help in
 theory with your high memory usage situations. I doubt you
 allocate a gigabyte each frame, so the actual generation 0
 content should be fairly low. Much of your memory usage should be
 allocations that will not be freed for long periods of time,
 while the per-frame and other short allocations should be fast to
 collect as there aren't many of them.


Yeah, it would probably be better, if it's possible.
Implementation needs to be considered from the perspective of embedded
systems with no OS or MMU, and as little as 64mb of ram (the smallest
modern systems).
Mid-range systems are 512mb and no MMU. 'next-gen' systems are
basically like little PC's with crappy OS's, so more likely a decent
GC is possible on a ps4/xbone... but very few have the luxury of
developing for just one system.

It occurred to me earlier that things like strings might enjoy their
own separate heap. And maybe some special logic for strings that
outlived their scope to be actively returned to their heap rather than
waiting for collection.
If the heap were successfully broken down into a suite of sub-heaps, I
have absolutely no idea how to make estimates about the performance of
this system, and if it would approach an acceptable level. I'm
skeptical it would, and it still won't decrease collection frequency.
But I'd be happy to be surprised.


 Depending on how tunable the GC is, I feel like it should be
 possible to get away with a GC even for soft real-time programs
 like games. The problem is it's hard to tell until we get a
 proper concurrent collector in D2, just like it's hard to tell
 how significant the impact of ARC is until we get an optimized
 implementation of it in the compiler. Neither of these is simple.
 I do quite like the idea of ARC, it's just something that someone
 would have to actually implement (well) in order to see how much
 of an impact it really has in D.


I understand the problem. The first hurdle is overcoming the hostility
against it though. There is a severe prejudice.


 For the truly low frequency
 situations, you could get away with a library type for ARC as
 well, and as you mentioned, for high frequency you would get
 around ARC regardless.


Yup.



 May 12 2014



prev sibling  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Monday, 12 May 2014 at 18:07:51 UTC, Kapps wrote:

 Depending on how tunable the GC is, I feel like it should be
 possible to get away with a GC even for soft real-time programs
 like games.


Even if you manage to make it work for game clients you also 
should make it work for low latency game servers, as code sharing 
is an important advantage.

What a game/world server requires differs a lot, but highly 
dynamic and flexible worlds have to keep the physics to a single 
node (or tight cluster) for a region. That means you want to have 
as many players as possible tied to that node.

In essence you want both performance, low latency, reliability, 
and little overhead in an evolutionary context (it has to support 
heavy modification over time).

My gut feeling is that a runtime satisfying one game design will 
not satisfy another one as long as one insists on one global GC. 
In essence, it will never really work well. IMO, the same goes 
for ARC since RC does not perform well with multi-threading even 
when you use near optimal patterns and strategies. If ARC is only 
to be used where speed does not matter then you might as well use 
shared_ptr.



 May 12 2014



  parent reply "Kapps" <opantm2+spam gmail.com>  writes:


On Monday, 12 May 2014 at 19:13:50 UTC, Ola Fosheim Grøstad wrote:

 On Monday, 12 May 2014 at 18:07:51 UTC, Kapps wrote:

 Depending on how tunable the GC is, I feel like it should be
 possible to get away with a GC even for soft real-time programs
 like games.


 Even if you manage to make it work for game clients you also 
 should make it work for low latency game servers, as code 
 sharing is an important advantage.

 What a game/world server requires differs a lot, but highly 
 dynamic and flexible worlds have to keep the physics to a 
 single node (or tight cluster) for a region. That means you 
 want to have as many players as possible tied to that node.

 In essence you want both performance, low latency, reliability, 
 and little overhead in an evolutionary context (it has to 
 support heavy modification over time).

 My gut feeling is that a runtime satisfying one game design 
 will not satisfy another one as long as one insists on one 
 global GC. In essence, it will never really work well. IMO, the 
 same goes for ARC since RC does not perform well with 
 multi-threading even when you use near optimal patterns and 
 strategies. If ARC is only to be used where speed does not 
 matter then you might as well use shared_ptr.


.NET allows configuring the garbage collector by specifying
workstation (concurrent, background [allow generation 0/1
collection while a generation 2 collection is going], one primary
heap and a large object heap) or server (not certain if
concurrent/background, but multiple heaps that get handled in
parallel during collections). Or in situations where you have
many processes running at once, disabling concurrent collection
to reduce context switching overhead. In reality, most people
leave the default concurrent collector, which is what I'd hope
the default for D would be, but if it was sufficiently tunable
something like vibe.d could decide to go with something more
similar to what .NET uses for servers (which ASP.NET uses by
default).

I haven't been able to find good concrete numbers online, but the
few sources I've found say that generation 0/1 collection tends
to take <1 to 2-3 milliseconds and is not run concurrently
because it's so short. This is quite sufficient for most
projects, but perhaps could be tweaked a bit more for certain
aspects like gaming, possibly even enabling concurrent collection
for generation 0/1, but I'm not sure if this works well or is
feasible. Still, the important thing is to get a good general one
to use first, like the default one .NET uses for workstation
applications.



 May 12 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Monday, 12 May 2014 at 22:27:06 UTC, Kapps wrote:

 because it's so short. This is quite sufficient for most
 projects, but perhaps could be tweaked a bit more for certain
 aspects like gaming, possibly even enabling concurrent 
 collection
 for generation 0/1, but I'm not sure if this works well or is
 feasible. Still, the important thing is to get a good general 
 one
 to use first, like the default one .NET uses for workstation
 applications.


I agree that getting a good (100% precise) GC is an important 
first step.  I am not so sure about generation based GC when you 
have a window on a world map that you move around which roughly 
is FIFO (first in, first out).

But to get good speed I think you are better off having multiple 
pools that can be released with no collection when a 
network-connection drops (if you have one conceptual pool per 
connection), and optimized allocators that give you 
pre-initialized objects etc.

In the ideal world all of this is transparent once you have 
specified your memory model (in detail), so you only have to 
issue a "new PlayerConnection" in the main logic of your program 
and can tweak the memory handling elsewhere. That is not the D 
way, from what I can tell from the forum posts so far, because 
"new" is going to stay tied to one global GC heap. So you have to 
write utility functions… which makes programs less legible.



 May 12 2014



prev sibling  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:

 2. you can have the non-pausible code running in a thread that 
 is not registered with the gc, so the gc won't pause it. This 
 requires that this thread not allocate gc memory, but it can 
 use gc memory allocated by other threads, as long as those 
 other threads retain a root to it.


This and  nogc is a very promising trend, but you should still be 
able to partion the GC search space.

The key to controlled real time performance is to partition the 
search space, that goes for anything algorithmic; memory 
management inclusive. That applies to any scheme like owned, ARC, 
GC etc. It makes little sense having to trace everything if only 
the physics engine is the one churning memory like crazy.

And fork() is not a solution without extensive structuring of 
allocations. Stuff like physics touch all pages the physics 
objects are onto like 100+ times per second, so you need to group 
allocations to pages based on usage patterns. (No point in 
forking if you get write traps on 50.000 pages the next time the 
physics engine run :-).


 3. D allows you to create and use any memory management scheme 
 you want. You are simply not locked into GC. For example, I 
 rewrote my Empire game into D and it did not do any allocation 
 at all - no GC, not even malloc. I know that you'll need to do 
 allocation, I'm just pointing out that GC allocations and 
 pauses are hardly inevitable.


This is no doubt the best approach for a MMO client. You have a 
window on the world and cache as much as possible both to memory 
and disk. Basically get as much memory from the OS as you can 
hoard (with headroom set by heuristics) when your application has 
focus and release caches that are outside the window when you 
loose focus to another application. This means you need a 
dedicated runtime for games that can delay GC collection and eat 
into the caches when you are low on computational resources.

You also need to distinguish between memory that is locked to RAM 
and memory that can swap. You should always lock memory for real 
time threads. So if you want to GC this, you need a GC that 
support multiple heaps.

(Some hardware might also distinguish between RAM that is 
accessible by the GPU or that has different characteristics in 
areas such as persistence or performance.)


 5. you can divide your app into multiple processes that 
 communicate via interprocess communication. One of them pausing 
 will not pause the others. You can even do things like turn off


Why processes and not threads with their own local GC?


 6. If you call C++ libs, they won't be allocating memory with 
 the D GC. D code can call C++ code. If you run those C++ libs


But what happens if that C++ code does "new 
HeapStuff(D_allocated_memory)" and then calls back to D? You 
cannot presume that C++ coders have the discipline to always 
allocate local memory from the stack, so basically you cannot GC 
collect while there are C++ functions on the stack. In order to 
get there the GC collector needs to understand the malloc heap 
and trace that one too.

Auditing all C++ libraries I want to use is too much work, and 
tracing the malloc heap is too time consuming, so at the end of 
the day you'll get a more robust environment by only scanning 
(tracing) the stacks when there is only D function calls on the 
stack, with a precise collector.

That means you need to partition the search space otherwise the 
collector might not run in time.

Freezing the world is really ugly. Most applications are actually 
soft real time.

Games are part hard real time, part soft real time. The 
difference between games and other applications is that there is 
less headroom so you have to do more work to make the "glitches" 
and "stuttering" occur sufficiently seldom to be perceived as 
acceptable by the end user. But games are not special.



 May 12 2014



prev sibling next sibling parent reply "Tommi" <tommitissari hotmail.com>  writes:


On Monday, 12 May 2014 at 00:50:24 UTC, Walter Bright wrote:

 On 5/11/2014 1:59 PM, Timon Gehr wrote:

 Borrowed pointers are not even superficially similar to near*. 
 They are
 compatible with everything else, because they can store data 
 that was borrowed
 from anywhere else.


 As long as those pointers don't escape. Am I right in that one 
 cannot store a borrowed pointer into a global data structure?


Perhaps:

struct Test {
     n: &'static int, // [1]
     m: int
}

static val: int = 123;
static mut t: Test = Test { n: &'static val, m: 0 };

fn main() {
     unsafe { // [2]
         let p = &mut t.m;
         *p = 456;
         println!("{} {}", *t.n, t.m); // prints: 123 456
     }
}

[1]: In order to create a static instance of 'Test', the 'n' 
field (which is a borrowed pointer) must be specified as to be 
pointing at a static immutable (int) variable.

[2]: Any use of static mutable data requires the use of an 
'unsafe' block (similar to  trusted in D)



 May 12 2014



  parent  "Tommi" <tommitissari hotmail.com>  writes:


On Monday, 12 May 2014 at 08:10:43 UTC, Tommi wrote:

 Perhaps: [..]


Somewhat surprisingly to me, you can later on change the borrowed 
pointer in the mutable static 'Test' to point at a mutable static 
int:

struct Test {
     n: &'static int
}

static old: int = 111;
static mut new: int = 222;
static mut t: Test = Test { n: &'static old };

fn main() {
     unsafe {
         println!("{}", *t.n); // prints: 111
         t.n = &new; // Can point to a different static
         println!("{}", *t.n); // prints: 222
         // ...but can't point to a local, e.g:
         // let v = 123;
         // t.n = &v; // error: `v` does not live long enough
         new = 333;
         println!("{}", *t.n); // prints: 333
     }
}



 May 12 2014



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 05/12/2014 02:50 AM, Walter Bright wrote:

 On 5/11/2014 1:59 PM, Timon Gehr wrote:

 On 05/11/2014 10:05 PM, Walter Bright wrote:

 That's clearly an additional benefit of the borrowed pointer notion. But
 have you examined generated Rust code for the cost of inc/dec? I
 haven't, but I don't see any way they could avoid this (very expensive)
 cost without borrowed pointers.


 Sure, but performance is the additional benefit.


 One constant theme in this thread, one I find baffling, is the regular
 dismissal of the performance implications of inc/dec.


Irrelevant, I'm not doing that.

(And again, reference counting is not the only allocation mechanism in 
Rust. AFAICT, most allocations use owned pointers.)


 Borrowed pointers are not necessary to support raw pointers  - this can be
(and is in some
 systems) supported by simply wrapping the raw pointer with a dummy
 reference count.
 ...


I have no idea what this part is trying to bring across.


 The reason for borrowed pointers is performance.


No, it is safety. Raw pointers give you all of the performance.


 Rust would be non-viable without them.


True in that it would fail to meet its design goals. Rust provides a 
tracing garbage collector as well, so it is at least as viable as D 
regarding performance of safe memory management. (Probably more, 
actually, because it does not provide precision-unfriendly constructs 
such as undiscriminated unions.)


 I strongly suggest writing a snippet in [[insert your favorite proven
 technology RC language here]] and disassembling the result, and have a
 look at what inc/dec entails.
 ...


I don't have trouble seeing the cost of reference counting. (And it's 
you who claimed that this is going to be the only memory allocation 
scheme in use in Rust code.)


 The thing is, if the compiler is capable of figuring out these lifetimes
 by examining the code,


 There are explicit lifetime annotations in function signatures.


 Yes, because the compiler cannot figure it out itself, so the programmer
 has to annotate.
 ...


You are saying 'if the compiler is capable of figuring out these 
lifetimes by examining the code, then ...' and then you are saying that 
the compiler is incapable of figuring them out itself. What is it that 
we are arguing here? Are you saying the Rust compiler should infer all 
memory management automatically or that it cannot possibly do that, or 
something else?


 It is simply not true that type systems are inherently restricted to
 checking
 trivial properties. They can be made as strong as mathematical logic
 without
 much fuss.


 Again, Rust would not need borrowed pointers nor the annotations for
 them if this knowledge could be deduced by the compiler. Heck, if the
 compiler can deduce lifetimes accurately,


It does not deduce anything. It checks that borrowed pointers do not 
outlive their source. Lifetime parameters are used to transport the 
required information across function signatures.


 you can get rid of GC and RC,
 and just have the compiler insert malloc/free in the right spots.
 ...


That's a form of GC, and I already acknowledged that global region 
inference exists, but noted that this is not what is used.


 Note that there is a Java version that does this partway, sometimes it
 will replace a GC object with a stack allocated one if it is successful
 in deducing that the object lifetime does not exceed the lifetime of the
 function.
 ...


I know. Also, inference is harder to control and less efficient than 
simply making the relevant information part of type signatures.

http://en.wikipedia.org/wiki/Region-based_memory_management#Region_inference

"This work was completed in 1995[9] and integrated into the ML Kit, a 
version of ML based on region allocation in place of garbage collection. 
This permitted a direct comparison between the two on medium-sized test 
programs, yielding widely varying results ("between 10 times faster and 
four times slower") depending on how "region-friendly" the program was; 
compile times, however, were on the order of minutes."


 Yes, one is & and the other is  .


 No, actually currently one is & and the other is RC<T> AFAIK.


 Then Rust changed again. The document I read on borrowed pointers was
 likely out of date, though it had no date on it.
 ...


Yes, most documents on Rust are at least slightly out of date.


 RC<T> is not more general. It cannot refer to stack-allocated data,
 for instance.


 So there is no general pointer type that has an unbounded lifetime?
 ...


How can it be general and have an unbounded lifetime and be safe?


 Sure, borrowing is very lightweight, but ultimately what is most
 important is
 that it solves the problem of multiple incompatible pointer types and
 makes the
 type system more expressive as well.


 Adding more pointer types makes a type system more expressive, by
 definition.
 ...


No. Also, this is irrelevant, because I was highlighting the 
_importance_ of the fact that it does in this case.


 A function that uses none of the specific pointer capabilities is more
 general,
 so what other choice of 'default' makes sense?


 A function that doesn't have restrictions on what can be done with the
 pointers passed to it.


You mean a function that only accepts GC-managed pointers?
(Not having _any_ restrictions at all does not make any sense.)
I don't see how this is better, because such a function is usable in 
less contexts.


 Borrowed pointers have restrictions on their
 usage - this is explicitly stated in the Rust documentation.
 ...


Values of every type have restrictions on their usage. If your function 
requires less capabilities from it's arguments, then it is usable in 
more cases. Why is this not the most sensible thing to do whenever possible?


 Convenience and reusable functions means using borrowed pointers
 whenever possible.


 Of course. And writing fast code means making your functions fast
 whenever possible!
 ...


What is your point here?


 It seems clear that the decisions of borrow/managed are going to pervade
 Rust code.


 But they are often obvious.


 I've written a lot of ref counted code in the past, enough to know that
 such is a pretty optimistic statement. Dealing with was a significant
 and ongoing drain on my time, especially when the programs and data
 structures got more complex.
 ...


You said you don't have any experience writing Rust code, and claimed 
that it is not a proven technology, so I fail to see how you can have 
any relevant experience with borrowing.


...


 Borrowed pointers are not even superficially similar to near*. They are
 compatible with everything else, because they can store data that was
 borrowed
 from anywhere else.


 As long as those pointers don't escape. Am I right in that one cannot
 store a borrowed pointer into a global data structure?


Sure. If you are going to escape a pointer, you need to think about how 
that memory is managed. That's how it is. And there are many options: 
You could simply transfer ownership, you might use reference counting, 
or a tracing GC.


 The similarity is
 that there are one way conversions from one to the other, and one of the
 types is more general.


And the difference is that the one you can convert to is faster here.


 I infer from your other statements about Rust
 that it doesn't actually have a general pointer type.


Memory can be borrowed no matter how it was allocated. That's what I 
call general. How do you fill that term?



 May 12 2014



  parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Timon Gehr:


 (Probably more, actually, because it does not provide 
 precision-unfriendly constructs such as undiscriminated unions.)


I suggested to add an optional method named "onGC" to unions that 
if present is called at run-time by the GC to know what's the 
real type of stored data, to make tracing more precise.

Bye,
bearophile



 May 12 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 4:35 AM, bearophile wrote:

 I suggested to add an optional method named "onGC" to unions that if present is
 called at run-time by the GC to know what's the real type of stored data, to
 make tracing more precise.


Unions of pointers are so rare in actual code that treating them conservatively 
is not a big problem.



 May 12 2014



 next sibling parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Walter Bright:


 Unions of pointers are so rare in actual code that treating 
 them conservatively is not a big problem.


std.variant.Algebraic is based on on std.variant.VariantN, and on 
std.variant.VariantN is based on an union, and often you use 
algebraic data types to represent trees and similar data 
structures that contain many references/pointers. Adding Adding 
an onGC() method to std.variant.VariantN you allow the GC to 
manage Algebraic well enough.

Bye,
bearophile



 May 12 2014



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 10:25 AM, bearophile wrote:

 Walter Bright:


 Unions of pointers are so rare in actual code that treating them
 conservatively is not a big problem.


 std.variant.Algebraic is based on on std.variant.VariantN, and on
 std.variant.VariantN is based on an union, and often you use algebraic data
 types to represent trees and similar data structures that contain many
 references/pointers. Adding Adding an onGC() method to std.variant.VariantN you
 allow the GC to manage Algebraic well enough.


BTW, the RTinfo can be used to discriminate unions.



 May 12 2014



 next sibling parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Walter Bright:


 BTW, the RTinfo can be used to discriminate unions.


I don't know if std.variant.VariantN is already using such 
RTinfo. I don't know much about RTinfo.

Bye,
bearophile



 May 12 2014



prev sibling  parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Walter Bright:


 On 5/12/2014 10:25 AM, bearophile wrote:

 Walter Bright:


 Unions of pointers are so rare in actual code that treating 
 them
 conservatively is not a big problem.


 std.variant.Algebraic is based on on std.variant.VariantN, and 
 on
 std.variant.VariantN is based on an union, and often you use 
 algebraic data
 types to represent trees and similar data structures that 
 contain many
 references/pointers. Adding Adding an onGC() method to 
 std.variant.VariantN you
 allow the GC to manage Algebraic well enough.


 BTW, the RTinfo can be used to discriminate unions.


Do I have to open a Bugzilla entry asking for VariantN to use 
such RTinfo to allow a more precise tracing of pointer-heavy 
Algebraics? Suggestions are welcome.

Bye,
bearophile



 May 14 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/12/14, 10:25 AM, bearophile wrote:

 Walter Bright:


 Unions of pointers are so rare in actual code that treating them
 conservatively is not a big problem.


 std.variant.Algebraic is based on on std.variant.VariantN, and on
 std.variant.VariantN is based on an union, and often you use algebraic
 data types to represent trees and similar data structures that contain
 many references/pointers. Adding Adding an onGC() method to
 std.variant.VariantN you allow the GC to manage Algebraic well enough.


I, too, felt the need of onGC() - actually preGC() - in my allocators 
implementation.

Specifically, a thread-local freelist would save a pointer to the root 
in thread-local storage (i.e. a traditional D global variable). That 
would thread through a number of free nodes available for allocation.

When a GC cycle occurs, it's okay if the list stays referenced; the GC 
will consider it "used" and won't do anything in particular about it. 
However, the GC cycle is a good opportunity to clean these freelists and 
offer the memory for other size classes, seeing as the freelists may 
grow unreasonably large and then just hold memory for no good reason.

A hook that nulls all freelist heads just as the collection process 
starts would be helpful.


Andrei



 May 12 2014



 next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


12-May-2014 22:08, Andrei Alexandrescu пишет:

 On 5/12/14, 10:25 AM, bearophile wrote:
 A hook that nulls all freelist heads just as the collection process
 starts would be helpful.


One word - weak pointers. Then head of freelist is weak and can be 
collected at whim.


 Andrei



-- 
Dmitry Olshansky



 May 12 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/12/14, 11:17 AM, Dmitry Olshansky wrote:

 12-May-2014 22:08, Andrei Alexandrescu пишет:

 On 5/12/14, 10:25 AM, bearophile wrote:
 A hook that nulls all freelist heads just as the collection process
 starts would be helpful.


 One word - weak pointers. Then head of freelist is weak and can be
 collected at whim.


Of course. My point here is that here you need simpler support than 
full-blown weak pointers. -- Andrei



 May 12 2014



prev sibling  parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Andrei Alexandrescu:


 I, too, felt the need of onGC() - actually preGC() - in my 
 allocators implementation.
 ...
 A hook that nulls all freelist heads just as the collection 
 process starts would be helpful.


How is this going to help increase tracing precision of unions 
(and Algebraic built on top of unions)?

Bye,
bearophile



 May 12 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/12/14, 12:59 PM, bearophile wrote:

 Andrei Alexandrescu:


 I, too, felt the need of onGC() - actually preGC() - in my allocators
 implementation.
 ...
 A hook that nulls all freelist heads just as the collection process
 starts would be helpful.


 How is this going to help increase tracing precision of unions (and
 Algebraic built on top of unions)?


How did I give the impression it has anything to do with unions? -- Andrei



 May 12 2014



  parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Andrei Alexandrescu:


 How did I give the impression it has anything to do with 
 unions? -- Andrei


OK, so yours is not an answer to my proposal, nor related to it.

Bye,
bearophile



 May 12 2014



prev sibling  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 13 May 2014 03:17, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/12/2014 4:35 AM, bearophile wrote:

 I suggested to add an optional method named "onGC" to unions that if
 present is
 called at run-time by the GC to know what's the real type of stored data,
 to
 make tracing more precise.



 Unions of pointers are so rare in actual code that treating them
 conservatively is not a big problem.


I find it fairly common.
I just searched through my code, and 7 out of 12 unions had pointers.



 May 12 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 10:31 AM, Manu via Digitalmars-d wrote:

 I just searched through my code, and 7 out of 12 unions had pointers.


Relative number of objects with unions, not declarations with unions!



 May 12 2014



  parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 13 May 2014 03:44, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/12/2014 10:31 AM, Manu via Digitalmars-d wrote:

 I just searched through my code, and 7 out of 12 unions had pointers.



 Relative number of objects with unions, not declarations with unions!


Ah, well I have 3 different tree/graph structures with unions, and
tree/graph nodes have a tendency to accumulate many instances.



 May 12 2014



prev sibling  parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 08:40, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/10/2014 8:54 AM, Manu via Digitalmars-d wrote:


 Recall too that D has significant opportunity to
 improve on ARC as implemented by other languages,


 You're essentially arguing that one is easy pickings and the other is
 impractically difficult, but they're actually about the same level.


That's not actually what I was arguing at all. I was arguing that D's
ARC offering would likely be superior to any that have come before,
due to inherent advantages in the language, and by all reports, D is
incapable of having the worlds best GC implemented successfully for
various reasons.

If it is possible to implement the worlds best GC in D, it's still not
a done deal though. We still need to consider; does the worlds best GC
remain incompatible with my workload at a fundamental level? What do
we do about destructors? If ARC is within 10% of the worlds best GC,
and both are equally attainable in D's context, is 10% a fair price to
pay for a more inclusive technology?
Who is it that will declare that 10% difference a deal-breaker?
(they're obviously not here now, since D's existing GC is not in that
ballpark)
With them in mind, and upon an ARC platform, can we offer those users
the GC as a library? Or perhaps a mechanism to disable RC code
generation in their local code and falling back upon the
cycle-collecting GC already in place to pick up the task?



 May 10 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/10/14, 6:33 AM, Manu via Digitalmars-d wrote:

 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.



 This is very, very, very wrong. -- Andrei


 Is there any evidence to support that assertion?


Of course, and I've shown it numerous times, only for it to be 
blissfully ignored. Probably the most recent and influential work on it is

http://research.microsoft.com/pubs/202163/rcix-oopsla-2013.pdf

of which FIRST sentence is "Despite some clear advantages and recent 
advances, reference counting remains a poor cousin to high-performance 
tracing garbage collectors." The paper reviews previous work on the 
matter, ALL of which shows that RC is behind GC in speed, and presents 
(for the first time in 2013) new work that brings about parity of RC 
with GC.


 I have many colleagues who work on phones. The patterns I see are that
 the iOS guys who use Obj-C never complain about it, and the GC guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should be a
 more populous group to draw samples from.


As competent engineers we can't afford to be ignorant of the existing 
research and practice showing a much more complex tradeoff space than a 
black/white distinction. We also can't do the capital mistake of 
engineering - the beauty of the part at the expense of the beauty of the 
whole.


Andrei



 May 10 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 02:15, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/10/14, 6:33 AM, Manu via Digitalmars-d wrote:

 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.




 This is very, very, very wrong. -- Andrei



 Is there any evidence to support that assertion?



 Of course, and I've shown it numerous times, only for it to be blissfully
 ignored. Probably the most recent and influential work on it is

 http://research.microsoft.com/pubs/202163/rcix-oopsla-2013.pdf

 of which FIRST sentence is "Despite some clear advantages and recent
 advances, reference counting remains a poor cousin to high-performance
 tracing garbage collectors." The paper reviews previous work on the matter,
 ALL of which shows that RC is behind GC in speed, and presents (for the
 first time in 2013) new work that brings about parity of RC with GC.


I saw that paper before. And I thought, if anything, it supported my
argument in the context of D.
The next sentence goes on to list the advantages of RC (issues we have
wrestled with, like destructors), and then goes on to say the recent
awesome RC is within 10% of "the fastest tracing collectors".
Are you suggesting that D's GC is among 'the fastest tracing
collectors'? Is such a GC possible in D?

Has anyone compared D's GC against 'the fastest tracing collectors'?
What is the difference? Does D fall within 10%?
If not, then chances are, according to this paper, RC would be
superior to the existing GC in terms of performance, as well as
reliability/predictability/functionality (destructors that work, not
incompatible with embedded software).

If an awesome RC is implementable in D, but an awesome GC is not, then
I can't see that even any performance argument can remain in the
debate. They are talking specifically about a GC-backed-RC too, to
handle cycles, so their implementation does not come at any cost to
safety or added complexity for users, and has the key advantage that
people like me may have the option to disable the cycle collector and
take manual responsibility if desired.


 I have many colleagues who work on phones. The patterns I see are that
 the iOS guys who use Obj-C never complain about it, and the GC guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should be a
 more populous group to draw samples from.



 As competent engineers we can't afford to be ignorant of the existing
 research and practice showing a much more complex tradeoff space than a
 black/white distinction. We also can't do the capital mistake of engineering
 - the beauty of the part at the expense of the beauty of the whole.


There is a black/white distinction though.
It you can't make the freezing go away, it is _incompatible_ with
certain classes of software.
I don't care what the performance difference is when this is the case.
If one option is incompatible and the other is not, then there is no
tradeoff.

There are also technical advantages to RC too, like destructors that
work, which I find very compelling.



 May 10 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/10/14, 10:03 AM, Manu via Digitalmars-d wrote:

 On 11 May 2014 02:15, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/10/14, 6:33 AM, Manu via Digitalmars-d wrote:

 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:

 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.


 This is very, very, very wrong. -- Andrei



 Is there any evidence to support that assertion?



 Of course, and I've shown it numerous times, only for it to be blissfully
 ignored. Probably the most recent and influential work on it is

 http://research.microsoft.com/pubs/202163/rcix-oopsla-2013.pdf

 of which FIRST sentence is "Despite some clear advantages and recent
 advances, reference counting remains a poor cousin to high-performance
 tracing garbage collectors." The paper reviews previous work on the matter,
 ALL of which shows that RC is behind GC in speed, and presents (for the
 first time in 2013) new work that brings about parity of RC with GC.


 I saw that paper before.


Then the right thing is to integrate it within one's worldview. It is 
difficult to reconcile your assertions and questions with the claim that 
you have read and internalized existing work on the topic.


 And I thought, if anything, it supported my
 argument in the context of D.


It doesn't seem that way to me. Your arguments so far seem to come from 
a staunch niche position that ignores factual realities and has no 
concern for the larger environment. Some of such arguments are factually 
incorrect, and just the minimal amends and qualifications are not 
sufficient.


 The next sentence goes on to list the advantages of RC (issues we have
 wrestled with, like destructors), and then goes on to say the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?


I believe it is.


 Has anyone compared D's GC against 'the fastest tracing collectors'?
 What is the difference? Does D fall within 10%?


In all likelihood, no.


 If not, then chances are, according to this paper, RC would be
 superior to the existing GC in terms of performance,


I disagree.


 as well as
 reliability/predictability/functionality (destructors that work, not
 incompatible with embedded software).


I partially agree, but this is getting rather difficult to assess. Let's 
say I'd agree it's a valid viewpoint.


 If an awesome RC is implementable in D, but an awesome GC is not, then
 I can't see that even any performance argument can remain in the
 debate. They are talking specifically about a GC-backed-RC too, to
 handle cycles, so their implementation does not come at any cost to
 safety or added complexity for users, and has the key advantage that
 people like me may have the option to disable the cycle collector and
 take manual responsibility if desired.


 I have many colleagues who work on phones. The patterns I see are that
 the iOS guys who use Obj-C never complain about it, and the GC guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should be a
 more populous group to draw samples from.



 As competent engineers we can't afford to be ignorant of the existing
 research and practice showing a much more complex tradeoff space than a
 black/white distinction. We also can't do the capital mistake of engineering
 - the beauty of the part at the expense of the beauty of the whole.


 There is a black/white distinction though.
 It you can't make the freezing go away, it is _incompatible_ with
 certain classes of software.
 I don't care what the performance difference is when this is the case.
 If one option is incompatible and the other is not, then there is no
 tradeoff.


Of course it would be nice to have better support for reference 
counting. My understanding is that you advocate that reference counting 
would be compulsive, which would preclude the GC option.


Andrei



 May 10 2014



 next sibling parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 03:54, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 5/10/14, 10:03 AM, Manu via Digitalmars-d wrote:

 There is a black/white distinction though.
 It you can't make the freezing go away, it is _incompatible_ with
 certain classes of software.
 I don't care what the performance difference is when this is the case.
 If one option is incompatible and the other is not, then there is no
 tradeoff.



 Of course it would be nice to have better support for reference counting. My
 understanding is that you advocate that reference counting would be
 compulsive, which would preclude the GC option.


As I've said many times, I'm really not married to ARC... it just
makes sense. In the years I've been hanging around here, I'm yet to
see anyone present a realistic path for D towards an acceptable GC
solution.
If anybody can do that, I'll reconsider my position. But it seems it's
becoming less and less likely to me.

What I know is that there is a solution which has a proven and
extremely successful track record in native languages and
embedded/realtime systems, and even solves ancillary problems like
destructors while at it. But most importantly, it doesn't exclude any
classes of software by nature.

If ARC were a foundation for D, I wonder if there's opportunity for
programs that may elect to disable the RC and leave the work
exclusively to the backing cycle collector, in the same way that I
might disable the cycle collector and take manual responsibility for
weak references...?

I may argue from a biased perspective, but placing more weight on some
particular negative qualities of a technology is not unreasonable when
they represent a showstopper. Which part of my issue with the GC is
factually incorrect? All it would take is for anyone competent to tell
me how a magic GC (which is compatible with D) will stop freezing in
the low/no-memory environment, and I'll shut up.
Even controlling it... I was initially wishing for something like an
incremental garbage collector, which I could allocate a maximum
periodic time-slice (1ms or less) on a regular basis. Not only was it
agreed that this is basically impossible (a bunch of collection
ground-work would need to be repeated at the start of each
micro-collect, probably representing more time than the total
time-slice... iirc), but it almost certainly loses any raw performance
advantage over ARC in such a case anyway, and it becomes moot.



 May 10 2014



prev sibling  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 10.05.2014 19:54, schrieb Andrei Alexandrescu:

 The next sentence goes on to list the advantages of RC (issues we have
 wrestled with, like destructors), and then goes on to say the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?


 I believe it is.


While it might be possible to implement a good GC in D it would require 
major changes in the language and its librariers. In my opinion it would 
be way more work to implement a propper GC than to implement ARC.

Every state of the art GC requires percise knowdelge of _all_ pointers. 
And thats exactly what we currently don't have in D.

Just to list a few issues with implementing this in D:

1) Calling C-Functions would require some kind of pinning API for the 
GC. But functions with C-calling convections are used ecessivly in 
druntime. So a lot of uneccessary pinning would happen within d-runtime 
just because of all the c-function hacks around the module system.

2) Most modern GC algorithms require write-barriers for pointers which 
are on the heap. This is a major problem for D, because structs can be 
both on the stack and on the heap. So if you call a method of a struct, 
the compiler can not know if the write barriers are actually neccessary 
(for heap based structs) or unneccssary (for stack based structs, e.g. 
inside classes).

3) Generating percise information of the location of pointers on the 
stack is going to be a major effort, because of structs, unions, 
exception handling and Ds ability to move struct instances freely around 
when dealing with exceptions.

4) As soon as you want to be fully percise you can not halt threads in 
random points, you will need to generate GC points on which the 
execution can be safely halted. This is going to require major effort 
again if it should be done efficiently.

5) As Ds strings are immutable(char)[] they can not be allocated on a 
thread local pool, because they must be freely shareable between 
threads, and there is currently no way to detect via the type system 
whean a immutable(char)[] is shared with another thread. So we would 
need to design a system were the sharing of memory blocks would be 
detected, so we could properly implement thread local GC pools. The same 
issue applies to __gshared.

Honenstly, D doesn't currently have any GC support at all. The only GC 
support we have is that the GC knows all threads and can halt them. And 
D knows which function to call, to destroy class objects. So basically 
we have the same level of GC support as C or C++. You could just use the 
Boem GC, plug it into C or C++ and have a GC that is as good as the D one.

So having a good GC in 5 years or having ARC in 1 year isn't a decision 
that is that hard for me.

Kind Regards
Benjamin Thaut



 May 11 2014



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 1:22 AM, Benjamin Thaut wrote:

 Honenstly, D doesn't currently have any GC support at all. The only GC support
 we have is that the GC knows all threads and can halt them. And D knows which
 function to call, to destroy class objects. So basically we have the same level
 of GC support as C or C++. You could just use the Boem GC, plug it into C or
C++
 and have a GC that is as good as the D one.


This is not quite correct. The Boehm GC knows nothing about the interiors of 
structs.

The D one does (or at least has the capability of doing so using RTinfo). This 
means that the D collector can be 'mostly' precise, the imprecision would be
for 
stack data.

The Boehm collector cannot move objects around, the D one can.



 May 11 2014



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 11.05.2014 10:53, schrieb Walter Bright:

 On 5/11/2014 1:22 AM, Benjamin Thaut wrote:

 This is not quite correct. The Boehm GC knows nothing about the
 interiors of structs.

 The D one does (or at least has the capability of doing so using
 RTinfo). This means that the D collector can be 'mostly' precise, the
 imprecision would be for stack data.


Mostly percise doesn't help. Its either fully percise or beeing stuck 
with a impercise mark & sweep. Also D's GC doesn't use RTInfo currently. 
Additionally Rainer Schuetzes implementation showed that using RTInfo 
with the current GC makes the collection actually slower at the price of 
beeing 'mostly' percise. So I argue that the level of GC support we have 
is still at the level of C/C++.


 The Boehm collector cannot move objects around, the D one can.


Oh it can? Really? I would love to see how well interfacing with any C 
library works, if the D garbage collector actually does that. I bet GTKd 
would break the first second this is implemented. Or any other C library 
that exchanges data with D for that matter. The D garbage collector can 
not simply move around objects because we don't support pinning.

Also I'm talking about what the D garbage collector currently actually 
does, and not what it could do. If we actually would implement arc, it 
would most likely take less time then a full blown proper gc and we 
would end up with better performance than we currently have. Beating the 
300% slowdown which the current GC imposes is not that hard.

If we however keep arguing what a GC could do, we will be stuck with the 
impercise mark & sweep forever. (or at least the next 5 years)



 May 11 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 2:48 AM, Benjamin Thaut wrote:

 Mostly percise doesn't help. Its either fully percise or beeing stuck with a
 impercise mark & sweep.


This is not correct. It helps because most of the false pointers will be in the 
heap, and the heap will be accurately scanned, nearly eliminating false 
references to garbage.



 Also D's GC doesn't use RTInfo currently.


Yes, but that is not an issue with the language design.



 Additionally Rainer Schuetzes implementation showed that using RTInfo with the
current GC
 makes the collection actually slower at the price of beeing 'mostly' percise.


As I recall, his implementation showed dramatic improvements in memory
consumption.

(All GC schemes are tradeoffs.)



 The Boehm collector cannot move objects around, the D one can.


 Oh it can? Really?


Yes. D, for example, requires that objects not be self-referential for this
reason.



 I would love to see how well interfacing with any C library
 works, if the D garbage collector actually does that.


The current D GC does not do that, but it can. It will work fine with C 
libraries as long as a pointer to the data is retained on the stack, as it 
naturally will be by passing it as a stack parameter to a C function. Roots 
found on the stack would automatically pin the object.

I know this works because I implemented it for a Java VM back in the 90's. 
Someone wrote a paper about a similar scheme calling it a "mostly copying 
collector".



 Also I'm talking about what the D garbage collector currently actually does,
and
 not what it could do.


I think it would be more useful to talk about what it could do without language 
changes.



 If we actually would implement arc, it would most likely
 take less time then a full blown proper gc and we would end up with better
 performance than we currently have. Beating the 300% slowdown which the current
 GC imposes is not that hard.

 If we however keep arguing what a GC could do, we will be stuck with the
 impercise mark & sweep forever. (or at least the next 5 years)


I believe the performance claims of pervasive ARC are unjustifiably optimistic 
and based on unproven technology that has never been implemented anywhere.



 May 11 2014



 next sibling parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 11.05.2014 22:33, Walter Bright wrote:

 The Boehm collector cannot move objects around, the D one can.


 Oh it can? Really?


 Yes. D, for example, requires that objects not be self-referential for
 this reason.


I don't think the GC would have problems with fixing up internal 
pointers to the object itself. self-referential is prohibited to allow 
moving structures by memcpy, e.g. as return value.



 May 11 2014



 next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 1:45 PM, Rainer Schuetze wrote:

 On 11.05.2014 22:33, Walter Bright wrote:

 The Boehm collector cannot move objects around, the D one can.


 Oh it can? Really?


 Yes. D, for example, requires that objects not be self-referential for
 this reason.


 I don't think the GC would have problems with fixing up internal pointers to
the
 object itself. self-referential is prohibited to allow moving structures by
 memcpy, e.g. as return value.


The earlier D design did not have postblit, and so there was no fixing up of 
internal pointers.



 May 11 2014



prev sibling  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Sunday, 11 May 2014 at 20:45:08 UTC, Rainer Schuetze wrote:

 On 11.05.2014 22:33, Walter Bright wrote:

 The Boehm collector cannot move objects around, the D one 
 can.


 Oh it can? Really?


 Yes. D, for example, requires that objects not be 
 self-referential for
 this reason.


 I don't think the GC would have problems with fixing up 
 internal pointers to the object itself. self-referential is 
 prohibited to allow moving structures by memcpy, e.g. as return 
 value.


Does this mean that you cannot safely implement something as 
basic as a circular linked list?



 May 11 2014



  parent  Rainer Schuetze <r.sagitario gmx.de>  writes:


On 12.05.2014 06:57, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Sunday, 11 May 2014 at 20:45:08 UTC, Rainer Schuetze wrote:

 On 11.05.2014 22:33, Walter Bright wrote:

 The Boehm collector cannot move objects around, the D one can.


 Oh it can? Really?


 Yes. D, for example, requires that objects not be self-referential for
 this reason.


 I don't think the GC would have problems with fixing up internal
 pointers to the object itself. self-referential is prohibited to allow
 moving structures by memcpy, e.g. as return value.


 Does this mean that you cannot safely implement something as basic as a
 circular linked list?


Depends on your implementation, but I would not expect that the nodes of 
your linked list could be subject to being moved (mostly happens on the 
stack AFAICT), as pointers to the node from other nodes would become 
invalid, too.



 May 12 2014



prev sibling  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Sun, 11 May 2014 16:33:04 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 5/11/2014 2:48 AM, Benjamin Thaut wrote:

 Mostly percise doesn't help. Its either fully percise or beeing stuck  
 with a
 impercise mark & sweep.


 This is not correct. It helps because most of the false pointers will be  
 in the heap, and the heap will be accurately scanned, nearly eliminating  
 false references to garbage.


It doesn't matter where the false pointers are. The largest issue with  
false pointers is not how many false pointers there are. It only matters  
how large the block is that it "points" at. The larger your blocks get,  
the more likely they are "pointed" at by the stack. On 32-bit systems,  
allocate 1/256th of your memory space (i.e. 16.5MB), and the likelihood of  
random data on the stack pointing at it is roughly 1/256. This problem is  
just about eliminated with 64-bit pointers.


 Yes. D, for example, requires that objects not be self-referential for  
 this reason.


As previously stated, self referencing does not preclude GC moving. This  
statement is simply false, you can self reference in D for objects. You  
cannot for structs, but not because of a possibility for the moving GC,  
but because of the requirement to be able to move a struct instance.

And in fact, even if it's forbidden, "requires" is too strong a word --  
there is no static or runtime prevention of this.

-Steve



 May 12 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:

 It doesn't matter where the false pointers are. The largest issue with false
 pointers is not how many false pointers there are. It only matters how large
the
 block is that it "points" at. The larger your blocks get, the more likely they
 are "pointed" at by the stack. On 32-bit systems, allocate 1/256th of your
 memory space (i.e. 16.5MB), and the likelihood of random data on the stack
 pointing at it is roughly 1/256. This problem is just about eliminated with
 64-bit pointers.


Generally, it is a bad idea to allocate such large blocks on the GC heap. GC's 
work best when the size of the objects being allocated is very small relative
to 
the size of the heap space.

Fortunately, it's a mathematical inevitability that large allocations relative 
to the GC size are rare, and so it isn't much of a pain to handle them manually.



 And in fact, even if it's forbidden, "requires" is too strong a word -- there
is
 no static or runtime prevention of this.


It's still forbidden. Andrei wrote a template that will verify this at runtime, 
but I don't recall its name.



 May 12 2014



 next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote:

 On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:

 pointing at it is roughly 1/256. This problem is just about 
 eliminated with
 64-bit pointers.




Not generally true. This presumes that the heap is not in the 
lower region of the address space and that you don't use 64 bit 
ints on the stack.


 Generally, it is a bad idea to allocate such large blocks on 
 the GC heap. GC's work best when the size of the objects being 
 allocated is very small relative to the size of the heap space.


Generally not true. This is a deficiency of not having a smart 
allocator / precise scanning that use available meta information 
properly (obtained statically or by profiling).


 Fortunately, it's a mathematical inevitability that large 
 allocations relative to the GC size are rare, and so it isn't 
 much of a pain to handle them manually.


Programmer pain is not measured in number of instances, but in 
terms of model complexity.

Ola.



 May 12 2014



  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Mon, 12 May 2014 14:14:28 -0400, Ola Fosheim Gr=C3=B8stad  =

<ola.fosheim.grostad+dlang gmail.com> wrote:


 On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote:

 On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:

 pointing at it is roughly 1/256. This problem is just about eliminat=






ed  =


 with
 64-bit pointers.




 Not generally true. This presumes that the heap is not in the lower  =



 region of the address space and that you don't use 64 bit ints on the =


 =


 stack.


I was thinking in terms of purely a random number happening to point at =
 =

heap data. Practically speaking, I don't know the true likelihood based =
on  =

the heap address scheme of 64-bit OSes, but I know that we always have a=
  =

complainer who will try and do an array-append test on 32-bit code, and =
 =

end up exhausting memory unexpectedly.

-Steve



 May 12 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Monday, 12 May 2014 at 21:22:09 UTC, Steven Schveighoffer 
wrote:

 On Mon, 12 May 2014 14:14:28 -0400, Ola Fosheim Grøstad 
 <ola.fosheim.grostad+dlang gmail.com> wrote:


 On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote:

 On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:

 pointing at it is roughly 1/256. This problem is just about 
 eliminated with
 64-bit pointers.




 Not generally true. This presumes that the heap is not in the 
 lower region of the address space and that you don't use 64 
 bit ints on the stack.


 I was thinking in terms of purely a random number happening to 
 point at heap data. Practically speaking, I don't know the true 
 likelihood based on the heap address scheme of 64-bit OSes, but


Wicked topic. In AMD64 mode hi-mem is usually reserved for kernel 
etc. Traditionally the unixy heap grew from low towards high 
addresses:

http://en.wikipedia.org/wiki/Sbrk

But that is legacy. I think mmap is it… :-P And layout is 
randomized to reduce the effect of buffer overflow etc.

:-(


 I know that we always have a complainer who will try and do an 
 array-append test on 32-bit code, and end up exhausting memory 
 unexpectedly.


Uhuh. Not focusing on precise collection gets ugly.



 May 12 2014



prev sibling  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Mon, 12 May 2014 13:52:20 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:

 It doesn't matter where the false pointers are. The largest issue with  
 false
 pointers is not how many false pointers there are. It only matters how  
 large the
 block is that it "points" at. The larger your blocks get, the more  
 likely they
 are "pointed" at by the stack. On 32-bit systems, allocate 1/256th of  
 your
 memory space (i.e. 16.5MB), and the likelihood of random data on the  
 stack
 pointing at it is roughly 1/256. This problem is just about eliminated  
 with
 64-bit pointers.


 Generally, it is a bad idea to allocate such large blocks on the GC  
 heap. GC's work best when the size of the objects being allocated is  
 very small relative to the size of the heap space.

 Fortunately, it's a mathematical inevitability that large allocations  
 relative to the GC size are rare, and so it isn't much of a pain to  
 handle them manually.


The issue arises when one allocates such a large block for temporary use  
repeatedly, but expects it to be collected between allocations. The  
consequences are extremely disastrous.

The workaround is simply to keep it around, but that's not always a  
scalable solution.


 And in fact, even if it's forbidden, "requires" is too strong a word --  
 there is
 no static or runtime prevention of this.


 It's still forbidden. Andrei wrote a template that will verify this at  
 runtime, but I don't recall its name.


Can you cite the spec where it says it's forbidden? Forgotten templates  
are not a convincing argument.

Regardless, Java can use a moving GC, and allows self references. The idea  
that self references prevent a moving GC is simply false. If you think  
about it a bit, you will understand why.

-Steve



 May 12 2014



 next sibling parent  "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Mon, 12 May 2014 17:32:09 -0400, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:



 The workaround is simply to keep it around, but that's not always a  
 scalable solution.


Sorry, actually you can free it. That's the correct workaround.

-Steve



 May 12 2014



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/12/2014 2:32 PM, Steven Schveighoffer wrote:

 It's still forbidden. Andrei wrote a template that will verify this at
 runtime, but I don't recall its name.


 Can you cite the spec where it says it's forbidden? Forgotten templates are not
 a convincing argument.

 Regardless, Java can use a moving GC, and allows self references. The idea that
 self references prevent a moving GC is simply false. If you think about it a
 bit, you will understand why.



I see this is not specified in the documentation. Not sure what happened here, 
but I'll have to think about it.



 May 12 2014



prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 1:22 AM, Benjamin Thaut wrote:

 2) Most modern GC algorithms require write-barriers for pointers which are on
 the heap. This is a major problem for D, because structs can be both on the
 stack and on the heap. So if you call a method of a struct, the compiler can
not
 know if the write barriers are actually neccessary (for heap based structs) or
 unneccssary (for stack based structs, e.g. inside classes).


I know about write barriers. The thing about D is, far fewer objects are 
allocated on the GC heap than in languages like Java, which don't have stack 
allocation at all.

The write barrier would have to be there for every write through a pointer.
This 
is justifiable where every pointer points to the GC heap, but I really do not 
believe it is justifiable for D, where only a minority does. D could not be 
performance competitive with C++ if write barriers were added.

D also cannot be performance competitive with C++ if pervasive ARC is used and 
memory safety is retained. Rust is attempting to solve this problem by using 
'borrowed' pointers, but this is unproven technology, see my reply to Manu
about it.



 May 11 2014



  parent reply "ponce" <contact gam3sfrommars.fr>  writes:


On Sunday, 11 May 2014 at 08:59:42 UTC, Walter Bright wrote:

 D also cannot be performance competitive with C++ if pervasive 
 ARC is used and memory safety is retained. Rust is attempting 
 to solve this problem by using 'borrowed' pointers, but this is 
 unproven technology, see my reply to Manu about it.


I work in a C++ shop and as I see it, resource management is 
becoming a solved problem:

- resource owners holds a std::unique_ptr<T> on them. Resource 
release is taken care by C++ destructors normally. That means to 
be exception-safe, each resource type must better have its class.
- resource users eventually "borrow" the resource by taking a raw 
pointer out of the unique pointer. What Rust would do with 
lifetimes here is ensure that the resource is still there since 
move semantics seems pervasive in this language. In C++ we ensure 
the resource holder outlive the users.
- std::shared_ptr is not needed with such constraints. This means 
no cycles in the object graph. TBH I have yet to find a 
dependency scheme that can't be done that way.

When I use D I can't help but think that releasing resources 
feels more manual and error-prone ("oops that resource should 
have been a struct not a class" and such traps).

I do not have huge concerns about D GC, but I would be glad to 
have more support for owned pointers (ie. Unique!T in Phobos or 
better). I have no idea how to make it safe ie. ensure the 
resource outlive its users.



 May 11 2014



  parent reply "sclytrack" <sclytrack fake.com>  writes:


On Sunday, 11 May 2014 at 11:42:37 UTC, ponce wrote:

 On Sunday, 11 May 2014 at 08:59:42 UTC, Walter Bright wrote:

 D also cannot be performance competitive with C++ if pervasive 
 ARC is used and memory safety is retained. Rust is attempting 
 to solve this problem by using 'borrowed' pointers, but this 
 is unproven technology, see my reply to Manu about it.


 I work in a C++ shop and as I see it, resource management is 
 becoming a solved problem:

 - resource owners holds a std::unique_ptr<T> on them. Resource 
 release is taken care by C++ destructors normally. That means 
 to be exception-safe, each resource type must better have its 
 class.
 - resource users eventually "borrow" the resource by taking a 
 raw pointer out of the unique pointer. What Rust would do with 
 lifetimes here is ensure that the resource is still there since 
 move semantics seems pervasive in this language. In C++ we 
 ensure the resource holder outlive the users.
 - std::shared_ptr is not needed with such constraints. This 
 means no cycles in the object graph. TBH I have yet to find a 
 dependency scheme that can't be done that way.

 When I use D I can't help but think that releasing resources 
 feels more manual and error-prone ("oops that resource should 
 have been a struct not a class" and such traps).

 I do not have huge concerns about D GC, but I would be glad to 
 have more support for owned pointers (ie. Unique!T in Phobos or 
 better). I have no idea how to make it safe ie. ensure the 
 resource outlive its users.


I like this owner/unique, borrow thing.

  is managed (currently reference counted)
~ is owner
      & is borrow

fn example3() -> int {
     let mut x = ~X {f: 3};
     let y = &x.f;
     x = ~X {f: 4};  // Error reported here.
     *y
}


According to Debian Rust is still too experimental to be packaged.
http://web.mit.edu/rust-lang_v0.8/doc/tutorial-borrowed-ptr.html


servo is written in Rust.
https://github.com/mozilla/servo

There is very little use of " ", it's mostly  "&" and "~". Heck I 
didn't find any   while casually browsing the code. It's like 
they are not using it at all.

I don't know Rust, it is the first day I look (read 
documentation) at it.



 May 11 2014



 next sibling parent reply "ponce" <contact gam3sfrommars.fr>  writes:


On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:

 There is very little use of " ", it's mostly  "&" and "~". Heck 
 I didn't find any   while casually browsing the code. It's like 
 they are not using it at all.


Similarly in current C++ std::shared_ptr is barely there while
std::unique_ptr is all over the place.



 May 11 2014



 next sibling parent reply "Francesco Cattoglio" <francesco.cattoglio gmail.com>  writes:


On Sunday, 11 May 2014 at 21:49:06 UTC, ponce wrote:

 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:

 There is very little use of " ", it's mostly  "&" and "~". 
 Heck I didn't find any   while casually browsing the code. 
 It's like they are not using it at all.


 Similarly in current C++ std::shared_ptr is barely there while
 std::unique_ptr is all over the place.


I guess that after burning their fingers with auto_ptr, people 
still have to realize that unique_ptr might be good to use.



 May 11 2014



  parent  "Francesco Cattoglio" <francesco.cattoglio gmail.com>  writes:


On Sunday, 11 May 2014 at 22:06:55 UTC, Francesco Cattoglio wrote:

 On Sunday, 11 May 2014 at 21:49:06 UTC, ponce wrote:

 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:

 There is very little use of " ", it's mostly  "&" and "~". 
 Heck I didn't find any   while casually browsing the code. 
 It's like they are not using it at all.


 Similarly in current C++ std::shared_ptr is barely there while
 std::unique_ptr is all over the place.


 I guess that after burning their fingers with auto_ptr, people 
 still have to realize that unique_ptr might be good to use.


sorry, my bad, I read it reversed :D



 May 11 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/11/14, 2:49 PM, ponce wrote:

 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:

 There is very little use of " ", it's mostly  "&" and "~". Heck I
 didn't find any   while casually browsing the code. It's like they are
 not using it at all.


 Similarly in current C++ std::shared_ptr is barely there while
 std::unique_ptr is all over the place.


The right tally here would include bald pointers as well. Also, I'm 
quite surprised by the confidence of this assertion. -- Andrei



 May 11 2014



 next sibling parent  "ponce" <contact gam3sfrommars.fr>  writes:


On Monday, 12 May 2014 at 00:44:54 UTC, Andrei Alexandrescu wrote:

 On 5/11/14, 2:49 PM, ponce wrote:

 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:

 There is very little use of " ", it's mostly  "&" and "~". 
 Heck I
 didn't find any   while casually browsing the code. It's like 
 they are
 not using it at all.


 Similarly in current C++ std::shared_ptr is barely there while
 std::unique_ptr is all over the place.


 The right tally here would include bald pointers as well. Also, 
 I'm quite surprised by the confidence of this assertion. -- 
 Andrei


Well I wa'm probably much biased indeed. I have no data.



 May 12 2014



prev sibling  parent  "ponce" <contact gam3sfrommars.fr>  writes:


On Monday, 12 May 2014 at 00:44:54 UTC, Andrei Alexandrescu wrote:

 On 5/11/14, 2:49 PM, ponce wrote:

 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:

 There is very little use of " ", it's mostly  "&" and "~". 
 Heck I
 didn't find any   while casually browsing the code. It's like 
 they are
 not using it at all.


 Similarly in current C++ std::shared_ptr is barely there while
 std::unique_ptr is all over the place.


 The right tally here would include bald pointers as well. Also, 
 I'm quite surprised by the confidence of this assertion. -- 
 Andrei


Well I was talking with no data indeed.



 May 12 2014



prev sibling next sibling parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 11.05.2014 23:43, schrieb sclytrack:

 On Sunday, 11 May 2014 at 11:42:37 UTC, ponce wrote:

 On Sunday, 11 May 2014 at 08:59:42 UTC, Walter Bright wrote:

 D also cannot be performance competitive with C++ if pervasive ARC is
 used and memory safety is retained. Rust is attempting to solve this
 problem by using 'borrowed' pointers, but this is unproven
 technology, see my reply to Manu about it.


 I work in a C++ shop and as I see it, resource management is becoming
 a solved problem:

 - resource owners holds a std::unique_ptr<T> on them. Resource release
 is taken care by C++ destructors normally. That means to be
 exception-safe, each resource type must better have its class.
 - resource users eventually "borrow" the resource by taking a raw
 pointer out of the unique pointer. What Rust would do with lifetimes
 here is ensure that the resource is still there since move semantics
 seems pervasive in this language. In C++ we ensure the resource holder
 outlive the users.
 - std::shared_ptr is not needed with such constraints. This means no
 cycles in the object graph. TBH I have yet to find a dependency scheme
 that can't be done that way.

 When I use D I can't help but think that releasing resources feels
 more manual and error-prone ("oops that resource should have been a
 struct not a class" and such traps).

 I do not have huge concerns about D GC, but I would be glad to have
 more support for owned pointers (ie. Unique!T in Phobos or better). I
 have no idea how to make it safe ie. ensure the resource outlive its
 users.


 I like this owner/unique, borrow thing.

   is managed (currently reference counted)
 ~ is owner
       & is borrow

 fn example3() -> int {
      let mut x = ~X {f: 3};
      let y = &x.f;
      x = ~X {f: 4};  // Error reported here.
      *y
 }


 According to Debian Rust is still too experimental to be packaged.
 http://web.mit.edu/rust-lang_v0.8/doc/tutorial-borrowed-ptr.html


 servo is written in Rust.
 https://github.com/mozilla/servo

 There is very little use of " ", it's mostly  "&" and "~". Heck I didn't
 find any   while casually browsing the code. It's like they are not
 using it at all.

 I don't know Rust, it is the first day I look (read documentation) at it.


Have you searched for RC<>() and GC<>() as well?

The   is gone from Rust and replaced by library types.

--
Paulo



 May 11 2014



prev sibling  parent  "Tommi" <tommitissari hotmail.com>  writes:


On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:

 I like this owner/unique, borrow thing.

   is managed (currently reference counted)
 ~ is owner
 & is borrow


I like it too. But a few notes:

1) The managed pointer  T has been deprecated and you should use 
the standard library types Gc<T> and Rc<T> instead.

2) The owned pointer ~T has been largely removed from the 
language and you should use the standard library type Box<T> 
instead.

The basic idea is that if a function needs to have ownership of 
its argument, the function should take its argument by value. And 
if the function doesn't need the ownership, it should take its 
argument either by a mutable or immutable reference (they don't 
like to call it "borrowed pointer" anymore, it's called simply a 
"reference" now). Owned types get moved by default when you pass 
them to a function that takes its argument by value. You call the 
'clone' method to make a copy of a variable of an owned type.



 May 12 2014



prev sibling  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 11.05.2014 10:22, Benjamin Thaut wrote:

 Am 10.05.2014 19:54, schrieb Andrei Alexandrescu:

 The next sentence goes on to list the advantages of RC (issues we have
 wrestled with, like destructors), and then goes on to say the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?


 I believe it is.


 While it might be possible to implement a good GC in D it would require
 major changes in the language and its librariers. In my opinion it would
 be way more work to implement a propper GC than to implement ARC.

 Every state of the art GC requires percise knowdelge of _all_ pointers.
 And thats exactly what we currently don't have in D.


I think most garbage collectors can work with a number of false 
pointers. The referenced memory area has to be treated as pinned and 
cannot be moved. Limiting the false pointers to stack and registers 
seems like a compromise, though most of the stack could even be 
annotated. Code for this does already exist in the debug info 
generation, though I suspect stack tracing could be unreliable.

Here's my current stance on the GC discussions:

I agree that the current GC is pretty lame, even if it were precise. 
"Stop-the-World" with complete tracing does not work for any interactive 
application that uses more than a few hundred MB of garbage collected 
memory (with or without soft-realtime requirements). Other applications 
with larger allocation requirements are easily dominated by collection 
time. Proposing to use manual memory management instead is admitting 
failure to me.

For a reasonable GC I currently see 2 possible directions:

1. Use a scheme that takes a snapshot of the heap, stack and registers 
at the moment of collection and do the actual collection in another 
thread/process while the application can continue to run. This is the 
way Leandro Lucarellas concurrent GC works 
(http://dconf.org/2013/talks/lucarella.html), but it relies on "fork" 
that doesn't exist on every OS/architecture. A manual copy of the memory 
won't scale to very large memory, though it might be compressed to 
possible pointers. Worst case it will need twice as much memory as the 
current heap.

It would be very interesting how far we can push this model on the 
supported platforms.

2. Change the compiler to emit (library defined) write barriers for 
modifications of (possible) pointers. This will allow to experiment with 
more sophisticated GC algorithms (if the write barrier is written in D, 
we might also need pointers without barriers to implement it). I know 
Walter is against this, and I am also not sure if this adds acceptable 
overhead, but we don't have proof of the opposite, too.

As we all know, the usual eager reference counting with atomic 
operations is not memory-safe, so my current favorite is "concurrent 
buffered reference counting" (see chapter 18.2/3 "The garbage collection 
handbook" by Richard Jones et al): reference count modifications are not 
performed directly by the write barrier, but it just logs the operation 
into a thread local buffer. This is then processed by a collector thread 
which also detects cycles (only on candidates which had their reference 
count decreased during the last cycle). Except for very large reference 
chains this scales with the number of executed pointer modifications and 
not with the heap size.



 May 11 2014



 next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote:

 On 11.05.2014 10:22, Benjamin Thaut wrote:

 Am 10.05.2014 19:54, schrieb Andrei Alexandrescu:

 The next sentence goes on to list the advantages of RC 
 (issues we have
 wrestled with, like destructors), and then goes on to say 
 the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?


 I believe it is.


 While it might be possible to implement a good GC in D it 
 would require
 major changes in the language and its librariers. In my 
 opinion it would
 be way more work to implement a propper GC than to implement 
 ARC.

 Every state of the art GC requires percise knowdelge of _all_ 
 pointers.
 And thats exactly what we currently don't have in D.


 I think most garbage collectors can work with a number of false 
 pointers. The referenced memory area has to be treated as 
 pinned and cannot be moved. Limiting the false pointers to 
 stack and registers seems like a compromise, though most of the 
 stack could even be annotated. Code for this does already exist 
 in the debug info generation, though I suspect stack tracing 
 could be unreliable.

 Here's my current stance on the GC discussions:

 I agree that the current GC is pretty lame, even if it were 
 precise. "Stop-the-World" with complete tracing does not work 
 for any interactive application that uses more than a few 
 hundred MB of garbage collected memory (with or without 
 soft-realtime requirements). Other applications with larger 
 allocation requirements are easily dominated by collection 
 time. Proposing to use manual memory management instead is 
 admitting failure to me.

 For a reasonable GC I currently see 2 possible directions:
Coventry
 1. Use a scheme that takes a snapshot of the heap, stack and 
 registers at the moment of collection and do the actual 
 collection in another thread/process while the application can 
 continue to run. This is the way Leandro Lucarellas concurrent 
 GC works (http://dconf.org/2013/talks/lucarella.html), but it 
 relies on "fork" that doesn't exist on every OS/architecture. A 
 manual copy of the memory won't scale to very large memory, 
 though it might be compressed to possible pointers. Worst case 
 it will need twice as much memory as the current heap.


a) Can't we do our own specialised alternative, instead of doing 
something as heavyweight as fork?

b) Surely that worst case is a very unlikely case, given a 
precise, concurrent garbage collector.

c) Does this actually help in the 99% cpu, 99% memory, 16ms per 
frame setting that Manu talks about?


 It would be very interesting how far we can push this model on 
 the supported platforms.

 2. Change the compiler to emit (library defined) write barriers 
 for modifications of (possible) pointers. This will allow to 
 experiment with more sophisticated GC algorithms (if the write 
 barrier is written in D, we might also need pointers without 
 barriers to implement it). I know Walter is against this, and I 
 am also not sure if this adds acceptable overhead, but we don't 
 have proof of the opposite, too.



Adding optional write-barriers would be great. It would allow us 
to do real experiments and stop the discussion being so 
hypothetical. It doesn't do any harm to add the capability, 
surely?



 As we all know, the usual eager reference counting with atomic 
 operations is not memory-safe, so my current favorite is 
 "concurrent buffered reference counting" (see chapter 18.2/3 
 "The garbage collection handbook" by Richard Jones et al): 
 reference count modifications are not performed directly by the 
 write barrier, but it just logs the operation into a thread 
 local buffer. This is then processed by a collector thread 
 which also detects cycles (only on candidates which had their 
 reference count decreased during the last cycle). Except for 
 very large reference chains this scales with the number of 
 executed pointer modifications and not with the heap size.





 May 11 2014



 next sibling parent  Rainer Schuetze <r.sagitario gmx.de>  writes:


On 11.05.2014 21:16, John Colvin wrote:

 On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote:

 For a reasonable GC I currently see 2 possible directions:
 Coventry
 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the
 memory won't scale to very large memory, though it might be compressed
 to possible pointers. Worst case it will need twice as much memory as
 the current heap.


 a) Can't we do our own specialised alternative, instead of doing
 something as heavyweight as fork?


If we don't want to make a copy with memcpy, we have to rely on the OS 
paging. I don't know linux too well, but on Windows the idea would be to 
use VirtualProtect with PAGE_WRITECOPY. This could allow control on page 
size granularity over what would be mapped to another process.


 b) Surely that worst case is a very unlikely case, given a precise,
 concurrent garbage collector.


Preciseness has little impact because copy-on-write protection works on 
page level. You will also pay for the page fault if you only write 
non-pointer data into the page.

It would help to sort memory chunks without pointers (they have 
attribute NO_SCAN) into separate pages that don't need page protection.


 c) Does this actually help in the 99% cpu, 99% memory, 16ms per frame
 setting that Manu talks about?


You'll need more memory, but given enough idle-time for a lower priority 
collector thread, a higher priority thread could run almost undisturbed. 
IIRC this meant about 40ms, as all threads need to be stopped for the 
time the fork is running.


 It would be very interesting how far we can push this model on the
 supported platforms.

 2. Change the compiler to emit (library defined) write barriers for
 modifications of (possible) pointers. This will allow to experiment
 with more sophisticated GC algorithms (if the write barrier is written
 in D, we might also need pointers without barriers to implement it). I
 know Walter is against this, and I am also not sure if this adds
 acceptable overhead, but we don't have proof of the opposite, too.



 Adding optional write-barriers would be great. It would allow us to do
 real experiments and stop the discussion being so hypothetical. It
 doesn't do any harm to add the capability, surely?


Sure, but it is also not easy ;-) I guess we would need some inspiration 
from existing implementation in other languages how to do it efficiently.



 May 11 2014



prev sibling  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


11-May-2014 23:16, John Colvin пишет:

 On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote:

 On 11.05.2014 10:22, Benjamin Thaut wrote:

 Am 10.05.2014 19:54, schrieb Andrei Alexandrescu:

 The next sentence goes on to list the advantages of RC (issues we have
 wrestled with, like destructors), and then goes on to say the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?


 I believe it is.


 While it might be possible to implement a good GC in D it would require
 major changes in the language and its librariers. In my opinion it would
 be way more work to implement a propper GC than to implement ARC.

 Every state of the art GC requires percise knowdelge of _all_ pointers.
 And thats exactly what we currently don't have in D.


 I think most garbage collectors can work with a number of false
 pointers. The referenced memory area has to be treated as pinned and
 cannot be moved. Limiting the false pointers to stack and registers
 seems like a compromise, though most of the stack could even be
 annotated. Code for this does already exist in the debug info
 generation, though I suspect stack tracing could be unreliable.

 Here's my current stance on the GC discussions:

 I agree that the current GC is pretty lame, even if it were precise.
 "Stop-the-World" with complete tracing does not work for any
 interactive application that uses more than a few hundred MB of
 garbage collected memory (with or without soft-realtime requirements).
 Other applications with larger allocation requirements are easily
 dominated by collection time. Proposing to use manual memory
 management instead is admitting failure to me.

 For a reasonable GC I currently see 2 possible directions:
 Coventry
 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the
 memory won't scale to very large memory, though it might be compressed
 to possible pointers. Worst case it will need twice as much memory as
 the current heap.


 a) Can't we do our own specialised alternative, instead of doing
 something as heavyweight as fork?


Provided heap was allocated as one big mmap-ed region (or File mapping 
in Win32) it should be straightforward to remap it as Copy-on-Write. 
This would though reduce virtual memory available in half (if not more 
provided we'd want to leave some space for C's malloc).

Would be cool to find a way to the allocation of thread stacks... else 
we'd have to copy them while threads are frozen.



-- 
Dmitry Olshansky



 May 11 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 3:42 PM, Dmitry Olshansky wrote:

 Provided heap was allocated as one big mmap-ed region (or File mapping in
Win32)
 it should be straightforward to remap it as Copy-on-Write. This would though
 reduce virtual memory available in half (if not more provided we'd want to
leave
 some space for C's malloc).


This is not an issue for 64 bit machines. I don't think it would be a big 
problem if we have an improved GC for 64 bit machines only.

64 bits have other advantages for a GC - very few false pointers.



 May 11 2014



prev sibling next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 5/11/2014 11:18 AM, Rainer Schuetze wrote:

 1. Use a scheme that takes a snapshot of the heap, stack and registers at the
 moment of collection and do the actual collection in another thread/process
 while the application can continue to run. This is the way Leandro Lucarellas
 concurrent GC works (http://dconf.org/2013/talks/lucarella.html), but it relies
 on "fork" that doesn't exist on every OS/architecture. A manual copy of the
 memory won't scale to very large memory, though it might be compressed to
 possible pointers. Worst case it will need twice as much memory as the current
 heap.

 It would be very interesting how far we can push this model on the supported
 platforms.


This is worth doing even if it is only possible on a subset of platforms. D 
should not be constrained to the worst of all the platforms.

My recollection is that Leandro ran into some serious reentrancy problems in
the 
Linux API. Was this ever resolved?



 May 11 2014



prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote:

 For a reasonable GC I currently see 2 possible directions:

 1. Use a scheme that takes a snapshot of the heap, stack and 
 registers at the moment of collection and do the actual 
 collection in another thread/process while the application can 
 continue to run. This is the way Leandro Lucarellas concurrent 
 GC works (http://dconf.org/2013/talks/lucarella.html), but it 
 relies on "fork" that doesn't exist on every OS/architecture. A 
 manual copy of the memory won't scale to very large memory, 
 though it might be compressed to possible pointers. Worst case 
 it will need twice as much memory as the current heap.

 It would be very interesting how far we can push this model on 
 the supported platforms.

 2. Change the compiler to emit (library defined) write barriers 
 for modifications of (possible) pointers. This will allow to 
 experiment with more sophisticated GC algorithms (if the write 
 barrier is written in D, we might also need pointers without 
 barriers to implement it). I know Walter is against this, and I 
 am also not sure if this adds acceptable overhead, but we don't 
 have proof of the opposite, too.

 As we all know, the usual eager reference counting with atomic 
 operations is not memory-safe, so my current favorite is 
 "concurrent buffered reference counting" (see chapter 18.2/3 
 "The garbage collection handbook" by Richard Jones et al): 
 reference count modifications are not performed directly by the 
 write barrier, but it just logs the operation into a thread 
 local buffer. This is then processed by a collector thread 
 which also detects cycles (only on candidates which had their 
 reference count decreased during the last cycle). Except for 
 very large reference chains this scales with the number of 
 executed pointer modifications and not with the heap size.


I'm surprised that you didn't include:

3. Thread-local GC, isolated zones (restricting where references 
to objects of a particular heap can be placed), exempting certain 
threads from GC completely, ...



 May 12 2014



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 12.05.2014 13:53, "Marc Schütz" <schuetzm gmx.net>" wrote:

 I'm surprised that you didn't include:

 3. Thread-local GC, isolated zones (restricting where references to
 objects of a particular heap can be placed), exempting certain threads
 from GC completely, ...


This comes up from time to time, but to me it is very blurry how this 
can work in reality.

Considering how "shared" is supposed to be used to be useful (do some 
locking, then cast away "shared") there is no guarantee by the language 
that any object is actually thread local (no references from other 
threads). Working with immutable (e.g. strings) is shared by design.



 May 12 2014



 next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:

 On 12.05.2014 13:53, "Marc Schütz" <schuetzm gmx.net>" wrote:

 I'm surprised that you didn't include:

 3. Thread-local GC, isolated zones (restricting where 
 references to
 objects of a particular heap can be placed), exempting certain 
 threads
 from GC completely, ...


 This comes up from time to time, but to me it is very blurry 
 how this can work in reality.

 Considering how "shared" is supposed to be used to be useful 
 (do some locking, then cast away "shared") there is no 
 guarantee by the language that any object is actually thread 
 local (no references from other threads). Working with 
 immutable (e.g. strings) is shared by design.


Yes, but only a part of the data is shared. I suspect the 
majority of the data in typical programs will be thread-local. If 
you use a message passing model, you can improve that even 
further (though it requires a way to move an object to another 
thread's heap). This way, you can - in the best case - avoid the 
shared heap completely.



 May 13 2014



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 13.05.2014 13:09, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:

 On 12.05.2014 13:53, "Marc Schütz" <schuetzm gmx.net>" wrote:

 I'm surprised that you didn't include:

 3. Thread-local GC, isolated zones (restricting where references to
 objects of a particular heap can be placed), exempting certain threads
 from GC completely, ...


 This comes up from time to time, but to me it is very blurry how this
 can work in reality.

 Considering how "shared" is supposed to be used to be useful (do some
 locking, then cast away "shared") there is no guarantee by the
 language that any object is actually thread local (no references from
 other threads). Working with immutable (e.g. strings) is shared by
 design.


 Yes, but only a part of the data is shared. I suspect the majority of
 the data in typical programs will be thread-local. If you use a message
 passing model, you can improve that even further (though it requires a
 way to move an object to another thread's heap). This way, you can - in
 the best case - avoid the shared heap completely.


Possible, but this is with a language without shared data (including 
immutable), not D. Safely transferring a large object tree from one 
thread to another will be very expensive. If you try to optimize that, 
the D compiler won't help you to guarantee memory safety.

Actually I don't have much experience with "shared" (I usually use 
__gshared if I need a shared global). Maybe I understand the idea better 
if you show what happens with this code when run with thread local GC:

class C { C next; }

shared(C) list;

C newC()
{
	return new C; // allocated on the local heap?
}	

void prependToList(C c)
{
	synchronized(list)
	{
		C lst = cast(C) list;
		c.next = lst;  // anything special happening here?
		list = cast(shared(C)) c;  // and here?
	}
}

void callMeFromMultipleThreads()
{
	prependToList(newC());
}



 May 13 2014



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 13 May 2014 at 19:50:52 UTC, Rainer Schuetze wrote:

 On 13.05.2014 13:09, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:

 On 12.05.2014 13:53, "Marc Schütz" <schuetzm gmx.net>" wrote:

 I'm surprised that you didn't include:

 3. Thread-local GC, isolated zones (restricting where 
 references to
 objects of a particular heap can be placed), exempting 
 certain threads
 from GC completely, ...


 This comes up from time to time, but to me it is very blurry 
 how this
 can work in reality.

 Considering how "shared" is supposed to be used to be useful 
 (do some
 locking, then cast away "shared") there is no guarantee by the
 language that any object is actually thread local (no 
 references from
 other threads). Working with immutable (e.g. strings) is 
 shared by
 design.


 Yes, but only a part of the data is shared. I suspect the 
 majority of
 the data in typical programs will be thread-local. If you use 
 a message
 passing model, you can improve that even further (though it 
 requires a
 way to move an object to another thread's heap). This way, you 
 can - in
 the best case - avoid the shared heap completely.


 Possible, but this is with a language without shared data 
 (including immutable), not D. Safely transferring a large 
 object tree from one thread to another will be very expensive. 
 If you try to optimize that, the D compiler won't help you to 
 guarantee memory safety.


It certainly requires a few modifications to the language. That's 
why I think the current discussions about uniqueness, scope, 
isolated are so important, because it would fit together very 
well with memory management concepts like ARC, isolated heaps, 
and safety. For example, if the uniqueness concept that Walter 
recently added to DMD were not only temporary (only used for 
implicit conversion to immutable, AFAIU), but a real, permanent 
type modifier, transferring such a tree can be made safe at 
practically no runtime costs, apart from notifying the GC about 
the changed ownership. (No copying needs to happen, because the 
important thing for thread-local heaps is actually just in which 
thread references to objects can be stored, not necessarily that 
the actual objects are placed into physically distinct heaps.)


 Actually I don't have much experience with "shared" (I usually 
 use __gshared if I need a shared global). Maybe I understand 
 the idea better if you show what happens with this code when 
 run with thread local GC:

 class C { C next; }

 shared(C) list;

 C newC()
 {
 	return new C; // allocated on the local heap?


Yes, because it doesn't say "new shared(C)".


 }	

 void prependToList(C c)
 {
 	synchronized(list)
 	{
 		C lst = cast(C) list;
 		c.next = lst;  // anything special happening here?
 		list = cast(shared(C)) c;  // and here?


The casts are a problem. You're basically lying to the compiler, 
so this cannot work. Instead, it should be something like:

     auto newC() {
         return new isolated(C);
     }

     void prependToList(isolated(C) c) {
         synchronized(list) {
             // transfer to shared heap
             // afterwards, c is no longer usable ("consumed")
             shared(C) tmp = c.makeShared();
             tmp.next = list;
             list = tmp;
         }
     }

This uses the "isolated" concept suggested by deadalnix here:
http://forum.dlang.org/thread/yiwcgyfzfbkzcavuqdwz forum.dlang.org?page=1


 	}
 }

 void callMeFromMultipleThreads()
 {
 	prependToList(newC());
 }





 May 14 2014



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 14.05.2014 12:56, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Tuesday, 13 May 2014 at 19:50:52 UTC, Rainer Schuetze wrote:

 class C { C next; }

 shared(C) list;

 C newC()
 {
     return new C; // allocated on the local heap?


 Yes, because it doesn't say "new shared(C)".


 }

 void prependToList(C c)
 {
     synchronized(list)
     {
         C lst = cast(C) list;
         c.next = lst;  // anything special happening here?
         list = cast(shared(C)) c;  // and here?


 The casts are a problem. You're basically lying to the compiler, so this
 cannot work. Instead, it should be something like:

      auto newC() {
          return new isolated(C);
      }

      void prependToList(isolated(C) c) {
          synchronized(list) {
              // transfer to shared heap
              // afterwards, c is no longer usable ("consumed")
              shared(C) tmp = c.makeShared();
              tmp.next = list;
              list = tmp;
          }
      }



Sorry for the late reply.

I currently don't know of a feasible way to replace casting away 
"shared" if you want to do more than just "tmp.next = list;", for 
example call "bool check(C c);" on tmp or list? I can't implement this 
with a shared object, because any operations on "shared" tend to not 
exist (ok, maybe I could for this very simple example, so consider C 
contains a container to operate on). Also, the object is already locked, 
I should not have to do it again in a function if it only accepts a 
shared C.

How can I prepend something to the list that is not isolated? I guess I 
cannot, as it might create references to same object both shared and 
unshared.

My gut feeling tells me that this ownership handling through the type 
system might work, but it is very restrictive and does not really 
contribute to the first goal of D listed on the front page: "convenience".



 This uses the "isolated" concept suggested by deadalnix here:
 http://forum.dlang.org/thread/yiwcgyfzfbkzcavuqdwz forum.dlang.org?page=1


     }
 }

 void callMeFromMultipleThreads()
 {
     prependToList(newC());
 }







 May 16 2014



 next sibling parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Friday, 16 May 2014 at 16:58:38 UTC, Rainer Schuetze wrote:

 On 14.05.2014 12:56, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Tuesday, 13 May 2014 at 19:50:52 UTC, Rainer Schuetze wrote:

 class C { C next; }

 shared(C) list;

 C newC()
 {
    return new C; // allocated on the local heap?


 Yes, because it doesn't say "new shared(C)".


 }

 void prependToList(C c)
 {
    synchronized(list)
    {
        C lst = cast(C) list;
        c.next = lst;  // anything special happening here?
        list = cast(shared(C)) c;  // and here?


 The casts are a problem. You're basically lying to the 
 compiler, so this
 cannot work. Instead, it should be something like:

     auto newC() {
         return new isolated(C);
     }

     void prependToList(isolated(C) c) {
         synchronized(list) {
             // transfer to shared heap
             // afterwards, c is no longer usable ("consumed")
             shared(C) tmp = c.makeShared();
             tmp.next = list;
             list = tmp;
         }
     }



 Sorry for the late reply.

 I currently don't know of a feasible way to replace casting 
 away "shared" if you want to do more than just "tmp.next = 
 list;", for example call "bool check(C c);" on tmp or list? I 
 can't implement this with a shared object, because any 
 operations on "shared" tend to not exist (ok, maybe I could for 
 this very simple example, so consider C contains a container to 
 operate on). Also, the object is already locked, I should not 
 have to do it again in a function if it only accepts a shared C.

 How can I prepend something to the list that is not isolated? I 
 guess I cannot, as it might create references to same object 
 both shared and unshared.

 My gut feeling tells me that this ownership handling through 
 the type system might work, but it is very restrictive and does 
 not really contribute to the first goal of D listed on the 
 front page: "convenience".


I haven't thought it through completely, of course, but I believe 
it doesn't have to be inconvenient. Maybe there is a nice 
solution. I already have something in mind, but I have to think 
it over first.



 May 17 2014



prev sibling  parent  Jerry <jlquinn optonline.net>  writes:


Rainer Schuetze <r.sagitario gmx.de> writes:


 On 14.05.2014 12:56, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Tuesday, 13 May 2014 at 19:50:52 UTC, Rainer Schuetze wrote:

 class C { C next; }

 shared(C) list;

 C newC()
 {
     return new C; // allocated on the local heap?


 Yes, because it doesn't say "new shared(C)".


 }

 void prependToList(C c)
 {
     synchronized(list)
     {
         C lst = cast(C) list;
         c.next = lst;  // anything special happening here?
         list = cast(shared(C)) c;  // and here?


 The casts are a problem. You're basically lying to the compiler, so this
 cannot work. Instead, it should be something like:

      auto newC() {
          return new isolated(C);
      }

      void prependToList(isolated(C) c) {
          synchronized(list) {
              // transfer to shared heap
              // afterwards, c is no longer usable ("consumed")
              shared(C) tmp = c.makeShared();
              tmp.next = list;
              list = tmp;
          }
      }



 Sorry for the late reply.

 I currently don't know of a feasible way to replace casting away "shared" if
 you want to do more than just "tmp.next = list;", for example call "bool
 check(C c);" on tmp or list? I can't implement this with a shared object,
 because any operations on "shared" tend to not exist (ok, maybe I could for
 this very simple example, so consider C contains a container to operate
 on). Also, the object is already locked, I should not have to do it again in a
 function if it only accepts a shared C.

 How can I prepend something to the list that is not isolated? I guess I
 cannot, as it might create references to same object both shared and unshared.

 My gut feeling tells me that this ownership handling through the type system
 might work, but it is very restrictive and does not really contribute to the
 first goal of D listed on the front page: "convenience".


Reading this, I had a thought.  What if synchronized temporarily
stripped the shared qualifier from list?  At the end of the synchronized
block, shared is restored.  Then:

void prependToList(C c) {
    synchronized(list) {       // list is now C, not shared(C)
        c.next = list;         // ok
        list = c;              // ok
        check(list);           // ok, not a shared object here
    }
    // list is now shared again.  move list to shared heap if not already
}

I'm sure I'm missing something, but using shared within a synchronized
block might be saner if this worked...

Jerry



 May 17 2014



prev sibling  parent reply "Wyatt" <wyatt.epp gmail.com>  writes:


On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:

 This comes up from time to time, but to me it is very blurry 
 how this can work in reality.


The paper I linked on Friday [0] presents a collector like this.  
Are there concerns I've missed that make that not applicable?


 Considering how "shared" is supposed to be used to be useful 
 (do some locking, then cast away "shared") there is no 
 guarantee by the language that any object is actually thread 
 local (no references from other threads). Working with 
 immutable (e.g. strings) is shared by design.


I'm not seeing much in the documentation, but from what I can 
tell (per the FAQ), shared in D just guarantees it's on the 
global heap?

-Wyatt

[0] 
https://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf



 May 13 2014



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 13.05.2014 14:20, Wyatt wrote:

 On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:

 This comes up from time to time, but to me it is very blurry how this
 can work in reality.


 The paper I linked on Friday [0] presents a collector like this. Are
 there concerns I've missed that make that not applicable?


I just read the first chapters, and according to that, existing local 
gcs needs write barriers, so we are back to my second proposal. The 
implementation in the paper even adds read barriers.


 Considering how "shared" is supposed to be used to be useful (do some
 locking, then cast away "shared") there is no guarantee by the
 language that any object is actually thread local (no references from
 other threads). Working with immutable (e.g. strings) is shared by
 design.


 I'm not seeing much in the documentation, but from what I can tell (per
 the FAQ), shared in D just guarantees it's on the global heap?


To me, shared is a type modifier that doesn't implicitely convert to 
anything else without casting.
If a global/static variable is shared, it is placed into the data 
segment of the executable image, not TLS. The heap is not involved here.



 -Wyatt

 [0]
 https://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf





 May 13 2014



  parent reply "Wyatt" <wyatt.epp gmail.com>  writes:


On Tuesday, 13 May 2014 at 19:56:20 UTC, Rainer Schuetze wrote:

 I just read the first chapters, and according to that, existing 
 local gcs needs write barriers, so we are back to my second 
 proposal. The implementation in the paper even adds read 
 barriers.


At this point, I suspect write barriers are simply a requirement 
for any modern scalable GC.  At the very least, the algorithms 
they enable are important if D is really committed to the 
pluggable GC concept.  (Given the ongoing discussion on 
performance tradeoffs of various GC methods and the needs of 
different groups, I'd even suggest it's a really important 
feature.)


 To me, shared is a type modifier that doesn't implicitely 
 convert to anything else without casting.


Interesting, maybe this should be clarified better.  Having 
skimmed back over chapter 13 of TDPL, my understanding of its 
semantics are that it only really enforces atomicity and 
execution order.  Also, this bit from near the beginning of 13.12 
states:
"For all numeric types and function pointers, shared-qualified 
values are convertible implicitly to and from unqualified values."

That sounds kind of at-odds with your interpretation...? :/

-Wyatt



 May 14 2014



  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 05/14/2014 06:41 PM, Wyatt wrote:

 To me, shared is a type modifier that doesn't implicitely convert to
 anything else without casting.


 Interesting, maybe this should be clarified better.  Having skimmed back
 over chapter 13 of TDPL, my understanding of its semantics are that it
 only really enforces atomicity and execution order.  Also, this bit from
 near the beginning of 13.12 states:
 "For all numeric types and function pointers, shared-qualified values
 are convertible implicitly to and from unqualified values."

 That sounds kind of at-odds with your interpretation...? :/


Not too much. Those are trivial cases (one copies the entire qualified 
data, so obviously one is free to change qualifiers as one wishes).



 May 14 2014



prev sibling  parent reply Martin Nowak <code dawg.eu>  writes:


On 05/11/2014 08:18 PM, Rainer Schuetze wrote:

 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the memory
 won't scale to very large memory, though it might be compressed to
 possible pointers. Worst case it will need twice as much memory as the
 current heap.


There is a problem with this scheme, copy-on-write is extremely 
expensive when a mutation happens. That's one page fault (context 
switch) + copying a whole page + mapping the new page. It's much worse 
with huge pages (2MB page size).



 May 12 2014



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 13.05.2014 00:15, Martin Nowak wrote:

 On 05/11/2014 08:18 PM, Rainer Schuetze wrote:

 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the memory
 won't scale to very large memory, though it might be compressed to
 possible pointers. Worst case it will need twice as much memory as the
 current heap.


 There is a problem with this scheme, copy-on-write is extremely
 expensive when a mutation happens. That's one page fault (context
 switch) + copying a whole page + mapping the new page.


I agree that this might be critical, but it is a one time cost per page. 
It seems unrealistic to do this with user mode exceptions, but the OS 
should have this optimized pretty well.


 It's much worse
 with huge pages (2MB page size).


How common are huge pages nowadays?



 May 12 2014



 next sibling parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 13 May 2014 at 06:12:46 UTC, Rainer Schuetze wrote:

 On 13.05.2014 00:15, Martin Nowak wrote:



 There is a problem with this scheme, copy-on-write is extremely
 expensive when a mutation happens. That's one page fault 
 (context
 switch) + copying a whole page + mapping the new page.


 I agree that this might be critical, but it is a one time cost 
 per page. It seems unrealistic to do this with user mode 
 exceptions, but the OS should have this optimized pretty well.


As I pointed out this won't help dynamic games that easily can 
touch 50000 pages per frame if you use a single global allocator. 
2000 cycles * 50K = 100K = frame drop. What's worse, if you are 
low on memory you will start to swap to disk (or compress pages).

So that means you have to optimize for collections, use dedicated 
allocators that keep dynamic data on the same pages etc... 
Basically you get the disadvantage of manual memory management 
and in the worst case the memory requirements of a copying GC 
without the benefits...



 May 13 2014



prev sibling  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Tue, 13 May 2014 02:12:44 -0400, Rainer Schuetze <r.sagitario gmx.de>  
wrote:


 On 13.05.2014 00:15, Martin Nowak wrote:

 On 05/11/2014 08:18 PM, Rainer Schuetze wrote:

 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the  
 memory
 won't scale to very large memory, though it might be compressed to
 possible pointers. Worst case it will need twice as much memory as the
 current heap.


 There is a problem with this scheme, copy-on-write is extremely
 expensive when a mutation happens. That's one page fault (context
 switch) + copying a whole page + mapping the new page.


 I agree that this might be critical, but it is a one time cost per page.  
 It seems unrealistic to do this with user mode exceptions, but the OS  
 should have this optimized pretty well.

  > It's much worse
  > with huge pages (2MB page size).

 How common are huge pages nowadays?


I know this is coming from a position of extreme ignorance, but why do we  
have to do copy on write? What about pause on write? In other words, if a  
thread tries to write to a page that's being used by the collector, it  
pauses the thread until the page is no longer being used by the GC.

This doesn't fix everything, but it's at least as good as today's GC,  
which preemptively pauses threads.

My ignorance is that I have no idea if this is even possible, and I also  
have no idea how it would affect performance.

-Steve



 May 13 2014



  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 13 May 2014 at 14:13:31 UTC, Steven Schveighoffer 
wrote:

 I know this is coming from a position of extreme ignorance, but 
 why do we have to do copy on write? What about pause on write?


Not sure how that will help? Pointers may still escape collection?

(but you get that with transactional memory, on the cache level)



 May 13 2014



 next sibling parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 13 May 2014 at 16:14:07 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 13 May 2014 at 14:13:31 UTC, Steven Schveighoffer 
 wrote:
 Not sure how that will help? Pointers may still escape 
 collection?


(as in not being traced, leading to freeing live memory)



 May 13 2014



prev sibling  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Tue, 13 May 2014 12:14:06 -0400, Ola Fosheim Gr=C3=B8stad  =

<ola.fosheim.grostad+dlang gmail.com> wrote:


 On Tuesday, 13 May 2014 at 14:13:31 UTC, Steven Schveighoffer wrote:

 I know this is coming from a position of extreme ignorance, but why d=




o  =


 we have to do copy on write? What about pause on write?


 Not sure how that will help? Pointers may still escape collection?

 (but you get that with transactional memory, on the cache level)


My understanding is that the way the fork collector works is that it mak=
es  =

pages copy-on-write. Then if the original process writes to one of the  =

pages, the page is copied, which may be expensive in terms of total memo=
ry  =

and time consumed.

The idea I had was to make them pause-on-write. This means, when the  =

original process attempts to write to the page, it gets a page-fault,  =

which pauses the thread until the collector is done with it. This causes=
  =

the same halting that normally happens with stop-the-world, but only  =

on-demand, instead of preemptively. If a thread is doing only reads, or =
is  =

only touching non-Scanned memory, it continues.

A collector may be able to take advantage of this knowledge to avoid as =
 =

many pauses as possible, but I'm not sure.

Just an idea, I make no claims to its actual benefits :)

-Steve



 May 13 2014



 next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com>  writes:


"Steven Schveighoffer"  wrote in message 
news:op.xfs6jhp3eav7ka stevens-macbook-pro-2.local...


 If a thread is doing only reads, or is  only touching non-Scanned memory, 
 it continues.


How long do threads usually go without touching the stack? 



 May 13 2014



  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Tue, 13 May 2014 13:36:17 -0400, Daniel Murphy  
<yebbliesnospam gmail.com> wrote:


 "Steven Schveighoffer"  wrote in message  
 news:op.xfs6jhp3eav7ka stevens-macbook-pro-2.local...


 If a thread is doing only reads, or is  only touching non-Scanned  
 memory, it continues.


 How long do threads usually go without touching the stack?


The stack would have to be COW.

-Steve



 May 13 2014



 next sibling parent  "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Tue, 13 May 2014 13:37:05 -0400, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 On Tue, 13 May 2014 13:36:17 -0400, Daniel Murphy  
 <yebbliesnospam gmail.com> wrote:


 "Steven Schveighoffer"  wrote in message  
 news:op.xfs6jhp3eav7ka stevens-macbook-pro-2.local...


 If a thread is doing only reads, or is  only touching non-Scanned  
 memory, it continues.


 How long do threads usually go without touching the stack?


 The stack would have to be COW.


No sorry, the stack only contains roots, so it can be marked as "safe to  
use" pretty quickly.

-Steve



 May 13 2014



prev sibling  parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 13 May 2014 at 17:37:05 UTC, Steven Schveighoffer 
wrote:

 On Tue, 13 May 2014 13:36:17 -0400, Daniel Murphy 
 <yebbliesnospam gmail.com> wrote:


 "Steven Schveighoffer"  wrote in message 
 news:op.xfs6jhp3eav7ka stevens-macbook-pro-2.local...


 If a thread is doing only reads, or is  only touching 
 non-Scanned memory, it continues.


 How long do threads usually go without touching the stack?


 The stack would have to be COW.


When you fork a process, all of its pages are COW. What you 
propose is much more complicated. You would either first need to 
fork(), then somehow remap certain pages as shared. The remapping 
would have to happen in the parent process. I don't know whether 
there are even interfaces to do this.



 May 13 2014



prev sibling  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 13 May 2014 at 17:22:19 UTC, Steven Schveighoffer 
wrote:

 The idea I had was to make them pause-on-write. This means, 
 when the original process attempts to write to the page, it 
 gets a page-fault, which pauses the thread until the collector 
 is done with it. This causes the same halting that normally 
 happens with stop-the-world, but only on-demand, instead of 
 preemptively. If a thread is doing only reads, or is only 
 touching non-Scanned memory, it continues.


That would require a separate page table and would freeze threads 
real fast for global GC. It can work for local heaps like caches? 
I guess you could set up shared memory between two processes and 
use that as your heap.


 Just an idea, I make no claims to its actual benefits :)


Another idea is to use transactions and roll back the collector 
if someone does a write, for small heaps with few writes... 
Principle: run the collector when a core is idle and yield to 
cores that do real work.

But I dont think current gen CPUs can handle big transactions...



 May 13 2014



prev sibling  parent  Mike Parker <aldacron gmail.com>  writes:


On 5/10/2014 6:05 AM, Wyatt wrote:

 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:

 Let's also bear in mind that Java's GC is worlds ahead of D's.


 Is Sun/Oracle reference implementation actually any good?


Yes. Given all the man hours that have gone into it over the years, it 
would be surprising if it weren't. Actually, though, there isn't really 
one collector that ships with the JRE anymore. There are different 
implementations, each using a different strategy, that can be selected 
at JVM startup. Furthermore, each collector is tunable so if you aren't 
satisfied with the default collector out of the box, you can use the 
JVM's instrumentation to find the most optimal match for your app's 
usage patterns. See [1] for some details.

[1] http://www.infoq.com/articles/Java_Garbage_Collection_Distilled



 May 10 2014



prev sibling  parent  "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 6 May 2014 at 22:07:15 UTC, Xavier Bigand wrote:

 Le 06/05/2014 13:39, Paulo Pinto a écrit :

 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d 
 wrote:

 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link 
 or describe
 me completely state and problems of current garbage 
 collection and
 other
 resource management? It help me in finding of existence 
 solution (at
 least theoretical).



 The major issue with the garbage collector is that it's not
 guaranteed to
 run a collection. When a collection is run the GC will call 
 the
 destructors
 for the objects it collects. If there's no guarantee a 
 collection is run
 there can be no guarantee that destructors are called. A 
 collection is
 usually run when allocating new memory and there's not 
 enough memory
 available.


 I think it's also an important consideration that GC is 
 incompatible
 with low-memory and real-time environments.

 ...


 I guess outside the gaming world, embedded and real-time seem 
 to be
 getting lots of Java and .NET love:

 https://www.aicas.com/cms/


 Is that VM full compatible with Java specifications and 
 standard frameworks?



Yes. That is the point of having a Java VM certification process.

It still baffles me that many aren't aware Sun/Oracle JVM is 
*only* the reference implementation.


 http://www.is2t.com/products/

 http://www.mountaineer.org/netmf-for-stm32/

 Just a small sample of the partners providing the said support.


 --
 Paulo



 Android works well, I love my nexus, it proves to me that it's 
 possible to create really smooth applications based completely 
 on Java (not 100% of that) but if we compare the Nexus 5 to 
 iPhone 4 :
 Memory : 2 GB RAM vs 512 MB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8
 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery

 And compared to an iPhone 5s
 Memory : 2 GB RAM vs 1 GB RAM
 CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone
 Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery

 It's maybe not really significant but the majority of Android 
 devices that have acceptable performances have a lot of memory, 
 a quad cores CPU and an heavy battery.

 So that cool Java can run smoothly but at which price? I think 
 the margin of Apple produce is unbelievable.


MicroEJ VMs can run in ARM Cortex-M systems. Do you know what 
that means?

A VM taking 28 Kbytes of flash, less than 1.5 Kbytes RAM with a 
boot time of just 2 ms, with a 120 MHz CPU.

Android Dalvik VM sucks. It has not been improved since Android 
2.3, other than additional tweaks on 4.3, and Google only did 
improve it to get it to a good enough state.

Hopefully ART will fix this.

--
Paulo



 May 07 2014



prev sibling  parent  Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 06/05/2014 12:58, Manu via Digitalmars-d a écrit :

 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).



 The major issue with the garbage collector is that it's not guaranteed to
 run a collection. When a collection is run the GC will call the destructors
 for the objects it collects. If there's no guarantee a collection is run
 there can be no guarantee that destructors are called. A collection is
 usually run when allocating new memory and there's not enough memory
 available.


 I think it's also an important consideration that GC is incompatible
 with low-memory and real-time environments.

 The trouble is, GC runs more frequently as the amount of available
 free memory decreases, and trace&collect takes a long time, long
 enough that realtime software typically can't tolerate the pause. If
 the pause is super rare (minutes/hours apart), maybe it's acceptable,
 but if it's closer to every call to alloc (ie, when there is little/no
 free memory), then you realise it's fundamentally incompatible.
 Then consider that this is typically the default operating state of
 embedded systems; to have no free memory at runtime, and therefore D
 is effectively incompatible with a subset of computers...
 specifically, games consoles; a gigantic industry, and one of the few
 that still absolutely relies on native code where D is particularly
 compelling.

 If people want to call D a native systems language, this seems like a
 serious conflict with that idea. A major subset of the language is
 incompatible with realtime or embedded use (surely among the most
 compelling use cases remaining for native code these days!). Usually
 the argument is that I represent a niche, and therefore I should
 accept the restrictions of my environment, and revert to full manual
 memory management for my work, while letting everyone else have their
 fun.
 That is a broken argument, not only because it leaves very little
 motivation to leave C++, but also because regardless of whether I
 adopt such restrictions, library authors won't, and we don't live in
 the 1980's where I can write a few lines of self-contained assembly
 and call it a program. To say I can't depend on libraries is
 practically absurd. Games are gigantic software projects, and depend
 on heaps of libraries.

 The GC effectively eliminates libraries from embedded/realtime users,
 which I see as an absolute deal-breaker alone, irrespective of a
 further bunch more restrictions it places on that subset of users
 within their own code :/

 I'll be happy with any solution that works, really. But as I see it,
 reference counting is the only garbage collection technology I know
 which will reliably clean up eagerly as things fall out of scope (also
 addressing this issue with destructors as bonus), and as such, it
 seems the logical solution. ARC is a demonstrable success in Obj-C,
 and works well in realtime and embedded systems. D has type system
 improvements which would theoretically allow for significant
 improvements over Obj-C.


+1

D seems to mainly interests C++ developers that generally aren't 
interested by dynamically types language and based on a GC.
But we always have the possibility to link C++ to a higher language for 
some part of applications.
IMO system languages have to continue working with manual memory management.



 May 06 2014



prev sibling  parent  Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 06/05/2014 08:33, Jacob Carlborg a écrit :

 On 06/05/14 08:07, HaraldZealot wrote:


 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).


 The major issue with the garbage collector is that it's not guaranteed
 to run a collection. When a collection is run the GC will call the
 destructors for the objects it collects. If there's no guarantee a
 collection is run there can be no guarantee that destructors are called.
 A collection is usually run when allocating new memory and there's not
 enough memory available.


For me that working almost every days on mobile devices, that not a good 
thing to release the memory when there is some new allocations.

Took an application with the notion of projects, here is a classical 
sequence users will do :
  1) Launch the application
  2) Open a project (heavy load on memory)
  3) Do some modifications (adding few allocations)
  4) Switch to the an other project

Please consider those kind of restrictions (from iOS and/or Android) :
  - After memory warning if you don't release memory enough fast your 
application will be killed
  - If you lock the main thread for too much time your application will 
be killed

Now how we implement transition between step 3 and 4 :
  - Saving the current project
  - Release manually all his memory and other resources can't be easily 
shared
  - At this point we are almost sure that memory and resources 
footprints are to the minimum amount (except leaks)
  - We can load the new project and allocate memory safely

Now with the GC, we tend to saturate the memory of the device which will 
cause memory warning with a chance of crash of the application. It's 
slower than manual release cause of number of scans,...
Even on devices like PC, GC's when they don't share memory pool between 
applications tends to reduce resources for other applications (bad for 
multitask OS).
So with a GC for this kind of steps we need shutdown the GC, and force 
the collect where it's interesting.

IMO, manual management is not the best, GC neither, but with good tools 
memory issues can be easy to find and fix. Apple have great tools for 
that on iOS, their memory analyser find leaks without any performance 
slow down just like if it's supported by the hardware!!!



 May 06 2014



prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 6 May 2014 at 06:07:41 UTC, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link or 
 describe me completely state and problems of current garbage 
 collection and other resource management? It help me in finding 
 of existence solution (at least theoretical).


A precise scanning GC is the only robust general solution. RC 
with weak pointers can only account for a subset of all possible 
models.

But I agree with you, the language should be redesigned to have a 
GC friendly set of D constructs where FFI is followed by 
programmer guaranteed postconditions (specified by library 
authors).

In other words the  nogc appoach is not sufficient, a  gc 
approach is needed.



 May 05 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 6 May 2014 at 06:39:19 UTC, Ola Fosheim Grøstad wrote:

 a GC friendly set of D constructs where FFI is followed by 
 programmer guaranteed postconditions (specified by library 
 authors).


Btw "postcondition" is the wrong term. In order to be robust need 
to specify how far into the GC heap a FFI can go while it is 
live. Consider callbacks from C to D. You don't know the layout 
of the stack frame, so you have to scan the stack assuming 
anything unknown is a pointer.

But that is not sufficient, because the C code may be holding 
references to the GC heap in a malloc'ed struct. So, you have to 
scan all reachable malloc'ed memory too.

At the end of the day some discipline is required for system 
level programming because the robust solutions are very expensive.



 May 06 2014



prev sibling next sibling parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 6 May 2014 at 06:07:41 UTC, HaraldZealot wrote:

 I have to say that all this discussion (more precisely the 
 understanding on the side of key developers) make me very upset.

 It's good that Andrei agreed with impossibility of the 
 harebrained disallowing of the class destructors. But I was 
 very surprise, that so thought go to such head, because such 
 solution contradicts the D spirit totally. There are many 
 language which are very popular and have many dark moments in 
 their design. I (and I think not only me) go to the D not for 
 its popularity, but for its clarity, power and SANITY (that 
 bases on strong guaranties). The strong solutions found on the 
 strong decision makes D itself. (Examples of such strong 
 solutions: immutabilities, default unshareness, struct and 
 class as distinct being). And way that leads us in state where 
 stucts have dtors and classes haven't but allow struct with 
 dtor as member and people have to call struct dtor manually, 
 isn't D way. Because such way relies on programmers discipline, 
 but how Andrei has written "If there one thing that decades of 
 computing have taught us, it must be that discipline-oriented 
 programming does not scale."[TDPL, p. 399].


Actually, IMO the clarity and sanity you talk about are a strong 
argument _for_ the deprecation of destructors for GC objects:

Premise 1: Destructors are for things that _must_ be done when an 
object is destroyed. Destroying (or leaking) an object without 
calling its destructor can (and often will) lead to wrong program 
behavior and is therefore an error.

Premise 2: Tracing GCs are by their nature non-deterministic and 
can not guarantee proper destruction of objects. This is true 
even if full type information about the managed objects is 
available (which is currently the case only for classes), as long 
as they are non-precise (which, realistically, in a systems 
programming language they will always be at least to some degree).

Conclusion: It's an error to have objects with destructors be 
managed by the GC. Per Andrei's quote about "discipline-oriented 
programming", this should therefore be made impossible.

(Apart from that, there are tons of things that you aren't 
allowed to do in a destructor called from the current GC, but 
there's currently no way to detect them statically => again 
"discipline-oriented programming".)

Of course, how exactly this is achieved is up for debate. Also, 
maybe one of the premises is false...



 May 06 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/5/14, 11:07 PM, HaraldZealot wrote:

 I have to say that all this discussion (more precisely the understanding
 on the side of key developers) make me very upset.


This is a misunderstanding of the situation. This is brainstorming. 
There has to be a public place in which ideas can be discussed freely, 
no matter how radical. Otherwise we'd need to create a cabal and only 
present finalized decisions to the public. We can't get to a community 
censorship so strong so as to disallow unencumbered discussion of 
radical ideas. All those who got up in arms about this thread, I compel 
you to reconsider.

Andrei



 May 06 2014



 next sibling parent  "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Tuesday, 6 May 2014 at 14:29:01 UTC, Andrei Alexandrescu wrote:

 On 5/5/14, 11:07 PM, HaraldZealot wrote:

 I have to say that all this discussion (more precisely the 
 understanding
 on the side of key developers) make me very upset.


 This is a misunderstanding of the situation. This is 
 brainstorming. There has to be a public place in which ideas 
 can be discussed freely, no matter how radical. Otherwise we'd 
 need to create a cabal and only present finalized decisions to 
 the public. We can't get to a community censorship so strong so 
 as to disallow unencumbered discussion of radical ideas. All 
 those who got up in arms about this thread, I compel you to 
 reconsider.

 Andrei


+1

with(devilsAdvocateMode)
{
    I think people are a bit used to the situation you describe,
    hence the misunderstanding w.r.t. the approach you're trying
    here
}



 May 06 2014



prev sibling  parent reply "HaraldZealot" <harald_zealot tut.by>  writes:


 This is a misunderstanding of the situation. This is 
 brainstorming. There has to be a public place in which ideas 
 can be discussed freely, no matter how radical.


This is another misunderstanding :)). Not radicalism upset me, 
but proposal to create one more hole instead exist hole from man, 
that touched me solidness in code by his book. It were 
surprising. But it's only emotion.



 May 06 2014



 next sibling parent  "HaraldZealot" <harald_zealot tut.by>  writes:


 This is another misunderstanding :)). Not radicalism upset me, 
 but proposal to create one more hole instead exist hole from 
 man, that touched me solidness in code by his book. It were 
 surprising. But it's only emotion.


But yes, It may possible as troll mode in brainstorming.



 May 06 2014



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/6/14, 8:19 AM, HaraldZealot wrote:

 This is a misunderstanding of the situation. This is brainstorming.
 There has to be a public place in which ideas can be discussed freely,
 no matter how radical.


 This is another misunderstanding :)). Not radicalism upset me, but
 proposal to create one more hole instead exist hole from man, that
 touched me solidness in code by his book. It were surprising. But it's
 only emotion.


Leave emotion aside and use reasoning. -- Andrei



 May 06 2014



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 06/05/14 05:51, HaraldZealot wrote:


 Manu, can you direct me what is ARC? This abbreviation is very
 misgooglly.


Automatic Reference Counting. Like regular RC but the compiler 
automatically inserts calls to release/free.

-- 
/Jacob Carlborg



 May 05 2014



  parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 6 May 2014 16:28, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 06/05/14 05:51, HaraldZealot wrote:


 Manu, can you direct me what is ARC? This abbreviation is very
 misgooglly.



 Automatic Reference Counting. Like regular RC but the compiler automatically
 inserts calls to release/free.


And further, more importantly, automatically *eliminates* redundant
calls to add/dec ref, which I think has much greater potential in D's
type system than in Obj-C.



 May 05 2014



prev sibling next sibling parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 6 May 2014 at 03:40:47 UTC, Manu via Digitalmars-d
wrote:

 On 3 May 2014 18:49, Benjamin Thaut via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:

 Walter and I have had a long chat in which we figured our 
 current
 offering of abstractions could be improved. Here are some 
 thoughts.
 There's a lot of work ahead of us on that and I wanted to 
 make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls 
 from within
 the GC entirely. It makes for a faster and better GC, but the 
 real
 reason here is that destructors are philosophically bankrupt 
 in a GC
 environment. I think there's no need to argue that in this 
 community.

 The GC never guarantees calling destructors even today, so 
 this decision
 would be just a point in the definition space (albeit an 
 extreme one).

 That means classes that need cleanup (either directly or by 
 having
 fields that are structs with destructors) would need to 
 garner that by
 other means, such as reference counting or manual. We're 
 considering
 deprecating ~this() for classes in the future.

 Also, we're considering a revamp of built-in slices, as 
 follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently 
 lowered into
 RCSlice!T, defined inside object.d. That type would occupy 
 THREE words,
 one of which being a pointer to a reference count. That type 
 would
 redefine all slice primitives to update the reference count 
 accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit 
 cast(void[])
 will be allowed, and will ignore the reference count (so if a 
 void[]
 extracted from a T[] via a cast outlives all slices, dangling 
 pointers
 will ensue).

 I foresee any number of theoretical and practical issues with 
 this
 approach. Let's discuss some of them here.


 Thanks,

 Andrei



 Honestly, that sounds like the entierly wrong apporach to me. 
 Your
 approaching the problem in this way:

 "We can not implement a propper GC in D because the language 
 design prevents
 us from doing so. So lets remove destructors to migate the 
 issue of false
 pointers."

 While the approach should be.

 "The language does not allow to implement a propper GC 
 (anything else then
 dirty mark & sweep), what needs to be changed to allow a 
 implementation of a
 more sophisticated GC."


 Couldn't agree more.
 Abandoning destructors is a disaster.
 Without destructors, you effectively have manual memory 
 management, or
 rather, manual 'resource' management, which is basically the 
 same
 thing, even if you have a GC.
 It totally undermines the point of memory management as a 
 foundational
 element of the language if most things are to require manual
 release/finalisation/destruction or whatever you wanna call it.




 which heavily
 relies on resource management. So basically every class in 
 there inherits

 the finalizer

 Basically the
 entire codebase feels like manual memory management. You have 
 to think about
 manually destroying every class and the entire advantage of 
 having a GC,
 e.g. not having to think about memory management and thus 
 beeing more

 syntax. Do we
 really want that for D?


 This is interesting to hear someone else say this. I have 
 always found

 memory
 management in practise too.
 I've ranted enough about it already, but I have come to the firm
 conclusion that the entire premise of a mark&sweep GC is 
 practically
 corrupt. Especially in D.

 experience that
 absolutely parallels your example, I realise that GC's failure 
 extends
 into far more cases than just the ones I'm usually representing.

 I also maintain that GC isn't future-proof in essence. 
 Computers grow
 exponentially, and GC performance inversely tracks the volume of
 memory in the system. Anything with an exponential growth curve 
 is
 fundamentally not future-proof.
 I predict a 2025 Wikipedia entry: "GC was a cute idea that 
 existed for
 a few years in the early 2000's while memory ranged in the 
 100's mb -
 few gb's, but quickly became unsustainable as computer 
 technology
 advanced".


Java Azul VM GC was already handling 1 TB in 2010.

http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf

GC is not the only way of doing automatic memory management, but
this ongoing discussion steams more from D's current GC status
and respective phobia in C world, and less from what a modern GC
is capable of.

--
Paulo



 May 06 2014



 next sibling parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Paulo Pinto:


 Java Azul VM GC was already handling 1 TB in 2010.

 http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf

 GC is not the only way of doing automatic memory management, but
 this ongoing discussion steams more from D's current GC status
 and respective phobia in C world, and less from what a modern GC
 is capable of.


On the other hand I don't think you will have available in the D 
world the amount of work done on the Azul GC. This is the problem 
with looking at Java.

Bye,
bearophile



 May 06 2014



  parent  "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 6 May 2014 at 07:34:13 UTC, bearophile wrote:

 Paulo Pinto:


 Java Azul VM GC was already handling 1 TB in 2010.

 http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf

 GC is not the only way of doing automatic memory management, 
 but
 this ongoing discussion steams more from D's current GC status
 and respective phobia in C world, and less from what a modern 
 GC
 is capable of.


 On the other hand I don't think you will have available in the 
 D world the amount of work done on the Azul GC. This is the 
 problem with looking at Java.

 Bye,
 bearophile


Hence my remark that this whole discussion is more about finding 
working alternatives in the D world and less about what 
industrial class GCs are capable of.

I was just doing a heads up, as people tend to misunderstand GCs.

--
Paulo



 May 06 2014



prev sibling  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 6 May 2014 17:16, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 6 May 2014 at 03:40:47 UTC, Manu via Digitalmars-d
 wrote:
 Java Azul VM GC was already handling 1 TB in 2010.


Is D compatible with this GC? And how does it go in an environment
with 128mb of ram, and no free memory?



 http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf

 GC is not the only way of doing automatic memory management, but
 this ongoing discussion steams more from D's current GC status
 and respective phobia in C world, and less from what a modern GC
 is capable of.


It's not a 'phobia', it's an incompatibility. Show me a 'modern' GC
that's compatible with realtime+embedded, and also compatible with D,
and then we can talk... I'm sure there's a very good reason nobody has
raised such a technology for consideration.
Framing the C world in this way is completely unfair. I've spent years
here trying to reconcile my relationship with the GC. I've actually
defended the GC on numerous occasions. I don't want to reject the GC,
it's very convenient... but I think it's reached the point where I
have to face the music. I want to know the plan, and if it remains
incompatible with my industry, then I can't reasonably continue to
advocate D in my field, can I?

If it were my language, I know what I'd do. It may require some
radical changes, and I would do it because I simply had _no other
choice_. Please, make suggestions! It's been years, and I'm still
waiting to hear another realistic proposal.



 May 06 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 6 May 2014 at 11:35:53 UTC, Manu via Digitalmars-d 
wrote:

 Please, make suggestions! It's been years, and I'm still 
 waiting to hear another realistic proposal.


1. Use  nogc and C++ style allocation for hard real time threads 
(render logic).

2. Use segmented GC for soft real time threads (world logic).

Lobby for more restrictions/better runtime support from the 
compiler for fast precise single-pass regional  (local) garbage 
collection. As a bonus you will get faster exception handling too 
if identifying stack frames is faster.

Lobby for restrictions:

- C may only call back to  nogc D-functions so you can collect 
without C-frames on the stack.

- Allow  omitting framepointers, global call optimization and 
stack optimizations for  nogc  nothrow threads.

etc.

The key to performance is to allow specifying constraints that 
the compiler and runtime can assume to hold.

The problem with the D community is a lack of priority of system 
level programming which is at odds with "safe", "no discipline 


No constraints == no performance.



 May 06 2014



prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 06.05.2014 05:40, schrieb Manu via Digitalmars-d:

 Does ~this() actually work, or just usually work?



Back when I actually used D's GC, it did usually work. There were a few 
workarounds because destructors would be called in the wrong thread, but 
other than that it did work for me.

By now I call my destructors manually like in C++. I do full manual 
memory management (I have a modified version of druntime that does not 
even contain the GC anymore). I always liked D because of its 
metaprogramming capabilities and not because of the automatic memory 
management. So I use D as a C++ with better meta programming. But this 
also means that I do not use phobos. I actually tried using only parts 
of phobos, but the dependency hell in phobos results in all modules 
being pulled in as soon as you use one central one, so I stopped using 
most of phobos alltogether. The annoying part, as you already pointed 
out, is that I can't use any third party libraries. So I'm living in my 
own little D world.

So I would be disappointed if classes suddently don't have destructors 
anymore. Most annoying would be the lack of compiler generated field 
destructors (e.g. automatically calling the destructor of each struct in 
the class). I could work around that, but it would be another anoyance 
over C++ and as they are piling up it becomes more and more viable to 
just go back to C++.

 I support the notion that if the GC isn't removed as a foundational
 feature of D, then destructors should probably be removed from D.
 That said, I really want my destructors, and would be very upset to
 see them go. So... ARC?


I think ARC could work, but should be extended with some sort of 
ownership notation. Often a block of memory (e.g. an array of data) is 
exclusivly owned by a single object. So it would be absolutly 
uneccessary to reference count that block of memory. Instead I would 
want something like Rust has, borrowed pointers. (We actually already 
have that, "scope" but its not defined nor implemented for anything but 
delegates)

Kind Regards
Benjamin Thaut



 May 11 2014



  parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 11 May 2014 17:52, Benjamin Thaut via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am 06.05.2014 05:40, schrieb Manu via Digitalmars-d:


 I support the notion that if the GC isn't removed as a foundational
 feature of D, then destructors should probably be removed from D.

 That said, I really want my destructors, and would be very upset to
 see them go. So... ARC?


 I think ARC could work, but should be extended with some sort of ownership
 notation. Often a block of memory (e.g. an array of data) is exclusivly
 owned by a single object. So it would be absolutly uneccessary to reference
 count that block of memory. Instead I would want something like Rust has,
 borrowed pointers. (We actually already have that, "scope" but its not
 defined nor implemented for anything but delegates)


Indeed, I also imagine that implementation of 'scope' would allow for
a really decent ARC experience. D already has some advantages over
other languages, but that one would be big.



 May 11 2014



 next sibling parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Sunday, 11 May 2014 at 09:53:59 UTC, Manu via Digitalmars-d 
wrote:

 On 11 May 2014 17:52, Benjamin Thaut via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Am 06.05.2014 05:40, schrieb Manu via Digitalmars-d:


 I support the notion that if the GC isn't removed as a 
 foundational
 feature of D, then destructors should probably be removed 
 from D.

 That said, I really want my destructors, and would be very 
 upset to
 see them go. So... ARC?


 I think ARC could work, but should be extended with some sort 
 of ownership
 notation. Often a block of memory (e.g. an array of data) is 
 exclusivly
 owned by a single object. So it would be absolutly uneccessary 
 to reference
 count that block of memory. Instead I would want something 
 like Rust has,
 borrowed pointers. (We actually already have that, "scope" but 
 its not
 defined nor implemented for anything but delegates)


 Indeed, I also imagine that implementation of 'scope' would 
 allow for
 a really decent ARC experience. D already has some advantages 
 over
 other languages, but that one would be big.


Yes, together with an opImplicitCast of some sort it could 
probably even be implemented as a pure library type 
(std.typecons.RefCounted). std.typecons.scoped can also benefit 
from this.

This would allow safe implicit casting to non-RC types, i.e. 
passing to a function that accepts a non-RC scope parameter. This 
is extremely important if we want to be able to use RC types only 
in some cases; otherwise it would need to "infect" everything in 
order to be safe.



 May 11 2014



prev sibling  parent reply "w0rp" <devw0rp gmail.com>  writes:


The vast majority of software, at least as far as I can see, use 
web services. That makes up the vast majority of software on my 
Android phone. Garbage collection is definitely applicable for 
web servers, so there is a huge area where D and a garbage 
collector can apply nicely. I think the arguement that the vast 
majority of software should be real time now is very weak, I 
wouldn't argue that. I would simply argue that garbage collection 
isn't applicable to real time software, because that is a given.

I'm really not sure how anything but a manual memory management 
allocation scheme could work with real time software. It seems to 
me that if you are writing software where any cost in time is 
absolutely critical, and you must know exactly when you are 
allocated and deallocating, then the best you can hope to do is 
to write these things yourself.

I don't think it's possible for a computer out there to manage 
time for you at the most fundamental level, managing memory. If I 
was to write a real time application, I would not interact with a 
garbage collector and use primarily small data structures on a 
stack. If I needed to allocate objects on a heap, I would use 
something I could resize and destroy pretty manually, or at least 
in a scoped manner, like std::vector. I can't see how garbage 
collection or automatic reference counting would help me. I would 
want to have primarily scoped or unique references to data, not 
shared references.



 May 11 2014



  parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 11.05.2014 12:57, schrieb w0rp:

 The vast majority of software, at least as far as I can see, use web
 services. That makes up the vast majority of software on my Android
 phone. Garbage collection is definitely applicable for web servers, so
 there is a huge area where D and a garbage collector can apply nicely. I
 think the arguement that the vast majority of software should be real
 time now is very weak, I wouldn't argue that. I would simply argue that
 garbage collection isn't applicable to real time software, because that
 is a given.

 I'm really not sure how anything but a manual memory management
 allocation scheme could work with real time software. It seems to me
 that if you are writing software where any cost in time is absolutely
 critical, and you must know exactly when you are allocated and
 deallocating, then the best you can hope to do is to write these things
 yourself.

 I don't think it's possible for a computer out there to manage time for
 you at the most fundamental level, managing memory. If I was to write a
 real time application, I would not interact with a garbage collector and
 use primarily small data structures on a stack. If I needed to allocate
 objects on a heap, I would use something I could resize and destroy
 pretty manually, or at least in a scoped manner, like std::vector. I
 can't see how garbage collection or automatic reference counting would
 help me. I would want to have primarily scoped or unique references to
 data, not shared references.


Apparently the customers of Aicas, Aonix, IBM and IS2T have a different 
opinion.

--
Paulo



 May 11 2014



prev sibling  parent  "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Tuesday, 6 May 2014 at 03:40:47 UTC, Manu via Digitalmars-d 
wrote:

 On 3 May 2014 18:49, Benjamin Thaut via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:

 Walter and I have had a long chat in which we figured our 
 current
 offering of abstractions could be improved. Here are some 
 thoughts.
 There's a lot of work ahead of us on that and I wanted to 
 make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls 
 from within
 the GC entirely. It makes for a faster and better GC, but the 
 real
 reason here is that destructors are philosophically bankrupt 
 in a GC
 environment. I think there's no need to argue that in this 
 community.

 The GC never guarantees calling destructors even today, so 
 this decision
 would be just a point in the definition space (albeit an 
 extreme one).

 That means classes that need cleanup (either directly or by 
 having
 fields that are structs with destructors) would need to 
 garner that by
 other means, such as reference counting or manual. We're 
 considering
 deprecating ~this() for classes in the future.

 Also, we're considering a revamp of built-in slices, as 
 follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently 
 lowered into
 RCSlice!T, defined inside object.d. That type would occupy 
 THREE words,
 one of which being a pointer to a reference count. That type 
 would
 redefine all slice primitives to update the reference count 
 accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit 
 cast(void[])
 will be allowed, and will ignore the reference count (so if a 
 void[]
 extracted from a T[] via a cast outlives all slices, dangling 
 pointers
 will ensue).

 I foresee any number of theoretical and practical issues with 
 this
 approach. Let's discuss some of them here.


 Thanks,

 Andrei



 Honestly, that sounds like the entierly wrong apporach to me. 
 Your
 approaching the problem in this way:

 "We can not implement a propper GC in D because the language 
 design prevents
 us from doing so. So lets remove destructors to migate the 
 issue of false
 pointers."

 While the approach should be.

 "The language does not allow to implement a propper GC 
 (anything else then
 dirty mark & sweep), what needs to be changed to allow a 
 implementation of a
 more sophisticated GC."


 Couldn't agree more.
 Abandoning destructors is a disaster.
 Without destructors, you effectively have manual memory 
 management, or
 rather, manual 'resource' management, which is basically the 
 same
 thing, even if you have a GC.
 It totally undermines the point of memory management as a 
 foundational
 element of the language if most things are to require manual
 release/finalisation/destruction or whatever you wanna call it.




 which heavily
 relies on resource management. So basically every class in 
 there inherits

 the finalizer

 Basically the
 entire codebase feels like manual memory management. You have 
 to think about
 manually destroying every class and the entire advantage of 
 having a GC,
 e.g. not having to think about memory management and thus 
 beeing more

 syntax. Do we
 really want that for D?


 This is interesting to hear someone else say this. I have 
 always found

 memory
 management in practise too.
 I've ranted enough about it already, but I have come to the firm
 conclusion that the entire premise of a mark&sweep GC is 
 practically
 corrupt. Especially in D.

 experience that
 absolutely parallels your example, I realise that GC's failure 
 extends
 into far more cases than just the ones I'm usually representing.

 I also maintain that GC isn't future-proof in essence. 
 Computers grow
 exponentially, and GC performance inversely tracks the volume of
 memory in the system. Anything with an exponential growth curve 
 is
 fundamentally not future-proof.
 I predict a 2025 Wikipedia entry: "GC was a cute idea that 
 existed for
 a few years in the early 2000's while memory ranged in the 
 100's mb -
 few gb's, but quickly became unsustainable as computer 
 technology
 advanced".



 And what if I want unsafe slices of structs with destructors, 
 for
 performance? Maybe I perfectly know that the memory behind the 
 slice will
 outlive the slice, and I don't want the overhead of all the 
 reference
 counthing behind it?

 If you actually deprecate ~this, there would be two options 
 for me.
 1) Migrate my entire codebase to some user defiend finalizer 
 function (which
 doesn't have compiler support), which would be a lot of work.


 Does ~this() actually work, or just usually work?



 2) Quit D. (which is becomeing more and more an option when 
 reading the
 recent news group discussions.)


 I'm starting to fear the same outcome for myself.
 I don't have any idea how to reconcile this problem in my 
 working
 environment, and very little community sympathy. I'm not seeing 
 real
 solutions emerging, and the only one I can imagine that's even
 theoretically possible is wildly unpopular (ARC).
 For years, I just (naively?) assumed that the GC was immature, 
 and
 would improve with time. Never gave it much thought; assumed 
 there
 were people much smarter than me with a plan...

 I can't imagine any way out of this without significant breaking
 changes to the type system. Probably a new pointer type at 
 least.


 This thread is starting to convince me that the GC should 
 probably be
 repositioned as a convenience library provided *beside* the 
 language,
 rather than a foundation of the language. It should be 
 exclusively
 opt-in, not opt-out, and upon opting-in, you accept the 
 associated
 problems.

 actually
 broken too hadn't occurred to me until now, but if I had to 
 nominate

 certainly
 it; it's built on a GC, but isn't really compatible with it 
 either.

 little
 value in the feature, and a definite tendency to produce 
 unreliability
 by users who aren't experts on garbage collection and presume it
 should 'just work'.

 I support the notion that if the GC isn't removed as a 
 foundational
 feature of D, then destructors should probably be removed from 
 D.
 That said, I really want my destructors, and would be very 
 upset to
 see them go. So... ARC?


You make some good arguments for ARC, but I think most of the 
discussion here is lacking in expertise. We need at least one 
serious memory management expert, possibly academic, to really 
set straight what is and isn't possible in D.



 May 11 2014



prev sibling  parent  "Dicebot" <public dicebot.lv>  writes:


Initial post in this thread makes focus on a change that does not 
fix anything and implies silent semantical breakage. I am glad 
Andrei has reconsidered it but feels like key problem is not that 
proposal itself was bad but that it does not even try to solve.

Real issue being deterministic destruction and/or deallocation of 
polymorphic objects. Class destructors are bad for that indeed. 
But I want some better replacement before prohibiting anything.

There is a very practical use case that highlights existing 
problem, it was discussed few months ago in exception performance 
thread. To speed up throwing of non-const exceptions it is 
desirable to use pool of exception objects. However currently 
there is no place where to put code for releasing object back 
into pool. One cannot use destructor because it will never be 
called if pool keeps the reference. One cannot use reference 
counting struct wrapper because breaks polymorphic catching of 
exceptions.

Any good solution to be proposed should be capable of solving 
this problem.



 May 05 2014