www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - RFC: reference counted Throwable

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
As discussed, having exception objects being GC-allocated is clearly a 
large liability that we need to address. They prevent otherwise careful 
functions from being  nogc so they affect even apps that otherwise would 
be okay with a little litter here and there.

The Throwable hierarchy is somewhat separate from everything else, which 
makes it a great starting point for investigating an automated reference 
count approach. Here's what I'm thinking.

First, there must be some compiler flag -nogc or something, which 
triggers the RC exceptions. All modules of an application must be 
compiled with this flag if it is to work (such that one module can throw 
an exception caught by the other). Of course a lot of refinement needs 
to be added here (what happens if one tries to link modules built with 
and without -nogc, allowing people to detect the flag programmatically 
by using version(nogc) etc).

If -nogc is passed, the compiler severs the inheritance relationship 
between Throwable and Object, making it impossible to convert a 
Throwable to an Object. From then henceforth, Throwable and Object form 
a two-rooted forest. (In all likelihood we'll later add an RCObject root 
that Throwable inherits.)

Whenever a reference to a Throwable is copied about, passed to 
functions, the compiler inserts appropriately calls to e.g. incRef and 
decRef. (Compiler may assume they cancel each other for optimization 
purposes.) Implementation of these is up to the runtime library. Null 
checking may be left to either the compiler or the library (in the 
latter case, the functions must be nonmember). However it seems the 
compiler may have an advantage because it can elide some of these checks.

Once we get this going, we should accumulate good experience that we can 
later apply to generalizing this approach to more objects. Also, if 
things go well we may as well define _always_ (whether GC or not) 
Throwable to be reference counted; seems like a good fit for all programs.

Please chime in with thoughts.


Andrei
Sep 19 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-19 17:32, Andrei Alexandrescu wrote:


 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.) Implementation of these is up to the runtime library.


Are you suggesting we implement ARC?

-- 
/Jacob Carlborg
Sep 19 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 12:42 PM, Jacob Carlborg wrote:

 On 2014-09-19 17:32, Andrei Alexandrescu wrote:


 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.) Implementation of these is up to the runtime library.


 Are you suggesting we implement ARC?


Yes. -- Andrei
Sep 19 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 19 September 2014 at 22:14:08 UTC, Andrei Alexandrescu 
wrote:

 On 9/19/14, 12:42 PM, Jacob Carlborg wrote:

 On 2014-09-19 17:32, Andrei Alexandrescu wrote:


 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. 
 incRef and
 decRef. (Compiler may assume they cancel each other for 
 optimization
 purposes.) Implementation of these is up to the runtime 
 library.


 Are you suggesting we implement ARC?


 Yes. -- Andrei


I don't think ARC is needed.

library RC + borrowing + uniqueness/moving = WIN
Sep 20 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz wrote:

 On Friday, 19 September 2014 at 22:14:08 UTC, Andrei 
 Alexandrescu wrote:

 On 9/19/14, 12:42 PM, Jacob Carlborg wrote:

 On 2014-09-19 17:32, Andrei Alexandrescu wrote:


 Whenever a reference to a Throwable is copied about, passed 
 to
 functions, the compiler inserts appropriately calls to e.g. 
 incRef and
 decRef. (Compiler may assume they cancel each other for 
 optimization
 purposes.) Implementation of these is up to the runtime 
 library.


 Are you suggesting we implement ARC?


 Yes. -- Andrei


 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


You can't do polymorphic entity RC (like exceptions) without at 
least some help from compiler because language currently does not 
provide tools for lifetime control of classes. At least _some_ 
form of ARC is necessary.
Sep 20 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 08:32:55 UTC, Dicebot wrote:

 On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz 
 wrote:

 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


 You can't do polymorphic entity RC (like exceptions) without at 
 least some help from compiler because language currently does 
 not provide tools for lifetime control of classes. At least 
 _some_ form of ARC is necessary.


I think we can, using templated alias this. We're not there yet, 
but it's probably feasable, seeing that Igor Stepanov already 
implemented multiple alias this [1]. With that and maybe a little 
opDispatch magic, it should be possible to making wrappers that 
are implicitly convertible to wrappers of parents of their inner 
types.

Granted, for exceptions there's more needed: There needs to be 
support for throwing and catching these wrappers, and for 
catching wrappers of derived types, too. But note that 
throw/catch itself doesn't involve copying, it's a moving 
operation, which might make it easier.

[1] https://github.com/D-Programming-Language/dmd/pull/3998
Sep 20 2014
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 09:05:24 UTC, Marc Schütz wrote:

 On Saturday, 20 September 2014 at 08:32:55 UTC, Dicebot wrote:

 On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz 
 wrote:

 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


 You can't do polymorphic entity RC (like exceptions) without 
 at least some help from compiler because language currently 
 does not provide tools for lifetime control of classes. At 
 least _some_ form of ARC is necessary.


 I think we can, using templated alias this. We're not there 
 yet, but it's probably feasable, seeing that Igor Stepanov 
 already implemented multiple alias this [1]. With that and 
 maybe a little opDispatch magic, it should be possible to 
 making wrappers that are implicitly convertible to wrappers of 
 parents of their inner types.

 Granted, for exceptions there's more needed: There needs to be 
 support for throwing and catching these wrappers, and for 
 catching wrappers of derived types, too. But note that 
 throw/catch itself doesn't involve copying, it's a moving 
 operation, which might make it easier.

 [1] https://github.com/D-Programming-Language/dmd/pull/3998


Thinking further about it, we don't need templated `alias this`, 
because we can just have one `alias this` that returns 
`scope!this(T)` (this needs to be there anyway to be safe). It 
would automatically behave like a normal class with regard to 
polymorphy.

What's still needed then is a little magic to actually be able to 
throw and catch the wrapper.
Sep 20 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 09:05:24 UTC, Marc Schütz wrote:

 On Saturday, 20 September 2014 at 08:32:55 UTC, Dicebot wrote:

 On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz 
 wrote:

 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


 You can't do polymorphic entity RC (like exceptions) without 
 at least some help from compiler because language currently 
 does not provide tools for lifetime control of classes. At 
 least _some_ form of ARC is necessary.


 I think we can, using templated alias this. We're not there 
 yet, but it's probably feasable, seeing that Igor Stepanov 
 already implemented multiple alias this [1]. With that and 
 maybe a little opDispatch magic, it should be possible to 
 making wrappers that are implicitly convertible to wrappers of 
 parents of their inner types.

 Granted, for exceptions there's more needed: There needs to be 
 support for throwing and catching these wrappers, and for 
 catching wrappers of derived types, too. But note that 
 throw/catch itself doesn't involve copying, it's a moving 
 operation, which might make it easier.

 [1] https://github.com/D-Programming-Language/dmd/pull/3998


Yeah but implicitly convertible to what? If you convert it to 
`Throwable` your reference counting facilities are circumvented 
resulting in dangling exception reference at point where 
`Throwable` gets caught.

Special casing catching such wrappers to still preserve original 
ref-counted type while pretending to be Throwable at call site 
sounds like a terrible hack, much worse than any sort of ARC 
complexity.
Sep 20 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 09:19:21 UTC, Dicebot wrote:

 On Saturday, 20 September 2014 at 09:05:24 UTC, Marc Schütz 
 wrote:

 On Saturday, 20 September 2014 at 08:32:55 UTC, Dicebot wrote:

 On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz 
 wrote:

 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


 You can't do polymorphic entity RC (like exceptions) without 
 at least some help from compiler because language currently 
 does not provide tools for lifetime control of classes. At 
 least _some_ form of ARC is necessary.


 I think we can, using templated alias this. We're not there 
 yet, but it's probably feasable, seeing that Igor Stepanov 
 already implemented multiple alias this [1]. With that and 
 maybe a little opDispatch magic, it should be possible to 
 making wrappers that are implicitly convertible to wrappers of 
 parents of their inner types.

 Granted, for exceptions there's more needed: There needs to be 
 support for throwing and catching these wrappers, and for 
 catching wrappers of derived types, too. But note that 
 throw/catch itself doesn't involve copying, it's a moving 
 operation, which might make it easier.

 [1] https://github.com/D-Programming-Language/dmd/pull/3998


 Yeah but implicitly convertible to what? If you convert it to 
 `Throwable` your reference counting facilities are circumvented 
 resulting in dangling exception reference at point where 
 `Throwable` gets caught.


As I said, throw/catch is at its core a moving operation. For 
classes this isn't important (they are references), but for an RC 
wrapper it would be, so we could specify that.

Move-constructing from an lvalue in the context of D can be seen 
as a three-step process:

1) create a temporary, initialize with T.init
2) bit-swap the variable with the temporary
3) destroy the variable

It can be seen that the value that is to be moved (the RC 
wrapper) must be non-const (only at the head, it may still be 
tail-const).

Now, any generic RC type that wants to be implicitly convertible 
to its payload type must do this via borrowing in order to be 
safe (see my proposal at [1]). Using const-borrowing, we can 
guarantee that the wrapper will not be thrown (or moved, in 
general) as long as borrowed references to its payload exist:

     struct RC(T) {
         // ...
         T _payload;
         scope!(const this) borrow() {
             return _payload;
         }
         alias borrow this;
     }

This already solves avoiding dangling references to the 
exception: No references can be left behind when the exception is 
thrown, and the wrapper will not be destroyed, but moved, thus 
not releasing the exception's memory.

The second part is really "just" some way to transport the 
wrapper via the exception mechanism, including support for 
catching wrappers of derived exception types.


 Special casing catching such wrappers to still preserve 
 original ref-counted type while pretending to be Throwable at 
 call site sounds like a terrible hack, much worse than any sort 
 of ARC complexity.


IMO it would only a hack _if_ they were indeed special cased. I'm 
sure we can find a more generic mechanism that would allow this 
to be implemented cleanly. Note that the other requirements I 
described above (borrowing, move semantics) are also things that 
just happen to be usable here: they are desirable in general, 
exceptions are just one possible application.

[1] http://wiki.dlang.org/User:Schuetzm/scope
Sep 20 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 11:50:28 UTC, Marc Schütz wrote:

 On Saturday, 20 September 2014 at 09:19:21 UTC, Dicebot wrote:

 Yeah but implicitly convertible to what? If you convert it to 
 `Throwable` your reference counting facilities are 
 circumvented resulting in dangling exception reference at 
 point where `Throwable` gets caught.


 As I said, throw/catch is at its core a moving operation. For 
 classes this isn't important (they are references), but for an 
 RC wrapper it would be, so we could specify that.

 Move-constructing from an lvalue in the context of D can be 
 seen as a three-step process:

 1) create a temporary, initialize with T.init
 2) bit-swap the variable with the temporary
 3) destroy the variable

 It can be seen that the value that is to be moved (the RC 
 wrapper) must be non-const (only at the head, it may still be 
 tail-const).

 Now, any generic RC type that wants to be implicitly 
 convertible to its payload type must do this via borrowing in 
 order to be safe (see my proposal at [1]). Using 
 const-borrowing, we can guarantee that the wrapper will not be 
 thrown (or moved, in general) as long as borrowed references to 
 its payload exist:

     struct RC(T) {
         // ...
         T _payload;
         scope!(const this) borrow() {
             return _payload;
         }
         alias borrow this;
     }

 This already solves avoiding dangling references to the 
 exception: No references can be left behind when the exception 
 is thrown, and the wrapper will not be destroyed, but moved, 
 thus not releasing the exception's memory.

 The second part is really "just" some way to transport the 
 wrapper via the exception mechanism, including support for 
 catching wrappers of derived exception types.


 Special casing catching such wrappers to still preserve 
 original ref-counted type while pretending to be Throwable at 
 call site sounds like a terrible hack, much worse than any 
 sort of ARC complexity.


 IMO it would only a hack _if_ they were indeed special cased. 
 I'm sure we can find a more generic mechanism that would allow 
 this to be implemented cleanly. Note that the other 
 requirements I described above (borrowing, move semantics) are 
 also things that just happen to be usable here: they are 
 desirable in general, exceptions are just one possible 
 application.

 [1] http://wiki.dlang.org/User:Schuetzm/scope


The mechanism might simply be to allow anything to be thrown that
a) supports moving, e.g. provides a method `release()` that 
returns a unique expression (this requires moving and uniqueness 
to be specified first, which is a good idea anyway), and
b) is implicitly convertible to Throwable.
Sep 20 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 1:20 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Friday, 19 September 2014 at 22:14:08 UTC, Andrei Alexandrescu wrote:

 On 9/19/14, 12:42 PM, Jacob Carlborg wrote:

 On 2014-09-19 17:32, Andrei Alexandrescu wrote:


 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.) Implementation of these is up to the runtime library.


 Are you suggesting we implement ARC?


 Yes. -- Andrei


 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


s/WIN/Rust/
Sep 20 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 20 September 2014 at 16:27:33 UTC, Andrei 
Alexandrescu wrote:

 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


 s/WIN/Rust/


Well Rust does some things well. We are going in the same 
direction anyway (type qualifier, uniqueness) expect we do it in 
an ad hoc manner that is guaranteed to yield a C++ish result.
Sep 20 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 5:30 PM, deadalnix wrote:

 On Saturday, 20 September 2014 at 16:27:33 UTC, Andrei Alexandrescu wrote:

 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


 s/WIN/Rust/


 Well Rust does some things well. We are going in the same direction
 anyway (type qualifier, uniqueness) expect we do it in an ad hoc manner
 that is guaranteed to yield a C++ish result.


Rust looked a lot more exciting when I didn't know much about it. -- Andrei
Sep 20 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:


 Rust looked a lot more exciting when I didn't know much about 
 it.


I didn't remember ever seeing you excited about Rust :-) In past 
you (rightfully) didn't comment much about Rust. But do you have 
more defined ideas about it now? Do you still think D has a 
chance against Rust?

Bye,
bearophile
Sep 20 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 02:10:13 UTC, bearophile wrote:

 Andrei Alexandrescu:


 Rust looked a lot more exciting when I didn't know much about 
 it.


 I didn't remember ever seeing you excited about Rust :-) In 
 past you (rightfully) didn't comment much about Rust. But do 
 you have more defined ideas about it now? Do you still think D 
 has a chance against Rust?

 Bye,
 bearophile


It is clear that rust has some serious limitations.
  - It interact badly with the existing world (other languages).
  - It is slow to compile.
  - Constraints too much the dev in some paradigms, which 
obviously won't fit all area of programming.
  - It bets on many unproven mechanism (ie see error handling). 
Some of them may be real hit, but I doubt ALL of them will be, 
and these which won't be will have to be worked around, adding 
complexity.
  - The macro system is plain horrible, and the only way to do 
code generation.

Rust has many cool goodies, but it is generally too radical.
Sep 20 2014
parent reply Nick Treleaven <ntrel-public yahoo.co.uk> writes:
On 21/09/2014 03:35, deadalnix wrote:

   - It is slow to compile.


Surely that's not an inherent property of Rust?


   - Constraints too much the dev in some paradigms, which obviously
 won't fit all area of programming.


Absolutely. The unique mutable borrow rule too often prevents even 
reading a variable, this is a big pain IMO. I wonder if this is partly 
because Rust doesn't have const, only immutable.


   - The macro system is plain horrible,


The syntax is not great (I heard it may change), but I think the 
capabilities are actually pretty good.


 and the only way to do code generation.


I think you're wrong, you can add syntax extensions such as the 
compile-time regex support:

http://blog.burntsushi.net/rust-regex-syntax-extensions

"Luckily, the second way to write a macro is via a syntax extension 
(also known as a “procedural macro”). This is done by a compile time 
hook that lets you execute arbitrary Rust code, rewrite the abstract 
syntax to whatever you want and opens up access to the Rust compiler’s 
parser. In short, you get a lot of power. And it’s enough to implement 
native regexes. The obvious drawback is that they are more difficult to 
implement."
Sep 22 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Nick Treleaven:


  - It is slow to compile.


 Surely that's not an inherent property of Rust?


We don't know yet. Perhaps Rust type inference needs lot of 
computations.

Bye,
bearophile
Sep 22 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 7:10 PM, bearophile wrote:

 Andrei Alexandrescu:


 Rust looked a lot more exciting when I didn't know much about it.


 I didn't remember ever seeing you excited about Rust :-) In past you
 (rightfully) didn't comment much about Rust. But do you have more
 defined ideas about it now? Do you still think D has a chance against Rust?


I don't think Rust has a chance against D. -- Andrei
Sep 20 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 21.09.2014 04:50, schrieb Andrei Alexandrescu:

 On 9/20/14, 7:10 PM, bearophile wrote:

 Andrei Alexandrescu:


 Rust looked a lot more exciting when I didn't know much about it.


 I didn't remember ever seeing you excited about Rust :-) In past you
 (rightfully) didn't comment much about Rust. But do you have more
 defined ideas about it now? Do you still think D has a chance against
 Rust?


 I don't think Rust has a chance against D. -- Andrei


The real question is which language, from all that want to replace C++, 
will eventually get a place at an OS vendors SDK.

So far, the winning ones seem to be Swift on Apple side, and .NET 


Maybe someone in the commercial UNIX (FOSS is too bound with C), real 
time or embedded OS space?

--
Paulo
Sep 20 2014
parent reply "Cliff" <cliff.s.hudson gmail.com> writes:
On Sunday, 21 September 2014 at 04:59:12 UTC, Paulo Pinto wrote:

 Am 21.09.2014 04:50, schrieb Andrei Alexandrescu:

 On 9/20/14, 7:10 PM, bearophile wrote:

 Andrei Alexandrescu:


 Rust looked a lot more exciting when I didn't know much 
 about it.


 I didn't remember ever seeing you excited about Rust :-) In 
 past you
 (rightfully) didn't comment much about Rust. But do you have 
 more
 defined ideas about it now? Do you still think D has a chance 
 against
 Rust?


 I don't think Rust has a chance against D. -- Andrei


 The real question is which language, from all that want to 
 replace C++, will eventually get a place at an OS vendors SDK.

 So far, the winning ones seem to be Swift on Apple side, and 
 .NET Native-C++/CLX on Microsoft side (who knows what are they 


 Maybe someone in the commercial UNIX (FOSS is too bound with 
 C), real time or embedded OS space?

 --
 Paulo


Interop, interop, interop.  Walter and Andrei are right when they 
talk about the importance of C++ interop - not only do you get to 
leverage those libraries, but it reduces the barrier to entry for 
D in more environments.

Swift will never be more important than Objective C was - which 
is to say it'll be the main development language on Apple 
products and probably nothing else.  That has real value, but the 
limits on it are pretty hard and fast (which says more about 
Apple than the language itself.)

.NET suffers a similar problem in spite of the community's best 
efforts with Mono - it'll always be a distant 2nd (or 5th or 
20th) on other platforms.  And on Windows, C++ won't get 
supplanted by .NET absent a sea-change in the mindset of the 
Windows OS group - which is notoriously resistant to change (and 
they have a colossal existing code base which isn't likely to 
benefit from the kind of inflection point Apple had moving to a 
BSD and porting/rewriting scads of code.)

So C/C++ is it for universal languages, really (outside of the 
web server space, where you have a large Java deployment.)  I 
don't think D needs to be the next .NET (of any flavor) or the 
next Swift, and I don't see as it is being positioned that way 
either - the target to me is clearly C/C++.  It doesn't need to 
compete with languages that have lesser universality, though it 
should (and does) borrow the good ideas from those languages.

I don't think D needs to look at *replacing* C++ in the near or 
mid term either - it still needs to convince people it deserves a 
place at the table.  And the easiest way to do that is to get 
this C++ interop story really nailed down, and make sure D's 
warts are smaller than C++'s.  And, of course, the GC strawman 
that native programmers always claim is more important than it 
really is.  I like the threads going on currently about ARC and 
related technologies - there's a real chance to innovate here.
Sep 20 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 05:55:20 UTC, Cliff wrote:

 .NET suffers a similar problem in spite of the community's best 
 efforts with Mono - it'll always be a distant 2nd (or 5th or 
 20th) on other platforms.  And on Windows, C++ won't get 
 supplanted by .NET absent a sea-change in the mindset of the 
 Windows OS group - which is notoriously resistant to change 
 (and they have a colossal existing code base which isn't likely 
 to benefit from the kind of inflection point Apple had moving 
 to a BSD and porting/rewriting scads of code.)


A bit OT: but Mono seems to be a popular platform for desktop 
application on linux. I have no idea why.
Sep 20 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 21.09.2014 08:05, schrieb deadalnix:

 On Sunday, 21 September 2014 at 05:55:20 UTC, Cliff wrote:

 .NET suffers a similar problem in spite of the community's best
 efforts with Mono - it'll always be a distant 2nd (or 5th or 20th) on
 other platforms.  And on Windows, C++ won't get supplanted by .NET
 absent a sea-change in the mindset of the Windows OS group - which is
 notoriously resistant to change (and they have a colossal existing
 code base which isn't likely to benefit from the kind of inflection
 point Apple had moving to a BSD and porting/rewriting scads of code.)


 A bit OT: but Mono seems to be a popular platform for desktop
 application on linux. I have no idea why.


I think it may come from C++ hate that GNU/Linux enjoys, or at least it 
used to enjoy.

Back in the early days suggesting using C++ was an heresy, although 
corporate developers were slowly migrating to it in commercial UNIX 
settings.

So you got to do CORBA in C, OO in C (Gtk) and so on.

When I used to take part in gtkmm discussions  (early 2000), the usual C 
vs C++ discussions on GNOME mailing lists were quite common.

Qt and KDE were a no go for many, due to licesing issues.

So C++ usage was limited to those of us already into C++ that weren't 
that keen in using plain C. A minority,

Meanwhile, Microsoft released .NET and Miguel and others, which already 
had ported Microsoft technologies (Bonobo(COM), Evolution(Outlook)...) 
started Mono.

Given that they were also GNOME maintainers, there was this big 
discussion going on about GNOME becoming tainted with Mono.


C and uses gobject as runtime system.

I used to be in the "we don't want .NET on GNU/Linux" side.

Nowadays I can only congratulate Miguel and the others that persisted 
against naysayers like myself. Specially given what happened with Oracle 
vs Google.

Which Xamarin and Microsoft took advantage of with the "Community 
license for open source".

Slowly, as the amount of users started to increase, I guess the 
GNU/Linux opinion about Mono, started to change.

So if you want to use a modern programming language, with bindings to
most common UI frameworks and ability to generate fast code, there 
aren't many other options.

Here I would say D could also fit the bill, but lacks visibility by the 
average GNU/Linux coder on the playground discussions.


This is just an opinion, maybe I am too far of the real reasons.

--
Paulo
Sep 20 2014
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-21 07:55, Cliff wrote:


 Swift will never be more important than Objective C was - which is to
 say it'll be the main development language on Apple products and
 probably nothing else.  That has real value, but the limits on it are
 pretty hard and fast (which says more about Apple than the language
 itself.)

 .NET suffers a similar problem in spite of the community's best efforts
 with Mono - it'll always be a distant 2nd (or 5th or 20th) on other
 platforms.  And on Windows, C++ won't get supplanted by .NET absent a
 sea-change in the mindset of the Windows OS group - which is notoriously
 resistant to change (and they have a colossal existing code base which
 isn't likely to benefit from the kind of inflection point Apple had
 moving to a BSD and porting/rewriting scads of code.)


Unfortunately the user base is so large on these platforms that it's 
enough for these languages to only work on one particular platform and 
the developers will invest a huge amount work on it anyway. The 
languages will have a large user base even though they only work on one 
platform.

-- 
/Jacob Carlborg
Sep 21 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-21 07:55, Cliff wrote:


 .NET suffers a similar problem in spite of the community's best efforts
 with Mono - it'll always be a distant 2nd (or 5th or 20th) on other
 platforms.  And on Windows, C++ won't get supplanted by .NET absent a
 sea-change in the mindset of the Windows OS group - which is notoriously
 resistant to change (and they have a colossal existing code base which
 isn't likely to benefit from the kind of inflection point Apple had
 moving to a BSD and porting/rewriting scads of code.)


Actually, Microsoft, with the help of Xamarin seems to aim for 
cross-paltform with Mono, at least for mobile platforms.

-- 
/Jacob Carlborg
Sep 21 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 21 September 2014 at 01:46:13 UTC, Andrei Alexandrescu 
wrote:

 Rust looked a lot more exciting when I didn't know much about 
 it. -- Andrei


I think it is interesting that Rust is kinda converging on only 
having the scope semantics in the language and the rest in the 
library.

(of course i'm biased to notice that cuz that's what I want here 
too!)
Sep 20 2014
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 01:46:13 UTC, Andrei Alexandrescu 
wrote:

 On 9/20/14, 5:30 PM, deadalnix wrote:

 On Saturday, 20 September 2014 at 16:27:33 UTC, Andrei 
 Alexandrescu wrote:

 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN


 s/WIN/Rust/


 Well Rust does some things well. We are going in the same 
 direction
 anyway (type qualifier, uniqueness) expect we do it in an ad 
 hoc manner
 that is guaranteed to yield a C++ish result.


 Rust looked a lot more exciting when I didn't know much about 
 it. -- Andrei


Ho come on, there is a reason why I'm here and not on rust's 
mailing list. I'm aware of various shortcomings of rust. Reading 
my message as a promotion of rust is a gross strawman.

And here is the thing. You and Walter closed one door to this 
kind of feature when discussed with Bartoz a while ago and won't 
admit you are wrong. Then whole thing degenerate in some 
ridiculous church war where all rhetorical trick and logical 
fallacy is good to use.

That is pretty much how it degenerated with the concept vs static 
if discussion. The whole debate become completely ridiculous 
(yes, your dismissal of concept is almost as ridiculous as 
Bjarne's rebuttal of static if). Because at this point, this is 
not about what is best, but about who is right.

Back to the subject, that door was closed few years ago. Now we 
are discussing pile of hacks again and again and again. The final 
complexity is WAY higher than the initial proposal. Be it 
Bartoz's, rust's, isolated or whatever variation of the concept, 
this is pretty much a given by now that it is needed.

As far as I'm concerned, isolated/owned (not even talking about 
rust's concepts like burrowing) on the GC should probably be 
valid in  nogc, and only the promotion to TL, shared or immutable 
heap should be invalid in  nogc. You combine that with require 
that you throw isolated/owned/whatever, and the whole discussion 
we are having here becomes moot (as well as various it can reduce 
friction between the GC world and the non GC world greatly).
Sep 20 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:


 Are you suggesting we implement ARC?


 Yes. -- Andrei


I think it's better first to design & implement a first version 
of memory ownership management, and then later add a reference 
counting scheme.

Bye,
bearophile
Sep 20 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 5:17 AM, bearophile wrote:

 Andrei Alexandrescu:


 Are you suggesting we implement ARC?


 Yes. -- Andrei


 I think it's better first to design & implement a first version of
 memory ownership management, and then later add a reference counting
 scheme.


That's not quite informative. -- Andrei
Sep 20 2014
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu 
wrote:

 First, there must be some compiler flag -nogc or something, 
 which triggers the RC exceptions. All modules of an application 
 must be compiled with this flag if it is to work (such that one 
 module can throw an exception caught by the other). Of course a 
 lot of refinement needs to be added here (what happens if one 
 tries to link modules built with and without -nogc, allowing 
 people to detect the flag programmatically by using 
 version(nogc) etc).


Won't using -nogc preclude using this in Phobos/Druntime?

How about instead of a new switch, change the name mangling of 
anything that inherits from RCObject? E.g. 'C' is used for 
classes, 'R' could be used for reference-counting classes. Then 
an ordinary version block can be used to switch between the two.
Sep 19 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 8:32 AM, Andrei Alexandrescu wrote:

 As discussed, having exception objects being GC-allocated is clearly a large
 liability that we need to address. They prevent otherwise careful functions
from
 being  nogc so they affect even apps that otherwise would be okay with a little
 litter here and there.

 The Throwable hierarchy is somewhat separate from everything else, which makes
 it a great starting point for investigating an automated reference count
 approach. Here's what I'm thinking.

 First, there must be some compiler flag -nogc or something, which triggers the
 RC exceptions. All modules of an application must be compiled with this flag if
 it is to work (such that one module can throw an exception caught by the
other).
 Of course a lot of refinement needs to be added here (what happens if one tries
 to link modules built with and without -nogc, allowing people to detect the
flag
 programmatically by using version(nogc) etc).


Having a compiler switch to change the behavior of every module in incompatible 
ways would be a disastrous balkanization. It has to be done in such a way that 
the ARC and the existing exceptions can coexist.



 If -nogc is passed, the compiler severs the inheritance relationship between
 Throwable and Object, making it impossible to convert a Throwable to an Object.
  From then henceforth, Throwable and Object form a two-rooted forest. (In all
 likelihood we'll later add an RCObject root that Throwable inherits.)

 Whenever a reference to a Throwable is copied about, passed to functions, the
 compiler inserts appropriately calls to e.g. incRef and decRef. (Compiler may
 assume they cancel each other for optimization purposes.) Implementation of
 these is up to the runtime library. Null checking may be left to either the
 compiler or the library (in the latter case, the functions must be nonmember).
 However it seems the compiler may have an advantage because it can elide some
of
 these checks.

 Once we get this going, we should accumulate good experience that we can later
 apply to generalizing this approach to more objects. Also, if things go well we
 may as well define _always_ (whether GC or not) Throwable to be reference
 counted; seems like a good fit for all programs.

 Please chime in with thoughts.


 Andrei
Sep 19 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 6:18 PM, Walter Bright wrote:

 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be done
 in such a way that the ARC and the existing exceptions can coexist.


Could you please elaborate why the disaster? -- Andrei
Sep 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 6:48 PM, Andrei Alexandrescu wrote:

 On 9/19/14, 6:18 PM, Walter Bright wrote:

 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be done
 in such a way that the ARC and the existing exceptions can coexist.


 Could you please elaborate why the disaster? -- Andrei


1. Every library for D will have to come in two versions. One which works, and 
the other is likely untested. This will divide the D community in half.

2. RC and GC engender different styles of programming. The idea that one can 
successfully switch between them with merely a compiler switch is a fantasy.
You 
cannot write non-trivial programs that function well and pay no attention to
how 
memory management is done.
Sep 19 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 7:20 PM, Walter Bright wrote:

 On 9/19/2014 6:48 PM, Andrei Alexandrescu wrote:

 On 9/19/14, 6:18 PM, Walter Bright wrote:

 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be done
 in such a way that the ARC and the existing exceptions can coexist.


 Could you please elaborate why the disaster? -- Andrei


 1. Every library for D will have to come in two versions. One which
 works, and the other is likely untested. This will divide the D
 community in half.


I understand where you're coming from. And you're right that we're 
looking at an important change - as big as -m64 vs. -m32 or porting to a 
new platform. It's big. But it's needed, soon. And it would be, it seems 
to me, a mistake to approach this big change as a small change that we 
can sneak in.

This judgment - that RC vs. GC would balkanize the community - has 
become a prejudice that is worth revisiting. Just trotting it out again 
won't do.


 2. RC and GC engender different styles of programming. The idea that one
 can successfully switch between them with merely a compiler switch is a
 fantasy. You cannot write non-trivial programs that function well and
 pay no attention to how memory management is done.


I think we're at a point where we owe it to ourselves to consider 
realizing the fantasy.


Andrei
Sep 19 2014
next sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 20.09.2014 06:43, schrieb Andrei Alexandrescu:

 On 9/19/14, 7:20 PM, Walter Bright wrote:

 On 9/19/2014 6:48 PM, Andrei Alexandrescu wrote:

 On 9/19/14, 6:18 PM, Walter Bright wrote:

 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be
 done
 in such a way that the ARC and the existing exceptions can coexist.


 Could you please elaborate why the disaster? -- Andrei


 1. Every library for D will have to come in two versions. One which
 works, and the other is likely untested. This will divide the D
 community in half.


 I understand where you're coming from. And you're right that we're
 looking at an important change - as big as -m64 vs. -m32 or porting to a
 new platform. It's big. But it's needed, soon. And it would be, it seems
 to me, a mistake to approach this big change as a small change that we
 can sneak in.

 This judgment - that RC vs. GC would balkanize the community - has
 become a prejudice that is worth revisiting. Just trotting it out again
 won't do.


 2. RC and GC engender different styles of programming. The idea that one
 can successfully switch between them with merely a compiler switch is a
 fantasy. You cannot write non-trivial programs that function well and
 pay no attention to how memory management is done.


 I think we're at a point where we owe it to ourselves to consider
 realizing the fantasy.


 Andrei


This is one of the reasons why the Objective-C GC failed.

Mixing Frameworks compiled with both modes never worked properly.

--
Paulo
Sep 19 2014
next sibling parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 06:28:11 UTC, Paulo Pinto wrote:

 This is one of the reasons why the Objective-C GC failed.

 Mixing Frameworks compiled with both modes never worked 
 properly.


Yeah, mixing is bad unless you template eveything. But in pure 
nogc it would work.

Except, I think most pure nogc projects want to eliminate 
rtti/exceptions too...
Sep 20 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 07:35:47 UTC, Ola Fosheim 
Grostad wrote:

 Except, I think most pure nogc projects want to eliminate 
 rtti/exceptions too...


It should be possible to do on top of plain  nogc Phobos by 
simply hooking into druntime _d_throw function and providing some 
own domain-specific handler instead. Of course it doesn't help if 
exceptions are used to indicate casual data processing signals 
but it is something else to be fixed in Phobos :)
Sep 20 2014
prev sibling parent reply "Olivier Pisano" <olivier.pisano laposte.net> writes:
On Saturday, 20 September 2014 at 06:28:11 UTC, Paulo Pinto wrote:

 This is one of the reasons why the Objective-C GC failed.

 Mixing Frameworks compiled with both modes never worked 
 properly.

 --
 Paulo


Yes this was a huge failure to take into account.

Linking code where Throwable inherits from Object with code where
Throwable inherits from RCObject... :(

If making the GC completely optional is a must, then error
handling shouldn't rely on it at all, no? What about completely
switching exception handling to RC ? Would it have an impact on
memory safety since exeption handling mecanism is somehow
"magical code generated by the compiler" ?
Sep 20 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 12:46 AM, Olivier Pisano wrote:

 If making the GC completely optional is a must, then error
 handling shouldn't rely on it at all, no? What about completely
 switching exception handling to RC ? Would it have an impact on
 memory safety since exeption handling mecanism is somehow
 "magical code generated by the compiler" ?


The more I think of it the more sensible this is. Exceptions are 
unlikely to create cycles, not copied extensively, and are generally 
short lived. So an RC scheme backed by malloc/free seems to be most 
appropriate.

There would be breakage, though: Throwable would not be convertible to 
Object. I wonder what the impact in the real world that would cause.


Andrei
Sep 20 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 20 Sep 2014 09:07:20 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 9/20/14, 12:46 AM, Olivier Pisano wrote:

 If making the GC completely optional is a must, then error
 handling shouldn't rely on it at all, no? What about completely
 switching exception handling to RC ? Would it have an impact on
 memory safety since exeption handling mecanism is somehow
 "magical code generated by the compiler" ?


 
 The more I think of it the more sensible this is. Exceptions are 
 unlikely to create cycles, not copied extensively, and are generally 
 short lived. So an RC scheme backed by malloc/free seems to be most 
 appropriate.
 
 There would be breakage, though: Throwable would not be convertible
 to Object. I wonder what the impact in the real world that would
 cause.
 
 
 Andrei
 


+1, replace it completely with malloc/free.

However, for backwards compatibility malloced exceptions probably still
have to be added as roots to the GC, at least if they refer GC
allocated data. This should be somehow optional however.
Sep 21 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 09:01:45 UTC, Johannes Pfau wrote:

 +1, replace it completely with malloc/free.

 However, for backwards compatibility malloced exceptions 
 probably still
 have to be added as roots to the GC, at least if they refer GC
 allocated data. This should be somehow optional however.


Yeah, so the GC do not scan in it, and the error message 
disappear randomly under your feet and you segfault when trying 
to read. Looks like a great idea.
Sep 21 2014
parent Johannes Pfau <nospam example.com> writes:
Am Sun, 21 Sep 2014 09:35:40 +0000
schrieb "deadalnix" <deadalnix gmail.com>:


 On Sunday, 21 September 2014 at 09:01:45 UTC, Johannes Pfau wrote:

 +1, replace it completely with malloc/free.

 However, for backwards compatibility malloced exceptions 
 probably still
 have to be added as roots to the GC, at least if they refer GC
 allocated data. This should be somehow optional however.


 
 Yeah, so the GC do not scan in it, and the error message 
 disappear randomly under your feet and you segfault when trying 
 to read. Looks like a great idea.


If you always need to add exceptions to GC roots the whole proposal of
reference counting exceptions is useless as you depend on the GC
exactly in the same way as if you allocate from the GC. Andrei initially
proposed pure reference-counted exceptions in -nogc which would suffer
from the same issue.

And storing GC-allocated messages in exceptions is in most cases bad
design. If the message is a string literal there's no GC involved and
if you add 'information' to the message, making it dynamic, then:
* In order to process that information you need to extract it from the
  message string
* You create a dynamic string which might never be used (exception can
  be caught and ignored, etc.)

The correct design is to add the information to a variable in a
subclassed exception and produce the message in toString, which can
work entirely without a GC.

Example:
BAD:
throw new Exception("Timout occured, timeout = " ~
to!string(timeout))
GOOD: throw new TimeoutException("Timeout occured", timout)
Sep 21 2014
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On 9/20/2014 1:43 PM, Andrei Alexandrescu wrote:

 On 9/19/14, 7:20 PM, Walter Bright wrote:

 On 9/19/2014 6:48 PM, Andrei Alexandrescu wrote:

 On 9/19/14, 6:18 PM, Walter Bright wrote:

 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be
 done
 in such a way that the ARC and the existing exceptions can coexist.


 Could you please elaborate why the disaster? -- Andrei


 1. Every library for D will have to come in two versions. One which
 works, and the other is likely untested. This will divide the D
 community in half.


 I understand where you're coming from. And you're right that we're
 looking at an important change - as big as -m64 vs. -m32 or porting to a
 new platform. It's big. But it's needed, soon. And it would be, it seems
 to me, a mistake to approach this big change as a small change that we
 can sneak in.

 This judgment - that RC vs. GC would balkanize the community - has
 become a prejudice that is worth revisiting. Just trotting it out again
 won't do.


Unless I'm missing something, Walter is referring specifically to the 
-nogc switch (or any such switch) and not to the concept of RC vs GC in 
general. And on that I am in strong agreement, FWIW.

Recall back during the D1/D2, Phobos/Tango period, library maintainers 
had to either choose which D version and library to support (D1/Phobos, 
D1/Tango, D2/Phobos) or come up with a means of supporting all of them. 
Wrapper modules and mixins were a workable (though terribly annoying) 
solution for some cases, but not all. It was a nasty mess.

I don't know yet what the ultimate impact of a -nogc switch would be on 
library maintenance, but I have a strong suspicion that simple wrapper 
modules and mixins wouldn't be enough to maintain compatibility. If 
there is even the slightest possibility that library maintainers will 
find themselves in that same situation of having to choose a or b 
because maintaining both is too troublesome to bother, then all the 
implications need to be hashed out beforehand.

---
This email is free from viruses and malware because avast! Antivirus protection
is active.
http://www.avast.com
Sep 20 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 12:48 AM, Mike Parker wrote:

 I don't know yet what the ultimate impact of a -nogc switch would be on
 library maintenance, but I have a strong suspicion that simple wrapper
 modules and mixins wouldn't be enough to maintain compatibility. If
 there is even the slightest possibility that library maintainers will
 find themselves in that same situation of having to choose a or b
 because maintaining both is too troublesome to bother, then all the
 implications need to be hashed out beforehand.


We need to explore that. A possibility is to support coexistence and 
then have the option to use a tool statically pinpoint the uses of GC. 
-- Andrei
Sep 20 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 20 September 2014 at 16:15:45 UTC, Andrei 
Alexandrescu wrote:

 We need to explore that. A possibility is to support 
 coexistence and then have the option to use a tool statically 
 pinpoint the uses of GC. -- Andrei


What, *exactly*, does "uses of GC" mean? In other words, what 
specifically makes GC.malloc evil that must be avoided at any 
cost while C malloc (+ GC.addRange most likely) is an acceptable 
replacement?

A few things that come to mind are:

1) Obviously, GC.malloc can trigger a collection. But this can be 
easily disabled.

2) The GC lock? I don't know how malloc handles this though.

3) Is GC.free substantially different than C's free?

4) Programmers don't typically explicitly free GC memory... but 
we could.

5) Bookkeeping overhead? I know malloc has some too though, is 
this really a dealbreaker?

6) Public relations.

...that's all I can think of. What am I missing? Which one of 
these is actually causing the problem that we're supposed to be 
fixing here?
Sep 20 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 9:31 AM, Adam D. Ruppe wrote:

 On Saturday, 20 September 2014 at 16:15:45 UTC, Andrei Alexandrescu wrote:

 We need to explore that. A possibility is to support coexistence and
 then have the option to use a tool statically pinpoint the uses of GC.
 -- Andrei


 What, *exactly*, does "uses of GC" mean? In other words, what
 specifically makes GC.malloc evil that must be avoided at any cost while
 C malloc (+ GC.addRange most likely) is an acceptable replacement?

 A few things that come to mind are:

 1) Obviously, GC.malloc can trigger a collection. But this can be easily
 disabled.

 2) The GC lock? I don't know how malloc handles this though.

 3) Is GC.free substantially different than C's free?

 4) Programmers don't typically explicitly free GC memory... but we could.

 5) Bookkeeping overhead? I know malloc has some too though, is this
 really a dealbreaker?

 6) Public relations.

 ....that's all I can think of. What am I missing? Which one of these is
 actually causing the problem that we're supposed to be fixing here?


The only problem is that GC.malloc doesn't need to be paired by a call 
to GC.free, whereas malloc must.

Andrei
Sep 20 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 20 Sep 2014 10:17:06 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 9/20/14, 9:31 AM, Adam D. Ruppe wrote:

 On Saturday, 20 September 2014 at 16:15:45 UTC, Andrei Alexandrescu
 wrote:

 We need to explore that. A possibility is to support coexistence
 and then have the option to use a tool statically pinpoint the
 uses of GC. -- Andrei


 What, *exactly*, does "uses of GC" mean? In other words, what
 specifically makes GC.malloc evil that must be avoided at any cost
 while C malloc (+ GC.addRange most likely) is an acceptable
 replacement?

 A few things that come to mind are:

 1) Obviously, GC.malloc can trigger a collection. But this can be
 easily disabled.

 2) The GC lock? I don't know how malloc handles this though.

 3) Is GC.free substantially different than C's free?

 4) Programmers don't typically explicitly free GC memory... but we
 could.

 5) Bookkeeping overhead? I know malloc has some too though, is this
 really a dealbreaker?

 6) Public relations.

 ....that's all I can think of. What am I missing? Which one of
 these is actually causing the problem that we're supposed to be
 fixing here?


 
 The only problem is that GC.malloc doesn't need to be paired by a
 call to GC.free, whereas malloc must.
 
 Andrei
 


You're both missing another obvious point:

7) The GC needs to be implemented in D, in the druntime. Malloc is
   available everywhere. The GC keeps a separate heap IIRC which wastes
   memory if you mix D code and C code which uses malloc. Both are
   important points for embedded systems. A druntime without GC is
   much easier to port and produces smaller binaries.

of course in that case, GC.addRange is not an acceptable replacement.
Here 'use of GC' literally means use of GC ;-)
Sep 21 2014
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 21 September 2014 at 09:07:58 UTC, Johannes Pfau wrote:

 7) The GC needs to be implemented in D, in the druntime.


Eh, it is easy to write a stub function that just calls malloc. 
The main problem with that is what Andrei said: most code won't 
call free since it doesn't have to, so it would leak in those 
situations. But if you're writing code for this situation, it is 
easy enough to call free; you don't *have* to, but you *can* 
(unless it is  safe code, then you can't).
Sep 21 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-20 18:31, Adam D. Ruppe wrote:

 On Saturday, 20 September 2014 at 16:15:45 UTC, Andrei Alexandrescu wrote:

 We need to explore that. A possibility is to support coexistence and
 then have the option to use a tool statically pinpoint the uses of GC.
 -- Andrei


 What, *exactly*, does "uses of GC" mean? In other words, what
 specifically makes GC.malloc evil that must be avoided at any cost while
 C malloc (+ GC.addRange most likely) is an acceptable replacement?

 A few things that come to mind are:

 1) Obviously, GC.malloc can trigger a collection. But this can be easily
 disabled.

 2) The GC lock? I don't know how malloc handles this though.

 3) Is GC.free substantially different than C's free?

 4) Programmers don't typically explicitly free GC memory... but we could.

 5) Bookkeeping overhead? I know malloc has some too though, is this
 really a dealbreaker?

 6) Public relations.

 ...that's all I can think of. What am I missing? Which one of these is
 actually causing the problem that we're supposed to be fixing here?


I think it's mostly 1) and then on second place 2).

-- 
/Jacob Carlborg
Sep 21 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 12:48 AM, Mike Parker wrote:

 I don't know yet what the ultimate impact of a -nogc switch would be on
 library maintenance, but I have a strong suspicion that simple wrapper
 modules and mixins wouldn't be enough to maintain compatibility. If
 there is even the slightest possibility that library maintainers will
 find themselves in that same situation of having to choose a or b
 because maintaining both is too troublesome to bother, then all the
 implications need to be hashed out beforehand.


Agreed. -- Andrei
Sep 20 2014
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Saturday, 20 September 2014 at 01:19:05 UTC, Walter Bright 
wrote:

 Having a compiler switch to change the behavior of every module 
 in incompatible ways would be a disastrous balkanization.


Don't -version, -debug etc. have this problem anyway?

Anyway, if you change the name mangling, then you'll get link 
errors instead of mysterious crashes.


 It has to be done in such a way that the ARC and the existing 
 exceptions can coexist.


How?
Sep 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 7:04 PM, Vladimir Panteleev wrote:

 On Saturday, 20 September 2014 at 01:19:05 UTC, Walter Bright wrote:

 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization.


 Don't -version, -debug etc. have this problem anyway?


I submit that it is a poor practice to write code that way.



 Anyway, if you change the name mangling, then you'll get link errors instead of
 mysterious crashes.


 It has to be done in such a way that the ARC and the existing exceptions can
 coexist.


 How?


Good question. It's a challenge. But it has to be done, or D will divide in
half 
and both halves will fail.


We've had this discussion numerous times before - "throw the magic compiler 
switch" and D becomes an ARC system and everything is wonderful. It cannot
work. 
ARC and GC are not equivalent.
Sep 19 2014
next sibling parent reply "Daniel N" <ufo orbiting.us> writes:
On Saturday, 20 September 2014 at 02:26:49 UTC, Walter Bright 
wrote:

 Good question. It's a challenge. But it has to be done, or D 
 will divide in half and both halves will fail.


 We've had this discussion numerous times before - "throw the 
 magic compiler switch" and D becomes an ARC system and 
 everything is wonderful. It cannot work. ARC and GC are not 
 equivalent.


I basically agree with Walter on this one, no switch please, it's 
maintenance nightmare for library devs.

My proposal would be to permanently use ARC for Throwable, no 
flags.

What does the GC bring to exceptions that makes it sufficiently 
invaluable to warrant two parallel implementations? It can't be 
about performance, since _thrown_ exceptions are already in the 
slow path... Backwards compatibility?
Sep 19 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 8:13 PM, Daniel N wrote:

 What does the GC bring to exceptions that makes it sufficiently invaluable to
 warrant two parallel implementations? It can't be about performance, since
 _thrown_ exceptions are already in the slow path... Backwards compatibility?


Backwards compatibility.
Sep 19 2014
parent reply "Daniel N" <ufo orbiting.us> writes:
On Saturday, 20 September 2014 at 03:25:39 UTC, Walter Bright 
wrote:

 On 9/19/2014 8:13 PM, Daniel N wrote:

 What does the GC bring to exceptions that makes it 
 sufficiently invaluable to
 warrant two parallel implementations? It can't be about 
 performance, since
 _thrown_ exceptions are already in the slow path... Backwards 
 compatibility?


 Backwards compatibility.


Thanks!

As much as I applaud our efforts for Backwards compatibility, it 
only affects code which already has been written, whereas a 
switch would affect "all" code that will be written, imho the 
burden both on library and app devs would be less if we simply 
redefined Throwable.
Sep 19 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 8:45 PM, Daniel N wrote:

 As much as I applaud our efforts for Backwards compatibility, it only
 affects code which already has been written, whereas a switch would
 affect "all" code that will be written, imho the burden both on library
 and app devs would be less if we simply redefined Throwable.


We need -nogc in the same way we need -m32 and -m64. -- Andrei
Sep 19 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 8:13 PM, Daniel N wrote:

 On Saturday, 20 September 2014 at 02:26:49 UTC, Walter Bright wrote:

 Good question. It's a challenge. But it has to be done, or D will
 divide in half and both halves will fail.


 We've had this discussion numerous times before - "throw the magic
 compiler switch" and D becomes an ARC system and everything is
 wonderful. It cannot work. ARC and GC are not equivalent.


 I basically agree with Walter on this one, no switch please, it's
 maintenance nightmare for library devs.


Why is it a maintenance nightmare?


 My proposal would be to permanently use ARC for Throwable, no flags.


How about other objects? Throwable is but the first step, and a good one 
to inform larger designs. We can't force RC on all objects on all 
applications.


 What does the GC bring to exceptions that makes it sufficiently
 invaluable to warrant two parallel implementations? It can't be about
 performance, since _thrown_ exceptions are already in the slow path...
 Backwards compatibility?


Conversion from Throwable to Object.


Andrei
Sep 19 2014
parent "Daniel N" <ufo orbiting.us> writes:
On Saturday, 20 September 2014 at 04:48:33 UTC, Andrei
Alexandrescu wrote:

 I basically agree with Walter on this one, no switch please, 
 it's
 maintenance nightmare for library devs.


 Why is it a maintenance nightmare?


Sorry, let's defer this, I might have overstated the negative
side-effects before carefully considering methods to mitigate the
impact.


 My proposal would be to permanently use ARC for Throwable, no 
 flags.


 How about other objects? Throwable is but the first step, and a 
 good one to inform larger designs. We can't force RC on all 
 objects on all applications.


I was viewing everything from a totally different perspective:
Normally what prevents an API to be nogc is exceptions, even
_emplace_ falls victim of this, in my line of reasoning I was
counting on the ripple effect to get us quite far, the remaining
issues could be fixed by refining the API to make it possible to
avoid allocations if desired.

However, since you had a different goal in mind, your solution
starts making more sense to me, it's also very easy to flip a
switch and benchmark individual applications, maybe people from
both camps will be surprised at the results.


 What does the GC bring to exceptions that makes it sufficiently
 invaluable to warrant two parallel implementations? It can't 
 be about
 performance, since _thrown_ exceptions are already in the slow 
 path...
 Backwards compatibility?


 Conversion from Throwable to Object.


 Andrei


The conversion could allocate on the GC, which would be
functionally backwards compatible and if the performance
degradation is a concern the code could be updated to avoid the
superfluous allocation by staying in the RCObject hierarchy.

Daniel
Sep 20 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 7:26 PM, Walter Bright wrote:

 On 9/19/2014 7:04 PM, Vladimir Panteleev wrote:

 On Saturday, 20 September 2014 at 01:19:05 UTC, Walter Bright wrote:

 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization.


 Don't -version, -debug etc. have this problem anyway?


 I submit that it is a poor practice to write code that way.



 Anyway, if you change the name mangling, then you'll get link errors
 instead of
 mysterious crashes.


 It has to be done in such a way that the ARC and the existing
 exceptions can
 coexist.


 How?


 Good question. It's a challenge. But it has to be done, or D will divide
 in half and both halves will fail.


I don't think there will be division, and I don't think the parts will 
fail. I see it as enrichment.

We've said for years that D is usable without a GC. The time has come to 
make that into reality. To the extent it's automatic, we'll make it 
automatic. To the extent we'll require adjustment in use, we'll ask 
users to adjust use. The worst thing about that is to enter the design 
with a prejudice tied to our ankles.


 We've had this discussion numerous times before - "throw the magic
 compiler switch" and D becomes an ARC system and everything is
 wonderful. It cannot work. ARC and GC are not equivalent.


I think past discussions have been inconclusive at best and must be 
reopened.


Andrei
Sep 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 9:47 PM, Andrei Alexandrescu wrote:

 We've had this discussion numerous times before - "throw the magic
 compiler switch" and D becomes an ARC system and everything is
 wonderful. It cannot work. ARC and GC are not equivalent.


 I think past discussions have been inconclusive at best and must be reopened.


For starters,

1. ARC is slow and generates bloated code.

2. To deal with (1), existing ARC systems allow escapes from it. This has
severe 
implications for memory safety. Rust's entire type system appears to be
designed 
around trying to deal with this, and AFAIK they're the only ones who have
tried. 
C++ shared_ptr and ObjectiveC's ARC are not memory safe. They are not. Not not 
not, and they don't even try. :-)

3. ARC objects require an embedded count. This means they cannot be mixed with 
non-ARC objects. This is fundamentally DIFFERENT from how GC behaves, and we 
cannot pretend or wish this away or add a compiler switch to make it go away.


None of these points were addressed in previous discussions beyond the 
supposition of some nonexistent compiler technology that would make them go
away.

D can have ref counted objects, but it will not work with a compiler switch to 
switch them back and forth. They'll have to coexist peacefully with GC objects.
Sep 19 2014
next sibling parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 05:24:11 UTC, Walter Bright 
wrote:

 D can have ref counted objects, but it will not work with a 
 compiler switch to switch them back and forth. They'll have to 
 coexist peacefully with GC objects.


If you don't expose the refcount, write libraries as if you have 
only RC, and don't expose destructors then you probably can. Just 
turn inc/dec into dummies.
Sep 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 10:32 PM, Ola Fosheim Grostad wrote:

 On Saturday, 20 September 2014 at 05:24:11 UTC, Walter Bright wrote:

 D can have ref counted objects, but it will not work with a compiler switch to
 switch them back and forth. They'll have to coexist peacefully with GC objects.


 If you don't expose the refcount, write libraries as if you have only RC, and
 don't expose destructors then you probably can. Just turn inc/dec into dummies.


This doesn't address any of the 3 concerns.
Sep 19 2014
parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 06:01:19 UTC, Walter Bright 
wrote:

 This doesn't address any of the 3 concerns.


1. RC efficiency is architecture dependent. E.g. TSX is coming 
even if there is a bug in lower end CPUs. Suggest making 
performance oriented prototypes on different architectures before 
concluding.

2. As long as you have a RC pointer to an obj on the stack you 
can switch to regular references. No magic involved for the easy 
case, just semantic analysis.

3. True, but you can keep the refcount at a negative offset for 
new-based allocations. Besides it only affects those who do  nogc 
and they should know what they are doing.
Sep 20 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 12:17 AM, Ola Fosheim Grostad wrote:

 On Saturday, 20 September 2014 at 06:01:19 UTC, Walter Bright wrote:

 This doesn't address any of the 3 concerns.


 1. RC efficiency is architecture dependent.


Please show me the efficient assembler for the x86.



 E.g. TSX is coming even if there is
 a bug in lower end CPUs. Suggest making performance oriented prototypes on
 different architectures before concluding.


I'd love to see it.



 2. As long as you have a RC pointer to an obj on the stack you can switch to
 regular references. No magic involved for the easy case, just semantic
analysis.


As soon as you pass a reference to a function, that all goes out the window. 
There's a reason why Rust has invested so much effort in the notion of a 
"borrowed" pointer.



 3. True, but you can keep the refcount at a negative offset for new-based
 allocations. Besides it only affects those who do  nogc and they should know
 what they are doing.


If this is so simple, why doesn't everyone do it?
Sep 20 2014
parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 08:39:41 UTC, Walter Bright 
wrote:

 E.g. TSX is coming even if there is
 a bug in lower end CPUs. Suggest making performance oriented 
 prototypes on
 different architectures before concluding.


 I'd love to see it.


A 128 bit CAS instruction is at about 19-25 cycles, but a 
transaction on the other hand can by using xbegin/xend cover both 
refcounting, locking and rollback of multiple objects so you need 
cooperation from code gen. Basically all changes between 
xbegin/xend are kept in cache and written to memory upon success. 
On failure you have a slower fallback.

I don't know what that leads to in amortized cost reduction, but 
30-70% might be possible if it is done right.


 As soon as you pass a reference to a function, that all goes 
 out the window. There's a reason why Rust has invested so much 
 effort in the notion of a "borrowed" pointer.


The pure  nogc crowd care less about safety, but you should be 
able to track this using dataflow?


 3. True, but you can keep the refcount at a negative offset 
 for new-based
 allocations. Besides it only affects those who do  nogc and 
 they should know
 what they are doing.


 If this is so simple, why doesn't everyone do it?


Never said performance and thread safe RC was easy. It is 
probably difficult to get below 10 cycles for inc/dec pairs even 
with excellent code gen...? And probably closer to 40 cycles for 
regular code gen. Just guessing.
Sep 20 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 3:00 AM, Ola Fosheim Grostad wrote:

 On Saturday, 20 September 2014 at 08:39:41 UTC, Walter Bright wrote:

 E.g. TSX is coming even if there is
 a bug in lower end CPUs. Suggest making performance oriented prototypes on
 different architectures before concluding.


 I'd love to see it.


 A 128 bit CAS instruction is at about 19-25 cycles, but a transaction on the
 other hand can by using xbegin/xend cover both refcounting, locking and
rollback
 of multiple objects so you need cooperation from code gen. Basically all
changes
 between xbegin/xend are kept in cache and written to memory upon success. On
 failure you have a slower fallback.

 I don't know what that leads to in amortized cost reduction, but 30-70% might
be
 possible if it is done right.


Please show me the inc/dec optimized x86 code.




 As soon as you pass a reference to a function, that all goes out the window.
 There's a reason why Rust has invested so much effort in the notion of a
 "borrowed" pointer.


 The pure  nogc crowd care less about safety, but you should be able to track
 this using dataflow?


Suppose I pass a pointer to:

    void foo(T* p);

How do I do dataflow?



 3. True, but you can keep the refcount at a negative offset for new-based
 allocations. Besides it only affects those who do  nogc and they should know
 what they are doing.


 If this is so simple, why doesn't everyone do it?


 Never said performance and thread safe RC was easy. It is probably difficult to
 get below 10 cycles for inc/dec pairs even with excellent code gen...? And
 probably closer to 40 cycles for regular code gen. Just guessing.


The question was about mixing RC'd and non-RC'd objects.
Sep 20 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 20 September 2014 at 18:18:05 UTC, Walter Bright 
wrote:

 Please show me the inc/dec optimized x86 code.


There's no optimization of the inc/dec, you use regular inc/dec 
within a larger xbegin/xend block.

You amortize the cost over all the lock-prefixed instructions you 
would otherwise have executed in that block. There is no locking, 
there is a transaction instead. Only if the transaction fails do 
you execute the locking fallback code.


 Suppose I pass a pointer to:

    void foo(T* p);

 How do I do dataflow?


You mean a separate compilation unit that the compiler don't have 
access to? You don't, obviously.


 The question was about mixing RC'd and non-RC'd objects.


Well, if you are talking about negative offset for ref counts, 
then people do it. E.g. there are examples of systems that 
provide C-compatible strings with length at offset -4 and ref 
count at -8.

If you need to distinguish you can either use address-space info 
or keep a negative int in the non-RC'd object.
Sep 20 2014
next sibling parent "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 18:26:56 UTC, Ola Fosheim 
Grøstad wrote:

 On Saturday, 20 September 2014 at 18:18:05 UTC, Walter Bright 
 wrote:

 Please show me the inc/dec optimized x86 code.


 There's no optimization of the inc/dec, you use regular inc/dec 
 within a larger xbegin/xend block.


Actually you dont need to inc/dec if you do both in the 
transaction. You only need to read the refcount. If somebody else 
writes to the ref count the transaction fails.

So, you cannot succeed a transaction while somebody else decrease 
the ref count.

Sounds right?
Sep 20 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 11:26 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Saturday, 20 September 2014 at 18:18:05 UTC, Walter Bright wrote:

 Please show me the inc/dec optimized x86 code.


 There's no optimization of the inc/dec, you use regular inc/dec within a larger
 xbegin/xend block.

 You amortize the cost over all the lock-prefixed instructions you would
 otherwise have executed in that block. There is no locking, there is a
 transaction instead. Only if the transaction fails do you execute the locking
 fallback code.


I strongly suggest taking a look at C++ shared_ptr<T>, compile a simple
example, 
and examine the generated code. It is NOT AT ALL as simple as emitting an 
inc/dec pair. I.e. what happens if an exception is thrown between the inc and 
the dec?

If RC was as costless and simple as you are suggesting, everyone would be doing 
it and would have for decades.



 Suppose I pass a pointer to:

    void foo(T* p);

 How do I do dataflow?


 You mean a separate compilation unit that the compiler don't have access to?
You
 don't, obviously.


Right. "dataflow" is not the answer.



 The question was about mixing RC'd and non-RC'd objects.


 Well, if you are talking about negative offset for ref counts, then people do
 it. E.g. there are examples of systems that provide C-compatible strings with
 length at offset -4 and ref count at -8.


Now embed that object as a field in some other object.



 If you need to distinguish you can either use address-space info or keep a
 negative int in the non-RC'd object.


Now add that runtime distinguishing logic to the inc/dec code.
Sep 20 2014
parent "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 22:07:45 UTC, Walter Bright 
wrote:

 On 9/20/2014 11:26 AM, "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Saturday, 20 September 2014 at 18:18:05 UTC, Walter Bright 
 wrote:

 Please show me the inc/dec optimized x86 code.


 There's no optimization of the inc/dec, you use regular 
 inc/dec within a larger
 xbegin/xend block.

 You amortize the cost over all the lock-prefixed instructions 
 you would
 otherwise have executed in that block. There is no locking, 
 there is a
 transaction instead. Only if the transaction fails do you 
 execute the locking
 fallback code.


 I strongly suggest taking a look at C++ shared_ptr<T>, compile 
 a simple example, and examine the generated code. It is NOT AT 
 ALL as simple as emitting an inc/dec pair. I.e. what happens if 
 an exception is thrown between the inc and the dec?


C++ provides weak references. If you throw issue xabort.

Nobody have said it is free, and TSX is new tech. So what you can 
do, who knows?


 If RC was as costless and simple as you are suggesting, 
 everyone would be doing it and would have for decades.


Transactions are not free.


 Now embed that object as a field in some other object.


Uhm, I referred to new-allocated objects.


 Now add that runtime distinguishing logic to the inc/dec code.


Test N flag?
Sep 20 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 10:24 PM, Walter Bright wrote:

 On 9/19/2014 9:47 PM, Andrei Alexandrescu wrote:

 We've had this discussion numerous times before - "throw the magic
 compiler switch" and D becomes an ARC system and everything is
 wonderful. It cannot work. ARC and GC are not equivalent.


 I think past discussions have been inconclusive at best and must be
 reopened.


 For starters,

 1. ARC is slow and generates bloated code.


Agreed. It has the cost distributed a different way.


 2. To deal with (1), existing ARC systems allow escapes from it. This
 has severe implications for memory safety. Rust's entire type system
 appears to be designed around trying to deal with this, and AFAIK
 they're the only ones who have tried. C++ shared_ptr and ObjectiveC's
 ARC are not memory safe. They are not. Not not not, and they don't even
 try. :-)


That's fine. You can't have everything.


 3. ARC objects require an embedded count. This means they cannot be
 mixed with non-ARC objects. This is fundamentally DIFFERENT from how GC
 behaves, and we cannot pretend or wish this away or add a compiler
 switch to make it go away.


In nogc mode, everything is ARC so I'm not getting this.


 None of these points were addressed in previous discussions beyond the
 supposition of some nonexistent compiler technology that would make them
 go away.


That was indeed a crappy argument, and not advanced by me. I'm not 
advocating for that.


 D can have ref counted objects, but it will not work with a compiler
 switch to switch them back and forth. They'll have to coexist peacefully
 with GC objects.


We need to figure out a design. All I'm saying is we must not bring 
prejudice to the table. And a very basic point is: there will be a way 
to COMPLETELY disable the GC. That is a must.


Andrei
Sep 19 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 10:46 PM, Andrei Alexandrescu wrote:

 2. To deal with (1), existing ARC systems allow escapes from it. This
 has severe implications for memory safety. Rust's entire type system
 appears to be designed around trying to deal with this, and AFAIK
 they're the only ones who have tried. C++ shared_ptr and ObjectiveC's
 ARC are not memory safe. They are not. Not not not, and they don't even
 try. :-)


 That's fine. You can't have everything.


You're willing to dispense with memory safety?



 3. ARC objects require an embedded count. This means they cannot be
 mixed with non-ARC objects. This is fundamentally DIFFERENT from how GC
 behaves, and we cannot pretend or wish this away or add a compiler
 switch to make it go away.


 In nogc mode, everything is ARC so I'm not getting this.


Not even ARC languages try to make everything ARC - because it is BLOATED and 
SLOW. Not C++, not ObjectiveC, not Rust.



 D can have ref counted objects, but it will not work with a compiler
 switch to switch them back and forth. They'll have to coexist peacefully
 with GC objects.


 We need to figure out a design. All I'm saying is we must not bring prejudice
to
 the table. And a very basic point is: there will be a way to COMPLETELY disable
 the GC. That is a must.


It doesn't have to be disabled. Just don't call it. The GC is not a bridge
troll 
that is going to pop up and extract a toll even if you aren't allocating with
it.
Sep 19 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 11:15 PM, Walter Bright wrote:

 On 9/19/2014 10:46 PM, Andrei Alexandrescu wrote:

 2. To deal with (1), existing ARC systems allow escapes from it. This
 has severe implications for memory safety. Rust's entire type system
 appears to be designed around trying to deal with this, and AFAIK
 they're the only ones who have tried. C++ shared_ptr and ObjectiveC's
 ARC are not memory safe. They are not. Not not not, and they don't even
 try. :-)


 That's fine. You can't have everything.


 You're willing to dispense with memory safety?


No.


 3. ARC objects require an embedded count. This means they cannot be
 mixed with non-ARC objects. This is fundamentally DIFFERENT from how GC
 behaves, and we cannot pretend or wish this away or add a compiler
 switch to make it go away.


 In nogc mode, everything is ARC so I'm not getting this.


 Not even ARC languages try to make everything ARC - because it is
 BLOATED and SLOW. Not C++, not ObjectiveC, not Rust.


Agreed.


 D can have ref counted objects, but it will not work with a compiler
 switch to switch them back and forth. They'll have to coexist peacefully
 with GC objects.


 We need to figure out a design. All I'm saying is we must not bring
 prejudice to
 the table. And a very basic point is: there will be a way to
 COMPLETELY disable
 the GC. That is a must.


 It doesn't have to be disabled. Just don't call it. The GC is not a
 bridge troll that is going to pop up and extract a toll even if you
 aren't allocating with it.


Well obviously there's a hierarchy of desirability:

1. Using the GC inadvertently causes a compile-time error.

2. Using the GC inadvertently causes a link-time error.

3. Using the GC inadvertently causes a run-time error.

4. Using the GC inadvertently goes undetected.

I disagree that (4) is good for everyone.


Andrei
Sep 20 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 8:59 AM, Andrei Alexandrescu wrote:

 4. Using the GC inadvertently goes undetected.

 I disagree that (4) is good for everyone.


We've already implemented  nogc.
Sep 20 2014
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 05:46:44 UTC, Andrei 
Alexandrescu wrote:

 D can have ref counted objects, but it will not work with a 
 compiler
 switch to switch them back and forth. They'll have to coexist 
 peacefully
 with GC objects.


 We need to figure out a design. All I'm saying is we must not 
 bring prejudice to the table. And a very basic point is: there 
 will be a way to COMPLETELY disable the GC. That is a must.


The ref counted objects still need to be considered as roots for 
the GC. This limits how much "completely" can be achieved.
Sep 20 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 02:26:49 UTC, Walter Bright 
wrote:

 On 9/19/2014 7:04 PM, Vladimir Panteleev wrote:

 On Saturday, 20 September 2014 at 01:19:05 UTC, Walter Bright 
 wrote:

 Having a compiler switch to change the behavior of every 
 module in
 incompatible ways would be a disastrous balkanization.


 Don't -version, -debug etc. have this problem anyway?


 I submit that it is a poor practice to write code that way.



 Anyway, if you change the name mangling, then you'll get link 
 errors instead of
 mysterious crashes.


 It has to be done in such a way that the ARC and the existing 
 exceptions can
 coexist.


 How?


 Good question. It's a challenge. But it has to be done, or D 
 will divide in half and both halves will fail.


 We've had this discussion numerous times before - "throw the 
 magic compiler switch" and D becomes an ARC system and 
 everything is wonderful. It cannot work. ARC and GC are not 
 equivalent.


I am fully with Walter here. Only way to do it in acceptable way 
is to make both  reference counted exceptions and current ones 
co-exist. Anything else means that we will need to distribute 4 
builds of Phobos - debug-nogc, debug-gc, release-nogc, 
release-gc. Guess you can smell exponential explosion here as 
well as I do.
Sep 20 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 19 Sep 2014 08:32:38 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 First, there must be some compiler flag -nogc or something, which=20


please, no compiler switches! it's partly ok for "-release", it's not
very good for "-version" and it's awful to add another one which
completely alters the way something works. as much as i want to make
throwables nogc, it's better to change nothing than to add such switch.
Sep 19 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 8:30 PM, ketmar via Digitalmars-d wrote:

 On Fri, 19 Sep 2014 08:32:38 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:


 First, there must be some compiler flag -nogc or something, which


 please, no compiler switches! it's partly ok for "-release", it's not
 very good for "-version" and it's awful to add another one which
 completely alters the way something works. as much as i want to make
 throwables nogc, it's better to change nothing than to add such switch.


Changing nothing is not an option. Ideas? -- Andrei
Sep 19 2014
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 19 Sep 2014 21:53:32 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 Changing nothing is not an option. Ideas? -- Andrei


let compiler do it's ARC magic on Throwable and allow programmer to
plug in his own allocator. and drop out GC mechanics for Throwables
entirely (except scanning Throwable instance for GC-managed pointers).
Sep 19 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 04:53:32 UTC, Andrei 
Alexandrescu wrote:

 On 9/19/14, 8:30 PM, ketmar via Digitalmars-d wrote:

 On Fri, 19 Sep 2014 08:32:38 -0700
 Andrei Alexandrescu via Digitalmars-d 
 <digitalmars-d puremagic.com>
 wrote:


 First, there must be some compiler flag -nogc or something, 
 which


 please, no compiler switches! it's partly ok for "-release", 
 it's not
 very good for "-version" and it's awful to add another one 
 which
 completely alters the way something works. as much as i want 
 to make
 throwables nogc, it's better to change nothing than to add 
 such switch.


 Changing nothing is not an option. Ideas? -- Andrei


I have been thinking about something like `ARC` interface and 
`ARCException : Throwable, ARC` with all the automatic ref 
inc/dec calls you have mentioned added for any class that 
implements `ARC` interface.
Sep 20 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 07:29:52 UTC, Dicebot wrote:

 I have been thinking about something like `ARC` interface and 
 `ARCException : Throwable, ARC` with all the automatic ref 
 inc/dec calls you have mentioned added for any class that 
 implements `ARC` interface.


..and yes, I'd expect it to be the default one used by Phobos 
(possibly with some backing pool allocator). It isn't critical 
for GC-enabled applications to use GC-based exceptions but 
contrary is true.
Sep 20 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Consider:

https://github.com/D-Programming-Language/phobos/blob/master/std/c/windows/com.d#L218

This is D's support for COM objects. They have AddRef() and Release(), i.e.
they 
are reference counted. They do not inherit from Object. The refcounting is 
currently manual.

D could allow an additional root (not a replacement) for thrown Exceptions.
This 
root would be based on ComObject.

As a bonus, COM objects were designed for C++, and C++ implements them well, 
i.e. they would be a compatible object structure.

catch statements could be changed to catch ComThrowable, which would not catch 
Throwables, and catches for Throwables would not catch ComThrowables.
Sep 19 2014
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
19-Sep-2014 19:32, Andrei Alexandrescu пишет:

 As discussed, having exception objects being GC-allocated is clearly a
 large liability that we need to address. They prevent otherwise careful
 functions from being  nogc so they affect even apps that otherwise would
 be okay with a little litter here and there.

 The Throwable hierarchy is somewhat separate from everything else, which
 makes it a great starting point for investigating an automated reference
 count approach. Here's what I'm thinking.

 First, there must be some compiler flag -nogc or something, which
 triggers the RC exceptions. All modules of an application must be
 compiled with this flag if it is to work (such that one module can throw
 an exception caught by the other). Of course a lot of refinement needs
 to be added here (what happens if one tries to link modules built with
 and without -nogc, allowing people to detect the flag programmatically
 by using version(nogc) etc).

 If -nogc is passed, the compiler severs the inheritance relationship
 between Throwable and Object, making it impossible to convert a
 Throwable to an Object. From then henceforth, Throwable and Object form
 a two-rooted forest. (In all likelihood we'll later add an RCObject root
 that Throwable inherits.)

 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.) Implementation of these is up to the runtime library. Null
 checking may be left to either the compiler or the library (in the
 latter case, the functions must be nonmember). However it seems the
 compiler may have an advantage because it can elide some of these checks.

 Once we get this going, we should accumulate good experience that we can
 later apply to generalizing this approach to more objects. Also, if
 things go well we may as well define _always_ (whether GC or not)
 Throwable to be reference counted; seems like a good fit for all programs.


All good, it wasn't a good idea to have GC-ed exceptions in the first 
place (It's a truly rare thing to have cycles of exceptions). It doesn' 
t matter GC or  nogc - this must go.

Speaking of RC-ed Throwable - why adding a switch? Just make a 
transition in 2 steps - all exceptions inherit from RCObject (but stay 
essentially GC-ed), then hack compiler to perform ARC on RCObject 
descendants (maybe even with  GC.malloc/GC.free for starters, i.e. the 
same heap). Then we can mess with custom allocation of these.

I can understand a transitory switch, say -exception-ongc for the 
minority of folks that indeed DO rely on exceptions being GC and not 
counted.


-- 
Dmitry Olshansky
Sep 20 2014
prev sibling next sibling parent reply "Uranuz" <neuranuz gmail.com> writes:
I'm quite a noobie in memory models but from position of user of 
D language I have some ideas about syntax of switching between 
memory models. I think that having everywhere declarations using 
wrapper structs (like RefCounted) is not very user-friendly. But 
still we need some way to say to compiler how we want to do 
memory management. I have read tutorial about Rust language. It 
looks not very clear for me, but I like the idea that type of 
memory management is included in variable declaration.

Intead of wrapper struct like scoped!A or RefCounted!A we could 
declare variables with annotations (attributes in D) to say what 
memory model we want to use. Compiler should understand this 
annotation and create some code for memory management. I thing 
it's possible to declare some interface (maybe struct with 
compile-time duck typing) to support some user-defined memory 
models or modifications of basic (implemented in a language) 
memory models.

Why I am saying about annotations? Because we can annotate some 
function, class or just block at the module scope that should use 
some sort of memory management and compiler will create 
corresponding code. In that case we don't need to put wrapper 
struct around all variables that use ref counting. We just 
annotate some function or class declaration as ref-counted and 
that's it!

It's just a concept of what I like to see in the language design 
of D in future)) Of course there are a lot of problems in a 
practice.

Also I think that we shouldn't impose some way of memory model to 
programmer and let him choose his approach but syntax should be 
simple and clear as much as it could be. Also some defaults 
should be for most common cases and for *novice* users of D.

Waiting for critic or thoughts!!!))
Sep 20 2014
next sibling parent reply "Uranuz" <neuranuz gmail.com> writes:
Also it's interesting waht is the correspondence between memory 
management, memory model and allocators? I know that 
std.allocator is in development. I guess that it should be 
considered in complex.
Sep 20 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 1:46 AM, Uranuz wrote:

 Also it's interesting waht is the correspondence between memory
 management, memory model and allocators? I know that std.allocator is in
 development. I guess that it should be considered in complex.


1. Memory management: when are calls to allocation and deallocation 
primitives made?

2. Allocators: what are the allocation and deallocation primitives, and 
how are they implemented?

3. Memory model: how is memory effected by programs? (of interest for 
compiler implementers and thread safety)

There is occasional confusion among the three. I myself was in the woods 
for a while thinking that allocators should do memory management.


Andrei
Sep 20 2014
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 08:42:51 UTC, Uranuz wrote:

 I'm quite a noobie in memory models but from position of user 
 of D language I have some ideas about syntax of switching 
 between memory models. I think that having everywhere 
 declarations using wrapper structs (like RefCounted) is not 
 very user-friendly. But still we need some way to say to 
 compiler how we want to do memory management. I have read 
 tutorial about Rust language. It looks not very clear for me, 
 but I like the idea that type of memory management is included 
 in variable declaration.

 Intead of wrapper struct like scoped!A or RefCounted!A we could 
 declare variables with annotations (attributes in D) to say 
 what memory model we want to use. Compiler should understand 
 this annotation and create some code for memory management. I 
 thing it's possible to declare some interface (maybe struct 
 with compile-time duck typing) to support some user-defined 
 memory models or modifications of basic (implemented in a 
 language) memory models.

 Why I am saying about annotations? Because we can annotate some 
 function, class or just block at the module scope that should 
 use some sort of memory management and compiler will create 
 corresponding code. In that case we don't need to put wrapper 
 struct around all variables that use ref counting. We just 
 annotate some function or class declaration as ref-counted and 
 that's it!

 It's just a concept of what I like to see in the language 
 design of D in future)) Of course there are a lot of problems 
 in a practice.

 Also I think that we shouldn't impose some way of memory model 
 to programmer and let him choose his approach but syntax should 
 be simple and clear as much as it could be. Also some defaults 
 should be for most common cases and for *novice* users of D.

 Waiting for critic or thoughts!!!))


You are right that there is an intricate relationship between the 
various types of memory (better: ownership) management, 
allocators, ref-counting, uniqueness, moving etc, as you write in 
your other post. They interact in very specific ways. For this 
reason I don't think it is feasible or desirable to set the type 
of memory management "from the outside", the types in question 
need to know how their innards work. In the same vein, if your 
code uses a specific memory management strategy, it has to be 
written in a way conforming to it. Walter has already stated 
that: you cannot simply slap an attribute onto your code saying 
"please do reference counting", and expect it to work, especially 
not efficiently.

I think we can get a lot further if we work out said 
relationships, and create good general purpose wrapper types that 
implement the different strategies accordingly. I believe, 
borrowing is crucial to this, which is why I made a proposal 
about it [1], which also deals with the other related topics.

You do have a point about a simple, accessible syntax being 
important. Still, I don't see a big difference between 
 annotations and RC!T wrappers in this regard (except that the 
former could apply to entire sections of code, which is not a 
good idea IMO). And I think with auto/scope/const type deduction, 
code is quite pleasant to read.

[1] http://wiki.dlang.org/User:Schuetzm/scope
Sep 20 2014
prev sibling next sibling parent reply "ponce" <contact gam3sfrommars.fr> writes:
On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu 
wrote:

 Please chime in with thoughts.


Coudln't we throw value types instead? (like in C++)
Sep 20 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 09:05:15 UTC, ponce wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:

 Please chime in with thoughts.


 Coudln't we throw value types instead? (like in C++)


But how would you chain them? And they have to implement 
Throwable's interface.
Sep 20 2014
parent reply "ponce" <contact gam3sfrommars.fr> writes:
On Saturday, 20 September 2014 at 09:09:38 UTC, Marc Schütz wrote:

 On Saturday, 20 September 2014 at 09:05:15 UTC, ponce wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:

 Please chime in with thoughts.


 Coudln't we throw value types instead? (like in C++)


 But how would you chain them?


Owning pointer to next exception.


 And they have to implement Throwable's interface.


One of their member would. Speculative design:

struct ValueException
{
      Unique!ValueException next;
      Unique!Throwable content;
}

Language would change to throw structs and catch them (without
matching).
At the end of a catch bloc, its destructor is called.
This breaks polymorphic catch so it need sugar unfortunately.
Sep 20 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 15:28:10 UTC, ponce wrote:

 On Saturday, 20 September 2014 at 09:09:38 UTC, Marc Schütz 
 wrote:

 On Saturday, 20 September 2014 at 09:05:15 UTC, ponce wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:

 Please chime in with thoughts.


 Coudln't we throw value types instead? (like in C++)


 But how would you chain them?


 Owning pointer to next exception.


 And they have to implement Throwable's interface.


 One of their member would. Speculative design:

 struct ValueException
 {
      Unique!ValueException next;
      Unique!Throwable content;
 }

 Language would change to throw structs and catch them (without
 matching).
 At the end of a catch bloc, its destructor is called.
 This breaks polymorphic catch so it need sugar unfortunately.


Ah, now I see, you just want a wrapper that contains a Throwable.
I thought you wanted to throw _any_ value types, like in C++.
Sep 20 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 2:05 AM, ponce wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:

 Please chime in with thoughts.


 Coudln't we throw value types instead? (like in C++)


My knee-jerk reaction is we shouldn't. If considered, that would need to 
be well argued. We've derived good benefits from using polymorphic 
exceptions. -- Andrei
Sep 20 2014
parent reply "ponce" <contact gam3sfrommars.fr> writes:
On Saturday, 20 September 2014 at 16:41:20 UTC, Andrei 
Alexandrescu wrote:

 On 9/20/14, 2:05 AM, ponce wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:

 Please chime in with thoughts.


 Coudln't we throw value types instead? (like in C++)


 My knee-jerk reaction is we shouldn't. If considered, that 
 would need to be well argued. We've derived good benefits from 
 using polymorphic exceptions. -- Andrei


What do you think of the solution briefly outlined here:
http://forum.dlang.org/post/iqenumncmczqvprwutmg forum.dlang.org
Sep 20 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 11:17 AM, ponce wrote:

 On Saturday, 20 September 2014 at 16:41:20 UTC, Andrei Alexandrescu wrote:

 On 9/20/14, 2:05 AM, ponce wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:

 Please chime in with thoughts.


 Coudln't we throw value types instead? (like in C++)


 My knee-jerk reaction is we shouldn't. If considered, that would need
 to be well argued. We've derived good benefits from using polymorphic
 exceptions. -- Andrei


 What do you think of the solution briefly outlined here:
 http://forum.dlang.org/post/iqenumncmczqvprwutmg forum.dlang.org


The short answer is I wouldn't advise pursuing it. -- Andrei
Sep 20 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-19 17:32, Andrei Alexandrescu wrote:


 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.


Assuming this would eventually be implemented for regular classes, it 
would be nice if it could be made compatible with Objective-C ARC [1].

[1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html

-- 
/Jacob Carlborg
Sep 20 2014
next sibling parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 12:27:23 UTC, Jacob Carlborg 
wrote:

 Assuming this would eventually be implemented for regular 
 classes, it would be nice if it could be made compatible with 
 Objective-C ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html


Isn't Objective-C ARC known to be terribly efficient?
Sep 20 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 20.09.2014 15:40, schrieb Ola Fosheim Grostad:

 On Saturday, 20 September 2014 at 12:27:23 UTC, Jacob Carlborg wrote:

 Assuming this would eventually be implemented for regular classes, it
 would be nice if it could be made compatible with Objective-C ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html


 Isn't Objective-C ARC known to be terribly efficient?


It requires compiler support, though.

- Place retain/release at each place expected by Cocoa programming 
conventions, as if manually written

- Do a second pass with data-flow analysis to remove retain/release 
pairs that don't escape current scope.

If you have an Apple account search for:

"Transitioning to ARC Release Notes", ARC
"Memory Management Programming Guide for Core Foundation" - Cocoa 
conventions and allocators
"WWDC 2012: Adopting Automatic Reference Counting" - Overral description
"WWDC 2012: What's New in LLVM" - Some slides about ARC Optimizer

--
Paulo
Sep 20 2014
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 20 September 2014 at 14:33:40 UTC, Paulo Pinto wrote:

 Am 20.09.2014 15:40, schrieb Ola Fosheim Grostad:

 On Saturday, 20 September 2014 at 12:27:23 UTC, Jacob Carlborg 
 wrote:

 Assuming this would eventually be implemented for regular 
 classes, it
 would be nice if it could be made compatible with Objective-C 
 ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html


 Isn't Objective-C ARC known to be terribly efficient?




I'm really sorry, but that was a very confusing typo on my part.

I meant "INefficient"! :-)


 It requires compiler support, though.


:-) Yes, but I was more talking about the representation in 
memory based on reading the source code that can be found in the 
repository. I read it the last time ARC was discussed…

But, I am not 100% sure what code Apple actually ships.
Sep 20 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-20 16:33, Paulo Pinto wrote:


 It requires compiler support, though.


The first thing I asked in this thread was "Are you suggesting we 
implement ARC?" and the answer was "Yes" [1]. So it looks like Andrei 
already wants to implement ARC. My definition of ARC is that the 
compiler inserts the calls to retain/release (or whatever you call them).

[1] 
http://forum.dlang.org/thread/lvhiam$1pno$1 digitalmars.com#post-lvi0ve:2429il:241:40digitalmars.com

-- 
/Jacob Carlborg
Sep 20 2014
next sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 20.09.2014 17:08, schrieb Jacob Carlborg:

 On 2014-09-20 16:33, Paulo Pinto wrote:


 It requires compiler support, though.


 The first thing I asked in this thread was "Are you suggesting we
 implement ARC?" and the answer was "Yes" [1]. So it looks like Andrei
 already wants to implement ARC. My definition of ARC is that the
 compiler inserts the calls to retain/release (or whatever you call them).

 [1]
 http://forum.dlang.org/thread/lvhiam$1pno$1 digitalmars.com#post-lvi0ve:2429il:241:40digitalmars.com


I would say ARC == RC. I never saw a distinction in literature between 
both, before Apple used the term.

With all these discussions I have been digging into old papers, and due 
to that I am starting to change my opinion.

As it seems, RC with GC for cycle collection was more common than I 
thought of.

In the end, what matters is having automatic memory management, be it 
via GC, RC or compiler dataflow analysis.

 From an outsider point of view, I just think that whatever the final 
outcome, it should be in a way that avoids the runtime episode from 
repeating itself.

--
Paulo
Sep 20 2014
next sibling parent "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 16:52:59 UTC, Paulo Pinto wrote:

 As it seems, RC with GC for cycle collection was more common 
 than I thought of.


I once read a paper that suggested optimizing GC by having RC on 
all objects and stop inc/dec ref counting when it reached 5 or 
something. I guess the idea was to get rid of short lived objects 
to make the GC run less often while cutting RC overhead.
Sep 20 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-20 18:53, Paulo Pinto wrote:


 I would say ARC == RC. I never saw a distinction in literature between
 both, before Apple used the term.


I never saw the term ARC before Apple used it. I would say, ARC is a 
form of RC but RC doesn't not need imply ARC. BTW "Automatic Reference 
Counting" on Wikipedia [1] basically only talks about Apple's ARC.

[1] http://en.wikipedia.org/wiki/Automatic_Reference_Counting

-- 
/Jacob Carlborg
Sep 21 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 8:08 AM, Jacob Carlborg wrote:

 On 2014-09-20 16:33, Paulo Pinto wrote:


 It requires compiler support, though.


 The first thing I asked in this thread was "Are you suggesting we
 implement ARC?" and the answer was "Yes" [1]. So it looks like Andrei
 already wants to implement ARC. My definition of ARC is that the
 compiler inserts the calls to retain/release (or whatever you call them).

 [1]
 http://forum.dlang.org/thread/lvhiam$1pno$1 digitalmars.com#post-lvi0ve:2429il:241:40digitalmars.com


Please don't take me in a court of law. But yes, I am talking about the 
compiler inserting calls to increment and decrement reference counts. -- 
Andrei
Sep 20 2014
next sibling parent reply "Mike" <none none.com> writes:
On Saturday, 20 September 2014 at 16:56:32 UTC, Andrei 
Alexandrescu wrote:

 On 9/20/14, 8:08 AM, Jacob Carlborg wrote:

 On 2014-09-20 16:33, Paulo Pinto wrote:


 It requires compiler support, though.


 The first thing I asked in this thread was "Are you suggesting 
 we
 implement ARC?" and the answer was "Yes" [1]. So it looks like 
 Andrei
 already wants to implement ARC. My definition of ARC is that 
 the
 compiler inserts the calls to retain/release (or whatever you 
 call them).

 [1]
 http://forum.dlang.org/thread/lvhiam$1pno$1 digitalmars.com#post-lvi0ve:2429il:241:40digitalmars.com


 Please don't take me in a court of law. But yes, I am talking 
 about the compiler inserting calls to increment and decrement 
 reference counts. -- Andrei


I suggest the compiler insert calls to whatever "lifetime events" 
may cause an increment/decrement.  Then, the druntime can provide 
the implementation.

I believe with these runtime hooks in place platform-specific 
optimizations and even alternative memory managers can be 
experimented with.

I refer you to an enlightening quote I once found on a raytracing 
site that really captures the beauty of software:
"You know you've been raytracing too long when you've been asked 
how you did that thing you did by the author of the raytracer you 
used to do it."

Just build the capability, and let the community surprise you 
with their ingenuity.

Mike
Sep 20 2014
parent "Daniel N" <ufo orbiting.us> writes:
On Sunday, 21 September 2014 at 03:28:03 UTC, Mike wrote:

 I suggest the compiler insert calls to whatever "lifetime 
 events" may cause an increment/decrement.  Then, the druntime 
 can provide the implementation.

 I believe with these runtime hooks in place platform-specific 
 optimizations and even alternative memory managers can be 
 experimented with.


Your post gave me an idea already, so far we've only been 
focusing on Throwable, however... the solution for Error:s and 
Exception:s is actually asymmetric!

For the vast majority of applications the best possible 
implementation for Error:s is to simply leak the memory, this is 
definitely not true for Exceptions(but it could be true for a 
small subset of applications, short-lived tools).

Daniel
Sep 20 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-20 18:56, Andrei Alexandrescu wrote:


 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference counts. --
 Andrei


We do need to know what you're proposal is for. How else can we comment 
on it? Paulo Pinto's comment "It requires compiler support, though" 
suggests, at least to me, he didn't understand you suggested ARC, i.e. 
RC with compiler support.

-- 
/Jacob Carlborg
Sep 21 2014
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 21.09.2014 10:51, schrieb Jacob Carlborg:

 On 2014-09-20 18:56, Andrei Alexandrescu wrote:


 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference counts. --
 Andrei


 We do need to know what you're proposal is for. How else can we comment
 on it? Paulo Pinto's comment "It requires compiler support, though"
 suggests, at least to me, he didn't understand you suggested ARC, i.e.
 RC with compiler support.


Sorry but I guess I did understand it. There are only four ways of doing 
RC, regardless how you name them.

- programmer writes manually calls to increment/decrement counters, like 
old time Cocoa and COM

- library types which take advantage of operator overloading for 
increment/decrement operations, like *_ptr<>(), ComPtr<>() and RefCounted

- automatic increment/decrement operations via special type or code 
pattern recognition => compiler support like Swift, Objective-C, 
C++/CLI, C++/CLX, Swift, Cedar, Modula-2+

- automatic increment/decrement operations via special type or code 
pattern recognition, followed by code removal of needless call pairs =>
compiler support like Objective-C and Swift

So given Andrei's comment, compiler support is required.

The only thing that distinguishes Apple solution from other ones, is 
that they made a marketing name for a common RC optimization.

--
Paulo
Sep 21 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 1:51 AM, Jacob Carlborg wrote:

 On 2014-09-20 18:56, Andrei Alexandrescu wrote:


 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference counts. --
 Andrei


 We do need to know what you're proposal is for.


The title is an incomplete summary. Exceptions would be transparently 
reference counted. I.e. no changes in user code.

A little more detail on a design Walter has put together: define a new 
root class RCObject. Neither Object nor RCObject inherit each other. All 
classes that inherit RCObject are detected by the compiler as reference 
counted.

(For practical reasons, RCObject will implement IUnknown. There are a 
few random consequences of that (meaningless method QueryInterface, no 
class info, no monitor, COM-compatible layout) but IUnknown already has 
AddRef and Release and the compiler already recognizes IUnknown as a 
special case. Hooking RCObject there will leave us with two roots 
instead of adding a third one.)

Throwable will inherit RCObject. The compiler inserts AddRef and Release 
whenever copying RCObject (and derivatives) references around, of course 
following a null check.

The caveats discussed previously in http://goo.gl/Xl5U3z do apply.


Andrei
Sep 21 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 21 September 2014 at 15:03:09 UTC, Andrei Alexandrescu 
wrote:

 On 9/21/14, 1:51 AM, Jacob Carlborg wrote:

 On 2014-09-20 18:56, Andrei Alexandrescu wrote:


 Please don't take me in a court of law. But yes, I am talking 
 about the
 compiler inserting calls to increment and decrement reference 
 counts. --
 Andrei


 We do need to know what you're proposal is for.


 The title is an incomplete summary. Exceptions would be 
 transparently reference counted. I.e. no changes in user code.

 A little more detail on a design Walter has put together: 
 define a new root class RCObject. Neither Object nor RCObject 
 inherit each other. All classes that inherit RCObject are 
 detected by the compiler as reference counted.

 (For practical reasons, RCObject will implement IUnknown. There 
 are a few random consequences of that (meaningless method 
 QueryInterface, no class info, no monitor, COM-compatible 
 layout) but IUnknown already has AddRef and Release and the 
 compiler already recognizes IUnknown as a special case. Hooking 
 RCObject there will leave us with two roots instead of adding a 
 third one.)

 Throwable will inherit RCObject. The compiler inserts AddRef 
 and Release whenever copying RCObject (and derivatives) 
 references around, of course following a null check.

 The caveats discussed previously in http://goo.gl/Xl5U3z do 
 apply.


 Andrei


This makes much more sense to me than initial proposal. I may 
even be brave enough to say that I like it :)
Sep 21 2014
prev sibling next sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/21/14, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 (For practical reasons, RCObject will implement IUnknown.


IUnknown has this issue btw:
https://issues.dlang.org/show_bug.cgi?id=12607
Sep 21 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 9:42 AM, Andrej Mitrovic via Digitalmars-d wrote:

 On 9/21/14, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 (For practical reasons, RCObject will implement IUnknown.


 IUnknown has this issue btw:
 https://issues.dlang.org/show_bug.cgi?id=12607


Thanks, good to know. IMHO having RCObject inherit IUnknown is more of a 
distraction than a benefit, but I'll let Walter be the judge of that. -- 
Andrei
Sep 21 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 21 September 2014 at 18:10:53 UTC, Andrei Alexandrescu 
wrote:

 Thanks, good to know. IMHO having RCObject inherit IUnknown is 
 more of a distraction than a benefit, but I'll let Walter be 
 the judge of that. -- Andrei


Actually, I think it is a good idea to do the refcounting thing 
on COM objects anyway. I've used wrapper structs for that in the 
past, but if you don't do that right you can escape a reference 
which is freed, so putting it right on the object is kinda useful.

Of course, IUnknown AND RCObject could be refcounted roots.
Sep 21 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
BTW hmmm what about this:

interface Foo { }

class Bar : Foo, RCObject {}

class Pwned : Foo {}

void main() {
    Foo bar = new Bar();
    /* where is bar.Release() called? */
    Foo pwned = new Pwned();
    /* better hope pwned.Release() isn't called cuz that's 
impossible */
}


I *believe* IUnknown interfaces are special and a separate 
category from regular D interfaces, so this would be statically 
disallowed.

But with the RCObject, we need to be careful about implicit 
casting to interfaces transparently killing memory safety.

of course then we might be weakening their usefulness. gah we 
just need scope references, then we can pass the interface with 
confidence that it won't be escaped. yeah yeah i know i sound 
like a broken record.
Sep 21 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 11:20 AM, Adam D. Ruppe wrote:

 BTW hmmm what about this:

 interface Foo { }

 class Bar : Foo, RCObject {}

 class Pwned : Foo {}

 void main() {
     Foo bar = new Bar();
     /* where is bar.Release() called? */
     Foo pwned = new Pwned();
     /* better hope pwned.Release() isn't called cuz that's impossible */
 }


 I *believe* IUnknown interfaces are special and a separate category from
 regular D interfaces, so this would be statically disallowed.

 But with the RCObject, we need to be careful about implicit casting to
 interfaces transparently killing memory safety.


Good point.


 of course then we might be weakening their usefulness. gah we just need
 scope references, then we can pass the interface with confidence that it
 won't be escaped. yeah yeah i know i sound like a broken record.


I don't think scope references are workable. I'll explain why in a 
latter post.


Andrei
Sep 21 2014
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
21-Sep-2014 19:03, Andrei Alexandrescu пишет:

 On 9/21/14, 1:51 AM, Jacob Carlborg wrote:

 On 2014-09-20 18:56, Andrei Alexandrescu wrote:


 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference counts. --
 Andrei


 We do need to know what you're proposal is for.




[snip]

 (For practical reasons, RCObject will implement IUnknown. There are a
 few random consequences of that (meaningless method QueryInterface, no
 class info, no monitor, COM-compatible layout) but IUnknown already has
 AddRef and Release and the compiler already recognizes IUnknown as a
 special case. Hooking RCObject there will leave us with two roots
 instead of adding a third one.)


This is actually a nice choice since COM would then (eventually) benefit 
from whatever automatic ref-counting we embed into the compiler.
This is exactly what Visual C++ does now for new style COM object 
(inherited from IInspectable).

-- 
Dmitry Olshansky
Sep 21 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 21.09.2014 20:17, schrieb Dmitry Olshansky:

 21-Sep-2014 19:03, Andrei Alexandrescu пишет:

 On 9/21/14, 1:51 AM, Jacob Carlborg wrote:

 On 2014-09-20 18:56, Andrei Alexandrescu wrote:


 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference
 counts. --
 Andrei


 We do need to know what you're proposal is for.




 [snip]

 (For practical reasons, RCObject will implement IUnknown. There are a
 few random consequences of that (meaningless method QueryInterface, no
 class info, no monitor, COM-compatible layout) but IUnknown already has
 AddRef and Release and the compiler already recognizes IUnknown as a
 special case. Hooking RCObject there will leave us with two roots
 instead of adding a third one.)


 This is actually a nice choice since COM would then (eventually) benefit
 from whatever automatic ref-counting we embed into the compiler.
 This is exactly what Visual C++ does now for new style COM object
 (inherited from IInspectable).


Specially important since all new Windows API since XP are mostly COM based.

--
Paulo
Sep 21 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 5:27 AM, Jacob Carlborg wrote:

 Assuming this would eventually be implemented for regular classes, it would be
 nice if it could be made compatible with Objective-C ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html


There was a long thread about that a year back or so. It wasn't looking good.
Sep 20 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 5:27 AM, Jacob Carlborg wrote:

 On 2014-09-19 17:32, Andrei Alexandrescu wrote:


 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.


 Assuming this would eventually be implemented for regular classes, it would be
 nice if it could be made compatible with Objective-C ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html



http://www.digitalmars.com/d/archives/digitalmars/D/draft_proposal_for_ref_counting_in_D_211885.html
Sep 20 2014
prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
How often do you store an exception reference anyway that escapes 
a catch block? I think all this talk is overkill to solve a 
non-problem in 99% of practice.


Correct me if I'm wrong, but aren't *all* exceptions in a 
particular thread generally unreferenced at the end of a catch() 
block, unless the programmer explicitly escaped that reference?

If so, we don't need refcounting! Here's my solution:

1) Throwables are allocated in a separate thread-local memory 
pool than most other objects. The  nogc version just aborts the 
program if this pool ever runs out of memory. The gc version can 
do a collection cycle and grow the size if necessary. (Though if 
you need a lot of Throwable objects alive at once, I question wtf 
is up with your code...)

2) class Throwable adds a method, gcClone, which copies it to the 
regular GC heap for those cases when you do want to store it or 
pass it between threads or whatever.

3) A function,  system void clearExceptions() is added. You may 
manually call this when you are done handling exceptions in this 
thread.

4) Might also have  safe void prepareExceptions() which 
pre-allocates the pool. This could also be done automatically or 
on demand or whatever.



Advantages:

* Throwables are still GC managed as far as most code is 
concerned. The only time you need caution is if you call the new 
 system function to free the pool... and even then, you just make 
sure you don't escape those references before you do.

* If you want to avoid the GC, just manually clear your 
exceptions from time to time.

* Allocating/deallocating from this specialized memory pool is 
prolly faster than any other scheme anyway.

* No language changes required, this is all library stuff.

Disadvantages:

* There's a few manual steps to get all the benefit. (We could 
insert a call to clearExceptions at the end of catch blocks 
automatically, but without a static check to ensure the objects 
actually haven't escaped, this would break memory safety. But a 
manual call isn't that big of a deal)

* If you do it wrong, you'll be annoyed.

* ???



I really think this is a win. I've done a proof of concept before 
by hacking _d_newclass to use the pool based on the TypeInfo 
passed in, gives a speedup in all the simple cases I tried. But 
since that hack still uses new, it wouldn't pass the  nogc test. 
However, a library function like emplace could be  nogc and do 
the same thing. Since  nogc is an addition to the function, 
you'll be modifying it anyway, so changing "throw new" to 
whatever the new lib function is called can be done at the same 
time.

It is the catch code that have to worry about freeing the 
pool.... and even then, only if you want to avoid the GC. Normal 
code can just let the pool be collected when it is collected.

I think I'll write a little module we can play with.
Sep 20 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 14:31:36 UTC, Adam D. Ruppe 
wrote:

 How often do you store an exception reference anyway that 
 escapes a catch block? I think all this talk is overkill to 
 solve a non-problem in 99% of practice.


Pretty much any time you do fibers + async I/O : to emulate 
blocking API one needs to catch and store exceptions from I/O 
routines so that later those can be re-thrown from resumed fiber 
context.
Sep 20 2014
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 20 September 2014 at 14:33:21 UTC, Dicebot wrote:

 Pretty much any time you do fibers + async I/O : to emulate 
 blocking API one needs to catch and store exceptions from I/O 
 routines so that later those can be re-thrown from resumed 
 fiber context.


Ah, indeed, and that could have a great many of them alive at 
once. Blargh, it'd need a real allocator to handle freeing them 
out of order.

Nevertheless though, I still think the lifetime management there 
is simple enough for the user-programmer that freeing it manually 
isn't a big hassle and not worth making major changes to the 
language over.
Sep 20 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 15:05:38 UTC, Adam D. Ruppe 
wrote:

 On Saturday, 20 September 2014 at 14:33:21 UTC, Dicebot wrote:

 Pretty much any time you do fibers + async I/O : to emulate 
 blocking API one needs to catch and store exceptions from I/O 
 routines so that later those can be re-thrown from resumed 
 fiber context.


 Ah, indeed, and that could have a great many of them alive at 
 once. Blargh, it'd need a real allocator to handle freeing them 
 out of order.

 Nevertheless though, I still think the lifetime management 
 there is simple enough for the user-programmer that freeing it 
 manually isn't a big hassle and not worth making major changes 
 to the language over.


I don't follow. Lifetime is totally unknown there and completely 
up to actual application logic - from the point of view of the 
caller of something like `readFile()` it just a normal allocated 
exception that doesn't need any explicit management. Requiring 
any kind of manual handling would be a huge breaking change.
Sep 20 2014
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 20 September 2014 at 15:12:45 UTC, Dicebot wrote:

 Requiring any kind of manual handling would be a huge breaking 
 change.


I'm not talking about *requiring* it; I want it to be a garbage 
collected object that 1) allocating it never triggers a 
collection cycle (so it is a  nogc allocation) and 2) you're 
allowed to explicitly free it if you want to, but if you don't, 
the gc will get around to it eventually.



just offer a new  nogc allocation function which creates garbage 
but does not garbage collect.

This could be a new function or the new keyword could be modified 
to suppress collection when allocating Throwable or whatever.

Maybe I'm missing something though.
Sep 20 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 8:05 AM, Adam D. Ruppe wrote:

 Nevertheless though, I still think the lifetime management there is
 simple enough for the user-programmer that freeing it manually isn't a
 big hassle and not worth making major changes to the language over.


The language will change in major ways, and this is the warmup before 
that. -- Andrei
Sep 20 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 20 September 2014 at 16:55:08 UTC, Andrei 
Alexandrescu wrote:

 The language will change in major ways, and this is the warmup 
 before that. -- Andrei


Ugh. I'm very concerned about the direction this is going, 
replacing a system that works with something that offers little 
benefit and a lot of risk.
Sep 20 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 10:50 AM, Adam D. Ruppe wrote:

 On Saturday, 20 September 2014 at 16:55:08 UTC, Andrei Alexandrescu wrote:

 The language will change in major ways, and this is the warmup before
 that. -- Andrei


 Ugh. I'm very concerned about the direction this is going, replacing a
 system that works with something that offers little benefit and a lot of
 risk.


Fasten your seatbelt, it's gonna be a bumpy ride! :o)

Andrei
Sep 20 2014
parent reply "ixid" <nuaccount gmail.com> writes:
 Fasten your seatbelt, it's gonna be a bumpy ride! :o)

 Andrei


The fundamentalness of the changes seem to be sufficient that one 
could argue it's D3. If you're going to make major changes 
wouldn't it be worth a fuller break to address some of the other 
unresolved and seemingly pretty major issues such as 
const/immutable and ref?
Sep 22 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
ixid:


 The fundamentalness of the changes seem to be sufficient that 
 one could argue it's D3.


This seems a good idea.



 If you're going to make major changes wouldn't it be worth a 
 fuller break to address some of the other unresolved and 
 seemingly pretty major issues such as const/immutable and ref?


This seems a bad idea, because GC and scope tracking is a 
sufficiently large design & implementation space. You can't face 
all the large problems at the same time.

Bye,
bearophile
Sep 22 2014
parent reply "Piotrek" <p nonexistent.pl> writes:
On Monday, 22 September 2014 at 09:13:49 UTC, bearophile wrote:

 ixid:


 The fundamentalness of the changes seem to be sufficient that 
 one could argue it's D3.


 This seems a good idea.


No, it's not a good idea. Tweaking memory management shouldn't 
require the language branching. IMHO, this would be a suicide.

Piotrek
Sep 22 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Piotrek:


 No, it's not a good idea. Tweaking memory management shouldn't 
 require the language branching. IMHO, this would be a suicide.


I didn't meant the advancement as a language branching, but as a 
successive version that is (mostly) backwards compatible. 


Bye,
bearophile
Sep 22 2014
parent "Piotrek" <p nonexistent.pl> writes:
On Monday, 22 September 2014 at 15:18:00 UTC, bearophile wrote:

 Piotrek:


 No, it's not a good idea. Tweaking memory management shouldn't 
 require the language branching. IMHO, this would be a suicide.


 I didn't meant the advancement as a language branching, but as 
 a successive version that is (mostly) backwards compatible. 


 Bye,
 bearophile


I'm not sure how you define a branch, but I look at it from SCM 
pov. E.g.


means, MS has to keep fixing all versions in parallel.

Piotrek
Sep 22 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/22/14, 1:56 AM, ixid wrote:

 Fasten your seatbelt, it's gonna be a bumpy ride! :o)

 Andrei


 The fundamentalness of the changes seem to be sufficient that one could
 argue it's D3.


Let's aim for not.


 If you're going to make major changes wouldn't it be
 worth a fuller break to address some of the other unresolved and
 seemingly pretty major issues such as const/immutable and ref?


What are the major issues with const/immutable and ref?


Andrei
Sep 22 2014
next sibling parent "ixid" <nuaccount gmail.com> writes:
 What are the major issues with const/immutable and ref?


Const and immutable seem to be difficult to work with, Maxime 
wrote a piece about how difficult to use they were in practice. 
Ref is difficult to combine properly with generic templates, Manu 
covers that in another thread here.

Maxime's blog:
http://pointersgonewild.wordpress.com/2014/07/11/the-constness-problem/

Manu's post about the combinatorial complexity of ref for 
templates (which you've responded to earlier in the chain):
http://forum.dlang.org/post/mailman.1381.1411381413.5783.digitalmars-d puremagic.com
Sep 22 2014
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 23 September 2014 00:50, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 9/22/14, 1:56 AM, ixid wrote:


 Fasten your seatbelt, it's gonna be a bumpy ride! :o)

 Andrei


 The fundamentalness of the changes seem to be sufficient that one could
 argue it's D3.


 Let's aim for not.

  If you're going to make major changes wouldn't it be

 worth a fuller break to address some of the other unresolved and
 seemingly pretty major issues such as const/immutable and ref?


 What are the major issues with const/immutable and ref?



This is precisely why I've mostly given up on this NG...
Sep 22 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 7:33 AM, Dicebot wrote:

 On Saturday, 20 September 2014 at 14:31:36 UTC, Adam D. Ruppe wrote:

 How often do you store an exception reference anyway that escapes a
 catch block? I think all this talk is overkill to solve a non-problem
 in 99% of practice.


 Pretty much any time you do fibers + async I/O : to emulate blocking API
 one needs to catch and store exceptions from I/O routines so that later
 those can be re-thrown from resumed fiber context.


Interesting. Are those stored as Object or Throwable/Exception? -- Andrei
Sep 20 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 16:50:08 UTC, Andrei 
Alexandrescu wrote:

 On 9/20/14, 7:33 AM, Dicebot wrote:

 On Saturday, 20 September 2014 at 14:31:36 UTC, Adam D. Ruppe 
 wrote:

 How often do you store an exception reference anyway that 
 escapes a
 catch block? I think all this talk is overkill to solve a 
 non-problem
 in 99% of practice.


 Pretty much any time you do fibers + async I/O : to emulate 
 blocking API
 one needs to catch and store exceptions from I/O routines so 
 that later
 those can be re-thrown from resumed fiber context.


 Interesting. Are those stored as Object or Throwable/Exception? 
 -- Andrei


I'd expect Throwable. Erasing it to Object smells like a trouble 
for no good. Don't know how Sonke has implemented in vibe.d 
though.
Sep 21 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 7:31 AM, Adam D. Ruppe wrote:

 How often do you store an exception reference anyway that escapes a
 catch block? I think all this talk is overkill to solve a non-problem in
 99% of practice.


 Correct me if I'm wrong, but aren't *all* exceptions in a particular
 thread generally unreferenced at the end of a catch() block, unless the
 programmer explicitly escaped that reference?

 If so, we don't need refcounting! Here's my solution:


[snip]

Thanks. I personally favor RC based on malloc to this. It'll be safer 
and simpler to use. -- Andrei
Sep 20 2014
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 20 September 2014 at 14:31:36 UTC, Adam D. Ruppe 
wrote:

 How often do you store an exception reference anyway that 
 escapes a catch block? I think all this talk is overkill to 
 solve a non-problem in 99% of practice.


 Correct me if I'm wrong, but aren't *all* exceptions in a 
 particular thread generally unreferenced at the end of a 
 catch() block, unless the programmer explicitly escaped that 
 reference?

 If so, we don't need refcounting! Here's my solution:


I proposed something very similar in this thread[1], except I 
proposed a solution that is fully backwards-compatible except the 
rare case when exceptions are escaped, in which case it causes a 
compile-time error. Might be worth a look.

I don't think it's acceptable that code using Phobos is suddenly 
leaking until they add the new manual cleanup call.

[1] 
http://forum.dlang.org/post/stlslhjndgugecvmbowd forum.dlang.org
Sep 20 2014
prev sibling next sibling parent reply "Oren Tirosh" <orent hishome.net> writes:
Hi everyone. Unlurking to make my first comment here.

Here is an idea for making RC and GC coexist peacefully. I think 
this technique may be used to make the Throwable transition to RC 
while keeping full backward compatibility.

A Throwable object would have a reference counter to track the 
number of RC references to it. It would also have a flag 
indicating whether there are any outstanding GC references to the 
object as well. This flag may be implemented by setting the MSB 
of the reference counter.

RC references would be a different type from GC references. An RC 
reference may be converted to a GC reference by setting the MSB, 
indicating that there may be one or more uncounted references to 
the object and that it should not be destroyed until successfully 
reclaimed by the GC. This RC to GC conversion may happen when a 
reference to a Throwable leaves  nogc code and enters GC land. 
When a GC operation can find no more references to the Throwable 
it would clear the counter MSB. If the number of RC references is 
zero at this point the object would be immediately destroyed.

A crucial part of making this work is to ensure that RC 
references are *not* seen by GC scanning. This may be done by 
mangling the pointer value in some way. Alternatively, if 
mark-and-sweep is replaced with count-and-sweep the number of 
counted RC references may be deducted.

One way to mangle the pointer is to decrement the value to a 
lower memory address. Incrementing the pointer won't work because 
the D garbage collector must handle inner references for slices 
etc.  Any dereferencing of the RC reference would compensate for 
this offset by incrementing the field offset. An obvious piece of 
information to put in the location reserved by decrementing the 
pointer would the the reference counter itself. Conversion of RC 
reference to GC would require incrementing the pointer and 
setting the counter MSB. Converting GC reference to RC is done by 
just decrementing the pointer.

I'm sure there are many things I'm missing here, but could 
something along these line be made to work?
Sep 20 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 20 September 2014 at 19:48:14 UTC, Oren Tirosh wrote:

 Hi everyone. Unlurking to make my first comment here.

 Here is an idea for making RC and GC coexist peacefully. I 
 think this technique may be used to make the Throwable 
 transition to RC while keeping full backward compatibility.

 A Throwable object would have a reference counter to track the 
 number of RC references to it. It would also have a flag 
 indicating whether there are any outstanding GC references to 
 the object as well. This flag may be implemented by setting the 
 MSB of the reference counter.


I was thinking about this recently (and also, Andrei's talk at 
cppcon provided some interesting data).

Most reference count are small. I think it make sense for us to 
propose a ref count system, that allocate on the GC heap, and 
that do not free when the counter saturate.

It would help to:
  - get intrusive refcount without putting too much crap into 
objects (it is all about cache line).
  - get a way to escape RC object to the GC heap (by saturating 
the refcount).

The GC is still there, but do kick in randomly, it simply act as 
a safety net.

Also given the type qualifier in D, the whole synchronization 
thing is embeded in the type, so we can be safe and fast on that 
one. All of this can be done as library.
Sep 20 2014
parent reply "Oren Tirosh" <orent hishome.net> writes:
On Sunday, 21 September 2014 at 00:45:49 UTC, deadalnix wrote:

 On Saturday, 20 September 2014 at 19:48:14 UTC, Oren Tirosh 
 wrote:

 Hi everyone. Unlurking to make my first comment here.

 Here is an idea for making RC and GC coexist peacefully. I 
 think this technique may be used to make the Throwable 
 transition to RC while keeping full backward compatibility.

 A Throwable object would have a reference counter to track the 
 number of RC references to it. It would also have a flag 
 indicating whether there are any outstanding GC references to 
 the object as well. This flag may be implemented by setting 
 the MSB of the reference counter.


 I was thinking about this recently (and also, Andrei's talk at 
 cppcon provided some interesting data).

 Most reference count are small. I think it make sense for us to 
 propose a ref count system, that allocate on the GC heap, and 
 that do not free when the counter saturate.


This is not what I was proposing. What you are describing (small,
saturating RC) is one of the  schemes mentioned in this article:

Shahriyar, Rifat, Stephen M. Blackburn, and Daniel Frampton.
"Down for the count? Getting reference counting back in the
ring." ACM SIGPLAN Notices. Vol. 47. No. 11. ACM, 2012.
http://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf

Reading this article is recommended for everyone here. It
demonstrates that advanced implementations of reference counting
systems are viable competition for GC and even outperform them
under certain workloads. The authors have implemented several
reference counting and hybrid schemes for the Java runtime and
benchmarked them. One of the schemes uses a small, saturating
reference counter in the object header similar to what you have
described.


 It would help to:
  - get intrusive refcount without putting too much crap into 
 objects (it is all about cache line).
  - get a way to escape RC object to the GC heap (by saturating 
 the refcount).


My idea is, indeed, to enable RC objects to be participate in
classic GC, but not by saturation. The MSB is, by definition, big
enough so the real reference count will not reach it. I suggest
using it as a flag to indicate whether there are any GC
references to the object. If clear, the object may be destroyed
when the counter reaches 0. If set, it means that a GC cycle is
still required to find if there are any more outstanding
references to it.

* No modification to the binary structure of Throwable. It
remains fully link-compatible with object files generated by
older compiler versions.
* "new Throwable" is modified to allocate extra space for a
counter just before the instance and returns a pointer just past
it. The counter is initialized to 0 with MSB set.
* The deallocator is modified to clear the counter MSB and
actually deallocate the object only if the counter is zero.

That's it for regular (" gc") code.

In  nogc code, an alternative "new" is used that sets the counter
to 1 (MSB clear) and returns a pointer to the counter rather than
the instance that follows it. Whenver this reference is copied,
overwritten or goes out of scope the reference counter must be
updated and deallocated when it reaches 0. Dereferencing the
Throwable via this type of reference-counter pointer uses
different offsets to compensate for the presence of the counter.
This reference-counted reference may be automatically converted
to a regular pointer that is safely used in code not aware of
reference counting by setting the MSB of the reference counter
and incrementing the pointer to point to the instance. Taking the
address of a field inside an object referenced through a
reference-counter pointer is either disallowed or required to set
the counter MSB because reference counted references do not
support internal pointers and

This would provide full backward compatibility even with .o code
already compiled before the feature was introduced.


 The GC is still there, but do kick in randomly, it simply act 
 as a safety net.

 Also given the type qualifier in D, the whole synchronization 
 thing is embeded in the type, so we can be safe and fast on 
 that one. All of this can be done as library.


The point of my scheme is to ensure that a thread that contains
no GC references may be safely excluded from "stop the world"
when GC kicks in. This is currently the biggest "lie" in the D
story about GC being optional. While  nogc code will not trigger
collection, it is NOT excluded from random pauses caused by GC in
other threads.
Sep 21 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 12:18 PM, Oren Tirosh wrote:

 Shahriyar, Rifat, Stephen M. Blackburn, and Daniel Frampton.
 "Down for the count? Getting reference counting back in the
 ring." ACM SIGPLAN Notices. Vol. 47. No. 11. ACM, 2012.
 http://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf

 Reading this article is recommended for everyone here.


Yah, and also all other related publications by Blackburn et comp. Great 
work we need to learn from. -- Andrei
Sep 21 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu
wrote:

 As discussed, having exception objects being GC-allocated is 
 clearly a large liability that we need to address. They prevent 
 otherwise careful functions from being  nogc so they affect 
 even apps that otherwise would be okay with a little litter 
 here and there.

 The Throwable hierarchy is somewhat separate from everything 
 else, which makes it a great starting point for investigating 
 an automated reference count approach. Here's what I'm thinking.

 First, there must be some compiler flag -nogc or something, 
 which triggers the RC exceptions. All modules of an application 
 must be compiled with this flag if it is to work (such that one 
 module can throw an exception caught by the other). Of course a 
 lot of refinement needs to be added here (what happens if one 
 tries to link modules built with and without -nogc, allowing 
 people to detect the flag programmatically by using 
 version(nogc) etc).

 If -nogc is passed, the compiler severs the inheritance 
 relationship between Throwable and Object, making it impossible 
 to convert a Throwable to an Object. From then henceforth, 
 Throwable and Object form a two-rooted forest. (In all 
 likelihood we'll later add an RCObject root that Throwable 
 inherits.)

 Whenever a reference to a Throwable is copied about, passed to 
 functions, the compiler inserts appropriately calls to e.g. 
 incRef and decRef. (Compiler may assume they cancel each other 
 for optimization purposes.) Implementation of these is up to 
 the runtime library. Null checking may be left to either the 
 compiler or the library (in the latter case, the functions must 
 be nonmember). However it seems the compiler may have an 
 advantage because it can elide some of these checks.

 Once we get this going, we should accumulate good experience 
 that we can later apply to generalizing this approach to more 
 objects. Also, if things go well we may as well define _always_ 
 (whether GC or not) Throwable to be reference counted; seems 
 like a good fit for all programs.

 Please chime in with thoughts.


 Andrei


I guess it is time to put isolated back on the table.

You can throw only isolated. When isolated goes out of scope
without being consumed, the island is blasted out of existence.
Sep 20 2014
prev sibling next sibling parent "Chris Cain" <zshazz gmail.com> writes:
I'm just throwing a new ThisOutThereException:

What if we modified the exception hierarchy to have it so 
"Throwable"s are GC allocated but there's an alternate set of 
classes down another line that either must be manually managed or 
are RC. Thus, there's no breakage.

Something like:

class Throwable : __UnifiedThrowable ...

class NoGCThrowable : __UnifiedThrowable ...


That way any code that's catching/handling Throwables doesn't 
need to change but we start making it so that code that needs to 
be NoGC handles the NoGCThrowable types. This would mean code 
could be migrated gradually to the new method (and, if truly 
desireable, the regular throwables could be deprecated 
eventually).

Just an idea.
Sep 20 2014
prev sibling parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu 
wrote:

 Please chime in with thoughts.


Why don't we all focus our efforts on upgrading the current GC to 
a state-of-the GC making use of D's strongly typed memory model 
before discussing these things?

Potentially with the help of escape analysis, which could be made 
part of this effort, automatically avoiding the GC seems like the 
wisest choice in the long run.

AFAIK there has been quite a lot of efforts done here already by 
various persons.
Sep 21 2014
next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Sunday, 21 September 2014 at 19:36:01 UTC, Nordlöw wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:

 Please chime in with thoughts.


 Why don't we all focus our efforts on upgrading the current GC 
 to a state-of-the GC making use of D's strongly typed memory 
 model before discussing these things?


GC improvements are critical, but...

"As discussed, having exception objects being GC-allocated is 
clearly a
large liability that we need to address. They prevent otherwise 
careful
functions from being  nogc so they affect even apps that 
otherwise would
be okay with a little litter here and there."

No improvements to the GC can fix this.  nogc needs to be usable, 
whether you are a GC fan or not.
Sep 21 2014
next sibling parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Sunday, 21 September 2014 at 20:57:24 UTC, Peter Alexander 
wrote:

 GC improvements are critical, but...

 "As discussed, having exception objects being GC-allocated is 
 clearly a
 large liability that we need to address. They prevent otherwise 
 careful
 functions from being  nogc so they affect even apps that 
 otherwise would
 be okay with a little litter here and there."


Ok, got it. Thanks.
Sep 21 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 1:57 PM, Peter Alexander wrote:

 On Sunday, 21 September 2014 at 19:36:01 UTC, Nordlöw wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:

 Please chime in with thoughts.


 Why don't we all focus our efforts on upgrading the current GC to a
 state-of-the GC making use of D's strongly typed memory model before
 discussing these things?


 GC improvements are critical, but...

 "As discussed, having exception objects being GC-allocated is clearly a
 large liability that we need to address. They prevent otherwise careful
 functions from being  nogc so they affect even apps that otherwise would
 be okay with a little litter here and there."

 No improvements to the GC can fix this.  nogc needs to be usable,
 whether you are a GC fan or not.


Exactly and awesomely right. -- Andrei
Sep 21 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 20:57:24 UTC, Peter Alexander 
wrote:

 No improvements to the GC can fix this.  nogc needs to be 
 usable, whether you are a GC fan or not.


True.

To fix this, we need to add a pile of hack that take care of this 
specific use case. The end goal is to have a pile of hack for 
every single use case, as this is what C++ has and C++ is 
successful and we want to be successful.

Introducing a construct to manage ownership would obviously avoid 
the whole pile of hack, but hey, this would introduce complexity 
to the language, when the pile of hack do not.

AMIRITE ?
Sep 21 2014
parent "Cliff" <cliff.s.hudson gmail.com> writes:
On Sunday, 21 September 2014 at 23:32:29 UTC, deadalnix wrote:

 On Sunday, 21 September 2014 at 20:57:24 UTC, Peter Alexander 
 wrote:

 No improvements to the GC can fix this.  nogc needs to be 
 usable, whether you are a GC fan or not.


 True.

 To fix this, we need to add a pile of hack that take care of 
 this specific use case. The end goal is to have a pile of hack 
 for every single use case, as this is what C++ has and C++ is 
 successful and we want to be successful.

 Introducing a construct to manage ownership would obviously 
 avoid the whole pile of hack, but hey, this would introduce 
 complexity to the language, when the pile of hack do not.

 AMIRITE ?


The devolution of conversation in this thread to snide remarks, 
extreme sarcasm and thinly (if at all) veiled name calling is 
counterproductive to solving the problem.  Reasonable people can 
disagree without resorting to passive-aggression.
Sep 21 2014
prev sibling parent luka8088 <luka8088 owave.net> writes:
On 21.9.2014. 22:57, Peter Alexander wrote:

 On Sunday, 21 September 2014 at 19:36:01 UTC, Nordlöw wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:

 Please chime in with thoughts.


 Why don't we all focus our efforts on upgrading the current GC to a
 state-of-the GC making use of D's strongly typed memory model before
 discussing these things?


 GC improvements are critical, but...

 "As discussed, having exception objects being GC-allocated is clearly a
 large liability that we need to address. They prevent otherwise careful
 functions from being  nogc so they affect even apps that otherwise would
 be okay with a little litter here and there."

 No improvements to the GC can fix this.  nogc needs to be usable,
 whether you are a GC fan or not.


I think that what is being suggested is that upgrading GC would 
widespread the point of view on what can and should be done.

For example, now that ranges and mixins exist great ideas comes to mind, 
and without them we can only guess. I think that GC is in the same position.
Sep 22 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 12:35 PM, "Nordlöw" wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:

 Please chime in with thoughts.


 Why don't we all focus our efforts on upgrading the current GC to a
 state-of-the GC making use of D's strongly typed memory model before
 discussing these things?


Not sure I understand, but: an important niche is apps that COMPLETELY 
disallow garbage collection -- Andrei
Sep 21 2014
parent Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/21/2014 3:12 PM, Andrei Alexandrescu via Digitalmars-d wrote:

 On 9/21/14, 12:35 PM, "Nordlöw" wrote:

 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:

 Please chime in with thoughts.


 Why don't we all focus our efforts on upgrading the current GC to a
 state-of-the GC making use of D's strongly typed memory model before
 discussing these things?


 Not sure I understand, but: an important niche is apps that COMPLETELY
 disallow garbage collection -- Andrei


It's also far from being a zero sum game.  Work on one is not 
necessarily taking away from the other.  There needs to be more 'doing' 
that follows the discussions than there typically is.  But progress can 
and needs to be made on both (and many many other) fronts.
Sep 22 2014