www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Smart pointers instead of GC?

reply "Sven Over" <dlang svenover.de> writes:
Hello dear D community.

This is my very first post here. I've been intrigued with D for 
quite a while, and I would love to have a project to work on in 
D. I'm working as a C++ software engineer, and should be 
moderately up-to-date with best practices of C++ programming.

I've read a good part of "The D Programming Language" and must 
say that I *love* almost everything about D. Metaprogramming, 
which quickly gets very convoluted in C++ (to say the least), is 
just beautiful in D. The approach to multi-threaded programming, 
and the adoption of concepts of functional programming, are all 
pure gold.

The one thing that I am unsure about in D is garbage collection. 
I guess this is some sort of FAQ and may have been answered 
dozens of times. I apologise in advance.

The main reason I dislike garbage collection for is that 
destructors of objects on the heap are not called at defined 
times. Techniques like RAII 
(http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization) 
have become an often used pattern in my programming, and works 
nicely for short lived objects (on the stack), but by means of 
smart pointers (i.e. reference counting) also for long lived 
objects (on the heap).

Also, garbage collection often tends to increase the memory 
footprint of your software. You may be able to avoid this with 
some expert knowledge about the garbage collector, but this of 
course invalidates one of the main arguments in favour of GC, 
which is less complexity. I'd say that if you want to write 
memory- and CPU-efficient code, then you'll need a certain amount 
of understanding of your memory management. Be it your manual 
memory management, or the inner workings of your garbage 
collector.

To cut things short, even after giving it a lot of thought I 
still feel uncomfortable about garbage collection. I'm wondering 
whether it is viable to use the same smart pointer techniques as 
in C++: is there an implementation already of such smart 
pointers? Can I switch off GC in my D programs altogether? How 
much does the standard library rely on GC? I.e. would I have to 
write a alternative standard library to do this?

There is one blog post I found interesting to read in this 
context, which also underlines my worries: 
http://3d.benjamin-thaut.de/?p=20
Dec 25 2012
next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Tuesday, 25 December 2012 at 14:13:21 UTC, Sven Over wrote:

 To cut things short, even after giving it a lot of thought I 
 still feel uncomfortable about garbage collection. I'm 
 wondering whether it is viable to use the same smart pointer 
 techniques as in C++: is there an implementation already of 
 such smart pointers? Can I switch off GC in my D programs 
 altogether? How much does the standard library rely on GC? I.e. 
 would I have to write a alternative standard library to do this?


std.typecons.RefCounted!T

core.memory.GC.disable();

Phobos does rely on the GC to some extent. Most algorithms and 
ranges do not though.
Dec 25 2012
next sibling parent reply "Sven Over" <dlang svenover.de> writes:
 std.typecons.RefCounted!T

 core.memory.GC.disable();


Wow. That was easy.

I see, D's claim of being a multi-paradigm language is not false.


 Phobos does rely on the GC to some extent. Most algorithms and 
 ranges do not though.


Running (library) code that was written with GC in mind and 
turning GC off doesn't sound ideal.

But maybe this allows me to familiarise myself more with D. Who 
knows, maybe I can learn to stop worrying and love garbage 
collection.

Thanks for your help!
Dec 25 2012
next sibling parent reply "Leandro Lucarella" <leandro.lucarella sociomantic.com> writes:
On Tuesday, 25 December 2012 at 14:48:48 UTC, Sven Over wrote:

 Phobos does rely on the GC to some extent. Most algorithms and 
 ranges do not though.


 Running (library) code that was written with GC in mind and 
 turning GC off doesn't sound ideal.


It isn't if you care about memory leaks. And also bare in mind 
that some language constructs also relies on the GC, like some 
dynamic arrays operations (concatenation and appending for 
example). Right now is pretty hard to write D code that doesn't 
really use the GC at all.
Dec 25 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, December 25, 2012 16:33:27 Leandro Lucarella wrote:

 On Tuesday, 25 December 2012 at 14:48:48 UTC, Sven Over wrote:

 Phobos does rely on the GC to some extent. Most algorithms and
 ranges do not though.


 
 Running (library) code that was written with GC in mind and
 turning GC off doesn't sound ideal.


 
 It isn't if you care about memory leaks. And also bare in mind
 that some language constructs also relies on the GC, like some
 dynamic arrays operations (concatenation and appending for
 example). Right now is pretty hard to write D code that doesn't
 really use the GC at all.


There's also often no reason not to have the GC on and use it for certain stuff 
but use ref-counted objects as your main type of memory-management. Then, you 
avoid whatever issues the GC has in most cases but don't have to worry about 
what it takes to be able to not use it at all. For instance, arrays would 
probably be GC-allocated in general, since then you can use slices and 
whatnot, but maybe all of your user-defined objects on the heap could be 
malloced and freed with reference counts (though that sort of thing should be 
much easier once we have custom allocators - it's a bit more of a pain right 
now than it should be).

- Jonathan M Davis
Dec 25 2012
next sibling parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Tuesday, 25 December 2012 at 19:23:59 UTC, Jonathan M Davis 
wrote:

 On Tuesday, December 25, 2012 16:33:27 Leandro Lucarella wrote:

 On Tuesday, 25 December 2012 at 14:48:48 UTC, Sven Over wrote:

 Phobos does rely on the GC to some extent. Most algorithms 
 and
 ranges do not though.


 
 Running (library) code that was written with GC in mind and
 turning GC off doesn't sound ideal.


 
 It isn't if you care about memory leaks. And also bare in mind
 that some language constructs also relies on the GC, like some
 dynamic arrays operations (concatenation and appending for
 example). Right now is pretty hard to write D code that doesn't
 really use the GC at all.


 There's also often no reason not to have the GC on and use it 
 for certain stuff
 but use ref-counted objects as your main type of 
 memory-management. Then, you
 avoid whatever issues the GC has in most cases but don't have 
 to worry about
 what it takes to be able to not use it at all. For instance, 
 arrays would
 probably be GC-allocated in general, since then you can use 
 slices and
 whatnot, but maybe all of your user-defined objects on the heap 
 could be
 malloced and freed with reference counts (though that sort of 
 thing should be
 much easier once we have custom allocators - it's a bit more of 
 a pain right
 now than it should be).

 - Jonathan M Davis


In my experience its quite some work to remove the GC entierly, 
but it is possible. Also it makes writing Code also easier ans 
more effektive in some places because:
-You don't have to register manually allocated blocks with the gc
-You control the exact order of destruction
-You control the time of destruction
-You control in which thread a object gets destructed
-You will be encouraged to not write code that does do 100 
allocations just to format a string (because it will leak)

But obviously it has an impact on productivity. But I'm still 
more productive writing manual memory managed code in D then in 
C++.

Kind Regards
Benjamin Thaut
Dec 26 2012
parent reply "Sven Over" <dlang svenover.de> writes:
On Wednesday, 26 December 2012 at 11:43:55 UTC, Benjamin Thaut 
wrote:

 In my experience its quite some work to remove the GC entierly, 
 but it is possible.


Hi Benjamin! I've seen your druntime and thBase repos on GitHub. 
Very interesting stuff. As I have little (or to be honest: no) 
experience in D it's difficult for me to judge how much work 
needs to be done. Is the idea to replace Phobos completely?


 Also it makes writing Code also easier ans more effektive in 
 some places
 because:
 -You don't have to register manually allocated blocks with the 
 gc
 -You control the exact order of destruction
 -You control the time of destruction
 -You control in which thread a object gets destructed
 -You will be encouraged to not write code that does do 100 
 allocations just to format a string (because it will leak)


I personally agree with all your points, but I must admit that 
their relevance depends on the programming patterns you are 
using. Experienced GC programmers certainly favour patterns that 
work well with GC. I personally prefer a style of programming 
that makes use of destructors and relies on them being executed 
at the right time.


 But obviously it has an impact on productivity.


I'm not sure about this at all. I write software that is designed 
to offer a certain performance, and latency is a big issue. I 
don't believe that I can achieve that with a "just allocate 
memory and don't ever care" attitude. I would need some 
background knowledge about how the GC works, and write code that 
doesn't screw up badly. Manual memory management also requires me 
to have an understanding of the implications behind it. But there 
are well-tested helpers (e.g. smart pointers) that simplify the 
task a lot.

I do admit that, when performance (both in the sense of being 
fast and offering reliable performance) is not such a critical 
issue, then GC won't hurt and "just allocating" is fine and of 
course reduces development time. However, I tend to just use 
Python for this class problems.

It comes down to a matter of choice and personal preference. I'm 
not saying that GC is bad and shouldn't be used. Many people are 
very happy with it and it does a great job for them. But my 
personal preference is manual memory management (then again 
greatly automated by use of reference counting and smart 
pointers).
Dec 31 2012
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 31.12.2012 13:36, schrieb Sven Over:

 Hi Benjamin! I've seen your druntime and thBase repos on GitHub. Very
 interesting stuff. As I have little (or to be honest: no) experience in
 D it's difficult for me to judge how much work needs to be done. Is the
 idea to replace Phobos completely?


The idea is not to replace phobos completely. Currenly it only contains 
the functionality that I did need for my projects and that could not be 
used from phobos. std.traits, std.typetuple, and others are still used 
from phobos because they don't leak (or could be made non leaking with 
very small modifications).
thBase also contains a lot of functionality phobos does not have. E.g. 
containers, a better xml parser/writer, templates for automiatcally 
serializing and deserializing objects to/from xml files and much more.
Everything that I think could be usefull for other projects in the 
future gets added to thBase.

Knid Regards
Benjamin Thaut
Dec 31 2012
prev sibling parent reply "Sven Over" <dlang svenover.de> writes:
On Tuesday, 25 December 2012 at 19:23:59 UTC, Jonathan M Davis 
wrote:

 There's also often no reason not to have the GC on and use it 
 for certain stuff


One thing that really freaks me out is the fact that the garbage 
collector pauses the whole process, i.e. all threads.

In my job I'm writing backend services that power a big web site. 
Perfomance is key, as the response time of the data service in 
most cases directly adds to the page load time. The bare 
possibility that the whole service pauses for, say, 100ms is 
making me feel very uncomfortable.

We easily achieve the performance and reliability we need in C++, 
but I would love to give D a chance, as it solves many 
inconveniences of C++ in an elegant way. Metaprogramming and the 
threading model, just to name two.


 For instance, arrays would probably be GC-allocated in general, 
 since
 then you can use slices and whatnot,


A smart-pointer type for arrays can easily provide slices. It 
keeps a reference to the full array (which gets destructed when 
the last reference is dropped), but addresses a subrange.

Thanks everyone for all the replies!
Dec 31 2012
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 31 December 2012 at 12:14:22 UTC, Sven Over wrote:

 On Tuesday, 25 December 2012 at 19:23:59 UTC, Jonathan M Davis 
 wrote:

 There's also often no reason not to have the GC on and use it 
 for certain stuff


 One thing that really freaks me out is the fact that the 
 garbage collector pauses the whole process, i.e. all threads.

 In my job I'm writing backend services that power a big web 
 site. Perfomance is key, as the response time of the data 
 service in most cases directly adds to the page load time. The 
 bare possibility that the whole service pauses for, say, 100ms 
 is making me feel very uncomfortable.

 We easily achieve the performance and reliability we need in 
 C++, but I would love to give D a chance, as it solves many 
 inconveniences of C++ in an elegant way. Metaprogramming and 
 the threading model, just to name two.


 For instance, arrays would probably be GC-allocated in 
 general, since
 then you can use slices and whatnot,


 A smart-pointer type for arrays can easily provide slices. It 
 keeps a reference to the full array (which gets destructed when 
 the last reference is dropped), but addresses a subrange.

 Thanks everyone for all the replies!


I think the problem would be if you try to append to the slice 
(~): If the underlying array is not GC allocated, then it will 
append to a new GC allocated array (AFAIK).

As long as you don't append, I'd say you are fine.
Dec 31 2012
parent reply "Sven Over" <dlang svenover.de> writes:
On Monday, 31 December 2012 at 12:36:06 UTC, monarch_dodra wrote:

 On Monday, 31 December 2012 at 12:14:22 UTC, Sven Over wrote:



 A smart-pointer type for arrays can easily provide slices. It 
 keeps a reference to the full array (which gets destructed 
 when the last reference is dropped), but addresses a subrange.





 I think the problem would be if you try to append to the slice 
 (~): If the underlying array is not GC allocated, then it will 
 append to a new GC allocated array (AFAIK).

 As long as you don't append, I'd say you are fine.


Of course your smart-array-pointer type needs to implement the ~ 
operator and create a copy of the array in that case. I guess the 
full implementation would include something that would resemble 
that of std::vector in C++.

I should get started writing those types. Then I could either 
demonstrate that it does work, or understand that it doesn't (and 
why).
Dec 31 2012
next sibling parent "jerro" <a a.com> writes:
 Of course your smart-array-pointer type needs to implement the 
 ~ operator and create a copy of the array in that case. I guess 
 the full implementation would include something that would 
 resemble that of std::vector in C++.


std.container.Array already does that.
Dec 31 2012
prev sibling parent reply "Kiith-Sa" <kiithsacmp gmail.com> writes:
I think you're overthinking this way too much.

I'm writing a game engine, where latency is much more important
than in your case. Even 10ms would translate to visible jitter.
GC's not an issue, and disabling it permanently is very
counterproductive _unless_ you've exhausted all other options
(which you're unlikely to, as you can do everything you can do in
C++).  All you have to do is care about how you allocate and, if
GC seems to be an issue, profile to see _where_ the GC is being
called most and optimize those allocations.

Basic rules:

For classes with one or few instances (singletons, etc.), GC is
not an issue. For classes with hundreds-thousands of instances,
it might be an issue. Profile. For classes with more instances,
it probably is an issue. Profile, reuse instances, use structs,
manually allocate.  Arrays: To avoid reallocation on append, use
array.assumeSafeAppend() before the append (Assumes that the
array is not a slice of a bigger array that would get its data
overwritten).

If appending a lot, use std.array.Appender. If you need manually
allocated storage, use std.container.Array, or create a wrapper
around malloc/free. I'm using a templated wrapper that allows me
to record how much memory/instances was allocated with which
types.  Destructors: Structs are RAII like in C. And you can
easily create a malloc wrapper that will act exactly like
new/delete in C++.

Classes: call destroy(instance). Calls the dtor, but doesn't
deallocate the class (GC will do that later). Not much different
from deleting a class allocated through new in C++ (which is what
you do most of the time in C++).  If you absolutely need to free
the memory, create malloc/free wrappers for classes.  In my case,
classes are used for polymorphic stuff, which are usually single
or few-instance objects. Structs are used for most of the
many-instance objects (e.g. components of game entities), and are
usually stored in manually allocated arrays.

GC is _very_ useful for stuff like closures, which can greatly
simplify some code. Also, while I maintain that disabling GC
completely is a bad idea, it might be useful to disable it in a
small area of code _after_ profiling shows that GC is being
called too much there.

In my YAML parser library, I've found after profiling that 18% of
time was spent in GC. Most of that was in a small piece of code
that repeatedly allocated memory. Disabling GC before this code
and reenabling it after (scope(exit) to avoid leaking disabled
GC) resulted in GC taking ~2% of time, because many unnecessary
collects were consolidated into one after the GC was reenabled.
Memory usage didn't take a hit, because this was actually a
fairly small piece of code, not doing too many allocations per
call, just called very often.
Dec 31 2012
next sibling parent reply "pjmlp" <pjmlp progtools.org> writes:
On Monday, 31 December 2012 at 14:05:01 UTC, Kiith-Sa wrote:

 I think you're overthinking this way too much.

 I'm writing a game engine, where latency is much more important
 than in your case. Even 10ms would translate to visible jitter.
 GC's not an issue, and disabling it permanently is very
 counterproductive _unless_ you've exhausted all other options
 (which you're unlikely to, as you can do everything you can do 
 in
 C++).  All you have to do is care about how you allocate and, if
 GC seems to be an issue, profile to see _where_ the GC is being
 called most and optimize those allocations.

 Basic rules:

 For classes with one or few instances (singletons, etc.), GC is
 not an issue. For classes with hundreds-thousands of instances,
 it might be an issue. Profile. For classes with more instances,
 it probably is an issue. Profile, reuse instances, use structs,
 manually allocate.  Arrays: To avoid reallocation on append, use
 array.assumeSafeAppend() before the append (Assumes that the
 array is not a slice of a bigger array that would get its data
 overwritten).

 If appending a lot, use std.array.Appender. If you need manually
 allocated storage, use std.container.Array, or create a wrapper
 around malloc/free. I'm using a templated wrapper that allows me
 to record how much memory/instances was allocated with which
 types.  Destructors: Structs are RAII like in C. And you can
 easily create a malloc wrapper that will act exactly like
 new/delete in C++.

 Classes: call destroy(instance). Calls the dtor, but doesn't
 deallocate the class (GC will do that later). Not much different
 from deleting a class allocated through new in C++ (which is 
 what
 you do most of the time in C++).  If you absolutely need to free
 the memory, create malloc/free wrappers for classes.  In my 
 case,
 classes are used for polymorphic stuff, which are usually single
 or few-instance objects. Structs are used for most of the
 many-instance objects (e.g. components of game entities), and 
 are
 usually stored in manually allocated arrays.

 GC is _very_ useful for stuff like closures, which can greatly
 simplify some code. Also, while I maintain that disabling GC
 completely is a bad idea, it might be useful to disable it in a
 small area of code _after_ profiling shows that GC is being
 called too much there.

 In my YAML parser library, I've found after profiling that 18% 
 of
 time was spent in GC. Most of that was in a small piece of code
 that repeatedly allocated memory. Disabling GC before this code
 and reenabling it after (scope(exit) to avoid leaking disabled
 GC) resulted in GC taking ~2% of time, because many unnecessary
 collects were consolidated into one after the GC was reenabled.
 Memory usage didn't take a hit, because this was actually a
 fairly small piece of code, not doing too many allocations per
 call, just called very often.


I think it will still take a few generations of programmers until 
everyone is confortable with using GC enabled languages for 
systems programming scenarios.

Most people nowadays are not aware of what has already been done 
in academia, or are burned by bad experiences caused by whatever 
reasons.

This situation will only improve when developers start aproaching 
the same class of problems with open minds on how to solve them 
in new ways.

--
Paulo
Dec 31 2012
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 31 December 2012 at 14:43:27 UTC, pjmlp wrote:

 I think it will still take a few generations of programmers 
 until everyone is confortable with using GC enabled languages 
 for systems programming scenarios.

 Most people nowadays are not aware of what has already been 
 done in academia, or are burned by bad experiences caused by 
 whatever reasons.

 This situation will only improve when developers start 
 aproaching the same class of problems with open minds on how to 
 solve them in new ways.

 --
 Paulo


D's GC has some implementation issue as well.
Jan 01 2013
parent reply "DythroposTheImposter" <barf barf.com> writes:
  I'm interested in how the new LuaJIT GC ends up performing. But 
overall I can't say I have much hope for GC right now.

  GC/D = Generally Faster allocation. Has a cost associated with 
every living object.

  C++ = Generally Slower allocation, but while it is alive there 
is no cost.

  So as the heap grows, the GC language falls behind.

  This seems to be the case in every language I've looked at this 
uses a GC.
Jan 02 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 2 January 2013 at 11:41:33 UTC, 
DythroposTheImposter wrote:

  I'm interested in how the new LuaJIT GC ends up performing. 
 But overall I can't say I have much hope for GC right now.

  GC/D = Generally Faster allocation. Has a cost associated with 
 every living object.


True, however, GC + immutability allow new idoms that are plain 
impossible in a non GC world, and that are really efficients at 
reducing copies and allocations (and used in many high perf D 
libs).

The comparison you present here fail to take this into account.


  C++ = Generally Slower allocation, but while it is alive there 
 is no cost.


Counting reference have a cost. Plus, when the heap grow, the 
object that dies together tend to grow as well, which cause 
pauses.

Latency contrived programs, like video games, tends to avoid 
allocation altogether because of this. This obviously work both 
with GC and reference counting.


  So as the heap grows, the GC language falls behind.


That is more subtle than that. Reconsider what I wrote above, for 
instance that GC + immutability can be used to reduce allocations 
(so live heap size). You'll also have to consider that GC can be 
concurrent, when reference counting usually cannot (or loose some 
of its benefit compared to GC).


  This seems to be the case in every language I've looked at 
 this uses a GC.


Most GC language lack proper memory management, and sometime have 
design choice that are a nightmare for the GC (java have no value 
type for instance). You are comparing here way more than simply 
GC vs other memory management.

Considering the pro and cons of each, an hybrid approach seems 
like the most reasonable thing to do in any high performance 
program. Which is possible in D because of GC.free .
Jan 02 2013
next sibling parent reply "Thiez" <thiezz gmail.com> writes:
On Wednesday, 2 January 2013 at 12:32:01 UTC, deadalnix wrote:

 Most GC language lack proper memory management, and sometime 
 have design choice that are a nightmare for the GC (java have 
 no value type for instance).


Surely Java's primitive types (byte, short, int, long, float, 
double, boolean, and char) count as value types?
Jan 02 2013
next sibling parent "Mehrdad" <wfunction hotmail.com> writes:
On Wednesday, 2 January 2013 at 17:18:52 UTC, Thiez wrote:

 On Wednesday, 2 January 2013 at 12:32:01 UTC, deadalnix wrote:

 Most GC language lack proper memory management, and sometime 
 have design choice that are a nightmare for the GC (java have 
 no value type for instance).


 Surely Java's primitive types (byte, short, int, long, float, 
 double, boolean, and char) count as value types?


I think he meant non-primitives
Jan 02 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 2 January 2013 at 17:18:52 UTC, Thiez wrote:

 On Wednesday, 2 January 2013 at 12:32:01 UTC, deadalnix wrote:

 Most GC language lack proper memory management, and sometime 
 have design choice that are a nightmare for the GC (java have 
 no value type for instance).


 Surely Java's primitive types (byte, short, int, long, float, 
 double, boolean, and char) count as value types?


Yes, but they are irrelevant for the GC.
Jan 02 2013
prev sibling parent reply "Frustrated" <c1514843 drdrb.com> writes:
On Wednesday, 2 January 2013 at 12:32:01 UTC, deadalnix wrote:

 On Wednesday, 2 January 2013 at 11:41:33 UTC, 
 DythroposTheImposter wrote:

 I'm interested in how the new LuaJIT GC ends up performing. 
 But overall I can't say I have much hope for GC right now.

 GC/D = Generally Faster allocation. Has a cost associated with 
 every living object.


 True, however, GC + immutability allow new idoms that are plain 
 impossible in a non GC world, and that are really efficients at 
 reducing copies and allocations (and used in many high perf D 
 libs).


And the problem is that D uses GC as a sledgehammer. A GC
shouldn't be used for everything. If it is used for stuff it
works well at and not for anything and everything, then it
becomes useful.

Because the GC is used for everything in D it gets clogged up
with stuff like simple object allocation(which, of course, there
is a way to get around, but not easily) which effects it's
performance.

For example, suppose the GC was used only for closures and
nothing else. In this case, since closures are not used that much
compared to other things, the GC would have minimal impact on
performance. It could probably be ran once a month and not have
memory issues.

But since the GC does everything, you have to run it much more
often, which means the amortization is much higher regardless if
we even use closures.

Ultimately the best option is control. If we were not "forced" to
use the GC for everything but could enable it for different
things, then we could tweak it for best performance.

And why does Phobos/runtime require the GC in so many cases when
it could just use an internal buffer? So much effort has been put
in to make the GC work that it simply has neglected all those
that can't use it as it is.

Basically doing the exact opposite by making the GC more
ubiquitous in D. Makes people that have no problem with the GC
happy but those that do have problems just get more and more
frustrated.

What has happened is the GC has made the library writers lazy. It
doesn't belong in 99% of Phobos.
Feb 01 2014
parent reply "Sean Kelly" <sean invisibleduck.org> writes:
On Saturday, 1 February 2014 at 11:40:37 UTC, Frustrated wrote:

 And why does Phobos/runtime require the GC in so many cases when
 it could just use an internal buffer? So much effort has been 
 put
 in to make the GC work that it simply has neglected all those
 that can't use it as it is.


This is the crux of the problem.  It's not so much that D uses 
garbage collection as that Phobos is built in a way that prevents 
the reuse of existing buffers.  It becomes much harder to sell 
the language to a GC averse group if it turns out they can't use 
the standard library as well.  Though I guess Tango is an option 
here as a replacement.
Feb 01 2014
parent reply "Frustrated" <c1514843 drdrb.com> writes:
On Saturday, 1 February 2014 at 17:30:54 UTC, Sean Kelly wrote:

 On Saturday, 1 February 2014 at 11:40:37 UTC, Frustrated wrote:

 And why does Phobos/runtime require the GC in so many cases 
 when
 it could just use an internal buffer? So much effort has been 
 put
 in to make the GC work that it simply has neglected all those
 that can't use it as it is.


 This is the crux of the problem.  It's not so much that D uses 
 garbage collection as that Phobos is built in a way that 
 prevents the reuse of existing buffers.  It becomes much harder 
 to sell the language to a GC averse group if it turns out they 
 can't use the standard library as well.  Though I guess Tango 
 is an option here as a replacement.


Right, and because of the mentality that the GC is one size that
fits all, phobos got this way. That mentality is still pervasive.
I call it laziness.

I think that at some point phobos will need to be rewritten...
maybe more .net like(there sense of hierarchy and presentation is
excellent).  Maybe D needs a proper allocator sub system built in
to get to that point?
Feb 01 2014
next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Sat, 01 Feb 2014 09:57:41 -0800, Frustrated <c1514843 drdrb.com> wrote:


 On Saturday, 1 February 2014 at 17:30:54 UTC, Sean Kelly wrote:

 On Saturday, 1 February 2014 at 11:40:37 UTC, Frustrated wrote:

 And why does Phobos/runtime require the GC in so many cases when
 it could just use an internal buffer? So much effort has been put
 in to make the GC work that it simply has neglected all those
 that can't use it as it is.


 This is the crux of the problem.  It's not so much that D uses garbage  
 collection as that Phobos is built in a way that prevents the reuse of  
 existing buffers.  It becomes much harder to sell the language to a GC  
 averse group if it turns out they can't use the standard library as  
 well.  Though I guess Tango is an option here as a replacement.


 Right, and because of the mentality that the GC is one size that
 fits all, phobos got this way. That mentality is still pervasive.
 I call it laziness.

 I think that at some point phobos will need to be rewritten...
 maybe more .net like(there sense of hierarchy and presentation is
 excellent).  Maybe D needs a proper allocator sub system built in
 to get to that point?


You do realize that in .NET the BCL is not GC-optional right. It  
GC-allocates like a fiend. What they have is a monumentally better GC than  
D. .NET is actually proof that GC's can work quite well in all but the  
most extreme edge cases.

Effectively what you are saying is ".NET does it well with a GC so we need  
to rewrite Phobos to not use a GC" ... Um wait, what?

I am not saying to Phobos needs to allocate as much as it does, just that  
argument itself fails to met a basic standard of logical coherency.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator
Feb 01 2014
parent "Frustrated" <c1514843 drdrb.com> writes:
On Saturday, 1 February 2014 at 19:11:28 UTC, Adam Wilson wrote:

 On Sat, 01 Feb 2014 09:57:41 -0800, Frustrated 
 <c1514843 drdrb.com> wrote:


 On Saturday, 1 February 2014 at 17:30:54 UTC, Sean Kelly wrote:

 On Saturday, 1 February 2014 at 11:40:37 UTC, Frustrated 
 wrote:

 And why does Phobos/runtime require the GC in so many cases 
 when
 it could just use an internal buffer? So much effort has 
 been put
 in to make the GC work that it simply has neglected all those
 that can't use it as it is.


 This is the crux of the problem.  It's not so much that D 
 uses garbage collection as that Phobos is built in a way that 
 prevents the reuse of existing buffers.  It becomes much 
 harder to sell the language to a GC averse group if it turns 
 out they can't use the standard library as well.  Though I 
 guess Tango is an option here as a replacement.


 Right, and because of the mentality that the GC is one size 
 that
 fits all, phobos got this way. That mentality is still 
 pervasive.
 I call it laziness.

 I think that at some point phobos will need to be rewritten...
 maybe more .net like(there sense of hierarchy and presentation 
 is
 excellent).  Maybe D needs a proper allocator sub system built 
 in
 to get to that point?


 You do realize that in .NET the BCL is not GC-optional right. 
 It GC-allocates like a fiend. What they have is a monumentally 
 better GC than D. .NET is actually proof that GC's can work 
 quite well in all but the most extreme edge cases.

 Effectively what you are saying is ".NET does it well with a GC 
 so we need to rewrite Phobos to not use a GC" ... Um wait, what?

 I am not saying to Phobos needs to allocate as much as it does, 
 just that argument itself fails to met a basic standard of 
 logical coherency.


No, you read way more in that than what I said. I am not talking
about the .NET GC(I or I would have said ".NET GC"). You totally
fabricated what you think I "effectively said".

By writing the phobos like .NET, I mean, in terms of .NET's
organization and structure.
Feb 02 2014
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 02/01/2014 06:57 PM, Frustrated wrote:

 Right, and because of the mentality that the GC is one size that
 fits all, phobos got this way. That mentality is still pervasive. ...


In the D community? Hardly.


 I call it laziness.
 ...


Well, name-calling does not get anything fixed.
Feb 01 2014
prev sibling parent reply "pjmlp" <pjmlp progtools.org> writes:
On Wednesday, 2 January 2013 at 11:41:33 UTC, 
DythroposTheImposter wrote:

  I'm interested in how the new LuaJIT GC ends up performing. 
 But overall I can't say I have much hope for GC right now.

  GC/D = Generally Faster allocation. Has a cost associated with 
 every living object.

  C++ = Generally Slower allocation, but while it is alive there 
 is no cost.

  So as the heap grows, the GC language falls behind.

  This seems to be the case in every language I've looked at 
 this uses a GC.


As a former user from Native Oberon and BlueBottle OS back in the 
late 90's, I can attest that the GC was fast enough for powering 
a single user desktop operating system.

Surely there are cases where it is an issue, mainly due to 
hardware constraints.

While I am a GC supporter, I wouldn't like to fly in an airplane 
with a GC system, but a few missile radar systems are controled 
with GC based systems (Ground Master 400). So who knows?!

--
Paulo
Jan 02 2013
parent reply "Frustrated" <c1514843 drdrb.com> writes:
On Wednesday, 2 January 2013 at 22:46:20 UTC, pjmlp wrote:

 On Wednesday, 2 January 2013 at 11:41:33 UTC, 
 DythroposTheImposter wrote:

 I'm interested in how the new LuaJIT GC ends up performing. 
 But overall I can't say I have much hope for GC right now.

 GC/D = Generally Faster allocation. Has a cost associated with 
 every living object.

 C++ = Generally Slower allocation, but while it is alive there 
 is no cost.

 So as the heap grows, the GC language falls behind.

 This seems to be the case in every language I've looked at 
 this uses a GC.


 As a former user from Native Oberon and BlueBottle OS back in 
 the late 90's, I can attest that the GC was fast enough for 
 powering a single user desktop operating system.

 Surely there are cases where it is an issue, mainly due to 
 hardware constraints.

 While I am a GC supporter, I wouldn't like to fly in an 
 airplane with a GC system, but a few missile radar systems are 
 controled with GC based systems (Ground Master 400). So who 
 knows?!

 --
 Paulo



The problem isn't stability, as you manual management can be just
or even more error prone. The problem is with real time. If the
GC must stop the world then it might be a stability issue with
respect to the design. Audio and gaming are critical. While in
some instances it is only superficial, such as video games, but
absolutely a deal breaker with professional audio... even though
in a sense these are not as critical as, say, a missile system,
they are used much more often so are put to the test much more.
There is, also, redundancy built into such critical systems as
military defense... not so in a digital mixer for a PA that can't
pop and click because the GC has to compact the heap. Your
audience is going to be quite annoyed when they here a loud
pop(due to the audio cutting off and potentially going from a
high value to a low value which produces a very loud click) every
30 seconds, minute or so. Not to mention what that would do to
their hearing over the course of a night.

What if the GC decides to pause just at the wrong time when you
only needed just a little bit more time to get the audio buffer
filled? You are screwed. Manual management or ARC or whatever is
at least much more predictable. Sure you can disable the GC but
when you do that you just make things worse(potentially takes
longer to catch up).

I think with proper management these things can be avoided, but
as of now, since the D core relies so much on the GC, it's kinda
pointless unless you avoid all the goodies it provides. The
mindset that the GC is the ultimate way to deal with memory
management has led to the issues with D and the GC and why some
avoid D. If it would have been designed as optional, we wouldn't
be talking about these problems.
Jan 31 2014
parent "Frank Bauer" <y z.com> writes:
+1.

What it boils down to for me then, is:

If I want to write applications without low-latency requirements,
D is the perfect fit for me. D's templates, slices, CTFE instead
of a nasty macro or preprocessor language, compiler producing


If on the other hand, I do have low-latency requirements, I
simply don't want to jump through hoops, i.e. implement all
critical memory management C++ style by myself to prevent the GC
from kicking in. And If I even have to use GC profiling on top of
that, D's cleaner language concepts just don't cut it for me.
I'll take a look at Rust for that (admitted, not as well thought
out a language as D).
Jan 31 2014
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, December 31, 2012 15:05:00 Kiith-Sa wrote:

 I think you're overthinking this way too much.
 
 I'm writing a game engine, where latency is much more important
 than in your case. Even 10ms would translate to visible jitter.
 GC's not an issue, and disabling it permanently is very
 counterproductive _unless_ you've exhausted all other options
 (which you're unlikely to, as you can do everything you can do in
 C++).  All you have to do is care about how you allocate and, if
 GC seems to be an issue, profile to see _where_ the GC is being
 called most and optimize those allocations.
 
 Basic rules:


[snip]

If you've got this much figured out with reasonably clear guidelines, you 
should consider writing a blog post or article about it.

- Jonathan M Davis
Jan 01 2013
prev sibling parent reply "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Monday, 31 December 2012 at 14:05:01 UTC, Kiith-Sa wrote:

 All you have to do is care about how you allocate and, if
 GC seems to be an issue, profile to see _where_ the GC is being
 called most and optimize those allocations.


Bumping old thread:

There is an ever-increasing amount of newcomers in #d on IRC who 
end up asking how to avoid the GC or at least to be able to 
determine where implicit allocations happen. I think we should 
work on creating a wiki page and gather as much advice as 
possible on dealing with the GC, implicit allocations, real-time 
constraints, etc.
Jan 31 2014
parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 31 January 2014 at 19:08:10 UTC, Andrej Mitrovic wrote:

 On Monday, 31 December 2012 at 14:05:01 UTC, Kiith-Sa wrote:

 All you have to do is care about how you allocate and, if
 GC seems to be an issue, profile to see _where_ the GC is being
 called most and optimize those allocations.


 Bumping old thread:

 There is an ever-increasing amount of newcomers in #d on IRC 
 who end up asking how to avoid the GC or at least to be able to 
 determine where implicit allocations happen. I think we should 
 work on creating a wiki page and gather as much advice as 
 possible on dealing with the GC, implicit allocations, 
 real-time constraints, etc.


We should concentrate on implementing scope. scope could prevent 
many of such things.
E.g.:

{
      scope int[] arr = new int[count];
}

Since scope shall guarantee that the marked variable does not 
leave the scope, the compiler could safely assume that arr 
shouldn't handled with the GC.
With that information the code could safely rewritten to:

{
      int* tmp_arr_ptr = cast(int*) calloc(count, int.sizeof);
      scope(exit) free(tmp_arr_ptr);
      int[] arr = tmp_arr_ptr[0 .. count];
}

So no need to invoke the GC.
Since many objects have a small lifetime it is preferable to 
avoid the GC. With scope the user could mark such objects... and 
everyone wins. It is not perfect and does not cover all use cases 
but it would be a start.

Sadly I doubt that scope gets implemented (soon). And since scope 
classes was replaced with an ugly library solution I seriously 
doubt that this nice dream comes true. But it would be so nice. :)
Jan 31 2014
parent reply "Frank Bauer" <y z.com> writes:
Yes, that is what Rust calls an "owning pointer". While we are at 
it, we might as well dream of "non-owning" or "borrowed" 
pointers, a.k.a. references in Rust. They don't have an effect on 
memory-deallocation when they go out of scope, but they prevent 
the owning pointer they "borrow" from (i.e. are assigned from) 
from being reassigned to a different object as long as there are 
borrowing references in scope. There is more to owning and 
borrowed pointers, but I think that is the essence.

 Andrei (who I believe is the go-to (!) guy for all things 
memeory allocation related right now):
IIRC you mentioned that it was convenient to have the GC around 
for implementing the D language features. Would it just be a 
minor inconvenience to drop that dependency in the generated 
compiler output and replace it with new/delete or something 
equivalent to owning pointers, say over the next one or two 
years? Or would it be a major roadblock that would require too 
much redesign work? Maybe you could test the concept of owning 
and borrowed pointers internally for some compiler components 
before actually bringing them "to the surface" for us to play 
with, if it turns out useful. But only of course, if we could 
leave the rest of D as good as it is today.

I would really like to see the GC-free requesting crowd be taken 
a lttle more seriously. Without asking them to forego D features 
or do manual C++-style memory management. Javas and .NETs garbage 
collectors had enough time to mature and are still unsatisfactory 
for many applications.
Jan 31 2014
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 01.02.2014 05:33, schrieb Frank Bauer:

 Yes, that is what Rust calls an "owning pointer". While we are at it, we
 might as well dream of "non-owning" or "borrowed" pointers, a.k.a.
 references in Rust. They don't have an effect on memory-deallocation
 when they go out of scope, but they prevent the owning pointer they
 "borrow" from (i.e. are assigned from) from being reassigned to a
 different object as long as there are borrowing references in scope.
 There is more to owning and borrowed pointers, but I think that is the
 essence.

  Andrei (who I believe is the go-to (!) guy for all things memeory
 allocation related right now):
 IIRC you mentioned that it was convenient to have the GC around for
 implementing the D language features. Would it just be a minor
 inconvenience to drop that dependency in the generated compiler output
 and replace it with new/delete or something equivalent to owning
 pointers, say over the next one or two years? Or would it be a major
 roadblock that would require too much redesign work? Maybe you could
 test the concept of owning and borrowed pointers internally for some
 compiler components before actually bringing them "to the surface" for
 us to play with, if it turns out useful. But only of course, if we could
 leave the rest of D as good as it is today.

 I would really like to see the GC-free requesting crowd be taken a lttle
 more seriously. Without asking them to forego D features or do manual
 C++-style memory management. Javas and .NETs garbage collectors had
 enough time to mature and are still unsatisfactory for many applications.


Just to nitpick, reference counting is also GC from CS point of view. :)
Feb 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 1 February 2014 at 04:33:23 UTC, Frank Bauer wrote:

 Yes, that is what Rust calls an "owning pointer". While we are 
 at it, we might as well dream of "non-owning" or "borrowed" 
 pointers, a.k.a. references in Rust. They don't have an effect 
 on memory-deallocation when they go out of scope, but they 
 prevent the owning pointer they "borrow" from (i.e. are 
 assigned from) from being reassigned to a different object as 
 long as there are borrowing references in scope. There is more 
 to owning and borrowed pointers, but I think that is the 
 essence.


I think D can use simplified model with only `scope`. Using it as 
storage class will effectively make data owned (not necessarily 
pointer), using it as qualifier will allow to "borrow" scope 
reference/pointer, for example, to pass as function argument.
Feb 01 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/31/14, 8:33 PM, Frank Bauer wrote:

  Andrei (who I believe is the go-to (!) guy for all things memeory
 allocation related right now):
 IIRC you mentioned that it was convenient to have the GC around for
 implementing the D language features. Would it just be a minor
 inconvenience to drop that dependency in the generated compiler output
 and replace it with new/delete or something equivalent to owning
 pointers, say over the next one or two years? Or would it be a major
 roadblock that would require too much redesign work?


It would be a major effort. It's possible we have to do it.


 Maybe you could
 test the concept of owning and borrowed pointers internally for some
 compiler components before actually bringing them "to the surface" for
 us to play with, if it turns out useful. But only of course, if we could
 leave the rest of D as good as it is today.

 I would really like to see the GC-free requesting crowd be taken a lttle
 more seriously. Without asking them to forego D features or do manual
 C++-style memory management. Javas and .NETs garbage collectors had
 enough time to mature and are still unsatisfactory for many applications.


I consider memory allocation a major issue to focus on in 2014.


Andrei
Feb 01 2014
prev sibling next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 31.12.2012 13:14, schrieb Sven Over:

 A smart-pointer type for arrays can easily provide slices. It keeps a
 reference to the full array (which gets destructed when the last
 reference is dropped), but addresses a subrange.


I did exactly implement this in 
https://github.com/Ingrater/druntime/blob/master/src/core/refcounted.d
Basically I implemented a struct that mimics a regular D array as far as 
possible, but it is using reference counting under the hood.

Kind Regards
Benjamin Thaut
Dec 31 2012
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 31 December 2012 at 12:14:22 UTC, Sven Over wrote:

 On Tuesday, 25 December 2012 at 19:23:59 UTC, Jonathan M Davis 
 wrote:

 There's also often no reason not to have the GC on and use it 
 for certain stuff


 One thing that really freaks me out is the fact that the 
 garbage collector pauses the whole process, i.e. all threads.

 In my job I'm writing backend services that power a big web 
 site. Perfomance is key, as the response time of the data 
 service in most cases directly adds to the page load time. The 
 bare possibility that the whole service pauses for, say, 100ms 
 is making me feel very uncomfortable.


I understand that. However, refcounted stuff tends to die in 
cluster as well and create pauses.

The main issue here is clearly GC's implementation rather than 
the concept of GC in itself (which can be quite good at avoiding 
pauses if you are ready to make some other tradeoffs).


 We easily achieve the performance and reliability we need in 
 C++, but I would love to give D a chance, as it solves many 
 inconveniences of C++ in an elegant way. Metaprogramming and 
 the threading model, just to name two.


Here is something I tried in the past with some success : use 
RefCounted and GC.free . Doing so, you allocate in the GC heap, 
but you will limit greatly the amount of garbage that the GC have 
to collect by itself.

Note that in some cases, GC means greater performances (usually 
when associated with immutability), so disabling it entirely 
don't seems a good idea to me.
Jan 01 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 12/31/2012 7:14 AM, Sven Over wrote:

 In my job I'm writing backend services that power a big web site.
 Perfomance is key, as the response time of the data service in most
 cases directly adds to the page load time. The bare possibility that the
 whole service pauses for, say, 100ms is making me feel very uncomfortable.


FWIW, Vibe.d <http://vibed.org/> is very careful about avoiding 
unnecessary GC allocations.
Jan 31 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 31 December 2012 at 12:14:22 UTC, Sven Over wrote:

 In my job I'm writing backend services that power a big web 
 site. Perfomance is key, as the response time of the data 
 service in most cases directly adds to the page load time. The 
 bare possibility that the whole service pauses for, say, 100ms 
 is making me feel very uncomfortable.


Back to the original scenario:

Current CPUs have a measured peak throughput of 2-4GB per 100ms. 
So if you keep pointers clustered on cache lines in a way that is 
compatible with the AGCs prefetching patterns, then you ought to 
make it for most applications if the collector is optimized?

If your backend service (data) is replicated on multiple servers 
you could, of course, let the servers run the GC at different 
times organized with a load-balancer/router/task scheduler.

So if GC makes you more productive on a web server then I don't 
really see why you don't use it.

Now, that might not work out in the future though. If CPU caches 
keep growing to the extent that most of your working dataset sits 
in the cache and you have lots of rather slow main memory then 
the AGC process will become sloooow no matter how good the GC is.
Feb 03 2014
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:


  std.typecons.RefCounted!T

 core.memory.GC.disable();


 Wow. That was easy.

 I see, D's claim of being a multi-paradigm language is not false.



It's not a realistic suggestion. Everything you want to link uses the GC,
and the language its self also uses the GC. Unless you write software in
complete isolation and forego many valuable features, it's not a solution.


 Phobos does rely on the GC to some extent. Most algorithms and ranges do

 not though.


 Running (library) code that was written with GC in mind and turning GC off
 doesn't sound ideal.

 But maybe this allows me to familiarise myself more with D. Who knows,
 maybe I can learn to stop worrying and love garbage collection.

 Thanks for your help!


I've been trying to learn to love the GC for as long as I've been around
here. I really wanted to break that mental barrier, but it hasn't happened.

wishlist item for D is the ability to use a reference counted collector in
place of the built-in GC.
You're not alone :)

I write realtime and memory-constrained software (console games), and for
me, I think the biggest issue that can never be solved is the
non-deterministic nature of the collect cycles, and the unknowable memory
footprint of the application. You can't make any guarantees or predictions
about the GC, which is fundamentally incompatible with realtime software.
Language-level ARC would probably do quite nicely for the miscellaneous
allocations. Obviously, bulk allocations are still usually best handled in
a context sensitive manner; ie, regions/pools/freelists/whatever, but the
convenience of the GC paradigm does offer some interesting and massively
time-saving features to D.
Everyone will always refer you to RefCounted, which mangles your types and
pollutes your code, but aside from that, for ARC to be useful, it needs to
be supported at the language-level, such that the language/optimiser is
able to optimise out redundant incref/decref calls, and also that it is
compatible with immutable (you can't manage a refcount if the object is
immutable).
Jan 31 2014
next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Fri, 31 Jan 2014 21:29:04 -0800, Manu <turkeyman gmail.com> wrote:


 On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:


  std.typecons.RefCounted!T

 core.memory.GC.disable();


 Wow. That was easy.

 I see, D's claim of being a multi-paradigm language is not false.



 It's not a realistic suggestion. Everything you want to link uses the GC,
 and the language its self also uses the GC. Unless you write software in
 complete isolation and forego many valuable features, it's not a  
 solution.


  Phobos does rely on the GC to some extent. Most algorithms and ranges do

 not though.


 Running (library) code that was written with GC in mind and turning GC  
 off
 doesn't sound ideal.

 But maybe this allows me to familiarise myself more with D. Who knows,
 maybe I can learn to stop worrying and love garbage collection.

 Thanks for your help!


 I've been trying to learn to love the GC for as long as I've been around
 here. I really wanted to break that mental barrier, but it hasn't  
 happened.
 In fact, I am more than ever convinced that the GC won't do. My current  

 wishlist item for D is the ability to use a reference counted collector  
 in
 place of the built-in GC.
 You're not alone :)

 I write realtime and memory-constrained software (console games), and for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable memory
 footprint of the application. You can't make any guarantees or  
 predictions
 about the GC, which is fundamentally incompatible with realtime software.
 Language-level ARC would probably do quite nicely for the miscellaneous
 allocations. Obviously, bulk allocations are still usually best handled  
 in
 a context sensitive manner; ie, regions/pools/freelists/whatever, but the
 convenience of the GC paradigm does offer some interesting and massively
 time-saving features to D.
 Everyone will always refer you to RefCounted, which mangles your types  
 and
 pollutes your code, but aside from that, for ARC to be useful, it needs  
 to
 be supported at the language-level, such that the language/optimiser is
 able to optimise out redundant incref/decref calls, and also that it is
 compatible with immutable (you can't manage a refcount if the object is
 immutable).


The problem isn't GC's per se. But D's horribly naive implementation,  
games are written on GC languages now all the time (Unity/.NET). And let's  
be honest, games are kind of a speciality, games do things most programs  
will never do.

You might want to read the GC Handbook. GC's aren't bad, but most, like  
the D GC, are just to simplistic for common usage today.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator
Jan 31 2014
parent reply Manu <turkeyman gmail.com> writes:
On 1 February 2014 16:26, Adam Wilson <flyboynw gmail.com> wrote:


 On Fri, 31 Jan 2014 21:29:04 -0800, Manu <turkeyman gmail.com> wrote:

  On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:

   std.typecons.RefCounted!T

 core.memory.GC.disable();


 Wow. That was easy.

 I see, D's claim of being a multi-paradigm language is not false.



 It's not a realistic suggestion. Everything you want to link uses the GC,
 and the language its self also uses the GC. Unless you write software in
 complete isolation and forego many valuable features, it's not a solution.


  Phobos does rely on the GC to some extent. Most algorithms and ranges do


 not though.


 Running (library) code that was written with GC in mind and turning GC
 off
 doesn't sound ideal.

 But maybe this allows me to familiarise myself more with D. Who knows,
 maybe I can learn to stop worrying and love garbage collection.

 Thanks for your help!


 I've been trying to learn to love the GC for as long as I've been around
 here. I really wanted to break that mental barrier, but it hasn't
 happened.

 wishlist item for D is the ability to use a reference counted collector in
 place of the built-in GC.
 You're not alone :)

 I write realtime and memory-constrained software (console games), and for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable memory
 footprint of the application. You can't make any guarantees or predictions
 about the GC, which is fundamentally incompatible with realtime software.
 Language-level ARC would probably do quite nicely for the miscellaneous
 allocations. Obviously, bulk allocations are still usually best handled in
 a context sensitive manner; ie, regions/pools/freelists/whatever, but the
 convenience of the GC paradigm does offer some interesting and massively
 time-saving features to D.
 Everyone will always refer you to RefCounted, which mangles your types and
 pollutes your code, but aside from that, for ARC to be useful, it needs to
 be supported at the language-level, such that the language/optimiser is
 able to optimise out redundant incref/decref calls, and also that it is
 compatible with immutable (you can't manage a refcount if the object is
 immutable).


 The problem isn't GC's per se. But D's horribly naive implementation,
 games are written on GC languages now all the time (Unity/.NET). And let's
 be honest, games are kind of a speciality, games do things most programs
 will never do.

 You might want to read the GC Handbook. GC's aren't bad, but most, like
 the D GC, are just to simplistic for common usage today.



Maybe a sufficiently advanced GC could address the performance
non-determinism to an acceptable level, but you're still left with the
memory non-determinism, and the conundrum that when your heap approaches
full (which is _always_ on a games console), the GC has to work harder and
harder, and more often to try and keep the tiny little bit of overhead
available.
A GC heap by nature expects you to have lots of memory, and also lots of
FREE memory.

No serious console game I'm aware of has ever been written in a language
with a GC. Casual games, or games that don't attempt to raise the bar may
get away with it, but that's not the industry I work in.



 Jan 31 2014



 next sibling parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Fri, 31 Jan 2014 23:35:44 -0800, Manu <turkeyman gmail.com> wrote:


 On 1 February 2014 16:26, Adam Wilson <flyboynw gmail.com> wrote:


 On Fri, 31 Jan 2014 21:29:04 -0800, Manu <turkeyman gmail.com> wrote:

  On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:

   std.typecons.RefCounted!T

 core.memory.GC.disable();


 Wow. That was easy.

 I see, D's claim of being a multi-paradigm language is not false.



 It's not a realistic suggestion. Everything you want to link uses the  
 GC,
 and the language its self also uses the GC. Unless you write software  
 in
 complete isolation and forego many valuable features, it's not a  
 solution.


  Phobos does rely on the GC to some extent. Most algorithms and ranges  
 do


 not though.


 Running (library) code that was written with GC in mind and turning GC
 off
 doesn't sound ideal.

 But maybe this allows me to familiarise myself more with D. Who knows,
 maybe I can learn to stop worrying and love garbage collection.

 Thanks for your help!


 I've been trying to learn to love the GC for as long as I've been  
 around
 here. I really wanted to break that mental barrier, but it hasn't
 happened.
 In fact, I am more than ever convinced that the GC won't do. My  

 wishlist item for D is the ability to use a reference counted  
 collector in
 place of the built-in GC.
 You're not alone :)

 I write realtime and memory-constrained software (console games), and  
 for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable  
 memory
 footprint of the application. You can't make any guarantees or  
 predictions
 about the GC, which is fundamentally incompatible with realtime  
 software.
 Language-level ARC would probably do quite nicely for the miscellaneous
 allocations. Obviously, bulk allocations are still usually best  
 handled in
 a context sensitive manner; ie, regions/pools/freelists/whatever, but  
 the
 convenience of the GC paradigm does offer some interesting and  
 massively
 time-saving features to D.
 Everyone will always refer you to RefCounted, which mangles your types  
 and
 pollutes your code, but aside from that, for ARC to be useful, it  
 needs to
 be supported at the language-level, such that the language/optimiser is
 able to optimise out redundant incref/decref calls, and also that it is
 compatible with immutable (you can't manage a refcount if the object is
 immutable).


 The problem isn't GC's per se. But D's horribly naive implementation,
 games are written on GC languages now all the time (Unity/.NET). And  
 let's
 be honest, games are kind of a speciality, games do things most programs
 will never do.

 You might want to read the GC Handbook. GC's aren't bad, but most, like
 the D GC, are just to simplistic for common usage today.



 Maybe a sufficiently advanced GC could address the performance
 non-determinism to an acceptable level, but you're still left with the
 memory non-determinism, and the conundrum that when your heap approaches
 full (which is _always_ on a games console), the GC has to work harder  
 and
 harder, and more often to try and keep the tiny little bit of overhead
 available.
 A GC heap by nature expects you to have lots of memory, and also lots of
 FREE memory.

 No serious console game I'm aware of has ever been written in a language
 with a GC. Casual games, or games that don't attempt to raise the bar may
 get away with it, but that's not the industry I work in.


That's kind of my point. You're asking for massive changes throughout the  
entire compiler to support what is becoming more of an edge case, not less  
of one. For the vast majority of use cases, a GC is the right call and D  
has to cater to the majority if it wants to gain any significant mindshare  
at all. You don't grow by increasing specialization...

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



 next sibling parent reply "Frank Bauer" <x y.com>  writes:


On Saturday, 1 February 2014 at 09:27:18 UTC, Adam Wilson wrote:

 For the vast majority of use cases, a GC is the right call and 
 D has to cater to the majority if it wants to gain any 
 significant mindshare at all. You don't grow by increasing 
 specialization...



JITing their code but precisely for their forced use of a GC is a 
"minority" issue then? Again, please take those of us more 
serious who don't like a GC to interfere with their business and 
if it's only for "programming hygiene" reasons. As you consider 
handling low latency requirements undue "specialisation" anyways.

On Saturday, 1 February 2014 at 12:04:56 UTC, JR wrote:

 To my uneducated eyes, an ARC collector does seem like the 
 near-ideal solution


ARC would be a plus for heavily interdependent code. But it 
doesn't beat unique_ptr semantics in local use that free their 
memory immediately as they go out of scope or are reassigned.


 I would *dearly* love to have concurrency in whatever we end up 
 with, though. For a multi-core personal computer threads are 
 free lunches, or close enough so.


No they are not. I want to make good use of all the cores I have 
available and I don't want to share them a bit with the GC. It's 
myyy ... precious.

As you may have guessed, my criticism is harsh because I love the 
language :)



 Feb 01 2014



 next sibling parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 06:33:15 -0800, Frank Bauer <x y.com> wrote:


 On Saturday, 1 February 2014 at 09:27:18 UTC, Adam Wilson wrote:

 For the vast majority of use cases, a GC is the right call and D has to  
 cater to the majority if it wants to gain any significant mindshare at  
 all. You don't grow by increasing specialization...



 their code but precisely for their forced use of a GC is a "minority"  
 issue then? Again, please take those of us more serious who don't like a  
 GC to interfere with their business and if it's only for "programming  
 hygiene" reasons. As you consider handling low latency requirements  
 undue "specialisation" anyways.


Real-time or low-latency programming is absolutely specialization, and for  
more reasons than a GC. It might be what you do all day, but it's not what  
the majority of programmers do all day.

As far as catering to C++. TIOBE says that C++ made up 7.5% of programmer  

That's almost three-times the interest. Like it or not, C++ IS the  
minority and D's mission is not solely based on converting C++ users, many  
of whom can't convert anyways. The reason I picked those three languages  
is because they are D's primary audience.


 On Saturday, 1 February 2014 at 12:04:56 UTC, JR wrote:

 To my uneducated eyes, an ARC collector does seem like the near-ideal  
 solution


 ARC would be a plus for heavily interdependent code. But it doesn't beat  
 unique_ptr semantics in local use that free their memory immediately as  
 they go out of scope or are reassigned.


So you want C++ back then? Got it. But the goal of D is NOT to be a  
slightly better C++, it's a be a Safe, Simple, systems language with well  
defined semantics that just happens to trace it's lineage back to C++.


 I would *dearly* love to have concurrency in whatever we end up with,  
 though. For a multi-core personal computer threads are free lunches, or  
 close enough so.


 No they are not. I want to make good use of all the cores I have  
 available and I don't want to share them a bit with the GC. It's myyy  
 ... precious.


Ahem. You only have the illusion of control over the CPU, the OS scheduler  
set's the rules, all you can do is provide hints. Giving some up to the GC  
will not materially affect the performance of the your code since you  
never had a choice anyways.


 As you may have guessed, my criticism is harsh because I love the  
 language :)



-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



prev sibling  parent reply Manu <turkeyman gmail.com>  writes:


On 2 February 2014 00:33, Frank Bauer <x y.com> wrote:


 On Saturday, 1 February 2014 at 09:27:18 UTC, Adam Wilson wrote:


 For the vast majority of use cases, a GC is the right call and D has to
 cater to the majority if it wants to gain any significant mindshare at all.
 You don't grow by increasing specialization...



 their code but precisely for their forced use of a GC is a "minority" issue
 then? Again, please take those of us more serious who don't like a GC to
 interfere with their business and if it's only for "programming hygiene"
 reasons. As you consider handling low latency requirements undue
 "specialisation" anyways.


 On Saturday, 1 February 2014 at 12:04:56 UTC, JR wrote:


 To my uneducated eyes, an ARC collector does seem like the near-ideal
 solution


 ARC would be a plus for heavily interdependent code. But it doesn't beat
 unique_ptr semantics in local use that free their memory immediately as
 they go out of scope or are reassigned.



Why wouldn't ARC do this?

 I would *dearly* love to have concurrency in whatever we end up with,

 though. For a multi-core personal computer threads are free lunches, or
 close enough so.


 No they are not. I want to make good use of all the cores I have available
 and I don't want to share them a bit with the GC. It's myyy ... precious.

 As you may have guessed, my criticism is harsh because I love the language
 :)





 Feb 01 2014



  parent reply "Frank Bauer" <y z.com>  writes:


On Sunday, 2 February 2014 at 02:05:16 UTC, Manu wrote:

 On 2 February 2014 00:33, Frank Bauer <x y.com> wrote:

 ARC would be a plus for heavily interdependent code. But it 
 doesn't beat
 unique_ptr semantics in local use that free their memory 
 immediately as
 they go out of scope or are reassigned.



 Why wouldn't ARC do this?


Because in a local scope you could decide when to free the object 
at compile time (i.e. end of scope) and not drag a reference 
count around when it is not necessary?



 Feb 01 2014



  parent reply "Frank Bauer" <y z.com>  writes:


Just to clarify: I would like to have the following pointers. 
I'll start with the GC ones to maybe reach some consensus quicker 
;)

1. GC(T)
Pointer to a heap allocated T, garbage collected.

2. ARC(T)
Reference counted pointer to a heap allocated T, compiler manages 
lifetime automatically via retain/ release.

3. Owning(T)
Pointer to a heap allocated T, freed when pointer goes out of 
scope or is reassigned (some special interactions with 
Borrowed(T) on top of that).

4. Borrowed(T)
Pointer to a T, not necessarily heap allocated, has no impact on 
memory allocation/ deallocation of the T whatsoever, if assigned 
from an Owning(T) prevents that owning pointer from being freed 
or reassigned as long as the borrowed pointer is in scope.

Performance, or, should I say, meddling of the compiler and 
runtime in your code, is from worst to best.

We could argue over which pointer concepts would be granted the 
honor of receiving their own D definition syntax (new, ~,  , &, 
whatever).



 Feb 01 2014



 next sibling parent reply "Frank Bauer" <y z.com>  writes:


Of course, the big deal is: if I opt for ARC(T), Owning(T) and 
Borrowed(T) exclusively, the GC has no business running for a 
microsecond in my code or the standard library functions I call.



 Feb 01 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 19:15:38 -0800, Frank Bauer <y z.com> wrote:


 Of course, the big deal is: if I opt for ARC(T), Owning(T) and  
 Borrowed(T) exclusively, the GC has no business running for a  
 microsecond in my code or the standard library functions I call.


That's not a guarantee that can be reasonably made or demanded. You have  
no idea if a library you are using is written using the GC and turning  
those on would have no way to impact the allocation scheme of the library.  
As long as there is a possibility of a GC, D will have to pass an active  
GC instance to any libraries that are loaded.

The GC only runs right now when a GC allocation is requested. I've been  
saying this all over the forums but people seem to persist in the mindset  
that the GC is a black-box that runs at completely random intervals, for  
random intervals, all determined by voodoo and black magic. It's not, and  
it doesn't. The GC only runs when an allocation is requested that cannot  
be filled from the current available heap. If you don't use the GC, by  
definition it will not run, unless a library you depend on uses it.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



  parent reply "Frank Bauer" <y z.com>  writes:


On Sunday, 2 February 2014 at 03:38:08 UTC, Adam Wilson wrote:

 You have no idea if a library you are using is written using 
 the GC.


Well, the library writer might tell me, if she used GC(T) (the 
linker might as well) ...


 If you don't use the GC, by definition it will not run, unless 
 a library you depend on uses it.


big UNLESS!



 Feb 01 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 20:17:47 -0800, Frank Bauer <y z.com> wrote:


 On Sunday, 2 February 2014 at 03:38:08 UTC, Adam Wilson wrote:

 You have no idea if a library you are using is written using the GC.


 Well, the library writer might tell me, if she used GC(T) (the linker  
 might as well) ...


 If you don't use the GC, by definition it will not run, unless a  
 library you depend on uses it.


 big UNLESS!


Maybe, but my point is that even with no allocations in phobos, you'd have  
to verify that ALL the libraries you use don't GC allocate. That's why so  
many people in the ARC camp want to make it the only GC system in D.  
Multiple GC systems increase your cognitive load considerably.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



  parent reply "Frank Bauer" <y z.com>  writes:


On Sunday, 2 February 2014 at 04:23:35 UTC, Adam Wilson wrote:

 Maybe, but my point is that even with no allocations in phobos, 
 you'd have to verify that ALL the libraries you use don't GC 
 allocate. That's why so many people in the ARC camp want to 
 make it the only GC system in D. Multiple GC systems increase 
 your cognitive load considerably.


Good point. What we DON'T want is a library that comes in 
MySuperLib-GC, MySuperLib-ARC, MySuperLib-Owning and 
MySuperLib-Borrowed flavors.

But I don't think that is necessary.

A library function would typically take arguments by reference 
aka as a Borrowed(T). GC(T), ARC(T) and Owning(T) convert equally 
well to Borrowed(T). So somebody who uses only GC(T) in her code 
could use the library just as easily as someone who prefers 
Owning(T). The library would be compatible with the two, because 
it does not own the (reference to the) passed object. GC(T) or 
Owning(T) at the call site would free the object automatically 
(at the next collection cycle or when leaving the scope). 
Everybody is happy!

All that is left for the caller is to choose whether she wants to 
use a library that uses GC *internally*. Interaction is possible 
in every combination:

1. I use GC(T),     the library uses GC(T) internally     -> 
works (with GC),
2. I use Owning(T), the library uses GC(T) internally     -> 
works (with GC),
3. I use GC(T),     the library uses Owning(T) internally -> 
works (with GC),
4. I use Owning(T), the library uses Owning(T) internally -> 
works (w/o GC).



 Feb 01 2014



 next sibling parent  "Frank Bauer" <y z.com>  writes:


Of course to make everyone in 4 happy (i.e. me;) there would have 
to be a Phobos that uses Owning(T) internally exclusively.



 Feb 01 2014



prev sibling  parent reply "Martin Cejp" <minexew gmail.com>  writes:


On Sunday, 2 February 2014 at 05:12:05 UTC, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 04:23:35 UTC, Adam Wilson wrote:

 Maybe, but my point is that even with no allocations in 
 phobos, you'd have to verify that ALL the libraries you use 
 don't GC allocate. That's why so many people in the ARC camp 
 want to make it the only GC system in D. Multiple GC systems 
 increase your cognitive load considerably.


 Good point. What we DON'T want is a library that comes in 
 MySuperLib-GC, MySuperLib-ARC, MySuperLib-Owning and 
 MySuperLib-Borrowed flavors.

 But I don't think that is necessary.

 A library function would typically take arguments by reference 
 aka as a Borrowed(T). GC(T), ARC(T) and Owning(T) convert 
 equally well to Borrowed(T). So somebody who uses only GC(T) in 
 her code could use the library just as easily as someone who 
 prefers Owning(T). The library would be compatible with the 
 two, because it does not own the (reference to the) passed 
 object. GC(T) or Owning(T) at the call site would free the 
 object automatically (at the next collection cycle or when 
 leaving the scope). Everybody is happy!

 All that is left for the caller is to choose whether she wants 
 to use a library that uses GC *internally*. Interaction is 
 possible in every combination:

 1. I use GC(T),     the library uses GC(T) internally     -> 
 works (with GC),
 2. I use Owning(T), the library uses GC(T) internally     -> 
 works (with GC),
 3. I use GC(T),     the library uses Owning(T) internally -> 
 works (with GC),
 4. I use Owning(T), the library uses Owning(T) internally -> 
 works (w/o GC).


This exists. It's called C++.

I've pretty much given up on GC and do all my allocations in the 
real-time path manually. In a game, there shouldn't be that much 
stuff to allocate once live, anyway.



 Feb 02 2014



  parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 02.02.2014 10:03, schrieb Martin Cejp:

 On Sunday, 2 February 2014 at 05:12:05 UTC, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 04:23:35 UTC, Adam Wilson wrote:

 Maybe, but my point is that even with no allocations in phobos, you'd
 have to verify that ALL the libraries you use don't GC allocate.
 That's why so many people in the ARC camp want to make it the only GC
 system in D. Multiple GC systems increase your cognitive load
 considerably.


 Good point. What we DON'T want is a library that comes in
 MySuperLib-GC, MySuperLib-ARC, MySuperLib-Owning and
 MySuperLib-Borrowed flavors.

 But I don't think that is necessary.

 A library function would typically take arguments by reference aka as
 a Borrowed(T). GC(T), ARC(T) and Owning(T) convert equally well to
 Borrowed(T). So somebody who uses only GC(T) in her code could use the
 library just as easily as someone who prefers Owning(T). The library
 would be compatible with the two, because it does not own the
 (reference to the) passed object. GC(T) or Owning(T) at the call site
 would free the object automatically (at the next collection cycle or
 when leaving the scope). Everybody is happy!

 All that is left for the caller is to choose whether she wants to use
 a library that uses GC *internally*. Interaction is possible in every
 combination:

 1. I use GC(T),     the library uses GC(T) internally     -> works
 (with GC),
 2. I use Owning(T), the library uses GC(T) internally     -> works
 (with GC),
 3. I use GC(T),     the library uses Owning(T) internally -> works
 (with GC),
 4. I use Owning(T), the library uses Owning(T) internally -> works
 (w/o GC).


 This exists. It's called C++.

 I've pretty much given up on GC and do all my allocations in the
 real-time path manually. In a game, there shouldn't be that much stuff
 to allocate once live, anyway.


The problem with C++ is the C traps that live within and are the cause 
of many C++ gotchas, as described by Bjarne on the "The Design and 
Evolution of C++" book.

--
Paulo



 Feb 02 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/1/14, 6:59 PM, Frank Bauer wrote:

 Just to clarify: I would like to have the following pointers. I'll start
 with the GC ones to maybe reach some consensus quicker ;)

 1. GC(T)
 Pointer to a heap allocated T, garbage collected.

 2. ARC(T)
 Reference counted pointer to a heap allocated T, compiler manages
 lifetime automatically via retain/ release.

 3. Owning(T)
 Pointer to a heap allocated T, freed when pointer goes out of scope or
 is reassigned (some special interactions with Borrowed(T) on top of that).

 4. Borrowed(T)
 Pointer to a T, not necessarily heap allocated, has no impact on memory
 allocation/ deallocation of the T whatsoever, if assigned from an
 Owning(T) prevents that owning pointer from being freed or reassigned as
 long as the borrowed pointer is in scope.

 Performance, or, should I say, meddling of the compiler and runtime in
 your code, is from worst to best.

 We could argue over which pointer concepts would be granted the honor of
 receiving their own D definition syntax (new, ~,  , &, whatever).


Whoa, this won't work without an explosion in language complexity.

Andrei



 Feb 01 2014



 next sibling parent reply "Frank Bauer" <y z.com>  writes:


On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei Alexandrescu 
wrote:

 Whoa, this won't work without an explosion in language 
 complexity.

 Andrei


Only daydreaming ...



 Feb 01 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/1/14, 8:18 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei Alexandrescu wrote:

 Whoa, this won't work without an explosion in language complexity.

 Andrei


 Only daydreaming ...


No, it's a nightmare.

Andrei



 Feb 01 2014



 next sibling parent reply "Frank Bauer" <y z.com>  writes:


On Sunday, 2 February 2014 at 05:30:02 UTC, Andrei Alexandrescu 
wrote:

 On 2/1/14, 8:18 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei 
 Alexandrescu wrote:

 Whoa, this won't work without an explosion in language 
 complexity.

 Andrei


 Only daydreaming ...


 No, it's a nightmare.

 Andrei


^^ Lets do it while we're young ...



 Feb 01 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/1/14, 9:34 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 05:30:02 UTC, Andrei Alexandrescu wrote:

 On 2/1/14, 8:18 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei Alexandrescu wrote:

 Whoa, this won't work without an explosion in language complexity.

 Andrei


 Only daydreaming ...


 No, it's a nightmare.

 Andrei


 ^^ Lets do it while we're young ...


Sure. While you're young typecheck this:

class A {
    auto stuff() { ...; return this; }
}


Andrei



 Feb 01 2014



  parent reply "Frank Bauer" <y z.com>  writes:


On Sunday, 2 February 2014 at 05:41:53 UTC, Andrei Alexandrescu 
wrote:

 Sure. While you're young typecheck this:

 class A {
    auto stuff() { ...; return this; }
 }


 Andrei


It would be a Borrowed!A, I would say, no matter how an A is 
allocated. A GC!A, an ARC!A and an Owning!A convert equally well 
to a Borrowed!A. As long as the Borrowed!A is in use (i.e. scope) 
after stuff() returns, the A object must not be freed (compiler 
enforced). Would that make sense?



 Feb 01 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/1/14, 10:00 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 05:41:53 UTC, Andrei Alexandrescu wrote:

 Sure. While you're young typecheck this:

 class A {
    auto stuff() { ...; return this; }
 }


 Andrei


 It would be a Borrowed!A, I would say, no matter how an A is allocated.
 A GC!A, an ARC!A and an Owning!A convert equally well to a Borrowed!A.
 As long as the Borrowed!A is in use (i.e. scope) after stuff() returns,
 the A object must not be freed (compiler enforced). Would that make sense?


No.

What I'm driving at is that right now the type is A, as is in a variety 
of other situations. It would be a whole different language that would 
be wholly incompatible with D.


Andrei



 Feb 01 2014



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Sunday, 2 February 2014 at 07:35:21 UTC, Andrei Alexandrescu 
wrote:

 No.

 What I'm driving at is that right now the type is A, as is in a 
 variety of other situations. It would be a whole different 
 language that would be wholly incompatible with D.


What do you think about my `scope` proposal? It does not 
introduce any new syntax and should be somewhat in line with 
existing reference escape analysis.

By the way, Sonke Ludwig has recently named scope + 
unique/isolated qualifiers as one of more promising D 
improvements for vibe.d : "having this in the core language would 
open up a lot of potential for performance and safety 
improvements and I feel that it needs more advertisement to 
really reach the decision makers - and of course the general 
community"



 Feb 02 2014



 next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Sunday, 2 February 2014 at 10:53:36 UTC, Dicebot wrote:

 What do you think about my `scope` proposal? It does not 
 introduce any new syntax and should be somewhat in line with 
 existing reference escape analysis.


What's the link to your proposal? I think I've read it before and 
liked it (scope would rock) but not sure it is the same thing on 
my mind.



 Feb 02 2014



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Sunday, 2 February 2014 at 14:12:39 UTC, Adam D. Ruppe wrote:

 On Sunday, 2 February 2014 at 10:53:36 UTC, Dicebot wrote:

 What do you think about my `scope` proposal? It does not 
 introduce any new syntax and should be somewhat in line with 
 existing reference escape analysis.


 What's the link to your proposal? I think I've read it before 
 and liked it (scope would rock) but not sure it is the same 
 thing on my mind.


There is no DIP for it, if you mean that. rvalue reference DIP 
has been implying it but mostly it is just short generalization 
for info already spread around the docs: 
http://forum.dlang.org/post/uopnrbrjwkxgejbcojxa forum.dlang.org

It boils down to storage class that binds lifetime of entity to 
scope + qualifier that guarantees that no references/pointers to 
that data can be possibly stored at the end of scope in question. 
May need attribute inference to become usable with Phobos though.

I have originally wanted to write it down in DIP form but at that 
time Andrei has said that implementing scope is out of question. 
Wondering if his opinion has changed :)



 Feb 02 2014



  parent reply "Namespace" <rswhite4 googlemail.com>  writes:


 I have originally wanted to write it down in DIP form but at 
 that time Andrei has said that implementing scope is out of 
 question. Wondering if his opinion has changed :)


What? When did he say that? And why?



 Feb 02 2014



  parent  "Dicebot" <public dicebot.lv>  writes:


On Sunday, 2 February 2014 at 14:37:50 UTC, Namespace wrote:

 I have originally wanted to write it down in DIP form but at 
 that time Andrei has said that implementing scope is out of 
 question. Wondering if his opinion has changed :)


 What? When did he say that? And why?


Well, wording was a bit different but from DIP36 discussion 
thread 
(http://forum.dlang.org/post/ylebrhjnrrcajnvtthtt forum.dlang.org):


 It defines a new language feature instead of improving the 
 existing ones. At this point in the development of the 
 language, our preference should be putting the existing 
 features in good order.


+


 You consider "scope" a new language feature?


 Yes.


I can be wrong but I have interpreted this as "no plans to 
actually implement scope in any forseeable future"



 Feb 02 2014



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/2/14, 2:53 AM, Dicebot wrote:

 On Sunday, 2 February 2014 at 07:35:21 UTC, Andrei Alexandrescu wrote:

 No.

 What I'm driving at is that right now the type is A, as is in a
 variety of other situations. It would be a whole different language
 that would be wholly incompatible with D.


 What do you think about my `scope` proposal?


I think that's more likely to be made to work.

Andrei



 Feb 02 2014



prev sibling next sibling parent  "Frank Bauer" <y z.com>  writes:


On Sunday, 2 February 2014 at 07:35:21 UTC, Andrei Alexandrescu 
wrote:

 No.

 What I'm driving at is that right now the type is A, as is in a 
 variety of other situations. It would be a whole different 
 language that would be wholly incompatible with D.


 Andrei


Thx for giving it a thought though.



 Feb 02 2014



prev sibling next sibling parent  "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Sunday, 2 February 2014 at 07:35:21 UTC, Andrei Alexandrescu 
wrote:

 What I'm driving at is that right now the type is A, as is in a 
 variety of other situations. It would be a whole different 
 language that would be wholly incompatible with D.


A == Borrowed!A in my view. Consider this:

class A { A foo() { return this; } }

A a;

void crashy() {
    ubyte[__traits(classInstanceSize, A)] Abuffer;
    auto localA = emplace!A(Abuffer);

    a = localA.foo();
}


That's obviously wrong, when crashy returns, a is dead... but it 
has no way of knowing that.

Let's consider we wrote a nice little Owned!T. Internally, it 
keeps that buffer, disables postblit, frees in the destructor and 
might have a release method to transfer ownership. All stuff we 
can do in the library today.


Owned!A a;
void crashy() {
    Owned!A localA = Owned!A();
    a = localA; // prohibited, this(this) is annotated with 
 disable
    a = localA.foo(); // should ALSO be prohibited, cannot convert 
A to Owned!A
}


The class object itself doesn't know how it is stored. If it 
returns this, it MUST assume it is borrowed unless it explicitly 
allocates a new thing.

For all it knows, it was emplaced into a static array or 
malloced. A == Borrowed!A. typeof(this) == A.




So, what about the global a up there? I think there's two 
solutions:

1) ideally, Owned!A has something like this:

scope A borrow() { return payload; }
alias borrow this;

A a = localA; // legal, A is a local variable in the same or 
inner scope
globalA = localA; // illegal, since borrow returns scope, it 
isn't allowed to escape


2) a worse but perhaps doable option would be

A globalA; // illegal, non-local variable must not be borrowed

struct foo {
   A a; // now foo is itself considered a borrowed reference type 
so the rules apply to it
}

GC!A globalA; // ok cool



What's interesting about this is we could, in theory (and I've 
done a limited proof of concept), implement it in the library 
using a static assert in RTInfo!

anyway i g2g might talk more later



 Feb 02 2014



prev sibling  parent  "Frank Bauer" <y z.com>  writes:


+1 on making Phobos GC-free.

On Sunday, 2 February 2014 at 07:35:21 UTC, Andrei Alexandrescu 
wrote:

 On 2/1/14, 10:00 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 05:41:53 UTC, Andrei 
 Alexandrescu wrote:

 Sure. While you're young typecheck this:

 class A {
   auto stuff() { ...; return this; }
 }


 Andrei


 It would be a Borrowed!A, I would say, no matter how an A is 
 allocated.
 A GC!A, an ARC!A and an Owning!A convert equally well to a 
 Borrowed!A.
 As long as the Borrowed!A is in use (i.e. scope) after stuff() 
 returns,
 the A object must not be freed (compiler enforced). Would that 
 make sense?


 No.

 What I'm driving at is that right now the type is A, as is in a 
 variety of other situations. It would be a whole different 
 language that would be wholly incompatible with D.


 Andrei


Why is that? I propose the new memory regime to be 100% 
compatible with all existing D code. It is way more powerful than 
anything scope can do. And it seems a lot less work to me than 
going through GC optimization and custom allocator tracing hell.

Let's see how far we can get if we require none of the existing D 
code to break.

Example: this is current D code that runs unchanged and with the 
same semantics under the new memory regime. GC(T) and Borrowed(T) 
remain under the hood:

class A {
     A foo() { return this; }   // returns Borrowed!A
}

void main () {
     A m1 = new A;   // m1 is a GC!A
     A m2 = m1;      // m2 is a Borrowed!A
     A m3;           // Borrowed!A
     m3 = m1;        // fine: assignment from GC!A to Borrowed!A
     m3 = bar(m1);   // fine: m1 converted from GC!A to 
Borrowed!A, assignment from Borrowed!A to Borrowed!A
     m3 = bar(m2);   // fine: m2 already Borrowed!A, assignment 
from Borrowed!A to Borrowed!A
     m1 = new A;     // fine: can reasssign GC!A although there 
are outstanding references
     m1 = m2;        // fine: can reassign GC!A to a Borrowed!A
}

A bar(A a) {        // takes and returns Borrowed!A
     A b1 = a;
     return b1;
}

Now here it comes: if, on the other hand, I replace the first 
'new' with, say e.g., 'newOwning' (or '~', whatever), I get 
GC-free code:

A m1 = newOwning A; // m1 is an Owning!A, allocating on the heap 
and freeing when it goes out of scope

Then these lines above would be in error:

     m1 = new A;     // error: can't reassign to m1 because m2 and 
m3 still referencing m1 (same for newOwning)
     m1 = m2;        // error: can't assign Borrowed!A to Owning!A

ARC(T) could be implemented similarly.

Nothing changes one bit for those who want to use the GC as 
before. ARC(T) and Owning(T) would be *ADDED* on top of the 
existing language spec.

Emplacement new, as Adam mentioned, is not a problem: you always 
get to the emplaced object with a Borrowed!T. The underlying 
memory is managed separately and automatically as a GC!M, ARC!M 
or Owning!M (M could be an array of bytes, e.g.).



 Feb 02 2014



prev sibling  parent reply "JR" <zorael gmail.com>  writes:


On Sunday, 2 February 2014 at 05:30:02 UTC, Andrei Alexandrescu 
wrote:

 On 2/1/14, 8:18 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei 
 Alexandrescu wrote:

 Whoa, this won't work without an explosion in language 
 complexity.

 Andrei


 Only daydreaming ...


 No, it's a nightmare.

 Andrei


So, going forward, what would you say is the preferred direction 
to strive toward?

I *seem* to remember reading here that you and Walter were 
increasingly growing to favor ARC, but I can't find the post. 
(Memory bitrot on my part is more than likely.)



 Feb 02 2014



 next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Sunday, 2 February 2014 at 11:23:56 UTC, JR wrote:

 I *seem* to remember reading here that you and Walter were 
 increasingly growing to favor ARC, but I can't find the post.


Personally, I think ARC is a waste of time and none of the 
arguments I've seen for it are convincing. Improving the Gc 
implementation would be a much bigger win.



 Feb 02 2014



  parent reply Paulo Pinto <pjmlp progtools.org>  writes:


Am 02.02.2014 15:11, schrieb Adam D. Ruppe:

 On Sunday, 2 February 2014 at 11:23:56 UTC, JR wrote:

 I *seem* to remember reading here that you and Walter were
 increasingly growing to favor ARC, but I can't find the post.


 Personally, I think ARC is a waste of time and none of the arguments
 I've seen for it are convincing. Improving the Gc implementation would
 be a much bigger win.


It might be that ARC will be a solution given the recent work in 


I was even surprised to discover that Cedar actually used RC with a GC 
just for collecting cycles.

What I don't like is using Objective-C as a poster child for ARC, 
because the only reason it exists, is that Apple engineers weren't able 
to write a GC for Objective-C that didn't core dump left and right.

Apple of course sold the history in a different way.

--
Paulo



 Feb 02 2014



  parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Sun, 02 Feb 2014 06:48:28 -0800, Paulo Pinto <pjmlp progtools.org>  
wrote:


 Am 02.02.2014 15:11, schrieb Adam D. Ruppe:

 On Sunday, 2 February 2014 at 11:23:56 UTC, JR wrote:

 I *seem* to remember reading here that you and Walter were
 increasingly growing to favor ARC, but I can't find the post.


 Personally, I think ARC is a waste of time and none of the arguments
 I've seen for it are convincing. Improving the Gc implementation would
 be a much bigger win.


 It might be that ARC will be a solution given the recent work in  


 I was even surprised to discover that Cedar actually used RC with a GC  
 just for collecting cycles.

 What I don't like is using Objective-C as a poster child for ARC,  
 because the only reason it exists, is that Apple engineers weren't able  
 to write a GC for Objective-C that didn't core dump left and right.

 Apple of course sold the history in a different way.

 --
 Paulo


I just want to point out that in the comments of Joe Duffy's latest post  


very fact of it's nature require the language semantics to change, it's  

to a GC guru. For a ridiculously easy example, what do you do when one  
object implements .Dispose() and another one does not?

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 02 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/2/14, 3:23 AM, JR wrote:

 On Sunday, 2 February 2014 at 05:30:02 UTC, Andrei Alexandrescu wrote:

 On 2/1/14, 8:18 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei Alexandrescu wrote:

 Whoa, this won't work without an explosion in language complexity.

 Andrei


 Only daydreaming ...


 No, it's a nightmare.

 Andrei


 So, going forward, what would you say is the preferred direction to
 strive toward?

 I *seem* to remember reading here that you and Walter were increasingly
 growing to favor ARC, but I can't find the post. (Memory bitrot on my
 part is more than likely.)


I think of the following foci for the first half of 2014:

1. Add  nullable and provide a -nullable compiler flag to verify it. The 
attribute is inferred locally and for white-box functions (lambdas, 
templates), and required as annotation otherwise. References not 
annotated with  nullable are statically enforced to never be null.

2. Work on Phobos to see what can be done about avoiding unnecessary 
allocation. Most likely we'll need to also add a  nogc flag.

3. Work on adding tracing capabilities to allocators and see how to 
integrate them with the language and Phobos.

4. Work on the core language and druntime to see how to seamlessly 
accommodate alternate GC mechanisms such as reference counting.


Andrei



 Feb 02 2014



 next sibling parent  "Frustrated" <c1514843 drdrb.com>  writes:


On Sunday, 2 February 2014 at 16:55:35 UTC, Andrei Alexandrescu
wrote:

 On 2/2/14, 3:23 AM, JR wrote:

 On Sunday, 2 February 2014 at 05:30:02 UTC, Andrei 
 Alexandrescu wrote:

 On 2/1/14, 8:18 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei 
 Alexandrescu wrote:

 Whoa, this won't work without an explosion in language 
 complexity.

 Andrei


 Only daydreaming ...


 No, it's a nightmare.

 Andrei


 So, going forward, what would you say is the preferred 
 direction to
 strive toward?

 I *seem* to remember reading here that you and Walter were 
 increasingly
 growing to favor ARC, but I can't find the post. (Memory 
 bitrot on my
 part is more than likely.)


 I think of the following foci for the first half of 2014:

 1. Add  nullable and provide a -nullable compiler flag to 
 verify it. The attribute is inferred locally and for white-box 
 functions (lambdas, templates), and required as annotation 
 otherwise. References not annotated with  nullable are 
 statically enforced to never be null.

 2. Work on Phobos to see what can be done about avoiding 
 unnecessary allocation. Most likely we'll need to also add a 
  nogc flag.

 3. Work on adding tracing capabilities to allocators and see 
 how to integrate them with the language and Phobos.

 4. Work on the core language and druntime to see how to 
 seamlessly accommodate alternate GC mechanisms such as 
 reference counting.


 Andrei


Sounds like a step in the right direction. By having the nogc,
phobos could be progressively converted and one can verify when
modules, functions, and aggregates are gc dependent.



 Feb 02 2014



prev sibling next sibling parent reply "Namespace" <rswhite4 googlemail.com>  writes:


On Sunday, 2 February 2014 at 16:55:35 UTC, Andrei Alexandrescu 
wrote:

 On 2/2/14, 3:23 AM, JR wrote:

 On Sunday, 2 February 2014 at 05:30:02 UTC, Andrei 
 Alexandrescu wrote:

 On 2/1/14, 8:18 PM, Frank Bauer wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei 
 Alexandrescu wrote:

 Whoa, this won't work without an explosion in language 
 complexity.

 Andrei


 Only daydreaming ...


 No, it's a nightmare.

 Andrei


 So, going forward, what would you say is the preferred 
 direction to
 strive toward?

 I *seem* to remember reading here that you and Walter were 
 increasingly
 growing to favor ARC, but I can't find the post. (Memory 
 bitrot on my
 part is more than likely.)


 I think of the following foci for the first half of 2014:

 1. Add  nullable and provide a -nullable compiler flag to 
 verify it. The attribute is inferred locally and for white-box 
 functions (lambdas, templates), and required as annotation 
 otherwise. References not annotated with  nullable are 
 statically enforced to never be null.

 2. Work on Phobos to see what can be done about avoiding 
 unnecessary allocation. Most likely we'll need to also add a 
  nogc flag.

 3. Work on adding tracing capabilities to allocators and see 
 how to integrate them with the language and Phobos.

 4. Work on the core language and druntime to see how to 
 seamlessly accommodate alternate GC mechanisms such as 
 reference counting.


 Andrei






 Feb 02 2014



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/2/2014 9:30 AM, Namespace wrote:




Let me rephrase that as "why use a storage class rather than a type
constructor?"

An excellent question.

One of the big problems with a type constructor is we've already got a number
of 
them - const, immutable, shared, inout. These and their conversions have taken 
quite a while for people to get comfortable with, and there are still some 
issues in the language arising from their interactions.

Adding another type constructor will double the number of cases. I worry that D 
would collapse under the strain. This applies as well to type constructors for 
"borrowed", etc.

On the other hand, D took the risky leap into the unknown by making 'ref' a 
storage class rather than a type constructor (in C++ it's a type constructor). 
This has turned out to be a satisfying win. I suspect it will be for  nullable, 
too, though we have to work through the cases to find out for sure.



 Feb 02 2014



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/2/14, 12:05 PM, Walter Bright wrote:

 On 2/2/2014 9:30 AM, Namespace wrote:




 Let me rephrase that as "why use a storage class rather than a type
 constructor?"

 An excellent question.

 One of the big problems with a type constructor is we've already got a
 number of them - const, immutable, shared, inout.


This post is confused, sorry. const, immutable, shared, inout are 
qualifiers (in addition to being type constructors). General type 
constructors don't have a particular combinatorial problem.

Andrei



 Feb 02 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 2/2/2014 1:18 PM, Andrei Alexandrescu wrote:

 On 2/2/14, 12:05 PM, Walter Bright wrote:

 One of the big problems with a type constructor is we've already got a
 number of them - const, immutable, shared, inout.


 This post is confused, sorry. const, immutable, shared, inout are qualifiers
(in
 addition to being type constructors). General type constructors don't have a
 particular combinatorial problem.


Yes, I used the wrong term.



 Feb 02 2014



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 02/02/2014 09:05 PM, Walter Bright wrote:

 On 2/2/2014 9:30 AM, Namespace wrote:




 Let me rephrase that as "why use a storage class rather than a type
 constructor?"

 An excellent question.

 One of the big problems with a type constructor is we've already got a
 number of them - const, immutable, shared, inout. These and their
 conversions have taken quite a while for people to get comfortable with,
 and there are still some issues in the language arising from their
 interactions.
 ...


Nullable interacts with those type constructors basically in the same 
way as the fixed-size array type constructor [1]. Regarding implicit 
conversion, I think one good form to state the implicit conversion rules 
in, for later procedural implementation, is the following:

  A converts to B
──────────────────
  A converts to B?

   A converts to B
───────────────────
  A? converts to B?

  e converts to B?
────────────────── (flow analysis proves e
non-null)
  e converts to B

Where would be potentially bad interactions?


 Adding another type constructor will double the number of cases. I worry
 that D would collapse under the strain. This applies as well to type
 constructors for "borrowed", etc.
 ...


This sounds as if there was a structural problem in DMD. (Adding the 
relevant implicit conversion rules would be easy given the current state 
of my own front end implementation effort.) How are type qualifiers 
represented in the AST? How is the implicit conversion checker structured?


 On the other hand, D took the risky leap into the unknown by making
 'ref' a storage class rather than a type constructor (in C++ it's a type
 constructor). This has turned out to be a satisfying win. I suspect it
 will be for  nullable, too, though we have to work through the cases to
 find out for sure.


These cases have almost nothing in common. (e.g, 'ref' does not 
influence implicit conversion behaviour.)



 Feb 02 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/2/2014 3:40 PM, Timon Gehr wrote:

 Nullable interacts with those type constructors basically in the same way as
the
 fixed-size array type constructor [1]. Regarding implicit conversion, I think
 one good form to state the implicit conversion rules in, for later procedural
 implementation, is the following:

   A converts to B
 ──────────────────
   A converts to B?

    A converts to B
 ───────────────────
   A? converts to B?

   e converts to B?
 ────────────────── (flow analysis proves e
non-null)
   e converts to B

 Where would be potentially bad interactions?


For starters, grep through the source code for all the implicit conversions. 
Then think about how it fits in with match levels, function overloading, 
template overloading, partial ordering, ?:, covariance, contravariance, name 
mangling, TypeInfo, default initialization, will general functions have to be 
written twice (once for T, again for T?), auto function returns, and that's
just 
off the top of my head.

It's not just writing a truth table and throwing it over the wall.



 Feb 02 2014



  parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-02-03 04:06:00 +0000, Walter Bright <newshound2 digitalmars.com> said:


 For starters, grep through the source code for all the implicit 
 conversions. Then think about how it fits in with match levels, 
 function overloading, template overloading, partial ordering, ?:, 
 covariance, contravariance, name mangling, TypeInfo, default 
 initialization, will general functions have to be written twice (once 
 for T, again for T?), auto function returns, and that's just off the 
 top of my head.
 
 It's not just writing a truth table and throwing it over the wall.


But wouldn't that prevent all function arguments from being null at 
all? As well as function return values? That'll make it almost 
impossible to port existing code to this new D model, or interface with 
existing C functions.

It'd still be possible, because you could always use a struct to pass a 
 nullable pointer around, but then that struct will not work with 
partial ordering, ?:, covariance, and contravariance, making things 
quite annoying (just like Rebindable is for tail const/immutable 
classes).

As for writing functions twice, I don't see it. T is implicitly 
convertible to T?, so a single function with a T? parameter will work 
with both T and T? arguments.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 Feb 03 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/3/2014 3:45 AM, Michel Fortin wrote:

 As for writing functions twice, I don't see it. T is implicitly convertible to
 T?, so a single function with a T? parameter will work with both T and T?
 arguments.


It's the same issue as with const (which we dealt with by adding the 'inout' 
qualifier). It's the affect of the argument type on the return type.



 Feb 03 2014



  parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-02-03 19:53:58 +0000, Walter Bright <newshound2 digitalmars.com> said:


 On 2/3/2014 3:45 AM, Michel Fortin wrote:

 As for writing functions twice, I don't see it. T is implicitly convertible to
 T?, so a single function with a T? parameter will work with both T and T?
 arguments.


 
 It's the same issue as with const (which we dealt with by adding the 
 'inout' qualifier). It's the affect of the argument type on the return 
 type.


Const can represent either mutable or immutable data, so you have inout 
standing in as "same output as input". This is useful, necessary even, 
if your function returns some part of its input, like the value of a 
field.

In theory it could make sense to have something similar for nullable, 
where you have a qualifier standing for as "same nullability as input". 
That said, you can't return a part of the input if the value is null, 
and that part's nullability won't depend on its parent (nullability is 
not transitive), so it's not as useful. The only useful thing that 
comes to mind is a function where you return the passed argument 
directly, such as this one:

	Object? foo(Object? o)
	{
		if (o)
			writeln(o);

		return o;
	}

But, you could implement this as a template function if you absolutely 
need to transfer nullability, and the template will actually make 
things more efficient because the template instantiated with a 
non-nullable function argument will optimize away the unnecessary 
branch.

And to get to the root of this, I think it'd be much more useful to 
have a "same type as input" stand-in, because notice how in the above 
function if you pass a DerivedObject, you'll get an Object returned, 
losing the derived part? Exact same problem, and we've been living with 
it for decades.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 Feb 03 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/3/2014 7:24 PM, Michel Fortin wrote:

 But, you could implement this as a template function if you absolutely need to
 transfer nullability, and the template will actually make things more efficient
 because the template instantiated with a non-nullable function argument will
 optimize away the unnecessary branch.


Yes, you could use a template, but the idea was to avoid bloat in the
executable 
by having multiple sets of functions that differ only by their mangled name. 
Furthermore, templates cannot be used to create virtual functions.



 And to get to the root of this, I think it'd be much more useful to have a
"same
 type as input" stand-in, because notice how in the above function if you pass a
 DerivedObject, you'll get an Object returned, losing the derived part? Exact
 same problem, and we've been living with it for decades.


The const issue is a recurring theme in a lot of C++ code, the derived type one 
just doesn't seem to come up.

Also, what is being looked at to transfer is the qualifier, not the type, as
the 
type may actually change (such as returning a pointer to a field of the
argument).



 Feb 03 2014



  parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-02-04 03:45:53 +0000, Walter Bright <newshound2 digitalmars.com> said:


 On 2/3/2014 7:24 PM, Michel Fortin wrote:

 But, you could implement this as a template function if you absolutely need to
 transfer nullability, and the template will actually make things more efficient
 because the template instantiated with a non-nullable function argument will
 optimize away the unnecessary branch.


 
 Yes, you could use a template, but the idea was to avoid bloat in the 
 executable by having multiple sets of functions that differ only by 
 their mangled name. Furthermore, templates cannot be used to create 
 virtual functions.


Unlike with the mutable/const/immutable case, those templates actually 
won't only differ only by their mangled names, because any check for 
null will be quashed by the optimizer in the not-null variant.

You can write two function if you really need them to be virtual, but 
as explained below I doubt you'll need to do that anyway.


 And to get to the root of this, I think it'd be much more useful to 
 have a "same
 type as input" stand-in, because notice how in the above function if you pass a
 DerivedObject, you'll get an Object returned, losing the derived part? Exact
 same problem, and we've been living with it for decades.


 
 The const issue is a recurring theme in a lot of C++ code, the derived 
 type one just doesn't seem to come up.
 
 Also, what is being looked at to transfer is the qualifier, not the 
 type, as the type may actually change (such as returning a pointer to a 
 field of the argument).


Yes, but const in C++ applies to all the fields of an aggregate. Making 
a struct variable const means all its fields are seen as const. To 
write accessors that works with both const and non-const you need to 
duplicate each of them.

That's not the case for nullable. Fields do not become nullable because 
the pointer leading to the struct is nullable. You only have to write 
one accessor that takes a non-nullable pointer or ref, and let the 
parent context make sure it is not nullable before calling it. Even 
today, most accessors don't check "this" for null anyway, so we're just 
formalizing that contract and enforcing it statically.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 Feb 03 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/3/2014 8:17 PM, Michel Fortin wrote:

 That's not the case for nullable.


Yes, it is. If the function accepts a pointer to S and returns a pointer to
S.f, 
then a null pointer in means a null pointer out, and a completely different
type.



 Feb 03 2014



  parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-02-04 04:26:22 +0000, Walter Bright <newshound2 digitalmars.com> said:


 On 2/3/2014 8:17 PM, Michel Fortin wrote:

 That's not the case for nullable.


 
 Yes, it is. If the function accepts a pointer to S and returns a 
 pointer to S.f, then a null pointer in means a null pointer out, and a 
 completely different type.


For that to happen you'd have to have a function like this one:

	class MyObject { OtherObject value; }

	OtherObject? foo(MyObject? o)
	{
		if (o)
			return o.other;
		else
			return null;
	}

The corresponding not-nullable version would be like that (notice the 
efficiency gain):

	OtherObject foo(MyObject o)
	{
		return o.other; // no need to check for null here
	}

Honestly, I don't think writing accessors that check for null is that 
common, except to detect null pointers passed in error because of the 
lack of static checking. In general for an accessor it's better to let 
the parent function check for null and only implement the later version 
of the accessor. The parent function can do one null check before 
calling all the accessors it wants, and accessors don't have to check 
for null. You lose that efficiency if each accessors has to check for 
null.

For the rare cases where you actually want both versions to work, you 
can write them twice or use a template (except in a virtual context), 
and in both cases you keep the efficiency of not checking for null when 
the argument is not nullable.

In any case, I have yet to understand why  nullable as a storage class 
would be any better. How do you solve that problem with a storage class?

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 Feb 04 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/4/2014 4:23 AM, Michel Fortin wrote:

 For the rare cases where you actually want both versions to work,


I think you're making a vast assumption that the case is rare.

When I write utility functions, I want them to work on as wide a variety of 
inputs as possible. Otherwise, they are not very useful.


 you can write them twice  or use a template (except in a virtual context), and
in both cases
 you keep the efficiency of not checking for null when the argument is not
nullable.


That's just what I wish to avoid. Consider adding more pointer types - the 
combinatorics quickly explode. Heck, just have two pointer parameters, and you 
already have 4 cases.

I wonder how Rust deals with this.



 In any case, I have yet to understand why  nullable as a storage class would be
 any better. How do you solve that problem with a storage class?


Good question. I don't have an answer offhand.



 Feb 04 2014



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 4 February 2014 at 22:30:39 UTC, Walter Bright wrote:

 you can write them twice  or use a template (except in a 
 virtual context), and in both cases
 you keep the efficiency of not checking for null when the 
 argument is not nullable.


 That's just what I wish to avoid. Consider adding more pointer 
 types - the combinatorics quickly explode. Heck, just have two 
 pointer parameters, and you already have 4 cases.

 I wonder how Rust deals with this.


I think it is pretty much same thing as with using specialised 
types (i.e. SQLEscapedString) - your API explodes exponentially 
if you try to write code with methods/functions that support all 
of them but in practice you shouldn't. Same is here. Some 
functions should always deal with with nullable types and most - 
only with non-nullable. Tricky part is writing your code in such 
way that separation is clear but good language guidlines and 
examples can solve it.

Also non-nullable types should be implicitly cast to nullable 
parameters so you don't always need to support all cases 
distinctively.



 Feb 04 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/4/2014 2:39 PM, Dicebot wrote:

 Also non-nullable types should be implicitly cast to nullable parameters so you
 don't always need to support all cases distinctively.


The difficulty comes when needing to transfer the nullability of the argument
to 
the nullability of the return type. Implicit conversions don't help with that.

See inout.



 Feb 04 2014



 next sibling parent  "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Wednesday, 5 February 2014 at 00:47:27 UTC, Walter Bright 
wrote:

 The difficulty comes when needing to transfer the nullability 
 of the argument to the nullability of the return type. Implicit 
 conversions don't help with that.

 See inout.


You could always do it with a template; T identity(T t) { return 
t;} will literally work. (I'm pretty sure it would also work with 
const and immutable; the reason inout exists isn't that it is 
impossible, it is just to counter template bloat)

Is this common enough to warrant an addition, or would templates 
be good enough when it is needed? inout has a very common real 
world use case: slicing.

inout(char)* strchr(inout(char)* haystack, in char* needle);

What's the real world equivalent to strchr with regard to 
nullability?


* typeof(this) inside a class is never nullable. You can't call a 
method with a null this. So a method that returns this can always 
just return the regular (not null) type.

* this[x..y] is never nullable. Slicing a null reference is 
illegal, so again, no need to offer the nullable overload.

* Returning a field of a nullable type is illegal, dereferencing 
null to get to the fields is illegal. Whereas a struct might 
return inout(member_variable), it never makes sense to do 
inout_nullability(member_variable).


I can't think of a single real world case where you'd need 
this... and if I'm wrong, we can always use a template and/or 
revisit the problem later.



 Feb 04 2014



prev sibling  parent reply "Dicebot" <public dicebot.lv>  writes:


On Wednesday, 5 February 2014 at 00:47:27 UTC, Walter Bright 
wrote:

 On 2/4/2014 2:39 PM, Dicebot wrote:

 Also non-nullable types should be implicitly cast to nullable 
 parameters so you
 don't always need to support all cases distinctively.


 The difficulty comes when needing to transfer the nullability 
 of the argument to the nullability of the return type. Implicit 
 conversions don't help with that.

 See inout.


Why can't inout be used for that as well? Same for pure & co.

In my opinion "inout" should be simply generic placeholder saying 
"allow only code that works for all possible qualifiers and 
preserve those in return type".

But nature of nullable is such that you rarely want to transfer 
it. Is can be useful as return type sometimes but when you pass 
it around and process it you almost always want it to be 
non-nullable. There is no practical benefit in trying to provide 
all-allowing API's.



 Feb 05 2014



  parent reply "Meta" <jared771 gmail.com>  writes:


On Wednesday, 5 February 2014 at 15:56:44 UTC, Dicebot wrote:

 On Wednesday, 5 February 2014 at 00:47:27 UTC, Walter Bright 
 wrote:

 On 2/4/2014 2:39 PM, Dicebot wrote:

 Also non-nullable types should be implicitly cast to nullable 
 parameters so you
 don't always need to support all cases distinctively.


 The difficulty comes when needing to transfer the nullability 
 of the argument to the nullability of the return type. 
 Implicit conversions don't help with that.

 See inout.


 Why can't inout be used for that as well? Same for pure & co.

 In my opinion "inout" should be simply generic placeholder 
 saying "allow only code that works for all possible qualifiers 
 and preserve those in return type".

 But nature of nullable is such that you rarely want to transfer 
 it. Is can be useful as return type sometimes but when you pass 
 it around and process it you almost always want it to be 
 non-nullable. There is no practical benefit in trying to 
 provide all-allowing API's.


If null becomes an invalid value for object references and 
pointers, then we can use it to unambiguously mean "item does not 
exist", and it becomes Just Another Value, so it is useful to 
transfer in various circumstances.



 Feb 05 2014



  parent  "Dicebot" <public dicebot.lv>  writes:


On Wednesday, 5 February 2014 at 17:09:31 UTC, Meta wrote:

 If null becomes an invalid value for object references and 
 pointers, then we can use it to unambiguously mean "item does 
 not exist", and it becomes Just Another Value, so it is useful 
 to transfer in various circumstances.


Yes, in that cases you transfer it as Nullable!T which is 
supposed to be single generic way to pass both value and 
reference types that can possibly be undefined.

Judging by typical code I see though, such "does not exist" case 
is most commonly processed as

if (!param)
     return; // return null;

So if `param` becomes non-nullable you simply don't call 
processing function for null cases.



 Feb 05 2014



prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Tuesday, 4 February 2014 at 22:30:39 UTC, Walter Bright wrote:

 I wonder how Rust deals with this.


The only time ownership matters is if you are going to store the 
pointer. It is like the difference between a container and a 
range.

An algorithm doesn't need to know about the specifics of a 
container. Let's use average for example. We might write it in D:

int average(InputRange)(InputRange r) {
     int count = 0;
     int sum;
     while(!r.empty) {
          count++;
          sum += r.front;
          r.popFront();
     }
     return sum / count;
}

Now, this being a template, D will generate new code for a 
variety of types... but even if we replaced InputRange with a 
specific thing, let's call it int[], it is still usable by a 
variety of containers:

int average(int[] r) { /* same impl */ }


D has two containers built in that provide this range:

int[50] staticArray;
int[] dynamicArray = new int[](50);

average(staticArray[]); // works
average(dynamicArray); // works

Pointers also offer this:

int* pointer = cast(int*) malloc(50 * int.sizeof);
average(pointer[0 .. 50]);



Moreover, user-defined types can also provide this range:

struct Numbers {
     int[] opSlice() { return [1,2,3]; }
}

Numbers numbers;
average(numbers[]); // works

In theory, we could provide either an inputRangeObject or a slice 
for linked lists, lazy generators, anything. One function, any 
kind of input.


Of course, we could slice memory from any allocator. Heck, we saw 
three different allocations right here (with three different 
types! stack, gc, and malloc) all using the same function, 
without templating.




I'm sure none of this is new to you... and this is basically how 
the rust thing works too. Our usage of int[] (or the input range) 
are borrowed pointers. Algorithms are written in their terms.

The ownership type only matters when you store it. And turns out, 
this matters in D as well:

struct ManualArray(T) {
     size_t length;
     T* data;

     this(size_t len) { data = malloc(T.sizeof * len); length = 
len; }
     ~this() { free(data); }
     T[] opSlice() { return data[0 .. length]; }
      disable this(this); // copying this is wrong, don't allow it!
}

void main() {
     auto array = ManualArray!int(50);
     average(array[]); // works, reusing our pointer
}


But, borrowed comes into play if we store it:

int[] globalArray;
void foo(int[] array) {
     globalArray = array;
}

void bar() {
     auto array = ManualArray!int(50);
     foo(array[]); // uh oh
}

void main() {
    bar();
    globalArray[0] = 10; // crash likely, memory safety violated
}



Again, I'm sure none of this is new to you, but it illustrates 
owned vs borrowed: ManualArray is owned. Storing it is safe - it 
ensures its internal pointer is valid throughout its entire life 
time.

But ManualArray.opSlice returns a borrowed reference. Great for 
algorithms or any processing that doesn't escape the reference. 
Anything that would be written in terms of an input range is 
probably correct with this.

However, we stored the borrowed reference, which is a no-no. 
array went out of scope, freeing the memory, leaving the escaped 
borrowed reference in an invalid state.


Let's say we did want to store it. There's a few options: we 
could make our own copy or store the pre-made copy.

GC!(int[]) globalArray;
void foo(GC!(int[]) array) { globalArray = array; }


That's sane, the GC owns it and we specified that so storing it 
is cool.

We could also take a RefCounted!(int[]), if that's how we wanted 
to store it.


But let's say we wanted to store it with a different method. 
There's only two sane options:


void foo(int[] array) { globalArray = array.dup; }

Take a borrowed reference and make a copy of it. The function foo 
is in charge of allocating (here, we made a GC managed copy).


OR, don't implement that and force the user to decide:


void foo(GC!(int[]) array) {...}


user:

foo(ownedArray[]); // error, cannot implicitly convert int[] to 
GC!(int[])
int[50] stackArray;
foo(stackArray[]); // error, cannot implicitly convert int[] to 
GC!int[]


Now, the user makes the decision. It is going to be stored, the 
function signature says that up front by asking for a 
non-borrowed reference. They won't get a surprise crash when the 
globalArray later accesses stack or freed data. They have to deal 
with the error. They might not call the function, or they might 
do the .dup themselves. Either way, memory safety is preserved 
and inefficiencies are visible.



So, a function that stores a reference would only ever come in 
one or two signatures, regardless of how many:

1) the exact match for the callee's allocation strategy. The 
callee, knowing what the strategy is, can also be sanely 
responsible for freeing it. (A struct dtor, for example, knows 
that its members are malloced and can thus call free)

2) A generic borrowed type, e.g. input range or slice, which it 
then makes a private copy of it internally. Since these are 
arguably hidden allocations you might not even like these. 
Calling .dup (or whatever) at the call sight keeps the 
allocations visible.




So bottom line, you don't duplicate functions for the different 
types. You borrow references for processing (analogous to 
implementing algorithms with ranges) and own references for 
storing... which you need to know about, so only one type makes 
sense.



 Feb 04 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/4/2014 3:03 PM, Adam D. Ruppe wrote:

 So bottom line, you don't duplicate functions for the different types. You
 borrow references for processing (analogous to implementing algorithms with
 ranges) and own references for storing... which you need to know about, so only
 one type makes sense.


What happens with this:

     T identity(T t) { return t; }

?



 Feb 04 2014



  parent  "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Wednesday, 5 February 2014 at 00:49:25 UTC, Walter Bright 
wrote:

 What happens with this:

     T identity(T t) { return t; }


My previous post was with regard to nullability. This one is 
about ownership.

The answer is the same: if you need this, use a template, 
otherwise, you always deal with a specific type (if you are 
allocating a new copy or need to store the reference somewhere 
outside the owner's scope) or a borrowed type:

GC!Object obj;
static assert(typeof(identity(obj) == GC!Object);

and so on, you can test this today, D already has all these 
options.


If you don't have a template, how do you approach this? Well, 
this is really nothing new for one:

class Foo {}

Foo foo = new Foo();
Object obj = foo; // legal

Object identity(Object obj) { return obj; }

foo = identity(foo); // won't compile without an explicit cast



When you work with interfaces, some information is lost. That's a 
pain sometimes, but it is a feature too - the function that works 
on the interface doesn't need to care about anything else.

With class inheritance, we can address this with covariant 
overrides.... but only of it is a member function (method).



So let's just accept that the free function loses data. You can't 
get a container back from a range in the general case.


So, what about the method:

class Foo {
    typeof(this) identity() { return this; }
}


What is typeof(this)? I'll tell you: Foo! Here's why:

*) The internal this pointer is never null, and thus never needs 
to be nullable, so that's out of consideration (see my last 
message)

*) The internal this pointer does *not* own its memory. Calling 
"delete this;" is (I think) undefined behavior in D: the class 
might exist on the stack, or be allocated with malloc, and 
calling freeing the memory inside a method is likely to lead to a 
crash when the invariant is called upon returning anyway!


Since the object does not own its own memory, it must always 
assume this is a borrowed reference. Therefore:

* Escaping `this` is prohibited.

Foo globalFoo;
class Foo {
    // illegal, escapes a reference to a borrowed reference
    void storeThis() { globalFoo = this; }
}

Consider that this is wrong if the class is made with 
std.conv.emplace (or the deprecated scope storage class) on a 
stack buffer.

This is silently wrong today, it can leave a dangling reference. 
(though since emplace is not  safe, we do have that protection 
against such problems... though since we're talking about 
avoiding the GC here, emplace is something we probably want 
anyway)


* Returning this is always a borrowed pointer, unless you 
explicitly make a copy.

class Foo {
    Foo identity() { return this; }
}

GC!Foo foo = new Foo(); // owned by the Gc

Foo foo2 = foo.identity(); // borrowed reference, NOT GC!Foo


Therefore, we cannot escape it to a global that way either. If we 
want that, we have to work with foo directly, bypassing 
foo.identity.

A clone method would create a new owned thing, so it returns 
something more specific:

class Foo {
    GC!Foo clone() { auto foo = new GC; /* copy methods */ return 
foo; }
}


A clone method might also be templated on an allocator, and can 
thus change the return type as needed by the allocator. This 
wouldn't be virtual... but it kinda has to be.

class Foo {
     virtual GC!Foo clone() {...}
}
class Bar : Foo {
     override RC!Foo clone() {...}
}

You wouldn't want that anyway, since if you called it through the 
foo interface, you wouldn't know how to free it (does it need to 
run a struct dtor? The GC? The caller needs to know.)


Gotta pick an allocation method in the base class if you want it 
to work as a virtual function. So... yeah do that or use a 
non-virtual template. You can make it work.

A clone method typically wouldn't even be inout anyway so meh.

* A class is responsible for its member references but not 
itself. When the class' dtor is called, it needs to call member 
struct dtors, but does not call free(this) - leave that to the 
outside allcoator. This is already reality today (the GC calls 
the dtor before releasing the memory).



 Feb 04 2014



prev sibling next sibling parent  "woh" <wojw yahoo.com>  writes:


On Tuesday, 4 February 2014 at 22:30:39 UTC, Walter Bright wrote:

 On 2/4/2014 4:23 AM, Michel Fortin wrote:

 For the rare cases where you actually want both versions to 
 work,


 I think you're making a vast assumption that the case is rare.

 When I write utility functions, I want them to work on as wide 
 a variety of inputs as possible. Otherwise, they are not very 
 useful.


 you can write them twice  or use a template (except in a 
 virtual context), and in both cases
 you keep the efficiency of not checking for null when the 
 argument is not nullable.


 That's just what I wish to avoid. Consider adding more pointer 
 types - the combinatorics quickly explode. Heck, just have two 
 pointer parameters, and you already have 4 cases.

 I wonder how Rust deals with this.



 In any case, I have yet to understand why  nullable as a 
 storage class would be
 any better. How do you solve that problem with a storage class?


 Good question. I don't have an answer offhand.


  rust would use a reference, basically a pointer that can come
from any smart ptr type, and does not outlive its owner



 Feb 04 2014



prev sibling next sibling parent reply "Frank Bauer" <y z.com>  writes:


On Tuesday, 4 February 2014 at 22:30:39 UTC, Walter Bright wrote:

 That's just what I wish to avoid. Consider adding more pointer 
 types - the combinatorics quickly explode. Heck, just have two 
 pointer parameters, and you already have 4 cases.

 I wonder how Rust deals with this.


Rust does not have null pointers.

In cases where you want to mark an object of some type as having 
"no value", or being "null", you use a stdlib type Option<T>, 
which is basically a tagged union plus some static compiler 
guarantees:

enum Option<T> {
     Some(T),
     None
}

An Option<T> either contains a T (tagged 'Some' then) or nothing 
(tagged 'None').

If you have a function that takes an Option<T> you can also pass 
it a (pure, non-nullable) T by specifying Some(myT) as the 
parameter.



 Feb 04 2014



  parent reply "Frank Bauer" <y z.com>  writes:


Regarding different pointer types in Rust as function arguments:

A function that takes a borrowed pointer &T can also be called 
with an owning pointer ~T, an RC pointer Rc<T>, or a GC pointer 
Gc<T>. They all convert neatly to a &T. One function to rule them 
... err .. accomodate all.



 Feb 04 2014



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/4/2014 4:07 PM, Frank Bauer wrote:

 Regarding different pointer types in Rust as function arguments:

 A function that takes a borrowed pointer &T can also be called with an owning
 pointer ~T, an RC pointer Rc<T>, or a GC pointer Gc<T>. They all convert neatly
 to a &T. One function to rule them ... err .. accomodate all.


Again, what happens with:

     T identity(T t) { return t; }

? I.e. the transference of the argument pointer type to the return type?



 Feb 04 2014



 next sibling parent reply "Frank Bauer" <y z.com>  writes:


On Wednesday, 5 February 2014 at 00:50:55 UTC, Walter Bright 
wrote:

 Again, what happens with:

     T identity(T t) { return t; }

 ? I.e. the transference of the argument pointer type to the 
 return type?


As far as I see, in Rust the pointer type is lost on the return 
value as well, if your function takes a reference (borrowed 
pointer). But you can do:

fn identity(x: &T) -> T {
     return *T;
}

fn main() {
     let a = ~T;
     let r = ~identity(a);
}

That is: pass by reference and return a value type. If you then 
pass an owned pointer ~T, you can directly construct the returned 
value in heap allocated memory ~identity(a) and assign the 
resulting owned pointer. The compiler is smart enough to not do 
unnecessary copies.

That way the caller is responsible for the relationship between 
argument pointer type and result pointer type.

With a single function, you can freely mix all combinations of 
argument and return types. E.g. pass an owned pointer and 
construct a Gc<T> from the returned result.



 Feb 04 2014



 next sibling parent  "Frank Bauer" <y z.com>  writes:


Sorry,

      return *x;

of course.



 Feb 04 2014



prev sibling next sibling parent  "Frank Bauer" <y z.com>  writes:


On Wednesday, 5 February 2014 at 01:34:59 UTC, Frank Bauer wrote:

     let a = ~T;
     let r = ~identity(a);


The equivalent in D for a GC pointer could be:
     a = new T;
     r = new identity(a);

or, while we're at it:

     a2 = newOwn T;
     r2 = newOwn identity(a2);
     a3 = newARC T;
     r3 = newARC identity(a3);

or even:

     a4 = new T;
     r4 = newOwn identity(a4);

;)



 Feb 04 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/4/14, 5:34 PM, Frank Bauer wrote:

 On Wednesday, 5 February 2014 at 00:50:55 UTC, Walter Bright wrote:

 Again, what happens with:

     T identity(T t) { return t; }

 ? I.e. the transference of the argument pointer type to the return type?


 As far as I see, in Rust the pointer type is lost on the return value as
 well, if your function takes a reference (borrowed pointer). But you can
 do:

 fn identity(x: &T) -> T {
      return *T;
 }

 fn main() {
      let a = ~T;
      let r = ~identity(a);
 }

 That is: pass by reference and return a value type.


Looks like a major weakness.

Andrei



 Feb 04 2014



  parent  "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Wednesday, 5 February 2014 at 04:22:21 UTC, Andrei 
Alexandrescu wrote:

 Looks like a major weakness.


Even assuming that's true (something I don't agree with), it 
doesn't apply to D anyway since we can always template these 
functions!


I'd like to say again that multiple pointer types is something we 
already have in D:

* unique pointer (I think this is ~ in Rust but rust syntax looks 
like vomit to me) is achieved in D by disabling postblit with a 
struct and freeing in the dtor (RAII). It CAN offer a release 
method to transfer ownership.

* ref counted pointer is achieved with postblit adding ref, dtor 
releasing ref.

* GC pointer could be as simple as struct GC(T) {  trustme T t; 
alias t this; } (where  trustme disables the escape analysis 
checks I'd like to see implemented. Alternatively, if escape 
analysis was built in, we would make t private than then offer a 
getter property that returns scope T.)

* borrowed pointer is the regular references we have today, plus 
escape analysis.


We don't have to speculate how these work. Just write functions 
that use GC!T or RC!T or Unique!T - types we can go ahead and 
write immediately, they are pretty basic structs. Phobos even has 
unique and refcounted right now. We're already halfway there!



 Feb 04 2014



prev sibling  parent  "Matthias Einwag" <matthias.einwag googlemail.com>  writes:


 Again, what happens with:

    T identity(T t) { return t; }

 ? I.e. the transference of the argument pointer type to the 
 return type?


 As far as I see, in Rust the pointer type is lost on the return 
 value as well, if your function takes a reference (borrowed 
 pointer). But you can do:

 fn identity(x: &T) -> T {
     return *T;
 }

 fn main() {
     let a = ~T;
     let r = ~identity(a);
 }

 That is: pass by reference and return a value type. If you then 
 pass an owned pointer ~T, you can directly construct the 
 returned value in heap allocated memory ~identity(a) and assign 
 the resulting owned pointer. The compiler is smart enough to 
 not do unnecessary copies.


This would definetly make r an unbecessary heap copy.
Return *x would also only work for POD values, because for all 
others it would cause a (not allowed) move ibstead of a copy. For 
a copy you would have to return x.clone().

You can return by reference in rust, but you have to annotate 
everything with proper lifetimes.
Possibly incorrect example:
fn identity<'a>(x: &'a T) -> &'a {
x
}

I dont think the comparisons to rust make too much sense because 
the whole system works completely different. For D the question 
mainly is how to manage classes (and also interfaces to them) 
with GC or ARC. Rust at the moment can't manage any 
interfaces/traits with theit smart pointers.



 Feb 05 2014



prev sibling  parent  "Meta" <jared771 gmail.com>  writes:


On Wednesday, 5 February 2014 at 00:50:55 UTC, Walter Bright 
wrote:

 On 2/4/2014 4:07 PM, Frank Bauer wrote:

 Regarding different pointer types in Rust as function 
 arguments:

 A function that takes a borrowed pointer &T can also be called 
 with an owning
 pointer ~T, an RC pointer Rc<T>, or a GC pointer Gc<T>. They 
 all convert neatly
 to a &T. One function to rule them ... err .. accomodate all.


 Again, what happens with:

     T identity(T t) { return t; }

 ? I.e. the transference of the argument pointer type to the 
 return type?


The compiler already infers  safe nothrow pure on template 
functions. Couldn't it do the same for  nullable?



 Feb 04 2014



prev sibling  parent  "Matthias Einwag" <matthias.einwag googlemail.com>  writes:


 A function that takes a borrowed pointer &T can also be called 
 with an owning pointer ~T, an RC pointer Rc<T>, or a GC pointer 
 Gc<T>. They all convert neatly to a &T. One function to rule 
 them ... err .. accomodate all.


No. That would be nice to have, but it's actually not the case. 
The only type that automatically coerces to a borrowed reference 
is an owned pointer. For value references you fist have to create 
a borrowes ref through
&val. For Gc and Rc you have to pass something like 
ptr.borrow().borrow_mut ().borrow()



 Feb 04 2014



prev sibling  parent  Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-02-04 22:30:40 +0000, Walter Bright <newshound2 digitalmars.com> said:


 On 2/4/2014 4:23 AM, Michel Fortin wrote:

 For the rare cases where you actually want both versions to work,


 
 I think you're making a vast assumption that the case is rare.
 
 When I write utility functions, I want them to work on as wide a 
 variety of inputs as possible. Otherwise, they are not very useful.


Well, if you think it's needed I won't contradict you. But that does 
not bode well for the storage class solution. If you really think it's 
needed, then you could just double the '?' to give it a similar meaning 
to inout:

	Object?? passthrough(Object?? o)
	{
		// do something here
		return o;
	}

It looks elegant to me, even if I'm not convinced it is necessary.



 you can write them twice  or use a template (except in a virtual 
 context), and in both cases you keep the efficiency of not checking for 
 null when the argument is not nullable.


 
 That's just what I wish to avoid. Consider adding more pointer types - 
 the combinatorics quickly explode. Heck, just have two pointer 
 parameters, and you already have 4 cases.


So you are already worried we'll have to add more pointer types? ;-)


kinds of pointers:

	Object   o1; // not-nullable & not-unique
	Object?  o2; // nullable & not-unique



And like above, double the marker to make it work like inout:


	{
		// have some fun here
		return o;
	}

If you really think more pointer types will be added, then I suggest 
the implementation for this in the compiler be generic enough to 
support yes/no/maybe states for a couple of flags. Perhaps even 
user-defined flags for use by libraries (i.e. Dicebot's 
SQLEscapedString, or UnicodeNormalizedNFD, or why not PrimeNumber).

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 Feb 04 2014



prev sibling next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/2/14, 9:30 AM, Namespace wrote:




Because  nullable is already in the language.

Andrei



 Feb 02 2014



prev sibling  parent reply Shammah Chancellor <anonymous coward.com>  writes:


On 2014-02-02 17:30:27 +0000, Namespace said:





 nullable is not supposed to do the same thing as "Type?" (or rather 
the syntatic sugar for Nullabe<Type> template).   Nullable<type> is to 
enable Value types to be null. For example: Nullable<Double>

 nullable in D is supposed to allow reference types to contain null.  


-S



 Feb 04 2014



  parent reply "Namespace" <rswhite4 googlemail.com>  writes:


On Wednesday, 5 February 2014 at 00:12:11 UTC, Shammah Chancellor 
wrote:

 On 2014-02-02 17:30:27 +0000, Namespace said:



 ?


  nullable is not supposed to do the same thing as "Type?" (or 
 rather the syntatic sugar for Nullabe<Type> template).   
 Nullable<type> is to enable Value types to be null. For 
 example: Nullable<Double>

  nullable in D is supposed to allow reference types to contain 


 -S


class Foo { }

Foo? f;

It's the same.



 Feb 04 2014



  parent  "Shammah Chancellor" <anonymous coward.com>  writes:


On Wednesday, 5 February 2014 at 00:16:19 UTC, Namespace wrote:

 On Wednesday, 5 February 2014 at 00:12:11 UTC, Shammah 
 Chancellor wrote:

 On 2014-02-02 17:30:27 +0000, Namespace said:



 "Type?" ?


  nullable is not supposed to do the same thing as "Type?" (or 
 rather the syntatic sugar for Nullabe<Type> template).   
 Nullable<type> is to enable Value types to be null. For 
 example: Nullable<Double>

  nullable in D is supposed to allow reference types to contain 


 -S


 class Foo { }

 Foo? f;

 It's the same.


No it's not.



http://msdn.microsoft.com/en-us/library/system.nullable(v=vs.110).aspx

"A type is said to be nullable if it can be assigned a value or
can be assigned null, which means the type has no value
whatsoever. By default, all reference types, such as String, are
nullable, but all value types, such as Int32, are not.

using the ? notation after the value type. For example, int? in

that can be assigned null.
The Nullable class provides complementary support for the
Nullable<T> structure. The Nullable class supports obtaining the
underlying type of a nullable type, and comparison and equality
operations on pairs of nullable types whose underlying value type
does not support generic comparison and equality operations."



 Feb 05 2014



prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com>  writes:


On 2/2/14, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:

 2. Work on Phobos to see what can be done about avoiding unnecessary
 allocation. Most likely we'll need to also add a  nogc flag.


See Johannes'es pull here:
https://github.com/D-Programming-Language/dmd/pull/1886

Where he comments:

```
Note: This could also be a start to implement a -nogc parameter which
would check a whole module not to use the GC. This is less intrusive
than a  nogc attribute.
```



 Feb 02 2014



  parent reply Johannes Pfau <nospam example.com>  writes:


Am Sun, 2 Feb 2014 18:36:29 +0100
schrieb Andrej Mitrovic <andrej.mitrovich gmail.com>:


 On 2/2/14, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:

 2. Work on Phobos to see what can be done about avoiding unnecessary
 allocation. Most likely we'll need to also add a  nogc flag.


 
 See Johannes'es pull here:
 https://github.com/D-Programming-Language/dmd/pull/1886
 
 Where he comments:
 
 ```
 Note: This could also be a start to implement a -nogc parameter which
 would check a whole module not to use the GC. This is less intrusive
 than a  nogc attribute.
 ```


The last iteration actually implements -nogc and the pull request
also provides almost everything needed for  nogc. The attribute
should probably be added in a second pull request as I expect more
discussions there.



 Feb 02 2014



  parent  Andrej Mitrovic <andrej.mitrovich gmail.com>  writes:


On 2/2/14, Johannes Pfau <nospam example.com> wrote:

 The last iteration actually implements -nogc and the pull request
 also provides almost everything needed for  nogc.


Excellent. 2014 is looking great so far. :>



 Feb 02 2014



prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 02/02/2014 05:55 PM, Andrei Alexandrescu wrote:

 I think of the following foci for the first half of 2014:

 1. Add  nullable and provide a -nullable compiler flag to verify it. The
 attribute is inferred locally and for white-box functions (lambdas,
 templates), and required as annotation otherwise. References not
 annotated with  nullable are statically enforced to never be null.


- I assume that also, references annotated with  nullable are statically 
enforced to never be dereferenced?

- how to eg. create an array of nullable references? struct Nullable!T{ 
 nullable T t; alias t this; } Nullable!T[] foo; ? I think that would 
cause problems with IFTI.



 Feb 02 2014



 next sibling parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 02/02/2014 06:55 PM, Timon Gehr wrote:

 struct Nullable!T


struct Nullable(T). Non-uniformity of declaration and usage...



 Feb 02 2014



prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/2/2014 9:55 AM, Timon Gehr wrote:

 - how to eg. create an array of nullable references? struct Nullable!T{
  nullable T t; alias t this; } Nullable!T[] foo; ? I think that would cause
 problems with IFTI.


 nullable being a storage class, this couldn't be done with  nullable just as 
you cannot create an array of 'ref' types.



 Feb 02 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/2/14, 12:07 PM, Walter Bright wrote:

 On 2/2/2014 9:55 AM, Timon Gehr wrote:

 - how to eg. create an array of nullable references? struct Nullable!T{
  nullable T t; alias t this; } Nullable!T[] foo; ? I think that would
 cause
 problems with IFTI.


  nullable being a storage class, this couldn't be done with  nullable
 just as you cannot create an array of 'ref' types.


I don't see how this restriction would work. On the face of it it turns 
one cannot create an array of Objects that may or may not be null, which 
is a very odd restriction. Such arrays occur in a lot of places.

Andrei



 Feb 02 2014



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/2/14, 9:55 AM, Timon Gehr wrote:

 On 02/02/2014 05:55 PM, Andrei Alexandrescu wrote:

 I think of the following foci for the first half of 2014:

 1. Add  nullable and provide a -nullable compiler flag to verify it. The
 attribute is inferred locally and for white-box functions (lambdas,
 templates), and required as annotation otherwise. References not
 annotated with  nullable are statically enforced to never be null.


 - I assume that also, references annotated with  nullable are statically
 enforced to never be dereferenced?


Correct.


 - how to eg. create an array of nullable references? struct Nullable!T{
  nullable T t; alias t this; } Nullable!T[] foo; ? I think that would
 cause problems with IFTI.


I don't know. We'd need to figure out a way.


Andrei



 Feb 02 2014



prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/2/2014 8:55 AM, Andrei Alexandrescu wrote:

 1. Add  nullable and provide a -nullable compiler flag to verify it. The
 attribute is inferred locally and for white-box functions (lambdas, templates),
 and required as annotation otherwise. References not annotated with  nullable
 are statically enforced to never be null.


I want to clarify that  nullable would be a storage class, NOT a type 
constructor. This means it will apply to the variable itself, not its type.

For example:

 nullable T* p;
auto q = p;      // q is a T*, not  nullable

In a way, it would work much like ref, which is also a storage class and not a 
type constructor.



 Feb 02 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/2/14, 11:58 AM, Walter Bright wrote:

 On 2/2/2014 8:55 AM, Andrei Alexandrescu wrote:

 1. Add  nullable and provide a -nullable compiler flag to verify it. The
 attribute is inferred locally and for white-box functions (lambdas,
 templates),
 and required as annotation otherwise. References not annotated with
  nullable
 are statically enforced to never be null.


 I want to clarify that  nullable would be a storage class, NOT a type
 constructor. This means it will apply to the variable itself, not its type.

 For example:

  nullable T* p;
 auto q = p;      // q is a T*, not  nullable

 In a way, it would work much like ref, which is also a storage class and
 not a type constructor.


That creates problems with creating arrays of nullable objects, as Timon 
Gehr mentioned.

We need to think this through.

Andrei



 Feb 02 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 2/2/2014 1:16 PM, Andrei Alexandrescu wrote:

 That creates problems with creating arrays of nullable objects, as Timon Gehr
 mentioned.

 We need to think this through.


Yup.



 Feb 02 2014



prev sibling next sibling parent  "Joseph Cassman" <jc7919 outlook.com>  writes:


On Sunday, 2 February 2014 at 16:55:35 UTC, Andrei Alexandrescu 
wrote:

 I think of the following foci for the first half of 2014:

 1. Add  nullable and provide a -nullable compiler flag to 
 verify it. The attribute is inferred locally and for white-box 
 functions (lambdas, templates), and required as annotation 
 otherwise. References not annotated with  nullable are 
 statically enforced to never be null.

 2. Work on Phobos to see what can be done about avoiding 
 unnecessary allocation. Most likely we'll need to also add a 
  nogc flag.

 3. Work on adding tracing capabilities to allocators and see 
 how to integrate them with the language and Phobos.

 4. Work on the core language and druntime to see how to 
 seamlessly accommodate alternate GC mechanisms such as 
 reference counting.


 Andrei


This is an excellent direction to take the language in the 
near-term. It seems that some who would like to use D are afraid 
of the GC. Others have legitimate use-cases where the effort of 
manually managing memory is warranted, such as a 
memory-constrained environment. Being able to accommodate both, 
provide an integrated approach to allocation, and allow for 
ref-counting as a viable alternative to GC is a home run. Also 
will be nice to see Phobos memory usage trimmed down. I feel that 
these items mentioned above combined will put D into a niche that 
other languages will find it difficult to compete with: 
end-to-end fine-grained memory control. Nice!

Just my 10 cents.

Joseph



 Feb 03 2014



prev sibling  parent reply "Francesco Cattoglio" <francesco.cattoglio gmail.com>  writes:


On Sunday, 2 February 2014 at 16:55:35 UTC, Andrei Alexandrescu 
wrote:

 1. Add  nullable and provide a -nullable compiler flag to 
 verify it. The attribute is inferred locally and for white-box 
 functions (lambdas, templates), and required as annotation 
 otherwise. References not annotated with  nullable are 
 statically enforced to never be null.

 2. Work on Phobos to see what can be done about avoiding 
 unnecessary allocation. Most likely we'll need to also add a 
  nogc flag.

 3. Work on adding tracing capabilities to allocators and see 
 how to integrate them with the language and Phobos.

 4. Work on the core language and druntime to see how to 
 seamlessly accommodate alternate GC mechanisms such as 
 reference counting.


That looks nice, but one thing is missing:

5. Fix  property already :P

What is the last status about properties? As far as I understand 
pulls are in place and might get pick up in 2.066, right?



 Feb 04 2014



  parent  "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Wednesday, 5 February 2014 at 01:16:47 UTC, Francesco 
Cattoglio wrote:

 5. Fix  property already :P


yes, we should be able to do this with a minimum amount of pain 
and it'd be soo nice to have.



 Feb 04 2014



prev sibling  parent reply "Adam D. Ruppe" <destructionator gmail.com>  writes:


On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei Alexandrescu 
wrote:

 Whoa, this won't work without an explosion in language 
 complexity.


The complexity is already there, it is just hidden. But most the 
capacity to deal with it is already there too:

Owning!T is just a struct with ~this() { free(me); }. Referencing 
counting is just a struct with this(this) { payload.ref++; } 
~this() { payload.ref--; } (with a few optimizations to remove 
ref++; ref--;, which IMO should be there anyway!)

Borrowed is a reference we can use or copy, but must not free 
(delete is deprecated!) and ought not store.

GC is a reference that we don't have to free AND is ok to store.



The language just conflates the latter two. And the spec even 
gives us some hope that it isn't supposed to with the scope 
storage class, though IMO the difference is more of a type thing 
than a storage class thing. A GC reference should implicitly 
convert to a borrowed reference (e.g. alias slice this;) but not 
vice versa. The only way to go from borrowed -> GC (or any other 
really) would be cowboy it with cast() or copying the contents.



BTW I like a lot of the ideas Rust has in this regard, but I 
don't think we have to go all the way they did. But having:

char[] foo = new char[](1024);

and

char[1024] buffer;
char[] foo = buffer[];


both work and yield exactly the same type is trouble.



 Feb 01 2014



  parent reply "Frank Bauer" <y z.com>  writes:


On Sunday, 2 February 2014 at 04:26:16 UTC, Adam D. Ruppe wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei Alexandrescu 
 wrote:

 Whoa, this won't work without an explosion in language 
 complexity.


 The complexity is already there, it is just hidden. But most 
 the capacity to deal with it is already there too:

 Owning!T is just a struct with ~this() { free(me); }. 
 Referencing counting is just a struct with this(this) { 
 payload.ref++; } ~this() { payload.ref--; } (with a few 
 optimizations to remove ref++; ref--;, which IMO should be 
 there anyway!)

 Borrowed is a reference we can use or copy, but must not free 
 (delete is deprecated!) and ought not store.

 GC is a reference that we don't have to free AND is ok to store.



 The language just conflates the latter two. And the spec even 
 gives us some hope that it isn't supposed to with the scope 
 storage class, though IMO the difference is more of a type 
 thing than a storage class thing. A GC reference should 
 implicitly convert to a borrowed reference (e.g. alias slice 
 this;) but not vice versa. The only way to go from borrowed -> 
 GC (or any other really) would be cowboy it with cast() or 
 copying the contents.



 BTW I like a lot of the ideas Rust has in this regard, but I 
 don't think we have to go all the way they did. But having:

 char[] foo = new char[](1024);

 and

 char[1024] buffer;
 char[] foo = buffer[];


 both work and yield exactly the same type is trouble.



I see light ...

There is one little complication of owning pointers interacting 
with borrowed pointers that might prevent a straightforward 
approach (I'm sure there's more): owning pointers must not go out 
of scope or be reassigned as long as there are outstanding 
references to them alive (i.e. in scope as well).

I don't see ATM how this could be done at zero cost without 
intrinsic compiler support. Only the compiler could enforce this 
free of cost as far as I see.



 Feb 01 2014



  parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 20:43:37 -0800, Frank Bauer <y z.com> wrote:


 On Sunday, 2 February 2014 at 04:26:16 UTC, Adam D. Ruppe wrote:

 On Sunday, 2 February 2014 at 03:38:03 UTC, Andrei Alexandrescu wrote:

 Whoa, this won't work without an explosion in language complexity.


 The complexity is already there, it is just hidden. But most the  
 capacity to deal with it is already there too:

 Owning!T is just a struct with ~this() { free(me); }. Referencing  
 counting is just a struct with this(this) { payload.ref++; } ~this() {  
 payload.ref--; } (with a few optimizations to remove ref++; ref--;,  
 which IMO should be there anyway!)

 Borrowed is a reference we can use or copy, but must not free (delete  
 is deprecated!) and ought not store.

 GC is a reference that we don't have to free AND is ok to store.



 The language just conflates the latter two. And the spec even gives us  
 some hope that it isn't supposed to with the scope storage class,  
 though IMO the difference is more of a type thing than a storage class  
 thing. A GC reference should implicitly convert to a borrowed reference  
 (e.g. alias slice this;) but not vice versa. The only way to go from  
 borrowed -> GC (or any other really) would be cowboy it with cast() or  
 copying the contents.



 BTW I like a lot of the ideas Rust has in this regard, but I don't  
 think we have to go all the way they did. But having:

 char[] foo = new char[](1024);

 and

 char[1024] buffer;
 char[] foo = buffer[];


 both work and yield exactly the same type is trouble.



 I see light ...

 There is one little complication of owning pointers interacting with  
 borrowed pointers that might prevent a straightforward approach (I'm  
 sure there's more): owning pointers must not go out of scope or be  
 reassigned as long as there are outstanding references to them alive  
 (i.e. in scope as well).

 I don't see ATM how this could be done at zero cost without intrinsic  
 compiler support. Only the compiler could enforce this free of cost as  
 far as I see.


I believe it should be possible to argue for the compiler support as long  
as it doesn't add any language level complexity. I too like how this works.

However, full ARC presents quite a few more problems. You will certainly  
need to add at least one more keyword to the language ('arc' anyone?) and  
a metric ton of compiler support.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



prev sibling next sibling parent  Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-02-01 09:27:18 +0000, "Adam Wilson" <flyboynw gmail.com> said:


 That's kind of my point. You're asking for massive changes throughout 
 the  entire compiler to support what is becoming more of an edge case, 
 not less  of one. For the vast majority of use cases, a GC is the right 
 call and D  has to cater to the majority if it wants to gain any 
 significant mindshare  at all. You don't grow by increasing 
 specialization...


Keep in mind that improving the GC we have now is probably going to 
require pretty much the same changes in the compiler, perhaps even more 
drastic ones than ARC.

To make it short, to implement a concurrent GC the compiler must add 
some code for each pointer assignment and each pointer move. To 
implement ARC, the compiler must add some code only to pointer 
assignments.

The point is, unless we're satisfied with the current GC 
implementation, those massive changes to the compiler will need to be 
done anyway. From the compiler's point of view ARC or a better 
concurrent GC have pretty similar requirements. And given current D 
semantics (where stack variables are movable at will) ARC is easier to 
implement than a concurrent GC.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 Feb 01 2014



prev sibling  parent reply Manu <turkeyman gmail.com>  writes:


On 1 February 2014 19:27, Adam Wilson <flyboynw gmail.com> wrote:


 On Fri, 31 Jan 2014 23:35:44 -0800, Manu <turkeyman gmail.com> wrote:

  On 1 February 2014 16:26, Adam Wilson <flyboynw gmail.com> wrote:

  On Fri, 31 Jan 2014 21:29:04 -0800, Manu <turkeyman gmail.com> wrote:

  On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:


   std.typecons.RefCounted!T


 core.memory.GC.disable();


  Wow. That was easy.


 I see, D's claim of being a multi-paradigm language is not false.


 It's not a realistic suggestion. Everything you want to link uses the
 GC,
 and the language its self also uses the GC. Unless you write software in
 complete isolation and forego many valuable features, it's not a
 solution.


  Phobos does rely on the GC to some extent. Most algorithms and ranges
 do

  not though.

  Running (library) code that was written with GC in mind and turning


 GC
 off
 doesn't sound ideal.

 But maybe this allows me to familiarise myself more with D. Who knows,
 maybe I can learn to stop worrying and love garbage collection.

 Thanks for your help!


  I've been trying to learn to love the GC for as long as I've been


 around
 here. I really wanted to break that mental barrier, but it hasn't
 happened.
 In fact, I am more than ever convinced that the GC won't do. My current

 wishlist item for D is the ability to use a reference counted collector
 in
 place of the built-in GC.
 You're not alone :)

 I write realtime and memory-constrained software (console games), and
 for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable
 memory
 footprint of the application. You can't make any guarantees or
 predictions
 about the GC, which is fundamentally incompatible with realtime
 software.
 Language-level ARC would probably do quite nicely for the miscellaneous
 allocations. Obviously, bulk allocations are still usually best handled
 in
 a context sensitive manner; ie, regions/pools/freelists/whatever, but
 the
 convenience of the GC paradigm does offer some interesting and massively
 time-saving features to D.
 Everyone will always refer you to RefCounted, which mangles your types
 and
 pollutes your code, but aside from that, for ARC to be useful, it needs
 to
 be supported at the language-level, such that the language/optimiser is
 able to optimise out redundant incref/decref calls, and also that it is
 compatible with immutable (you can't manage a refcount if the object is
 immutable).


 The problem isn't GC's per se. But D's horribly naive implementation,
 games are written on GC languages now all the time (Unity/.NET). And
 let's
 be honest, games are kind of a speciality, games do things most programs
 will never do.

 You might want to read the GC Handbook. GC's aren't bad, but most, like
 the D GC, are just to simplistic for common usage today.



 Maybe a sufficiently advanced GC could address the performance
 non-determinism to an acceptable level, but you're still left with the
 memory non-determinism, and the conundrum that when your heap approaches
 full (which is _always_ on a games console), the GC has to work harder and
 harder, and more often to try and keep the tiny little bit of overhead
 available.
 A GC heap by nature expects you to have lots of memory, and also lots of
 FREE memory.

 No serious console game I'm aware of has ever been written in a language
 with a GC. Casual games, or games that don't attempt to raise the bar may
 get away with it, but that's not the industry I work in.


 That's kind of my point. You're asking for massive changes throughout the
 entire compiler to support what is becoming more of an edge case, not less
 of one. For the vast majority of use cases, a GC is the right call and D
 has to cater to the majority if it wants to gain any significant mindshare
 at all. You don't grow by increasing specialization...



Why is ARC any worse than GC? Why is it even a compromise at the high level?
Major players have been moving away from GC to ARC in recent years. It's
still a perfectly valid method of garbage collection, and it has the
advantage that it's intrinsically real-time compatible.

I don't think realtime software is becoming an edge case by any means,
maybe 'extreme' realtime is, but that doesn't change the fact that the GC
still causes problems for all realtime software.

I personally believe latency and stuttering is one of the biggest usability
hindrances in modern computing, and it will become a specific design focus
in software of the future. People are becoming less and less tolerant of
latency in all forms; just consider the success of iPhone compared to
competition, almost entirely attributable to the silky smooth UI
experience. It may also be a telling move that Apple switched to ARC around
the same time, but I don't know the details.

I also firmly believe that if D - a native compiled language familiar to
virtually all low-level programmers - doesn't have any ambition to service
the embedded space in the future, what will? And why not?
The GC is the only thing inhibiting D from being a successful match in that
context. ARC is far more appropriate, and will see it enter a lot more
fields.
What's the loss?

I think it's also telling that newcomers constantly raise it as a massive
concern, or even a deal-breaker. Would they feel the same about ARC? I
seriously doubt it. I wonder if a poll is in order...
Conversely, would any of the new-comers who are pro-GC feel any less happy
if it were ARC instead? $100 says they probably wouldn't even know, and
almost certainly wouldn't care.



 Feb 01 2014



 next sibling parent  "Dicebot" <public dicebot.lv>  writes:


On Saturday, 1 February 2014 at 19:35:08 UTC, Manu wrote:

 I also firmly believe that if D - a native compiled language 
 familiar to
 virtually all low-level programmers - doesn't have any ambition 
 to service
 the embedded space in the future, what will?


I feel like Rust have already won this. It has more suitable 
basic and has more developers -> evolves faster in general. Can 
be too late to even call it priority.

// only 50% trolling



 Feb 01 2014



prev sibling  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 02:15:42 -0800, Manu <turkeyman gmail.com> wrote:


 On 1 February 2014 19:27, Adam Wilson <flyboynw gmail.com> wrote:


 On Fri, 31 Jan 2014 23:35:44 -0800, Manu <turkeyman gmail.com> wrote:

  On 1 February 2014 16:26, Adam Wilson <flyboynw gmail.com> wrote:

  On Fri, 31 Jan 2014 21:29:04 -0800, Manu <turkeyman gmail.com> wrote:

  On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:


   std.typecons.RefCounted!T


 core.memory.GC.disable();


  Wow. That was easy.


 I see, D's claim of being a multi-paradigm language is not false.


 It's not a realistic suggestion. Everything you want to link uses the
 GC,
 and the language its self also uses the GC. Unless you write  
 software in
 complete isolation and forego many valuable features, it's not a
 solution.


  Phobos does rely on the GC to some extent. Most algorithms and  
 ranges
 do

  not though.

  Running (library) code that was written with GC in mind and  
 turning


 GC
 off
 doesn't sound ideal.

 But maybe this allows me to familiarise myself more with D. Who  
 knows,
 maybe I can learn to stop worrying and love garbage collection.

 Thanks for your help!


  I've been trying to learn to love the GC for as long as I've been


 around
 here. I really wanted to break that mental barrier, but it hasn't
 happened.
 In fact, I am more than ever convinced that the GC won't do. My  
 current

 wishlist item for D is the ability to use a reference counted  
 collector
 in
 place of the built-in GC.
 You're not alone :)

 I write realtime and memory-constrained software (console games), and
 for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable
 memory
 footprint of the application. You can't make any guarantees or
 predictions
 about the GC, which is fundamentally incompatible with realtime
 software.
 Language-level ARC would probably do quite nicely for the  
 miscellaneous
 allocations. Obviously, bulk allocations are still usually best  
 handled
 in
 a context sensitive manner; ie, regions/pools/freelists/whatever, but
 the
 convenience of the GC paradigm does offer some interesting and  
 massively
 time-saving features to D.
 Everyone will always refer you to RefCounted, which mangles your  
 types
 and
 pollutes your code, but aside from that, for ARC to be useful, it  
 needs
 to
 be supported at the language-level, such that the language/optimiser  
 is
 able to optimise out redundant incref/decref calls, and also that it  
 is
 compatible with immutable (you can't manage a refcount if the object  
 is
 immutable).


 The problem isn't GC's per se. But D's horribly naive implementation,
 games are written on GC languages now all the time (Unity/.NET). And
 let's
 be honest, games are kind of a speciality, games do things most  
 programs
 will never do.

 You might want to read the GC Handbook. GC's aren't bad, but most,  
 like
 the D GC, are just to simplistic for common usage today.



 Maybe a sufficiently advanced GC could address the performance
 non-determinism to an acceptable level, but you're still left with the
 memory non-determinism, and the conundrum that when your heap  
 approaches
 full (which is _always_ on a games console), the GC has to work harder  
 and
 harder, and more often to try and keep the tiny little bit of overhead
 available.
 A GC heap by nature expects you to have lots of memory, and also lots  
 of
 FREE memory.

 No serious console game I'm aware of has ever been written in a  
 language
 with a GC. Casual games, or games that don't attempt to raise the bar  
 may
 get away with it, but that's not the industry I work in.


 That's kind of my point. You're asking for massive changes throughout  
 the
 entire compiler to support what is becoming more of an edge case, not  
 less
 of one. For the vast majority of use cases, a GC is the right call and D
 has to cater to the majority if it wants to gain any significant  
 mindshare
 at all. You don't grow by increasing specialization...



 Why is ARC any worse than GC? Why is it even a compromise at the high  
 level?
 Major players have been moving away from GC to ARC in recent years. It's
 still a perfectly valid method of garbage collection, and it has the
 advantage that it's intrinsically real-time compatible.


Define Major Players? Because I only know about Apple, but they've been  
doing ARC for almost a decade, and IIRC, like GC's, it's not universally  
loved there either. Microsoft is desperately trying to get people to move  
back to C++ but so far the community has spoken with a resounding "You can  

interest in moving away from GC's probably because their GC is best in  
class. Even games are starting to bring in GC's (The Witcher 2 for  
example, and almost all of the mobile/casual games market, which is  
actually monetarily bigger than the triple-A games market.)


 I don't think realtime software is becoming an edge case by any means,
 maybe 'extreme' realtime is, but that doesn't change the fact that the GC
 still causes problems for all realtime software.


Yes, but my point is that there is very little real-time software written  
as a percentage of all software written, which, by definition, makes it an  
edge-case. Even vehicle control software is no longer completely  
real-time. [I just happen to know that because that's the industry I work  
in. Certain aspects are, with the rest scheduled out.] And more to the  
point, D has made no claim about it's suitability for RT software and I  
have seen little demand for it outside a very small very vocal minority  
that is convinced that it has the dynamic resource management panacea if  
everyone would just do as they say.


 I personally believe latency and stuttering is one of the biggest  
 usability
 hindrances in modern computing, and it will become a specific design  
 focus
 in software of the future. People are becoming less and less tolerant of
 latency in all forms; just consider the success of iPhone compared to
 competition, almost entirely attributable to the silky smooth UI
 experience. It may also be a telling move that Apple switched to ARC  
 around
 the same time, but I don't know the details.


I use .NET every day, seriously not one day goes by when I haven't touched  

Studio 2013 doesn't stutter often, and only when I am pulling in some  
massive third-party module that may or may not be well written.  
Ironically, it is VisualD, it of D code fame that most slows down VS for  

to have a problem with stuttering in my code that wasn't of my own devise.  
(If you forget to asynchronously call that web-service it WILL stutter)


having to worry about circular references or if my data prematurely falls  
out of scope or any other of the details that are associated with ARC. And  
for my not-effort, I pay an effective cost of 0. Win-win. You're demanding  
that to suit your needs, we make a massive philosophical and language  
change to D that will incur HIGHER cognitive load on programmers for  
something that will not measurably improve the general use case? Ahem,  
that's good for D how?


 I also firmly believe that if D - a native compiled language familiar to
 virtually all low-level programmers - doesn't have any ambition to  
 service
 the embedded space in the future, what will? And why not?
 The GC is the only thing inhibiting D from being a successful match in  
 that
 context. ARC is far more appropriate, and will see it enter a lot more
 fields.
 What's the loss?


Cognitive load. How many details does the programmer have to worry about  
per line of code. Ease-of-use. A GC is easier to use in practice. You can  
say well they should learn to use ARC because it's better (for certain  
definitions of better) but they won't. They'll just move on. I'd say  
that's a pretty big loss.

And based on my conversations with Walter, I don't think that D was ever  
intended to make a play for the embedded space. If D can be made to work  
there great, but Walter, as near as I can tell, has no interest in tying  
the language in knots to make it happen. So that's a non-issue. And let's  
be honest, the requirements of that space are fairly extreme.


 I think it's also telling that newcomers constantly raise it as a massive
 concern, or even a deal-breaker. Would they feel the same about ARC? I
 seriously doubt it. I wonder if a poll is in order...
 Conversely, would any of the new-comers who are pro-GC feel any less  
 happy
 if it were ARC instead? $100 says they probably wouldn't even know, and
 almost certainly wouldn't care.


I DON'T see a majority of newcomers raising an issue with the GC, I only  
see it from newcomers with some pretty extreme latency requirements,  
primarily for the real-time crowd. The majority of newcomers aren't  
interested in RT work. I think you're falling prey to confirmation bias  
here.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



 next sibling parent reply "Frank Bauer" <y z.com>  writes:


Granted, low latency apps are the minority. C++ apps are a 

is merit to a GC in all apps that don't have to squeeze the last 
bit of performance out of the machine, yes, the vast majority, I 
know.

But maybe we should ask ourselves if it makes sense to get the 
low latency developers on board, especially if D's syntax remains 
basically unchanged in the process. Heck, we could even keep 
'new' instead of Rusts '~' for owning pointers.

On Saturday, 1 February 2014 at 19:40:20 UTC, Adam Wilson wrote:

 On Sat, 01 Feb 2014 06:33:15 -0800, Frank Bauer <x y.com> wrote:

 ARC would be a plus for heavily interdependent code. But it 
 doesn't beat unique_ptr semantics in local use that free their 
 memory immediately as they go out of scope or are reassigned.


 So you want C++ back then? Got it.


Ahem ... no you don't. I tried to make my point as clear as 
possible. Unique_ptr *semantics*, e.g. Rusts 'let x = ~X' 
*without* GC is as easy to use as 'auto x = new X' *with* GC. In 
both cases the runtime takes care of freeing the object without 
user intervention.

So newbies should not be turned off, if that is your concern.

In general, I don't understand your arguments against dumping the 
GC if it does not make much difference in your code anyways? 
Getting rid of mandatory GC would not take away one iota of your 
D expressiveness. It means more work on the part of D's 
implementers, to be sure.

I'm close to Stuttgart and Ingolstadt where Mercedes Benz and 
Audi produce their flagship cars for a tiny minority of 
customers. Yet their reputation with the average buyer relies 
heavily on how well their top of the line products fare.

Do we want D to be the flagship?



 Feb 01 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 16:07:20 -0800, Frank Bauer <y z.com> wrote:


 Granted, low latency apps are the minority. C++ apps are a minority  

 GC in all apps that don't have to squeeze the last bit of performance  
 out of the machine, yes, the vast majority, I know.

 But maybe we should ask ourselves if it makes sense to get the low  
 latency developers on board, especially if D's syntax remains basically  
 unchanged in the process. Heck, we could even keep 'new' instead of  
 Rusts '~' for owning pointers.

 On Saturday, 1 February 2014 at 19:40:20 UTC, Adam Wilson wrote:

 On Sat, 01 Feb 2014 06:33:15 -0800, Frank Bauer <x y.com> wrote:

 ARC would be a plus for heavily interdependent code. But it doesn't  
 beat unique_ptr semantics in local use that free their memory  
 immediately as they go out of scope or are reassigned.


 So you want C++ back then? Got it.


 Ahem ... no you don't. I tried to make my point as clear as possible.  
 Unique_ptr *semantics*, e.g. Rusts 'let x = ~X' *without* GC is as easy  
 to use as 'auto x = new X' *with* GC. In both cases the runtime takes  
 care of freeing the object without user intervention.

 So newbies should not be turned off, if that is your concern.

 In general, I don't understand your arguments against dumping the GC if  
 it does not make much difference in your code anyways? Getting rid of  
 mandatory GC would not take away one iota of your D expressiveness. It  
 means more work on the part of D's implementers, to be sure.

 I'm close to Stuttgart and Ingolstadt where Mercedes Benz and Audi  
 produce their flagship cars for a tiny minority of customers. Yet their  
 reputation with the average buyer relies heavily on how well their top  
 of the line products fare.

 Do we want D to be the flagship?


The problem isn't syntax, it's semantics. It's not 'new' versus '~' (which  
by the way is not easy to type on a standard US keyboard, it's a shift+key  
that's in an an uncomfortable position). It's the number of rules and  
special cases around what happens when I do something. You're right, you  
are not asking for a big syntactic change, and I know that. What you are  
asking for is a massive semantics change, and that means new rules and  
most likely, more of them.


Top 100. On TIOBE, there are three non-GC languages in the entire top 20  
(C, C++, Obj-C) the rest feature a non-ARC GC of some form. I am not  
concerned with how the languages manage memory under-the-hood, only how  
they present their memory management semantics to the user.

It's interesting that you bring up Mercedes and Audi. Yes, their  
reputation is defined by their high-end products. But how many people  
actually buy an R8 or an SLK Black? The vast majority or everyday don't  
buy a completely bonkers super-car, they by cars that get the job done.  
Why do Toyota and Volkswagen produce the most cars in the world? Not  
because either company (yes, I am aware of VW's 13 name-plates, I'm  
talking about the VW name-plate here) is particularly renowned for  
producing supercars, but because their cars reliably get the job done. A  
Corolla is not the fastest, it's not the prettiest, and it's certainly not  
the most expensive. More Corolla's have been sold world-wide (40 million)  
than any other single car model in history.

You want a super-car, with all the fancy buttons and knobs and options,  
and ARC is very analogous to a super-car in this respect. In fact many of  
the complaints I see about Obj-C's ARC have to do with the semantics of  
it. ARC is by definition semantically more complex than a GC. But most  
people don't want those options, they just want to put the key in and have  
the car reliably do what they ask within reasonable boundaries and that is  
what a GC does best. The semantics of a good GC are nearly transparent.

But that's where the analogy breaks down. You can stratify car models. You  
can't deliver multiple exclusive strata in a programming language. A  
language is what it is, there are no multiple levels.

You say, let's bring on the low-latency developers to prove how fast D is  
and how it has all these great options. The problem is most programmers  
don't need those options on a day to day basis, but having to carry around  
the semantics for them everywhere is wearying. I say, let's fix the GC to  
so that the masses can do what they need to do without any fuss.

Most car drivers aren't Jeremy Clarkson. And most programmers aren't  
real-time embedded developers. A friend of mine has a saying "Sell to the  
masses, eat with the classes. Sell to the classes, eat with the masses."  
It's a catchy way of saying, you don't make money or gain popularity by  
catering to the most demanding of users or the most wealthy of clients.  
You gain popularity by catering to the NEEDS of the everyday user, or the  
middle-class client.

The reason I am being to vehement, is that I unlike most people here, I  
see D as needing to focus on making the experience better for the every  
day programmer, NOT the experience of the most demanding users. And as C++  
has proven, the two are distinct, writing a language that caters to the  
high-end often precludes the semantics that appeal to the everyday  
precisely because those semantics are too slow.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



 next sibling parent reply "Frank Bauer" <y z.com>  writes:


Agreed. We must keep the GC in D and not change its semantics 
(certainly its performance to be sure).

I would not even want to go the Rust way (their latest anyway, it 
keeps changing, mostly for the better) of relegating the GC to a 
library type Gc<T>. It should remain part of the language for the 
majority of developers who benefit from it.

Again, I agree, making life harder for people who wish to use GC 
would not be acceptable. I enjoy the GC for non-critical apps.

What I am for is an opt-in solution for GC, not necessarily, as 
the OP implies, favoring smart pointers over GC, but for both 
being on an equal footing.

Let developers who strive for ultimate performance choose their 
tool and let D set the standard for what is possible with a 
modern language.

It might be a minority issue overall, but a significant one here 
in these forums, from what I gather.

The D implementers will decide if the two ways of memory 
management can be reconciled and if it's worth the effort. +1 
from me on both.



 Feb 01 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 18:16:07 -0800, Frank Bauer <y z.com> wrote:


 Agreed. We must keep the GC in D and not change its semantics (certainly  
 its performance to be sure).

 I would not even want to go the Rust way (their latest anyway, it keeps  
 changing, mostly for the better) of relegating the GC to a library type  
 Gc<T>. It should remain part of the language for the majority of  
 developers who benefit from it.

 Again, I agree, making life harder for people who wish to use GC would  
 not be acceptable. I enjoy the GC for non-critical apps.

 What I am for is an opt-in solution for GC, not necessarily, as the OP  
 implies, favoring smart pointers over GC, but for both being on an equal  
 footing.

 Let developers who strive for ultimate performance choose their tool and  
 let D set the standard for what is possible with a modern language.

 It might be a minority issue overall, but a significant one here in  
 these forums, from what I gather.

 The D implementers will decide if the two ways of memory management can  
 be reconciled and if it's worth the effort. +1 from me on both.


I am for that as well. GC by default, then opt-in to other semantics, even  
if via new syntax. That will give the flexibility to do what you need to  
do when you need to do it without significantly increasing the costs to  
the common use case... THAT is a laudable position to take. And I'll  
happily root for support different resource management paradigms, because  
the one truth is that there is no one right way. But as soon as I start  
seeing arguments along the lines of "X is stupid because it doesn't work  
for my use case so we must replace it with Y" I'll start fighting.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



  parent  "Frank Bauer" <y z.com>  writes:


Yes, that would be stupid.



 Feb 01 2014



prev sibling next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Sunday, 2 February 2014 at 00:57:38 UTC, Adam Wilson wrote:

 ...


I often regret that D went so well as "yet another but faster" 
general-purpose language and is often viewed as such. And this is 
a niche overloaded with languages. Contrary to that, real-time / 
barebone stuff is rather narrow niche but there are only few 
usable languages there (and 0 good languages).



 Feb 02 2014



  parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 02.02.2014 11:44, schrieb Dicebot:

 On Sunday, 2 February 2014 at 00:57:38 UTC, Adam Wilson wrote:

 ...


 I often regret that D went so well as "yet another but faster"
 general-purpose language and is often viewed as such. And this is a
 niche overloaded with languages. Contrary to that, real-time / barebone
 stuff is rather narrow niche but there are only few usable languages
 there (and 0 good languages).


I consider Ada a good one.



 Feb 02 2014



prev sibling  parent reply John J <john.joyus gmail.com>  writes:


On 02/01/2014 07:57 PM, Adam Wilson wrote:

 On TIOBE, there are three non-GC languages in the entire
 top 20 (C, C++, Obj-C) the rest feature a non-ARC GC of some form.


Add "Pascal" and "Delphi/Object Pascal" to that.



 Feb 04 2014



  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 20:29:18 UTC, John J wrote:

 On 02/01/2014 07:57 PM, Adam Wilson wrote:

 On TIOBE, there are three non-GC languages in the entire
 top 20 (C, C++, Obj-C) the rest feature a non-ARC GC of some 
 form.


 Add "Pascal" and "Delphi/Object Pascal" to that.


And Python, which I believe primarily use ARC and then a weird GC 
of some form to catch cycles.



 Feb 04 2014



  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 20:55:30 UTC, Ola Fosheim Grøstad 
wrote:

 And Python, which I believe primarily use ARC and then a weird 
 GC of some form to catch cycles.


Oh, and PHP too.



 Feb 04 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 20:57:28 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 4 February 2014 at 20:55:30 UTC, Ola Fosheim 
 Grøstad wrote:

 And Python, which I believe primarily use ARC and then a weird 
 GC of some form to catch cycles.


 Oh, and PHP too.


Oh, and Perl 5…



 Feb 04 2014



prev sibling  parent reply Manu <turkeyman gmail.com>  writes:


On 2 February 2014 06:07, Adam Wilson <flyboynw gmail.com> wrote:


 On Sat, 01 Feb 2014 02:15:42 -0800, Manu <turkeyman gmail.com> wrote:


 Why is ARC any worse than GC? Why is it even a compromise at the high
 level?
 Major players have been moving away from GC to ARC in recent years. It's
 still a perfectly valid method of garbage collection, and it has the
 advantage that it's intrinsically real-time compatible.


 Define Major Players? Because I only know about Apple, but they've been
 doing ARC for almost a decade, and IIRC, like GC's, it's not universally
 loved there either. Microsoft is desperately trying to get people to move
 back to C++ but so far the community has spoken with a resounding "You can

 interest in moving away from GC's probably because their GC is best in
 class. Even games are starting to bring in GC's (The Witcher 2 for example,
 and almost all of the mobile/casual games market, which is actually
 monetarily bigger than the triple-A games market.)



You didn't say why ARC is worse than GC?

I have my suspicious that the witcher guys would have been perfectly
(perhaps MORE happy) with ARC, if it were possible.
It's possible to write a GC in C++, but it's impossible to use ARC without
compiler support, which C++ does not and never will have, and as a result,
it wasn't an option to them.
You can't say they picked one over the other when one simply wasn't
available. Their use of a GC shows they appreciate the convenience of
garbage collection, but it doesn't say anything about GC vs ARC.

 I don't think realtime software is becoming an edge case by any means,

 maybe 'extreme' realtime is, but that doesn't change the fact that the GC
 still causes problems for all realtime software.


 Yes, but my point is that there is very little real-time software written
 as a percentage of all software written, which, by definition, makes it an
 edge-case. Even vehicle control software is no longer completely real-time.
 [I just happen to know that because that's the industry I work in. Certain
 aspects are, with the rest scheduled out.] And more to the point, D has
 made no claim about it's suitability for RT software and I have seen little
 demand for it outside a very small very vocal minority that is convinced
 that it has the dynamic resource management panacea if everyone would just
 do as they say.



So answer, why is a GC so much preferable to ARC?
GC forces you into one (poorly implemented) paradigm, which is
fundamentally incompatible with many tasks.
ARC might force you to think about weak references... but is that really as
hard as you say?
It's possible an ARC solution might optionally have a GC in the background
to collect cycles (for those not interested in dealing with the cognitive
load of circular reference as you are worried). This is fine, as long as
the aggressive users can turn that off, and deal with their circular
reference issues directly. Under an approach like this there are far more
options in terms of how different users can approach the garbage collection
in their apps, and ALL USERS would still have access to phobos.
I don't see them as mutually exclusive. They are quite complimentary.

 I personally believe latency and stuttering is one of the biggest usability

 hindrances in modern computing, and it will become a specific design focus
 in software of the future. People are becoming less and less tolerant of
 latency in all forms; just consider the success of iPhone compared to
 competition, almost entirely attributable to the silky smooth UI
 experience. It may also be a telling move that Apple switched to ARC
 around
 the same time, but I don't know the details.


 I use .NET every day, seriously not one day goes by when I haven't touched

 Studio 2013 doesn't stutter often, and only when I am pulling in some
 massive third-party module that may or may not be well written. Ironically,
 it is VisualD, it of D code fame that most slows down VS for me. I write

 problem with stuttering in my code that wasn't of my own devise. (If you
 forget to asynchronously call that web-service it WILL stutter)


 having to worry about circular references or if my data prematurely falls
 out of scope or any other of the details that are associated with ARC. And
 for my not-effort, I pay an effective cost of 0. Win-win. You're demanding
 that to suit your needs, we make a massive philosophical and language
 change to D that will incur HIGHER cognitive load on programmers for
 something that will not measurably improve the general use case? Ahem,
 that's good for D how?



bugger-all memory? I can; it doesn't work (although to be fair, the GC is
not the only issue there).

It's not, D has a terrible GC, and there seems to be almost no motion to
fix or improve it.


to get it there?
And even then, what is the resource overhead? How much ambient memory
overhead/loss will I suffer as a consequence? What if my platform is only
single-core?

 I also firmly believe that if D - a native compiled language familiar to

 virtually all low-level programmers - doesn't have any ambition to service
 the embedded space in the future, what will? And why not?
 The GC is the only thing inhibiting D from being a successful match in
 that
 context. ARC is far more appropriate, and will see it enter a lot more
 fields.
 What's the loss?


 Cognitive load. How many details does the programmer have to worry about
 per line of code. Ease-of-use. A GC is easier to use in practice. You can
 say well they should learn to use ARC because it's better (for certain
 definitions of better) but they won't. They'll just move on. I'd say that's
 a pretty big loss.

 And based on my conversations with Walter, I don't think that D was ever
 intended to make a play for the embedded space. If D can be made to work
 there great, but Walter, as near as I can tell, has no interest in tying
 the language in knots to make it happen. So that's a non-issue. And let's
 be honest, the requirements of that space are fairly extreme.



But D ticks all the boxes, except that one... and it's an important field
that isn't covered by the rest of the landscape of emerging or trendy
languages.

 I think it's also telling that newcomers constantly raise it as a massive

 concern, or even a deal-breaker. Would they feel the same about ARC? I
 seriously doubt it. I wonder if a poll is in order...
 Conversely, would any of the new-comers who are pro-GC feel any less happy
 if it were ARC instead? $100 says they probably wouldn't even know, and
 almost certainly wouldn't care.


 I DON'T see a majority of newcomers raising an issue with the GC, I only
 see it from newcomers with some pretty extreme latency requirements,
 primarily for the real-time crowd. The majority of newcomers aren't
 interested in RT work. I think you're falling prey to confirmation bias
 here.



I never said anything about a majority, re-read my statement. But it does
come up all the time.
Are you surprised that the RT crowd are attracted to D? It's precisely what
they've been waiting for all these years.
There's just this one single stumbling block that seems to make every
single one of them very nervous.

I attended a small graphics/systems programming event yesterday, there were
about 40-50 people there. I hadn't caught up with the organisers for years,
they asked me what I had been up to, I mentioned I had been working a lot
with D in my spare time. They were interested, conversations went a lot
like this:

[among other comments]
"Oh yeah, I heard about that... we'd love you to come and give a talk about
it some time."
(more people overheard and approached the conversation)
"I was really interested in D, but then I realised that the GC isn't really
optional like the website claims, and stopped"
(more people approach)
"I tried it once, but I don't think it's practical for the same reason I

(small crowd, ~10 interested in hearing about it)


reach)
"so, can you actually use D without the GC?"
me: "Erm. Well..."

Etc.

There was really surprising interest, and I ended out drinking beers with 4
guys who were really interested to hear more about it.

There is a massive potential audience here, I've been saying for a long
time, D is the language that game devs and other realtime programmers want,
but they're very conservative, and this particular issue is an absolute red
flag.



 Feb 02 2014



  parent reply "Frank Bauer" <x y.com>  writes:


On Monday, 3 February 2014 at 02:58:59 UTC, Manu wrote:

 But D ticks all the boxes, except that one... and it's an 
 important field
 that isn't covered by the rest of the landscape of emerging or 
 trendy
 languages.


  I think it's also telling that newcomers constantly raise it 
 as a massive
 concern, or even a deal-breaker. Would they feel the same 
 about ARC? I
 seriously doubt it. I wonder if a poll is in order...







Agree with Manu 100% there and the rest of his post as well. I 
imagine standing in that little crowd he talked to.

Anyone asking for the addition of ARC or owning pointers to D, 
gets pretty much ignored. The topic is "Smart pointers instead of 
GC?", remember? People here seem to be more interested in 
diverting to nullable, scope and GC optimization. Telling, indeed.

And, yes, as I posted, I believe one could keep D's syntax 
unchanged, including GC allocation via new, by changing the 
memory regime under the hood and providing for ARCs and owning 
pointers on top of GC. It would be a lot of work taking into 
account storage classes, type construction, pointer interaction 
and what not), I understand. But it would be better spent than on 
more of the same garbage ... err ... collection.

So it's a done deal, then? D didn't go all the way to become a 
systems or highest performance language? Instead, it wants to 


Good luck with that. I don't think those guys are as open to 
change as the C++ guys, also given the higher complexity (i.e. 


IMHO, the D implementers should really reconsider what language 
features they would like to concentrate on if they want to be 
around in the years to come.



 Feb 03 2014



 next sibling parent reply "Frustrated" <c1514843 drdrb.com>  writes:


On Monday, 3 February 2014 at 14:57:35 UTC, Frank Bauer wrote:

 On Monday, 3 February 2014 at 02:58:59 UTC, Manu wrote:

 But D ticks all the boxes, except that one... and it's an 
 important field
 that isn't covered by the rest of the landscape of emerging or 
 trendy
 languages.


 I think it's also telling that newcomers constantly raise it 
 as a massive
 concern, or even a deal-breaker. Would they feel the same 
 about ARC? I
 seriously doubt it. I wonder if a poll is in order...







 Agree with Manu 100% there and the rest of his post as well. I 
 imagine standing in that little crowd he talked to.

 Anyone asking for the addition of ARC or owning pointers to D, 
 gets pretty much ignored. The topic is "Smart pointers instead 
 of GC?", remember? People here seem to be more interested in 
 diverting to nullable, scope and GC optimization. Telling, 
 indeed.

 And, yes, as I posted, I believe one could keep D's syntax 
 unchanged, including GC allocation via new, by changing the 
 memory regime under the hood and providing for ARCs and owning 
 pointers on top of GC. It would be a lot of work taking into 
 account storage classes, type construction, pointer interaction 
 and what not), I understand. But it would be better spent than 
 on more of the same garbage ... err ... collection.

 So it's a done deal, then? D didn't go all the way to become a 
 systems or highest performance language? Instead, it wants to 


 Good luck with that. I don't think those guys are as open to 
 change as the C++ guys, also given the higher complexity (i.e. 


 IMHO, the D implementers should really reconsider what language 
 features they would like to concentrate on if they want to be 
 around in the years to come.


It's pure laziness. That is the hole point of AGC, right? So you
don't have to pay attention to what you are doing.

Just wait. Google is working on it! They are working on cars that
drive themselves so you don't have to. There is no way to
override it. Google decides where you need to go and how fast.
You have no say so in the matter and if you want to take the
scenic route then you'll just have to get out and walk(from a
moving car BTW). If it so happens there is construction work and
you gotta take a detour, well, tell that to the judge or your
insurance company... it's not googles fault for not making the
car smart enough or giving you alternatives(remember, you are too
stupid to know anything).



 Feb 03 2014



 next sibling parent  "Frank Bauer" <x y.com>  writes:


On Monday, 3 February 2014 at 15:09:08 UTC, Frustrated wrote:

 It's pure laziness. That is the hole point of AGC, right? So you
 don't have to pay attention to what you are doing.


Well, aren't we all programmers because we are lazy? Othherwise 
we would be moving physical stuff around ;)

Certainly, Walter, Andrei and the rest were anything but lazy. 
Quite the opposite. But considering what they accomplished 
already, maybe they decide to go that extra mile in the coming 
years.

I actually enjoy the comfort of a GC cleaning up after me, if I 
don't have critical stuff to do. Other than that, you are right: 
if you know what you do in your code and given powerful tools, 
you can accomplish more (performance).



 Feb 03 2014



prev sibling  parent  "Daniel Murphy" <yebbliesnospam gmail.com>  writes:


"Frustrated"  wrote in message news:kpejxijjckravodhixpi forum.dlang.org... 

 It's pure laziness.


Pull requests welcome.



 Feb 03 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


I thought I made it clear that GC avoidance (which includes considering 
built-in reference counting) is a major focus of 2014.

Andrei



 Feb 03 2014



 next sibling parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes considering  
 built-in reference counting) is a major focus of 2014.

 Andrei


Andrei, I am sorry to report that anything other than complete removal of  
the GC and replacement with compiler generated ARC will be unacceptable to  
a certain, highly vocal, subset of D users. No arguments can be made to  
otherwise, regardless of validity. As far as they are concerned the  
discussion of ARC vs. GC is closed and decided. ARC is the only path  
forward to the bright and glorious future of D. ARC most efficiently  
solves all memory management problems ever encountered. Peer-Reviewed  
Research and the Scientific Method be damned! ALL HAIL ARC!

Sadly, although written as hyperbole, I feel that the above is fairly  
close to the actual position of the ARC crowd.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



 next sibling parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 12:21:14 -0800, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu  
 <SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes considering  
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than complete removal  
 of the GC and replacement with compiler generated ARC will be  
 unacceptable to a certain, highly vocal, subset of D users. No arguments  
 can be made to otherwise, regardless of validity. As far as they are  
 concerned the discussion of ARC vs. GC is closed and decided. ARC is the  
 only path forward to the bright and glorious future of D. ARC most  
 efficiently solves all memory management problems ever encountered.  
 Peer-Reviewed Research and the Scientific Method be damned! ALL HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above is fairly  
 close to the actual position of the ARC crowd.


That said, I do think that it should be possible to implement the two  
side-by-side, although it might require some new keywords.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


04-Feb-2014 00:21, Adam Wilson пишет:

 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes
 considering built-in reference counting) is a major focus of 2014.

 Andrei




...

 Sadly, although written as hyperbole, I feel that the above is fairly
 close to the actual position of the ARC crowd.


I won't be surprised that half of current GC problems are because it's 
simply too stupid as an allocator. I had some numbers that I'd need to 
dig up were GC.malloc was ~2x slower then malloc even with garbage 
collection disabled.

With that said I'd focus on ref-counting somehow coexisting with tracing 
GC. It doesn't go well at the moment - try creating dynamic arrays of 
std.stdio.File (that is ref-counted).

-- 
Dmitry Olshansky



 Feb 03 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 12:40:20 -0800, Dmitry Olshansky  =

<dmitry.olsh gmail.com> wrote:


 04-Feb-2014 00:21, Adam Wilson =D0=BF=D0=B8=D1=88=D0=B5=D1=82:

 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets=









 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes
 considering built-in reference counting) is a major focus of 2014.

 Andrei




 ...

 Sadly, although written as hyperbole, I feel that the above is fairly=





 close to the actual position of the ARC crowd.


 I won't be surprised that half of current GC problems are because it's=


  =


 simply too stupid as an allocator. I had some numbers that I'd need to=


  =


 dig up were GC.malloc was ~2x slower then malloc even with garbage  =



 collection disabled.

 With that said I'd focus on ref-counting somehow coexisting with traci=


ng  =


 GC. It doesn't go well at the moment - try creating dynamic arrays of =


 =


 std.stdio.File (that is ref-counted).


I will not defend the current GC, it is about as stupid as you can make =
 =

one and still have a functioning memory management system. It implements=
  =

exactly none of the optimizations recommended for Mark-Sweep GC's in the=
  =

GC Handbook.

That said, I firmly believe that wholesale replacement of the GC is  =

throwing the baby out with the bathwater. Effectively, the poor D GC  =

implementation has become an excuse to launch a crusade against all GC's=
  =

everywhere, never mind that Java and the .NET GC's are consistent exampl=
es  =

of just how good GC's can actually be.

-- =

Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



 next sibling parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Adam Wilson:


 I will not defend the current GC, it is about as stupid as you 
 can make one and still have a functioning memory management 
 system. It implements exactly none of the optimizations 
 recommended for Mark-Sweep GC's in the GC Handbook.


Why don't you open an enhancement request for each of those 
recommended optimizations (explaining as well as possible what to 
do and how and what to expect from them)? I have opened several 
"obvious" enhancement requests, and I have seen them fixed in 
some time.

Bye,
bearophile



 Feb 03 2014



prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Mon, Feb 03, 2014 at 12:52:53PM -0800, Adam Wilson wrote:

 On Mon, 03 Feb 2014 12:40:20 -0800, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:


[...]

I won't be surprised that half of current GC problems are because
it's simply too stupid as an allocator. I had some numbers that
I'd need to dig up were GC.malloc was ~2x slower then malloc even
with garbage collection disabled.

With that said I'd focus on ref-counting somehow coexisting with
tracing GC. It doesn't go well at the moment - try creating
dynamic arrays of std.stdio.File (that is ref-counted).


 
 I will not defend the current GC, it is about as stupid as you can
 make one and still have a functioning memory management system. It
 implements exactly none of the optimizations recommended for
 Mark-Sweep GC's in the GC Handbook.
 
 That said, I firmly believe that wholesale replacement of the GC is
 throwing the baby out with the bathwater. Effectively, the poor D GC
 implementation has become an excuse to launch a crusade against all
 GC's everywhere, never mind that Java and the .NET GC's are
 consistent examples of just how good GC's can actually be.


[...]

I vote for the wholesale replacement of the GC... with a better GC. :-P
Preferably one that is on par with modern Java GC's.

I agree that the GC phobia of some people is a little overboard. It's
what they need for their specific use case, but that doesn't really
apply in other use cases for D. (And I'm not convinced that ARC is the
be-all and end-all of memory management issues -- I've seen real-life
examples of ARCs gone horribly, horribly wrong, whereas had a GC been
used in the first place things wouldn't have gone down that route.)

Nevertheless, supporting both can't be a bad thing; after all, D is all
about choice (it's a breath of fresh air to *not* have to shoehorn every
square peg into the OO round hole, like you have to in Java), so having
the choice to *not* use the GC is a net improvement IMO, even if most D
users wouldn't care about the difference.


T

-- 
Маленькие детки - маленькие бедки.



 Feb 03 2014



  parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com>  writes:


On 2/3/2014 4:13 PM, H. S. Teoh wrote:

 I've seen real-life
 examples of ARCs gone horribly, horribly wrong, whereas had a GC been
 used in the first place things wouldn't have gone down that route.


I'm curious to hear more about this.



 Feb 03 2014



 next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 6:05 PM, Nick Sabalausky wrote:

 On 2/3/2014 4:13 PM, H. S. Teoh wrote:

 I've seen real-life
 examples of ARCs gone horribly, horribly wrong, whereas had a GC been
 used in the first place things wouldn't have gone down that route.


 I'm curious to hear more about this.


Me too.

Andrei



 Feb 03 2014



prev sibling next sibling parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 12:05, Nick Sabalausky <
SeeWebsiteToContactMe semitwist.com> wrote:


 On 2/3/2014 4:13 PM, H. S. Teoh wrote:


 I've seen real-life
 examples of ARCs gone horribly, horribly wrong, whereas had a GC been
 used in the first place things wouldn't have gone down that route.


 I'm curious to hear more about this.


Me too.



 Feb 03 2014



prev sibling next sibling parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 4 February 2014 at 02:05:07 UTC, Nick Sabalausky 
wrote:

 On 2/3/2014 4:13 PM, H. S. Teoh wrote:

 I've seen real-life
 examples of ARCs gone horribly, horribly wrong, whereas had a 
 GC been
 used in the first place things wouldn't have gone down that 
 route.


 I'm curious to hear more about this.


An example is when you have a huge graph and the root reaches it 
count == 0.

The time taken into a cascading deletes of the whole structure is 
similar to a stop-the-world GC pause.


--
Paulo



 Feb 03 2014



 next sibling parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 23:31:25 -0800, Paulo Pinto <pjmlp progtools.org>  
wrote:


 On Tuesday, 4 February 2014 at 02:05:07 UTC, Nick Sabalausky wrote:

 On 2/3/2014 4:13 PM, H. S. Teoh wrote:

 I've seen real-life
 examples of ARCs gone horribly, horribly wrong, whereas had a GC been
 used in the first place things wouldn't have gone down that route.


 I'm curious to hear more about this.


 An example is when you have a huge graph and the root reaches it count  
 == 0.

 The time taken into a cascading deletes of the whole structure is  
 similar to a stop-the-world GC pause.


 --
 Paulo


That's a generic example, but ARC proponents steadfastly maintain that it  
doesn't happen often enough in practice to be relevant. Mr. Teoh's  
examples of it would be most helpful. And there is more than one way ARC  
can go horribly wrong, like cyclic references inside a huge graph.  
Wouldn't THAT be fun! :-) I could design a huge graph that never gets  
freed in ARC in my sleep, I've built them before without trying hard in  


-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



prev sibling  parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 17:31, Paulo Pinto <pjmlp progtools.org> wrote:


 On Tuesday, 4 February 2014 at 02:05:07 UTC, Nick Sabalausky wrote:


 On 2/3/2014 4:13 PM, H. S. Teoh wrote:


 I've seen real-life
 examples of ARCs gone horribly, horribly wrong, whereas had a GC been
 used in the first place things wouldn't have gone down that route.


 I'm curious to hear more about this.


 An example is when you have a huge graph and the root reaches it count ==
 0.

 The time taken into a cascading deletes of the whole structure is similar
 to a stop-the-world GC pause.


Only if it's not deferred, and even then, if you're freeing a huge
structure like that, it's at a particular time where you planned to do so.
No realtime app I know of goes and free's a huge runtime graph mid-frame at
some random moment. That's just silly.

It's still easy to defer deletion under ARC if that's what you want to
do... it's all about _choice_ :)



 Feb 04 2014



prev sibling  parent  "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Tue, Feb 04, 2014 at 01:11:23PM +1000, Manu wrote:

 On 4 February 2014 12:05, Nick Sabalausky <
 SeeWebsiteToContactMe semitwist.com> wrote:
 

 On 2/3/2014 4:13 PM, H. S. Teoh wrote:


 I've seen real-life examples of ARCs gone horribly, horribly wrong,
 whereas had a GC been used in the first place things wouldn't have
 gone down that route.


 I'm curious to hear more about this.


 
 Me too.


Well, first I have to apologize for my misleading message, I meant
*reference counting systems*, not ARC specifically, since the case I had
in mind is a reference-counting system implemented in C, so obviously
it's not *A*RC.  :-(

But anyway, what happened is that in the past, we used to do just
straight malloc/free, but over time discovered that it was an unending
source of embarrassing pointer bugs and memory leaks. So at some point,
the team lead decided to implement a reference-counting system instead.

Which worked *better*, I suppose, but it was hardly the panacea that
people might think. It was still prone to pointer bugs and memory leaks
caused by not updating the reference counts properly -- something that
an ARC wouldn't have, I guess.

But then people started doing things with dtors that they shouldn't
do... like this one especially egregious abuse that I had in mind, where
there was a container pointing to a bunch of subobjects, and whenever
one of the subobjects got deleted, the rest must be deleted as well.
This was implemented by having the dtor of the subobject destruct the
container, which, according to refcount thinking, should "automatically"
clean up everything. Except that it didn't, because this implementation
was wrong on so many levels... the first being that the subobject can't
have its dtor invoked unless the container didn't contribute to the
reference count, and of course, once the dtor is invoked, it tries to
destruct the container by decreasing *its* reference count, which makes
one wonder who's keeping the reference to the container, and what
happens with the container's reference to the current subobject.  This
was the case that really needed a GC, because there are just too many
cross-references going on. Had we used a GC to begin with, we'd never
have gone down this slide of increasingly wrong code. And this was just
the one instance we discovered; who knows how many other such abuses are
still lurking in the code somewhere.

Now you may argue that an ARC wouldn't permit this sort of abuses, but
the fact of the matter is, the same people who can't write bug-free
malloc/free code are the same people who can't write bug-free reference
counting code, and since we have very little say in who we work with,
we're going to have to continue dealing with stupid bugs of this sort.
Even with an ARC, we're going to get people writing stupid code that
create memory leaks with cycles and what-not.  To fully address this
issue at its root, what we really need is a GC, since it eliminates an
entire class of memory bugs. Except that the decision makers are
hardcore C coders who have irrational GC phobia (and I used to be one of
them), so this will probably never happen in *this* job.  (Maybe I
should consider working for Sociomantic or something...)


T

-- 
Truth, Sir, is a cow which will give [skeptics] no more milk, and so
they are gone to milk the bull. -- Sam. Johnson



 Feb 04 2014



prev sibling next sibling parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 06:52, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 12:40:20 -0800, Dmitry Olshansky <
 dmitry.olsh gmail.com> wrote:

  04-Feb-2014 00:21, Adam Wilson =D0=BF=D0=B8=D1=88=D0=B5=D1=82:

 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

  On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes
 considering built-in reference counting) is a major focus of 2014.

 Andrei


  ...



 Sadly, although written as hyperbole, I feel that the above is fairly
 close to the actual position of the ARC crowd.


 I won't be surprised that half of current GC problems are because it's
 simply too stupid as an allocator. I had some numbers that I'd need to d=




ig

 up were GC.malloc was ~2x slower then malloc even with garbage collectio=




n

 disabled.

 With that said I'd focus on ref-counting somehow coexisting with tracing
 GC. It doesn't go well at the moment - try creating dynamic arrays of
 std.stdio.File (that is ref-counted).


 I will not defend the current GC, it is about as stupid as you can make
 one and still have a functioning memory management system. It implements
 exactly none of the optimizations recommended for Mark-Sweep GC's in the =


GC

 Handbook.

 That said, I firmly believe that wholesale replacement of the GC is
 throwing the baby out with the bathwater. Effectively, the poor D GC
 implementation has become an excuse to launch a crusade against all GC's
 everywhere, never mind that Java and the .NET GC's are consistent example=


s

 of just how good GC's can actually be.



Point me at a proposal for a GC that satisfies the problems in my prior
email, which is realistically implement-able in D, and I'll shut up.



 Feb 03 2014



prev sibling next sibling parent reply Manu <turkeyman gmail.com>  writes:


On 4 February 2014 06:52, Adam Wilson <flyboynw gmail.com> wrote:



Java and the .NET GC's are consistent examples of just how good GC's can

 actually be.



Also, neither languages are systems languages, or practical/appropriate on
embedded, or memory limited systems. What is your argument? That D not be a
systems language?
Additionally, if you are going to beat that drum, you need to prove that
either languages GC technology is even possible in the context of D,
otherwise it's a red herring.



 Feb 03 2014



  parent  "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 4 February 2014 at 01:13:09 UTC, Manu wrote:

 On 4 February 2014 06:52, Adam Wilson <flyboynw gmail.com> 
 wrote:



 Java and the .NET GC's are consistent examples of just how good 
 GC's can

 actually be.



 Also, neither languages are systems languages, or 
 practical/appropriate on
 embedded, or memory limited systems. What is your argument? 
 That D not be a
 systems language?
 Additionally, if you are going to beat that drum, you need to 
 prove that
 either languages GC technology is even possible in the context 
 of D,
 otherwise it's a red herring.


Well there are quite a few systems with an usable GC it seems,

Singularity research OS
http://research.microsoft.com/en-us/projects/Singularity/

Netduino hardware boards
http://www.netduino.com/netduino/

Cortex M3 boards with Oberon-07
http://www.astrobe.com/boards.htm

MirageOS

http://www.openmirage.org/


--
Paulo



 Feb 03 2014



prev sibling  parent  "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Mon, 03 Feb 2014 15:52:53 -0500, Adam Wilson <flyboynw gmail.com> wrote:


 That said, I firmly believe that wholesale replacement of the GC is  
 throwing the baby out with the bathwater. Effectively, the poor D GC  
 implementation has become an excuse to launch a crusade against all GC's  
 everywhere, never mind that Java and the .NET GC's are consistent  
 examples of just how good GC's can actually be.


AFAIK, Java's and .NET's GCs require precise collection. This is not  
possible in D (at least not fully). While I think we need to move more  
towards precise collection, we will never be as good as languages which  
have that restriction. Some GC options just aren't available.

I think ARC is good for many (most?) applications, and GC is good for many  
as well. We should have an option of both.

I still look forward to the day when I can write iPhone apps in D instead  
of objective-C :)

-Steve



 Feb 04 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 12:21 PM, Adam Wilson wrote:

 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes
 considering built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than complete removal
 of the GC and replacement with compiler generated ARC will be
 unacceptable to a certain, highly vocal, subset of D users.


Why would you be sorry? "It is what it is", "customer is always right" 
etc. If we deem that important then we should look into supporting that 
scenario.


 No arguments
 can be made to otherwise, regardless of validity. As far as they are
 concerned the discussion of ARC vs. GC is closed and decided. ARC is the
 only path forward to the bright and glorious future of D. ARC most
 efficiently solves all memory management problems ever encountered.
 Peer-Reviewed Research and the Scientific Method be damned! ALL HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above is fairly
 close to the actual position of the ARC crowd.


I'm going on a limb here, but it seems you're not convinced :o).


Andrei



 Feb 03 2014



  parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 12:51:20 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 12:21 PM, Adam Wilson wrote:

 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes
 considering built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than complete removal
 of the GC and replacement with compiler generated ARC will be
 unacceptable to a certain, highly vocal, subset of D users.


 Why would you be sorry? "It is what it is", "customer is always right"  
 etc. If we deem that important then we should look into supporting that  
 scenario.


There is a big difference between supporting a scenario, and forcing it  
down everyone's throats as The One True Path. I would actually get out an  
pull for supporting both, equally. Let me choose, I know more about the  
memory usage patterns of my cloud-based service than a game developer  
does. In some cases ARC is more efficient, in other cases, the extra  
effort required is more than I care to pay my programmers for. In fact,  
for all of our projects, the efficiency cost of the GC is microscopic  
compared to having to pay the developer/time cost of ARC or Manual  

Apple/Obj-C has consistently proved that all the hype about ARC being more  
efficient for free is untrue. There are plenty of complaints in that camp  
about having to hunt down heisenleaks based on ARC leaking because  
somebody forgot to mark something as weak.


 No arguments
 can be made to otherwise, regardless of validity. As far as they are
 concerned the discussion of ARC vs. GC is closed and decided. ARC is the
 only path forward to the bright and glorious future of D. ARC most
 efficiently solves all memory management problems ever encountered.
 Peer-Reviewed Research and the Scientific Method be damned! ALL HAIL  
 ARC!

 Sadly, although written as hyperbole, I feel that the above is fairly
 close to the actual position of the ARC crowd.


 I'm going on a limb here, but it seems you're not convinced :o).


 Andrei


Hehe, you might say that. I did buy the GC Handbook to learn more about  
GC's after all, not replace them with something that has demonstrable  
drawbacks. :-)

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



prev sibling next sibling parent  "Frustrated" <c1514843 drdrb.com>  writes:


On Monday, 3 February 2014 at 20:21:14 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to 
 D, gets
 pretty much ignored. The topic is "Smart pointers instead of 
 GC?",
 remember? People here seem to be more interested in diverting 
 to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes 
 considering built-in reference counting) is a major focus of 
 2014.

 Andrei


 Andrei, I am sorry to report that anything other than complete 
 removal of the GC and replacement with compiler generated ARC 
 will be unacceptable to a certain, highly vocal, subset of D 
 users. No arguments can be made to otherwise, regardless of 
 validity. As far as they are concerned the discussion of ARC 
 vs. GC is closed and decided. ARC is the only path forward to 
 the bright and glorious future of D. ARC most efficiently 
 solves all memory management problems ever encountered. 
 Peer-Reviewed Research and the Scientific Method be damned! ALL 
 HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above is 
 fairly close to the actual position of the ARC crowd.


No!! That seems quite arrogant! No one is trying to take your
toys away from you! We just ask that you allow us to play with
ours as we have done the same for you. It should be a two way
street. Just because you drive down it doesn't mean you own it.

Of course no one is demanding it be done, Anyone can choose to do
what they want. But many love D and what it does but are forced
to take the good with the bad... and there is no great reason for
that.

D has the potential to be one of those languages that have it
all. It can't when you keep it from growing. Removing the AGC
dependence that D has will be like giving water to a seed... and
we all want D to be successful, don't you?

After all, what's the harm in us both being able to play with our
toys? Or are you just the jealous time that doesn't like to see
anyone else happy?



 Feb 03 2014



prev sibling  parent reply Manu <turkeyman gmail.com>  writes:


On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

  On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than complete removal of
 the GC and replacement with compiler generated ARC will be unacceptable to
 a certain, highly vocal, subset of D users. No arguments can be made to
 otherwise, regardless of validity. As far as they are concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the only path
 forward to the bright and glorious future of D. ARC most efficiently solves
 all memory management problems ever encountered. Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above is fairly
 close to the actual position of the ARC crowd.



Don't be a dick.
I get the impression you don't actually read my posts. And I also feel like
you're a lot more dogmatic about this than you think I am.

I'm absolutely fine with GC in most applications, I really couldn't give
any shits if most people want a GC. I'm not dogmatic about it, and I've
**honestly** tried to love the GC for years now.
What I'm concerned about is that I have _no option_ to use D uninhibited
when I need to not have the GC.

These are the problems:
 * GC stalls for long periods time at completely un-predictable moments.
 * GC stalls become longer *and* more frequent as memory becomes less
available, and the working pool becomes larger (what a coincidence).
 * Memory footprint is unknowable, what if you don't have a virtual memory
manager? What if your total memory is measured in megabytes?
 * It's not possible to know when destruction of an object will happen,

cases, and supports the prior point.

Conclusion:
  GC is unfit for embedded systems. One of the most significant remaining
and compelling uses for a native systems language.

The only realistic path I am aware of is to use ARC, which IS a form of GC,
and allows a lot more flexibility in the front-end.
GC forces one very particular paradigm upon you.
ARC is a GC, but it has some complex properties __which can be addressed in
various ways__. Unlike a GC which is entirely inflexible.

You're not happy with ARC's cleaning objects up on the spot? Something that
many people WANT, but I understand zero cleanup times in the running
context is in other occasions a strength of GC; fine, just stick the
pointer on a dead list, and free it either later during idle time, or on
another thread. On the contrary, I haven't heard any proposal for a GC that
would allow it to operate in carefully controlled time-slices, or strictly
during idle-time.
Cycles are a problem with ARC? True, how much effort are you willing to
spend to mitigate the problem? None: run a secondary GC in the background
to collect cycles (yes, there is still a GC, but it has much less work to
do). Some: Disable background GC, manually require user specified weak
references and stuff. Note: A user-preferred combination of the 2 could
severely mitigate the workload of the background GC if it is still desired
to handle some complex situations, or user errors.
Are there any other disadvantages to ARC? I don't know of them if there are.

Is far as I can tell, an ARC collector could provide identical convenience
as the existing GC for anyone that simply doesn't care. It would also seem
that it could provide significantly more options and control for those that
do.

I am _yet to hear anyone present a realistic path forwards using any form
of GC_, so what else do I have to go with? Until I know of any other path
forward, I'll stand behind the only one I can see.
You're just repeating "I don't care about something that a significant
subset of D developers do care about, and I don't think any changes should
be made to support them".
As far as I know, a switch to ARC could be done in a way that 'regular'
users don't lose anything, or even notice... why is that so offensive?



 Feb 03 2014



 next sibling parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 17:04:08 -0800, Manu <turkeyman gmail.com> wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

  On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than complete removal  
 of
 the GC and replacement with compiler generated ARC will be unacceptable  
 to
 a certain, highly vocal, subset of D users. No arguments can be made to
 otherwise, regardless of validity. As far as they are concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the only path
 forward to the bright and glorious future of D. ARC most efficiently  
 solves
 all memory management problems ever encountered. Peer-Reviewed Research  
 and
 the Scientific Method be damned! ALL HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above is fairly
 close to the actual position of the ARC crowd.



 Don't be a dick.
 I get the impression you don't actually read my posts. And I also feel  
 like
 you're a lot more dogmatic about this than you think I am.

 I'm absolutely fine with GC in most applications, I really couldn't give
 any shits if most people want a GC. I'm not dogmatic about it, and I've
 **honestly** tried to love the GC for years now.
 What I'm concerned about is that I have _no option_ to use D uninhibited
 when I need to not have the GC.

 These are the problems:
  * GC stalls for long periods time at completely un-predictable moments.
  * GC stalls become longer *and* more frequent as memory becomes less
 available, and the working pool becomes larger (what a coincidence).
  * Memory footprint is unknowable, what if you don't have a virtual  
 memory
 manager? What if your total memory is measured in megabytes?
  * It's not possible to know when destruction of an object will happen,

 cases, and supports the prior point.

 Conclusion:
   GC is unfit for embedded systems. One of the most significant remaining
 and compelling uses for a native systems language.

 The only realistic path I am aware of is to use ARC, which IS a form of  
 GC,
 and allows a lot more flexibility in the front-end.
 GC forces one very particular paradigm upon you.
 ARC is a GC, but it has some complex properties __which can be addressed  
 in
 various ways__. Unlike a GC which is entirely inflexible.

 You're not happy with ARC's cleaning objects up on the spot? Something  
 that
 many people WANT, but I understand zero cleanup times in the running
 context is in other occasions a strength of GC; fine, just stick the
 pointer on a dead list, and free it either later during idle time, or on
 another thread. On the contrary, I haven't heard any proposal for a GC  
 that
 would allow it to operate in carefully controlled time-slices, or  
 strictly
 during idle-time.
 Cycles are a problem with ARC? True, how much effort are you willing to
 spend to mitigate the problem? None: run a secondary GC in the background
 to collect cycles (yes, there is still a GC, but it has much less work to
 do). Some: Disable background GC, manually require user specified weak
 references and stuff. Note: A user-preferred combination of the 2 could
 severely mitigate the workload of the background GC if it is still  
 desired
 to handle some complex situations, or user errors.
 Are there any other disadvantages to ARC? I don't know of them if there  
 are.

 Is far as I can tell, an ARC collector could provide identical  
 convenience
 as the existing GC for anyone that simply doesn't care. It would also  
 seem
 that it could provide significantly more options and control for those  
 that
 do.

 I am _yet to hear anyone present a realistic path forwards using any form
 of GC_, so what else do I have to go with? Until I know of any other path
 forward, I'll stand behind the only one I can see.
 You're just repeating "I don't care about something that a significant
 subset of D developers do care about, and I don't think any changes  
 should
 be made to support them".
 As far as I know, a switch to ARC could be done in a way that 'regular'
 users don't lose anything, or even notice... why is that so offensive?


I am not trying to be a dick. But I do feel like a small number of people  
are trying to gang up on me for daring to point out that the solution  
they've proposed solution might have bigger problems for other people than  
they care to admit.

You still haven't dealt with the cyclic reference problem in ARC. There is  
absolutely no way ARC can handle that without programmer input, therefore,  
it is simply not possible to switch D to ARC without adding some language  
support to deal with cyclic-refs. Ergo, it is simply not possible to  
seamlessly switch D to ARC without creating all kinds of havoc as people  
now how memory leaks where they didn't before. In order to support ARC the  
D language will necessarily have to grow/change to accommodate it. Apple  
devs constantly have trouble with cyclic-refs to this day.

I am not against supporting ARC side-by-side with the GC (I'm actually  
quite for it, I would love the flexibility), but it is unrealistic to make  
ARC the default option in D as that would subtly break all existing D  
code, something that Walter has point-blank refused to do in much smaller  
easier to find+fix cases. You can't grep for a weak-ref. So if that is  
what you are asking for, then yes, it will never happen in D.

Also, I don't think you've fully considered what the perf penalty actually  
is for a *good* ARC implementation. I just leafed through the P-Code in  
the GC Handbook for their ARC implementation, it's about 4x longer than  
what their best P-Code Mark-Sweep implementation is.

I would also like to point out that the GC Handbook points out six  
scientifically confirmed problems with ARC. (See Page 59)

1. RC imposes a time overhead on mutators in order to manipulate the  
counter.
2. Both the counter manipulation and pointer load/store operations MUST be  
atomic to prevent races.
3. Naive RC turns read ops into store ops to update the count.
4. No RC can reclaim cyclic data structures, which are much more common  
than is typically understood. [Bacon and Rajan 2001]
5. Counter must be the same size as the pointer, which can result in  
significant overhead for small objects.
6. RC can still pause. When the last head to a large pointer structure is  
deleted, RC MUST delete each descendant node.

Note that these are paraphrases of the book, not me talking. And these  
apply equally to ARC and vanilla RC.

Boehm demonstrated in one of his papers (2004) that thread-safe ARC may  
even lead to longer maximum pause times than a standard Tracing GC.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



 next sibling parent reply "Frustrated" <c1514843 drdrb.com>  writes:


On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu <turkeyman gmail.com> 
 wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> 
 wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to 
 D, gets
 pretty much ignored. The topic is "Smart pointers instead 
 of GC?",
 remember? People here seem to be more interested in 
 diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes 
 considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than 
 complete removal of
 the GC and replacement with compiler generated ARC will be 
 unacceptable to
 a certain, highly vocal, subset of D users. No arguments can 
 be made to
 otherwise, regardless of validity. As far as they are 
 concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the 
 only path
 forward to the bright and glorious future of D. ARC most 
 efficiently solves
 all memory management problems ever encountered. 
 Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above 
 is fairly
 close to the actual position of the ARC crowd.



 Don't be a dick.
 I get the impression you don't actually read my posts. And I 
 also feel like
 you're a lot more dogmatic about this than you think I am.

 I'm absolutely fine with GC in most applications, I really 
 couldn't give
 any shits if most people want a GC. I'm not dogmatic about it, 
 and I've
 **honestly** tried to love the GC for years now.
 What I'm concerned about is that I have _no option_ to use D 
 uninhibited
 when I need to not have the GC.

 These are the problems:
 * GC stalls for long periods time at completely un-predictable 
 moments.
 * GC stalls become longer *and* more frequent as memory 
 becomes less
 available, and the working pool becomes larger (what a 
 coincidence).
 * Memory footprint is unknowable, what if you don't have a 
 virtual memory
 manager? What if your total memory is measured in megabytes?
 * It's not possible to know when destruction of an object will 
 happen,

 in many
 cases, and supports the prior point.

 Conclusion:
  GC is unfit for embedded systems. One of the most significant 
 remaining
 and compelling uses for a native systems language.

 The only realistic path I am aware of is to use ARC, which IS 
 a form of GC,
 and allows a lot more flexibility in the front-end.
 GC forces one very particular paradigm upon you.
 ARC is a GC, but it has some complex properties __which can be 
 addressed in
 various ways__. Unlike a GC which is entirely inflexible.

 You're not happy with ARC's cleaning objects up on the spot? 
 Something that
 many people WANT, but I understand zero cleanup times in the 
 running
 context is in other occasions a strength of GC; fine, just 
 stick the
 pointer on a dead list, and free it either later during idle 
 time, or on
 another thread. On the contrary, I haven't heard any proposal 
 for a GC that
 would allow it to operate in carefully controlled time-slices, 
 or strictly
 during idle-time.
 Cycles are a problem with ARC? True, how much effort are you 
 willing to
 spend to mitigate the problem? None: run a secondary GC in the 
 background
 to collect cycles (yes, there is still a GC, but it has much 
 less work to
 do). Some: Disable background GC, manually require user 
 specified weak
 references and stuff. Note: A user-preferred combination of 
 the 2 could
 severely mitigate the workload of the background GC if it is 
 still desired
 to handle some complex situations, or user errors.
 Are there any other disadvantages to ARC? I don't know of them 
 if there are.

 Is far as I can tell, an ARC collector could provide identical 
 convenience
 as the existing GC for anyone that simply doesn't care. It 
 would also seem
 that it could provide significantly more options and control 
 for those that
 do.

 I am _yet to hear anyone present a realistic path forwards 
 using any form
 of GC_, so what else do I have to go with? Until I know of any 
 other path
 forward, I'll stand behind the only one I can see.
 You're just repeating "I don't care about something that a 
 significant
 subset of D developers do care about, and I don't think any 
 changes should
 be made to support them".
 As far as I know, a switch to ARC could be done in a way that 
 'regular'
 users don't lose anything, or even notice... why is that so 
 offensive?


 I am not trying to be a dick. But I do feel like a small number 
 of people are trying to gang up on me for daring to point out 
 that the solution they've proposed solution might have bigger 
 problems for other people than they care to admit.


The problem is that you are selfish. You won't let anyone in the
sand box to play with their toys... it's your sand box, even
though it's not. It's your toys, even though it's not. You pitch
a fit when someone comes into the sand box because you don't want
them their. They might mess up the sand.

Why can't you get it through your thick skull that just because
people are suggesting alternative methods does not mean that you
can't have your precious AGC? Also, why is it so difficult that
just because you have no need for real-time applications that
others don't either?

Your method of thinking is very arrogant and I wish you the best
of luck at the center of the universe... I hear it's a very
unfriendly place.



 Feb 03 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 17:54:22 -0800, Frustrated <c1514843 drdrb.com> wrote:


 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu <turkeyman gmail.com> wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, gets
 pretty much ignored. The topic is "Smart pointers instead of GC?",
 remember? People here seem to be more interested in diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes  
 considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than complete  
 removal of
 the GC and replacement with compiler generated ARC will be  
 unacceptable to
 a certain, highly vocal, subset of D users. No arguments can be made  
 to
 otherwise, regardless of validity. As far as they are concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the only path
 forward to the bright and glorious future of D. ARC most efficiently  
 solves
 all memory management problems ever encountered. Peer-Reviewed  
 Research and
 the Scientific Method be damned! ALL HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above is fairly
 close to the actual position of the ARC crowd.



 Don't be a dick.
 I get the impression you don't actually read my posts. And I also feel  
 like
 you're a lot more dogmatic about this than you think I am.

 I'm absolutely fine with GC in most applications, I really couldn't  
 give
 any shits if most people want a GC. I'm not dogmatic about it, and I've
 **honestly** tried to love the GC for years now.
 What I'm concerned about is that I have _no option_ to use D  
 uninhibited
 when I need to not have the GC.

 These are the problems:
 * GC stalls for long periods time at completely un-predictable moments.
 * GC stalls become longer *and* more frequent as memory becomes less
 available, and the working pool becomes larger (what a coincidence).
 * Memory footprint is unknowable, what if you don't have a virtual  
 memory
 manager? What if your total memory is measured in megabytes?
 * It's not possible to know when destruction of an object will happen,

 cases, and supports the prior point.

 Conclusion:
  GC is unfit for embedded systems. One of the most significant  
 remaining
 and compelling uses for a native systems language.

 The only realistic path I am aware of is to use ARC, which IS a form  
 of GC,
 and allows a lot more flexibility in the front-end.
 GC forces one very particular paradigm upon you.
 ARC is a GC, but it has some complex properties __which can be  
 addressed in
 various ways__. Unlike a GC which is entirely inflexible.

 You're not happy with ARC's cleaning objects up on the spot? Something  
 that
 many people WANT, but I understand zero cleanup times in the running
 context is in other occasions a strength of GC; fine, just stick the
 pointer on a dead list, and free it either later during idle time, or  
 on
 another thread. On the contrary, I haven't heard any proposal for a GC  
 that
 would allow it to operate in carefully controlled time-slices, or  
 strictly
 during idle-time.
 Cycles are a problem with ARC? True, how much effort are you willing to
 spend to mitigate the problem? None: run a secondary GC in the  
 background
 to collect cycles (yes, there is still a GC, but it has much less work  
 to
 do). Some: Disable background GC, manually require user specified weak
 references and stuff. Note: A user-preferred combination of the 2 could
 severely mitigate the workload of the background GC if it is still  
 desired
 to handle some complex situations, or user errors.
 Are there any other disadvantages to ARC? I don't know of them if  
 there are.

 Is far as I can tell, an ARC collector could provide identical  
 convenience
 as the existing GC for anyone that simply doesn't care. It would also  
 seem
 that it could provide significantly more options and control for those  
 that
 do.

 I am _yet to hear anyone present a realistic path forwards using any  
 form
 of GC_, so what else do I have to go with? Until I know of any other  
 path
 forward, I'll stand behind the only one I can see.
 You're just repeating "I don't care about something that a significant
 subset of D developers do care about, and I don't think any changes  
 should
 be made to support them".
 As far as I know, a switch to ARC could be done in a way that 'regular'
 users don't lose anything, or even notice... why is that so offensive?


 I am not trying to be a dick. But I do feel like a small number of  
 people are trying to gang up on me for daring to point out that the  
 solution they've proposed solution might have bigger problems for other  
 people than they care to admit.


 The problem is that you are selfish. You won't let anyone in the
 sand box to play with their toys... it's your sand box, even
 though it's not. It's your toys, even though it's not. You pitch
 a fit when someone comes into the sand box because you don't want
 them their. They might mess up the sand.

 Why can't you get it through your thick skull that just because
 people are suggesting alternative methods does not mean that you
 can't have your precious AGC? Also, why is it so difficult that
 just because you have no need for real-time applications that
 others don't either?

 Your method of thinking is very arrogant and I wish you the best
 of luck at the center of the universe... I hear it's a very
 unfriendly place.


Funny how you conveniently cut the thread immediately prior to paragraph  
where I say I would be all for supporting ARC side-by-side with the GC. It  
was only two paragraphs down, I can't imagine that it would've been that  
hard to include. I can only imagine that you left it off because it makes  
your name-calling look silly.

However I did point out that making ARC the DEFAULT is unlikely to ever  
happen. Walter has consistently refused to approve much smaller breaking  
changes in the past. Why would he all-of-a-sudden decide that now is the  
time for the biggest breaking change in the history of D? I am just  
pointing out that the creator of D is not likely to approve that pull  
request while he is still breathing. Please don't shoot the messenger.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



  parent  "Frustrated" <c1514843 drdrb.com>  writes:


On Tuesday, 4 February 2014 at 02:15:21 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 17:54:22 -0800, Frustrated 
 <c1514843 drdrb.com> wrote:


 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu 
 <turkeyman gmail.com> wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> 
 wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers 
 to D, gets
 pretty much ignored. The topic is "Smart pointers instead 
 of GC?",
 remember? People here seem to be more interested in 
 diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which 
 includes considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than 
 complete removal of
 the GC and replacement with compiler generated ARC will be 
 unacceptable to
 a certain, highly vocal, subset of D users. No arguments 
 can be made to
 otherwise, regardless of validity. As far as they are 
 concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the 
 only path
 forward to the bright and glorious future of D. ARC most 
 efficiently solves
 all memory management problems ever encountered. 
 Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above 
 is fairly
 close to the actual position of the ARC crowd.



 Don't be a dick.
 I get the impression you don't actually read my posts. And I 
 also feel like
 you're a lot more dogmatic about this than you think I am.

 I'm absolutely fine with GC in most applications, I really 
 couldn't give
 any shits if most people want a GC. I'm not dogmatic about 
 it, and I've
 **honestly** tried to love the GC for years now.
 What I'm concerned about is that I have _no option_ to use D 
 uninhibited
 when I need to not have the GC.

 These are the problems:
 * GC stalls for long periods time at completely 
 un-predictable moments.
 * GC stalls become longer *and* more frequent as memory 
 becomes less
 available, and the working pool becomes larger (what a 
 coincidence).
 * Memory footprint is unknowable, what if you don't have a 
 virtual memory
 manager? What if your total memory is measured in megabytes?
 * It's not possible to know when destruction of an object 
 will happen,

 annoying in many
 cases, and supports the prior point.

 Conclusion:
 GC is unfit for embedded systems. One of the most 
 significant remaining
 and compelling uses for a native systems language.

 The only realistic path I am aware of is to use ARC, which 
 IS a form of GC,
 and allows a lot more flexibility in the front-end.
 GC forces one very particular paradigm upon you.
 ARC is a GC, but it has some complex properties __which can 
 be addressed in
 various ways__. Unlike a GC which is entirely inflexible.

 You're not happy with ARC's cleaning objects up on the spot? 
 Something that
 many people WANT, but I understand zero cleanup times in the 
 running
 context is in other occasions a strength of GC; fine, just 
 stick the
 pointer on a dead list, and free it either later during idle 
 time, or on
 another thread. On the contrary, I haven't heard any 
 proposal for a GC that
 would allow it to operate in carefully controlled 
 time-slices, or strictly
 during idle-time.
 Cycles are a problem with ARC? True, how much effort are you 
 willing to
 spend to mitigate the problem? None: run a secondary GC in 
 the background
 to collect cycles (yes, there is still a GC, but it has much 
 less work to
 do). Some: Disable background GC, manually require user 
 specified weak
 references and stuff. Note: A user-preferred combination of 
 the 2 could
 severely mitigate the workload of the background GC if it is 
 still desired
 to handle some complex situations, or user errors.
 Are there any other disadvantages to ARC? I don't know of 
 them if there are.

 Is far as I can tell, an ARC collector could provide 
 identical convenience
 as the existing GC for anyone that simply doesn't care. It 
 would also seem
 that it could provide significantly more options and control 
 for those that
 do.

 I am _yet to hear anyone present a realistic path forwards 
 using any form
 of GC_, so what else do I have to go with? Until I know of 
 any other path
 forward, I'll stand behind the only one I can see.
 You're just repeating "I don't care about something that a 
 significant
 subset of D developers do care about, and I don't think any 
 changes should
 be made to support them".
 As far as I know, a switch to ARC could be done in a way 
 that 'regular'
 users don't lose anything, or even notice... why is that so 
 offensive?


 I am not trying to be a dick. But I do feel like a small 
 number of people are trying to gang up on me for daring to 
 point out that the solution they've proposed solution might 
 have bigger problems for other people than they care to admit.


 The problem is that you are selfish. You won't let anyone in 
 the
 sand box to play with their toys... it's your sand box, even
 though it's not. It's your toys, even though it's not. You 
 pitch
 a fit when someone comes into the sand box because you don't 
 want
 them their. They might mess up the sand.

 Why can't you get it through your thick skull that just because
 people are suggesting alternative methods does not mean that 
 you
 can't have your precious AGC? Also, why is it so difficult that
 just because you have no need for real-time applications that
 others don't either?

 Your method of thinking is very arrogant and I wish you the 
 best
 of luck at the center of the universe... I hear it's a very
 unfriendly place.


 Funny how you conveniently cut the thread immediately prior to 
 paragraph where I say I would be all for supporting ARC 
 side-by-side with the GC. It was only two paragraphs down, I 
 can't imagine that it would've been that hard to include. I can 
 only imagine that you left it off because it makes your 
 name-calling look silly.

 However I did point out that making ARC the DEFAULT is unlikely 
 to ever happen. Walter has consistently refused to approve much 
 smaller breaking changes in the past. Why would he 
 all-of-a-sudden decide that now is the time for the biggest 
 breaking change in the history of D? I am just pointing out 
 that the creator of D is not likely to approve that pull 
 request while he is still breathing. Please don't shoot the 
 messenger.


It has nothing to do with side by side. Who in their right mind
would argue that the AGC is bad in all cases? No one. Who wants
to deal with manual memory management when writing programs that
do not have real-time requirements?

But the fact remains, "D is AGC dependent" and "Some programs
perform poorly when used with AGC" means that D is a bad choice
for those programs, correct? So, your logic is that we are forced
to use D's GC if we want to use D because you think if ARC or MMM
is implemented you'll be forced to use it... and, of course, you
don't write RT apps so it's not your problem.

All we are asking is that D not be GC dependent. This allows one
to write RT apps, embedded apps, etc.

THIS DOES NOT MEAN you can't use a GC to handle your MM. It just
means there are other options, and by your pretentious stance
against "us" tells us you don't want D to change.

Why not join our team? We want AGC, ARC, MMM, MRC, etc. To do
this though D and phobos have to be lifted from the grips of the
GC. IT DOESN'T MEAN you'll have to manage the memory of phobo's.
You can still use the GC just like you do now without any
changes. You'll never know the difference. BUT WE WILL.

It will be the perfect world. We get what we want and you get
what you want. As it stands you, since you have been arguing
points against us, you must not agree with us, which means you do
not want us to be happy. Yet we want both of us to be happy. So
you must be wrong, unless you know for a fact that we both can't
be happy, which is non-sense(cause we can have AGC and ARC side
by side).

So which is it? Are you with us or against us? Do you want to rid
D and Phobos of the hardcoded AGC and allow for other methods of
MM? (hopefully you are intelligent enough to understand that this
doesn't mean that the AGC will go away or be any different to use)

[I'm also not in favor of ridding D of the AGC all together... I
don't think anyone is, it's there, it works for the most part for
most applications... it is handy and useful in many cases... BUT
NOT ALL. So stop pretending it is, and lets solve the problem]



 Feb 03 2014



prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 1. RC imposes a time overhead on mutators in order to 
 manipulate the counter.


Usually you just do a retain() when your thread attain ownership 
(at the root of the call tree). If you use embedded counters you 
most likely want the data in the cache anyway, but the cacheline 
is made dirty. So it cost you a write back. That affects reads of 
small objects more than writes.

(The C++ implementation does not use embedded counters.)


 2. Both the counter manipulation and pointer load/store 
 operations MUST be atomic to prevent races.


Not with transactional memory. You revert to locking, and you 
probably want that kind of synchronization anyway if you run 
multi threaded.


 3. Naive RC turns read ops into store ops to update the count


If we assume naive RC, then we should assume naive GC.


 4. No RC can reclaim cyclic data structures, which are much 
 more common than is typically understood. [Bacon and Rajan 2001]


Weak pointers are sufficient to break cycles. You need a model, 
but you should have one anyway.

In cases where you build temporary data structures you can just 
avoid RC altogether and just use a pool, then free the entire 
pool.


 5. Counter must be the same size as the pointer, which can 
 result in significant overhead for small objects.





 6. RC can still pause. When the last head to a large pointer 
 structure is deleted, RC MUST delete each descendant node.


Not really, you can postpone the deletion by pushing it onto a 
queue. The free it not by node at your own leisure...


 Note that these are paraphrases of the book, not me talking. 
 And these apply equally to ARC and vanilla RC.


You don't  use RC alone, you use it with fully owned pointers, 
region pools, etc.

GC is simpler and more uniform, and that's it.



 Feb 03 2014



 next sibling parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 18:14:36 -0800, Ola Fosheim Gr=F8stad  =

<ola.fosheim.grostad+dlang gmail.com> wrote:


 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 1. RC imposes a time overhead on mutators in order to manipulate the =




 =


 counter.


 Usually you just do a retain() when your thread attain ownership (at t=


he  =


 root of the call tree). If you use embedded counters you most likely  =



 want the data in the cache anyway, but the cacheline is made dirty. So=


  =


 it cost you a write back. That affects reads of small objects more tha=


n  =


 writes.

 (The C++ implementation does not use embedded counters.)


 2. Both the counter manipulation and pointer load/store operations MU=




ST  =


 be atomic to prevent races.


 Not with transactional memory. You revert to locking, and you probably=


  =


 want that kind of synchronization anyway if you run multi threaded.


 3. Naive RC turns read ops into store ops to update the count


 If we assume naive RC, then we should assume naive GC.


Indeed D is a naive GC, but that's not the point. Naive RC incurs a  =

penalty that even Naive GC's don't.


 4. No RC can reclaim cyclic data structures, which are much more comm=




on  =


 than is typically understood. [Bacon and Rajan 2001]


 Weak pointers are sufficient to break cycles. You need a model, but yo=


u  =


 should have one anyway.


Yes, they absolutely are. However, supporting them at the automatic leve=
l  =

requires adding keywords to the language. If ARC is made the default in =
D  =

you will automatically have memory leaks where before you did not, and  =

programmer will have to carefully scan the code by hand to find everythi=
ng  =

that needs to be marked with a 'weak' keyword. It's an extremely  =

frustrating and error-prone process. The compiler cannot detect  =

cyclic-refs, it's a halting problem.


 In cases where you build temporary data structures you can just avoid =


RC  =


 altogether and just use a pool, then free the entire pool.


 5. Counter must be the same size as the pointer, which can result in =




 =


 significant overhead for small objects.





Yes, which can theoretically be as many as a 64-bit integer. In a system=
s  =

language you have to design for what is possible, not merely likely.  =

Because someday, some nutcase, somewhere is going to overflow the counte=
r,  =

and given that it's hidden by the compiler, will be VERY tricky to figur=
e  =

out. And he will have a perfectly rational reason for doing so.


 6. RC can still pause. When the last head to a large pointer structur=




e  =


 is deleted, RC MUST delete each descendant node.


 Not really, you can postpone the deletion by pushing it onto a queue. =


 =


 The free it not by node at your own leisure...


The GC Handbook deals with that idea directly: "Unfortunately, this  =

technique allows large garbage structures to be hidden by smaller ones, =
 =

and hence increases overall space requirements. [Boehm, 2004]" - The  =

Garbage Collection Handbook


 Note that these are paraphrases of the book, not me talking. And thes=




e  =


 apply equally to ARC and vanilla RC.


 You don't  use RC alone, you use it with fully owned pointers, region =


 =


 pools, etc.

 GC is simpler and more uniform, and that's it.


That may be "it", but it's a pretty big "it". Simplicity and uniformity =
 =

are VERY important.

-- =

Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



prev sibling  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 18:14:36 -0800, Ola Fosheim Gr=F8stad  =

<ola.fosheim.grostad+dlang gmail.com> wrote:


 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 1. RC imposes a time overhead on mutators in order to manipulate the =




 =


 counter.


 Usually you just do a retain() when your thread attain ownership (at t=


he  =


 root of the call tree). If you use embedded counters you most likely  =



 want the data in the cache anyway, but the cacheline is made dirty. So=


  =


 it cost you a write back. That affects reads of small objects more tha=


n  =


 writes.

 (The C++ implementation does not use embedded counters.)


There are many more ways to modify the count than a thread.


 2. Both the counter manipulation and pointer load/store operations MU=




ST  =


 be atomic to prevent races.


 Not with transactional memory. You revert to locking, and you probably=


  =


 want that kind of synchronization anyway if you run multi threaded.


Transactional memory incurs a massive perf penalty beyond even what a lo=
ck  =

would. You might be able to do better than a lock with Intel's TSX, but =
 =

that is brand new, available on Haswells only, and even then only the  =

high-end models. I doubt you'll be able to sell TM-ARC to perf conscienc=
e  =

users as the perf will most likely be much worse than a GC for the next =
 =

decade.


-- =

Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



  parent  "woh" <wojw yahoo.com>  writes:


  please this GC handbook sound good, can u post it? A queue uses 
more memory than not a queue? Fuk I did not know, I so dumb. Live 
longer more memory used, what///.


  Tuesday, 4 February 2014 at 02:33:50 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 18:14:36 -0800, Ola Fosheim Grøstad 
 <ola.fosheim.grostad+dlang gmail.com> wrote:


 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 1. RC imposes a time overhead on mutators in order to 
 manipulate the counter.


 Usually you just do a retain() when your thread attain 
 ownership (at the root of the call tree). If you use embedded 
 counters you most likely want the data in the cache anyway, 
 but the cacheline is made dirty. So it cost you a write back. 
 That affects reads of small objects more than writes.

 (The C++ implementation does not use embedded counters.)


 There are many more ways to modify the count than a thread.


 2. Both the counter manipulation and pointer load/store 
 operations MUST be atomic to prevent races.


 Not with transactional memory. You revert to locking, and you 
 probably want that kind of synchronization anyway if you run 
 multi threaded.


 Transactional memory incurs a massive perf penalty beyond even 
 what a lock would. You might be able to do better than a lock 
 with Intel's TSX, but that is brand new, available on Haswells 
 only, and even then only the high-end models. I doubt you'll be 
 able to sell TM-ARC to perf conscience users as the perf will 
 most likely be much worse than a GC for the next decade.





 Feb 03 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 5:36 PM, Adam Wilson wrote:

 You still haven't dealt with the cyclic reference problem in ARC. There
 is absolutely no way ARC can handle that without programmer input,
 therefore, it is simply not possible to switch D to ARC without adding
 some language support to deal with cyclic-refs. Ergo, it is simply not
 possible to seamlessly switch D to ARC without creating all kinds of
 havoc as people now how memory leaks where they didn't before. In order
 to support ARC the D language will necessarily have to grow/change to
 accommodate it. Apple devs constantly have trouble with cyclic-refs to
 this day.


The stock response: weak pointers. But I think the best solution is to 
allow some form of automatic reference counting backed up by the GC, 
which will lift cycles.

Andrei



 Feb 03 2014



 next sibling parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 12:57, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org

 wrote:



 On 2/3/14, 5:36 PM, Adam Wilson wrote:


 You still haven't dealt with the cyclic reference problem in ARC. There
 is absolutely no way ARC can handle that without programmer input,
 therefore, it is simply not possible to switch D to ARC without adding
 some language support to deal with cyclic-refs. Ergo, it is simply not
 possible to seamlessly switch D to ARC without creating all kinds of
 havoc as people now how memory leaks where they didn't before. In order
 to support ARC the D language will necessarily have to grow/change to
 accommodate it. Apple devs constantly have trouble with cyclic-refs to
 this day.


 The stock response: weak pointers. But I think the best solution is to
 allow some form of automatic reference counting backed up by the GC, which
 will lift cycles.



I agree, I suggested that a few times, and Adam ignored it each time.
With this approach, performance/aggressive users would be able to disable
the backing GC, and deal with cycles manually, taking responsibility for
leaking themselves.
All other users would be able to ignore the problem and have it collected
by the backing GC, business as usual.

The advantage here is *choice*. Users would then have a fully automated
system and/or have quite articulate control over it's influence on their
app.



 Feb 03 2014



prev sibling next sibling parent reply Manu <turkeyman gmail.com>  writes:


On 4 February 2014 13:31, Manu <turkeyman gmail.com> wrote:


 On 4 February 2014 12:57, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 5:36 PM, Adam Wilson wrote:


 You still haven't dealt with the cyclic reference problem in ARC. There
 is absolutely no way ARC can handle that without programmer input,
 therefore, it is simply not possible to switch D to ARC without adding
 some language support to deal with cyclic-refs. Ergo, it is simply not
 possible to seamlessly switch D to ARC without creating all kinds of
 havoc as people now how memory leaks where they didn't before. In order
 to support ARC the D language will necessarily have to grow/change to
 accommodate it. Apple devs constantly have trouble with cyclic-refs to
 this day.


 The stock response: weak pointers. But I think the best solution is to
 allow some form of automatic reference counting backed up by the GC, which
 will lift cycles.



 I agree, I suggested that a few times, and Adam ignored it each time.
 With this approach, performance/aggressive users would be able to disable
 the backing GC, and deal with cycles manually, taking responsibility for
 leaking themselves.
 All other users would be able to ignore the problem and have it collected
 by the backing GC, business as usual.

 The advantage here is *choice*. Users would then have a fully automated
 system and/or have quite articulate control over it's influence on their
 app.


Oh yeah, and the important detail, that it would apply to phobos and other
libraries.
The massive generally un-discussed problem here, is that libraries will
almost always use whatever is default and/or most convenient. I have no
control over the library code I link; if I can't influence the memory
collection routine running behind libraries I'm using, there's a good
chance I simply can't use that library.

I tend to think if we had a GC backed ARC system, then phobos would be
required to support 'gc-disabled' usage, ie, handle weak references
internally (I don't think this is too much to ask for the standard
library), making it useful to the 'gc-disabled' crowd, but the GC would
remain collecting cycles and such for all the regular D users, who don't
want to worry about such troubles. I think the majority of library authors
would then follow the phobos standard of handling weak references manually,
which would result in those libraries not being excluded from realtime use.

The majority of trivial allocations don't produce cycles; closures,
strings, temporary arrays and working sets. These are the allocations that
do occur regularly. Major allocations of complex runtime data that may
contain cycles don't happen at runtime in realtime apps, but these little
trivial/temporary allocations do (and should). Since that sort of transient
data doesn't run the risk of cycles, then it will reliably clean up
quickly, safely, and immediately under ARC, even with backing GC disabled.



 Feb 03 2014



  parent reply Michel Fortin <michel.fortin michelf.ca>  writes:


On 2014-02-04 03:45:33 +0000, Manu <turkeyman gmail.com> said:


 The majority of trivial allocations don't produce cycles; closures,
 strings, temporary arrays and working sets.


Oh, no. Beware of closures. That's a frequent problem in Objective-C 
ARC, even for those who understand ARC well. You have to be very 
careful of closures creating cycles if you use them as callbacks. For 
instance, you have a view that has a pointer to the model and sets a 
callback for the model to call when something change to update the 
view; that's a cycle and you need to use a weak ref to the view within 
the closure. Pretty common pattern.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca



 Feb 03 2014



 next sibling parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 14:19, Michel Fortin <michel.fortin michelf.ca> wrote:


 On 2014-02-04 03:45:33 +0000, Manu <turkeyman gmail.com> said:

  The majority of trivial allocations don't produce cycles; closures,

 strings, temporary arrays and working sets.


 Oh, no. Beware of closures. That's a frequent problem in Objective-C ARC,
 even for those who understand ARC well. You have to be very careful of
 closures creating cycles if you use them as callbacks. For instance, you
 have a view that has a pointer to the model and sets a callback for the
 model to call when something change to update the view; that's a cycle and
 you need to use a weak ref to the view within the closure. Pretty common
 pattern.



Ah right. Interesting.
It sounds like this could be addressed easily though by the API that
manages the 'event' handler. If you use a raw delegate, maybe it's a
problem, but it sounds like an event-programming construct, and that
usually requires an event class that can handle multiple subscribers, as
soon as that exists, it's easy to hide the weak reference behind the event
api...



 Feb 03 2014



prev sibling  parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 14:27, Manu <turkeyman gmail.com> wrote:


 On 4 February 2014 14:19, Michel Fortin <michel.fortin michelf.ca> wrote:


 On 2014-02-04 03:45:33 +0000, Manu <turkeyman gmail.com> said:

  The majority of trivial allocations don't produce cycles; closures,

 strings, temporary arrays and working sets.


 Oh, no. Beware of closures. That's a frequent problem in Objective-C ARC,
 even for those who understand ARC well. You have to be very careful of
 closures creating cycles if you use them as callbacks. For instance, you
 have a view that has a pointer to the model and sets a callback for the
 model to call when something change to update the view; that's a cycle and
 you need to use a weak ref to the view within the closure. Pretty common
 pattern.



 Ah right. Interesting.
 It sounds like this could be addressed easily though by the API that
 manages the 'event' handler. If you use a raw delegate, maybe it's a
 problem, but it sounds like an event-programming construct, and that
 usually requires an event class that can handle multiple subscribers, as
 soon as that exists, it's easy to hide the weak reference behind the event
 api...


I can say that all the closures that I've observed in my D apps so far have
been entirely trivial.



 Feb 03 2014



prev sibling next sibling parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 18:57:00 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 5:36 PM, Adam Wilson wrote:

 You still haven't dealt with the cyclic reference problem in ARC. There
 is absolutely no way ARC can handle that without programmer input,
 therefore, it is simply not possible to switch D to ARC without adding
 some language support to deal with cyclic-refs. Ergo, it is simply not
 possible to seamlessly switch D to ARC without creating all kinds of
 havoc as people now how memory leaks where they didn't before. In order
 to support ARC the D language will necessarily have to grow/change to
 accommodate it. Apple devs constantly have trouble with cyclic-refs to
 this day.


 The stock response: weak pointers. But I think the best solution is to  
 allow some form of automatic reference counting backed up by the GC,  
 which will lift cycles.

 Andrei


The immediate problem that I can see here is you're now paying for TWO GC  
algorithms. There is no traditional GC without a Mark phase (unless it's a  
copying collector, which will scare off the Embedded guys), and the mark  
phase is actually typically the longer portion of the pause. If you have  
ARC backed up by a GC you'll still have to mark+collect which means the GC  
still has to track ARC memory and then when a collection is needed, mark  
and collect. This means that you might reduce the total number of pauses,  
but you won't eliminate them. That in turn makes it an invalid tool for  
RT/Embedded purposes. And of course we still have the costs of ARC. Manu  
still can't rely on pause-free (although ARC isn't either) memory  
management, and the embedded guys still have to pay the costs in heap size  
to support the GC.

Going the other way, GC is default with ARC support on the side, is not as  
troublesome from an implementation standpoint because the GC does not have  
to be taught about the ARC memory. This means that ARC memory is free of  
being tracked by the GC and the GC has less overall memory to track which  
makes collection cycles faster. However, I don't think that the  
RT/Embedded guys will like this either, because it means you are still  
paying for the GC at some point, and they'll never know for sure if a  
library they are using is going to GC-allocate (and collect) when they  
don't expect it.

The only way I can see to make the ARC crowd happy is to completely  
replace the GC entirely, along with the attendant language changes (new  
keywords, etc) that are probably along the lines of Rust. I strongly  
believe that the reason we've never seen a GC backed ARC system is because  
in practice it doesn't completely solve any of the problems with either  
system but costs quite a bit more than either system on it's own.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



  parent reply Manu <turkeyman gmail.com>  writes:


On 4 February 2014 15:23, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 18:57:00 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

  On 2/3/14, 5:36 PM, Adam Wilson wrote:

 You still haven't dealt with the cyclic reference problem in ARC. There
 is absolutely no way ARC can handle that without programmer input,
 therefore, it is simply not possible to switch D to ARC without adding
 some language support to deal with cyclic-refs. Ergo, it is simply not
 possible to seamlessly switch D to ARC without creating all kinds of
 havoc as people now how memory leaks where they didn't before. In order
 to support ARC the D language will necessarily have to grow/change to
 accommodate it. Apple devs constantly have trouble with cyclic-refs to
 this day.


 The stock response: weak pointers. But I think the best solution is to
 allow some form of automatic reference counting backed up by the GC, which
 will lift cycles.

 Andrei


 The immediate problem that I can see here is you're now paying for TWO GC
 algorithms. There is no traditional GC without a Mark phase (unless it's a
 copying collector, which will scare off the Embedded guys), and the mark
 phase is actually typically the longer portion of the pause. If you have
 ARC backed up by a GC you'll still have to mark+collect which means the GC
 still has to track ARC memory and then when a collection is needed, mark
 and collect. This means that you might reduce the total number of pauses,
 but you won't eliminate them. That in turn makes it an invalid tool for
 RT/Embedded purposes. And of course we still have the costs of ARC. Manu
 still can't rely on pause-free (although ARC isn't either) memory
 management, and the embedded guys still have to pay the costs in heap size
 to support the GC.


So, the way I see this working in general, is that because in the majority
case, ARC would release memory immediately freeing up memory regularly, an
alloc that would have usually triggered a collect will happen far, far less
often.
Practically, this means that the mark phase, which you say is the longest
phase, would be performed far less often.

For me and my kind, I think the typical approach would be to turn off the
backing GC, and rely on marking weak references correctly.
This satisfies my requirements, and I also lose nothing in terms of
facilities in Phobos or other libraries (assuming that those libraries have
also marked weak references correctly, which I expect phobos would
absolutely be required to do).

This serves both worlds nicely, I retain access to libraries since they use
the same allocator, the GC remains (and is run less often) for those that
want care-free memory management, and for RT/embedded users, they can
*practically* disable the GC, and take responsibility for weak references
themselves, which I'm happy to do.


Going the other way, GC is default with ARC support on the side, is not as

 troublesome from an implementation standpoint because the GC does not have
 to be taught about the ARC memory. This means that ARC memory is free of
 being tracked by the GC and the GC has less overall memory to track which
 makes collection cycles faster. However, I don't think that the RT/Embedded
 guys will like this either, because it means you are still paying for the
 GC at some point, and they'll never know for sure if a library they are
 using is going to GC-allocate (and collect) when they don't expect it.


It also means that phobos and other libraries will use the GC because it's
the default. Correct, I don't see this as a valid solution. In fact, I
don't see it as a solution at all.
Where would implicit allocations like strings, concatenations, closures be
allocated?
I might as well just use RefCounted, I don't see this offering anything
much more than that.

The only way I can see to make the ARC crowd happy is to completely replace

 the GC entirely, along with the attendant language changes (new keywords,
 etc) that are probably along the lines of Rust. I strongly believe that the
 reason we've never seen a GC backed ARC system is because in practice it
 doesn't completely solve any of the problems with either system but costs
 quite a bit more than either system on it's own.



Really? [refer to my first paragraph in the reply]
It seems to me like ARC in front of a GC would result in the GC running far
less collect cycles. And the ARC opposition would be absolved of having to
tediously mark weak references. Also, the GC opposition can turn the GC
off, and everything will still work (assuming they take care of their
cycles).
I don't really see the disadvantage here, except that the
only-GC-at-all-costs-I-won't-even-consider-ARC crowd would gain a
ref-count, but they would also gain the advantage where the GC would run
less collect cycles. That would probably balance out.

I'm certainly it would be better than what we have, and in theory, everyone
would be satisfied.



 Feb 03 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 22:12:18 -0800, Manu <turkeyman gmail.com> wrote:


 On 4 February 2014 15:23, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 18:57:00 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

  On 2/3/14, 5:36 PM, Adam Wilson wrote:

 You still haven't dealt with the cyclic reference problem in ARC.  
 There
 is absolutely no way ARC can handle that without programmer input,
 therefore, it is simply not possible to switch D to ARC without adding
 some language support to deal with cyclic-refs. Ergo, it is simply not
 possible to seamlessly switch D to ARC without creating all kinds of
 havoc as people now how memory leaks where they didn't before. In  
 order
 to support ARC the D language will necessarily have to grow/change to
 accommodate it. Apple devs constantly have trouble with cyclic-refs to
 this day.


 The stock response: weak pointers. But I think the best solution is to
 allow some form of automatic reference counting backed up by the GC,  
 which
 will lift cycles.

 Andrei


 The immediate problem that I can see here is you're now paying for TWO  
 GC
 algorithms. There is no traditional GC without a Mark phase (unless  
 it's a
 copying collector, which will scare off the Embedded guys), and the mark
 phase is actually typically the longer portion of the pause. If you have
 ARC backed up by a GC you'll still have to mark+collect which means the  
 GC
 still has to track ARC memory and then when a collection is needed, mark
 and collect. This means that you might reduce the total number of  
 pauses,
 but you won't eliminate them. That in turn makes it an invalid tool for
 RT/Embedded purposes. And of course we still have the costs of ARC. Manu
 still can't rely on pause-free (although ARC isn't either) memory
 management, and the embedded guys still have to pay the costs in heap  
 size
 to support the GC.


 So, the way I see this working in general, is that because in the  
 majority
 case, ARC would release memory immediately freeing up memory regularly,  
 an
 alloc that would have usually triggered a collect will happen far, far  
 less
 often.
 Practically, this means that the mark phase, which you say is the longest
 phase, would be performed far less often.


Well, if you want the ARC memory to share the heap with the GC the ARC  
memory will need to be tracked and marked by the GC. Otherwise the GC  
might try to allocate over the top of ARC memory and vice versa. This  
means that every time you run a collection you're still marking all ARC+GC  
memory, that will induce a pause. And the GC will still STW-collect on  
random allocations, and it will still have to Mark all ARC memory to make  
sure it's still valid. So yes, there will be fewer pauses, but they will  
still be there.


 For me and my kind, I think the typical approach would be to turn off the
 backing GC, and rely on marking weak references correctly.
 This satisfies my requirements, and I also lose nothing in terms of
 facilities in Phobos or other libraries (assuming that those libraries  
 have
 also marked weak references correctly, which I expect phobos would
 absolutely be required to do).

 This serves both worlds nicely, I retain access to libraries since they  
 use
 the same allocator, the GC remains (and is run less often) for those that
 want care-free memory management, and for RT/embedded users, they can
 *practically* disable the GC, and take responsibility for weak references
 themselves, which I'm happy to do.


 Going the other way, GC is default with ARC support on the side, is not  
 as

 troublesome from an implementation standpoint because the GC does not  
 have
 to be taught about the ARC memory. This means that ARC memory is free of
 being tracked by the GC and the GC has less overall memory to track  
 which
 makes collection cycles faster. However, I don't think that the  
 RT/Embedded
 guys will like this either, because it means you are still paying for  
 the
 GC at some point, and they'll never know for sure if a library they are
 using is going to GC-allocate (and collect) when they don't expect it.


 It also means that phobos and other libraries will use the GC because  
 it's
 the default. Correct, I don't see this as a valid solution. In fact, I
 don't see it as a solution at all.
 Where would implicit allocations like strings, concatenations, closures  
 be
 allocated?
 I might as well just use RefCounted, I don't see this offering anything
 much more than that.

 The only way I can see to make the ARC crowd happy is to completely  
 replace

 the GC entirely, along with the attendant language changes (new  
 keywords,
 etc) that are probably along the lines of Rust. I strongly believe that  
 the
 reason we've never seen a GC backed ARC system is because in practice it
 doesn't completely solve any of the problems with either system but  
 costs
 quite a bit more than either system on it's own.



 Really? [refer to my first paragraph in the reply]
 It seems to me like ARC in front of a GC would result in the GC running  
 far
 less collect cycles. And the ARC opposition would be absolved of having  
 to
 tediously mark weak references. Also, the GC opposition can turn the GC
 off, and everything will still work (assuming they take care of their
 cycles).
 I don't really see the disadvantage here, except that the
 only-GC-at-all-costs-I-won't-even-consider-ARC crowd would gain a
 ref-count, but they would also gain the advantage where the GC would run
 less collect cycles. That would probably balance out.

 I'm certainly it would be better than what we have, and in theory,  
 everyone
 would be satisfied.


I'm not convinced. Mostly, because it's not likely going to be good news  
for the GC crowd. First, now there are two GC algos running unpredictably  
at different times, so while you *might* experience a perf win in ARC-only  
mode, we'll probably pay for it in GC-backed ARC mode, because you still  
have the chance at non-deterministic pause lengths with ARC and you have  
the GC pauses, and they happen at different times (GC pause on allocate,  
ARC pause on delete). Each individual pause length *might* be shorter, but  
there is no guarantee of that, but you end up paying more time on the  
whole than you would otherwise, remembering that with the GC on, the slow  
part of the collection has to be performed on all memory, not just the GC  
memory. So yes you might delete a bit less, but you're marking just as  
much, and you've still got those pesky ARC pauses to deal with. And in  
basically everything but games you measure time spent on resource  
management as a portion of CPU cycles over time, not time spent per frame.

That ref-count you hand-wave can actually cost quite a lot. Implementing  
ARC properly can have some serious perf implications on pointer-ops and  
count-ops due to the need to make sure that everything is performed  
atomically. And since this is a compiler thing, you can't say "Don't  
atomically operate here because I will never do anything that might race."  
because the compiler has to assume that at some point you will and the  
compiler cannot detect which mode it needs, or if a library will ruin your  
day. The only way you could get around this is with yet more syntax hints  
for the compiler like ' notatomic'.

Very quickly ARC starts needing a lot of specialized syntax to make it  
work efficiently. And that's not good for "Modern Convenience".

However, you don't have to perform everything atomically with a GC as the  
collect phase can always be performed concurrently on a separate thread  
and in most cases, the mark phase can do stop-the-thread instead of  
stop-the-world and in some cases, it will never stop anything at all. That  
can very easily result in pause times that are less than ARC on average.  
So now I've got a marginal improvement in the speed of ARC over the GC at  
best, and I still haven't paid for the GC.

And if we disable the GC to get the speed back we now require that  
everyone on the team learns the specialized rules and syntax for  
cyclic-refs. That might be relatively easy for someone coming from C++,  

statistically the more likely person to come to D. And indeed would be  
more than enough to stop my company moving to D.

I've seen you say more than once that you can't bond with the GC, and  
believe me I understand, if you search back through the forums, you'll  
find one of the first things I did when I got here was complain about the  
GC. But what you're saying is "I can't bond with this horrible GC so we  
need to throw it out and rebuild the compiler to support ARC." All I am  
saying is "I can't bond with the horrible GC, so why don't we make a  
better one, that doesn't ruin responsiveness, because I've seen it done in  
other places and there is no technical reason D can't do the same, or  
better." Now that I've started to read the GC Handbook I am starting to  
suspect that using D, there might be a way to create a completely  
pause-less GC. Don't hold me too it, I don't know enough yet, but D has  
some unique capabilities that might just make it possible.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



 next sibling parent reply "Araq" <rumpf_a web.de>  writes:


 I've seen you say more than once that you can't bond with the 
 GC, and believe me I understand, if you search back through the 
 forums, you'll find one of the first things I did when I got 
 here was complain about the GC. But what you're saying is "I 
 can't bond with this horrible GC so we need to throw it out and 
 rebuild the compiler to support ARC." All I am saying is "I 
 can't bond with the horrible GC, so why don't we make a better 
 one, that doesn't ruin responsiveness, because I've seen it 
 done in other places and there is no technical reason D can't 
 do the same, or better."


There are lots of technical reasons why D's GC is just the way it 

in this forum a lot of times, you simply chose to ignore them:

1. D doesn't restrict pointers, interior pointers abound 
especially thanks to the design of slices. Interior pointers are 
a problem for every high performance GC algorithm I'm aware of.
2. D's design doesn't easily allow for read or write barriers in 
general. You simply cannot make a GC behave without barriers.
3. Unions cause some (minor) problems.

In general the type system doesn't distinguish between GC'ed 
memory and manually managed memory at all.



 Now that I've started to read the GC Handbook I am starting to 
 suspect that using D, there might be a way to create a 
 completely pause-less GC. Don't hold me too it, I don't know 
 enough yet, but D has some unique capabilities that might just 
 make it possible.


That's just bullshit. The opposite is true: Of the big players D 
is easily the language that is most hostile to an efficient GC 
implementation.



 Feb 04 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 13:03:24 UTC, Araq wrote:

 1. D doesn't restrict pointers, interior pointers abound 
 especially thanks to the design of slices. Interior pointers 
 are a problem for every high performance GC algorithm I'm aware 
 of.


Interior pointers can always be replaced with a pointer + offset, 
so in the case of slices it can be resolved at the implementation 
level at some memory trade off? In the case of internal pointers 
elsewhere you would have to introduce a new type 
"pointer+offset", but that would break C-interop.


 2. D's design doesn't easily allow for read or write barriers 
 in general. You simply cannot make a GC behave without barriers.


It is difficult without affecting the C-compatible semantics. 
There are some interesting options on 64 bit CPUs where you in 
theory can do compaction to a fresh region and put hardware traps 
on the old region, rewriting pointers trying to access the old 
region or converting misses to exceptions on the fly (assuming 
you can have a barrier/isolate that prevents comparison of 
old/new pointers).

Another obvious solution is thread local GC for thread local 
memory or segmented GC (collect for only a specific pool, refuse 
pointers to escape the pool, and put a lock on that pool during 
GC).


 That's just bullshit. The opposite is true: Of the big players 
 D is easily the language that is most hostile to an efficient 
 GC implementation.


Because C is hostile to GC. So you either forsake C-compatibility 
or stay inefficient?



 Feb 04 2014



prev sibling  parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 17:50, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 22:12:18 -0800, Manu <turkeyman gmail.com> wrote:


 So, the way I see this working in general, is that because in the majority
 case, ARC would release memory immediately freeing up memory regularly, an
 alloc that would have usually triggered a collect will happen far, far
 less
 often.
 Practically, this means that the mark phase, which you say is the longest
 phase, would be performed far less often.


 Well, if you want the ARC memory to share the heap with the GC the ARC
 memory will need to be tracked and marked by the GC. Otherwise the GC might
 try to allocate over the top of ARC memory and vice versa. This means that
 every time you run a collection you're still marking all ARC+GC memory,
 that will induce a pause. And the GC will still STW-collect on random
 allocations, and it will still have to Mark all ARC memory to make sure
 it's still valid. So yes, there will be fewer pauses, but they will still
 be there.


I'm not bothered in the least. At that stage, I will have turned the GC
off, and I'll handle weak references myself. The GC crowd are then welcome
to go on and continue improving the GC in whatever way they plan to do so.

 For me and my kind, I think the typical approach would be to turn off the

 backing GC, and rely on marking weak references correctly.
 This satisfies my requirements, and I also lose nothing in terms of
 facilities in Phobos or other libraries (assuming that those libraries
 have
 also marked weak references correctly, which I expect phobos would
 absolutely be required to do).

 This serves both worlds nicely, I retain access to libraries since they
 use
 the same allocator, the GC remains (and is run less often) for those that
 want care-free memory management, and for RT/embedded users, they can
 *practically* disable the GC, and take responsibility for weak references
 themselves, which I'm happy to do.


 Going the other way, GC is default with ARC support on the side, is not as


 troublesome from an implementation standpoint because the GC does not
 have
 to be taught about the ARC memory. This means that ARC memory is free of
 being tracked by the GC and the GC has less overall memory to track which
 makes collection cycles faster. However, I don't think that the
 RT/Embedded
 guys will like this either, because it means you are still paying for the
 GC at some point, and they'll never know for sure if a library they are
 using is going to GC-allocate (and collect) when they don't expect it.


 It also means that phobos and other libraries will use the GC because it's
 the default. Correct, I don't see this as a valid solution. In fact, I
 don't see it as a solution at all.
 Where would implicit allocations like strings, concatenations, closures be
 allocated?
 I might as well just use RefCounted, I don't see this offering anything
 much more than that.

 The only way I can see to make the ARC crowd happy is to completely
 replace


 the GC entirely, along with the attendant language changes (new keywords,
 etc) that are probably along the lines of Rust. I strongly believe that
 the
 reason we've never seen a GC backed ARC system is because in practice it
 doesn't completely solve any of the problems with either system but costs
 quite a bit more than either system on it's own.



 Really? [refer to my first paragraph in the reply]
 It seems to me like ARC in front of a GC would result in the GC running
 far
 less collect cycles. And the ARC opposition would be absolved of having to
 tediously mark weak references. Also, the GC opposition can turn the GC
 off, and everything will still work (assuming they take care of their
 cycles).
 I don't really see the disadvantage here, except that the
 only-GC-at-all-costs-I-won't-even-consider-ARC crowd would gain a
 ref-count, but they would also gain the advantage where the GC would run
 less collect cycles. That would probably balance out.

 I'm certainly it would be better than what we have, and in theory,
 everyone
 would be satisfied.


 I'm not convinced. Mostly, because it's not likely going to be good news
 for the GC crowd. First, now there are two GC algos running unpredictably
 at different times, so while you *might* experience a perf win in ARC-only
 mode, we'll probably pay for it in GC-backed ARC mode, because you still
 have the chance at non-deterministic pause lengths with ARC and you have
 the GC pauses, and they happen at different times (GC pause on allocate,
 ARC pause on delete).



I don't understand. How is ARC non-deterministic? It seems entirely
deterministic to me. And if you want to, you can defer destruction to idle
time if you fancy.
Sure, the GC may pause from time to time, but you already have that anyway.
In this case, it'll run much less often.

Each individual pause length *might* be shorter, but there is no guarantee

 of that, but you end up paying more time on the whole than you would
 otherwise, remembering that with the GC on, the slow part of the collection
 has to be performed on all memory, not just the GC memory. So yes you might
 delete a bit less, but you're marking just as much, and you've still got
 those pesky ARC pauses to deal with.



Again, what ARC pauses? You mean object destruction time? Defer destruction
if you find cleaning up on the spot to be expensive. GC will always have to
scan all memory that is allocated. The fact that it's scanning ARC memory
is precisely the point (catch cycles), and no additional cost in scan load,
since that memory would all be GC memory anyway if ARC wasn't present.

And in basically everything but games you measure time spent on resource

 management as a portion of CPU cycles over time, not time spent per frame.


I suspect that spending less time doing GC scan's will result in a win
overall. I have nothing to back that up, but it's a strong suspicion.
Object destruction, which you seem to have a problem with under ARC, still
happens even with a GC, just at some unknown time. It's not clear to me
what the additional ARC cost is (other than the obvious inc and dec)...
except it facilitates spending less time doing GC collection, which is
probably a significant saving.

That ref-count you hand-wave can actually cost quite a lot. Implementing

 ARC properly can have some serious perf implications on pointer-ops and
 count-ops due to the need to make sure that everything is performed
 atomically.



I don't think this is true. There's no need to perform the ref fiddling
with atomic operations unless it's shared.
Everyone expects additional costs for synchronisation of shared objects.

And since this is a compiler thing, you can't say "Don't atomically operate

 here because I will never do anything that might race." because the
 compiler has to assume that at some point you will and the compiler cannot
 detect which mode it needs, or if a library will ruin your day. The only
 way you could get around this is with yet more syntax hints for the
 compiler like ' notatomic'.


Ummm. I think D makes a clear assumption that if something isn't marked
shared, that it doesn't have to compile code to protect against races.
That's the whole point of making shared an explicit attribute.
What you say is true in C++ which can't distinguish the cases, I don't
think it applies in D.

Very quickly ARC starts needing a lot of specialized syntax to make it work

 efficiently. And that's not good for "Modern Convenience".


Other than a weak attribute, what does it need? I'm not aware of anything
else.

However, you don't have to perform everything atomically with a GC as the

 collect phase can always be performed concurrently on a separate thread and
 in most cases,



You don't have to perform everything atomically in ARC, and the GC is
definitely like that now.

the mark phase can do stop-the-thread instead of stop-the-world and in some

 cases, it will never stop anything at all.



If I only have one core?
ARC doesn't need to mark at all, that cost is effectively distributed among
inc/dec ref's, and I'm confident ++ and -- operations performed only on
relevant data and only when it's copied is much cheaper than the GC
scanning the whole heap, and applying all that logic to determine what
things are pointers that it needs to follow and what not.

That can very easily result in pause times that are less than ARC on

 average. So now I've got a marginal improvement in the speed of ARC over
 the GC at best, and I still haven't paid for the GC.


What pause does the ARC produce?

Are you paying an ambient cost for the GC? When the ARC is doing it's job,
the GC wouldn't run. When too many un-handled cycles add up, the GC might
run a scan. If/when the GC does run a scan, it's precisely because you
_haven't_ already paid the cost via the ARC; it missed it due to cycle, no
cost was paid, no time was lost, it was just collected by the GC instead. I
don't see how you can sum the costs of the 2 collection mechanisms in this
case.
Either the ARC cleans it up, and the GC doesn't. Or the GC cleans it up
because the ARC didn't.

ARC destruction can easily be deferred, and unlike a mark phase which MUST
be executed entirely in one step, it is possible to process deferred ARC
object destruction at leisure, using only idle time for instance, and
arbitrary lengths of time are easily supported.

And if we disable the GC to get the speed back


I still don't follow, we never lost anything, we only moved it.

we now require that everyone on the team learns the specialized rules and

 syntax for cyclic-refs. That might be relatively easy for someone coming

 which is statistically the more likely person to come to D. And indeed
 would be more than enough to stop my company moving to D.


Don't turn the GC off. Indeed, that would be silly for most applications.
You need to clarify how moving some collection cost from the GC to ARC
makes it more expensive? I still can't see it. As far as I can see,
everything the ARC cleans up is burden lifted from the GC, it could only
result in the GC running less often, and ARC is not by nature more
expensive than GC. I suspect ARC has a lower net cost, since ++/--, only on
relevant things, and only when they're copied, is probably a lot less
complicated work than a full mark phase, which touches everything, and
follows many pointers, mostly unnecessarily. GC mark phase is quite a large
workload, increases proportionally to the working data set, and it's also
an absolute dcache disaster. No way inc/dec could compare to that workload,
particularly as the heap grows large or nears capacity.

I've seen you say more than once that you can't bond with the GC, and

 believe me I understand, if you search back through the forums, you'll find
 one of the first things I did when I got here was complain about the GC.
 But what you're saying is "I can't bond with this horrible GC so we need to
 throw it out and rebuild the compiler to support ARC." All I am saying is
 "I can't bond with the horrible GC, so why don't we make a better one, that
 doesn't ruin responsiveness, because I've seen it done in other places and
 there is no technical reason D can't do the same, or better." Now that I've
 started to read the GC Handbook I am starting to suspect that using D,
 there might be a way to create a completely pause-less GC. Don't hold me
 too it, I don't know enough yet, but D has some unique capabilities that
 might just make it possible.



Well when you know, I'm all ears. Truly, I am. But I can't imagine a GC
that will work acceptably in environments such as limited memory, realtime,
single core, or various combinations of those.
I also get the feeling you haven't thought through the cost distribution of
a GC backed ARC solution. Either that, or I haven't done my math correctly,
which I'd be happy to have demonstrated where I went wrong.



 Feb 04 2014



prev sibling  parent  "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 4 February 2014 at 02:57:00 UTC, Andrei Alexandrescu 
wrote:

 On 2/3/14, 5:36 PM, Adam Wilson wrote:

 You still haven't dealt with the cyclic reference problem in 
 ARC. There
 is absolutely no way ARC can handle that without programmer 
 input,
 therefore, it is simply not possible to switch D to ARC 
 without adding
 some language support to deal with cyclic-refs. Ergo, it is 
 simply not
 possible to seamlessly switch D to ARC without creating all 
 kinds of
 havoc as people now how memory leaks where they didn't before. 
 In order
 to support ARC the D language will necessarily have to 
 grow/change to
 accommodate it. Apple devs constantly have trouble with 
 cyclic-refs to
 this day.


 The stock response: weak pointers. But I think the best 
 solution is to allow some form of automatic reference counting 
 backed up by the GC, which will lift cycles.

 Andrei


This is the approach taken by Cedar on the Mesa system.

"On Adding Garbage Collection and
Runtime Types to a Strongly-Typed,
Statically-Checked, Concurrent Language"

http://www.textfiles.com/bitsavers/pdf/xerox/parc/techReports/CSL-84-7_On_Adding_Garbage_Collection_and_Runtime_Types_to_a_Strongly-Typed_Statically-Checked_Concurrent_Language.pdf

--
Paulo



 Feb 04 2014



prev sibling next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 5:36 PM, Adam Wilson wrote:

 Boehm demonstrated in one of his papers (2004) that thread-safe ARC may
 even lead to longer maximum pause times than a standard Tracing GC.


At least this paper claims RC/GC parity has been reached:

http://research.microsoft.com/pubs/202163/rcix-oopsla-2013.pdf


Andrei



 Feb 03 2014



prev sibling  parent reply "ed" <growlercab gmail.com>  writes:


On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu <turkeyman gmail.com> 
 wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> 
 wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to 
 D, gets
 pretty much ignored. The topic is "Smart pointers instead 
 of GC?",
 remember? People here seem to be more interested in 
 diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes 
 considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than 
 complete removal of
 the GC and replacement with compiler generated ARC will be 
 unacceptable to
 a certain, highly vocal, subset of D users. No arguments can 
 be made to
 otherwise, regardless of validity. As far as they are 
 concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the 
 only path
 forward to the bright and glorious future of D. ARC most 
 efficiently solves
 all memory management problems ever encountered. 
 Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!

 Sadly, although written as hyperbole, I feel that the above 
 is fairly
 close to the actual position of the ARC crowd.



 Don't be a dick.
 I get the impression you don't actually read my posts. And I 
 also feel like
 you're a lot more dogmatic about this than you think I am.

 I'm absolutely fine with GC in most applications, I really 
 couldn't give
 any shits if most people want a GC. I'm not dogmatic about it, 
 and I've
 **honestly** tried to love the GC for years now.
 What I'm concerned about is that I have _no option_ to use D 
 uninhibited
 when I need to not have the GC.

 These are the problems:
 * GC stalls for long periods time at completely un-predictable 
 moments.
 * GC stalls become longer *and* more frequent as memory 
 becomes less
 available, and the working pool becomes larger (what a 
 coincidence).
 * Memory footprint is unknowable, what if you don't have a 
 virtual memory
 manager? What if your total memory is measured in megabytes?
 * It's not possible to know when destruction of an object will 
 happen,

 in many
 cases, and supports the prior point.

 Conclusion:
  GC is unfit for embedded systems. One of the most significant 
 remaining
 and compelling uses for a native systems language.

 The only realistic path I am aware of is to use ARC, which IS 
 a form of GC,
 and allows a lot more flexibility in the front-end.
 GC forces one very particular paradigm upon you.
 ARC is a GC, but it has some complex properties __which can be 
 addressed in
 various ways__. Unlike a GC which is entirely inflexible.

 You're not happy with ARC's cleaning objects up on the spot? 
 Something that
 many people WANT, but I understand zero cleanup times in the 
 running
 context is in other occasions a strength of GC; fine, just 
 stick the
 pointer on a dead list, and free it either later during idle 
 time, or on
 another thread. On the contrary, I haven't heard any proposal 
 for a GC that
 would allow it to operate in carefully controlled time-slices, 
 or strictly
 during idle-time.
 Cycles are a problem with ARC? True, how much effort are you 
 willing to
 spend to mitigate the problem? None: run a secondary GC in the 
 background
 to collect cycles (yes, there is still a GC, but it has much 
 less work to
 do). Some: Disable background GC, manually require user 
 specified weak
 references and stuff. Note: A user-preferred combination of 
 the 2 could
 severely mitigate the workload of the background GC if it is 
 still desired
 to handle some complex situations, or user errors.
 Are there any other disadvantages to ARC? I don't know of them 
 if there are.

 Is far as I can tell, an ARC collector could provide identical 
 convenience
 as the existing GC for anyone that simply doesn't care. It 
 would also seem
 that it could provide significantly more options and control 
 for those that
 do.

 I am _yet to hear anyone present a realistic path forwards 
 using any form
 of GC_, so what else do I have to go with? Until I know of any 
 other path
 forward, I'll stand behind the only one I can see.
 You're just repeating "I don't care about something that a 
 significant
 subset of D developers do care about, and I don't think any 
 changes should
 be made to support them".
 As far as I know, a switch to ARC could be done in a way that 
 'regular'
 users don't lose anything, or even notice... why is that so 
 offensive?


 I am not trying to be a dick. But I do feel like a small number 
 of people are trying to gang up on me for daring to point out 
 that the solution they've proposed solution might have bigger 
 problems for other people than they care to admit.

 You still haven't dealt with the cyclic reference problem in 
 ARC. There is absolutely no way ARC can handle that without 
 programmer input, therefore, it is simply not possible to 
 switch D to ARC without adding some language support to deal 
 with cyclic-refs. Ergo, it is simply not possible to seamlessly 
 switch D to ARC without creating all kinds of havoc as people 
 now how memory leaks where they didn't before. In order to 
 support ARC the D language will necessarily have to grow/change 
 to accommodate it. Apple devs constantly have trouble with 
 cyclic-refs to this day.

 I am not against supporting ARC side-by-side with the GC (I'm 
 actually quite for it, I would love the flexibility), but it is 
 unrealistic to make ARC the default option in D as that would 
 subtly break all existing D code, something that Walter has 
 point-blank refused to do in much smaller easier to find+fix 
 cases. You can't grep for a weak-ref. So if that is what you 
 are asking for, then yes, it will never happen in D.

 Also, I don't think you've fully considered what the perf 
 penalty actually is for a *good* ARC implementation. I just 
 leafed through the P-Code in the GC Handbook for their ARC 
 implementation, it's about 4x longer than what their best 
 P-Code Mark-Sweep implementation is.

 I would also like to point out that the GC Handbook points out 
 six scientifically confirmed problems with ARC. (See Page 59)

 1. RC imposes a time overhead on mutators in order to 
 manipulate the counter.
 2. Both the counter manipulation and pointer load/store 
 operations MUST be atomic to prevent races.
 3. Naive RC turns read ops into store ops to update the count.
 4. No RC can reclaim cyclic data structures, which are much 
 more common than is typically understood. [Bacon and Rajan 2001]
 5. Counter must be the same size as the pointer, which can 
 result in significant overhead for small objects.
 6. RC can still pause. When the last head to a large pointer 
 structure is deleted, RC MUST delete each descendant node.

 Note that these are paraphrases of the book, not me talking. 
 And these apply equally to ARC and vanilla RC.

 Boehm demonstrated in one of his papers (2004) that thread-safe 
 ARC may even lead to longer maximum pause times than a standard 
 Tracing GC.


Most of us know and understand the issues with ARC and that with 
a GC. Many of us have seen how they play out in systems level 
development. There is a good reason all serious driver and 
embedded development is done in C/C++.

A language is the compiler+std as one unit. If Phobos depends on 
the GC, D depends on the GC. If Phobos isn't systems level ready, 
D isn't systems level ready. I've heard arguments here that you 
can turn off the GC, but that equates to rewriting functions that 
already exists in Phobos and not using any third-party library. 
Why would anyone seriously consider that as an option? Embedded 
C++ has std:: and third-party libraries where memory is under 
control?

Realistically D as a systems language isn't even at the hobby 
stage. I understand people are working in this area, as am I. 
Eagerly testing and trying to contribute where possible. That's 
the fun part of D, watching and helping it mature into everything 
it can be.

Cheers,
ed



 Feb 03 2014



  parent reply "Don" <x nospam.com>  writes:


On Tuesday, 4 February 2014 at 03:43:53 UTC, ed wrote:

 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu <turkeyman gmail.com> 
 wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> 
 wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers 
 to D, gets
 pretty much ignored. The topic is "Smart pointers instead 
 of GC?",
 remember? People here seem to be more interested in 
 diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes 
 considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than 
 complete removal of
 the GC and replacement with compiler generated ARC will be 
 unacceptable to
 a certain, highly vocal, subset of D users. No arguments can 
 be made to
 otherwise, regardless of validity. As far as they are 
 concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the 
 only path
 forward to the bright and glorious future of D. ARC most 
 efficiently solves
 all memory management problems ever encountered. 
 Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!









 Most of us know and understand the issues with ARC and that 
 with a GC. Many of us have seen how they play out in systems 
 level development. There is a good reason all serious driver 
 and embedded development is done in C/C++.

 A language is the compiler+std as one unit. If Phobos depends 
 on the GC, D depends on the GC. If Phobos isn't systems level 
 ready, D isn't systems level ready. I've heard arguments here 
 that you can turn off the GC, but that equates to rewriting 
 functions that already exists in Phobos and not using any 
 third-party library.


At Sociomantic, that is exactly what we have done. Phobos is 
almost completely unusable at the present time.

I personally don't think that ARC would make much difference. The 
problem is that *far* too much garbage is being created. And it's 
completely unnecessary in most cases.

To take an extreme example, even a pauseless, perfect GC, 
wouldn't make std.json acceptable.



 Why would anyone seriously consider that as an option? Embedded 
 C++ has std:: and third-party libraries where memory is under 
 control?

 Realistically D as a systems language isn't even at the hobby 
 stage.


We're using D as a systems language on a global commercial scale. 
And no, we don't use malloc/free. We just don't use Phobos.



 Feb 04 2014



 next sibling parent  "ed" <growlercab gmail.com>  writes:


On Tuesday, 4 February 2014 at 09:59:07 UTC, Don wrote:

 On Tuesday, 4 February 2014 at 03:43:53 UTC, ed wrote:

 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu 
 <turkeyman gmail.com> wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> 
 wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers 
 to D, gets
 pretty much ignored. The topic is "Smart pointers instead 
 of GC?",
 remember? People here seem to be more interested in 
 diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which 
 includes considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than 
 complete removal of
 the GC and replacement with compiler generated ARC will be 
 unacceptable to
 a certain, highly vocal, subset of D users. No arguments 
 can be made to
 otherwise, regardless of validity. As far as they are 
 concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the 
 only path
 forward to the bright and glorious future of D. ARC most 
 efficiently solves
 all memory management problems ever encountered. 
 Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!









 Most of us know and understand the issues with ARC and that 
 with a GC. Many of us have seen how they play out in systems 
 level development. There is a good reason all serious driver 
 and embedded development is done in C/C++.

 A language is the compiler+std as one unit. If Phobos depends 
 on the GC, D depends on the GC. If Phobos isn't systems level 
 ready, D isn't systems level ready. I've heard arguments here 
 that you can turn off the GC, but that equates to rewriting 
 functions that already exists in Phobos and not using any 
 third-party library.


 At Sociomantic, that is exactly what we have done. Phobos is 
 almost completely unusable at the present time.

 I personally don't think that ARC would make much difference. 
 The problem is that *far* too much garbage is being created. 
 And it's completely unnecessary in most cases.

 To take an extreme example, even a pauseless, perfect GC, 
 wouldn't make std.json acceptable.



 Why would anyone seriously consider that as an option? 
 Embedded C++ has std:: and third-party libraries where memory 
 is under control?

 Realistically D as a systems language isn't even at the hobby 
 stage.


 We're using D as a systems language on a global commercial 
 scale. And no, we don't use malloc/free. We just don't use 
 Phobos.


OK, I stand corrected! :D

Thanks,
ed



 Feb 04 2014



prev sibling next sibling parent reply "ed" <growlercab gmail.com>  writes:


On Tuesday, 4 February 2014 at 09:59:07 UTC, Don wrote:

 On Tuesday, 4 February 2014 at 03:43:53 UTC, ed wrote:

 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu 
 <turkeyman gmail.com> wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> 
 wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers 
 to D, gets
 pretty much ignored. The topic is "Smart pointers instead 
 of GC?",
 remember? People here seem to be more interested in 
 diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which 
 includes considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than 
 complete removal of
 the GC and replacement with compiler generated ARC will be 
 unacceptable to
 a certain, highly vocal, subset of D users. No arguments 
 can be made to
 otherwise, regardless of validity. As far as they are 
 concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the 
 only path
 forward to the bright and glorious future of D. ARC most 
 efficiently solves
 all memory management problems ever encountered. 
 Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!









 Most of us know and understand the issues with ARC and that 
 with a GC. Many of us have seen how they play out in systems 
 level development. There is a good reason all serious driver 
 and embedded development is done in C/C++.

 A language is the compiler+std as one unit. If Phobos depends 
 on the GC, D depends on the GC. If Phobos isn't systems level 
 ready, D isn't systems level ready. I've heard arguments here 
 that you can turn off the GC, but that equates to rewriting 
 functions that already exists in Phobos and not using any 
 third-party library.


 At Sociomantic, that is exactly what we have done. Phobos is 
 almost completely unusable at the present time.

 I personally don't think that ARC would make much difference. 
 The problem is that *far* too much garbage is being created. 
 And it's completely unnecessary in most cases.

 To take an extreme example, even a pauseless, perfect GC, 
 wouldn't make std.json acceptable.



 Why would anyone seriously consider that as an option? 
 Embedded C++ has std:: and third-party libraries where memory 
 is under control?

 Realistically D as a systems language isn't even at the hobby 
 stage.


 We're using D as a systems language on a global commercial 
 scale. And no, we don't use malloc/free. We just don't use 
 Phobos.


Maybe I'm thinking too much embedded, which I admit isn't fair on 
D and at this stage maybe not a realistic comparison.

Our projects are Siemens medical devices so it is 90% embedded, a 
different level of system perhaps. They too would be on a global 
scale and I'd love to get D on them :-)

Cheers,
ed



 Feb 04 2014



  parent reply "Don" <x nospam.com>  writes:


On Tuesday, 4 February 2014 at 10:32:26 UTC, ed wrote:

 On Tuesday, 4 February 2014 at 09:59:07 UTC, Don wrote:

 On Tuesday, 4 February 2014 at 03:43:53 UTC, ed wrote:

 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson 
 wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu 
 <turkeyman gmail.com> wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> 
 wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers 
 to D, gets
 pretty much ignored. The topic is "Smart pointers 
 instead of GC?",
 remember? People here seem to be more interested in 
 diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which 
 includes considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than 
 complete removal of
 the GC and replacement with compiler generated ARC will be 
 unacceptable to
 a certain, highly vocal, subset of D users. No arguments 
 can be made to
 otherwise, regardless of validity. As far as they are 
 concerned the
 discussion of ARC vs. GC is closed and decided. ARC is the 
 only path
 forward to the bright and glorious future of D. ARC most 
 efficiently solves
 all memory management problems ever encountered. 
 Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!









 Most of us know and understand the issues with ARC and that 
 with a GC. Many of us have seen how they play out in systems 
 level development. There is a good reason all serious driver 
 and embedded development is done in C/C++.

 A language is the compiler+std as one unit. If Phobos depends 
 on the GC, D depends on the GC. If Phobos isn't systems level 
 ready, D isn't systems level ready. I've heard arguments here 
 that you can turn off the GC, but that equates to rewriting 
 functions that already exists in Phobos and not using any 
 third-party library.


 At Sociomantic, that is exactly what we have done. Phobos is 
 almost completely unusable at the present time.

 I personally don't think that ARC would make much difference. 
 The problem is that *far* too much garbage is being created. 
 And it's completely unnecessary in most cases.

 To take an extreme example, even a pauseless, perfect GC, 
 wouldn't make std.json acceptable.



 Why would anyone seriously consider that as an option? 
 Embedded C++ has std:: and third-party libraries where memory 
 is under control?

 Realistically D as a systems language isn't even at the hobby 
 stage.


 We're using D as a systems language on a global commercial 
 scale. And no, we don't use malloc/free. We just don't use 
 Phobos.


 Maybe I'm thinking too much embedded, which I admit isn't fair 
 on D and at this stage maybe not a realistic comparison.


Yeah, I dunno what "systems language" means really. In practice 
it seems to mean "competes with C++" and that's how I use it. And 
C++ had some problems getting into the embedded market.

Though even D as "a better C" is a surprisingly nice language.



 Our projects are Siemens medical devices so it is 90% embedded, 
 a different level of system perhaps. They too would be on a 
 global scale and I'd love to get D on them :-)


Yeah. I can imagine that's a tough assignment.



 Feb 04 2014



  parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 4 February 2014 at 11:13:03 UTC, Don wrote:

 On Tuesday, 4 February 2014 at 10:32:26 UTC, ed wrote:

 On Tuesday, 4 February 2014 at 09:59:07 UTC, Don wrote:

 On Tuesday, 4 February 2014 at 03:43:53 UTC, ed wrote:

 On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson 
 wrote:

 On Mon, 03 Feb 2014 17:04:08 -0800, Manu 
 <turkeyman gmail.com> wrote:


 On 4 February 2014 06:21, Adam Wilson <flyboynw gmail.com> 
 wrote:


 On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning 
 pointers to D, gets
 pretty much ignored. The topic is "Smart pointers 
 instead of GC?",
 remember? People here seem to be more interested in 
 diverting to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which 
 includes considering
 built-in reference counting) is a major focus of 2014.

 Andrei


 Andrei, I am sorry to report that anything other than 
 complete removal of
 the GC and replacement with compiler generated ARC will 
 be unacceptable to
 a certain, highly vocal, subset of D users. No arguments 
 can be made to
 otherwise, regardless of validity. As far as they are 
 concerned the
 discussion of ARC vs. GC is closed and decided. ARC is 
 the only path
 forward to the bright and glorious future of D. ARC most 
 efficiently solves
 all memory management problems ever encountered. 
 Peer-Reviewed Research and
 the Scientific Method be damned! ALL HAIL ARC!









 Most of us know and understand the issues with ARC and that 
 with a GC. Many of us have seen how they play out in systems 
 level development. There is a good reason all serious driver 
 and embedded development is done in C/C++.

 A language is the compiler+std as one unit. If Phobos 
 depends on the GC, D depends on the GC. If Phobos isn't 
 systems level ready, D isn't systems level ready. I've heard 
 arguments here that you can turn off the GC, but that 
 equates to rewriting functions that already exists in Phobos 
 and not using any third-party library.


 At Sociomantic, that is exactly what we have done. Phobos is 
 almost completely unusable at the present time.

 I personally don't think that ARC would make much difference. 
 The problem is that *far* too much garbage is being created. 
 And it's completely unnecessary in most cases.

 To take an extreme example, even a pauseless, perfect GC, 
 wouldn't make std.json acceptable.



 Why would anyone seriously consider that as an option? 
 Embedded C++ has std:: and third-party libraries where 
 memory is under control?

 Realistically D as a systems language isn't even at the 
 hobby stage.


 We're using D as a systems language on a global commercial 
 scale. And no, we don't use malloc/free. We just don't use 
 Phobos.


 Maybe I'm thinking too much embedded, which I admit isn't fair 
 on D and at this stage maybe not a realistic comparison.


 Yeah, I dunno what "systems language" means really.


For me it means you can write an OS with it, even if some tiny 
parts require the use of Assembly glue.

--
Paulo



 Feb 04 2014



 next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 11:57:25 UTC, Paulo Pinto wrote:

 On Tuesday, 4 February 2014 at 11:13:03 UTC, Don wrote:



 Yeah, I dunno what "systems language" means really.


 For me it means you can write an OS with it, even if some tiny 
 parts require the use of Assembly glue.



Pretty close to my definition, but I would add:

- you should be able to write a performant OS with it
- you should be able to use all core language features when doing 
so
- the mapping to hardware should be obvious
- thus the runtime should be transparent/easy to grasp

Basically I think a system level language should provide a 
minimal library and runtime that you can extend to a OS-level 
library that you use for implementing all services on your OS.

I think GC+Phobos should be an extension of such a minimal set up.

---

I don't think ARC is needed for D, because I think you use RC 
only when needed when writing latency sensitive code and use 
primarily other strategies (unique pointers, embedded objects 
etc).

I do think that interop with C++ is a good strategy. Language 
level support for C++11 pointer types would be a good addition 
now that they are being used quite extensively. Interop with C++ 
is more important than Objective-C.

Having a system library core that is GC free would be nice. I 
think this should be done on the name space level. I.e. having 
one optimized "core" section that is 100% GC free and one more 
extensive "std" section that requires GC to be available.

I don't think it is realistic to have all libraries being non-GC. 
It is better to have useful libraries that are somewhat 
inefficient than not having them due to complexity issue.

It is better to have a very easy to use and extensible to XML DOM 
library than to have a more obfuscated and less extensible XML 
DOM library without GC. But the XML parser library should be GC 
free.



 Feb 04 2014



  parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 4 February 2014 at 12:46:58 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 4 February 2014 at 11:57:25 UTC, Paulo Pinto wrote:

 On Tuesday, 4 February 2014 at 11:13:03 UTC, Don wrote:



 Yeah, I dunno what "systems language" means really.


 For me it means you can write an OS with it, even if some tiny 
 parts require the use of Assembly glue.



 Pretty close to my definition, but I would add:

 - you should be able to write a performant OS with it
 - you should be able to use all core language features when 
 doing so
 - the mapping to hardware should be obvious
 - thus the runtime should be transparent/easy to grasp

 Basically I think a system level language should provide a 
 minimal library and runtime that you can extend to a OS-level 
 library that you use for implementing all services on your OS.

 I think GC+Phobos should be an extension of such a minimal set 
 up.

 ---

 I don't think ARC is needed for D, because I think you use RC 
 only when needed when writing latency sensitive code and use 
 primarily other strategies (unique pointers, embedded objects 
 etc).

 I do think that interop with C++ is a good strategy. Language 
 level support for C++11 pointer types would be a good addition 
 now that they are being used quite extensively. Interop with 
 C++ is more important than Objective-C.

 Having a system library core that is GC free would be nice. I 
 think this should be done on the name space level. I.e. having 
 one optimized "core" section that is 100% GC free and one more 
 extensive "std" section that requires GC to be available.

 I don't think it is realistic to have all libraries being 
 non-GC. It is better to have useful libraries that are somewhat 
 inefficient than not having them due to complexity issue.

 It is better to have a very easy to use and extensible to XML 
 DOM library than to have a more obfuscated and less extensible 
 XML DOM library without GC. But the XML parser library should 
 be GC free.



In Oberon and Modula-3's control over GC tends to be a  bit 
easier than D's current design, because of a few points:

- No implicit allocations outside NEW
- No closures
- You can use untraced pointers in unsafe modules.

However it would be nice to know if anyone here has real life 
experience with Lisp Machines and how was their performance.

--
Paulo



 Feb 04 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 13:52:38 UTC, Paulo Pinto wrote:

 - No implicit allocations outside NEW
 - No closures


True, on the OS level, in drivers and in real time call backs you 
want everything to be explicit.


 - You can use untraced pointers in unsafe modules.


I assume it would be possible to add  notrace as an attribute to 
pointers/references in D so that you can avoid tracing objects 
that are known to be in the pool of traced objects?

The best would be to have "notrace-pointer-types" and programmer 
specified "trace functions" that are more optimal. E.g. keeping a 
list of all active Nodes and have your own function for scanning 
it for outgoing pointers, then keep all pointers to Nodes 
non-traceable.



 Feb 04 2014



prev sibling  parent  "Don" <x nospam.com>  writes:


On Tuesday, 4 February 2014 at 11:57:25 UTC, Paulo Pinto wrote:

 On Tuesday, 4 February 2014 at 11:13:03 UTC, Don wrote:

 On Tuesday, 4 February 2014 at 10:32:26 UTC, ed wrote:

 Realistically D as a systems language isn't even at the 
 hobby stage.


 We're using D as a systems language on a global commercial 
 scale. And no, we don't use malloc/free. We just don't use 
 Phobos.


 Maybe I'm thinking too much embedded, which I admit isn't 
 fair on D and at this stage maybe not a realistic comparison.


 Yeah, I dunno what "systems language" means really.


 For me it means you can write an OS with it, even if some tiny 
 parts require the use of Assembly glue.


I know that's the *origin* of the term, but I don't think that's 
the *meaning*.

I mean, that's a really obscure task. People almost never write a 
new OS, and languages have claimed to be systems languages 
without ever having done it.

AFAICT what it really means is "like C" in some vaguely-defined 
sense.



 Feb 04 2014



prev sibling next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/4/14, 1:59 AM, Don wrote:

 We're using D as a systems language on a global commercial scale. And
 no, we don't use malloc/free. We just don't use Phobos.


What do you use instead of malloc/free?

Andrei



 Feb 04 2014



prev sibling next sibling parent  "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 4 February 2014 at 09:59:07 UTC, Don wrote:

 We're using D as a systems language on a global commercial 
 scale. And no, we don't use malloc/free. We just don't use 
 Phobos.


With all respect I don't consider Sociomantic projects examples 
of systems-level programming (now that I have had a chance to 
inspect those closely ;)). Soft real-time performance critical 
system is a better description.



 Feb 04 2014



prev sibling  parent reply Manu <turkeyman gmail.com>  writes:


On 4 February 2014 19:59, Don <x nospam.com> wrote:


 On Tuesday, 4 February 2014 at 03:43:53 UTC, ed wrote:


 Most of us know and understand the issues with ARC and that with a GC.
 Many of us have seen how they play out in systems level development. There
 is a good reason all serious driver and embedded development is done in
 C/C++.

 A language is the compiler+std as one unit. If Phobos depends on the GC,
 D depends on the GC. If Phobos isn't systems level ready, D isn't systems
 level ready. I've heard arguments here that you can turn off the GC, but
 that equates to rewriting functions that already exists in Phobos and not
 using any third-party library.


 At Sociomantic, that is exactly what we have done. Phobos is almost
 completely unusable at the present time.

 I personally don't think that ARC would make much difference. The problem
 is that *far* too much garbage is being created. And it's completely
 unnecessary in most cases.


I agree here. A new collector can't distract from the task of reducing the
amount of garbage produced in the first place.

I've never suggested ARC will make a wild difference in terms of
performance in the standard use case, that's not the point (although I do
imagine it would be faster).
I'm saying that ARC is not fundamentally incompatible with many kinds of
workloads, and offers the application a level of flexibility that's not
available under a GC alone. It's an enabler for some whole new industries
to use D with confidence.



 Feb 04 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Wednesday, 5 February 2014 at 02:00:00 UTC, Manu wrote:

 I'm saying that ARC is not fundamentally incompatible with many 
 kinds of
 workloads, and offers the application a level of flexibility 
 that's not
 available under a GC alone. It's an enabler for some whole new 
 industries
 to use D with confidence.


That's true, ARC will improve latencies. There is a reason for 
why languages like Python, Perl and Php up till now have been 
primarily ARC based, but are moving more towards GC as CPU speed 
has increased. The freeze-the-world time GC provides was 
unacceptable 15-20 years ago, even for non-realtime stuff.

It is also true that ARC+GC will make the GC run very seldom, 
perhaps never, depending on the thresholds you set up and the 
patterns you use.



 Feb 04 2014



prev sibling  parent reply Jerry <jlquinn optonline.net>  writes:


Manu <turkeyman gmail.com> writes:


 These are the problems:
  * GC stalls for long periods time at completely un-predictable moments.
  * GC stalls become longer *and* more frequent as memory becomes less
 available, and the working pool becomes larger (what a coincidence).
  * Memory footprint is unknowable, what if you don't have a virtual memory
 manager? What if your total memory is measured in megabytes?
  * It's not possible to know when destruction of an object will happen, which

 supports the prior point.

 Conclusion:
   GC is unfit for embedded systems. One of the most significant remaining and
 compelling uses for a native systems language.


Hi Manu,

Doing a bit of searching, I came across the Metronome GC that IBM came
up with.  It appears to try to address the problems you raise:

http://researcher.watson.ibm.com/researcher/view_project_subpage.php?id=175

Any thoughts?

Jerry



 Feb 03 2014



  parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 15:22, Jerry <jlquinn optonline.net> wrote:


 Manu <turkeyman gmail.com> writes:


 These are the problems:
  * GC stalls for long periods time at completely un-predictable moments.
  * GC stalls become longer *and* more frequent as memory becomes less
 available, and the working pool becomes larger (what a coincidence).
  * Memory footprint is unknowable, what if you don't have a virtual


 memory

 manager? What if your total memory is measured in megabytes?
  * It's not possible to know when destruction of an object will happen,


 which




 and

 supports the prior point.

 Conclusion:
   GC is unfit for embedded systems. One of the most significant


 remaining and

 compelling uses for a native systems language.


 Hi Manu,

 Doing a bit of searching, I came across the Metronome GC that IBM came
 up with.  It appears to try to address the problems you raise:

 http://researcher.watson.ibm.com/researcher/view_project_subpage.php?id=175

 Any thoughts?


Interesting. I'll read :)
I know nothing about it, so I can't really comment. And I'm also not an
authority on what can/can't be done in terms of GC implementation in the
context of D.



 Feb 03 2014



prev sibling next sibling parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Monday, 3 February 2014 at 20:02:30 UTC, Andrei Alexandrescu 
wrote:

 I thought I made it clear that GC avoidance (which includes 
 considering built-in reference counting) is a major focus of 
 2014.


Please make it so that you can easily write thread-level 
allocator agnostic code without passing the allocator around. 
E.g. so that you can start writing code with GC, but properly 
release memory, then when you are certain that your model does 
not introduce cycles replace it with a cheaper allocation model 
with no additional code changes.



 Feb 03 2014



prev sibling next sibling parent reply "Frank Bauer" <y z.com>  writes:


Andrei Alexandrescu wrote:


 2. Work on Phobos to see what can be done about avoiding 
 unnecessary allocation. Most likely we'll need to also add a 
  nogc flag.

 ...

 4. Work on the core language and druntime to see how to 
 seamlessly accommodate alternate GC mechanisms such as 
 reference counting.

 ...

 I thought I made it clear that GC avoidance (which includes 
 considering built-in reference counting) is a major focus of 
 2014.

 Andrei


My apologies then. Not apologizing for stressing this point over 
and over again, though :)

Just to steal some people's thunder: I want the GC to remain in D 
as it is. This is not hyperbole or cynicism. I mean it. Having 
the GC around to clean up after me for noncritical apps is a 
blessing. Adding ARC and owning pointers would make my (and 
others) day though.

BTW: Why is most everybody who doesn't like GC obsessed with ARC 
pointers? I like them for where they are appropriate. But what 
about a simpler owning pointer that frees automatically when it 
goes out of scope. It comes at exactly zero runtime cost, just as 
a "dumb" pointer. You don't have to drag a reference count around 
with you. It is just a pointer in memory. All its semantics are 
enforced at compile time. Throw borrowed pointers in the mix, 
i.e. references to owning pointers that become "frozen" while 
there are outstanding references to them, and you're all set.

I think this would really be necessary (and sufficient!) to get 
Rust off your back and whatever half baked language stuff MS is 
intentionally leaking these days. At least IMHO and others from 
the *vocal minority*.

I'm not quite sure that I understand what you mean by GC 
avoidance being a major focus of 2014 though. In the long term, 
can I look forward to writing an absolutely, like in 100 %, like 
in guaranteed, GC free D app with all of current D's and Phobos' 
features if I choose to? Or does it mean: well for the most part 
it will avoid the GC, but if you're unlucky the GC might still 
kick in if you don't pay attention and when you least expect it?



 Feb 03 2014



 next sibling parent reply "woh" <wojw yahoo.com>  writes:


  Any system that forces a single way of handling memory as the 
only viable method, be it GC( as D currently does), or ARC, is 
undesirable to me

   Rust seems to have found a very nice model, and even cpp with 
value/unique/rc/weak is IMO superior to what D currently offers

On Monday, 3 February 2014 at 22:51:50 UTC, Frank Bauer wrote:

 Andrei Alexandrescu wrote:


 2. Work on Phobos to see what can be done about avoiding 
 unnecessary allocation. Most likely we'll need to also add a 
  nogc flag.

 ...

 4. Work on the core language and druntime to see how to 
 seamlessly accommodate alternate GC mechanisms such as 
 reference counting.

 ...

 I thought I made it clear that GC avoidance (which includes 
 considering built-in reference counting) is a major focus of 
 2014.

 Andrei


 My apologies then. Not apologizing for stressing this point 
 over and over again, though :)

 Just to steal some people's thunder: I want the GC to remain in 
 D as it is. This is not hyperbole or cynicism. I mean it. 
 Having the GC around to clean up after me for noncritical apps 
 is a blessing. Adding ARC and owning pointers would make my 
 (and others) day though.

 BTW: Why is most everybody who doesn't like GC obsessed with 
 ARC pointers? I like them for where they are appropriate. But 
 what about a simpler owning pointer that frees automatically 
 when it goes out of scope. It comes at exactly zero runtime 
 cost, just as a "dumb" pointer. You don't have to drag a 
 reference count around with you. It is just a pointer in 
 memory. All its semantics are enforced at compile time. Throw 
 borrowed pointers in the mix, i.e. references to owning 
 pointers that become "frozen" while there are outstanding 
 references to them, and you're all set.

 I think this would really be necessary (and sufficient!) to get 
 Rust off your back and whatever half baked language stuff MS is 
 intentionally leaking these days. At least IMHO and others from 
 the *vocal minority*.

 I'm not quite sure that I understand what you mean by GC 
 avoidance being a major focus of 2014 though. In the long term, 
 can I look forward to writing an absolutely, like in 100 %, 
 like in guaranteed, GC free D app with all of current D's and 
 Phobos' features if I choose to? Or does it mean: well for the 
 most part it will avoid the GC, but if you're unlucky the GC 
 might still kick in if you don't pay attention and when you 
 least expect it?





 Feb 03 2014



 next sibling parent reply "Frank Bauer" <y z.com>  writes:


On Monday, 3 February 2014 at 23:13:07 UTC, woh wrote:


   Rust seems to have found a very nice model ...


How cool would that be: a language with the memory allocation 
features of Rust but with stronger syntax and semantics roots in 
the C / C++ world?



 Feb 03 2014



  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 00:08:28 UTC, Frank Bauer wrote:

 How cool would that be: a language with the memory allocation 
 features of Rust but with stronger syntax and semantics roots 
 in the C / C++ world?


It is really tempting to write a new parser for Rust, isn't it? 
:-)



 Feb 03 2014



  parent  "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 4 February 2014 at 00:15:34 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 4 February 2014 at 00:08:28 UTC, Frank Bauer wrote:

 How cool would that be: a language with the memory allocation 
 features of Rust but with stronger syntax and semantics roots 
 in the C / C++ world?


 It is really tempting to write a new parser for Rust, isn't it? 
 :-)


Us ML folks don't see a need for it.



 Feb 04 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 3:13 PM, woh wrote:

   Any system that forces a single way of handling memory as the only
 viable method, be it GC( as D currently does), or ARC, is undesirable to me

    Rust seems to have found a very nice model, and even cpp with
 value/unique/rc/weak is IMO superior to what D currently offers


I think Rust expended too much language complexity on that one issue.

Andrei



 Feb 03 2014



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 4 February 2014 at 02:51:21 UTC, Andrei Alexandrescu 
wrote:

   Rust seems to have found a very nice model, and even cpp with
 value/unique/rc/weak is IMO superior to what D currently offers


 I think Rust expended too much language complexity on that one 
 issue.

 Andrei


By the way, while this statement was true for initial design, 
they have recently moved to much more simple model, replacing 
most of more complicated pointer types with library solutions. I 
think those who refer to Rust example are more likely to have in 
mind that new model and your judgement seems to be based on 
previous one.



 Feb 04 2014



  parent reply "Matthias Einwag" <matthias.einwag googlemail.com>  writes:


 By the way, while this statement was true for initial design, 
 they have recently moved to much more simple model, replacing 
 most of more complicated pointer types with library solutions. 
 I think those who refer to Rust example are more likely to have 
 in mind that new model and your judgement seems to be based on 
 previous one.


In my opinion the new model is even harder. As you can only put 
immutable data in their Gc/Rc structures you end up with things 
like Rc<RefMut<Type>> or Rc<Cell<Type>> and must exactly 
understand what each of those does. For me already the library 
solution in C++ (shared_ptr<Type>) is somewhat annoying to type 
when you compare that to the current D solution or other managed 
languages. Dereferencing these types in Rust is also very hard at 
the moment, but they plan do get it easier.

Freely mixing ARC and GC is also not that easy in my opinion: As 
soon as you have a garbage collected object anywhere in your 
object hierarchy basically anything beneath that will be also 
"garbage collected" - in the sense of only deallocated when the 
GC runs. And if you used RAII semantics in some of your ARC 
managed objects you might wonder why their resources are not 
immediatly releasd, but also when the GC runs.



 Feb 04 2014



  parent  "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 4 February 2014 at 18:19:56 UTC, Matthias Einwag 
wrote:

 In my opinion the new model is even harder. ... <skip>


I also find exact implementation considerably over-engineered but 
I would never propose to just copy stuff to Rust as-is. What I do 
propose to is to acknowledge general principle - build base 
language upon allocation-ignorant primitives that provide 
ownership semantics and move all advanced memory management to 
the library. I sincerely believe that `std .allocator` alone as 
presented by Andrei will give D a huge edge when exploring 
similar approach.



 Feb 04 2014



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/3/2014 3:13 PM, woh wrote:

   Any system that forces a single way of handling memory as the only viable
 method, be it GC( as D currently does)


This is incorrect. You can use malloc/free in D, as well as write & use your
own 
allocators, and even write your own ref counted types.



 Feb 03 2014



  parent reply "Frank Bauer" <y z.com>  writes:


On Tuesday, 4 February 2014 at 06:47:03 UTC, Walter Bright wrote:

 On 2/3/2014 7:03 PM, Adam Wilson wrote:

 Note that ObjC has special syntax to handle weak pointers. 
 It's not well
 understood by many.


 Sounds like explicitly managed memory is hardly worse.

 On 2/3/2014 3:13 PM, woh wrote:

  Any system that forces a single way of handling memory as the 
 only viable
 method, be it GC( as D currently does)


 This is incorrect. You can use malloc/free in D, as well as 
 write & use your own allocators, and even write your own ref 
 counted types.

 On 2/3/2014 1:42 PM, Shammah Chancellor wrote:

 It's also probably
 possible to create a drop-in replacement for the GC to do 
 something else.


 It certainly is possible. There's nothing magic about the 
 current GC, it's just library code.



Andrei Alexandrescu wrote:

 2. Work on Phobos to see what can be done about avoiding 
 unnecessary allocation. Most likely we'll need to also add a 
  nogc flag.
 ...
 4. Work on the core language and druntime to see how to 
 seamlessly accommodate alternate GC mechanisms such as 
 reference counting.
 ...
 I thought I made it clear that GC avoidance (which includes 
 considering built-in reference counting) is a major focus of 
 2014.

 Andrei


Im totally confused: Walter, do you back what Andrei says or do 
we have a good cop / bad cop situation here?



 Feb 04 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/4/14, 6:26 AM, Frank Bauer wrote:

 On Tuesday, 4 February 2014 at 06:47:03 UTC, Walter Bright wrote:

 On 2/3/2014 7:03 PM, Adam Wilson wrote:

 Note that ObjC has special syntax to handle weak pointers. It's not well
 understood by many.


 Sounds like explicitly managed memory is hardly worse.

 On 2/3/2014 3:13 PM, woh wrote:

  Any system that forces a single way of handling memory as the only
 viable
 method, be it GC( as D currently does)


 This is incorrect. You can use malloc/free in D, as well as write &
 use your own allocators, and even write your own ref counted types.

 On 2/3/2014 1:42 PM, Shammah Chancellor wrote:

 It's also probably
 possible to create a drop-in replacement for the GC to do something
 else.


 It certainly is possible. There's nothing magic about the current GC,
 it's just library code.



 Andrei Alexandrescu wrote:

 2. Work on Phobos to see what can be done about avoiding unnecessary
 allocation. Most likely we'll need to also add a  nogc flag.
 ...
 4. Work on the core language and druntime to see how to seamlessly
 accommodate alternate GC mechanisms such as reference counting.
 ...
 I thought I made it clear that GC avoidance (which includes
 considering built-in reference counting) is a major focus of 2014.

 Andrei


 Im totally confused: Walter, do you back what Andrei says or do we have
 a good cop / bad cop situation here?


There's no contradiction. Walter and I are on the same page.

Andrei



 Feb 04 2014



prev sibling next sibling parent  "Frank Bauer" <y z.com>  writes:


An owning pointer is the perfect garbage collector: it gets rid 
of the garbage exactly at the moment it goes out of scope.



 Feb 03 2014



prev sibling next sibling parent reply "Frustrated" <c1514843 drdrb.com>  writes:


On Monday, 3 February 2014 at 21:42:59 UTC, Shammah Chancellor
wrote:

 You can always force the GC to run between cycles in your game, 
 and
 turn off automatic sweeps.  This is how most games operate 
 nowadays.
 It's also probably possible to create a drop-in replacement for 
 the GC
 to do something else.   I could see if being *VERY* useful to 
 make the
 GC take a compile-time parameter to select which GC engine is 
 used.



This is just non-sense. Maybe this is why modern games suck then?
How do you guarantee that the GC won't kick in at the most
inopportune times? Oh, you manually trigger it? When? Right at
the moment when the player is about to down the boss after a 45
min fight?

Oh, right.. you just segfault cause there is no memory left.

On Monday, 3 February 2014 at 22:51:50 UTC, Frank Bauer wrote:


 I'm not quite sure that I understand what you mean by GC 
 avoidance being a major focus of 2014 though. In the long term, 
 can I look forward to writing an absolutely, like in 100 %, 
 like in guaranteed, GC free D app with all of current D's and 
 Phobos' features if I choose to? Or does it mean: well for the 
 most part it will avoid the GC, but if you're unlucky the GC 
 might still kick in if you don't pay attention and when you 
 least expect it?


It's either got to be 100% or nothing. The only issue of the GC
is the non-determinism.... or if you do corner it and trigger it
manually you end up with exactly the types of problems Mr.
Chancellor thinks doesn't exist... i.e., the longer you have to
put off the GC the worse it becomes(the more time it takes to run
or the less memory you have to work with).

It might work ok with some concurrent AGC that you can use for
non-critical parts. e.g., have phobo's use the GC for non-real
time sections of the app(boot up, menu's for games, etc...) then
disable it and use ARC for when the app is meant for optimal
and/or deterministic performance.

One could argue that if one goes this around why not use ARC or
manual memory mangement(MMM?) the whole time... but by using the
GC during the non-critical parts of the program one can focus
less on memory leaks as usual with the GC.

What would be nice is to be able to write code that is oblivious
to how it's memory is managed, but be able to switch between
different methods.

[Sartup->menu's and other non-performance section of the game] <-
use GC
[real-time areas of game] <- use ARC or manual



 Feb 03 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 15:26:22 -0800, Frustrated <c1514843 drdrb.com> wrote:


 On Monday, 3 February 2014 at 21:42:59 UTC, Shammah Chancellor
 wrote:

 You can always force the GC to run between cycles in your game, and
 turn off automatic sweeps.  This is how most games operate nowadays.
 It's also probably possible to create a drop-in replacement for the GC
 to do something else.   I could see if being *VERY* useful to make the
 GC take a compile-time parameter to select which GC engine is used.



 This is just non-sense. Maybe this is why modern games suck then?
 How do you guarantee that the GC won't kick in at the most
 inopportune times? Oh, you manually trigger it? When? Right at
 the moment when the player is about to down the boss after a 45
 min fight?

 Oh, right.. you just segfault cause there is no memory left.

 On Monday, 3 February 2014 at 22:51:50 UTC, Frank Bauer wrote:


 I'm not quite sure that I understand what you mean by GC avoidance  
 being a major focus of 2014 though. In the long term, can I look  
 forward to writing an absolutely, like in 100 %, like in guaranteed, GC  
 free D app with all of current D's and Phobos' features if I choose to?  
 Or does it mean: well for the most part it will avoid the GC, but if  
 you're unlucky the GC might still kick in if you don't pay attention  
 and when you least expect it?


 It's either got to be 100% or nothing. The only issue of the GC
 is the non-determinism.... or if you do corner it and trigger it
 manually you end up with exactly the types of problems Mr.
 Chancellor thinks doesn't exist... i.e., the longer you have to
 put off the GC the worse it becomes(the more time it takes to run
 or the less memory you have to work with).


Why is this myth of non-determinism still alive? The only truly  
non-deterministic GC's are concurrent collectors, but alas concurrent  
collects don't routinely stop-the-world either, so there really aren't any  
pauses to complain about. D's Mark-Sweep GC is *perfectly* deterministic.  
It can  *only* pause on allocation. Ergo you can determine exactly which  
allocation caused the problem. You might not expect the function you  
called to GC-allocate, but that doesn't make it non-deterministic, just  
not what you expected. Please, stop blaming your missed expectations on  
the GC. This non-determinism thing is a red herring that is repeated over  
and over by people who obviously have no idea what they are talking about.


 It might work ok with some concurrent AGC that you can use for
 non-critical parts. e.g., have phobo's use the GC for non-real
 time sections of the app(boot up, menu's for games, etc...) then
 disable it and use ARC for when the app is meant for optimal
 and/or deterministic performance.

 One could argue that if one goes this around why not use ARC or
 manual memory mangement(MMM?) the whole time... but by using the
 GC during the non-critical parts of the program one can focus
 less on memory leaks as usual with the GC.

 What would be nice is to be able to write code that is oblivious
 to how it's memory is managed, but be able to switch between
 different methods.

 [Sartup->menu's and other non-performance section of the game] <-
 use GC
 [real-time areas of game] <- use ARC or manual



-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



 next sibling parent  "Frustrated" <c1514843 drdrb.com>  writes:


On Tuesday, 4 February 2014 at 00:19:53 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 15:26:22 -0800, Frustrated 
 <c1514843 drdrb.com> wrote:


 On Monday, 3 February 2014 at 21:42:59 UTC, Shammah Chancellor
 wrote:

 You can always force the GC to run between cycles in your 
 game, and
 turn off automatic sweeps.  This is how most games operate 
 nowadays.
 It's also probably possible to create a drop-in replacement 
 for the GC
 to do something else.   I could see if being *VERY* useful to 
 make the
 GC take a compile-time parameter to select which GC engine is 
 used.



 This is just non-sense. Maybe this is why modern games suck 
 then?
 How do you guarantee that the GC won't kick in at the most
 inopportune times? Oh, you manually trigger it? When? Right at
 the moment when the player is about to down the boss after a 45
 min fight?

 Oh, right.. you just segfault cause there is no memory left.

 On Monday, 3 February 2014 at 22:51:50 UTC, Frank Bauer wrote:


 I'm not quite sure that I understand what you mean by GC 
 avoidance being a major focus of 2014 though. In the long 
 term, can I look forward to writing an absolutely, like in 
 100 %, like in guaranteed, GC free D app with all of current 
 D's and Phobos' features if I choose to? Or does it mean: 
 well for the most part it will avoid the GC, but if you're 
 unlucky the GC might still kick in if you don't pay attention 
 and when you least expect it?


 It's either got to be 100% or nothing. The only issue of the GC
 is the non-determinism.... or if you do corner it and trigger 
 it
 manually you end up with exactly the types of problems Mr.
 Chancellor thinks doesn't exist... i.e., the longer you have to
 put off the GC the worse it becomes(the more time it takes to 
 run
 or the less memory you have to work with).


 Why is this myth of non-determinism still alive? The only truly 
 non-deterministic GC's are concurrent collectors, but alas 
 concurrent collects don't routinely stop-the-world either, so 
 there really aren't any pauses to complain about. D's 
 Mark-Sweep GC is *perfectly* deterministic. It can  *only* 
 pause on allocation. Ergo you can determine exactly which 
 allocation caused the problem. You might not expect the 
 function you called to GC-allocate, but that doesn't make it 
 non-deterministic, just not what you expected. Please, stop 
 blaming your missed expectations on the GC. This 
 non-determinism thing is a red herring that is repeated over 
 and over by people who obviously have no idea what they are 
 talking about.


What I want you to do then is tell me exactly which allocation
will stop the world. You claim it is deterministic so it should
be obvious... hell, it shouldn't even require any code to monitor
what the GC is doing. With manual allocation and deallocation I
can see exactly when the memory will be allocated and free'ed.
That is deterministic.

Just because you want to use an all encompassing definition that
is meaningless doesn't mean you are right. RNG's are
deterministic using your definition, but that is a useless
definition.



 Feb 03 2014



prev sibling  parent  Manu <turkeyman gmail.com>  writes:


On 4 February 2014 10:19, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 15:26:22 -0800, Frustrated <c1514843 drdrb.com> wrote:

  On Monday, 3 February 2014 at 21:42:59 UTC, Shammah Chancellor

 wrote:


 You can always force the GC to run between cycles in your game, and
 turn off automatic sweeps.  This is how most games operate nowadays.
 It's also probably possible to create a drop-in replacement for the GC
 to do something else.   I could see if being *VERY* useful to make the
 GC take a compile-time parameter to select which GC engine is used.



 This is just non-sense. Maybe this is why modern games suck then?
 How do you guarantee that the GC won't kick in at the most
 inopportune times? Oh, you manually trigger it? When? Right at
 the moment when the player is about to down the boss after a 45
 min fight?

 Oh, right.. you just segfault cause there is no memory left.

 On Monday, 3 February 2014 at 22:51:50 UTC, Frank Bauer wrote:

  I'm not quite sure that I understand what you mean by GC avoidance being

 a major focus of 2014 though. In the long term, can I look forward to
 writing an absolutely, like in 100 %, like in guaranteed, GC free D app
 with all of current D's and Phobos' features if I choose to? Or does it
 mean: well for the most part it will avoid the GC, but if you're unlucky
 the GC might still kick in if you don't pay attention and when you least
 expect it?


 It's either got to be 100% or nothing. The only issue of the GC
 is the non-determinism.... or if you do corner it and trigger it
 manually you end up with exactly the types of problems Mr.
 Chancellor thinks doesn't exist... i.e., the longer you have to
 put off the GC the worse it becomes(the more time it takes to run
 or the less memory you have to work with).


 Why is this myth of non-determinism still alive? The only truly
 non-deterministic GC's are concurrent collectors, but alas concurrent
 collects don't routinely stop-the-world either, so there really aren't any
 pauses to complain about. D's Mark-Sweep GC is *perfectly* deterministic.
 It can  *only* pause on allocation. Ergo you can determine exactly which
 allocation caused the problem. You might not expect the function you called
 to GC-allocate, but that doesn't make it non-deterministic, just not what
 you expected. Please, stop blaming your missed expectations on the GC. This
 non-determinism thing is a red herring that is repeated over and over by
 people who obviously have no idea what they are talking about.



Your assertion makes the assumption that people who write huge complex
programs have completely control over their code, and/or depend on zero
libraries, which is a ridiculous notion.
I'm quite sick of people making claims like that. If the default is to do
something incompatible with my use case, and I depend on any libraries
(including phobos), then it's safe to say, I'm running code that is
incompatible with my use case.
What are my options then?

Additionally, in D, you don't have to type 'new' to allocate memory, it
happens all the time... a closure here, a concatenate there. These are core
language features, and to say that I'm meant to avoid them in my million
loc program, authored by perhaps hundreds of programmers, because I need to
be certain where every alloc is being issued from, is quite unrealistic to
say the least.



 Feb 03 2014



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 2:51 PM, Frank Bauer wrote:

 I'm not quite sure that I understand what you mean by GC avoidance being
 a major focus of 2014 though. In the long term, can I look forward to
 writing an absolutely, like in 100 %, like in guaranteed, GC free D app
 with all of current D's and Phobos' features if I choose to? Or does it
 mean: well for the most part it will avoid the GC, but if you're unlucky
 the GC might still kick in if you don't pay attention and when you least
 expect it?


Depends on how we (including YOU) design it.

Andrei



 Feb 03 2014



prev sibling  parent reply "Frustrated" <c1514843 drdrb.com>  writes:


On Monday, 3 February 2014 at 20:02:30 UTC, Andrei Alexandrescu
wrote:

 On 2/3/14, 6:57 AM, Frank Bauer wrote:

 Anyone asking for the addition of ARC or owning pointers to D, 
 gets
 pretty much ignored. The topic is "Smart pointers instead of 
 GC?",
 remember? People here seem to be more interested in diverting 
 to
 nullable, scope and GC optimization. Telling, indeed.


 I thought I made it clear that GC avoidance (which includes 
 considering built-in reference counting) is a major focus of 
 2014.

 Andrei


I hope the same mistakes are not repeated of the past:

This means that whatever is chosen, removing the hard dependence
on any automatic memory management should be avoided.

It would be nice to be able have D work in all areas of
programming. I think allowing one to turn off all automatic
memory management, or even allowing one to choose which
allocation method to use per function/class/module get about
99.99 of the use cases. You allow people that want the freedom of
now worrying about deallocation to use the AGC and those that
need every drop of performance can go the manual route.

Maybe such a general approach is difficult to implement? But
surely it would be worth it. From this thread, it's obvious that
there are plenty of people interested in its success.

Getting D runtime off any specific memory allocation scheme would
seem to be the priority? Then Phobos? then we party like it's
1999?

It would be nice if one could simply write some allocator, drop
it into D, and everything work out dandy. e.g., I want to try out
a new super fast AGC like metronome GC, I write the code for it,
tell D to use it, and then reap the benefits.



 Feb 04 2014



  parent reply "ed" <growlercab gmail.com>  writes:


On Tuesday, 4 February 2014 at 12:03:31 UTC, Frustrated wrote:
[snip]

 It would be nice if one could simply write some allocator, drop
 it into D, and everything work out dandy. e.g., I want to try 
 out
 a new super fast AGC like metronome GC, I write the code for it,
 tell D to use it, and then reap the benefits.


You can write your own GC now and drop it in, we did as an 
experiment.

If you're talking about a dropping in an allocator that is 
another matter, an allocator is not a GC.


Cheers,
ed



 Feb 04 2014



  parent  "Frustrated" <c1514843 drdrb.com>  writes:


On Tuesday, 4 February 2014 at 12:18:22 UTC, ed wrote:

 On Tuesday, 4 February 2014 at 12:03:31 UTC, Frustrated wrote:
 [snip]

 It would be nice if one could simply write some allocator, drop
 it into D, and everything work out dandy. e.g., I want to try 
 out
 a new super fast AGC like metronome GC, I write the code for 
 it,
 tell D to use it, and then reap the benefits.


 You can write your own GC now and drop it in, we did as an 
 experiment.

 If you're talking about a dropping in an allocator that is 
 another matter, an allocator is not a GC.


 Cheers,
 ed


Yes, you can do a lot with D, even rewrite D runtime to remove GC
dependent stuff, or write your own Phobos library. It is not easy
and D is not *designed* to do that.

AGC is about automatic deallocation so you don't have to *free*
the memory you allocate. To do this you have to allocate through
the GC so it knows when you allocate. So a GC is an allocator.

Regardless, I would simply like to have, as I'm sure many would,
the ability to set how D handles memory.

1. No memory management at all - Fast as possible. No memory is
ever free'ed. Similar to how DMD works.

2. Manual memory management - Pretty fast but old school.
Predictable but error prone. Different allocating schemes for
dealing with efficiency. Some apps require this to be performant.

3. Automatic memory management - Easy to use = Lazy mentality.
Safer. Most apps would not need anything more than this.

4. Combination of the above - The hard part. How to make all this
stuff plug and play, work well together, easy to use, etc...





std.allocators goes a long way in the right direction as does
std.allocators.gc. All I'm wanting is a unified way to use it all
and to get D and phobos off the hard dependence of the GC.



 Feb 04 2014



prev sibling next sibling parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 01.02.2014 08:35, schrieb Manu:

 On 1 February 2014 16:26, Adam Wilson <flyboynw gmail.com
 ...

 No serious console game I'm aware of has ever been written in a language
 with a GC. Casual games, or games that don't attempt to raise the bar
 may get away with it, but that's not the industry I work in.


Not sure how much serious you consider The Witcher 2 for the XBox 360, 
they used a GC in their engine.

http://www.makinggames.de/index.php/magazin/2155_porting_the_witcher_2_on_xbox_360

Yes, they faced some issues with it, but it wasn't thrown away, rather 
optimized how it was being used.

--
Paulo



 Feb 01 2014



prev sibling  parent reply Shammah Chancellor <anonymous coward.com>  writes:


On 2014-02-01 07:35:44 +0000, Manu said:


 On 1 February 2014 16:26, Adam Wilson <flyboynw gmail.com> wrote:
 On Fri, 31 Jan 2014 21:29:04 -0800, Manu <turkeyman gmail.com> wrote:
 
 On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:
 
  std.typecons.RefCounted!T
 
 core.memory.GC.disable();
 
 
 Wow. That was easy.
 
 I see, D's claim of being a multi-paradigm language is not false.
 
 
 It's not a realistic suggestion. Everything you want to link uses the GC,
 and the language its self also uses the GC. Unless you write software in
 complete isolation and forego many valuable features, it's not a solution.
 
 
  Phobos does rely on the GC to some extent. Most algorithms and ranges do
 not though.
 
 
 Running (library) code that was written with GC in mind and turning GC off
 doesn't sound ideal.
 
 But maybe this allows me to familiarise myself more with D. Who knows,
 maybe I can learn to stop worrying and love garbage collection.
 
 Thanks for your help!
 
 
 I've been trying to learn to love the GC for as long as I've been around
 here. I really wanted to break that mental barrier, but it hasn't happened.

 wishlist item for D is the ability to use a reference counted collector in
 place of the built-in GC.
 You're not alone :)
 
 I write realtime and memory-constrained software (console games), and for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable memory
 footprint of the application. You can't make any guarantees or predictions
 about the GC, which is fundamentally incompatible with realtime software.
 Language-level ARC would probably do quite nicely for the miscellaneous
 allocations. Obviously, bulk allocations are still usually best handled in
 a context sensitive manner; ie, regions/pools/freelists/whatever, but the
 convenience of the GC paradigm does offer some interesting and massively
 time-saving features to D.
 Everyone will always refer you to RefCounted, which mangles your types and
 pollutes your code, but aside from that, for ARC to be useful, it needs to
 be supported at the language-level, such that the language/optimiser is
 able to optimise out redundant incref/decref calls, and also that it is
 compatible with immutable (you can't manage a refcount if the object is
 immutable).
 
 The problem isn't GC's per se. But D's horribly naive implementation, 
 games are written on GC languages now all the time (Unity/.NET). And 
 let's be honest, games are kind of a speciality, games do things most 
 programs will never do.
 
 You might want to read the GC Handbook. GC's aren't bad, but most, like 
 the D GC, are just to simplistic for common usage today.
 
 Maybe a sufficiently advanced GC could address the performance 
 non-determinism to an acceptable level, but you're still left with the 
 memory non-determinism, and the conundrum that when your heap 
 approaches full (which is _always_ on a games console), the GC has to 
 work harder and harder, and more often to try and keep the tiny little 
 bit of overhead available.
 A GC heap by nature expects you to have lots of memory, and also lots 
 of FREE memory.
 
 No serious console game I'm aware of has ever been written in a 
 language with a GC. Casual games, or games that don't attempt to raise 
 the bar may get away with it, but that's not the industry I work in.


You can always force the GC to run between cycles in your game, and 
turn off automatic sweeps.  This is how most games operate nowadays.   
It's also probably possible to create a drop-in replacement for the GC 
to do something else.   I could see if being *VERY* useful to make the 
GC take a compile-time parameter to select which GC engine is used.  



 Feb 03 2014



 next sibling parent reply "woh" <wojw yahoo.com>  writes:


  ur right I never thought of that, I bet all them game devs never 
thought of it either, they so dumb.  I bet they never tried to 
use a GC, what fools!  Endless graphs of traced objects, oh yes 
oh yes!  It only runs when I allocate, oh what a fool I've been, 
please castigate me harder!


so good




On Monday, 3 February 2014 at 21:42:59 UTC, Shammah Chancellor 
wrote:

 On 2014-02-01 07:35:44 +0000, Manu said:


 On 1 February 2014 16:26, Adam Wilson <flyboynw gmail.com> 
 wrote:
 On Fri, 31 Jan 2014 21:29:04 -0800, Manu <turkeyman gmail.com> 
 wrote:
 
 On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:
 
  std.typecons.RefCounted!T
 
 core.memory.GC.disable();
 
 
 Wow. That was easy.
 
 I see, D's claim of being a multi-paradigm language is not 
 false.
 
 
 It's not a realistic suggestion. Everything you want to link 
 uses the GC,
 and the language its self also uses the GC. Unless you write 
 software in
 complete isolation and forego many valuable features, it's not 
 a solution.
 
 
  Phobos does rely on the GC to some extent. Most algorithms 
 and ranges do
 not though.
 
 
 Running (library) code that was written with GC in mind and 
 turning GC off
 doesn't sound ideal.
 
 But maybe this allows me to familiarise myself more with D. 
 Who knows,
 maybe I can learn to stop worrying and love garbage collection.
 
 Thanks for your help!
 
 
 I've been trying to learn to love the GC for as long as I've 
 been around
 here. I really wanted to break that mental barrier, but it 
 hasn't happened.
 In fact, I am more than ever convinced that the GC won't do. 

 wishlist item for D is the ability to use a reference counted 
 collector in
 place of the built-in GC.
 You're not alone :)
 
 I write realtime and memory-constrained software (console 
 games), and for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the 
 unknowable memory
 footprint of the application. You can't make any guarantees or 
 predictions
 about the GC, which is fundamentally incompatible with 
 realtime software.
 Language-level ARC would probably do quite nicely for the 
 miscellaneous
 allocations. Obviously, bulk allocations are still usually 
 best handled in
 a context sensitive manner; ie, 
 regions/pools/freelists/whatever, but the
 convenience of the GC paradigm does offer some interesting and 
 massively
 time-saving features to D.
 Everyone will always refer you to RefCounted, which mangles 
 your types and
 pollutes your code, but aside from that, for ARC to be useful, 
 it needs to
 be supported at the language-level, such that the 
 language/optimiser is
 able to optimise out redundant incref/decref calls, and also 
 that it is
 compatible with immutable (you can't manage a refcount if the 
 object is
 immutable).
 
 The problem isn't GC's per se. But D's horribly naive 
 implementation, games are written on GC languages now all the 
 time (Unity/.NET). And let's be honest, games are kind of a 
 speciality, games do things most programs will never do.
 
 You might want to read the GC Handbook. GC's aren't bad, but 
 most, like the D GC, are just to simplistic for common usage 
 today.
 
 Maybe a sufficiently advanced GC could address the performance 
 non-determinism to an acceptable level, but you're still left 
 with the memory non-determinism, and the conundrum that when 
 your heap approaches full (which is _always_ on a games 
 console), the GC has to work harder and harder, and more often 
 to try and keep the tiny little bit of overhead available.
 A GC heap by nature expects you to have lots of memory, and 
 also lots of FREE memory.
 
 No serious console game I'm aware of has ever been written in 
 a language with a GC. Casual games, or games that don't 
 attempt to raise the bar may get away with it, but that's not 
 the industry I work in.


 You can always force the GC to run between cycles in your game, 
 and
 turn off automatic sweeps.  This is how most games operate 
 nowadays.
 It's also probably possible to create a drop-in replacement for 
 the GC
 to do something else.   I could see if being *VERY* useful to 
 make the
 GC take a compile-time parameter to select which GC engine is 
 used.





 Feb 03 2014



 next sibling parent  "Frank Bauer" <y z.com>  writes:


On Monday, 3 February 2014 at 23:00:23 UTC, woh wrote:

  ur right I never thought of that, I bet all them game devs 
 never thought of it either, they so dumb.  I bet they never 
 tried to use a GC, what fools!  Endless graphs of traced 
 objects, oh yes oh yes!  It only runs when I allocate, oh what 
 a fool I've been, please castigate me harder!


 so good


+1



 Feb 03 2014



prev sibling next sibling parent reply "NoUseForAName" <no spam.com>  writes:


On Monday, 3 February 2014 at 23:00:23 UTC, woh wrote:

  ur right I never thought of that, I bet all them game devs 
 never thought of it either, they so dumb.  I bet they never 
 tried to use a GC, what fools!  Endless graphs of traced 
 objects, oh yes oh yes!  It only runs when I allocate, oh what 
 a fool I've been, please castigate me harder!


Also people should consider that Apple (unlike C++ game devs) did 
not have a tradition of contempt for GC. In fact they tried GC 
*before* they switched to ARC. The pro-GC camp always likes to 
pretend that the anti-GC one is just ignorant, rejecting GC based 
on prejudice not experience but Apple rejected GC based on 
experience.

GCed Objective-C did not allow them to deliver the user 
experience they wanted (on mobile), because of the related 
latency issues. So they switched to automated ref counting. It is 
not in question that ref counting sacrifices throughput (compared 
to an advanced GC) but for interactive, user facing applications 
latency is much more important.

You can do soft-real time with GC as long as the GC is 
incremental (D's is not) and you heavily rely on object reuse. 
That is what I am doing with LuaJIT right now and the frame rates 
are nice and constant indeed. However, you pay a high price for 
that. Object reuse means writing additional code, makes things 
more complex and error-prone, which is why your average app 
developer does not do it.. and should not have to do it.

Apple had to come up with a solution which does not assume that 
the developers will be careful about allocations. The performance 
of the apps in the iOS app store are ultimately part of the user 
experience so ARC is the right solution because it means that 
your average iOS app written by Joe Coder will not have latency 
issues or at least less latency issues compared to any GC-based 
solution.

I think it is an interesting decision for the D development team 
to make. Do you want a language which can achieve low latency *if 
used carefully* or one which sacrifices maximal throughput 
performance for less latency issues in the common case.

I see no obvious answer to that. I have read D has recently been 
used for some server system at Facebook, ref counting usually 
degrades performance in that area. It is no coincidence that Java 
shines on the server as a high performance solution while Java is 
a synonym for dog slow memory hog on the desktop and mighty 
unpopular there because of that. The whole Java ecosystem from 
the VM to the libraries is optimized for enterprise server use 
cases, for throughput, scalability, and robustness, not for 
making responsive GUIs (and low latency in general) or for memory 
use.

If D wants to be the new Java GC is the way to go, but no heap 
allocation happy GCed language will ever challenge C/C++ on the 
desktop.

Which reminds me of another major company who paddled back on GC 
based on experience: Microsoft. Do you remember the talk back 

technology stack of the future "managed code" everywhere, C/C++ 

Microsoft Java, shoveling enterprise CRUD in the server room. 
Microsoft is hosting "Going Native" conferences nowadays, 
declaring their present and future dedication to C++ (again) and 
they based the new WinRT on ref counting not GC.



 Feb 03 2014



 next sibling parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 16:24:52 -0800, NoUseForAName <no spam.com> wrote:


 On Monday, 3 February 2014 at 23:00:23 UTC, woh wrote:

  ur right I never thought of that, I bet all them game devs never  
 thought of it either, they so dumb.  I bet they never tried to use a  
 GC, what fools!  Endless graphs of traced objects, oh yes oh yes!  It  
 only runs when I allocate, oh what a fool I've been, please castigate  
 me harder!


 Also people should consider that Apple (unlike C++ game devs) did not  
 have a tradition of contempt for GC. In fact they tried GC *before* they  
 switched to ARC. The pro-GC camp always likes to pretend that the  
 anti-GC one is just ignorant, rejecting GC based on prejudice not  
 experience but Apple rejected GC based on experience.

 GCed Objective-C did not allow them to deliver the user experience they  
 wanted (on mobile), because of the related latency issues. So they  
 switched to automated ref counting. It is not in question that ref  
 counting sacrifices throughput (compared to an advanced GC) but for  
 interactive, user facing applications latency is much more important.


That may be the case, but StackOverflow shows that ARC hasn't been panacea  
in Apple land either. Way to many people don't understand ARC and how to  
use it, and subsequently beg for help understanding heisenleaks and weak  
references. ARC places a higher cognitive load on the programmer than a GC  
does. And Android runs just fine with GC'ed apps, but ARC guys don't want  
to talk about Google's successes there.


 You can do soft-real time with GC as long as the GC is incremental (D's  
 is not) and you heavily rely on object reuse. That is what I am doing  
 with LuaJIT right now and the frame rates are nice and constant indeed.  
 However, you pay a high price for that. Object reuse means writing  
 additional code, makes things more complex and error-prone, which is why  
 your average app developer does not do it.. and should not have to do it.

 Apple had to come up with a solution which does not assume that the  
 developers will be careful about allocations. The performance of the  
 apps in the iOS app store are ultimately part of the user experience so  
 ARC is the right solution because it means that your average iOS app  
 written by Joe Coder will not have latency issues or at least less  
 latency issues compared to any GC-based solution.

 I think it is an interesting decision for the D development team to  
 make. Do you want a language which can achieve low latency *if used  
 carefully* or one which sacrifices maximal throughput performance for  
 less latency issues in the common case.

 I see no obvious answer to that. I have read D has recently been used  
 for some server system at Facebook, ref counting usually degrades  
 performance in that area. It is no coincidence that Java shines on the  
 server as a high performance solution while Java is a synonym for dog  
 slow memory hog on the desktop and mighty unpopular there because of  
 that. The whole Java ecosystem from the VM to the libraries is optimized  
 for enterprise server use cases, for throughput, scalability, and  
 robustness, not for making responsive GUIs (and low latency in general)  
 or for memory use.


Ahem. Wrong. See: WinForms, WPF, Silverlight. All extremely successful GUI  
toolkits that are not known for GC related problems. I've been working  
with WPF since 2005, I can say the biggest performance problem with it by  
far is the naive rendering of rounded corners, the GC has NEVER caused a  
hitch.


 If D wants to be the new Java GC is the way to go, but no heap  
 allocation happy GCed language will ever challenge C/C++ on the desktop.


So that's why nearly every desktop app (for Windows at least, but that's  
the overwhelming majority) that started development since .NET came out is  



 Which reminds me of another major company who paddled back on GC based  

 were new? Microsoft totally wanted that to be the technology stack of  
 the future "managed code" everywhere, C/C++ becoming "legacy". However,  

 CRUD in the server room. Microsoft is hosting "Going Native" conferences  
 nowadays, declaring their present and future dedication to C++ (again)  
 and they based the new WinRT on ref counting not GC.


This is primarily due to Internal Microsoft Politics rather than any  
desire of the Microsoft Developer Community to jettison .NET. I won't dive  
in to those right now, there are plenty of places on the web that tell the  
story better than I could.

I remember sitting in the Build 2011 Keynotes thinking "They just cost  
themselves two years, minimum." Turns out I my estimate was low. They have  
tried in vain for over two years get devs to write WinRT apps using  
C++/CX. They've also failed miserably. I forget the exact percentage, but  

rest are VB.NET, only something like 3% of all apps in the Windows Store  
are C++/CX. Server apps are written almost universally in .NET languages,  
such that they didn't even bother making C++/CX capable of building  
servers. Microsoft recently had to do a mea culpa and reinvest in .NET  
(Build 2013).

Microsoft might be trying to get us all to move to C++ with Hats, but they  
are failing miserably in real terms, despite what the rah-rah section of  
Microsoft would have you believe.

At the end of the day, the overwhelming majority of the MS Dev Community  
it quite happy with the performance of the GC. They even gave us more  
performance tweaking options in the latest release. You'll not find any  
ARC fans in that world. And given that the MS Dev Community eclipses  
Apple's in real terms, that's important to note.



-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



 next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 00:49:04 UTC, Adam Wilson wrote:

 Ahem. Wrong. See: WinForms, WPF, Silverlight. All extremely 
 successful GUI toolkits that are not known for GC related 
 problems. I've been working with WPF since 2005, I can say the 
 biggest performance problem with it by far is the naive 
 rendering of rounded corners, the GC has NEVER caused a hitch.


According to Wikipedia:

«While the majority of WPF is in managed code, the composition 
engine which renders the WPF applications is a native component. 
It is named Media Integration Layer (MIL) and resides in 
milcore.dll. It interfaces directly with DirectX and provides 
basic support for 2D and 3D surfaces, timer-controlled 
manipulation of contents of a surface with a view to exposing 
animation constructs at a higher level, and compositing the 
individual elements of a WPF application into a final 3D "scene" 
that represents the UI of the application and renders it to the 
screen.»

So, Microsoft does not think that GC is suitable for real time 
interactive graphics. And they are right.



 Feb 03 2014



  parent reply "Eric Suen" <eric.suen.tech gmail.com>  writes:


"Ola Fosheim Gr?stad" <ola.fosheim.grostad+dlang gmail.com>"> On Tuesday, 4
February 2014 at 00:49:04 UTC, Adam Wilson 
wrote:

 Ahem. Wrong. See: WinForms, WPF, Silverlight. All extremely successful GUI
toolkits that are not known for GC related 
 problems. I've been working with WPF since 2005, I can say the biggest
performance problem with it by far is the 
 naive rendering of rounded corners, the GC has NEVER caused a hitch.


 According to Wikipedia:

 ?While the majority of WPF is in managed code, the composition engine which
renders the WPF applications is a native 
 component. It is named Media Integration Layer (MIL) and resides in
milcore.dll. It interfaces directly with DirectX 
 and provides basic support for 2D and 3D surfaces, timer-controlled
manipulation of contents of a surface with a view 
 to exposing animation constructs at a higher level, and compositing the
individual elements of a WPF application into 
 a final 3D "scene" that represents the UI of the application and renders it to
the screen.?

 So, Microsoft does not think that GC is suitable for real time interactive
graphics. And they are right.


As long as other code is in managed code, there is GC running at
 background, no matter your code write in whatever high performance
language, it will be affect by GC anyway.  so that "So, Microsoft does
not think that GC is suitable for real time > interactive graphics.
And they are right." is only your opinion. you don't know the real reason
behind MS's decision. some resource need release ASAP, so you can't
rely on GC. or maybe bacause of overhead when call low API in managed
code. 



 Feb 03 2014



 next sibling parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 23:05:35 -0800, Eric Suen <eric.suen.tech gmail.com>  
wrote:


 "Ola Fosheim Gr?stad" <ola.fosheim.grostad+dlang gmail.com>"> On  
 Tuesday, 4 February 2014 at 00:49:04 UTC, Adam Wilson
 wrote:

 Ahem. Wrong. See: WinForms, WPF, Silverlight. All extremely successful  
 GUI toolkits that are not known for GC related
 problems. I've been working with WPF since 2005, I can say the biggest  
 performance problem with it by far is the
 naive rendering of rounded corners, the GC has NEVER caused a hitch.


 According to Wikipedia:

 ?While the majority of WPF is in managed code, the composition engine  
 which renders the WPF applications is a native
 component. It is named Media Integration Layer (MIL) and resides in  
 milcore.dll. It interfaces directly with DirectX
 and provides basic support for 2D and 3D surfaces, timer-controlled  
 manipulation of contents of a surface with a view
 to exposing animation constructs at a higher level, and compositing the  
 individual elements of a WPF application into
 a final 3D "scene" that represents the UI of the application and  
 renders it to the screen.?

 So, Microsoft does not think that GC is suitable for real time  
 interactive graphics. And they are right.


 As long as other code is in managed code, there is GC running at
  background, no matter your code write in whatever high performance
 language, it will be affect by GC anyway.  so that "So, Microsoft does
 not think that GC is suitable for real time > interactive graphics.
 And they are right." is only your opinion. you don't know the real reason
 behind MS's decision. some resource need release ASAP, so you can't
 rely on GC. or maybe bacause of overhead when call low API in managed
 code.


Actually the reason is that DirectX is a specialized Native COM-Like API  
that is NOT compatible with normal COM and therefore not compatible with  
.NET COM Interop. Milcore is a low-level immediate API. Most of the  
legwork in WPF is done in managed code. And yes, the GC would effect the  
performance regardless of whether or not the rendering API is native  
because so much of WPF is managed code.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 07:15:17 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 23:05:35 -0800, Eric Suen 
 <eric.suen.tech gmail.com> wrote:


 "Ola Fosheim Gr?stad" <ola.fosheim.grostad+dlang gmail.com>">





 So, Microsoft does not think that GC is suitable for real 
 time interactive graphics. And they are right.







 And they are right." is only your opinion. you don't know the 
 real reason
 behind MS's decision. some resource need release ASAP, so you





 Actually the reason is that DirectX is a specialized Native 
 COM-Like API that is NOT compatible with normal COM and 
 therefore not compatible with .NET COM Interop. Milcore is a


Quoting
http://msdn.microsoft.com/en-us/library/ms750441(v=vs.110).aspx :

«Milcore is written in unmanaged code in order to enable tight 
integration with DirectX. All display in WPF is done through the 
DirectX engine, allowing for efficient hardware and software 
rendering. WPF also required fine control over memory and 
execution. The composition engine in milcore is extremely 
performance sensitive, and required giving up many advantages of 
the CLR to gain performance.»

Please note that Microsoft knows their stuff: «WPF also required 
fine control over memory and execution.» and «required giving up 
many advantages of the CLR to gain performance».

WPF maintains a retained mode shadow-tree of the composition 
elements of the scene graph to get more responsive applications. 
i.e. to avoid having client code blocking rendering:

«There is a very important architectural detail to notice here – 
the entire tree of visuals and drawing instructions is cached. In 
graphics terms, WPF uses a retained rendering system. This 
enables the system to repaint at high refresh rates without the 
composition system blocking on callbacks to user code. This helps 
prevent the appearance of an unresponsive application.»

But… I don't think it was a good idea to go for full 
back-to-front rendering.



 Feb 04 2014



prev sibling  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 07:06:07 UTC, Eric Suen wrote:

 As long as other code is in managed code, there is GC running at
  background, no matter your code write in whatever high 
 performance
 language, it will be affect by GC anyway.  so that "So, 
 Microsoft does
 not think that GC is suitable for real time > interactive 
 graphics.
 And they are right." is only your opinion. you don't know the 
 real reason
 behind MS's decision. some resource need release ASAP, so you 
 can't
 rely on GC. or maybe bacause of overhead when call low API in 
 managed
 code.


Anyone who has tried to use Direct 3D from managed code knows 
why. They had the foundation to do everything from manage code, 
but chose not to.

You can get smooth animations in GC javascript on browsers too, 
but only if relying on the C++ CSS transition engine which does 
the frame-by-frame interpolation for you. But I expect javascript 
to get better at this if/when using true isolates. If the memory 
pool is small enough GC can work out ok, but for very low 
latency/overhead you use the stack/pools, not even malloc is good 
enough then.



 Feb 04 2014



prev sibling next sibling parent reply "NoUseForAName" <no spam.com>  writes:


On Tuesday, 4 February 2014 at 00:49:04 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 16:24:52 -0800, NoUseForAName <no spam.com> 
 wrote:


On Tuesday, 4 February 2014 at 00:49:04 UTC, Adam Wilson wrote:

 That may be the case, but StackOverflow shows that ARC hasn't 
 been panacea in Apple land either. Way to many people don't 
 understand ARC and how to use it, and subsequently beg for help 
 understanding heisenleaks and weak references.


Your point? ARC addressed the latency issues, I never said it was 
without challenges of its own.


ARC places a higher cognitive load on the programmer than a GC 
does.


Yes, it does. But the whole "not thinking about allocations" 
thing comes at the often unacceptable cost of unresponsive apps.


And
 Android runs just fine with GC'ed apps, but ARC guys don't want 
 to talk about Google's successes there.


Google's success there is that demanding apps are written using 
the NDK.


 Ahem. Wrong. See: WinForms, WPF, Silverlight. All extremely 
 successful GUI toolkits that are not known for GC related 
 problems.


Silverlight is dead and was an utter failure. WinForms and WPF 
have an uncertain future. Neither has ever been used much in end 
user applications.

I would also like to say that the typical .NET or Java developer 
has lost all sense of what an efficient app feels like. E.g. 
someone who works with Eclipse all day will of course consider 
about everything else lightweight and snappy.


 So that's why nearly every desktop app (for Windows at least, 
 but that's the overwhelming majority) that started development 



That is simply not true. The set of widely popular Windows 
desktop applications is basically .NET free. However, maybe you 
misunderstood me because - I admit - my phrasing was unclear. 
When I said "desktop" I meant end user desktop applications and 
games. Not enterprise/government desktop CRUD apps which are 
forced upon office workers who cannot reject them because of 
their horrible performance. I would not be surprised if most of 
those are indeed written in .NET (if not written in Java).


 only something like 3% of all apps in the Windows Store are 
 C++/CX.


Does anybody actually use Windows Store? Frankly I do not know 
anyone who does.


 Server apps are written almost universally in .NET languages


Eh.. yes. I said myself that Java and Java-likes rule that domain.

Again, if D wants to compete with Java (or Microsoft's version of 
it) there is nothing wrong with GC.



 Feb 03 2014



  parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 17:48:42 -0800, NoUseForAName <no spam.com> wrote:


 On Tuesday, 4 February 2014 at 00:49:04 UTC, Adam Wilson wrote:

 On Mon, 03 Feb 2014 16:24:52 -0800, NoUseForAName <no spam.com> wrote:


 On Tuesday, 4 February 2014 at 00:49:04 UTC, Adam Wilson wrote:

 That may be the case, but StackOverflow shows that ARC hasn't been  
 panacea in Apple land either. Way to many people don't understand ARC  
 and how to use it, and subsequently beg for help understanding  
 heisenleaks and weak references.


 Your point? ARC addressed the latency issues, I never said it was  
 without challenges of its own.


 ARC places a higher cognitive load on the programmer than a GC does.


 Yes, it does. But the whole "not thinking about allocations" thing comes  
 at the often unacceptable cost of unresponsive apps.


So basically, all programmers who use GC's are too stupid to understand  
ARC and should therefore either learn it because it is categorically  
superior, or face the ridicule of their superior brethren? A tad  
unrealistic there, most use a GC for purely practical reasons, namely  
their workload is such that without the benefit of not having to worry  
about memory they'd be even further behind, and subsequently spend even  
more time away from family and loved-ones.


 And
 Android runs just fine with GC'ed apps, but ARC guys don't want to talk  
 about Google's successes there.


 Google's success there is that demanding apps are written using the NDK.


Fair enough.


 Ahem. Wrong. See: WinForms, WPF, Silverlight. All extremely successful  
 GUI toolkits that are not known for GC related problems.


 Silverlight is dead and was an utter failure. WinForms and WPF have an  
 uncertain future. Neither has ever been used much in end user  
 applications.


And how certain are you of this? Because that assertion doesn't match the  
data MS has collected.


 I would also like to say that the typical .NET or Java developer has  
 lost all sense of what an efficient app feels like. E.g. someone who  
 works with Eclipse all day will of course consider about everything else  
 lightweight and snappy.


"I don't agree with their choices or preferences so they are wrong."


 So that's why nearly every desktop app (for Windows at least, but  
 that's the overwhelming majority) that started development since .NET  



 That is simply not true. The set of widely popular Windows desktop  
 applications is basically .NET free. However, maybe you misunderstood me  
 because - I admit - my phrasing was unclear. When I said "desktop" I  
 meant end user desktop applications and games. Not enterprise/government  
 desktop CRUD apps which are forced upon office workers who cannot reject  
 them because of their horrible performance. I would not be surprised if  
 most of those are indeed written in .NET (if not written in Java).


So now you're tightening your definition of desktop app to something you  
think you can win the argument on? Not a very powerful argument. I am  
using the industry standard definition: A desktop app is anything that  
runs on a desktop OS regardless of usage. Example, Visual Studio is both a  
.NET/WPF app AND a desktop app.


 only something like 3% of all apps in the Windows Store are C++/CX.


 Does anybody actually use Windows Store? Frankly I do not know anyone  
 who does.


LOL, I tend to agree, but that's a strawman, and besides the point. I was  
merely commenting on the code makeup of the Windows Store, not it's usage.


 Server apps are written almost universally in .NET languages


 Eh.. yes. I said myself that Java and Java-likes rule that domain.

 Again, if D wants to compete with Java (or Microsoft's version of it)  
 there is nothing wrong with GC.


D absolutely DOES want to get compete here, otherwise we might as well  
evict Andrei and Facebook right now.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



prev sibling next sibling parent reply Manu <turkeyman gmail.com>  writes:


On 4 February 2014 10:49, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 16:24:52 -0800, NoUseForAName <no spam.com> wrote:

  On Monday, 3 February 2014 at 23:00:23 UTC, woh wrote:

  ur right I never thought of that, I bet all them game devs never
 thought of it either, they so dumb.  I bet they never tried to use a GC,
 what fools!  Endless graphs of traced objects, oh yes oh yes!  It only runs
 when I allocate, oh what a fool I've been, please castigate me harder!


 Also people should consider that Apple (unlike C++ game devs) did not
 have a tradition of contempt for GC. In fact they tried GC *before* they
 switched to ARC. The pro-GC camp always likes to pretend that the anti-GC
 one is just ignorant, rejecting GC based on prejudice not experience but
 Apple rejected GC based on experience.

 GCed Objective-C did not allow them to deliver the user experience they
 wanted (on mobile), because of the related latency issues. So they switched
 to automated ref counting. It is not in question that ref counting
 sacrifices throughput (compared to an advanced GC) but for interactive,
 user facing applications latency is much more important.


 That may be the case, but StackOverflow shows that ARC hasn't been panacea
 in Apple land either. Way to many people don't understand ARC and how to
 use it, and subsequently beg for help understanding heisenleaks and weak
 references. ARC places a higher cognitive load on the programmer than a GC
 does. And Android runs just fine with GC'ed apps, but ARC guys don't want
 to talk about Google's successes there.



I'd have trouble disagreeing more; Android is the essence of why Java
should never be used for user-facing applications.
Android is jerky and jittery, has random pauses and lockups all the time,
and games on android always jitter and drop frames. Most high-end games on
android now are written in C++ as a means to mitigate that problem, but
then you're back writing C++. Yay!
iOS is silky smooth by comparison to Android.
I'm sure this isn't entirely attributable to the GC, or Java in general,
but it can't possibly be used as an example of success. Precisely the
opposite if anything. Games on Android make gamedevs who care about smooth
interactivity's brains bleed.

 You can do soft-real time with GC as long as the GC is incremental (D's is

 not) and you heavily rely on object reuse. That is what I am doing with
 LuaJIT right now and the frame rates are nice and constant indeed. However,
 you pay a high price for that. Object reuse means writing additional code,
 makes things more complex and error-prone, which is why your average app
 developer does not do it.. and should not have to do it.

 Apple had to come up with a solution which does not assume that the
 developers will be careful about allocations. The performance of the apps
 in the iOS app store are ultimately part of the user experience so ARC is
 the right solution because it means that your average iOS app written by
 Joe Coder will not have latency issues or at least less latency issues
 compared to any GC-based solution.

 I think it is an interesting decision for the D development team to make.
 Do you want a language which can achieve low latency *if used carefully* or
 one which sacrifices maximal throughput performance for less latency issues
 in the common case.

 I see no obvious answer to that. I have read D has recently been used for
 some server system at Facebook, ref counting usually degrades performance
 in that area. It is no coincidence that Java shines on the server as a high
 performance solution while Java is a synonym for dog slow memory hog on the
 desktop and mighty unpopular there because of that. The whole Java
 ecosystem from the VM to the libraries is optimized for enterprise server
 use cases, for throughput, scalability, and robustness, not for making
 responsive GUIs (and low latency in general) or for memory use.


 Ahem. Wrong. See: WinForms, WPF, Silverlight. All extremely successful GUI
 toolkits that are not known for GC related problems. I've been working with
 WPF since 2005, I can say the biggest performance problem with it by far is
 the naive rendering of rounded corners, the GC has NEVER caused a hitch.



On a modern many ghz PC with many cores, and many gb of ram (most of which
is unallocated), a hardware virtual memory manager, and a mature RTOS.
Computers come in all shapes and sizes. D is positioned as a systems
language, last time I checked... or else I don't know what I'm doing here.

 If D wants to be the new Java GC is the way to go, but no heap allocation

 happy GCed language will ever challenge C/C++ on the desktop.


 So that's why nearly every desktop app (for Windows at least, but that's
 the overwhelming majority) that started development since .NET came out is





it is productive and awesome; has an amazing dev infrastructure, dev
environment, well integrated GUI toolkits, debugger works awesome, docs are
excellent, etc.
Correlation does not imply causality.

the GC as it's biggest flaw.




moments notice.



 Feb 03 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 5:51 PM, Manu wrote:

 I'd have trouble disagreeing more; Android is the essence of why Java
 should never be used for user-facing applications.
 Android is jerky and jittery, has random pauses and lockups all the
 time, and games on android always jitter and drop frames. Most high-end
 games on android now are written in C++ as a means to mitigate that
 problem, but then you're back writing C++. Yay!
 iOS is silky smooth by comparison to Android.


Kinda difficult to explain the market success of Android.

Andrei



 Feb 03 2014



 next sibling parent reply Manu <turkeyman gmail.com>  writes:


On 4 February 2014 12:59, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org

 wrote:



 On 2/3/14, 5:51 PM, Manu wrote:


 I'd have trouble disagreeing more; Android is the essence of why Java
 should never be used for user-facing applications.
 Android is jerky and jittery, has random pauses and lockups all the
 time, and games on android always jitter and drop frames. Most high-end
 games on android now are written in C++ as a means to mitigate that
 problem, but then you're back writing C++. Yay!
 iOS is silky smooth by comparison to Android.


 Kinda difficult to explain the market success of Android.



I think it's easy to explain.
1. It's aggressively backed by the biggest technology company in the world.
2. It's free for product vendors.
3. For all product vendors at the curve of Android's success, it presented
a realistic and well supported (by Google) competition to Apple, who were
running away with the industry. Everybody had to compete with Apple, but
didn't have the resources to realistically compete on their own. Nokia for
instance were certainly in the best position to produce serious
competition, but they fumbled multiple times. I suspect Google won because
they're Google, and it is free.


Java, except it's even better. If Java was a significant factor in
Android's success, I'd argue that WindowsMobile should have been equally
successful.
I think it's safe to say, that's not the case.
Personally, I suspect that Java was actually a barrier to entry in early
Android, and even possibly the reason that it took as it did for Android to
take root.
It's most certainly the reason that Android had absolutely no games on it
for so many years. They eventually released the NDK, and games finally
appeared. There were years between Angry Birds success in iPhone, and any
serious games appearing on Android.
There were years where if you wanted to play games in your mobile device,
you had to get an iDevice. It's finally levelled now that the indisputable
success of Android is absolute (and the NDK is available).



 Feb 03 2014



 next sibling parent  Nick Sabalausky <SeeWebsiteToContactMe semitwist.com>  writes:


On 2/3/2014 10:30 PM, Manu wrote:

 On 4 February 2014 12:59, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org

 wrote:

 Kinda difficult to explain the market success of Android.




Not really. With PalmOS long dead (RIP) and MS famously just flailing 
around on anything battery-powered, Android has been the only realistic 
option for those who want a handheld computer, but don't want to sell 
all their freedoms to Apple. Plus it's Google.


 2. It's free for product vendors.


No.

I mean, on the surface yea it appears free, but realistically it's on 
the decline: Device manufacturers (partly under pressure from asswhores 
like Verizon, no doubt) have been moving towards a system where normal, 
affordable, contract-subsidized devices (not that I'm a fan of the 
contracts) are locked-down as much as possible [1], even using eFuses 
now, and thereby rendered impractical for average developers (since you 
can't freely test on multiple OS versions).

But, GOLLY GEE!! FOR *ONLY SEVERAL HUNDREDS* OF PURELY OUT-OF-POCKET 
DOLLARS, these douchebag corporations like Google or Samsung will 
happily sell you a "developer" phone that restricts FEWER of your 
freedoms! Isn't that great? And best of all, Google gets to pretend 
being an Android developer is still free! Get it? Because the extra $$$ 
developers pay is for the psuedo-optional *wink wink nudge nudge* 
"developer phone" instead of a "developer account" like mean-ol'-Apple.

[1] Not only the usual locked bootloaders, but see also Knox-secured 
bootloader. And no, I'm not being tin-foil-hat here, I've actually been 
bit by that and had a device bricked because of that forced-update 
Knox-secured bullshit, just because I needed to switch between 4.3 and 
4.1.2 which Samsung decided I shouldn't be allowed to do (Knox eFuse). 
Luckily it was still under warranty and pretty much all salespeople and 
support reps are about as intelligent as your average house pet, so I 
was able to get a replacement...at least after they finally got tired of 
screwing up the replacement orders (hmmm, a DOA device...a wrong device 
from a completely different manufacturer...).



 Feb 03 2014



prev sibling next sibling parent  "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 4 February 2014 at 03:30:58 UTC, Manu wrote:

 On 4 February 2014 12:59, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org

 wrote:



 On 2/3/14, 5:51 PM, Manu wrote:


 I'd have trouble disagreeing more; Android is the essence of 
 why Java
 should never be used for user-facing applications.
 Android is jerky and jittery, has random pauses and lockups 
 all the
 time, and games on android always jitter and drop frames. 
 Most high-end
 games on android now are written in C++ as a means to 
 mitigate that
 problem, but then you're back writing C++. Yay!
 iOS is silky smooth by comparison to Android.


 Kinda difficult to explain the market success of Android.



 I think it's easy to explain.
 1. It's aggressively backed by the biggest technology company 
 in the world.
 2. It's free for product vendors.
 3. For all product vendors at the curve of Android's success, 
 it presented
 a realistic and well supported (by Google) competition to 
 Apple, who were
 running away with the industry. Everybody had to compete with 
 Apple, but
 didn't have the resources to realistically compete on their 
 own. Nokia for
 instance were certainly in the best position to produce serious
 competition, but they fumbled multiple times. I suspect Google 
 won because
 they're Google, and it is free.


 same as
 Java, except it's even better. If Java was a significant factor 
 in
 Android's success, I'd argue that WindowsMobile should have 
 been equally
 successful.
 I think it's safe to say, that's not the case.
 Personally, I suspect that Java was actually a barrier to entry 
 in early
 Android, and even possibly the reason that it took as it did 
 for Android to
 take root.
 It's most certainly the reason that Android had absolutely no 
 games on it
 for so many years. They eventually released the NDK, and games 
 finally
 appeared. There were years between Angry Birds success in 
 iPhone, and any
 serious games appearing on Android.
 There were years where if you wanted to play games in your 
 mobile device,
 you had to get an iDevice. It's finally levelled now that the 
 indisputable
 success of Android is absolute (and the NDK is available).


You forget to mention that the NDK has a very restrained set of 
APIs.

If you want to do interact with the OS besides audio and OpenGL, 
it is JNI Ad nauseam because Google sees the NDK as a minor 
inconvenience and all APIs are Java based.

Even Java native methods need to be called via their Java class 
and are not accessible to NDK code.

--
Paulo



 Feb 04 2014



prev sibling  parent  Xavier Bigand <flamaros.xavier gmail.com>  writes:


Le 04/02/2014 04:30, Manu a écrit :

 On 4 February 2014 12:59, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org <mailto:SeeWebsiteForEmail erdani.org>>
 wrote:

     On 2/3/14, 5:51 PM, Manu wrote:

         I'd have trouble disagreeing more; Android is the essence of why
         Java
         should never be used for user-facing applications.
         Android is jerky and jittery, has random pauses and lockups all the
         time, and games on android always jitter and drop frames. Most
         high-end
         games on android now are written in C++ as a means to mitigate that
         problem, but then you're back writing C++. Yay!
         iOS is silky smooth by comparison to Android.


     Kinda difficult to explain the market success of Android.


 I think it's easy to explain.
 1. It's aggressively backed by the biggest technology company in the world.
 2. It's free for product vendors.
 3. For all product vendors at the curve of Android's success, it
 presented a realistic and well supported (by Google) competition to
 Apple, who were running away with the industry. Everybody had to compete
 with Apple, but didn't have the resources to realistically compete on
 their own. Nokia for instance were certainly in the best position to
 produce serious competition, but they fumbled multiple times. I suspect
 Google won because they're Google, and it is free.


 Java, except it's even better. If Java was a significant factor in
 Android's success, I'd argue that WindowsMobile should have been equally
 successful.
 I think it's safe to say, that's not the case.
 Personally, I suspect that Java was actually a barrier to entry in early
 Android, and even possibly the reason that it took as it did for Android
 to take root.
 It's most certainly the reason that Android had absolutely no games on
 it for so many years. They eventually released the NDK, and games
 finally appeared. There were years between Angry Birds success in
 iPhone, and any serious games appearing on Android.
 There were years where if you wanted to play games in your mobile
 device, you had to get an iDevice. It's finally levelled now that the
 indisputable success of Android is absolute (and the NDK is available).


I work almost every day on android with NDK and Qt. Tools still poor and 
there is a lot of issues. I found one critical last week in the ndk : an 
error with a python dll causing gdb launch failure, so sometimes you 
just haven't debugger...

Developing games on such devices can be a real pain, apk can't exceed 
50Mo and updating files with the obb package must be done manually.

We never got those kind of difficulties with iOS even at the beginning.

PS : I just buy a Nexus 5 to replace my old Motorola defy, and there is 
absolutely no pauses everything is perfectly smooth. I had some doubts 
if it was possible before buying it cause of Java,... No I just think 
all constructors doesn't delivers good drivers and put completely buggy 
additional components making the bad reputation of Android.
On my defy I always got bad touch events,... It was a real pain to send 
an SMS :-{



 Feb 04 2014



prev sibling next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 4 February 2014 at 02:59:11 UTC, Andrei Alexandrescu 
wrote:

 On 2/3/14, 5:51 PM, Manu wrote:

 I'd have trouble disagreeing more; Android is the essence of 
 why Java
 should never be used for user-facing applications.
 Android is jerky and jittery, has random pauses and lockups 
 all the
 time, and games on android always jitter and drop frames. Most 
 high-end
 games on android now are written in C++ as a means to mitigate 
 that
 problem, but then you're back writing C++. Yay!
 iOS is silky smooth by comparison to Android.


 Kinda difficult to explain the market success of Android.

 Andrei


Market success is probably 10% about technology and 90% about PR, 
filling the niche, finding right time etc.



 Feb 04 2014



  parent reply "Dicebot" <public dicebot.lv>  writes:


There is a lot of discussion ongoing about ARC vs GC but in 
practice forcing either of those is unacceptable. Language that 
is strongly coupled with hard-coded memory model will inevitably 
fail in some domain.

For me perfect solution would have been to use an allocator 
concept as language basis instead and let you chose any 
conformant allocator for built-in language features. With both GC 
and ARC available in Phobos / druntime.

Bottom-Up design is simply superior to Top-Down when it comes to 
tools.



 Feb 04 2014



 next sibling parent  "Frustrated" <c1514843 drdrb.com>  writes:


On Tuesday, 4 February 2014 at 13:18:51 UTC, Dicebot wrote:

 There is a lot of discussion ongoing about ARC vs GC but in 
 practice forcing either of those is unacceptable. Language that 
 is strongly coupled with hard-coded memory model will 
 inevitably fail in some domain.

 For me perfect solution would have been to use an allocator 
 concept as language basis instead and let you chose any 
 conformant allocator for built-in language features. With both 
 GC and ARC available in Phobos / druntime.

 Bottom-Up design is simply superior to Top-Down when it comes 
 to tools.


And who could disagree with that? Yet we have many ;/

I guess it's all about fear. There are people that use the GC and
think they will never need more. They think "changing it" will
cause them to have to learn something new, which is difficult for
them. Hence they rather keep their world the same than allow
someone else the satisfaction of making things better for them.
What they don't realize is that their world won't change... so
maybe it's a bit of ignorance too?



 Feb 04 2014



prev sibling  parent reply "Frank Bauer" <y z.com>  writes:


On Tuesday, 4 February 2014 at 13:18:51 UTC, Dicebot wrote:

 For me perfect solution would have been to use an allocator 
 concept as language basis instead and let you chose any 
 conformant allocator for built-in language features. With both 
 GC and ARC available in Phobos / druntime.


That is exactly what Rust did succsessfully.

http://pcwalton.github.io/blog/2013/06/02/removing-garbage-collection-from-the-rust-language/

got pulled.

They deprecated the   pointer, which is a managed poiter 
equivalent to what D's 'new' produces, to a library solution.

They now have basically only ~T and &T left in the language. ~T 
being an owned pointer that frees at the end of its scope and &T, 
a borrowed pointer (or reference, but *not* RC) with no direct 
effect on memory management.

There is Rc<T> and Gc<T> in the library for those who wish to use 
them.

To quote from the above link:





Programmers don’t know which to use, since some operations are 
available with ~ and some operations are available with  . 
Actually, we were confused on this point for a long time as 
well—it wasn’t clear whether ~ or   would become dominant. We 
debated for a long time which to present first, ~ or  . However, 
as the language and community evolved, and coding standards 
became more settled, a clear winner emerged: the owning pointer 
~. In practice, the rule has been that programmers should use ~ 
to allocate in all circumstances except when they have no way of 
knowing precisely when the object in question should be freed.
<<

This is what happens if you give users choice.

One of Rust's priorities is to have a runtime footprint as small 
as possible. I would give priority to not break existing D code. 
So keep GC'd 'new' in the language (the runtime will never get as 
small as Rust's; don't care) and see if it is possible to add 
owned and ARC pointers from there.



 Feb 04 2014



 next sibling parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Tuesday, 4 February 2014 at 14:19:36 UTC, Frank Bauer wrote:

 On Tuesday, 4 February 2014 at 13:18:51 UTC, Dicebot wrote:

 For me perfect solution would have been to use an allocator 
 concept as language basis instead and let you chose any 
 conformant allocator for built-in language features. With both 
 GC and ARC available in Phobos / druntime.


 That is exactly what Rust did succsessfully.

 http://pcwalton.github.io/blog/2013/06/02/removing-garbage-collection-from-the-rust-language/

 got pulled.

 They deprecated the   pointer, which is a managed poiter 
 equivalent to what D's 'new' produces, to a library solution.

 They now have basically only ~T and &T left in the language. ~T 
 being an owned pointer that frees at the end of its scope and 
 &T, a borrowed pointer (or reference, but *not* RC) with no 
 direct effect on memory management.

 There is Rc<T> and Gc<T> in the library for those who wish to 
 use them.

 To quote from the above link:





 Programmers don’t know which to use, since some operations are 
 available with ~ and some operations are available with  . 
 Actually, we were confused on this point for a long time as 
 well—it wasn’t clear whether ~ or   would become dominant. We 
 debated for a long time which to present first, ~ or  . 
 However, as the language and community evolved, and coding 
 standards became more settled, a clear winner emerged: the 
 owning pointer ~. In practice, the rule has been that 
 programmers should use ~ to allocate in all circumstances 
 except when they have no way of knowing precisely when the 
 object in question should be freed.
 <<

 This is what happens if you give users choice.

 One of Rust's priorities is to have a runtime footprint as 
 small as possible. I would give priority to not break existing 
 D code. So keep GC'd 'new' in the language (the runtime will 
 never get as small as Rust's; don't care) and see if it is 
 possible to add owned and ARC pointers from there.


Rust still remains to prove itself on the mainstream market.

While D already has commercial users.



 Feb 04 2014



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 4 February 2014 at 14:22:41 UTC, Paulo Pinto wrote:

 Rust still remains to prove itself on the mainstream market.

 While D already has commercial users.


It is dangerous position. D is not that far ahead to appeal to 
own authority.

"..it takes all the running you can do, to keep in the same 
place. If you want to get somewhere else, you must run at least 
twice as fast as that!"



 Feb 04 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/4/14, 6:45 AM, Dicebot wrote:

 On Tuesday, 4 February 2014 at 14:22:41 UTC, Paulo Pinto wrote:

 Rust still remains to prove itself on the mainstream market.

 While D already has commercial users.


 It is dangerous position. D is not that far ahead to appeal to own
 authority.


I agree.

Andrei



 Feb 04 2014



prev sibling  parent  "woh" <wojw yahoo.com>  writes:


unique is best, should be default, down with GC!!


Tuesday, 4 February 2014 at 14:19:36 UTC, Frank Bauer wrote:

 On Tuesday, 4 February 2014 at 13:18:51 UTC, Dicebot wrote:

 For me perfect solution would have been to use an allocator 
 concept as language basis instead and let you chose any 
 conformant allocator for built-in language features. With both 
 GC and ARC available in Phobos / druntime.


 That is exactly what Rust did succsessfully.

 http://pcwalton.github.io/blog/2013/06/02/removing-garbage-collection-from-the-rust-language/

 got pulled.

 They deprecated the   pointer, which is a managed poiter 
 equivalent to what D's 'new' produces, to a library solution.

 They now have basically only ~T and &T left in the language. ~T 
 being an owned pointer that frees at the end of its scope and 
 &T, a borrowed pointer (or reference, but *not* RC) with no 
 direct effect on memory management.

 There is Rc<T> and Gc<T> in the library for those who wish to 
 use them.

 To quote from the above link:





 Programmers don’t know which to use, since some operations are 
 available with ~ and some operations are available with  . 
 Actually, we were confused on this point for a long time as 
 well—it wasn’t clear whether ~ or   would become dominant. We 
 debated for a long time which to present first, ~ or  . 
 However, as the language and community evolved, and coding 
 standards became more settled, a clear winner emerged: the 
 owning pointer ~. In practice, the rule has been that 
 programmers should use ~ to allocate in all circumstances 
 except when they have no way of knowing precisely when the 
 object in question should be freed.
 <<

 This is what happens if you give users choice.

 One of Rust's priorities is to have a runtime footprint as 
 small as possible. I would give priority to not break existing 
 D code. So keep GC'd 'new' in the language (the runtime will 
 never get as small as Rust's; don't care) and see if it is 
 possible to add owned and ARC pointers from there.





 Feb 04 2014



prev sibling  parent  "renoX" <renozyx gmail.com>  writes:


On Tuesday, 4 February 2014 at 02:59:11 UTC, Andrei Alexandrescu 
wrote:

 On 2/3/14, 5:51 PM, Manu wrote:

 I'd have trouble disagreeing more; Android is the essence of 
 why Java
 should never be used for user-facing applications.
 Android is jerky and jittery, has random pauses and lockups 
 all the
 time, and games on android always jitter and drop frames. Most 
 high-end
 games on android now are written in C++ as a means to mitigate 
 that
 problem, but then you're back writing C++. Yay!
 iOS is silky smooth by comparison to Android.


 Kinda difficult to explain the market success of Android.

 Andrei


Are you agreeing or disagreeing with his point?
I don't know about other Android phones, but on my Galaxy S3 what 
he says is true and just the other day I had to reboot my phone 
as it was very, very slow, the reboot made it "fast" again..

I don't like Apple (nor Microsoft), so I'm stuck with Android, 
but this doesn't mean that I consider that Android is good.. For 
me it's like Windows in the early days, not very good but usable.

renoX



 Feb 05 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/3/14, 4:49 PM, Adam Wilson wrote:

 That may be the case, but StackOverflow shows that ARC hasn't been
 panacea in Apple land either. Way to many people don't understand ARC
 and how to use it, and subsequently beg for help understanding
 heisenleaks and weak references. ARC places a higher cognitive load on
 the programmer than a GC does. And Android runs just fine with GC'ed
 apps, but ARC guys don't want to talk about Google's successes there.


Links please.

Andrei



 Feb 03 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 18:53:39 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 4:49 PM, Adam Wilson wrote:

 That may be the case, but StackOverflow shows that ARC hasn't been
 panacea in Apple land either. Way to many people don't understand ARC
 and how to use it, and subsequently beg for help understanding
 heisenleaks and weak references. ARC places a higher cognitive load on
 the programmer than a GC does. And Android runs just fine with GC'ed
 apps, but ARC guys don't want to talk about Google's successes there.


 Links please.

 Andrei


Easier to link a couple of Google searches:
https://www.google.com/search?q=ios+stack+overflow+cyclic+reference&oq=stack+overflow+ios+cyclic+reference

Note that some of those are related to circular imports as well. But the  
problems definitely show up in that list.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 19:00:30 -0800, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 18:53:39 -0800, Andrei Alexandrescu  
 <SeeWebsiteForEmail erdani.org> wrote:


 On 2/3/14, 4:49 PM, Adam Wilson wrote:

 That may be the case, but StackOverflow shows that ARC hasn't been
 panacea in Apple land either. Way to many people don't understand ARC
 and how to use it, and subsequently beg for help understanding
 heisenleaks and weak references. ARC places a higher cognitive load on
 the programmer than a GC does. And Android runs just fine with GC'ed
 apps, but ARC guys don't want to talk about Google's successes there.


 Links please.

 Andrei


 Easier to link a couple of Google searches:
 https://www.google.com/search?q=ios+stack+overflow+cyclic+reference&oq=stack+overflow+ios+cyclic+reference

 Note that some of those are related to circular imports as well. But the  
 problems definitely show up in that list.


Note that ObjC has special syntax to handle weak pointers. It's not well  
understood by many. And for D it would create an explosion of leaks in  
existing code when there previously were none, due to the need to mark all  
weak-refs explicitly.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/3/2014 7:03 PM, Adam Wilson wrote:

 Note that ObjC has special syntax to handle weak pointers. It's not well
 understood by many.


Sounds like explicitly managed memory is hardly worse.



 Feb 03 2014



  parent reply "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 22:39:02 -0800, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 2/3/2014 7:03 PM, Adam Wilson wrote:

 Note that ObjC has special syntax to handle weak pointers. It's not well
 understood by many.


 Sounds like explicitly managed memory is hardly worse.


Well special syntax is needed to make ARC work because of the requirement  
to explicitly mark a pointer as a weak-reference. It would also mean that  
all existing D code would subtly break (leak memory) in any case where a  
weak-reference is required and was previously handled transparently by the  
GC. I won't say that manually managed memory is worse, because it does  
automatically clean up most of the trash, but ARC still allows to shot  
yourself in the foot easily enough.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



  parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Mon, 03 Feb 2014 22:51:39 -0800, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 22:39:02 -0800, Walter Bright  
 <newshound2 digitalmars.com> wrote:


 On 2/3/2014 7:03 PM, Adam Wilson wrote:

 Note that ObjC has special syntax to handle weak pointers. It's not  
 well
 understood by many.


 Sounds like explicitly managed memory is hardly worse.


 Well special syntax is needed to make ARC work because of the  
 requirement to explicitly mark a pointer as a weak-reference. It would  
 also mean that all existing D code would subtly break (leak memory) in  
 any case where a weak-reference is required and was previously handled  
 transparently by the GC. I won't say that manually managed memory is  
 worse, because it does automatically clean up most of the trash, but ARC  
 still allows to shot yourself in the foot easily enough.


Oops. I won't say that ARC is worse than Manual Memory Management.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 03 2014



prev sibling next sibling parent  Nick Sabalausky <SeeWebsiteToContactMe semitwist.com>  writes:


On 2/3/2014 7:49 PM, Adam Wilson wrote:

 And Android runs just fine with GC'ed
 apps, but ARC guys don't want to talk about Google's successes there.


FWIW, my Galaxy S3 randomly gets a lot of irritating second-or-two bouts 
of complete unresponsiveness.



 Feb 03 2014



prev sibling  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Mon, 03 Feb 2014 19:49:04 -0500, Adam Wilson <flyboynw gmail.com> wrote:


 On Mon, 03 Feb 2014 16:24:52 -0800, NoUseForAName <no spam.com> wrote:


 On Monday, 3 February 2014 at 23:00:23 UTC, woh wrote:

  ur right I never thought of that, I bet all them game devs never  
 thought of it either, they so dumb.  I bet they never tried to use a  
 GC, what fools!  Endless graphs of traced objects, oh yes oh yes!  It  
 only runs when I allocate, oh what a fool I've been, please castigate  
 me harder!


 Also people should consider that Apple (unlike C++ game devs) did not  
 have a tradition of contempt for GC. In fact they tried GC *before*  
 they switched to ARC. The pro-GC camp always likes to pretend that the  
 anti-GC one is just ignorant, rejecting GC based on prejudice not  
 experience but Apple rejected GC based on experience.

 GCed Objective-C did not allow them to deliver the user experience they  
 wanted (on mobile), because of the related latency issues. So they  
 switched to automated ref counting. It is not in question that ref  
 counting sacrifices throughput (compared to an advanced GC) but for  
 interactive, user facing applications latency is much more important.


 That may be the case, but StackOverflow shows that ARC hasn't been  
 panacea in Apple land either. Way to many people don't understand ARC  
 and how to use it, and subsequently beg for help understanding  
 heisenleaks and weak references. ARC places a higher cognitive load on  
 the programmer than a GC does. And Android runs just fine with GC'ed  
 apps, but ARC guys don't want to talk about Google's successes there.


Where you have to be cognizant is avoiding cycles. Plain and simple. And  
it's not that difficult. The compiler takes care of the rest. It's  
somewhat magical I suppose :) Managing your autorelease pools can make a  
difference, but is not necessarily critical.

I think when working on embedded systems, it is quite important to  
understand the limitations and strengths of the language/hardware you are  
using, much more so than on a full PC/Server. Where ARC really beats the  
GC is on memory usage. It's very very close to malloc/free usage.

I would love to see D attain an ARC system, especially if it can link  
seamlessly with Objective-C. What I don't know is if one can replace a GC  
with an ARC memory manager without having to port any high-level code. I  
don't think that is the case. Which leads me to think -- is it even  
possible to write druntime/phobos in an agnostic way? If it isn't, what  
subset can be done that way?

-Steve



 Feb 04 2014



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/4/14, 11:28 AM, Steven Schveighoffer wrote:

 Where you have to be cognizant is avoiding cycles. Plain and simple. And
 it's not that difficult.


Do you have evidence to back that up?

Andrei



 Feb 04 2014



  parent  "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Tue, 04 Feb 2014 14:31:13 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 2/4/14, 11:28 AM, Steven Schveighoffer wrote:

 Where you have to be cognizant is avoiding cycles. Plain and simple. And
 it's not that difficult.


 Do you have evidence to back that up?


Does personal experience count? I can say that the main project I have  
worked on in Objective-C uses manual reference counting, and when I have  
done newer, but shorter/smaller, projects in ARC, it is much much easier  
to deal with. Maybe my point of reference is tainted :)

If you use xcode to add member variables, it usually chooses the correct  
attributes for them (weak or not).

But the code the compiler generates for ARC is pretty good. It has quirky  
rules for naming methods and how you have to return objects (either  
autorelease or retained), and will not let you do the wrong thing. You  
also can't do ANY manual memory management in ARC code, it fails to  
compile.

Trickier aspects are dealing with C structs, which are NOT reference  
counted, so you have to deal with those manually using malloc/free (not  
really different from D).

I will say that it really helps to have an understanding of how ARC is  
actually implemented in the compiler (that is, the process it goes through  
to add/elide reference increments and decrements).

-Steve



 Feb 04 2014



prev sibling  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 4 February 2014 at 19:28:08 UTC, Steven Schveighoffer 
wrote:

 Where you have to be cognizant is avoiding cycles. Plain and 
 simple. And it's not that difficult. The compiler takes care of 
 the rest. It's somewhat magical I suppose :) Managing your 
 autorelease pools can make a difference, but is not necessarily 
 critical.


My experience is that, just like null reference, it is highly 
dependent on the type of code you are facing.

Typically, you'll have trouble with complex graph of objects.



 Feb 04 2014



  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Tue, 04 Feb 2014 15:10:48 -0500, deadalnix <deadalnix gmail.com> wrote:


 On Tuesday, 4 February 2014 at 19:28:08 UTC, Steven Schveighoffer wrote:

 Where you have to be cognizant is avoiding cycles. Plain and simple.  
 And it's not that difficult. The compiler takes care of the rest. It's  
 somewhat magical I suppose :) Managing your autorelease pools can make  
 a difference, but is not necessarily critical.


 My experience is that, just like null reference, it is highly dependent  
 on the type of code you are facing.

 Typically, you'll have trouble with complex graph of objects.


This is very true. I have only dealt with acyclic (except for parent  
pointers) graphs or linked-lists.

For specialized cases, you will have to NULL out the pointers to/from a  
node that has to be deleted (essentially a manual delete).

But it works VERY well for linked-lists/queues. My app is I/O intensive,  
so I have a lot of that.

-Steve



 Feb 04 2014



  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 4 February 2014 at 20:16:20 UTC, Steven Schveighoffer 
wrote:

 On Tue, 04 Feb 2014 15:10:48 -0500, deadalnix 
 <deadalnix gmail.com> wrote:


 On Tuesday, 4 February 2014 at 19:28:08 UTC, Steven 
 Schveighoffer wrote:

 Where you have to be cognizant is avoiding cycles. Plain and 
 simple. And it's not that difficult. The compiler takes care 
 of the rest. It's somewhat magical I suppose :) Managing your 
 autorelease pools can make a difference, but is not 
 necessarily critical.


 My experience is that, just like null reference, it is highly 
 dependent on the type of code you are facing.

 Typically, you'll have trouble with complex graph of objects.


 This is very true. I have only dealt with acyclic (except for 
 parent pointers) graphs or linked-lists.

 For specialized cases, you will have to NULL out the pointers 
 to/from a node that has to be deleted (essentially a manual 
 delete).

 But it works VERY well for linked-lists/queues. My app is I/O 
 intensive, so I have a lot of that.

 -Steve


Yes.

Also, another issue comes up when you share data across threads.

Core can share a cache line in read mode, but can't in write 
mode. That mean that updating the reference count will cause 
contention on the cache line (core will have to fight for the 
cache line ownership). That is why immutability + GC is so 
efficient in a highly concurrent system, and ref counting would 
ruin that.



 Feb 04 2014



 next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 21:02:37 UTC, deadalnix wrote:

 Core can share a cache line in read mode, but can't in write 
 mode.


Why not? Level 3 is shared, but it has a latency of 30-40 cycles 
or so.


 That mean that updating the reference count will cause 
 contention on the cache line (core will have to fight for the 
 cache line ownership).


If they access it simultaneously and it stays in the local caches.


 That is why immutability + GC is so efficient in a highly 
 concurrent system, and ref counting would ruin that.


If you blindly use ARC rather than sane RC. There is no reason to 
up the ref count if the data structure is "owned" while 
processing it. Which is a good reason to avoid ARC and use 
regular RC. If you know that the entire graph has a 1+ count, you 
don't need to do any ref counting while processing it.



 Feb 04 2014



  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Tue, 04 Feb 2014 16:37:20 -0500, Ola Fosheim Gr=C3=B8stad  =

<ola.fosheim.grostad+dlang gmail.com> wrote:


 If you blindly use ARC rather than sane RC. There is no reason to up t=


he  =


 ref count if the data structure is "owned" while processing it. Which =


is  =


 a good reason to avoid ARC and use regular RC. If you know that the  =



 entire graph has a 1+ count, you don't need to do any ref counting whi=


le  =


 processing it.


This is an important point. However, the default is to be safe, and good=
  =

enough for most. As long as there is a way to go into "I know what I'm  =

doing" mode for encapsulated low-level code, the default should be to do=
  =

ARC. ARC should handle simple cases, like if the reference only ever  =

exists as a stack local.

One could also say you could mark a memory region as no-scan if you know=
  =

that the data it points at will never be collected for its entire  =

existence in a GC environment.

I think some mix of GC and ref counting for D would be extremely useful.=


In my major Objective C project, I use manual reference counting, and I =
do  =

avoid extra retains for redundant pointers. I've also had a few bugs  =

because of that :)

-Steve



 Feb 04 2014



 next sibling parent reply Paulo Pinto <pjmlp progtools.org>  writes:


Am 04.02.2014 23:17, schrieb Steven Schveighoffer:

 On Tue, 04 Feb 2014 16:37:20 -0500, Ola Fosheim Grøstad
 <ola.fosheim.grostad+dlang gmail.com> wrote:


 If you blindly use ARC rather than sane RC. There is no reason to up
 the ref count if the data structure is "owned" while processing it.
 Which is a good reason to avoid ARC and use regular RC. If you know
 that the entire graph has a 1+ count, you don't need to do any ref
 counting while processing it.


 This is an important point. However, the default is to be safe, and good
 enough for most. As long as there is a way to go into "I know what I'm
 doing" mode for encapsulated low-level code, the default should be to do
 ARC. ARC should handle simple cases, like if the reference only ever
 exists as a stack local.

 One could also say you could mark a memory region as no-scan if you know
 that the data it points at will never be collected for its entire
 existence in a GC environment.

 I think some mix of GC and ref counting for D would be extremely useful.

 In my major Objective C project, I use manual reference counting, and I
 do avoid extra retains for redundant pointers. I've also had a few bugs
 because of that :)

 -Steve


How big is your team?

Based on my experience, this type of coding falls apart in the typical 
corporate team size.

--
Paulo



 Feb 04 2014



  parent  "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Tue, 04 Feb 2014 17:27:46 -0500, Paulo Pinto <pjmlp progtools.org>  =

wrote:


 Am 04.02.2014 23:17, schrieb Steven Schveighoffer:

 On Tue, 04 Feb 2014 16:37:20 -0500, Ola Fosheim Gr=C3=B8stad
 <ola.fosheim.grostad+dlang gmail.com> wrote:


 If you blindly use ARC rather than sane RC. There is no reason to up=







 the ref count if the data structure is "owned" while processing it.
 Which is a good reason to avoid ARC and use regular RC. If you know
 that the entire graph has a 1+ count, you don't need to do any ref
 counting while processing it.


 This is an important point. However, the default is to be safe, and g=




ood

 enough for most. As long as there is a way to go into "I know what I'=




m

 doing" mode for encapsulated low-level code, the default should be to=




 do

 ARC. ARC should handle simple cases, like if the reference only ever
 exists as a stack local.

 One could also say you could mark a memory region as no-scan if you k=




now

 that the data it points at will never be collected for its entire
 existence in a GC environment.

 I think some mix of GC and ref counting for D would be extremely usef=




ul.

 In my major Objective C project, I use manual reference counting, and=




 I

 do avoid extra retains for redundant pointers. I've also had a few bu=




gs

 because of that :)


 How big is your team?


2


 Based on my experience, this type of coding falls apart in the typical=


  =


 corporate team size.


It's not clear what "this type of coding" refers to, but I'll assume you=
  =

targeted the last paragraph.

If it's encapsulated (and it is), it should be relatively untouched. In =
 =

fact, I haven't touched the linked list part of it for a long time.

Note that the MRC was not really our choice, the project existed before =
 =

ARC did.

-Steve



 Feb 04 2014



prev sibling  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 22:17:40 UTC, Steven Schveighoffer 
wrote:

 This is an important point. However, the default is to be safe, 
 and good enough for most.


Yes, absolutely. I am all for progressive refinement, starting 
out with something conceptual  and very close to pseudocode, with 
all the convenience of high level programming. Then replace the 
parts that does not perform.

In Objective-C you don't really care that much about performance 
though, Cocoa provides very versatile libraries doing a lot of 
the work for you, but you have to go one step down to get speed 
without libraries. Inefficiencies are less noticeable in a heavy 
library/premade-components environment like Cococa than in a more 
barebones hit-the-iron-C-like approach.


 One could also say you could mark a memory region as no-scan if 
 you know that the data it points at will never be collected for 
 its entire existence in a GC environment.


Yes, I agree. You could have the same type in different pools and 
mark one of the pools as no-scan. You could have pool-local 
GC/segmented GC or even type-limited GC with whole program 
analysis (only collecting a specific class/subclasses).


 I think some mix of GC and ref counting for D would be 
 extremely useful.


Yes, I don't want to throw out GC, but I think it would be more 
valuable and more realistic to have C++ reference semantics than 
advanced compiler inferred ARC, though I don't mind ARC, I think 
it probably is harder to get right for a more pure C systems 
level language than for Objective-C with Cocoa.

So, I think ARC is right for Cocoa, but not so sure about how 
well it works in other environments.



 Feb 04 2014



prev sibling  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Tuesday, 4 February 2014 at 21:02:37 UTC, deadalnix wrote:

 Core can share a cache line in read mode, but can't in write 
 mode. That mean that updating the reference count will cause 
 contention on the cache line (core will have to fight for the 
 cache line ownership). That is why immutability + GC is so 
 efficient in a highly concurrent system, and ref counting would 
 ruin that.


Just curious, maybe I got it a bit wrong in my previous reply. It 
is indeed possible to set the memory type on some CPUs such that 
you don't get a penalty using some special MTRR (memory type 
range registers). If the memory type is write-back (regular 
memory) you get a 43 cycle latency if the cache line might be in 
multiple local caches and is not dirty (read), 60+ cycle latency 
if it is dirty (write), and 20-30 cycles latency on local miss, 
but L3 hit. (if I got that right) But does D actually make sure 
that immutable types sit in non-write-back memory marked by MTRR?



 Feb 04 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Wednesday, 5 February 2014 at 00:53:06 UTC, Ola Fosheim 
Grøstad wrote:

 But does D actually make sure that immutable types sit in 
 non-write-back memory marked by MTRR?


Guess I have to end this monologue by concluding that changing 
the PAT is the only reasonable way I have found so far for doing 
this. Which is too expensive for short-lived data.

So this strategy of turning immutable into non-WB memory type in 
order to avoid cache coherency latencies are only going to work 
in a OS-free environment AFAIK. (Like BareMetal OS)



 Feb 04 2014



prev sibling  parent  Paulo Pinto <pjmlp progtools.org>  writes:


Am 04.02.2014 01:24, schrieb NoUseForAName:

 On Monday, 3 February 2014 at 23:00:23 UTC, woh wrote:

  ur right I never thought of that, I bet all them game devs never
 thought of it either, they so dumb.  I bet they never tried to use a
 GC, what fools!  Endless graphs of traced objects, oh yes oh yes!  It
 only runs when I allocate, oh what a fool I've been, please castigate
 me harder!


 Also people should consider that Apple (unlike C++ game devs) did not
 have a tradition of contempt for GC. In fact they tried GC *before* they
 switched to ARC. The pro-GC camp always likes to pretend that the
 anti-GC one is just ignorant, rejecting GC based on prejudice not
 experience but Apple rejected GC based on experience.

 GCed Objective-C did not allow them to deliver the user experience they
 wanted (on mobile), because of the related latency issues. So they
 switched to automated ref counting. It is not in question that ref
 counting sacrifices throughput (compared to an advanced GC) but for
 interactive, user facing applications latency is much more important.


The reality of those hanging around Apple developer forums before ARC 
existed, was more related to crashes caused by Apple's implementation 
and their interaction with third party libraries. Specially the amount 
of special use cases one needed to take care of.

They *failed* to produce a stable GC implementation for Objective-C, 
that is the reason, not better performance.

--
Paulo



 Feb 04 2014



prev sibling  parent  Shammah Chancellor <anonymous coward.com>  writes:


On 2014-02-03 23:00:22 +0000, woh said:

 
 On Monday, 3 February 2014 at 21:42:59 UTC, Shammah Chancellor wrote:

 On 2014-02-01 07:35:44 +0000, Manu said:
 

 On 1 February 2014 16:26, Adam Wilson <flyboynw gmail.com> wrote:
 On Fri, 31 Jan 2014 21:29:04 -0800, Manu <turkeyman gmail.com> wrote:
 
 On 26 December 2012 00:48, Sven Over <dlang svenover.de> wrote:
 
  std.typecons.RefCounted!T
 
 core.memory.GC.disable();
 
 
 Wow. That was easy.
 
 I see, D's claim of being a multi-paradigm language is not false.
 
 
 It's not a realistic suggestion. Everything you want to link uses the GC,
 and the language its self also uses the GC. Unless you write software in
 complete isolation and forego many valuable features, it's not a solution.
 
 
  Phobos does rely on the GC to some extent. Most algorithms and ranges do
 not though.
 
 
 Running (library) code that was written with GC in mind and turning GC off
 doesn't sound ideal.
 
 But maybe this allows me to familiarise myself more with D. Who knows,
 maybe I can learn to stop worrying and love garbage collection.
 
 Thanks for your help!
 
 
 I've been trying to learn to love the GC for as long as I've been around
 here. I really wanted to break that mental barrier, but it hasn't happened.

 wishlist item for D is the ability to use a reference counted collector in
 place of the built-in GC.
 You're not alone :)
 
 I write realtime and memory-constrained software (console games), and for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable memory
 footprint of the application. You can't make any guarantees or predictions
 about the GC, which is fundamentally incompatible with realtime software.
 Language-level ARC would probably do quite nicely for the miscellaneous
 allocations. Obviously, bulk allocations are still usually best handled in
 a context sensitive manner; ie, regions/pools/freelists/whatever, but the
 convenience of the GC paradigm does offer some interesting and massively
 time-saving features to D.
 Everyone will always refer you to RefCounted, which mangles your types and
 pollutes your code, but aside from that, for ARC to be useful, it needs to
 be supported at the language-level, such that the language/optimiser is
 able to optimise out redundant incref/decref calls, and also that it is
 compatible with immutable (you can't manage a refcount if the object is
 immutable).
 
 The problem isn't GC's per se. But D's horribly naive implementation, 
 games are written on GC languages now all the time (Unity/.NET). And 
 let's be honest, games are kind of a speciality, games do things most 
 programs will never do.
 
 You might want to read the GC Handbook. GC's aren't bad, but most, like 
 the D GC, are just to simplistic for common usage today.
 
 Maybe a sufficiently advanced GC could address the performance 
 non-determinism to an acceptable level, but you're still left with the 
 memory non-determinism, and the conundrum that when your heap 
 approaches full (which is _always_ on a games console), the GC has to 
 work harder and harder, and more often to try and keep the tiny little 
 bit of overhead available.
 A GC heap by nature expects you to have lots of memory, and also lots 
 of FREE memory.
 
 No serious console game I'm aware of has ever been written in a 
 language with a GC. Casual games, or games that don't attempt to raise 
 the bar may get away with it, but that's not the industry I work in.


 
 You can always force the GC to run between cycles in your game, and
 turn off automatic sweeps.  This is how most games operate nowadays.
 It's also probably possible to create a drop-in replacement for the GC
 to do something else.   I could see if being *VERY* useful to make the
 GC take a compile-time parameter to select which GC engine is used.





   ur right I never thought of that, I bet all them game devs never 
 thought of it either, they so dumb.  I bet they never tried to use a 
 GC, what fools!  Endless graphs of traced objects, oh yes oh yes!  It 
 only runs when I allocate, oh what a fool I've been, please castigate 
 me harder!
 




First of all, bottom quoting is evil.  Second of all, your response is 
immature.  Thirdly, I am not adam.  And fourthly, I specifically 
mention that many games are currently using garbage collection.

-S.



 Feb 04 2014



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 2/3/2014 1:42 PM, Shammah Chancellor wrote:

 It's also probably
 possible to create a drop-in replacement for the GC to do something else.


It certainly is possible. There's nothing magic about the current GC, it's just 
library code.



 Feb 03 2014



  parent reply Shammah Chancellor <anonymous coward.com>  writes:


On 2014-02-04 06:50:51 +0000, Walter Bright said:


 On 2/3/2014 1:42 PM, Shammah Chancellor wrote:

 It's also probably
 possible to create a drop-in replacement for the GC to do something else.


 
 It certainly is possible. There's nothing magic about the current GC, 
 it's just library code.


Is it possible that we add some additional functionality to the GC API 
so that it could do ARC?  I took a look at it, and it seems that right 
now, there'd be no way to implement ARC.

-S.



 Feb 04 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 2/4/2014 4:10 AM, Shammah Chancellor wrote:

 On 2014-02-04 06:50:51 +0000, Walter Bright said:


 On 2/3/2014 1:42 PM, Shammah Chancellor wrote:

 It's also probably
 possible to create a drop-in replacement for the GC to do something else.


 It certainly is possible. There's nothing magic about the current GC, it's
 just library code.


 Is it possible that we add some additional functionality to the GC API so that
 it could do ARC?  I took a look at it, and it seems that right now, there'd be
 no way to implement ARC.


ARC would require compiler changes, so it cannot be drop in.



 Feb 04 2014



prev sibling next sibling parent reply Paulo Pinto <pjmlp progtools.org>  writes:


Am 01.02.2014 06:29, schrieb Manu:

 On 26 December 2012 00:48, Sven Over <dlang svenover.de
 <mailto:dlang svenover.de>> wrote:

         std.typecons.RefCounted!T

         core.memory.GC.disable();


     Wow. That was easy.

     I see, D's claim of being a multi-paradigm language is not false.


 It's not a realistic suggestion. Everything you want to link uses the
 GC, and the language its self also uses the GC. Unless you write
 software in complete isolation and forego many valuable features, it's
 not a solution.


         Phobos does rely on the GC to some extent. Most algorithms and
         ranges do not though.


     Running (library) code that was written with GC in mind and turning
     GC off doesn't sound ideal.

     But maybe this allows me to familiarise myself more with D. Who
     knows, maybe I can learn to stop worrying and love garbage collection.

     Thanks for your help!


 I've been trying to learn to love the GC for as long as I've been around
 here. I really wanted to break that mental barrier, but it hasn't happened.
 In fact, I am more than ever convinced that the GC won't do. My current

 collector in place of the built-in GC.
 You're not alone :)

 I write realtime and memory-constrained software (console games), and
 for me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable
 memory footprint of the application. You can't make any guarantees or
 predictions about the GC, which is fundamentally incompatible with
 realtime software.



Meanwhile Unity and similar engines are becoming widespread, with C++ 
being pushed all the way to the bottom on the stack.

At least from what I hear in the gaming communities I hop around.

What is your experience there?

--
Paulo



 Feb 01 2014



 next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com>  writes:


On 2/1/2014 3:20 AM, Paulo Pinto wrote:

 Meanwhile Unity and similar engines are becoming widespread, with C++
 being pushed all the way to the bottom on the stack.

 At least from what I hear in the gaming communities I hop around.

 What is your experience there?


It's not uncommon for Unity-engine games to drop down to C++ to optimize 
bottlenecks. Somewhat of a pain since you have to deal with the CLR <-> 
unmanaged marshalling mess. (I do really wish Unity would expose an 
optional C/C++ API so you wouldn't need a pointless CLR layer sandwiched 
between the C++-based engine and any non-CLR game code...)

Keep in mind too that while Unity is certainly powerful, the games 
written with it aren't usually the ultra-AAA titles (like Battlefield, 
Crysis or Last of Us, for example) where a major part of the business is 
"wow everyone with never-before-seen technical achievements or risk 
failing to recoup the gigantic development costs".

Come to think of it, I wonder what language Frostbite 3 uses for 
game-level "scripting" code. From what I've heard about them, Frostbite 
3 and Unreal Engine 4 both sound like they have some notable 
similarities with Unity3D (from the standpoint of the user-experience 
for game developers), although AIUI Unreal Engine 4 still uses C++ for 
game code (unless the engine user wants to to do lua or something on 
their own). I imagine Frostbite's probably the same, C++, but I haven't 
actually heard anything.



 Feb 01 2014



  parent reply "develop32" <develop32 gmail.com>  writes:


On Saturday, 1 February 2014 at 12:29:10 UTC, Nick Sabalausky 
wrote:

 Come to think of it, I wonder what language Frostbite 3 uses 
 for game-level "scripting" code. From what I've heard about 
 them, Frostbite 3 and Unreal Engine 4 both sound like they have 
 some notable similarities with Unity3D (from the standpoint of 
 the user-experience for game developers), although AIUI Unreal 
 Engine 4 still uses C++ for game code (unless the engine user 
 wants to to do lua or something on their own). I imagine 
 Frostbite's probably the same, C++, but I haven't actually 
 heard anything.


Frostbite has an option for both Lua and C++. C++ is the 
preferred one.

Unreal Engine went from Unrealscript to C++ and everyone 
applauded that.



 Feb 01 2014



 next sibling parent reply Paulo Pinto <pjmlp progtools.org>  writes:


Am 01.02.2014 13:35, schrieb develop32:

 On Saturday, 1 February 2014 at 12:29:10 UTC, Nick Sabalausky wrote:

 Come to think of it, I wonder what language Frostbite 3 uses for
 game-level "scripting" code. From what I've heard about them,
 Frostbite 3 and Unreal Engine 4 both sound like they have some notable
 similarities with Unity3D (from the standpoint of the user-experience
 for game developers), although AIUI Unreal Engine 4 still uses C++ for
 game code (unless the engine user wants to to do lua or something on
 their own). I imagine Frostbite's probably the same, C++, but I
 haven't actually heard anything.


 Frostbite has an option for both Lua and C++. C++ is the preferred one.

 Unreal Engine went from Unrealscript to C++ and everyone applauded that.


Do you have any experience with Project Anarchy?

--
Paulo



 Feb 01 2014



  parent  "develop32" <develop32 gmail.com>  writes:


On Saturday, 1 February 2014 at 16:37:23 UTC, Paulo Pinto wrote:


 Do you have any experience with Project Anarchy?

 --
 Paulo


No I have not. I haven't got a smartphone yet, so I guess that 
explains it.



 Feb 01 2014



prev sibling  parent  Manu <turkeyman gmail.com>  writes:


On 1 February 2014 22:35, develop32 <develop32 gmail.com> wrote:


 On Saturday, 1 February 2014 at 12:29:10 UTC, Nick Sabalausky wrote:


 Come to think of it, I wonder what language Frostbite 3 uses for
 game-level "scripting" code. From what I've heard about them, Frostbite 3
 and Unreal Engine 4 both sound like they have some notable similarities
 with Unity3D (from the standpoint of the user-experience for game
 developers), although AIUI Unreal Engine 4 still uses C++ for game code
 (unless the engine user wants to to do lua or something on their own). I
 imagine Frostbite's probably the same, C++, but I haven't actually heard
 anything.


 Frostbite has an option for both Lua and C++. C++ is the preferred one.

 Unreal Engine went from Unrealscript to C++ and everyone applauded that.


Correct, this is the general trend I see. idTech also uses C/C++ DLL's for
'scripting'.

At Remedy, we were keen to migrate our scripts to native code in the same
way, except using D instead of C++. The proof of concept is solid.



 Feb 01 2014



prev sibling  parent  Manu <turkeyman gmail.com>  writes:


On 1 February 2014 18:20, Paulo Pinto <pjmlp progtools.org> wrote:


 Am 01.02.2014 06:29, schrieb Manu:


 On 26 December 2012 00:48, Sven Over <dlang svenover.de
 <mailto:dlang svenover.de>> wrote:

         std.typecons.RefCounted!T

         core.memory.GC.disable();


     Wow. That was easy.

     I see, D's claim of being a multi-paradigm language is not false.


 It's not a realistic suggestion. Everything you want to link uses the
 GC, and the language its self also uses the GC. Unless you write
 software in complete isolation and forego many valuable features, it's
 not a solution.


         Phobos does rely on the GC to some extent. Most algorithms and
         ranges do not though.


     Running (library) code that was written with GC in mind and turning
     GC off doesn't sound ideal.

     But maybe this allows me to familiarise myself more with D. Who
     knows, maybe I can learn to stop worrying and love garbage collection.

     Thanks for your help!


 I've been trying to learn to love the GC for as long as I've been around
 here. I really wanted to break that mental barrier, but it hasn't
 happened.
 In fact, I am more than ever convinced that the GC won't do. My current

 collector in place of the built-in GC.
 You're not alone :)

 I write realtime and memory-constrained software (console games), and
 for me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable
 memory footprint of the application. You can't make any guarantees or
 predictions about the GC, which is fundamentally incompatible with
 realtime software.



 Meanwhile Unity and similar engines are becoming widespread, with C++
 being pushed all the way to the bottom on the stack.

 At least from what I hear in the gaming communities I hop around.

 What is your experience there?


Unity is indeed popular, for casual/indy games.
AAA/'big games' show no signs of moving away from C++. The 'next gen' has
enough memory for GC (still can't afford the time though), but handhelds
and small devices are a bigger market these days.
It's true that there are less 'big games' on handhelds, which are the
future of resource-limited devices, but I think that rift is closing
quickly.



 Feb 01 2014



prev sibling  parent reply "JR" <zorael gmail.com>  writes:


On Saturday, 1 February 2014 at 05:36:44 UTC, Manu wrote:

 I write realtime and memory-constrained software (console 
 games), and for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the 
 unknowable memory
 footprint of the application. You can't make any guarantees or 
 predictions
 about the GC, which is fundamentally incompatible with realtime 
 software.


(tried to manually fix ugly linebreaks here, so apologies if it 
turns out even worse.)

(Maybe this would be better posted in D.learn; if so I'll 
crosspost.)

In your opinion, of how much value would deadlining be? As in, 
"okay handyman, you may sweep the floor now BUT ONLY FOR 6 
MILLISECONDS; whatever's left after that you'll have to take care 
of next time, your pride as a professional Sweeper be damned"?

It obviously doesn't address memory footprint, but you would get 
the illusion of determinism in cases similar to where 
race-to-idle approaches work. Inarguably, this wouldn't apply if 
the goal is to render as many frames per second as possible, such 
as for non-console shooters where tearing is not a concern but 
latency is very much so.

I'm very much a layman in this field, but I'm trying to soak up 
as much knowledge as possible, and most of it from threads like 
these. To my uneducated eyes, an ARC collector does seem like the 
near-ideal solution -- assuming, as always, the code is written 
with the GC in mind. But am I right in gathering that it solves 
two-thirds of the problem? You don't need to scan the managed 
heap, but when memory is actually freed is still 
non-deterministic and may incur pauses, though not necessarily a 
program-wide stop. Aye?

At the same time, Lucarella's dconf slides were very, very 
attractive. I gather that allocations themselves may become 
slower with a concurrent collector, but collection times in 
essence become non-issues. Technically parallelism doesn't equate 
to free CPU time; but that it more or less *is* assuming there is 
a cores/thread to spare. Wrong?

Lastly, am I right in understanding precise collectors as 
identical to the stop-the-world collector we currently have, but 
with a smarter allocation scheme resulting in a smaller managed 
heap to scan? With the additional bonus of less false pointers. 
If so, this would seem like a good improvement to the current 
implementation, with the next increment in the same direction 
being a generational gc.

I would *dearly* love to have concurrency in whatever we end up 
with, though. For a multi-core personal computer threads are free 
lunches, or close enough so. Concurrentgate and all that jazz.



 Feb 01 2014



 next sibling parent reply "develop32" <develop32 gmail.com>  writes:


On Saturday, 1 February 2014 at 12:04:56 UTC, JR wrote:

 In your opinion, of how much value would deadlining be? As in, 
 "okay handyman, you may sweep the floor now BUT ONLY FOR 6 
 MILLISECONDS;


Unrelated to the whole post note: 6ms is too high, I would allow 
max 1ms a frame for a game I'm developing and even that is 
somewhat painful.

If I turn on GC on the game I'm making, it takes 80ms every frame 
when memory usage is 800mb (shown on task manager). This was 
actually surprising, previously it was 6ms, good that I made my 
engine GC independent.



 Feb 01 2014



 next sibling parent reply "JR" <zorael gmail.com>  writes:


On Saturday, 1 February 2014 at 12:20:33 UTC, develop32 wrote:

 If I turn on GC on the game I'm making, it takes 80ms every 
 frame when memory usage is 800mb (shown on task manager). This 
 was actually surprising, previously it was 6ms, good that I 
 made my engine GC independent.


You wouldn't happen to have a blog post someplace with 
reflections on the steps you took to avoid the GC, by any chance? 
Hint hint! I'm genuinely curious.

Or is it merely a matter of manual allocation?



 Feb 01 2014



  parent  "develop32" <develop32 gmail.com>  writes:


On Saturday, 1 February 2014 at 13:16:02 UTC, JR wrote:

 On Saturday, 1 February 2014 at 12:20:33 UTC, develop32 wrote:

 If I turn on GC on the game I'm making, it takes 80ms every 
 frame when memory usage is 800mb (shown on task manager). This 
 was actually surprising, previously it was 6ms, good that I 
 made my engine GC independent.


 You wouldn't happen to have a blog post someplace with 
 reflections on the steps you took to avoid the GC, by any 
 chance? Hint hint! I'm genuinely curious.

 Or is it merely a matter of manual allocation?


I don't have a blog...

It is mostly a matter of memory reuse (LOTS of it) and manual 
freeing when needed.



 Feb 01 2014



prev sibling  parent reply "Francesco Cattoglio" <francesco.cattoglio gmail.com>  writes:


On Saturday, 1 February 2014 at 12:20:33 UTC, develop32 wrote:

 On Saturday, 1 February 2014 at 12:04:56 UTC, JR wrote:

 In your opinion, of how much value would deadlining be? As in, 
 "okay handyman, you may sweep the floor now BUT ONLY FOR 6 
 MILLISECONDS;


 Unrelated to the whole post note: 6ms is too high, I would 
 allow max 1ms a frame for a game I'm developing and even that 
 is somewhat painful.

 If I turn on GC on the game I'm making, it takes 80ms every 
 frame when memory usage is 800mb (shown on task manager). This 
 was actually surprising, previously it was 6ms, good that I 
 made my engine GC independent.


Wow! That's interesting! What kind of game are you working on?



 Feb 01 2014



  parent  "develop32" <develop32 gmail.com>  writes:


On Saturday, 1 February 2014 at 14:24:17 UTC, Francesco Cattoglio 
wrote:

 On Saturday, 1 February 2014 at 12:20:33 UTC, develop32 wrote:

 On Saturday, 1 February 2014 at 12:04:56 UTC, JR wrote:

 In your opinion, of how much value would deadlining be? As 
 in, "okay handyman, you may sweep the floor now BUT ONLY FOR 
 6 MILLISECONDS;


 Unrelated to the whole post note: 6ms is too high, I would 
 allow max 1ms a frame for a game I'm developing and even that 
 is somewhat painful.

 If I turn on GC on the game I'm making, it takes 80ms every 
 frame when memory usage is 800mb (shown on task manager). This 
 was actually surprising, previously it was 6ms, good that I 
 made my engine GC independent.


 Wow! That's interesting! What kind of game are you working on?


Don't want to reveal too much now, its early in development, plan 
to have a proper trailer few months later.

Its an economical space sim, set in an asteroid field of 3000km 
radius, filled with 100k asteroids, hundred stations and few 
hundred AI ships. All of the characters/stations are simulated 
simultaneously.

http://imgur.com/QAKzJWb



 Feb 01 2014



prev sibling next sibling parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 04:04:54 -0800, JR <zorael gmail.com> wrote:


 On Saturday, 1 February 2014 at 05:36:44 UTC, Manu wrote:

 I write realtime and memory-constrained software (console games), and  
 for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable  
 memory
 footprint of the application. You can't make any guarantees or  
 predictions
 about the GC, which is fundamentally incompatible with realtime  
 software.


 (tried to manually fix ugly linebreaks here, so apologies if it turns  
 out even worse.)

 (Maybe this would be better posted in D.learn; if so I'll crosspost.)

 In your opinion, of how much value would deadlining be? As in, "okay  
 handyman, you may sweep the floor now BUT ONLY FOR 6 MILLISECONDS;  
 whatever's left after that you'll have to take care of next time, your  
 pride as a professional Sweeper be damned"?

 It obviously doesn't address memory footprint, but you would get the  
 illusion of determinism in cases similar to where race-to-idle  
 approaches work. Inarguably, this wouldn't apply if the goal is to  
 render as many frames per second as possible, such as for non-console  
 shooters where tearing is not a concern but latency is very much so.

 I'm very much a layman in this field, but I'm trying to soak up as much  
 knowledge as possible, and most of it from threads like these. To my  
 uneducated eyes, an ARC collector does seem like the near-ideal solution  
 -- assuming, as always, the code is written with the GC in mind. But am  
 I right in gathering that it solves two-thirds of the problem? You don't  
 need to scan the managed heap, but when memory is actually freed is  
 still non-deterministic and may incur pauses, though not necessarily a  
 program-wide stop. Aye?


It would only not be a program wide stop if you had multiple threads  
running, otherwise yes, ARC can still Stop-The-World for a  
non-deterministic period of time, because you the programmer have no idea  
how long that collection cycle will last. Also note that this is just a  
shuffling of where the collection happens. In D's GC a collection can  
happen any time you attempt to allocate, whereas in ARC you eagerly  
collect when you delete, because if you don't eagerly collect you'll have  
a memory leak. Also you can't make ARC concurrent.


 At the same time, Lucarella's dconf slides were very, very attractive. I  
 gather that allocations themselves may become slower with a concurrent  
 collector, but collection times in essence become non-issues.  
 Technically parallelism doesn't equate to free CPU time; but that it  
 more or less *is* assuming there is a cores/thread to spare. Wrong?


Essentially yes, concurrency does get rid of MOST of the STW aspects of  
GC's. However, most modern GC's are generational and typically their are  
one or two generations that are not collected concurrently. In .NET both  
Generations 0 and 1 are not collected concurrently because they can be  
collected very quickly, more quickly than cost of enabling concurrent  
collection support on allocation.

For example, I use WPF for almost every project I do at work. WPF is a  
retained-mode GUI API based on DirectX 9, and it has a 60FPS render speed  
requirement. The only time I have seen the rendering bog down due to the  
GC is when there are a LOT of animations starting and stopping. Otherwise  
it's almost always because of WPF's horrifically naive rendering code.


 Lastly, am I right in understanding precise collectors as identical to  
 the stop-the-world collector we currently have, but with a smarter  
 allocation scheme resulting in a smaller managed heap to scan? With the  
 additional bonus of less false pointers. If so, this would seem like a  
 good improvement to the current implementation, with the next increment  
 in the same direction being a generational gc.


Correct, precision won't change the STW nature of the GC, just make it so  
there is much less to scan/collect in the first place, and believe it or  
not, the difference can be huge. See Rainer Schutze's for more information  
on a precise collector in D: http://dconf.org/2013/talks/schuetze.html


 I would *dearly* love to have concurrency in whatever we end up with,  
 though. For a multi-core personal computer threads are free lunches, or  
 close enough so. Concurrentgate and all that jazz.


You and me both, this is the way all GC's are headed, I can't think of a  
major GC-language that doesn't have a Concurrent-Generational-Incremental  
GC. :-)

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



prev sibling  parent reply Manu <turkeyman gmail.com>  writes:


On 1 February 2014 22:04, JR <zorael gmail.com> wrote:


 On Saturday, 1 February 2014 at 05:36:44 UTC, Manu wrote:


 I write realtime and memory-constrained software (console games), and for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable memory
 footprint of the application. You can't make any guarantees or predictions
 about the GC, which is fundamentally incompatible with realtime software.


 (tried to manually fix ugly linebreaks here, so apologies if it turns out
 even worse.)

 (Maybe this would be better posted in D.learn; if so I'll crosspost.)

 In your opinion, of how much value would deadlining be? As in, "okay
 handyman, you may sweep the floor now BUT ONLY FOR 6 MILLISECONDS;
 whatever's left after that you'll have to take care of next time, your
 pride as a professional Sweeper be damned"?


This has been my only suggestion for years. Although 6ms is way too much
(almost half a frame), 500us is more realistic (a little over 3% of a
frame, still quite a lot of time).

It obviously doesn't address memory footprint, but you would get the

 illusion of determinism in cases similar to where race-to-idle approaches
 work. Inarguably, this wouldn't apply if the goal is to render as many
 frames per second as possible, such as for non-console shooters where
 tearing is not a concern but latency is very much so.


If it were running a small amount every frame, maybe the memory footprint
would stay within a predictable level... don't know. I'd be interested in
experimenting with this, but as far as I know, nobody knows how to do it.

I'm very much a layman in this field, but I'm trying to soak up as much

 knowledge as possible, and most of it from threads like these. To my
 uneducated eyes, an ARC collector does seem like the near-ideal solution --
 assuming, as always, the code is written with the GC in mind. But am I
 right in gathering that it solves two-thirds of the problem? You don't need
 to scan the managed heap, but when memory is actually freed is still
 non-deterministic and may incur pauses, though not necessarily a
 program-wide stop. Aye?


I'm not sure what you mean. If you mean pauses because it's simply doing
work to free things, then that's deterministically issued workload just
like anything else and can (should) be budgeted. It doesn't disappear in a
GC context, it just happens in bursts at unpredictable times.
Like you say, it doesn't stop the world, no need for that. Additionally, it
leaves room for case-by-case flexibility in destruction approach. If
something takes some time to destroy, and you're okay to have it hang
around for a while, you can easily plonk it in a dead list waiting for some
idle time to work through it; thus truly allows the opportunity to use idle
time to clean stuff up since you don't need to scan the heap every time, a
list of things waiting to be destroyed is already known.
You could also add dead objects to a list which can be processed by another
thread (without stopping all threads).
There is far more opportunity to implement the destruction pattern that
suits your application, and for those not interested in implementing their
own destruction patterns, it would be easy to offer a library function
setDestructionPattern(patternType); which would configure a particular
policy application-wide, and they'd never have to think about it again.

At the same time, Lucarella's dconf slides were very, very attractive. I

 gather that allocations themselves may become slower with a concurrent
 collector, but collection times in essence become non-issues. Technically
 parallelism doesn't equate to free CPU time; but that it more or less *is*
 assuming there is a cores/thread to spare. Wrong?


IIRC, it assumes a particular operating system, and required significant
resource overhead. I recall being impressed, but concluding that it wasn't
a solution that could be merged into D across the board.
I'd need to go back and revisit the slides...

Lastly, am I right in understanding precise collectors as identical to the

 stop-the-world collector we currently have, but with a smarter allocation
 scheme resulting in a smaller managed heap to scan? With the additional
 bonus of less false pointers. If so, this would seem like a good
 improvement to the current implementation, with the next increment in the
 same direction being a generational gc.


I think the reduction in heap to scan is due to only following what are
known to be pointers, but I don't think that reduces the scan volume, the
memory footprint of your app is the same (more because there's new
metadata), but it wouldn't chase false pointers anymore.
I recall most agreed that it was a good improvement (although it
benchmarked something like 5% slower), I would appreciate it simply for the
improved precision... but it doesn't solve the problem in any way. The
practical application for me is that it wouldn't leak (much) memory.

I would *dearly* love to have concurrency in whatever we end up with,

 though. For a multi-core personal computer threads are free lunches, or
 close enough so. Concurrentgate and all that jazz.


The kicker for me with the whole GC thing, is that as long as I've been
talking about it around here (among a whole bunch of experts), I've been
saying that an acceptable GC would require to 1. not stop the world, 2.
support incremental collection/time-slicing, so I can budget it a maximum
amount of time per frame.
I'm yet to hear anyone suggest how they can even IMAGINE writing a
collector like that. As far as I can tell, the consensus is, that it's
impossible.
Certainly, nobody has ever given any suggestions how it could possibly be
done in the future, or steps in that direction, which says a lot to me.
I've abandoned the idea as unrealistic fantasy.



 Feb 01 2014



  parent  "Adam Wilson" <flyboynw gmail.com>  writes:


On Sat, 01 Feb 2014 17:58:01 -0800, Manu <turkeyman gmail.com> wrote:


 On 1 February 2014 22:04, JR <zorael gmail.com> wrote:


 On Saturday, 1 February 2014 at 05:36:44 UTC, Manu wrote:


 I write realtime and memory-constrained software (console games), and  
 for
 me, I think the biggest issue that can never be solved is the
 non-deterministic nature of the collect cycles, and the unknowable  
 memory
 footprint of the application. You can't make any guarantees or  
 predictions
 about the GC, which is fundamentally incompatible with realtime  
 software.


 (tried to manually fix ugly linebreaks here, so apologies if it turns  
 out
 even worse.)

 (Maybe this would be better posted in D.learn; if so I'll crosspost.)

 In your opinion, of how much value would deadlining be? As in, "okay
 handyman, you may sweep the floor now BUT ONLY FOR 6 MILLISECONDS;
 whatever's left after that you'll have to take care of next time, your
 pride as a professional Sweeper be damned"?


 This has been my only suggestion for years. Although 6ms is way too much
 (almost half a frame), 500us is more realistic (a little over 3% of a
 frame, still quite a lot of time).

 It obviously doesn't address memory footprint, but you would get the

 illusion of determinism in cases similar to where race-to-idle  
 approaches
 work. Inarguably, this wouldn't apply if the goal is to render as many
 frames per second as possible, such as for non-console shooters where
 tearing is not a concern but latency is very much so.


 If it were running a small amount every frame, maybe the memory footprint
 would stay within a predictable level... don't know. I'd be interested in
 experimenting with this, but as far as I know, nobody knows how to do it.

 I'm very much a layman in this field, but I'm trying to soak up as much

 knowledge as possible, and most of it from threads like these. To my
 uneducated eyes, an ARC collector does seem like the near-ideal  
 solution --
 assuming, as always, the code is written with the GC in mind. But am I
 right in gathering that it solves two-thirds of the problem? You don't  
 need
 to scan the managed heap, but when memory is actually freed is still
 non-deterministic and may incur pauses, though not necessarily a
 program-wide stop. Aye?


 I'm not sure what you mean. If you mean pauses because it's simply doing
 work to free things, then that's deterministically issued workload just
 like anything else and can (should) be budgeted. It doesn't disappear in  
 a
 GC context, it just happens in bursts at unpredictable times.
 Like you say, it doesn't stop the world, no need for that. Additionally,  
 it
 leaves room for case-by-case flexibility in destruction approach. If
 something takes some time to destroy, and you're okay to have it hang
 around for a while, you can easily plonk it in a dead list waiting for  
 some
 idle time to work through it; thus truly allows the opportunity to use  
 idle
 time to clean stuff up since you don't need to scan the heap every time,  
 a
 list of things waiting to be destroyed is already known.
 You could also add dead objects to a list which can be processed by  
 another
 thread (without stopping all threads).
 There is far more opportunity to implement the destruction pattern that
 suits your application, and for those not interested in implementing  
 their
 own destruction patterns, it would be easy to offer a library function
 setDestructionPattern(patternType); which would configure a particular
 policy application-wide, and they'd never have to think about it again.

 At the same time, Lucarella's dconf slides were very, very attractive. I

 gather that allocations themselves may become slower with a concurrent
 collector, but collection times in essence become non-issues.  
 Technically
 parallelism doesn't equate to free CPU time; but that it more or less  
 *is*
 assuming there is a cores/thread to spare. Wrong?


 IIRC, it assumes a particular operating system, and required significant
 resource overhead. I recall being impressed, but concluding that it  
 wasn't
 a solution that could be merged into D across the board.
 I'd need to go back and revisit the slides...

 Lastly, am I right in understanding precise collectors as identical to  
 the

 stop-the-world collector we currently have, but with a smarter  
 allocation
 scheme resulting in a smaller managed heap to scan? With the additional
 bonus of less false pointers. If so, this would seem like a good
 improvement to the current implementation, with the next increment in  
 the
 same direction being a generational gc.


 I think the reduction in heap to scan is due to only following what are
 known to be pointers, but I don't think that reduces the scan volume, the
 memory footprint of your app is the same (more because there's new
 metadata), but it wouldn't chase false pointers anymore.
 I recall most agreed that it was a good improvement (although it
 benchmarked something like 5% slower), I would appreciate it simply for  
 the
 improved precision... but it doesn't solve the problem in any way. The
 practical application for me is that it wouldn't leak (much) memory.

 I would *dearly* love to have concurrency in whatever we end up with,

 though. For a multi-core personal computer threads are free lunches, or
 close enough so. Concurrentgate and all that jazz.


 The kicker for me with the whole GC thing, is that as long as I've been
 talking about it around here (among a whole bunch of experts), I've been
 saying that an acceptable GC would require to 1. not stop the world, 2.
 support incremental collection/time-slicing, so I can budget it a maximum
 amount of time per frame.
 I'm yet to hear anyone suggest how they can even IMAGINE writing a
 collector like that. As far as I can tell, the consensus is, that it's
 impossible.
 Certainly, nobody has ever given any suggestions how it could possibly be
 done in the future, or steps in that direction, which says a lot to me.
 I've abandoned the idea as unrealistic fantasy.


It is impossible with a conservative GC such as the current D  
implementation. Boehm tried with his C++ collector but could never get  
beyond the basic Mark-LazySweep because C++ enforces a conservative  
mindset.

When you have precision a number of options open up. Firstly, you can now  
parallelize all phases because pointers are precisely known. And second,  
you can start to do things like moving (a prereq for generational  
collection) because you don't have to pin things that you conservatively  
think are pointers. Now that you have a generational collector, it's  
trivial to do an incremental collection on one or more generations.

So therefore precision is the first step, and more to the point it's  
already been done in D, VisualD uses a precise version of the stock D GC  
and special build of the compiler with precise information emitted at  
compile time. That was what Rainer's talk was about at DConf last year.  
Getting this merged into the mainline will mean that there is finally  
enough information for us to go nuts building an actually useful GC in D,  
but until then you really can't progress beyond what we have right now.

It's quite possible, it's just a matter of getting the worked vetted at  
merged.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator



 Feb 01 2014



prev sibling  parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com>  writes:


 std.typecons.RefCounted!T


Does this work equally well with T being both a value objects 
(struct) and reference objects (class)?



 Feb 04 2014



  parent  "Brad Anderson" <eco gnuk.net>  writes:


On Tuesday, 4 February 2014 at 20:53:50 UTC, Nordlöw wrote:

 std.typecons.RefCounted!T


 Does this work equally well with T being both a value objects 
 (struct) and reference objects (class)?


RefCounted doesn't support classes yet but that's just because 
nobody has taken the time to implement it. There also needs to be 
a WeakRef added in order to round out the smart pointer support.

Once that's done Phobos will at least have parity with C++ (it 
already has Unique).



 Feb 04 2014



prev sibling  parent reply Stewart Gordon <smjg_1998 yahoo.com>  writes:


On 25/12/2012 14:13, Sven Over wrote:
<snip>

 Also, garbage collection often tends to increase the memory footprint of
 your software.


ISTM you can't really make a like-for-like comparison, since non-GC 
programs vary a lot in how much they leak memory, which becomes almost a 
non-issue when you have GC.

I say "almost" because GC systems often aren't perfect.  For instance, a 
GC might mistake some sequence of four or eight bytes for a pointer and 
keep hold of the allocated memory it is pointing to.  Java has been 
known to leak memory in certain circumstances, such as not disposing a 
window when you've finished with it, or creating a thread but never 
starting it.

But there is something called packratting, which is a mistake at the 
code level of keeping a pointer hanging around for longer than necessary 
and therefore preventing whatever it's pointing to from being GC'd.


 You may be able to avoid this with some expert knowledge
 about the garbage collector, but this of course invalidates one of the
 main arguments in favour of GC, which is less complexity. I'd say that
 if you want to write memory- and CPU-efficient code, then you'll need a
 certain amount of understanding of your memory management. Be it your
 manual memory management, or the inner workings of your garbage collector.

 To cut things short, even after giving it a lot of thought I still feel
 uncomfortable about garbage collection.


That means you're uncomfortable about reference-counted smart pointers, 
because these are a form of garbage collection. :)


 I'm wondering whether it is
 viable to use the same smart pointer techniques as in C++: is there an
 implementation already of such smart pointers? Can I switch off GC in my
 D programs altogether? How much does the standard library rely on GC?
 I.e. would I have to write a alternative standard library to do this?


Good luck at getting your alternative library accepted as part of the D 
standard. :)

Seriously though, it's as much built-in language functionality as the 
standard library that relies on GC.  Any of the following will allocate 
memory from the GC heap:

- create anything with the 'new' keyword
- increase the length of a dynamic array
- concatenate arrays
- duplicate an array with .dup or .idup
- put data into an associative array

Which means sooner or later the garbage will need to be collected, 
unless the program is such that all memory allocation will be done at 
startup and then released only on exit.  Of course, whether the program 
can reasonably be written in this way depends on the nature of the program.


 There is one blog post I found interesting to read in this context,
 which also underlines my worries: http://3d.benjamin-thaut.de/?p=20


An interesting set of observations.  Looking at this one:
"Calls to the druntime invariant handler are emitted in release build 
also and there is no way to turn them off. Even if the class does not 
have any invariants the invariant handler will always be called, walk 
the class hirarchy and generate multiple cache misses without actually 
doing anything."

There seem to be two bugs here: that these calls are generated even in 
release mode, and that they are generated even if the class has no 
invariants.  One of us needs to experiment with this a bit more and get 
these issues filed in Bugzilla.

Stewart.



 Jan 01 2013



  parent  "Mehrdad" <wfunction hotmail.com>  writes:


On Tuesday, 1 January 2013 at 16:31:55 UTC, Stewart Gordon wrote:

 But there is something called packratting, which is a mistake 
 at the code level of keeping a pointer hanging around for 
 longer than necessary and therefore preventing whatever it's 
 pointing to from being GC'd.



I've heard it called "midlife crisis" :)



 Jan 02 2013