www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - RFC: moving forward with nogc Phobos

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Back when I've first introduced RCString I hinted that we have a larger 
strategy in mind. Here it is.

The basic tenet of the approach is to reckon and act on the fact that 
memory allocation (the subject of allocators) is an entirely distinct 
topic from memory management, and more generally resource management. 
This clarifies that it would be wrong to approach alternatives to GC in 
Phobos by means of allocators. GC is not only an approach to memory 
allocation, but also an approach to memory management. Reducing it to 
either one is a mistake. In hindsight this looks rather obvious but it 
has caused me and many people better than myself a lot of headache.

That said allocators are nice to have and use, and I will definitely 
follow up with std.allocator. However, std.allocator is not the key to a 
 nogc Phobos.

Nor are ranges. There is an attitude that either output ranges, or input 
ranges in conjunction with lazy computation, would solve the issue of 
creating garbage. 
https://github.com/D-Programming-Language/phobos/pull/2423 is a good 
illustration of the latter approach: a range would be lazily created by 
chaining stuff together. A range-based approach would take us further 
than the allocators, but I see the following issues with it:

(a) the whole approach doesn't stand scrutiny for non-linear outputs, 
e.g. outputting some sort of associative array or really any composite 
type quickly becomes tenuous either with an output range (eager) or with 
exposing an input range (lazy);

(b) makes the style of programming without GC radically different, and 
much more cumbersome, than programming with GC; as a consequence, 
programmers who consider changing one approach to another, or 
implementing an algorithm neutral to it, are looking at a major rewrite;

(c) would make D/ nogc a poor cousin of C++. This is quite out of 
character; technically, I have long gotten used to seeing most elaborate 
C++ code like poor emulation of simple D idioms. But C++ has spent years 
and decades taking to perfection an approach without a tracing garbage 
collector. A departure from that would need to be superior, and that 
doesn't seem to be the case with range-based approaches.

===========

Now that we clarified that these existing attempts are not going to work 
well, the question remains what does. For Phobos I'm thinking of 
defining and using three policies:

enum MemoryManagementPolicy { gc, rc, mrc }
immutable
     gc = ResourceManagementPolicy.gc,
     rc = ResourceManagementPolicy.rc,
     mrc = ResourceManagementPolicy.mrc;

The three policies are:

(a) gc is the classic garbage-collected style of management;

(b) rc is a reference-counted style still backed by the GC, i.e. the GC 
will still be able to pick up cycles and other kinds of leaks.

(c) mrc is a reference-counted style backed by malloc.

(It should be possible to collapse rc and mrc together and make the 
distinction dynamically, at runtime. I'm distinguishing them statically 
here for expository purposes.)

The policy is a template parameter to functions in Phobos (and 
elsewhere), and informs the functions e.g. what types to return. Consider:

auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2 ext)
if (...)
{
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
}

On the caller side:

auto p1 = setExtension("hello", ".txt"); // fine, use gc
auto p2 = setExtension!gc("hello", ".txt"); // same
auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc

So by default it's going to continue being business as usual, but 
certain functions will allow passing in a (defaulted) policy for memory 
management.

Destroy!


Andrei
Sep 29 2014
next sibling parent reply Daniel Kozak via Digitalmars-d <digitalmars-d puremagic.com> writes:
V Mon, 29 Sep 2014 03:49:52 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
napsáno:


 Back when I've first introduced RCString I hinted that we have a
 larger strategy in mind. Here it is.
 
 The basic tenet of the approach is to reckon and act on the fact that 
 memory allocation (the subject of allocators) is an entirely distinct 
 topic from memory management, and more generally resource management. 
 This clarifies that it would be wrong to approach alternatives to GC
 in Phobos by means of allocators. GC is not only an approach to
 memory allocation, but also an approach to memory management.
 Reducing it to either one is a mistake. In hindsight this looks
 rather obvious but it has caused me and many people better than
 myself a lot of headache.
 
 That said allocators are nice to have and use, and I will definitely 
 follow up with std.allocator. However, std.allocator is not the key
 to a  nogc Phobos.
 
 Nor are ranges. There is an attitude that either output ranges, or
 input ranges in conjunction with lazy computation, would solve the
 issue of creating garbage. 
 https://github.com/D-Programming-Language/phobos/pull/2423 is a good 
 illustration of the latter approach: a range would be lazily created
 by chaining stuff together. A range-based approach would take us
 further than the allocators, but I see the following issues with it:
 
 (a) the whole approach doesn't stand scrutiny for non-linear outputs, 
 e.g. outputting some sort of associative array or really any
 composite type quickly becomes tenuous either with an output range
 (eager) or with exposing an input range (lazy);
 
 (b) makes the style of programming without GC radically different,
 and much more cumbersome, than programming with GC; as a consequence, 
 programmers who consider changing one approach to another, or 
 implementing an algorithm neutral to it, are looking at a major
 rewrite;
 
 (c) would make D/ nogc a poor cousin of C++. This is quite out of 
 character; technically, I have long gotten used to seeing most
 elaborate C++ code like poor emulation of simple D idioms. But C++
 has spent years and decades taking to perfection an approach without
 a tracing garbage collector. A departure from that would need to be
 superior, and that doesn't seem to be the case with range-based
 approaches.
 
 ===========
 
 Now that we clarified that these existing attempts are not going to
 work well, the question remains what does. For Phobos I'm thinking of 
 defining and using three policies:
 
 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
      gc = ResourceManagementPolicy.gc,
      rc = ResourceManagementPolicy.rc,
      mrc = ResourceManagementPolicy.mrc;
 
 The three policies are:
 
 (a) gc is the classic garbage-collected style of management;
 
 (b) rc is a reference-counted style still backed by the GC, i.e. the
 GC will still be able to pick up cycles and other kinds of leaks.
 
 (c) mrc is a reference-counted style backed by malloc.
 
 (It should be possible to collapse rc and mrc together and make the 
 distinction dynamically, at runtime. I'm distinguishing them
 statically here for expository purposes.)
 
 The policy is a template parameter to functions in Phobos (and 
 elsewhere), and informs the functions e.g. what types to return.
 Consider:
 
 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path,
 R2 ext) if (...)
 {
      static if (mmp == gc) alias S = string;
      else alias S = RCString;
      S result;
      ...
      return result;
 }
 
 On the caller side:
 
 auto p1 = setExtension("hello", ".txt"); // fine, use gc
 auto p2 = setExtension!gc("hello", ".txt"); // same
 auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc
 
 So by default it's going to continue being business as usual, but 
 certain functions will allow passing in a (defaulted) policy for
 memory management.
 
 Destroy!
 
 
 Andrei


I would add something like this:

 DefaultMemoryManagementPolicy(rc)
module A;

void main() {
    auto p1 = setExtension("hello", ".txt"); // use rc
}
Sep 29 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 4:03 AM, Daniel Kozak via Digitalmars-d wrote:

 I would add something like this:

  DefaultMemoryManagementPolicy(rc)
 module A;

 void main() {
      auto p1 = setExtension("hello", ".txt"); // use rc
 }


(please don't overquote!)

Yah, I realized I forgot to mention this: if we play our cards right, a 
lot of code will build in both approaches to memory management by just 
flipping a switch. In particular, the switch can be defaulted to 
something else.

I was thinking of leaving it to the user:

module A;
immutable myMMP = rc;

void main() {
     auto p1 = setExtension!myMMP("hello", ".txt");
}


Andrei
Sep 29 2014
prev sibling next sibling parent "Daniel N" <ufo orbiting.us> writes:
On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu 
wrote:

 Back when I've first introduced RCString I hinted that we have 
 a larger strategy in mind. Here it is.

 The policy is a template parameter to functions in Phobos (and 
 elsewhere), and informs the functions e.g. what types to 
 return. Consider:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 
 path, R2 ext)
 if (...)
 {
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
 }


How about having something like ResourceManagementPolicy.infer, 
which under the hood could work something like below... you could 
combine it with your original suggestion, with an overridable 
MemoryManagementPolicy(just removed it to make the example 
shorter)

auto setExtension(R1, R2)(R1 path, R2 ext)
if (...)
{
     static if(functionAttributes!(__traits(parent, setExtension)) 
& FunctionAttribute.nogc)
       alias S = RCString;
     else
       alias S = string;
     ...
     return result;
}

Daniel N
Sep 29 2014
prev sibling next sibling parent reply "eles" <eles eles.com> writes:
On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu 
wrote:



 entirely distinct topic


Finally!
Sep 29 2014
parent "eles" <eles eles.com> writes:
On Monday, 29 September 2014 at 11:37:00 UTC, eles wrote:

 On Monday, 29 September 2014 at 10:49:53 UTC, Andrei 
 Alexandrescu wrote:



 entirely distinct topic


 Finally!


Sorry, enthusiasm. I really think this is the key for doing the 
management of all resources in the right way. For me, the memory 
should be seen as a resource that simply happens to have the 
possibility of being manageable in a more flexible way and with 
specific constraints.

For example, with respect to other kind of resources, you could 
use a lazy approach to deallocate memory, as unlike many other 
resources memory is like money: is fungible [1]. Other resources 
are not. OTOH, the memory comes with some of its own quirks, such 
as the cycles (these could be, in theory, possible for other kind 
of resources, but are exceptions).

Memory management is not necessarily deterministic neither. Other 
resources might require determinism, however.

[1] http://en.wikipedia.org/wiki/Fungibility
Sep 29 2014
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu 
wrote:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 
 path, R2 ext)
 if (...)
 {
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
 }


Is this practically feasible without blowing up Phobos several 
times in size and complexity?

And I'm not sure adding a template parameter to every function is 
going to work well, what with all the existing template 
parameters - especially the optional ones.
Sep 29 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 5:06 AM, Vladimir Panteleev wrote:

 On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2
 ext)
 if (...)
 {
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
 }


 Is this practically feasible without blowing up Phobos several times in
 size and complexity?


I believe so. For the most part implementations will be identical - just 
look at the RCString primitives, which are virtually the same as string's.


 And I'm not sure adding a template parameter to every function is going
 to work well, what with all the existing template parameters -
 especially the optional ones.


Not all functions, just those that allocate. I agree there will be a few 
decisions to be made there.


Andrei
Sep 29 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
Any assumption that library code can go away with some set of 
pre-defined allocation strategies is crap. This whole discussion 
was about how important it is to move allocation decisions to 
user code (ranges are just one tool to achieve that, Don has been 
presenting examples of how we do that with plain arrays in DConf 
2014 talk).

In that regard allocators + ranges are still the way to go in my 
opinion. Yes, sometimes those result in very hard to use API - 
providing GC-heavy but friendly alternatives for those shouldn't 
do any harm. But in general full decoupling of algorithms from 
allocations is necessary. If that makes D poor cousin of C++ we 
may have a learn few tricks from C++.
Sep 29 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 5:29 AM, Dicebot wrote:

 Any assumption that library code can go away with some set of
 pre-defined allocation strategies is crap. This whole discussion was
 about how important it is to move allocation decisions to user code
 (ranges are just one tool to achieve that, Don has been presenting
 examples of how we do that with plain arrays in DConf 2014 talk).


That's making exactly the confusion I was - that memory allocation 
strategy is the same as memory management strategy.


 In that regard allocators + ranges are still the way to go in my
 opinion. Yes, sometimes those result in very hard to use API - providing
 GC-heavy but friendly alternatives for those shouldn't do any harm. But
 in general full decoupling of algorithms from allocations is necessary.
 If that makes D poor cousin of C++ we may have a learn few tricks from C++.


As long as things are trivial they can be done with relative ease, 
albeit with more pain. But consider e.g. the recent JSON library by 
Sönke. It needs to create a lookup data structure and return things like 
strings from it. What primitives do you think could it define?


Andrei
Sep 29 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 29 September 2014 at 15:18:40 UTC, Andrei Alexandrescu 
wrote:

 On 9/29/14, 5:29 AM, Dicebot wrote:

 Any assumption that library code can go away with some set of
 pre-defined allocation strategies is crap. This whole 
 discussion was
 about how important it is to move allocation decisions to user 
 code
 (ranges are just one tool to achieve that, Don has been 
 presenting
 examples of how we do that with plain arrays in DConf 2014 
 talk).


 That's making exactly the confusion I was - that memory 
 allocation strategy is the same as memory management strategy.


Yes but neither decision belongs to library code except for very 
rare cases.


 In that regard allocators + ranges are still the way to go in 
 my
 opinion. Yes, sometimes those result in very hard to use API - 
 providing
 GC-heavy but friendly alternatives for those shouldn't do any 
 harm. But
 in general full decoupling of algorithms from allocations is 
 necessary.
 If that makes D poor cousin of C++ we may have a learn few 
 tricks from C++.


 As long as things are trivial they can be done with relative 
 ease, albeit with more pain. But consider e.g. the recent JSON 
 library by Sönke. It needs to create a lookup data structure 
 and return things like strings from it. What primitives do you 
 think could it define?


Sounds like it may have to define own kind of allocator with 
certain implementation restrictions (and implement it in terms of 
GC by default). I have not actually read the code for that 
proposal so hard to guess. Will need to do it if it really 
matters.
Sep 29 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 8:53 AM, Dicebot wrote:

 On Monday, 29 September 2014 at 15:18:40 UTC, Andrei Alexandrescu wrote:

 On 9/29/14, 5:29 AM, Dicebot wrote:

 Any assumption that library code can go away with some set of
 pre-defined allocation strategies is crap. This whole discussion was
 about how important it is to move allocation decisions to user code
 (ranges are just one tool to achieve that, Don has been presenting
 examples of how we do that with plain arrays in DConf 2014 talk).


 That's making exactly the confusion I was - that memory allocation
 strategy is the same as memory management strategy.


 Yes but neither decision belongs to library code except for very rare
 cases.


You just assert it, so all I can say is "I understand you believe this". 
I've motivated my argument. You may want to do the same for yours.


 In that regard allocators + ranges are still the way to go in my
 opinion. Yes, sometimes those result in very hard to use API - providing
 GC-heavy but friendly alternatives for those shouldn't do any harm. But
 in general full decoupling of algorithms from allocations is necessary.
 If that makes D poor cousin of C++ we may have a learn few tricks
 from C++.


 As long as things are trivial they can be done with relative ease,
 albeit with more pain. But consider e.g. the recent JSON library by
 Sönke. It needs to create a lookup data structure and return things
 like strings from it. What primitives do you think could it define?


 Sounds like it may have to define own kind of allocator with certain
 implementation restrictions (and implement it in terms of GC by
 default). I have not actually read the code for that proposal so hard to
 guess. Will need to do it if it really matters.


So you don't have an answer. And again you are confusing memory 
allocation with memory management.

I have sketched an approach that works and will take us to Phobos being 
most transparently usable with tracing collection or with reference 
counting. Part of that is RCString (and generally reference counted 
slices and hashtables), and another part is the  refcounted attribute 
for classes. I will push it through. If you have any objections, it 
would be great if you argued them properly.


Thanks,

Andrei
Sep 29 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 29 September 2014 at 17:04:54 UTC, Andrei Alexandrescu 
wrote:

 Yes but neither decision belongs to library code except for 
 very rare
 cases.


 You just assert it, so all I can say is "I understand you 
 believe this". I've motivated my argument. You may want to do 
 the same for yours.


I probably have missed the part with arguments :) Your reasoning 
is not fundamentally different from "GC should be enough" but 
extended to several options from single one.

My argument is simple - one can't forsee everything. I remember 
reading book of one guy who has been advocating thing called 
"policy-based design", you may know him ;) Was quite impressed 
with the simple but practical basic idea - decoupling parts of 
the implementation that are not inherently related.


 So you don't have an answer. And again you are confusing memory 
 allocation with memory management.


Yes, sorry, I don't have an answer. Or time do deeply dive into 
the code unless it is really important or my direct 
responsibility.

Unfortunately, I don't see an answer how your proposal fits our 
code either. Most of Sociomantic code relies on using arrays as 
ref arguments to avoid creating of new GC roots (no, we don't 
need/want to switch to ARC). This was several times called as the 
reason why Phobos in its current shape is largely unusable for 
out needs even when D2 switch is finished. I don't see how 
proposal in original post changes that.
Sep 29 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 10:19 AM, Dicebot wrote:

 On Monday, 29 September 2014 at 17:04:54 UTC, Andrei Alexandrescu wrote:

 Yes but neither decision belongs to library code except for very rare
 cases.


 You just assert it, so all I can say is "I understand you believe
 this". I've motivated my argument. You may want to do the same for yours.


 I probably have missed the part with arguments :)


No problem, let me paste it again:


 The basic tenet of the approach is to reckon and act on the fact that memory
allocation (the subject of allocators) is an entirely distinct topic from
memory management, and more generally resource management. This clarifies that
it would be wrong to approach alternatives to GC in Phobos by means of
allocators. GC is not only an approach to memory allocation, but also an
approach to memory management. Reducing it to either one is a mistake. In
hindsight this looks rather obvious but it has caused me and many people better
than myself a lot of headache.

 That said allocators are nice to have and use, and I will definitely follow up
with std.allocator. However, std.allocator is not the key to a  nogc Phobos.

 Nor are ranges. There is an attitude that either output ranges, or input
ranges in conjunction with lazy computation, would solve the issue of creating
garbage. https://github.com/D-Programming-Language/phobos/pull/2423 is a good
illustration of the latter approach: a range would be lazily created by
chaining stuff together. A range-based approach would take us further than the
allocators, but I see the following issues with it:

 (a) the whole approach doesn't stand scrutiny for non-linear outputs, e.g.
outputting some sort of associative array or really any composite type quickly
becomes tenuous either with an output range (eager) or with exposing an input
range (lazy);

 (b) makes the style of programming without GC radically different, and much
more cumbersome, than programming with GC; as a consequence, programmers who
consider changing one approach to another, or implementing an algorithm neutral
to it, are looking at a major rewrite;

 (c) would make D/ nogc a poor cousin of C++. This is quite out of character;
technically, I have long gotten used to seeing most elaborate C++ code like
poor emulation of simple D idioms. But C++ has spent years and decades taking
to perfection an approach without a tracing garbage collector. A departure from
that would need to be superior, and that doesn't seem to be the case with
range-based approaches.


=================


 Your reasoning is not
 fundamentally different from "GC should be enough" but extended to
 several options from single one.


Where's RC in the "GC should be enough"?


 My argument is simple - one can't forsee everything. I remember reading
 book of one guy who has been advocating thing called "policy-based
 design", you may know him ;) Was quite impressed with the simple but
 practical basic idea - decoupling parts of the implementation that are
 not inherently related.


Totally. Then it would be great if you trusted the guy when he makes a 
judgment call in which reasonable people may disagree.

There are many memory /allocation/ policies but precious few memory 
/management/ policies. I only know "manual", "scoped", "reference 
counted", and "tracing" based on... the last 50 years of software 
development.


 So you don't have an answer. And again you are confusing memory
 allocation with memory management.


 Yes, sorry, I don't have an answer. Or time do deeply dive into the code
 unless it is really important or my direct responsibility.

 Unfortunately, I don't see an answer how your proposal fits our code
 either. Most of Sociomantic code relies on using arrays as ref arguments
 to avoid creating of new GC roots (no, we don't need/want to switch to
 ARC). This was several times called as the reason why Phobos in its
 current shape is largely unusable for out needs even when D2 switch is
 finished. I don't see how proposal in original post changes that.


Passing arrays by reference is plenty adequate with all memory 
management strategies. You'll need to wait and see how the proposal 
changes that, but if you naysay, back it up.


Andrei
Sep 29 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 29 September 2014 at 22:18:38 UTC, Andrei Alexandrescu 
wrote:

 Passing arrays by reference is plenty adequate with all memory 
 management strategies. You'll need to wait and see how the 
 proposal changes that, but if you naysay, back it up.


Resisting to go on meaningless argument on other points, this 
pretty much says that focus on things that are important for me 
is abandoned in favor of something that mostly doesn't matter. Am 
I supposed to be happy? :) Am I supposed to be twice as happy 
when you propose to close pull requests that do help because of 
this proposal?

I am waiting for what comes next but right now "not impressed" is 
most optimistic way to put this. Sorry :(
Sep 29 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 3:43 PM, Dicebot wrote:

 On Monday, 29 September 2014 at 22:18:38 UTC, Andrei Alexandrescu wrote:

 Passing arrays by reference is plenty adequate with all memory
 management strategies. You'll need to wait and see how the proposal
 changes that, but if you naysay, back it up.


 Resisting to go on meaningless argument on other points, this pretty
 much says that focus on things that are important for me is abandoned in
 favor of something that mostly doesn't matter. Am I supposed to be
 happy? :) Am I supposed to be twice as happy when you propose to close
 pull requests that do help because of this proposal?

 I am waiting for what comes next but right now "not impressed" is most
 optimistic way to put this. Sorry :(


I trust you'll be. -- Andrei
Sep 29 2014
prev sibling parent "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Monday, 29 September 2014 at 12:29:33 UTC, Dicebot wrote:

 Any assumption that library code can go away with some set of 
 pre-defined allocation strategies is crap. This whole 
 discussion was about how important it is to move allocation 
 decisions to user code (ranges are just one tool to achieve 
 that, Don has been presenting examples of how we do that with 
 plain arrays in DConf 2014 talk).


I think the key to this sort of issue is to try and get as much 
functionality in Phobos marked  nogc as possible. After that, 
building new library-like functionality into a DUB package that 
assumes  nogc and only uses the  nogc code in Phobos would be the 
next step. Should that get to a state where it's popular and 
supported, pulling it in as std.nogc.* might make sense, but 
trying to redo Phobos as a manual memory collection library is 
infeasible.

Were I your company, I'd start working on leading such an effort.

Unlike Tango, I don't think a development like this would split 
the community nor the community's resources in a useless fashion.
Sep 29 2014
prev sibling next sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 29.09.2014 12:49, schrieb Andrei Alexandrescu:

 [...]

 The three policies are:

 (a) gc is the classic garbage-collected style of management;

 (b) rc is a reference-counted style still backed by the GC, i.e. the GC
 will still be able to pick up cycles and other kinds of leaks.

 (c) mrc is a reference-counted style backed by malloc.

 (It should be possible to collapse rc and mrc together and make the
 distinction dynamically, at runtime. I'm distinguishing them statically
 here for expository purposes.)

 ...


Personally, I would go just for (b) with compiler support for 
increment/decrement removal, as I think it will be too complex having to 
support everything and this will complicate all libraries.

Anyway, that was just my 0.02€. Stepping out the thread as I just toy 
around with D and cannot add much more to the discussion.

--
Paulo
Sep 29 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 10:16 AM, Paulo Pinto wrote:

 Personally, I would go just for (b) with compiler support for
 increment/decrement removal, as I think it will be too complex having to
 support everything and this will complicate all libraries.


Compiler already knows (after inlining) that ++i and --i cancel each 
other, so we should be in good shape there. -- Andrei
Sep 29 2014
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 29 Sep 2014 15:04:03 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 9/29/14, 10:16 AM, Paulo Pinto wrote:

 Personally, I would go just for (b) with compiler support for
 increment/decrement removal, as I think it will be too complex having to
 support everything and this will complicate all libraries.


=20
 Compiler already knows (after inlining) that ++i and --i cancel each=20
 other, so we should be in good shape there. -- Andrei


That helps with very small, inlined functions until Marc
Sch=C3=BCtz's work on borrowed pointers makes it redundant by
unifying scoped copies of GC, RC and stack pointers.
In any case inc/dec elision is an optimization and and not an
enabling feature. It sure is on the radar and can be improved
later on.

--=20
Marco
Sep 30 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 30 September 2014 08:04, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/29/14, 10:16 AM, Paulo Pinto wrote:

 Personally, I would go just for (b) with compiler support for
 increment/decrement removal, as I think it will be too complex having to
 support everything and this will complicate all libraries.



 Compiler already knows (after inlining) that ++i and --i cancel each other,
 so we should be in good shape there. -- Andrei


The compiler doesn't know that MyLibrary_AddRef(Thing *t); and
MyLibrary_DecRef(Thing *t); cancel eachother out though...
rc needs primitives that the compiler understands implicitly, so that
rc logic can be more complex than ++i/--i;
Sep 30 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 1 October 2014 at 01:26:45 UTC, Manu via
Digitalmars-d wrote:

 On 30 September 2014 08:04, Andrei Alexandrescu via 
 Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 9/29/14, 10:16 AM, Paulo Pinto wrote:

 Personally, I would go just for (b) with compiler support for
 increment/decrement removal, as I think it will be too 
 complex having to
 support everything and this will complicate all libraries.



 Compiler already knows (after inlining) that ++i and --i 
 cancel each other,
 so we should be in good shape there. -- Andrei


 The compiler doesn't know that MyLibrary_AddRef(Thing *t); and
 MyLibrary_DecRef(Thing *t); cancel eachother out though...
 rc needs primitives that the compiler understands implicitly, 
 so that
 rc logic can be more complex than ++i/--i;


Even with simply i++ and i--, the information that they always go
by pair is lost on the compiler in many cases.
Sep 30 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-29 12:49, Andrei Alexandrescu wrote:


 Now that we clarified that these existing attempts are not going to work
 well, the question remains what does. For Phobos I'm thinking of
 defining and using three policies:

 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
      gc = ResourceManagementPolicy.gc,
      rc = ResourceManagementPolicy.rc,
      mrc = ResourceManagementPolicy.mrc;

 The three policies are:

 (a) gc is the classic garbage-collected style of management;

 (b) rc is a reference-counted style still backed by the GC, i.e. the GC
 will still be able to pick up cycles and other kinds of leaks.

 (c) mrc is a reference-counted style backed by malloc.

 (It should be possible to collapse rc and mrc together and make the
 distinction dynamically, at runtime. I'm distinguishing them statically
 here for expository purposes.)

 The policy is a template parameter to functions in Phobos (and
 elsewhere), and informs the functions e.g. what types to return. Consider:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2 ext)
 if (...)
 {
      static if (mmp == gc) alias S = string;
      else alias S = RCString;
      S result;
      ...
      return result;
 }

 On the caller side:

 auto p1 = setExtension("hello", ".txt"); // fine, use gc
 auto p2 = setExtension!gc("hello", ".txt"); // same
 auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc


How does allocators fit in this? Will it be an additional argument to 
the function. Or a separate stack that one can push and pop allocators to?

-- 
/Jacob Carlborg
Sep 29 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 10:25 AM, Jacob Carlborg wrote:

 How does allocators fit in this? Will it be an additional argument to
 the function. Or a separate stack that one can push and pop allocators to?


There would be one allocator per thread (changeable) deferring to a 
global interlocked allocator. Most algorithms would just use whatever 
allocator is installed.

I know the notion of a thread-local and then global allocator is liable 
to cause some an apoplexy attack. But it's time to model things as they 
are - memory is a global resource and it ought to be treated as such. No 
need to pass allocators around except for special cases.


Andrei
Sep 29 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Mon, 29 Sep 2014 15:11:26 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 9/29/14, 10:25 AM, Jacob Carlborg wrote:

 How does allocators fit in this? Will it be an additional argument
 to the function. Or a separate stack that one can push and pop
 allocators to?


 
 There would be one allocator per thread (changeable) deferring to a 
 global interlocked allocator. Most algorithms would just use whatever 
 allocator is installed.
 
 I know the notion of a thread-local and then global allocator is
 liable to cause some an apoplexy attack. But it's time to model
 things as they are - memory is a global resource and it ought to be
 treated as such. No need to pass allocators around except for special
 cases.
 
 
 Andrei
 



 No need to pass allocators around except for special
 cases.


So you propose RC + global/thread local allocators as the solution for
all memory related problems as 'memory management is not allocation'.
And you claim that using output ranges / providing buffers / allocators
is not an option because it only works in some special cases?

What if I don't want automated memory _management_? What if I want a
function to use a stack buffer? Or if I want to free manually?

If I want std.string.toStringz to put the result into a temporary stack
buffer your solution doesn't help at all. Passing an ouput range,
allocator or buffer would all solve this.
Sep 30 2014
next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Tuesday, 30 September 2014 at 08:34:26 UTC, Johannes Pfau 
wrote:

 What if I don't want automated memory _management_? What if I 
 want a
 function to use a stack buffer? Or if I want to free manually?


Agreed. This is the common case we need to solve for, but this is 
memory allocation, not management. I'm not sure where manual 
management fits into Andrei's scheme. Andrei, could you give an 
example of, e.g. how toStringz would work with a stack buffer in 
your proposed scheme?

Another thought: if we use a template parameter, what's the story 
for virtual functions (e.g. Object.toString)? They can't be 
templated.
Sep 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/30/14, 3:41 AM, Peter Alexander wrote:

 On Tuesday, 30 September 2014 at 08:34:26 UTC, Johannes Pfau wrote:

 What if I don't want automated memory _management_? What if I want a
 function to use a stack buffer? Or if I want to free manually?


 Agreed. This is the common case we need to solve for, but this is memory
 allocation, not management. I'm not sure where manual management fits
 into Andrei's scheme. Andrei, could you give an example of, e.g. how
 toStringz would work with a stack buffer in your proposed scheme?


There would be no possibility to do that. I mean it's not there but it 
can be added e.g. as a "manual" option of performing memory management. 
The "manual" overloads for functions would require an output range 
parameter. Not all functions might support a "manual" option - that'd be 
rejected statically.


 Another thought: if we use a template parameter, what's the story for
 virtual functions (e.g. Object.toString)? They can't be templated.


Good point. We need to think about that.


Andrei
Sep 30 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 30/09/14 14:29, Andrei Alexandrescu wrote:


 Good point. We need to think about that.


Weren't all methods in Object supposed to be lifted out from Object anyway?

-- 
/Jacob Carlborg
Sep 30 2014
parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, September 30, 2014 15:18:17 Jacob Carlborg via Digitalmars-d 
wrote:

 On 30/09/14 14:29, Andrei Alexandrescu wrote:

 Good point. We need to think about that.


 Weren't all methods in Object supposed to be lifted out from Object anyway?


Yes, but not much work has been done on it, and the little work that has been
done is blocked by at least one compiler bug:

https://issues.dlang.org/show_bug.cgi?id=12537

- Jonathan M Davis
Oct 28 2014
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Tue, 30 Sep 2014 05:29:55 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 

 Another thought: if we use a template parameter, what's the story
 for virtual functions (e.g. Object.toString)? They can't be
 templated.


 
 Good point. We need to think about that.
 


Passing buffers or sink delegates (like we already do for toString) is
possible for some functions. For toString it works fine. Then implement
to!RCString(object) using the toString(sink delegate) overload.

For all other functions RC is indeed difficult, probably only possible
with different manually written overloads (and a dummy parameter as we
can't overload on return type)?
Sep 30 2014
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tuesday, 30 September 2014 at 08:34:26 UTC, Johannes Pfau 
wrote:

 What if I don't want automated memory _management_? What if I 
 want a
 function to use a stack buffer? Or if I want to free manually?

 If I want std.string.toStringz to put the result into a 
 temporary stack
 buffer your solution doesn't help at all. Passing an ouput 
 range,
 allocator or buffer would all solve this.


I don't understand, why wouldn't you be able to temporarily set 
the thread-local allocator to use the stack buffer, and restore 
it once done?
Sep 30 2014
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 30 Sep 2014 10:47:54 +0000
schrieb "Vladimir Panteleev" <vladimir thecybershadow.net>:


 On Tuesday, 30 September 2014 at 08:34:26 UTC, Johannes Pfau 
 wrote:

 What if I don't want automated memory _management_? What if I 
 want a
 function to use a stack buffer? Or if I want to free manually?

 If I want std.string.toStringz to put the result into a 
 temporary stack
 buffer your solution doesn't help at all. Passing an ouput 
 range,
 allocator or buffer would all solve this.


 
 I don't understand, why wouldn't you be able to temporarily set 
 the thread-local allocator to use the stack buffer, and restore 
 it once done?


That's possible but insanely dangerous in case you forget to reset the
thread allocator. Also storing stack pointers in global state (even
thread-local) is dangerous, for example interaction with fibers could
lead to bugs, etc. (What if I set the allocator to a stack allocator
and call a function which yields from a Fiber?).

You also loose all possibilities to use 'scope' or a similar mechanism
to prevent escaping a stack pointer. 

Also a stack buffer is not a complete allocator, but in some
cases like toStringz it works even better than allocators (less
overhead as you know the required buffer size before calling toStringz
and there's only one allocation)

And it is a hack. Of course you can provide a wrapper which does
oldAlloc = threadLocalAllocator;
threadLocalAllocator = stackbuf;
func();
scope(exit)
    threadLocalAllocator = oldAlloc;

But how could anybody think this is good API design? I think I'd rather
fork the required Phobos functions instead of using such a wrapper.
Sep 30 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 30 September 2014 at 12:02:10 UTC, Johannes Pfau 
wrote:

 That's possible but insanely dangerous in case you forget to 
 reset the
 thread allocator. Also storing stack pointers in global state 
 (even
 thread-local) is dangerous, for example interaction with fibers 
 could
 lead to bugs, etc. (What if I set the allocator to a stack 
 allocator
 and call a function which yields from a Fiber?).

 You also loose all possibilities to use 'scope' or a similar 
 mechanism
 to prevent escaping a stack pointer.


Yes, I agree. One option would be to have thread-local region 
allocator that can only be used for "scoped" allocation. That is, 
only for allocations that are not assigned to globals or can get 
stuck in fibers and that are returned to the calling function. 
That way the context can free the region when done and you can 
get away with little allocation overhead if used prudently.

I also don't agree with the sentiment that allocation/management 
can be kept fully separate. If you have a region allocator that 
is refcounted it most certainly is interrelated with a fairly 
tight coupling.

Also the idea exposed in this thread that release()/retain() is 
purely arithmetic and can be optimized as such is quite wrong. 
retain() is conceptually a locking construct on a memory region 
that prevents reuse. I've made a case for TSX, but one can 
probably come up with other multi-threaded examples.

These hacks are not making D more attractive to people who find 
C++ lacking in elegance.

Actually, creating a phobos light with nothrow, nogc, a light 
runtime and basic building blocks such as intrinsics to build 
your own RC with compiler support sounds like a more interesting 
option.

I am really not interested in library provided allocators or RC. 
If I am not going to use malloc/GC then I want to write my own 
and have dedicated allocators for the most common objects.

I think it is quite reasonable that people who want to take the 
difficult road of not using GC at all also have to do some extra 
work, but provide a clean slate to work from!
Sep 30 2014
next sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 30 September 2014 at 12:32:08 UTC, Ola Fosheim 
Grøstad wrote:

 On Tuesday, 30 September 2014 at 12:02:10 UTC, Johannes Pfau 
 wrote:

 ...



 Also the idea exposed in this thread that release()/retain() 
 is


 purely arithmetic and can be optimized as such is quite wrong. 
 retain() is conceptually a locking construct on a memory region 
 that prevents reuse. I've made a case for TSX, but one can 
 probably come up with other multi-threaded examples.


It works when two big ifs come together.

- inside the same scope (e.g. function level)

- when the referece is not shared between threads.

While it is of limited applicability, Objective-C (and eventually 
Swift) codebases prove it helps in most real life use cases.

--
Paulo
Sep 30 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 30 September 2014 at 12:51:25 UTC, Paulo  Pinto wrote:

 It works when two big ifs come together.

 - inside the same scope (e.g. function level)

 - when the referece is not shared between threads.

 While it is of limited applicability, Objective-C (and 
 eventually Swift) codebases prove it helps in most real life 
 use cases.


But Objective-C has thread safe ref-counting?!

If it isn't thread safe it is of very limited utility, you can 
usually get away with unique_ptr in single threaded scenarios.
Sep 30 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 30.09.2014 14:55, schrieb "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>":

 On Tuesday, 30 September 2014 at 12:51:25 UTC, Paulo  Pinto wrote:

 It works when two big ifs come together.

 - inside the same scope (e.g. function level)

 - when the referece is not shared between threads.

 While it is of limited applicability, Objective-C (and eventually
 Swift) codebases prove it helps in most real life use cases.


 But Objective-C has thread safe ref-counting?!

 If it isn't thread safe it is of very limited utility, you can usually
 get away with unique_ptr in single threaded scenarios.


Did you read my second bullet?
Sep 30 2014
parent "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Tuesday, 30 September 2014 at 20:13:38 UTC, Paulo Pinto wrote:

 Am 30.09.2014 14:55, schrieb "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>":

 On Tuesday, 30 September 2014 at 12:51:25 UTC, Paulo  Pinto 
 wrote:

 It works when two big ifs come together.

 - inside the same scope (e.g. function level)

 - when the referece is not shared between threads.

 While it is of limited applicability, Objective-C (and 
 eventually
 Swift) codebases prove it helps in most real life use cases.


 But Objective-C has thread safe ref-counting?!

 If it isn't thread safe it is of very limited utility, you can 
 usually
 get away with unique_ptr in single threaded scenarios.


 Did you read my second bullet?


Yes? I dont want builtin rc default for single threaded use 
cases. I do want it when references are shared between threads, 
e.g. for cache objects.
Sep 30 2014
prev sibling parent "Mike" <none none.com> writes:
On Tuesday, 30 September 2014 at 12:32:08 UTC, Ola Fosheim 
Grøstad wrote:

 ...basic building blocks such as intrinsics to build your own 
 RC with compiler support sounds like a more interesting option.


I agree.
Sep 30 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/30/14, 3:47 AM, Vladimir Panteleev wrote:

 On Tuesday, 30 September 2014 at 08:34:26 UTC, Johannes Pfau wrote:

 What if I don't want automated memory _management_? What if I want a
 function to use a stack buffer? Or if I want to free manually?

 If I want std.string.toStringz to put the result into a temporary stack
 buffer your solution doesn't help at all. Passing an ouput range,
 allocator or buffer would all solve this.


 I don't understand, why wouldn't you be able to temporarily set the
 thread-local allocator to use the stack buffer, and restore it once done?


That's doable, but you don't get to place the string at a _specific_ 
buffer. -- Andrei
Sep 30 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/30/14, 1:34 AM, Johannes Pfau wrote:

 So you propose RC + global/thread local allocators as the solution for
 all memory related problems as 'memory management is not allocation'.
 And you claim that using output ranges / providing buffers / allocators
 is not an option because it only works in some special cases?


Correct. I assume you meant an irony/sarcasm somewhere :o).


 What if I don't want automated memory _management_? What if I want a
 function to use a stack buffer? Or if I want to free manually?

 If I want std.string.toStringz to put the result into a temporary stack
 buffer your solution doesn't help at all. Passing an ouput range,
 allocator or buffer would all solve this.


Correct. The output of toStringz would be either a GC string or an RC 
string.


Andrei
Sep 30 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 30 Sep 2014 05:23:29 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 9/30/14, 1:34 AM, Johannes Pfau wrote:

 So you propose RC + global/thread local allocators as the solution
 for all memory related problems as 'memory management is not
 allocation'. And you claim that using output ranges / providing
 buffers / allocators is not an option because it only works in some
 special cases?


 
 Correct. I assume you meant an irony/sarcasm somewhere :o).


The sarcasm is supposed to be here: '_all_ memory related problems' ;-)

I guess my point is that although RC is useful in some cases output
ranges / sink delegates / pre-allocated buffers are still necessary in
other cases and RC is not the solution for _everything_.

As Manu often pointed out sometimes you do not want any dynamic
allocation (toStringz in games is a good example) and here RC doesn't
help.

Another example is format which can already write to output ranges and
uses sink delegates internally. That's a much better abstraction than
simply returning a reference counted string (allocated with a thread
local allocator). Using sink delegates internally is also more
efficient than creating temporary RCStrings. And sometimes there's no
allocation at all this way (directly writing to a socket/file). 


 

 What if I don't want automated memory _management_? What if I want a
 function to use a stack buffer? Or if I want to free manually?

 If I want std.string.toStringz to put the result into a temporary
 stack buffer your solution doesn't help at all. Passing an ouput
 range, allocator or buffer would all solve this.


 
 Correct. The output of toStringz would be either a GC string or an RC 
 string.


But why not provide 3 overloads then?

toStringz(OutputRange)
string toStringz(Policy) //char*, actually
RCString toStringz(Policy)

The notion I got from some of your posts is that you're opposed to such
overloads, or did I misinterpret that?
Sep 30 2014
next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Tuesday, 30 September 2014 at 16:49:48 UTC, Johannes Pfau
wrote:

 I guess my point is that although RC is useful in some cases 
 output
 ranges / sink delegates / pre-allocated buffers are still 
 necessary in
 other cases and RC is not the solution for _everything_.


Yes, I'm hoping this is an adjunct to changes in Phobos to reduce
the frequency of implicit allocation in general.  The less
garbage that's generated, the less GC vs. RC actually matters.
Sep 30 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/30/14, 9:49 AM, Johannes Pfau wrote:

 I guess my point is that although RC is useful in some cases output
 ranges / sink delegates / pre-allocated buffers are still necessary in
 other cases and RC is not the solution for _everything_.


Agreed.


 As Manu often pointed out sometimes you do not want any dynamic
 allocation (toStringz in games is a good example) and here RC doesn't
 help.

 Another example is format which can already write to output ranges and
 uses sink delegates internally. That's a much better abstraction than
 simply returning a reference counted string (allocated with a thread
 local allocator). Using sink delegates internally is also more
 efficient than creating temporary RCStrings. And sometimes there's no
 allocation at all this way (directly writing to a socket/file).


Agreed.


 What if I don't want automated memory _management_? What if I want a
 function to use a stack buffer? Or if I want to free manually?

 If I want std.string.toStringz to put the result into a temporary
 stack buffer your solution doesn't help at all. Passing an ouput
 range, allocator or buffer would all solve this.


 Correct. The output of toStringz would be either a GC string or an RC
 string.


 But why not provide 3 overloads then?

 toStringz(OutputRange)
 string toStringz(Policy) //char*, actually
 RCString toStringz(Policy)

 The notion I got from some of your posts is that you're opposed to such
 overloads, or did I misinterpret that?


I'm not opposed. Here's what I think.

As an approach to using Phobos without a GC, it's been suggested that we 
supplement garbage-creating functions with new functions that use output 
ranges everywhere, or lazy ranges everywhere.

I think a better approach is to make memory management a policy that 
makes convenient use of reference counting possible. So instead of 
garbage there'd be reference counted stuff.

Of course, to the extent using lazy computation and/or output ranges is 
a good thing to have for various reasons, they remain valid techniques 
that are and will continue being used in Phobos.

My point is that acknowledging and systematically using reference 
counted types is an essential part of the entire approach.


Andrei
Oct 01 2014
parent Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Oct 2014 02:21:44 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 9/30/14, 9:49 AM, Johannes Pfau wrote:

 I guess my point is that although RC is useful in some cases output
 ranges / sink delegates / pre-allocated buffers are still necessary
 in other cases and RC is not the solution for _everything_.


 
 Agreed.
 

 As Manu often pointed out sometimes you do not want any dynamic
 allocation (toStringz in games is a good example) and here RC
 doesn't help.

 Another example is format which can already write to output ranges
 and uses sink delegates internally. That's a much better
 abstraction than simply returning a reference counted string
 (allocated with a thread local allocator). Using sink delegates
 internally is also more efficient than creating temporary
 RCStrings. And sometimes there's no allocation at all this way
 (directly writing to a socket/file).


 
 Agreed.
 

 What if I don't want automated memory _management_? What if I
 want a function to use a stack buffer? Or if I want to free
 manually?

 If I want std.string.toStringz to put the result into a temporary
 stack buffer your solution doesn't help at all. Passing an ouput
 range, allocator or buffer would all solve this.


 Correct. The output of toStringz would be either a GC string or an
 RC string.


 But why not provide 3 overloads then?

 toStringz(OutputRange)
 string toStringz(Policy) //char*, actually
 RCString toStringz(Policy)

 The notion I got from some of your posts is that you're opposed to
 such overloads, or did I misinterpret that?


 
 I'm not opposed. Here's what I think.
 
 As an approach to using Phobos without a GC, it's been suggested that
 we supplement garbage-creating functions with new functions that use
 output ranges everywhere, or lazy ranges everywhere.
 
 I think a better approach is to make memory management a policy that 
 makes convenient use of reference counting possible. So instead of 
 garbage there'd be reference counted stuff.
 
 Of course, to the extent using lazy computation and/or output ranges
 is a good thing to have for various reasons, they remain valid
 techniques that are and will continue being used in Phobos.
 
 My point is that acknowledging and systematically using reference 
 counted types is an essential part of the entire approach.
 
 
 Andrei
 
 


OK then I got you wrong and I agree with everything you wrote above.
Thanks for clarifying.
Oct 06 2014
prev sibling next sibling parent "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu 
wrote:

 On the caller side:

 auto p1 = setExtension("hello", ".txt"); // fine, use gc
 auto p2 = setExtension!gc("hello", ".txt"); // same
 auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc

 So by default it's going to continue being business as usual, 
 but certain functions will allow passing in a (defaulted) 
 policy for memory management.


Forcing someone (or rather, a team of someones) to call into the 
library in a consistent fashion like this seems like a rather 
risky venture. I suppose that you could add some special compiler 
checks to make sure that people are being consistent, but I'd 
probably rather see some way of templating modules so that the 
chances for human error are reduced.

--- foo.d ---
module std.foo(GC = gc);

void bar() {
    static if (gc) {
       ...
    }
}

--- usercode.d ---
import std.foo!rc;

void fooCaller() {
     bar();
}

Though truthfully, I'd rather it be a compiler flag. But I 
presume that there's an issue with that, which it is too early 
for my brain to think of.
Sep 29 2014
prev sibling next sibling parent reply Shammah Chancellor <email domain.com> writes:
On 2014-09-29 10:49:52 +0000, Andrei Alexandrescu said:


 Back when I've first introduced RCString I hinted that we have a larger 
 strategy in mind. Here it is.
 
 The basic tenet of the approach is to reckon and act on the fact that 
 memory allocation (the subject of allocators) is an entirely distinct 
 topic from memory management, and more generally resource management. 
 This clarifies that it would be wrong to approach alternatives to GC in 
 Phobos by means of allocators. GC is not only an approach to memory 
 allocation, but also an approach to memory management. Reducing it to 
 either one is a mistake. In hindsight this looks rather obvious but it 
 has caused me and many people better than myself a lot of headache.
 
 That said allocators are nice to have and use, and I will definitely 
 follow up with std.allocator. However, std.allocator is not the key to 
 a  nogc Phobos.
 
 Nor are ranges. There is an attitude that either output ranges, or 
 input ranges in conjunction with lazy computation, would solve the 
 issue of creating garbage. 
 https://github.com/D-Programming-Language/phobos/pull/2423 is a good 
 illustration of the latter approach: a range would be lazily created by 
 chaining stuff together. A range-based approach would take us further 
 than the allocators, but I see the following issues with it:
 
 (a) the whole approach doesn't stand scrutiny for non-linear outputs, 
 e.g. outputting some sort of associative array or really any composite 
 type quickly becomes tenuous either with an output range (eager) or 
 with exposing an input range (lazy);
 
 (b) makes the style of programming without GC radically different, and 
 much more cumbersome, than programming with GC; as a consequence, 
 programmers who consider changing one approach to another, or 
 implementing an algorithm neutral to it, are looking at a major rewrite;
 
 (c) would make D/ nogc a poor cousin of C++. This is quite out of 
 character; technically, I have long gotten used to seeing most 
 elaborate C++ code like poor emulation of simple D idioms. But C++ has 
 spent years and decades taking to perfection an approach without a 
 tracing garbage collector. A departure from that would need to be 
 superior, and that doesn't seem to be the case with range-based 
 approaches.
 
 ===========
 
 Now that we clarified that these existing attempts are not going to 
 work well, the question remains what does. For Phobos I'm thinking of 
 defining and using three policies:
 
 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
      gc = ResourceManagementPolicy.gc,
      rc = ResourceManagementPolicy.rc,
      mrc = ResourceManagementPolicy.mrc;
 
 The three policies are:
 
 (a) gc is the classic garbage-collected style of management;
 
 (b) rc is a reference-counted style still backed by the GC, i.e. the GC 
 will still be able to pick up cycles and other kinds of leaks.
 
 (c) mrc is a reference-counted style backed by malloc.
 
 (It should be possible to collapse rc and mrc together and make the 
 distinction dynamically, at runtime. I'm distinguishing them statically 
 here for expository purposes.)
 
 The policy is a template parameter to functions in Phobos (and 
 elsewhere), and informs the functions e.g. what types to return. 
 Consider:
 
 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2 ext)
 if (...)
 {
      static if (mmp == gc) alias S = string;
      else alias S = RCString;
      S result;
      ...
      return result;
 }
 
 On the caller side:
 
 auto p1 = setExtension("hello", ".txt"); // fine, use gc
 auto p2 = setExtension!gc("hello", ".txt"); // same
 auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc
 
 So by default it's going to continue being business as usual, but 
 certain functions will allow passing in a (defaulted) policy for memory 
 management.
 
 Destroy!
 
 
 Andrei


I don't like the idea of having to pass in template parameters 
everywhere -- even for allocators.  Is there some way we could have 
"allocator contexts"?

E.G.

with( auto allocator = ReferencedCounted() )
{
	auto foo = setExtension("hello", "txt");
}

ReferenceCounted() could replace a thread-local "new" delegate with 
something it has, and when it goes out of scope, it would reset it to 
whatever it was before.   This would create some runtime overhead -- 
but I'm not sure how much more than already exists.

-Shammah
Sep 29 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 11:44 AM, Shammah Chancellor wrote:

 I don't like the idea of having to pass in template parameters
 everywhere -- even for allocators.


I agree.


 Is there some way we could have
 "allocator contexts"?

 E.G.

 with( auto allocator = ReferencedCounted() )


Don't confuse memory allocation with memory management. There's no such 
a thing as a "reference counted allocator".


Andrei
Sep 29 2014
next sibling parent Shammah Chancellor <email domain.com> writes:
On 2014-09-29 22:15:33 +0000, Andrei Alexandrescu said:


 On 9/29/14, 11:44 AM, Shammah Chancellor wrote:

 I don't like the idea of having to pass in template parameters
 everywhere -- even for allocators.


 
 I agree.
 

 Is there some way we could have
 "allocator contexts"?
 
 E.G.
 
 with( auto allocator = ReferencedCounted() )


 
 Don't confuse memory allocation with memory management. There's no such 
 a thing as a "reference counted allocator".
 
 Andrei


Sure, but combining the two could be very useful -- as we have noticed 
with a allocators that work off of a garbage collector.  With regards 
to reference counting, you could implement one that automatically wraps 
the type in an RC struct and proxies them.   Being able to redefined 
aliases during different sections of compilation would be required 
though.
Sep 29 2014
prev sibling parent "Daniel N" <ufo orbiting.us> writes:
On Monday, 29 September 2014 at 22:15:32 UTC, Andrei Alexandrescu 
wrote:

 On 9/29/14, 11:44 AM, Shammah Chancellor wrote:

 I don't like the idea of having to pass in template parameters
 everywhere -- even for allocators.


 I agree.


There was a solution earlier in this thread which avoids that 
problem. When a function is annotated with  nogc there's 
sufficient info to chose the correct implementation without any 
parameters, it's already known whether we are instantiated from a 
 nogc block or not.
Sep 29 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 11:44 AM, Shammah Chancellor wrote:

 I don't like the idea of having to pass in template parameters
 everywhere -- even for allocators.  Is there some way we could have
 "allocator contexts"?

 E.G.

 with( auto allocator = ReferencedCounted() )
 {
      auto foo = setExtension("hello", "txt");
 }

 ReferenceCounted() could replace a thread-local "new" delegate with
 something it has, and when it goes out of scope, it would reset it to
 whatever it was before.   This would create some runtime overhead -- but
 I'm not sure how much more than already exists.


I'm not sure whether we can do this within D's type system. -- Andrei
Oct 01 2014
prev sibling next sibling parent reply "Uranuz" <neuranuz gmail.com> writes:
 auto p1 = setExtension("hello", ".txt"); // fine, use gc
 auto p2 = setExtension!gc("hello", ".txt"); // same
 auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc

 So by default it's going to continue being business as usual, 
 but certain functions will allow passing in a (defaulted) 
 policy for memory management.

 Destroy!


I'll try to destroy ;) Before thinking out some answers to this
problem let me ask a little more questions.

1. As far as I understand allocation and memory management of
entities like class (Object), dynamic arrays and associative
arrays is part of language/ runtime. What is proposed here is
*fix* to standart library. But that allocation and MM happening
via GC is not *fault* of standart library but is predefined
behaviour of D lang itself and it's runtime. The standard library
becomes a `hostage` of runtime library in this situation. Do you
really sure that we should "fix" standart library in that way?
For me it looks like implementing struts for standard lib (which
is not broken yet ;) ) in order to compensate behaviour of
runtime lib.

2. Second question is slightly oftopic, but I still want put it
there. What I dislike about ranges and standart library is that
it's hard to understand what is the returned value of library
function. I have some *pedals* (front, popFront) to push and do
some magic. Of course it was made for purpose of making universal
algorithms. But the mor I use ranges, *auto* then less I believe
that I use static-typed language. What is wanted to make code
clear is having distinct variable declaration with specification
of it's type. With all of these auto's logic of programme becomes
unclear, because data structures are unclear. So I came to the
question: is the memory management or allocation policy
syntacticaly part of declaration or is it a inner implementation
detail that should not be shown in decl?

Should rc and gc string look simillar or not?

string str1 = makeGCString("test");
string str2 = makeRCString("test");

// --- vs ---

GCString str1 = "test";
RCString str2 = "test";

// --- or ---

String!GC str1 = "test";
String!RC str2 = "test";

// --- or even ---
 gc string str1 = "test";
 rc string str2 = "test";

As far as I understand currently we will have:
string str1 = "test";
RCString str2 = "test";

So another question is why the same object "string" is
implemented as different types. Array and struct (class)?

3. Should algorithms based on range interface care about
allocation? Range is about iteration and access to elements but
not about allocation and memory mangement.

I would like to have attributes  rc,  gc (or like these) to
switch MM-policy versus *String!RC* or *RCString* but we cannot
apply attributes to literal. Passing to allgorithm something like
this:

find(  rc "test",  rc "t" )

is syntactically incorrect. But we can use this form:

find( RCString("test"), RCString("t") )

But above form is more verbose. As continuation of this question
I have next question.

4. How to deal with literals? How to make them ref-counted?

I ask this because even when writing RCString("test")
syntactically expression "test" is still GC-managed literal. I
pass GC-managed literal into struct to make it RC-managed. Why
just not make it RC from the start?

Adding some additional template parameter to algrorithm wil not
fix this. It is a problem of D itself and it's runtime library.


So I assume that std lib is not broken this way and we should not
try to fix it this way. Thanks for attention.
Sep 29 2014
next sibling parent "Mike" <none none.com> writes:
On Monday, 29 September 2014 at 20:07:41 UTC, Uranuz wrote:


 1. As far as I understand allocation and memory management of
 entities like class (Object), dynamic arrays and associative
 arrays is part of language/ runtime. What is proposed here is
 *fix* to standart library. But that allocation and MM happening
 via GC is not *fault* of standart library but is predefined
 behaviour of D lang itself and it's runtime. The standard 
 library
 becomes a `hostage` of runtime library in this situation. Do you
 really sure that we should "fix" standart library in that way?
 For me it looks like implementing struts for standard lib (which
 is not broken yet ;) ) in order to compensate behaviour of
 runtime lib.


This really hits the nail on the head, and I think your other 
comments and questions are also quite insightful.

IMO the proposal that started this thread,  nogc, and -vgc are 
all beating around the bush rather than addressing the 
fundamental problem.

Mike
Sep 29 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 1:07 PM, Uranuz wrote:

 1. As far as I understand allocation and memory management of
 entities like class (Object), dynamic arrays and associative
 arrays is part of language/ runtime. What is proposed here is
 *fix* to standart library. But that allocation and MM happening
 via GC is not *fault* of standart library but is predefined
 behaviour of D lang itself and it's runtime. The standard library
 becomes a `hostage` of runtime library in this situation. Do you
 really sure that we should "fix" standart library in that way?
 For me it looks like implementing struts for standard lib (which
 is not broken yet ;) ) in order to compensate behaviour of
 runtime lib.


The change will be to both the runtime and the standard library.


 2. Second question is slightly oftopic, but I still want put it
 there. What I dislike about ranges and standart library is that
 it's hard to understand what is the returned value of library
 function. I have some *pedals* (front, popFront) to push and do
 some magic. Of course it was made for purpose of making universal
 algorithms. But the mor I use ranges, *auto* then less I believe
 that I use static-typed language. What is wanted to make code
 clear is having distinct variable declaration with specification
 of it's type. With all of these auto's logic of programme becomes
 unclear, because data structures are unclear. So I came to the
 question: is the memory management or allocation policy
 syntacticaly part of declaration or is it a inner implementation
 detail that should not be shown in decl?


Sadly this is the way things are going (not only in D, but other 
languages such as C++, Haskell, Scala, etc). Type proliferation has 
costs, but also a ton of benefits.

Most often the memory management policy will be part of function 
signatures because it affects data type definitions.


 Should rc and gc string look simillar or not?

 string str1 = makeGCString("test");
 string str2 = makeRCString("test");

 // --- vs ---

 GCString str1 = "test";
 RCString str2 = "test";

 // --- or ---

 String!GC str1 = "test";
 String!RC str2 = "test";

 // --- or even ---
  gc string str1 = "test";
  rc string str2 = "test";

 As far as I understand currently we will have:
 string str1 = "test";
 RCString str2 = "test";


Per Sean's idea things would go GC.string vs. RC.string, where GC and RC 
are two memory management policies (simple structs defining aliases and 
probably a few primitives).


 So another question is why the same object "string" is
 implemented as different types. Array and struct (class)?


A reference counted string has a different layout than immutable(char)[].


 3. Should algorithms based on range interface care about
 allocation? Range is about iteration and access to elements but
 not about allocation and memory mangement.


Most don't.


 I would like to have attributes  rc,  gc (or like these) to
 switch MM-policy versus *String!RC* or *RCString* but we cannot
 apply attributes to literal. Passing to allgorithm something like
 this:

 find(  rc "test",  rc "t" )

 is syntactically incorrect. But we can use this form:

 find( RCString("test"), RCString("t") )

 But above form is more verbose. As continuation of this question
 I have next question.


If language changes are necessary, we will make language changes. I'm 
trying first to explore solutions within the language.


 4. How to deal with literals? How to make them ref-counted?


I don't know yet.


 I ask this because even when writing RCString("test")
 syntactically expression "test" is still GC-managed literal. I
 pass GC-managed literal into struct to make it RC-managed. Why
 just not make it RC from the start?

 Adding some additional template parameter to algrorithm wil not
 fix this. It is a problem of D itself and it's runtime library.


I understand. The problem is actually worse with array literals, which 
are silently dynamically allocated on the garbage-collected heap:

auto s = "hello"; // at least there's no allocation
auto a = [1, 2, 3]; // dynamic allocation

A language-based solution would change array literal syntax. A 
library-based solution would leave array literals with today's syntax 
and semantics and offer a controlled alternative a la:

auto a = MyMemPolicy.array(1, 2, 3); // cool


 So I assume that std lib is not broken this way and we should not
 try to fix it this way. Thanks for attention.


And thanks for your great points.


Andrei
Oct 01 2014
prev sibling next sibling parent reply "Freddy" <Hexagonalstar64 gmail.com> writes:
On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu
wrote:

 Back when I've first introduced RCString I hinted that we have 
 a larger strategy in mind. Here it is.

 The basic tenet of the approach is to reckon and act on the 
 fact that memory allocation (the subject of allocators) is an 
 entirely distinct topic from memory management, and more 
 generally resource management. This clarifies that it would be 
 wrong to approach alternatives to GC in Phobos by means of 
 allocators. GC is not only an approach to memory allocation, 
 but also an approach to memory management. Reducing it to 
 either one is a mistake. In hindsight this looks rather obvious 
 but it has caused me and many people better than myself a lot 
 of headache.

 That said allocators are nice to have and use, and I will 
 definitely follow up with std.allocator. However, std.allocator 
 is not the key to a  nogc Phobos.

 Nor are ranges. There is an attitude that either output ranges, 
 or input ranges in conjunction with lazy computation, would 
 solve the issue of creating garbage. 
 https://github.com/D-Programming-Language/phobos/pull/2423 is a 
 good illustration of the latter approach: a range would be 
 lazily created by chaining stuff together. A range-based 
 approach would take us further than the allocators, but I see 
 the following issues with it:

 (a) the whole approach doesn't stand scrutiny for non-linear 
 outputs, e.g. outputting some sort of associative array or 
 really any composite type quickly becomes tenuous either with 
 an output range (eager) or with exposing an input range (lazy);

 (b) makes the style of programming without GC radically 
 different, and much more cumbersome, than programming with GC; 
 as a consequence, programmers who consider changing one 
 approach to another, or implementing an algorithm neutral to 
 it, are looking at a major rewrite;

 (c) would make D/ nogc a poor cousin of C++. This is quite out 
 of character; technically, I have long gotten used to seeing 
 most elaborate C++ code like poor emulation of simple D idioms. 
 But C++ has spent years and decades taking to perfection an 
 approach without a tracing garbage collector. A departure from 
 that would need to be superior, and that doesn't seem to be the 
 case with range-based approaches.

 ===========

 Now that we clarified that these existing attempts are not 
 going to work well, the question remains what does. For Phobos 
 I'm thinking of defining and using three policies:

 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
     gc = ResourceManagementPolicy.gc,
     rc = ResourceManagementPolicy.rc,
     mrc = ResourceManagementPolicy.mrc;

 The three policies are:

 (a) gc is the classic garbage-collected style of management;

 (b) rc is a reference-counted style still backed by the GC, 
 i.e. the GC will still be able to pick up cycles and other 
 kinds of leaks.

 (c) mrc is a reference-counted style backed by malloc.

 (It should be possible to collapse rc and mrc together and make 
 the distinction dynamically, at runtime. I'm distinguishing 
 them statically here for expository purposes.)

 The policy is a template parameter to functions in Phobos (and 
 elsewhere), and informs the functions e.g. what types to 
 return. Consider:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 
 path, R2 ext)
 if (...)
 {
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
 }

 On the caller side:

 auto p1 = setExtension("hello", ".txt"); // fine, use gc
 auto p2 = setExtension!gc("hello", ".txt"); // same
 auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc

 So by default it's going to continue being business as usual, 
 but certain functions will allow passing in a (defaulted) 
 policy for memory management.

 Destroy!


 Andrei


Internally we should have something like:

---
template String(MemoryManagementPolicy mmp=gc){
      /++ ... +/
}
auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1
path, R2 ext)
if (...)
{
      auto result=String!mmp();
      /++ +/
}
----

or maybe even allowing user types in the template argument(the
original purpose of templates)

---
auto setExtension(String = string, R1, R2)(R1
path, R2){
      /++ +/
}
----
Sep 29 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/29/14, 3:11 PM, Freddy wrote:


 Internally we should have something like:

 ---
 template String(MemoryManagementPolicy mmp=gc){
       /++ ... +/
 }
 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1
 path, R2 ext)
 if (...)
 {
       auto result=String!mmp();
       /++ +/
 }
 ----

 or maybe even allowing user types in the template argument(the
 original purpose of templates)

 ---
 auto setExtension(String = string, R1, R2)(R1
 path, R2){
       /++ +/
 }


That's correct. -- Andrei



 Sep 29 2014



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/29/14, 3:11 PM, Freddy wrote:


 Internally we should have something like:

 ---
 template String(MemoryManagementPolicy mmp=gc){
       /++ ... +/
 }
 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1
 path, R2 ext)
 if (...)
 {
       auto result=String!mmp();
       /++ +/
 }
 ----

 or maybe even allowing user types in the template argument(the
 original purpose of templates)

 ---
 auto setExtension(String = string, R1, R2)(R1
 path, R2){
       /++ +/
 }
 ----


Good idea, and it seems Sean's is even better because it groups 
everything related to memory management where it belongs - in the memory 
management policy. -- Andrei



 Oct 01 2014



prev sibling next sibling parent reply "Foo" <Foo test.de>  writes:


I hate the fact that this will produce template bloat for each 
function/method.
I'm also in favor of "let the user pick", but I would use a 
global variable:

----
enum MemoryManagementPolicy { gc, rc, mrc }
immutable
     gc = ResourceManagementPolicy.gc,
     rc = ResourceManagementPolicy.rc,
     mrc = ResourceManagementPolicy.mrc;

auto RMP = gc;
----

and in my code:

----
RMP = rc;
string str = "foo"; // compiler knows -> ref counted
// ...
RMP = gc;
string str2 = "bar"; // normal behaviour restored
----



 Sep 30 2014



 next sibling parent  "Foo" <Foo test.de>  writes:


On Tuesday, 30 September 2014 at 13:38:43 UTC, Foo wrote:

 I hate the fact that this will produce template bloat for each 
 function/method.
 I'm also in favor of "let the user pick", but I would use a 
 global variable:

 ----
 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
     gc = ResourceManagementPolicy.gc,
     rc = ResourceManagementPolicy.rc,
     mrc = ResourceManagementPolicy.mrc;

 auto RMP = gc;
 ----

 and in my code:

 ----
 RMP = rc;
 string str = "foo"; // compiler knows -> ref counted
 // ...
 RMP = gc;
 string str2 = "bar"; // normal behaviour restored
 ----


Of course each method/function in Phobos should use the global
RMP.



 Sep 30 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 6:38 AM, Foo wrote:

 I hate the fact that this will produce template bloat for each
 function/method.
 I'm also in favor of "let the user pick", but I would use a global
 variable:

 ----
 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
      gc = ResourceManagementPolicy.gc,
      rc = ResourceManagementPolicy.rc,
      mrc = ResourceManagementPolicy.mrc;

 auto RMP = gc;
 ----

 and in my code:

 ----
 RMP = rc;
 string str = "foo"; // compiler knows -> ref counted
 // ...
 RMP = gc;
 string str2 = "bar"; // normal behaviour restored
 ----


This won't work because the type of "string" is different for RC vs. GC. 
-- Andrei



 Sep 30 2014



  parent reply "Foo" <Foo test.de>  writes:


On Tuesday, 30 September 2014 at 13:59:23 UTC, Andrei
Alexandrescu wrote:

 On 9/30/14, 6:38 AM, Foo wrote:

 I hate the fact that this will produce template bloat for each
 function/method.
 I'm also in favor of "let the user pick", but I would use a 
 global
 variable:

 ----
 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
     gc = ResourceManagementPolicy.gc,
     rc = ResourceManagementPolicy.rc,
     mrc = ResourceManagementPolicy.mrc;

 auto RMP = gc;
 ----

 and in my code:

 ----
 RMP = rc;
 string str = "foo"; // compiler knows -> ref counted
 // ...
 RMP = gc;
 string str2 = "bar"; // normal behaviour restored
 ----


 This won't work because the type of "string" is different for 
 RC vs. GC. -- Andrei


But it would work for phobos functions without template bloat.



 Sep 30 2014



 next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 30 September 2014 at 14:05:43 UTC, Foo wrote:

 On Tuesday, 30 September 2014 at 13:59:23 UTC, Andrei
 Alexandrescu wrote:

 On 9/30/14, 6:38 AM, Foo wrote:
 This won't work because the type of "string" is different for 
 RC vs. GC. -- Andrei


 But it would work for phobos functions without template bloat.


Only for internal allocations. If the functions want to return 
something, the type must known.



 Sep 30 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 7:13 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Tuesday, 30 September 2014 at 14:05:43 UTC, Foo wrote:

 On Tuesday, 30 September 2014 at 13:59:23 UTC, Andrei
 Alexandrescu wrote:

 On 9/30/14, 6:38 AM, Foo wrote:
 This won't work because the type of "string" is different for RC vs.
 GC. -- Andrei


 But it would work for phobos functions without template bloat.


 Only for internal allocations. If the functions want to return
 something, the type must known.


Ah, now I understand the point. Thanks. -- Andrei



 Sep 30 2014



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 7:05 AM, Foo wrote:

 On Tuesday, 30 September 2014 at 13:59:23 UTC, Andrei
 Alexandrescu wrote:

 On 9/30/14, 6:38 AM, Foo wrote:

 I hate the fact that this will produce template bloat for each
 function/method.
 I'm also in favor of "let the user pick", but I would use a global
 variable:

 ----
 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
     gc = ResourceManagementPolicy.gc,
     rc = ResourceManagementPolicy.rc,
     mrc = ResourceManagementPolicy.mrc;

 auto RMP = gc;
 ----

 and in my code:

 ----
 RMP = rc;
 string str = "foo"; // compiler knows -> ref counted
 // ...
 RMP = gc;
 string str2 = "bar"; // normal behaviour restored
 ----


 This won't work because the type of "string" is different for RC vs.
 GC. -- Andrei


 But it would work for phobos functions without template bloat.


How is the fact there's less bloat relevant for code that doesn't work? 
I.e. it doesn't compile. It needs to return string for GC and RCString 
for RC.

Andrei



 Sep 30 2014



prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu 
wrote:

 Back when I've first introduced RCString I hinted that we have 
 a larger strategy in mind. Here it is.

 The basic tenet of the approach is to reckon and act on the 
 fact that memory allocation (the subject of allocators) is an 
 entirely distinct topic from memory management, and more 
 generally resource management. This clarifies that it would be 
 wrong to approach alternatives to GC in Phobos by means of 
 allocators. GC is not only an approach to memory allocation, 
 but also an approach to memory management. Reducing it to 
 either one is a mistake. In hindsight this looks rather obvious 
 but it has caused me and many people better than myself a lot 
 of headache.

 That said allocators are nice to have and use, and I will 
 definitely follow up with std.allocator. However, std.allocator 
 is not the key to a  nogc Phobos.

 Nor are ranges. There is an attitude that either output ranges, 
 or input ranges in conjunction with lazy computation, would 
 solve the issue of creating garbage. 
 https://github.com/D-Programming-Language/phobos/pull/2423 is a 
 good illustration of the latter approach: a range would be 
 lazily created by chaining stuff together. A range-based 
 approach would take us further than the allocators, but I see 
 the following issues with it:

 (a) the whole approach doesn't stand scrutiny for non-linear 
 outputs, e.g. outputting some sort of associative array or 
 really any composite type quickly becomes tenuous either with 
 an output range (eager) or with exposing an input range (lazy);

 (b) makes the style of programming without GC radically 
 different, and much more cumbersome, than programming with GC; 
 as a consequence, programmers who consider changing one 
 approach to another, or implementing an algorithm neutral to 
 it, are looking at a major rewrite;

 (c) would make D/ nogc a poor cousin of C++. This is quite out 
 of character; technically, I have long gotten used to seeing 
 most elaborate C++ code like poor emulation of simple D idioms. 
 But C++ has spent years and decades taking to perfection an 
 approach without a tracing garbage collector. A departure from 
 that would need to be superior, and that doesn't seem to be the 
 case with range-based approaches.

 ===========

 Now that we clarified that these existing attempts are not 
 going to work well, the question remains what does. For Phobos 
 I'm thinking of defining and using three policies:

 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
     gc = ResourceManagementPolicy.gc,
     rc = ResourceManagementPolicy.rc,
     mrc = ResourceManagementPolicy.mrc;

 The three policies are:

 (a) gc is the classic garbage-collected style of management;

 (b) rc is a reference-counted style still backed by the GC, 
 i.e. the GC will still be able to pick up cycles and other 
 kinds of leaks.

 (c) mrc is a reference-counted style backed by malloc.

 (It should be possible to collapse rc and mrc together and make 
 the distinction dynamically, at runtime. I'm distinguishing 
 them statically here for expository purposes.)

 The policy is a template parameter to functions in Phobos (and 
 elsewhere), and informs the functions e.g. what types to 
 return. Consider:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 
 path, R2 ext)
 if (...)
 {
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
 }

 On the caller side:

 auto p1 = setExtension("hello", ".txt"); // fine, use gc
 auto p2 = setExtension!gc("hello", ".txt"); // same
 auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc

 So by default it's going to continue being business as usual, 
 but certain functions will allow passing in a (defaulted) 
 policy for memory management.

 Destroy!


 Andrei


Instead of adding a new template parameter to every function 
(which won't necessarily play nicely with existing IFTI and 
variadic templates), why not allow template modules?

import stringRC = std.string!rc;
import stringGC = std.string!gc;


// in std/string.d
module std.string(MemoryManagementPolicy mmp)

pure  trusted S capitalize(S)(S s)
     if (isSomeString!S)
{
     //...

     static if(mmp == MemoryManagementPolicy.gc)
     {
         //...
     }
     else static if .......
}



 Sep 30 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 7:07 AM, John Colvin wrote:

 Instead of adding a new template parameter to every function (which
 won't necessarily play nicely with existing IFTI and variadic
 templates), why not allow template modules?


Nice idea, but let's try and explore possibilities within the existing 
rich language. If a need for new language features arises, I trust we'll 
see it. -- Andrei



 Oct 01 2014



prev sibling next sibling parent reply "Sean Kelly" <sean invisibleduck.org>  writes:


On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu 
wrote:

 The policy is a template parameter to functions in Phobos (and 
 elsewhere), and informs the functions e.g. what types to 
 return. Consider:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 
 path, R2 ext)
 if (...)
 {
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
 }


Is this for exposition purposes or actually how you expect it to 
work?  Quite honestly, I can't imagine how I could write a 
template function in D that needs to work with this approach.

As much as I hate to say it, this is pretty much exactly what C++ 
allocators were designed for.  They handle allocation, sure, but 
they also hold aliases for all relevant types for the data being 
allocated.  If the MemoryManagementPolicy enum were replaced with 
an alias to a type that I could use to at least obtain relevant 
aliases, that would be something.  But even that approach 
dramatically complicates code that uses it.

Having written standards-compliant containers in C++, I honestly 
can't imagine the average user writing code that works this way.  
Once you assert that the reference type may be a pointer or it 
may be some complex proxy to data stored elsewhere, a lot of 
composability pretty much flies right out the window.

For example, I have an implementation of C++ 
unordered_map/set/etc designed to be a customizable cache, so one 
of its template arguments is a policy type that allows eviction 
behavior to be chosen at declaration time.  Maybe the cache is 
size-limited, maybe it's age-limited, maybe it's a combination of 
the two or something even more complicated.  The problem is that 
the container defines all the aliases relating to the underlying 
data, but the policy, which needs to be aware of these, is passed 
as a template argument to this container.

To make something that's fully aware of C++ allocators then, I'd 
have to define a small type that takes the container template 
arguments (the contained type and the allocator type) and 
generates the aliases and pass this to the policy, which in turn 
passes the type through to the underlying container so it can 
declare its public aliases and whatever else is true 
standards-compliant fashion (or let the container derive this 
itself, but then you run into the potential for disagreement).  
And while this is possible, doing so would complicate the 
creation of the cache policies to the point where it subverts 
their intent, which was to make it easy for the user to tune the 
behavior of the cache to their own particular needs by defining a 
simple type which implements a few functions.  Ultimately, I 
decided against this approach for the cache container and decided 
to restrict the allocators to those which defined a pointer to T 
as T* so the policies could be coded with basically no knowledge 
of the underlying storage.

So... while I support the goal you're aiming at, I want to see a 
much more comprehensive example of how this will work and how it 
will affect code written by D *users*.  Because it isn't enough 
for Phobos to be written this way.  Basically all D code will 
have to take this into account for the strategy to be truly 
viable.  Simply outlining one of the most basic functions in 
Phobos, which already looks like it will have a static 
conditional at the beginning and *need to be aware of the fact 
that an RCString type exists* makes me terrified of what a 
realistic example will look like.



 Sep 30 2014



 next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Tue, Sep 30, 2014 at 04:10:43PM +0000, Sean Kelly via Digitalmars-d wrote:

 On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:

The policy is a template parameter to functions in Phobos (and
elsewhere), and informs the functions e.g. what types to return.
Consider:

auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2
ext)
if (...)
{
    static if (mmp == gc) alias S = string;
    else alias S = RCString;
    S result;
    ...
    return result;
}


 
 Is this for exposition purposes or actually how you expect it to work?
 Quite honestly, I can't imagine how I could write a template function
 in D that needs to work with this approach.
 
 As much as I hate to say it, this is pretty much exactly what C++
 allocators were designed for.  They handle allocation, sure, but they
 also hold aliases for all relevant types for the data being allocated.


[...]

 So... while I support the goal you're aiming at, I want to see a much
 more comprehensive example of how this will work and how it will
 affect code written by D *users*.  Because it isn't enough for Phobos
 to be written this way.  Basically all D code will have to take this
 into account for the strategy to be truly viable.  Simply outlining
 one of the most basic functions in Phobos, which already looks like it
 will have a static conditional at the beginning and *need to be aware
 of the fact that an RCString type exists* makes me terrified of what a
 realistic example will look like.


Yeah, this echoes my concern. This looks not that much different, from a
user's POV, from C++ containers' allocator template parameters. Yes I
know we're not talking about *allocators* per se but about *memory
management*, but I'm talking about the need to explicitly pass mmp to
*every* *single* *function* if you desire anything but the default. How
many people actually *use* the allocator parameter in STL? Certainly,
many people do... but the code is anything but readable / maintainable.

Not only that, but every single function will have to handle this
parameter somehow, and if static if's at the top of the function is what
we're starting with, I fear seeing what we end up with.

Furthermore, in order for this to actually work, it has to be percolated
throughout the entire codebase -- any D library that even remotely uses
Phobos for anything will have to percolate this parameter throughout its
API -- at least, any part of the API that might potentially use a Phobos
function. Otherwise, you still have the situation where a given D
library doesn't allow the user to select a memory management scheme, and
internally calls Phobos functions with the default settings. So this
still doesn't solve the problem that today, people who need to use  nogc
can't use a lot of existing libraries because the library depends on the
GC, even if it doesn't assume anything about the MM scheme, but just
happens to call some obscure Phobos function with the default MM
parameter. The only way this could work was if *every* D library author
voluntarily rewrites a lot of code in order to percolate this MM
parameter through to the API, on the off-chance that some obscure user
somewhere might have need to use it. I don't see much likelihood of this
actually happening.

Then there's the matter of functions like parseJSON() that needs to
allocate nodes and return a tree (or whatever) of these nodes. Note that
they need to *allocate*, not just know what kind of memory management
model is to be used. So how do you propose to address this? Via another
parameter (compile-time or otherwise) to specify which allocator to use?
So how does the memory management parameter solve anything then? And how
would such a thing be implemented? Using a 3-way static-if branch in
every single point in parseJSON where it needs to allocate nodes? We
could just as well write it in C++, if that's the case.

This proposal has many glaring holes that need to be fixed before it can
be viable.


T

-- 
EMACS = Extremely Massive And Cumbersome System



 Sep 30 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 10:33 AM, H. S. Teoh via Digitalmars-d wrote:

 Yeah, this echoes my concern. This looks not that much different, from a
 user's POV, from C++ containers' allocator template parameters. Yes I
 know we're not talking about*allocators*  per se but about *memory
 management*, but I'm talking about the need to explicitly pass mmp to
 *every*  *single*  *function*  if you desire anything but the default. How
 many people actually*use*  the allocator parameter in STL? Certainly,
 many people do... but the code is anything but readable / maintainable.


The parallel with STL allocators is interesting, but I'm not worried 
about it that much. I don't want to go off on a tangent but I'm fairly 
certain std::allocator is hard to use for entirely different reasons 
than the intended use patterns of MemoryManagementPolicy.


 Not only that, but every single function will have to handle this
 parameter somehow, and if static if's at the top of the function is what
 we're starting with, I fear seeing what we end up with.


Apparently Sean's idea would take care of that.


 Furthermore, in order for this to actually work, it has to be percolated
 throughout the entire codebase -- any D library that even remotely uses
 Phobos for anything will have to percolate this parameter throughout its
 API -- at least, any part of the API that might potentially use a Phobos
 function.


Yes, but that's entirely expected. We're adding genuinely new 
functionality to Phobos.


 Otherwise, you still have the situation where a given D
 library doesn't allow the user to select a memory management scheme, and
 internally calls Phobos functions with the default settings.


Correct.


 So this
 still doesn't solve the problem that today, people who need to use  nogc
 can't use a lot of existing libraries because the library depends on the
 GC, even if it doesn't assume anything about the MM scheme, but just
 happens to call some obscure Phobos function with the default MM
 parameter. The only way this could work was if*every*  D library author
 voluntarily rewrites a lot of code in order to percolate this MM
 parameter through to the API, on the off-chance that some obscure user
 somewhere might have need to use it. I don't see much likelihood of this
 actually happening.


A simple way to put this is Libraries that use the GC will continue to 
use the GC. There's no way around that unless we choose to break them all.


 Then there's the matter of functions like parseJSON() that needs to
 allocate nodes and return a tree (or whatever) of these nodes. Note that
 they need to*allocate*, not just know what kind of memory management
 model is to be used. So how do you propose to address this? Via another
 parameter (compile-time or otherwise) to specify which allocator to use?
 So how does the memory management parameter solve anything then? And how
 would such a thing be implemented? Using a 3-way static-if branch in
 every single point in parseJSON where it needs to allocate nodes? We
 could just as well write it in C++, if that's the case.


parseJSON() would get a memory management policy parameter, and will use 
the currently installed memory allocator for allocation.


 This proposal has many glaring holes that need to be fixed before it can
 be viable.


Affirmative. That's why it's an RFC, very far from a proposal. I'm glad 
I got a bunch of good ideas.


Andrei



 Oct 01 2014



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 9:10 AM, Sean Kelly wrote:

 On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:

 The policy is a template parameter to functions in Phobos (and
 elsewhere), and informs the functions e.g. what types to return.
 Consider:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2
 ext)
 if (...)
 {
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
 }


 Is this for exposition purposes or actually how you expect it to work?


That's pretty much what it would take. The key here is that RCString is 
almost a drop-in replacement for string, so the code using it is almost 
identical. There will be places where code needs to be replaced, e.g.

auto s = "literal";

would need to become

S s = "literal";

So creation of strings will change a bit, but overall there's not a lot 
of churn.


 Quite honestly, I can't imagine how I could write a template function
 in D that needs to work with this approach.


You mean write a function that accepts a memory management policy, or a 
function that uses one?


 As much as I hate to say it, this is pretty much exactly what C++
 allocators were designed for.  They handle allocation, sure, but they
 also hold aliases for all relevant types for the data being allocated.
 If the MemoryManagementPolicy enum were replaced with an alias to a type
 that I could use to at least obtain relevant aliases, that would be
 something.  But even that approach dramatically complicates code that
 uses it.


I think making MemoryManagementPolicy a meaningful type is a great idea. 
It would e.g. define the string type, so the code becomes:

auto setExtension(alias MemoryManagementPolicy = gc, R1, R2)(R1 path, R2 
ext)
if (...)
{
     MemoryManagementPolicy.string result;
     ...
     return result;
}

This is a lot more general and extensible. Thanks!

Why do you think there'd be dramatic complication of code? (Granted, at 
some point we must acknowledge that some egg breaking is necessary for 
the proverbial omelette.)


 Having written standards-compliant containers in C++, I honestly can't
 imagine the average user writing code that works this way. Once you
 assert that the reference type may be a pointer or it may be some
 complex proxy to data stored elsewhere, a lot of composability pretty
 much flies right out the window.


The thing is, again, we must make some changes if we want D to be usable 
without a GC. One of them is e.g. to not allocate built-in slices all 
over the place.


 For example, I have an implementation of C++ unordered_map/set/etc
 designed to be a customizable cache, so one of its template arguments is
 a policy type that allows eviction behavior to be chosen at declaration
 time.  Maybe the cache is size-limited, maybe it's age-limited, maybe
 it's a combination of the two or something even more complicated.  The
 problem is that the container defines all the aliases relating to the
 underlying data, but the policy, which needs to be aware of these, is
 passed as a template argument to this container.

 To make something that's fully aware of C++ allocators then, I'd have to
 define a small type that takes the container template arguments (the
 contained type and the allocator type) and generates the aliases and
 pass this to the policy, which in turn passes the type through to the
 underlying container so it can declare its public aliases and whatever
 else is true standards-compliant fashion (or let the container derive
 this itself, but then you run into the potential for disagreement). And
 while this is possible, doing so would complicate the creation of the
 cache policies to the point where it subverts their intent, which was to
 make it easy for the user to tune the behavior of the cache to their own
 particular needs by defining a simple type which implements a few
 functions.  Ultimately, I decided against this approach for the cache
 container and decided to restrict the allocators to those which defined
 a pointer to T as T* so the policies could be coded with basically no
 knowledge of the underlying storage.


That sounds like a rather involved artifact. Hopefully we can leverage 
D's better expressiveness to make building such complex libraries easier.


 So... while I support the goal you're aiming at, I want to see a much
 more comprehensive example of how this will work and how it will affect
 code written by D *users*.


Agreed.


 Because it isn't enough for Phobos to be
 written this way.  Basically all D code will have to take this into
 account for the strategy to be truly viable.  Simply outlining one of
 the most basic functions in Phobos, which already looks like it will
 have a static conditional at the beginning and *need to be aware of the
 fact that an RCString type exists* makes me terrified of what a
 realistic example will look like.


That would be overreacting :o).


Andrei



 Oct 01 2014



 next sibling parent reply "Sean Kelly" <sean invisibleduck.org>  writes:


On Wednesday, 1 October 2014 at 08:55:55 UTC, Andrei Alexandrescu 
wrote:

 On 9/30/14, 9:10 AM, Sean Kelly wrote:

 Is this for exposition purposes or actually how you expect it 
 to work?


 That's pretty much what it would take. The key here is that 
 RCString is almost a drop-in replacement for string, so the 
 code using it is almost identical. There will be places where 
 code needs to be replaced, e.g.

 auto s = "literal";

 would need to become

 S s = "literal";

 So creation of strings will change a bit, but overall there's 
 not a lot of churn.


I'm confused.  Is this a general-purpose solution or just one 
that switches between string and RCString?



 Oct 01 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/1/14, 6:52 AM, Sean Kelly wrote:

 On Wednesday, 1 October 2014 at 08:55:55 UTC, Andrei Alexandrescu wrote:

 On 9/30/14, 9:10 AM, Sean Kelly wrote:

 Is this for exposition purposes or actually how you expect it to work?


 That's pretty much what it would take. The key here is that RCString
 is almost a drop-in replacement for string, so the code using it is
 almost identical. There will be places where code needs to be
 replaced, e.g.

 auto s = "literal";

 would need to become

 S s = "literal";

 So creation of strings will change a bit, but overall there's not a
 lot of churn.


 I'm confused.  Is this a general-purpose solution or just one that
 switches between string and RCString?


General purpose since your suggested change. -- Andrei



 Oct 01 2014



prev sibling  parent reply "Sean Kelly" <sean invisibleduck.org>  writes:


On Wednesday, 1 October 2014 at 08:55:55 UTC, Andrei Alexandrescu 
wrote:

 On 9/30/14, 9:10 AM, Sean Kelly wrote:


 Quite honestly, I can't imagine how I could write a template 
 function in D that needs to work with this approach.


 You mean write a function that accepts a memory management 
 policy, or a function that uses one?


Both, I suppose?  A static if block at the top of each function 
that must be aware of every RC type the user may expect?  What if 
it's a user-defined RC type and this function is in Phobos?



 As much as I hate to say it, this is pretty much exactly what 
 C++
 allocators were designed for.  They handle allocation, sure, 
 but they
 also hold aliases for all relevant types for the data being 
 allocated.
 If the MemoryManagementPolicy enum were replaced with an alias 
 to a type that I could use to at least obtain relevant 
 aliases, that would be something.  But even that approach 
 dramatically complicates code that uses it.


 I think making MemoryManagementPolicy a meaningful type is a 
 great idea. It would e.g. define the string type, so the code 
 becomes:

 auto setExtension(alias MemoryManagementPolicy = gc, R1, R2)(R1 
 path, R2 ext)
 if (...)
 {
     MemoryManagementPolicy.string result;
     ...
     return result;
 }

 This is a lot more general and extensible. Thanks!

 Why do you think there'd be dramatic complication of code? 
 (Granted, at some point we must acknowledge that some egg 
 breaking is necessary for the proverbial omelette.)


 From my experience with C++ containers.  Having an alias for a 
type is okay, but bank of aliases where one is a pointer to the 
type, one is a const pointer to the type, etc, makes writing the 
involved code feel really unnatural.



 The thing is, again, we must make some changes if we want D to 
 be usable without a GC. One of them is e.g. to not allocate 
 built-in slices all over the place.


So let the user supply a scratch buffer that will hold the 
result?  With the RC approach we're still allocating, they just 
aren't built-in slices, correct?



 That would be overreacting :o).


I hope it is :-)



 Oct 01 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/1/14, 7:03 AM, Sean Kelly wrote:

 So let the user supply a scratch buffer that will hold the result?  With
 the RC approach we're still allocating, they just aren't built-in
 slices, correct?


Correct. -- Andrei



 Oct 01 2014



prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


29-Sep-2014 14:49, Andrei Alexandrescu пишет:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2 ext)
 if (...)
 {
      static if (mmp == gc) alias S = string;
      else alias S = RCString;
      S result;
      ...
      return result;
 }


Incredible code bloat? Boilerplate in each function for the win?
I'm at loss as to how it would make things better.


-- 
Dmitry Olshansky



 Sep 30 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 11:06 AM, Dmitry Olshansky wrote:

 29-Sep-2014 14:49, Andrei Alexandrescu пишет:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2
 ext)
 if (...)
 {
      static if (mmp == gc) alias S = string;
      else alias S = RCString;
      S result;
      ...
      return result;
 }


 Incredible code bloat? Boilerplate in each function for the win?
 I'm at loss as to how it would make things better.


Sean's idea to make string an alias of the policy takes care of this 
concern. -- Andrei



 Oct 01 2014



  parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Wed, Oct 01, 2014 at 02:51:08AM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 9/30/14, 11:06 AM, Dmitry Olshansky wrote:

29-Sep-2014 14:49, Andrei Alexandrescu пишет:

auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2
ext)
if (...)
{
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
}


Incredible code bloat? Boilerplate in each function for the win?
I'm at loss as to how it would make things better.


 
 Sean's idea to make string an alias of the policy takes care of this
 concern. -- Andrei


But Sean's idea only takes strings into account. Strings aren't the only
allocated resource Phobos needs to deal with. So extrapolating from that
idea, each memory management struct (or whatever other aggregate we end
up using), say call it MMP, will have to define MMP.string, MMP.jsonNode
(since parseJSON() need to allocate not only strings but JSON nodes),
MMP.redBlackTreeNode, MMP.listNode, MMP.userDefinedNode, ...

Nope, still don't see how this could work. Please clarify, kthx.


T

-- 
Sometimes the best solution to morale problems is just to fire all of the
unhappy people. -- despair.com



 Oct 01 2014



 next sibling parent  "Kiith-Sa" <kiithsacmp gmail.com>  writes:


On Wednesday, 1 October 2014 at 17:53:43 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 On Wed, Oct 01, 2014 at 02:51:08AM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 On 9/30/14, 11:06 AM, Dmitry Olshansky wrote:

29-Sep-2014 14:49, Andrei Alexandrescu пишет:

auto setExtension(MemoryManagementPolicy mmp = gc, R1, 
R2)(R1 path, R2
ext)
if (...)
{
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
}


Incredible code bloat? Boilerplate in each function for the 
win?
I'm at loss as to how it would make things better.


 
 Sean's idea to make string an alias of the policy takes care 
 of this
 concern. -- Andrei


 But Sean's idea only takes strings into account. Strings aren't 
 the only
 allocated resource Phobos needs to deal with. So extrapolating 
 from that
 idea, each memory management struct (or whatever other 
 aggregate we end
 up using), say call it MMP, will have to define MMP.string, 
 MMP.jsonNode
 (since parseJSON() need to allocate not only strings but JSON 
 nodes),
 MMP.redBlackTreeNode, MMP.listNode, MMP.userDefinedNode, ...

 Nope, still don't see how this could work. Please clarify, kthx.


 T



MMP.Ref!redBlackTreeNode ?

(where Ref is e.g. a ref-counted pointer type (like RefCounted 
but with class support) for RC MMP but plain GC reference for GC 
MMP, etc.)

I kinda like this idea, since it might possibly allow 
user-defined memory management policies (which wouldn't get 
special compiler treatment that e.g. RC may need, though).



 Oct 01 2014



prev sibling next sibling parent reply "Sean Kelly" <sean invisibleduck.org>  writes:


On Wednesday, 1 October 2014 at 17:53:43 UTC, H. S. Teoh via
Digitalmars-d wrote:

 But Sean's idea only takes strings into account. Strings aren't 
 the only
 allocated resource Phobos needs to deal with. So extrapolating 
 from that
 idea, each memory management struct (or whatever other 
 aggregate we end
 up using), say call it MMP, will have to define MMP.string, 
 MMP.jsonNode
 (since parseJSON() need to allocate not only strings but JSON 
 nodes),
 MMP.redBlackTreeNode, MMP.listNode, MMP.userDefinedNode, ...

 Nope, still don't see how this could work. Please clarify, kthx.


Assuming you're willing to take the memoryModel type as a
template argument, I imagine we could do something where the user
can specialize the memoryModel for their own types, a bit like
how information is derived for iterators in C++.  The problem is
that this still means passing the memoryModel in as a template
argument.  What I'd really want is for it to be a global, except
that templated virtuals is logically impossible.  I guess
something could maybe be sorted out via a factory design, but
that's not terribly D-like.  I'm at a loss for how to make this
memoryModel thing work the way I'd actually want it to if I were
to use it.



 Oct 01 2014



  parent  "Cliff" <cliff.s.hudson gmail.com>  writes:


On Wednesday, 1 October 2014 at 18:37:50 UTC, Sean Kelly wrote:

 On Wednesday, 1 October 2014 at 17:53:43 UTC, H. S. Teoh via
 Digitalmars-d wrote:

 But Sean's idea only takes strings into account. Strings 
 aren't the only
 allocated resource Phobos needs to deal with. So extrapolating 
 from that
 idea, each memory management struct (or whatever other 
 aggregate we end
 up using), say call it MMP, will have to define MMP.string, 
 MMP.jsonNode
 (since parseJSON() need to allocate not only strings but JSON 
 nodes),
 MMP.redBlackTreeNode, MMP.listNode, MMP.userDefinedNode, ...

 Nope, still don't see how this could work. Please clarify, 
 kthx.


 Assuming you're willing to take the memoryModel type as a
 template argument, I imagine we could do something where the 
 user
 can specialize the memoryModel for their own types, a bit like
 how information is derived for iterators in C++.  The problem is
 that this still means passing the memoryModel in as a template
 argument.  What I'd really want is for it to be a global, except
 that templated virtuals is logically impossible.  I guess
 something could maybe be sorted out via a factory design, but
 that's not terribly D-like.  I'm at a loss for how to make this
 memoryModel thing work the way I'd actually want it to if I were
 to use it.


If you were to forget D restrictions for a moment, and consider 
an idealized language, how would you express this?  Maybe 
providing that will trigger some ideas from people beyond what we 
have seen so far by removing implied restrictions.



 Oct 01 2014



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/1/14, 10:51 AM, H. S. Teoh via Digitalmars-d wrote:

 But Sean's idea only takes strings into account. Strings aren't the only
 allocated resource Phobos needs to deal with. So extrapolating from that
 idea, each memory management struct (or whatever other aggregate we end
 up using), say call it MMP, will have to define MMP.string, MMP.jsonNode
 (since parseJSON() need to allocate not only strings but JSON nodes),
 MMP.redBlackTreeNode, MMP.listNode, MMP.userDefinedNode, ...

 Nope, still don't see how this could work. Please clarify, kthx.


There's management for T[], pointers to structs, pointers to class 
objects, associative arrays, and that covers everything. -- Andrei



 Oct 01 2014



prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


Ok, here are my few cents:

On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu 
wrote:

 Back when I've first introduced RCString I hinted that we have 
 a larger strategy in mind. Here it is.

 The basic tenet of the approach is to reckon and act on the 
 fact that memory allocation (the subject of allocators) is an 
 entirely distinct topic from memory management, and more 
 generally resource management. This clarifies that it would be 
 wrong to approach alternatives to GC in Phobos by means of 
 allocators. GC is not only an approach to memory allocation, 
 but also an approach to memory management. Reducing it to 
 either one is a mistake. In hindsight this looks rather obvious 
 but it has caused me and many people better than myself a lot 
 of headache.


I would argue that GC is at its core _only_ a memory management 
strategy. It just so happens that the one in D's runtime also 
comes with an allocator, with which it is tightly integrated. In 
theory, a GC can work with any (and multiple) allocators, and you 
could of course also call GC.free() manually, because, as you 
say, management and allocation are entirely distinct topics.


 That said allocators are nice to have and use, and I will 
 definitely follow up with std.allocator. However, std.allocator 
 is not the key to a  nogc Phobos.


Agreed.


 Nor are ranges. There is an attitude that either output ranges, 
 or input ranges in conjunction with lazy computation, would 
 solve the issue of creating garbage. 
 https://github.com/D-Programming-Language/phobos/pull/2423 is a 
 good illustration of the latter approach: a range would be 
 lazily created by chaining stuff together. A range-based 
 approach would take us further than the allocators, but I see 
 the following issues with it:

 (a) the whole approach doesn't stand scrutiny for non-linear 
 outputs, e.g. outputting some sort of associative array or 
 really any composite type quickly becomes tenuous either with 
 an output range (eager) or with exposing an input range (lazy);

 (b) makes the style of programming without GC radically 
 different, and much more cumbersome, than programming with GC; 
 as a consequence, programmers who consider changing one 
 approach to another, or implementing an algorithm neutral to 
 it, are looking at a major rewrite;

 (c) would make D/ nogc a poor cousin of C++. This is quite out 
 of character; technically, I have long gotten used to seeing 
 most elaborate C++ code like poor emulation of simple D idioms. 
 But C++ has spent years and decades taking to perfection an 
 approach without a tracing garbage collector. A departure from 
 that would need to be superior, and that doesn't seem to be the 
 case with range-based approaches.


I agree with this, too.


 ===========

 Now that we clarified that these existing attempts are not 
 going to work well, the question remains what does. For Phobos 
 I'm thinking of defining and using three policies:

 enum MemoryManagementPolicy { gc, rc, mrc }
 immutable
     gc = ResourceManagementPolicy.gc,
     rc = ResourceManagementPolicy.rc,
     mrc = ResourceManagementPolicy.mrc;

 The three policies are:

 (a) gc is the classic garbage-collected style of management;

 (b) rc is a reference-counted style still backed by the GC, 
 i.e. the GC will still be able to pick up cycles and other 
 kinds of leaks.

 (c) mrc is a reference-counted style backed by malloc.

 (It should be possible to collapse rc and mrc together and make 
 the distinction dynamically, at runtime. I'm distinguishing 
 them statically here for expository purposes.)

 The policy is a template parameter to functions in Phobos (and 
 elsewhere), and informs the functions e.g. what types to 
 return. Consider:

 auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 
 path, R2 ext)
 if (...)
 {
     static if (mmp == gc) alias S = string;
     else alias S = RCString;
     S result;
     ...
     return result;
 }

 On the caller side:

 auto p1 = setExtension("hello", ".txt"); // fine, use gc
 auto p2 = setExtension!gc("hello", ".txt"); // same
 auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc

 So by default it's going to continue being business as usual, 
 but certain functions will allow passing in a (defaulted) 
 policy for memory management.


This, however, I disagree with strongly. For one thing - this has 
already been noted by others - it would make the functions' 
implementation extremely ugly (`static if` hell), it would make 
them harder to unit test, and from a user's point of view, it's 
very tedious and might interfere badly with UFCS.

But more importantly, IMO, it's the wrong thing to do. These 
functions shouldn't know anything about memory management policy 
at all. They allocate, which means they need to know about 
_allocation_ policy, but memory _management_ policy needs to be 
decided by the user.

Now, your suggestion in a way still leaves that decision to the 
user, but does so in a very intrusive way, by passing a template 
flag. This is clearly a violation of the separation of concerns. 
Contrary to the typical case, implementation details of the 
user's code leak into the library code, and not the other way 
round, but that's just as bad.

I'm convinced this isn't necessary. Let's take `setExtension()` 
as an example, standing in for any of a class of similar 
functions. This function allocates memory, returns it, and 
abandons it; it gives up ownership of the memory. The fact that 
the memory has been freshly allocated means that it is (head) 
unique, and therefore the caller (= library user) can take over 
the ownership. This, in turn, means that the caller can decide 
how she wants to manage it.

(I'll try to make a sketch on how this can be implemented in 
another post.)

As a conclusion, I would say that APIs should strive for the 
following principles, in this order:

1. Avoid allocation altogether, for example by laziness (ranges), 
or by accepting sinks.

2. If allocations are necessary (or desirable, to make the API 
more easily usable), try hard to return a unique value (this of 
course needs to be expressed in the return type).

3. If both of the above fails, only then return a GCed pointer, 
or alternatively provide several variants of the function (though 
this shouldn't be necessary often). An interesting alternative: 
Instead of passing a flag directly describing the policy, pass 
the function a type that it should wrap it's return value in.

As for the _allocation_ strategy: It indeed needs to be 
configurable, but here, the same objections against a template 
parameter apply. As the allocator doesn't necessarily need to be 
part of the type, a (thread) global variable can be used to 
specify it. This lends itself well to idioms like

     with(MyAllocator alloc) {
         // ...
     }


 Destroy!


Done :-)



 Sep 30 2014



 next sibling parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 30 September 2014 at 19:10:19 UTC, Marc Schütz wrote:

 I'm convinced this isn't necessary. Let's take `setExtension()` 
 as an example, standing in for any of a class of similar 
 functions. This function allocates memory, returns it, and 
 abandons it; it gives up ownership of the memory. The fact that 
 the memory has been freshly allocated means that it is (head) 
 unique, and therefore the caller (= library user) can take over 
 the ownership. This, in turn, means that the caller can decide 
 how she wants to manage it.

 (I'll try to make a sketch on how this can be implemented in 
 another post.)


Ok. What we need for it:

1)  unique, or a way to expressly specify uniqueness on a 
function's return type, as well as restrict function params by it 
(and preferably overloading on uniqueness). DMD already has this 
concept internally, it just needs to be formalized.

2) A few modifications to RefCounted to be constructable from 
unique values.

3) A wrapper type similar to std.typecons.Unique, that also 
supports moving. Let's called it Owned(T).

4) Borrowing.

setExtension() can then look like this:

     Owned!string setExtension(in char[] path, in char[] ext);

To be used:

     void saveFileAs(in char[] name) {
         import std.path: setExtension;
         import std.file: write;
         name.                    // scope const(char[])
             setExtension("txt"). // Owned!string
             write(data);
     }

The Owned(T) value implicitly converts to `scope!this(T)` via 
alias this; it can therefore be conveniently passed to 
std.file.write() (which already takes the filename as `in`) 
without copying or moving. The value then is released 
automatically at the end of the statement, because it is only a 
temporary and is not assigned to a variable.

For transferring ownership:

     RefCounted!string[] filenames;
     // ...
     filenames ~= name.setExtension("txt").release;

`Owned!T.release()` returns the payload as a unique value, and 
resets the payload to it's init value (in this case `null`). 
RefCounted's constructor then accepts this unique value and takes 
ownership of it. When the Owned value's destructor is called, it 
finds the payload to be null and doesn't free the memory. 
Inlining and subsequent optimization can turn the destructor into 
a no-op in this case.

Optionally, Owned!T can provide an `alias this` to its release 
method; in this case, the method doesn't need to be called 
explicitly. It is however debatable whether being explicit with 
moving isn't the better choice.



 Sep 30 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 12:10 PM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 I would argue that GC is at its core _only_ a memory management
 strategy. It just so happens that the one in D's runtime also comes with
 an allocator, with which it is tightly integrated. In theory, a GC can
 work with any (and multiple) allocators, and you could of course also
 call GC.free() manually, because, as you say, management and allocation
 are entirely distinct topics.


I'm not very sure. A GC might need to interoperate closely with the 
allocator. -- Andrei



 Oct 01 2014



  parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Wednesday, 1 October 2014 at 09:52:46 UTC, Andrei Alexandrescu 
wrote:

 On 9/30/14, 12:10 PM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 I would argue that GC is at its core _only_ a memory management
 strategy. It just so happens that the one in D's runtime also 
 comes with
 an allocator, with which it is tightly integrated. In theory, 
 a GC can
 work with any (and multiple) allocators, and you could of 
 course also
 call GC.free() manually, because, as you say, management and 
 allocation
 are entirely distinct topics.


 I'm not very sure. A GC might need to interoperate closely with 
 the allocator. -- Andrei


It needs to know what to scan (ideally with type info), and which 
allocator to release memory with, but it doesn't need to be an 
allocator itself. It certainly helps with the implementation, but 
ideally there would be a well defined interface between 
allocators and GCs, so that both can be plugged in as desired, 
even with multiple GCs in parallel.



 Oct 01 2014



prev sibling  parent reply "Oren Tirosh" <orent hishome.net>  writes:


On Tuesday, 30 September 2014 at 19:10:19 UTC, Marc Schütz wrote:

 [...]

 I'm convinced this isn't necessary. Let's take `setExtension()` 
 as an example, standing in for any of a class of similar 
 functions. This function allocates memory, returns it, and 
 abandons it; it gives up ownership of the memory. The fact that 
 the memory has been freshly allocated means that it is (head) 
 unique, and therefore the caller (= library user) can take over 
 the ownership. This, in turn, means that the caller can decide 
 how she wants to manage it.


Bingo. Have some way to mark the function return type as a unique 
pointer. This does not imply full-fledged unique pointer type 
support in the language - just enough to have the caller ensure 
continuity of memory management policy from there.

One problem with actually implementing this is that using 
reference counting as a memory management policy requires extra 
space for the reference counter in the object, just as garbage 
collection requires support for scanning and identification of 
interior object memory range. While allocation and memory 
management may be quite independent in theory, practical high 
performance implementations tend to be intimately related.


 (I'll try to make a sketch on how this can be implemented in 
 another post.)


Do elaborate!


 As a conclusion, I would say that APIs should strive for the 
 following principles, in this order:

 1. Avoid allocation altogether, for example by laziness 
 (ranges), or by accepting sinks.

 2. If allocations are necessary (or desirable, to make the API 
 more easily usable), try hard to return a unique value (this of 
 course needs to be expressed in the return type).

 3. If both of the above fails, only then return a GCed pointer, 
 or alternatively provide several variants of the function 
 (though this shouldn't be necessary often). An interesting 
 alternative: Instead of passing a flag directly describing the 
 policy, pass the function a type that it should wrap it's 
 return value in.

 As for the _allocation_ strategy: It indeed needs to be 
 configurable, but here, the same objections against a template 
 parameter apply. As the allocator doesn't necessarily need to 
 be part of the type, a (thread) global variable can be used to 
 specify it. This lends itself well to idioms like

     with(MyAllocator alloc) {
         // ...
     }


Assuming there is some dependency between the allocator and the 
memory management policy I guess this would be initialized on 
thread start that cannot be modified later. All code running 
inside the thread would need to either match the configured 
policy, not handle any kind of pointers or use a limited subset 
of unique pointers. Another way to ensure that code can run on 
either RC or GC is to make certain objects (specifically, 
Exceptions) always allocate a reference counter, regardless of 
the currently configured policy.



 Oct 01 2014



 next sibling parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Oren Tirosh:


 Bingo. Have some way to mark the function return type as a 
 unique pointer. This does not imply full-fledged unique pointer 
 type support in the language


Let's have full-fledged memory zones tracking in the D type 
system :-)

Bye,
bearophile



 Oct 01 2014



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/1/14, 8:48 AM, Oren Tirosh wrote:

 On Tuesday, 30 September 2014 at 19:10:19 UTC, Marc Schütz wrote:

 [...]

 I'm convinced this isn't necessary. Let's take `setExtension()` as an
 example, standing in for any of a class of similar functions. This
 function allocates memory, returns it, and abandons it; it gives up
 ownership of the memory. The fact that the memory has been freshly
 allocated means that it is (head) unique, and therefore the caller (=
 library user) can take over the ownership. This, in turn, means that
 the caller can decide how she wants to manage it.


 Bingo. Have some way to mark the function return type as a unique
 pointer.


I'm skeptical about this approach (though clearly we need to explore it 
for e.g. passing ownership of data across threads). For strings and 
other "casual" objects I think we should focus on GC/RC strategies. This 
is because people do things like:

auto s = setExtension(s1, s2);

and then attempt to use s as a regular variable (copy it etc). Making s 
unique would make usage quite surprising and cumbersome.


Andrei



 Oct 01 2014



 next sibling parent reply "Oren T" <orent hishome.net>  writes:


On Wednesday, 1 October 2014 at 17:13:38 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/14, 8:48 AM, Oren Tirosh wrote:

 On Tuesday, 30 September 2014 at 19:10:19 UTC, Marc Schütz 
 wrote:

 [...]

 I'm convinced this isn't necessary. Let's take 
 `setExtension()` as an
 example, standing in for any of a class of similar functions. 
 This
 function allocates memory, returns it, and abandons it; it 
 gives up
 ownership of the memory. The fact that the memory has been 
 freshly
 allocated means that it is (head) unique, and therefore the 
 caller (=
 library user) can take over the ownership. This, in turn, 
 means that
 the caller can decide how she wants to manage it.


 Bingo. Have some way to mark the function return type as a 
 unique
 pointer.


 I'm skeptical about this approach (though clearly we need to 
 explore it for e.g. passing ownership of data across threads). 
 For strings and other "casual" objects I think we should focus 
 on GC/RC strategies. This is because people do things like:

 auto s = setExtension(s1, s2);

 and then attempt to use s as a regular variable (copy it etc). 
 Making s unique would make usage quite surprising and 
 cumbersome.


The idea is that the unique property is very short-lived: the 
caller immediately assigns it to a pointer of the appropriate 
policy: either RC or GC. This keeps the callee agnostic of the 
chosen policy and does not require templating multiple versions 
of the code. The allocator configured for the thread must match 
the generated code at the call site i.e. if the caller uses RC 
pointers the allocator must allocate space for the reference 
counter (at negative offset to keep compatibility).



 Oct 01 2014



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/1/14, 10:25 AM, Oren T wrote:

 The idea is that the unique property is very short-lived: the caller
 immediately assigns it to a pointer of the appropriate policy: either RC
 or GC. This keeps the callee agnostic of the chosen policy and does not
 require templating multiple versions of the code. The allocator
 configured for the thread must match the generated code at the call site
 i.e. if the caller uses RC pointers the allocator must allocate space
 for the reference counter (at negative offset to keep compatibility).


This all... looks arcane. I'm not sure how it can even made to work if 
user code just uses "auto". -- Andrei



 Oct 01 2014



 next sibling parent  "Oren T" <orent hishome.net>  writes:


On Wednesday, 1 October 2014 at 17:33:34 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/14, 10:25 AM, Oren T wrote:

 The idea is that the unique property is very short-lived: the 
 caller
 immediately assigns it to a pointer of the appropriate policy: 
 either RC
 or GC. This keeps the callee agnostic of the chosen policy and 
 does not
 require templating multiple versions of the code. The allocator
 configured for the thread must match the generated code at the 
 call site
 i.e. if the caller uses RC pointers the allocator must 
 allocate space
 for the reference counter (at negative offset to keep 
 compatibility).


 This all... looks arcane. I'm not sure how it can even made to 
 work if user code just uses "auto". -- Andrei


At the moment,  nogc code can't call any function returning a 
pointer. Under this scheme  nogc is allowed to call either code 
that returns an explicitly RC ty



 Oct 01 2014



prev sibling  parent  "Oren T" <orent hishome.net>  writes:


On Wednesday, 1 October 2014 at 17:33:34 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/14, 10:25 AM, Oren T wrote:

 The idea is that the unique property is very short-lived: the 
 caller
 immediately assigns it to a pointer of the appropriate policy: 
 either RC
 or GC. This keeps the callee agnostic of the chosen policy and 
 does not
 require templating multiple versions of the code. The allocator
 configured for the thread must match the generated code at the 
 call site
 i.e. if the caller uses RC pointers the allocator must 
 allocate space
 for the reference counter (at negative offset to keep 
 compatibility).


 This all... looks arcane. I'm not sure how it can even made to 
 work if user code just uses "auto". -- Andrei


At the moment,  nogc code can't call any function returning a 
pointer. Under this scheme  nogc is allowed to call either code 
that returns an explicitly RC type (Exception, RCString) or code 
returning an "agnostic" unique pointer that may be used from 
either  gc or  nogc code.
I already see some holes and problems, but I wonder if something 
along these lines may be made to work.



 Oct 01 2014



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 01/10/14 19:25, Oren T wrote:


 The idea is that the unique property is very short-lived: the caller
 immediately assigns it to a pointer of the appropriate policy: either RC
 or GC. This keeps the callee agnostic of the chosen policy and does not
 require templating multiple versions of the code. The allocator
 configured for the thread must match the generated code at the call site
 i.e. if the caller uses RC pointers the allocator must allocate space
 for the reference counter (at negative offset to keep compatibility).


Can't we do something like this, or it might be what you're proposing:

Foo foo () { return new Foo; }

 gc a = foo(); // a contains an instance of Foo allocated with the GC
 rc b = foo(); // b contains an instance of Foo allocated with the RC 
allocator

-- 
/Jacob Carlborg



 Oct 01 2014



  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Thursday, 2 October 2014 at 06:29:24 UTC, Jacob Carlborg wrote:

  gc a = foo(); // a contains an instance of Foo allocated with 
 the GC
  rc b = foo(); // b contains an instance of Foo allocated with 
 the RC allocator


That would be better, but how do you deal with "bar(foo())" ? 
Context dependent instantiation is a semantic challenge when you 
also have overloading, but I guess you can get somewhere if you 
make whole program optimization mandatory and use a 
state-of-the-art constraint solver to handle the type system. 
Could lead you to NP-complete type resolution? But still doable 
(in most cases).

I think you basically have 2 realistic choices if you want 
easy-going syntax for the end user:

1. implement rc everywhere in standard libraries and make it 
possible to turn off rc in a call-chain by having compiler 
support (and whole program optimization). To support manual 
management you need some kind of protocol for traversing 
allocated data-structures to free them.

e.g.:

define memory strategy  malloc = 
some…manual…allocation…strategy…description;

auto a = bar(foo()); //  use gc or rc based on compiler flag
auto a =  rc( bar(foo()) ); // use rc in a gc context
auto a =  malloc( bar(foo()) ); // manual management (requires a 
protocol for traversal of recursive datastructures)


2. provide allocation strategy as a parameter

e.g.:

auto a = foo(); // alloc with gc
auto a = foo!rc(); // alloc with rc
auto a = foo!malloc(); // alloc with malloc

But going the C++ way of having explicit allocators and 
non-embedded reference counters (double indirectio) probably is 
the easier solution in terms of bringing D to completion.

How many years are you going to spend on making D ref count by 
default in a flawless and performant manner? Sure having RC being 
as easy to use as GC is a nice idea, but if it turns out to be 
either slower or more bug ridden than GC, then what is the point?

Note that:

1. A write to a ref-count means the 64 bytes cacheline is dirty 
and has to be written back to memory. So you don't write 4 bytes, 
  you write to 64 bytes. That's pretty expensive.

2. The memory bus is increasingly becoming the bottle neck of 
hardware architectures.

=> RC everywhere without heavy duty compiler/hardware support is 
a bad long term idea.



 Oct 02 2014



  parent reply Jacob Carlborg <doob me.com>  writes:


On 02/10/14 11:41, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:


 That would be better, but how do you deal with "bar(foo())" ? Context
 dependent instantiation is a semantic challenge when you also have
 overloading, but I guess you can get somewhere if you make whole program
 optimization mandatory and use a state-of-the-art constraint solver to
 handle the type system. Could lead you to NP-complete type resolution?
 But still doable (in most cases).


I haven't really thought how it could be implemented but I was hoping 
that the caller could magically decide the allocation strategy instead 
of the callee. It looks like Rust is doing something like that but I 
haven't looked at it in detail.

-- 
/Jacob Carlborg



 Oct 02 2014



  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Thursday, 2 October 2014 at 11:41:14 UTC, Jacob Carlborg wrote:

 I haven't really thought how it could be implemented but I was 
 hoping that the caller could magically decide the allocation 
 strategy instead of the callee. It looks like Rust is doing 
 something like that but I haven't looked at it in detail.


I haven't looked at Rust in detail, but doesn't the Rust compiler 
take full control over memory management? I think that is a good 
idea, but it is at odds with D's general direction.



 Oct 02 2014



  parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Thursday, 2 October 2014 at 13:29:58 UTC, Ola Fosheim Grøstad
wrote:

 On Thursday, 2 October 2014 at 11:41:14 UTC, Jacob Carlborg 
 wrote:

 I haven't really thought how it could be implemented but I was 
 hoping that the caller could magically decide the allocation 
 strategy instead of the callee. It looks like Rust is doing 
 something like that but I haven't looked at it in detail.


 I haven't looked at Rust in detail, but doesn't the Rust 
 compiler take full control over memory management? I think that 
 is a good idea, but it is at odds with D's general direction.


Rust makes use of the type system and the borrow checker to
validate how the pointers are being used.

The usual errors when dealing with pointers are compile time
errors in Rust.

--
Paulo



 Oct 02 2014



  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Thursday, 2 October 2014 at 18:52:18 UTC, Paulo Pinto wrote:

 Rust makes use of the type system and the borrow checker to
 validate how the pointers are being used.

 The usual errors when dealing with pointers are compile time
 errors in Rust.


They constrain usage so that you cannot share mutable objects. It 
is described in reasonable high level here:

http://doc.rust-lang.org/0.11.0/rust.html#memory-and-concurrency-models

But is sketchy on implementation details, semantic restrictions 
that follows and consequences when interacting with foreign code 
etc.



 Oct 02 2014



  parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Thursday, 2 October 2014 at 19:45:17 UTC, Ola Fosheim Grøstad 
wrote:

 But is sketchy on implementation details, semantic restrictions 
 that follows and consequences when interacting with foreign 
 code etc.


Some Rust details. «sendable» means that a reference can be 
transferred to another thread (or task/fiber/whatever).

 From http://doc.rust-lang.org/std/gc/ :

«The Gc type provides shared ownership of an immutable value. 
Destruction is not deterministic, and will occur some time 
between every Gc handle being gone and the end of the task. The 
garbage collector is task-local so Gc<T> is not sendable.»

 From http://doc.rust-lang.org/std/rc/index.html :

«The Rc type provides shared ownership of an immutable value. 
Destruction is deterministic, and will occur as soon as the last 
owner is gone. It is marked as non-sendable because it avoids the 
overhead of atomic reference counting.

The downgrade method can be used to create a non-owning Weak 
pointer to the box. A Weak pointer can be upgraded to an Rc 
pointer, but will return None if the value has already been 
freed.»

So… they don't really solve the issues a  nogc version of D 
should be able to deal with beyond having built-in unique_ptr 
style semantics?

Or?



 Oct 02 2014



  parent reply "Paulo Pinto" <pjmlp progtools.org>  writes:


On Thursday, 2 October 2014 at 20:10:42 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 2 October 2014 at 19:45:17 UTC, Ola Fosheim 
 Grøstad wrote:

 But is sketchy on implementation details, semantic 
 restrictions that follows and consequences when interacting 
 with foreign code etc.


 Some Rust details. «sendable» means that a reference can be 
 transferred to another thread (or task/fiber/whatever).

 From http://doc.rust-lang.org/std/gc/ :

 «The Gc type provides shared ownership of an immutable value. 
 Destruction is not deterministic, and will occur some time 
 between every Gc handle being gone and the end of the task. The 
 garbage collector is task-local so Gc<T> is not sendable.»

 From http://doc.rust-lang.org/std/rc/index.html :

 «The Rc type provides shared ownership of an immutable value. 
 Destruction is deterministic, and will occur as soon as the 
 last owner is gone. It is marked as non-sendable because it 
 avoids the overhead of atomic reference counting.

 The downgrade method can be used to create a non-owning Weak 
 pointer to the box. A Weak pointer can be upgraded to an Rc 
 pointer, but will return None if the value has already been 
 freed.»

 So… they don't really solve the issues a  nogc version of D 
 should be able to deal with beyond having built-in unique_ptr 
 style semantics?

 Or?


The Gc type is gone as of this week.

https://github.com/rust-lang/meeting-minutes/blob/master/weekly-meetings/2014-09-30.md



 Oct 02 2014



  parent  "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=  writes:


On Thursday, 2 October 2014 at 20:42:16 UTC, Paulo Pinto wrote:

 The Gc type is gone as of this week.

 https://github.com/rust-lang/meeting-minutes/blob/master/weekly-meetings/2014-09-30.md


Thanks, apparently they do it because they want to make a proper 
tracing gc available later:

https://github.com/pnkfelix/rfcs/blob/fsk-remove-refcounting-gc-of-t/active/0000-remove-refcounting-gc-of-t.md



 Oct 02 2014



prev sibling  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Wednesday, 1 October 2014 at 17:13:38 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/14, 8:48 AM, Oren Tirosh wrote:

 Bingo. Have some way to mark the function return type as a 
 unique
 pointer.


 I'm skeptical about this approach (though clearly we need to 
 explore it for e.g. passing ownership of data across threads). 
 For strings and other "casual" objects I think we should focus 
 on GC/RC strategies. This is because people do things like:

 auto s = setExtension(s1, s2);

 and then attempt to use s as a regular variable (copy it etc). 
 Making s unique would make usage quite surprising and 
 cumbersome.


Sure? I already showed in an example how it is possible to chain 
calls seamlessly that return unique objects. The users would only 
notice it when they are trying to make a real copy (i.e. not 
borrowing). Do you think this happens frequently enough to be of 
concern?



 Oct 01 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/1/14, 1:56 PM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Wednesday, 1 October 2014 at 17:13:38 UTC, Andrei Alexandrescu wrote:

 On 10/1/14, 8:48 AM, Oren Tirosh wrote:

 Bingo. Have some way to mark the function return type as a unique
 pointer.


 I'm skeptical about this approach (though clearly we need to explore
 it for e.g. passing ownership of data across threads). For strings and
 other "casual" objects I think we should focus on GC/RC strategies.
 This is because people do things like:

 auto s = setExtension(s1, s2);

 and then attempt to use s as a regular variable (copy it etc). Making
 s unique would make usage quite surprising and cumbersome.


 Sure? I already showed in an example how it is possible to chain calls
 seamlessly that return unique objects. The users would only notice it
 when they are trying to make a real copy (i.e. not borrowing). Do you
 think this happens frequently enough to be of concern?


I'd think so. -- Andrei



 Oct 01 2014



prev sibling next sibling parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Wednesday, 1 October 2014 at 15:48:39 UTC, Oren Tirosh wrote:

 On Tuesday, 30 September 2014 at 19:10:19 UTC, Marc Schütz 
 wrote:
 One problem with actually implementing this is that using 
 reference counting as a memory management policy requires extra 
 space for the reference counter in the object, just as garbage 
 collection requires support for scanning and identification of 
 interior object memory range. While allocation and memory 
 management may be quite independent in theory, practical high 
 performance implementations tend to be intimately related.


 (I'll try to make a sketch on how this can be implemented in 
 another post.)


 Do elaborate!


 As a conclusion, I would say that APIs should strive for the 
 following principles, in this order:

 1. Avoid allocation altogether, for example by laziness 
 (ranges), or by accepting sinks.

 2. If allocations are necessary (or desirable, to make the API 
 more easily usable), try hard to return a unique value (this 
 of course needs to be expressed in the return type).

 3. If both of the above fails, only then return a GCed 
 pointer, or alternatively provide several variants of the 
 function (though this shouldn't be necessary often). An 
 interesting alternative: Instead of passing a flag directly 
 describing the policy, pass the function a type that it should 
 wrap it's return value in.

 As for the _allocation_ strategy: It indeed needs to be 
 configurable, but here, the same objections against a template 
 parameter apply. As the allocator doesn't necessarily need to 
 be part of the type, a (thread) global variable can be used to 
 specify it. This lends itself well to idioms like

    with(MyAllocator alloc) {
        // ...
    }


 Assuming there is some dependency between the allocator and the 
 memory management policy I guess this would be initialized on 
 thread start that cannot be modified later. All code running 
 inside the thread would need to either match the configured 
 policy, not handle any kind of pointers or use a limited subset 
 of unique pointers. Another way to ensure that code can run on 
 either RC or GC is to make certain objects (specifically, 
 Exceptions) always allocate a reference counter, regardless of 
 the currently configured policy.


I don't have all answers to these questions. Still, I'm convinced 
this is doable.

A straight-forwarding and general way to convert a unique object 
to a ref-counted one is to allocate new memory for it plus the 
reference count, move the original object into it, and release 
the original memory. This is safe, because there can be no 
external pointers to the object, as it is unique. Of course, this 
can be optimized if the allocator supports extending an 
allocation. It could then preallocate a few extra bytes at the 
end to make the extend operation always succeed, similar to your 
suggestion to always allocate a reference counter.

I think the most difficult part is to find an efficient and 
user-friendly way for the wrapper types to get at the allocator. 
Maybe the allocators should all implement an interface (a real 
one, not duck-typing). The wrappers (Owned, RC) can then include 
a pointer to the allocator (or for RC, embed it next to the 
reference count). This would make it possible to specify a 
(thread) global default allocator at runtime, which all library 
functions use by convention (for example let's call it `alloc`, 
then they would call `alloc.make!MyStruct()`). At the same time, 
it is safe to change the default allocator at any time, and to 
use different allocators in parallel in the same thread.

The alternative is obviously a template parameter to the function 
that returns the unique object. But this unfortunately is then 
not restricted to just the function, but "infects" the return 
type, too. And from there, it needs to spread to the RC wrapper, 
or any containers. Thus we'd have incompatible RC types, which I 
would imagine would be very inconvenient and restrictive. 
Besides, it would probably be too tedious to specify the 
allocator everywhere.

Therfore, I think the additional cost of an allocator interface 
pointer is worth it. For Owned!T (with T being a pointer or 
reference), it would just be two words, which we can return 
efficiently. We already have slices doing that, and AFAIK there's 
no significantly worse performance because of them.



 Oct 01 2014



prev sibling  parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Wednesday, 1 October 2014 at 15:48:39 UTC, Oren Tirosh wrote:

 On Tuesday, 30 September 2014 at 19:10:19 UTC, Marc Schütz 
 wrote:

 (I'll try to make a sketch on how this can be implemented in 
 another post.)


 Do elaborate!


Here's an example implementation of what I have in mind (totally 
untested and won't compile because of `scope`):

http://wiki.dlang.org/User:Schuetzm/RC,_Owned_and_allocators

This is just a sketch to explain the general idea. Some things 
probably won't work as implemented, especially the disable 
postblit and opAssign() of Owned!T. I think it needs to implement 
implicit moving, otherwise one would have to call `release()` 
everywhere.

As in the other post, the function that produces the value 
returns Owned!T. The types don't require  unique however 
(although integration with DMD's idea of unique would still be 
useful).

Because of auto-borrowing via alias this, Owned!T and RC!T both 
can pass their payloads to functions that accept them by `scope`. 
The ref-count is not touched for borrowing.

Usage examples:

     Owned!string setExtension(in char[] path, in char[] ext);

     void saveFileAs(in char[] name) {
         import std.path: setExtension;
         import std.file: write;
         name.                    // scope const(char[])
             setExtension("txt"). // Owned!string
             write(data);
     }

     RC!string[] stringList;

     void addToGlobalList(scope RC!string s) {
         stringList ~= s;    // increments ref-count
     }

     RC!string foo;
     addToGlobalList(foo);   // borrowing doesn't change ref-count

     auto newFileName = "hello-world".setExtension("txt");
     auto tmp1 = newFileName;       // ERROR: cannot copy
     scope tmp2 = newFileName;      // OK, borrowing
     foo = newFileName;             // ERROR: cannot copy
     foo = newFileName.release();   // OK, move
     auto bar = newFileName.toRC(); // ditto



 Oct 02 2014



prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 29 September 2014 20:49, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 [...]

 Destroy!

 Andrei


I generally like the idea, but my immediate concern is that it implies
that every function that may deal with allocation is a template.
This interferes with C/C++ compatibility in a pretty big way. Or more
generally, the idea of a lib. Does this mean that a lib will be
required to produce code for every permutation of functions according
to memory management strategy? Usually libs don't contain code for
uninstantiated templates.

With this in place, I worry that traditional use of libs, separate
compilation, external language linkage, etc, all become very
problematic.
Pervasive templates can only work well if all code is D code, and if
all code is compiled together.
Most non-OSS industry doesn't ship source, they ship libs. And if libs
are to become impractical, then dependencies become a problem; instead
of linking libphobos.so, you pretty much have to compile phobos
together with your app (already basically true for phobos, but it's
fairly unique).
What if that were a much larger library? What if you have 10s of
dependencies all distributed in this manner? Does it scale?

I guess this doesn't matter if this is only a proposal for phobos...
but I suspect the pattern will become pervasive if it works, and yeah,
I'm not sure where that leads.



 Sep 30 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 6:53 PM, Manu via Digitalmars-d wrote:

 I generally like the idea, but my immediate concern is that it implies
 that every function that may deal with allocation is a template.
 This interferes with C/C++ compatibility in a pretty big way. Or more
 generally, the idea of a lib. Does this mean that a lib will be
 required to produce code for every permutation of functions according
 to memory management strategy? Usually libs don't contain code for
 uninstantiated templates.


If a lib chooses one specific memory management policy, it can of course 
be non-templated with regard to that. If it wants to offer its users the 
choice, it would probably have to offer some templates.


 With this in place, I worry that traditional use of libs, separate
 compilation, external language linkage, etc, all become very
 problematic.
 Pervasive templates can only work well if all code is D code, and if
 all code is compiled together.
 Most non-OSS industry doesn't ship source, they ship libs. And if libs
 are to become impractical, then dependencies become a problem; instead
 of linking libphobos.so, you pretty much have to compile phobos
 together with your app (already basically true for phobos, but it's
 fairly unique).
 What if that were a much larger library? What if you have 10s of
 dependencies all distributed in this manner? Does it scale?

 I guess this doesn't matter if this is only a proposal for phobos...
 but I suspect the pattern will become pervasive if it works, and yeah,
 I'm not sure where that leads.


Thanks for the point. I submit that Phobos has and will be different 
from other D libraries; as the standard library, it has the role of 
supporting widely varying needs, and as such it makes a lot of sense to 
make it highly generic and configurable. Libraries that are for specific 
domains can avail themselves of a narrower design scope.


Andrei



 Oct 01 2014



prev sibling next sibling parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com>  writes:


On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu 
wrote:

 Back when I've first introduced RCString I hinted that we have 
 a larger strategy in mind. Here it is.


Slightly related :)

https://github.com/D-Programming-Language/phobos/pull/2573



 Sep 30 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/14, 10:46 PM, "Nordlöw" wrote:

 On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:

 Back when I've first introduced RCString I hinted that we have a
 larger strategy in mind. Here it is.


 Slightly related :)

 https://github.com/D-Programming-Language/phobos/pull/2573


Nice, thanks! -- Andrei



 Oct 01 2014



prev sibling  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


29-Sep-2014 14:49, Andrei Alexandrescu пишет:

 Back when I've first introduced RCString I hinted that we have a larger
 strategy in mind. Here it is.


[snip]

I think it would be well worth it to actually do a bit of research.
Before we get into the fry and spill blood (or LOCs) everywhere.

Can we:

1. Present a list of allocating functions.

2. What they (currently) allocate: string, T[], V[K] or something else.

3. See what alternatives they have (that do not allocate if any).

4. Plot course for these that do not have. (Just listing how function 
signature would change is good enough).

Thanks!

P.S. If there are no takers I'd get do myself it in a week or so.

-- 
Dmitry Olshansky



 Oct 03 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/3/14, 11:27 AM, Dmitry Olshansky wrote:

 29-Sep-2014 14:49, Andrei Alexandrescu пишет:

 Back when I've first introduced RCString I hinted that we have a larger
 strategy in mind. Here it is.


 [snip]

 I think it would be well worth it to actually do a bit of research.
 Before we get into the fry and spill blood (or LOCs) everywhere.

 Can we:

 1. Present a list of allocating functions.


Awesome. I just started 
http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage and I 
encourage us all to add to it (sorted by module and then by artifact name).


 2. What they (currently) allocate: string, T[], V[K] or something else.


Mention that in the "Possible Fix(es)" column.


 3. See what alternatives they have (that do not allocate if any).


Yah.


 4. Plot course for these that do not have. (Just listing how function
 signature would change is good enough).


Yah.


 Thanks!

 P.S. If there are no takers I'd get do myself it in a week or so.


Let's all get this rolling!


Andrei



 Oct 03 2014



  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


03-Oct-2014 23:50, Andrei Alexandrescu пишет:

 On 10/3/14, 11:27 AM, Dmitry Olshansky wrote:

 29-Sep-2014 14:49, Andrei Alexandrescu пишет:

 Back when I've first introduced RCString I hinted that we have a larger
 strategy in mind. Here it is.


 [snip]

 I think it would be well worth it to actually do a bit of research.
 Before we get into the fry and spill blood (or LOCs) everywhere.

 Can we:

 1. Present a list of allocating functions.


 Awesome. I just started
 http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage and I
 encourage us all to add to it (sorted by module and then by artifact name).


Glad you liked it.

Being in favor of automation as a start I just toggled -vgc flag in 
Win64 makefile and built phobos. Raw data (CSV) is here:

https://gist.github.com/anonymous/763adcd62ab60a66e9d8

Time to mine it...

-- 
Dmitry Olshansky



 Oct 03 2014



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/3/14, 1:18 PM, Dmitry Olshansky wrote:

 03-Oct-2014 23:50, Andrei Alexandrescu пишет:

 On 10/3/14, 11:27 AM, Dmitry Olshansky wrote:

 29-Sep-2014 14:49, Andrei Alexandrescu пишет:

 Back when I've first introduced RCString I hinted that we have a larger
 strategy in mind. Here it is.


 [snip]

 I think it would be well worth it to actually do a bit of research.
 Before we get into the fry and spill blood (or LOCs) everywhere.

 Can we:

 1. Present a list of allocating functions.


 Awesome. I just started
 http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage and I
 encourage us all to add to it (sorted by module and then by artifact
 name).


 Glad you liked it.

 Being in favor of automation as a start I just toggled -vgc flag in
 Win64 makefile and built phobos. Raw data (CSV) is here:

 https://gist.github.com/anonymous/763adcd62ab60a66e9d8

 Time to mine it...


D script that generates wikitable from that -> awesomeness. -- Andrei



 Oct 03 2014



  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


04-Oct-2014 00:21, Andrei Alexandrescu пишет:

 On 10/3/14, 1:18 PM, Dmitry Olshansky wrote:

 03-Oct-2014 23:50, Andrei Alexandrescu пишет:

 On 10/3/14, 11:27 AM, Dmitry Olshansky wrote:

 29-Sep-2014 14:49, Andrei Alexandrescu пишет:

 Back when I've first introduced RCString I hinted that we have a
 larger
 strategy in mind. Here it is.


 [snip]

 I think it would be well worth it to actually do a bit of research.
 Before we get into the fry and spill blood (or LOCs) everywhere.

 Can we:

 1. Present a list of allocating functions.


 Awesome. I just started
 http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage and I
 encourage us all to add to it (sorted by module and then by artifact
 name).


 Glad you liked it.

 Being in favor of automation as a start I just toggled -vgc flag in
 Win64 makefile and built phobos. Raw data (CSV) is here:

 https://gist.github.com/anonymous/763adcd62ab60a66e9d8

 Time to mine it...


 D script that generates wikitable from that -> awesomeness. -- Andrei


I'm on it. With GitHub source links. D's regex rocks ;)


-- 
Dmitry Olshansky



 Oct 03 2014



  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


04-Oct-2014 00:21, Dmitry Olshansky пишет:

 04-Oct-2014 00:21, Andrei Alexandrescu пишет:

 On 10/3/14, 1:18 PM, Dmitry Olshansky wrote:

 03-Oct-2014 23:50, Andrei Alexandrescu пишет:






[snip]

 Glad you liked it.

 Being in favor of automation as a start I just toggled -vgc flag in
 Win64 makefile and built phobos. Raw data (CSV) is here:

 https://gist.github.com/anonymous/763adcd62ab60a66e9d8

 Time to mine it...


 D script that generates wikitable from that -> awesomeness. -- Andrei


 I'm on it. With GitHub source links. D's regex rocks ;)


Forgot my wiki credentials. Anyhow I got passable Markdown page fairly 
quickly. Looks like this:

https://github.com/DmitryOlshansky/phobos/wiki/Phobos-GC-happy-list!

Tool to get it, anybody feel free to take over from here:

https://gist.github.com/anonymous/dc0000d3b801a7bedff0

Takes DMD's output from stdin, so:

make -f posix.mak | ./this_script

(Needs -vgc flag obviously)

-- 
Dmitry Olshansky



 Oct 03 2014



  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


04-Oct-2014 00:42, Dmitry Olshansky пишет:

 04-Oct-2014 00:21, Dmitry Olshansky пишет:

 04-Oct-2014 00:21, Andrei Alexandrescu пишет:

 On 10/3/14, 1:18 PM, Dmitry Olshansky wrote:

 03-Oct-2014 23:50, Andrei Alexandrescu пишет:






 [snip]

 Glad you liked it.

 Being in favor of automation as a start I just toggled -vgc flag in
 Win64 makefile and built phobos. Raw data (CSV) is here:

 https://gist.github.com/anonymous/763adcd62ab60a66e9d8

 Time to mine it...


 D script that generates wikitable from that -> awesomeness. -- Andrei


 I'm on it. With GitHub source links. D's regex rocks ;)


 Forgot my wiki credentials. Anyhow I got passable Markdown page fairly
 quickly. Looks like this:

 https://github.com/DmitryOlshansky/phobos/wiki/Phobos-GC-happy-list!


Ehm, rather (without '!' at the end):
https://github.com/DmitryOlshansky/phobos/wiki/Phobos-GC-happy-list

-- 
Dmitry Olshansky



 Oct 03 2014



  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


04-Oct-2014 00:56, Dmitry Olshansky пишет:

 04-Oct-2014 00:42, Dmitry Olshansky пишет:

 04-Oct-2014 00:21, Dmitry Olshansky пишет:

 04-Oct-2014 00:21, Andrei Alexandrescu пишет:

 On 10/3/14, 1:18 PM, Dmitry Olshansky wrote:

 03-Oct-2014 23:50, Andrei Alexandrescu пишет:






 [snip]

 Glad you liked it.

 Being in favor of automation as a start I just toggled -vgc flag in
 Win64 makefile and built phobos. Raw data (CSV) is here:

 https://gist.github.com/anonymous/763adcd62ab60a66e9d8

 Time to mine it...


 D script that generates wikitable from that -> awesomeness. -- Andrei








Got it:
https://gist.github.com/DmitryOlshansky/d718be4ec12158cf2f02

Tries hard to detect class & function name (it's all on heuristics + 
regex... e-hm) and generates mediawiki table.

DWiki won't let me edit it, but the output is here:
https://gist.github.com/DmitryOlshansky/341aa7f6d6f0d53ffc59

Anybody with a proper D parser may do a way better job ;)

-- 
Dmitry Olshansky



 Oct 03 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/3/14, 3:59 PM, Dmitry Olshansky wrote:

 Got it:
 https://gist.github.com/DmitryOlshansky/d718be4ec12158cf2f02

 Tries hard to detect class & function name (it's all on heuristics +
 regex... e-hm) and generates mediawiki table.

 DWiki won't let me edit it, but the output is here:
 https://gist.github.com/DmitryOlshansky/341aa7f6d6f0d53ffc59

 Anybody with a proper D parser may do a way better job ;)


Tried to insert it, looks weird. Probably it would be most effective if 
you fixed your wiki account. Thanks! -- Andrei



 Oct 03 2014