www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Full closures

reply Frank Benoit <keinfarbton googlemail.com> writes:
Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
i think the current full closure implementation should take a redesign.

My suggestion:
Let D have 2 kind of delegate closures (I don't know how to call this 
correctly)
local delegates and heap delegates

local delegate are that from D1 without modifiction.
They can access every local variable in the surrounding scope, no heap 
allocation if the addess is taken. It is expected the adress is not used 
outside the allowed scope.

heap delegates
- can only access variables from the surrounding scope, if they are 
marked as 'const', 'invariant' or 'final'. Which means, they are not 
expected to changed after initialization.
- with instantiation of the delegate, a heap allocated frame is used to 
store a copy of the reference 'final' varialbles for the delegate.
That means in case of a loop, the delegate get a new heap allocation 
with each iteration.

foreach( element; container ){
   const c = element;
   logLater( new { writefln(c); });
}
Aug 15 2008
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Frank,


 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043 i
 think the current full closure implementation should take a redesign.
 
 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this
 correctly)
 local delegates and heap delegates
 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no heap
 allocation if the addess is taken. It is expected the adress is not
 used
 outside the allowed scope.
 heap delegates
 - can only access variables from the surrounding scope, if they are
 marked as 'const', 'invariant' or 'final'. Which means, they are not
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used
 to
 store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation
 with each iteration.
 foreach( element; container ){
 const c = element;
 logLater( new { writefln(c); });
 }


Sounds reasonable. I like that it makes the semantics of referenced variables 
explicit.

However I would amend it by allowing access to non const (etc.) function 
arguments (which would be copied on function entry) and variables who's scope 
is exactly that of the function.

void fn(int thisArg)
{
  int andThis;
  {
    int butNotThis;
  }
}

These would greatly increase the practicality of the design without adding 
ambiguities. 

On second thought, the function args might be bit of a trick. I could live 
without them.
Aug 15 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
BCS wrote:

 Reply to Frank,
 

 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043 i
 think the current full closure implementation should take a redesign.

 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this
 correctly)
 local delegates and heap delegates
 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no heap
 allocation if the addess is taken. It is expected the adress is not
 used
 outside the allowed scope.
 heap delegates
 - can only access variables from the surrounding scope, if they are
 marked as 'const', 'invariant' or 'final'. Which means, they are not
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used
 to
 store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation
 with each iteration.
 foreach( element; container ){
 const c = element;
 logLater( new { writefln(c); });
 }


 
 Sounds reasonable. I like that it makes the semantics of referenced 
 variables explicit.
 
 However I would amend it by allowing access to non const (etc.) function 
 arguments (which would be copied on function entry) and variables who's 
 scope is exactly that of the function.
 
 void fn(int thisArg)
 {
  int andThis;
  {
    int butNotThis;
  }
 }
 
 These would greatly increase the practicality of the design without 
 adding ambiguities.
 On second thought, the function args might be bit of a trick. I could 
 live without them.


Ewww, -- changing the scope of the variable should never change its 
behavior, IMO.
Aug 15 2008
parent reply BCS <ao pathlink.com> writes:
Reply to Robert,


 BCS wrote:
 

 [...] allowing access to non const (etc.)
 function arguments (which would be copied on function entry) and
 variables who's scope is exactly that of the function.
 


 Ewww, -- changing the scope of the variable should never change its
 behavior, IMO.
 


That is a good point, but how else you you get a real closure that has mutable 
state?

int delegate() Seq()
{
   int at = 0;
   int Next() { return at++; }
   return &Next;
}
Aug 15 2008
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
BCS wrote:

 
 That is a good point, but how else you you get a real closure that has 
 mutable state?
 
 int delegate() Seq()
 {
   int at = 0;
   int Next() { return at++; }
   return &Next;
 }
 
 


Err.... just have a closure like D has now?... one that allows access to 
any visible variable?

-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Aug 18 2008
parent reply BCS <ao pathlink.com> writes:
Reply to Bruno,


 BCS wrote:
 

 That is a good point, but how else you you get a real closure that
 has mutable state?
 
 int delegate() Seq()
 {
 int at = 0;
 int Next() { return at++; }
 return &Next;
 }


 Err.... just have a closure like D has now?... one that allows access
 to any visible variable?
 


Franks local delegates won't work because they aren't valid after the other 
function returns, his heap delegates won't work because the variables they 
can access can't be altered.
Aug 18 2008
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
BCS wrote:

 Reply to Bruno,
 

 BCS wrote:


 That is a good point, but how else you you get a real closure that
 has mutable state?

 int delegate() Seq()
 {
 int at = 0;
 int Next() { return at++; }
 return &Next;
 }


 Err.... just have a closure like D has now?... one that allows access
 to any visible variable?


 
 Franks local delegates won't work because they aren't valid after the 
 other function returns, his heap delegates won't work because the 
 variables they can access can't be altered.
 
 


I feel there is a misunderstanding here. I wasn't talking about Frank's 
closures, but the one D currently has. What exactly did you mean with 
"how else [do] you get a real closure that has mutable state?" ?

-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Aug 25 2008
parent BCS <ao pathlink.com> writes:
Reply to Bruno,


 BCS wrote:
 

 Reply to Bruno,
 

 BCS wrote:
 

 That is a good point, but how else you you get a real closure that
 has mutable state?
 
 int delegate() Seq()
 {
 int at = 0;
 int Next() { return at++; }
 return &Next;
 }


 Err.... just have a closure like D has now?... one that allows
 access to any visible variable?
 


 Franks local delegates won't work because they aren't valid after the
 other function returns, his heap delegates won't work because the
 variables they can access can't be altered.
 


 I feel there is a misunderstanding here. I wasn't talking about
 Frank's closures, but the one D currently has. What exactly did you
 mean with "how else [do] you get a real closure that has mutable
 state?" ?
 


The problem with the current ones is that they aren't valid in some cases 
if the outer function leave the scope the delegate is defined in.


void Foo()
{
   int delegate() Bar;
   for(int m = 3; m; m--)
   {
       const int i = m;
       Ber = {return i;}
   }
   {
       int j = 6;
       int k = Bar();
       //k could be 6
   }
}
Aug 25 2008
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Frank Benoit wrote:

 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
 i think the current full closure implementation should take a redesign.
 
 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this 
 correctly)
 local delegates and heap delegates
 
 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no heap 
 allocation if the addess is taken. It is expected the adress is not used 
 outside the allowed scope.
 


Refresh my memory on why would that be wanted. Is it because the 
compiler incorrectly heap allocates some variables in a situation that 
is statically verifiable that such allocation wasn't necessary?
Or is it more of a programmer's help, to avoid him/her making mistakes 
and writing code that inadvertently causes locals to be heap allocated?



 heap delegates
 - can only access variables from the surrounding scope, if they are 
 marked as 'const', 'invariant' or 'final'. Which means, they are not 
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used to 
 store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation 
 with each iteration.
 
 foreach( element; container ){
   const c = element;
   logLater( new { writefln(c); });
 }
 
 


Why would we want such version of "heap delegates" that are more 
restrictive in power than D's current full closures?



-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Aug 18 2008
parent reply Frank Benoit <keinfarbton googlemail.com> writes:
Bruno Medeiros schrieb:

 Frank Benoit wrote:

 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
 i think the current full closure implementation should take a redesign.

 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this 
 correctly)
 local delegates and heap delegates

 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no heap 
 allocation if the addess is taken. It is expected the adress is not 
 used outside the allowed scope.


 
 Refresh my memory on why would that be wanted. Is it because the 
 compiler incorrectly heap allocates some variables in a situation that 
 is statically verifiable that such allocation wasn't necessary?
 Or is it more of a programmer's help, to avoid him/her making mistakes 
 and writing code that inadvertently causes locals to be heap allocated?
 


the D1 nested functions or anonymous classes can access local variable 
from the surrounding scope. local vars are located on the stack. if the 
reference to the nested function of anonymous class is escaping the 
local scope (by storing it somewhere or passing it or returning it) and 
the surrounding scope if left, the variables are no more valid. Running 
the nested function or anonymous class accessing those variable will 
result in crashes.

So the D1 feature is very good in performance, but it should be taken 
care or the livetime of the accessed variables.

The D2 "full closure" feature allocated the whole stack frame on the 
stack, if the compile detects the "take address from nested function".


 

 heap delegates
 - can only access variables from the surrounding scope, if they are 
 marked as 'const', 'invariant' or 'final'. Which means, they are not 
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used 
 to store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation 
 with each iteration.

 foreach( element; container ){
   const c = element;
   logLater( new { writefln(c); });
 }


 
 Why would we want such version of "heap delegates" that are more 
 restrictive in power than D's current full closures?
 


The current D2 approach of simply allocating the whole stack frame looks 
good on the first sight, but IMHO it is not sufficient.
- it breaks the old semantic
- it always do heap allocation, but the old semantic is an important D 
feature.
- it make invariant/const data change value, if used in a loop.

See the link to the bug report.

This is why i think, for the case of "heap delegates" the access should 
be restricted to constant data, and that data should be copied to the 
delegate, instead of heap-ing the surrounding scope stack frame.
Aug 18 2008
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Frank Benoit wrote:

 Bruno Medeiros schrieb:

 Frank Benoit wrote:

 Like shown in http://d.puremagic.com/issues/show_bug.cgi?id=2043
 i think the current full closure implementation should take a redesign.

 My suggestion:
 Let D have 2 kind of delegate closures (I don't know how to call this 
 correctly)
 local delegates and heap delegates

 local delegate are that from D1 without modifiction.
 They can access every local variable in the surrounding scope, no 
 heap allocation if the addess is taken. It is expected the adress is 
 not used outside the allowed scope.


 Refresh my memory on why would that be wanted. Is it because the 
 compiler incorrectly heap allocates some variables in a situation that 
 is statically verifiable that such allocation wasn't necessary?
 Or is it more of a programmer's help, to avoid him/her making mistakes 
 and writing code that inadvertently causes locals to be heap allocated?


 
 the D1 nested functions or anonymous classes can access local variable 
 from the surrounding scope. local vars are located on the stack. if the 
 reference to the nested function of anonymous class is escaping the 
 local scope (by storing it somewhere or passing it or returning it) and 
 the surrounding scope if left, the variables are no more valid. Running 
 the nested function or anonymous class accessing those variable will 
 result in crashes.
 
 So the D1 feature is very good in performance, but it should be taken 
 care or the livetime of the accessed variables.
 
 The D2 "full closure" feature allocated the whole stack frame on the 
 stack, if the compile detects the "take address from nested function".
 


Agh, I didn't think it would heap allocate that easily. That's the most 
conservative approach, and it "indeed puts a serious burden on the use 
of scoped closures".
So I agree we either need:
* a way to specify scoped closures/delegates.
* a smarter compiler that better detects scoped delegates. I'd say it 
doesn't have to be 100% accurate, but should be better than the simple 
heuristic "take address from nested function".




 heap delegates
 - can only access variables from the surrounding scope, if they are 
 marked as 'const', 'invariant' or 'final'. Which means, they are not 
 expected to changed after initialization.
 - with instantiation of the delegate, a heap allocated frame is used 
 to store a copy of the reference 'final' varialbles for the delegate.
 That means in case of a loop, the delegate get a new heap allocation 
 with each iteration.

 foreach( element; container ){
   const c = element;
   logLater( new { writefln(c); });
 }


 Why would we want such version of "heap delegates" that are more 
 restrictive in power than D's current full closures?


 
 The current D2 approach of simply allocating the whole stack frame looks 
 good on the first sight, but IMHO it is not sufficient.
 - it breaks the old semantic


D2 breaks with a lot of things. And I fully agree that D shouldn't be 
bound to D1.0 compatibility (at least not on such early D stage)


 - it always do heap allocation, but the old semantic is an important D 
 feature.


Your "heap closures" also do heap allocation. (although they may 
deferred to only when it's strictly needed, such as when the closure 
delegate is evaluated/created, instead of when the outer variable is 
declared.


 - it make invariant/const data change value, if used in a loop.
 
 See the link to the bug report.


Yeah, but that's a bug, it should go away.



-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Aug 18 2008
parent BCS <ao pathlink.com> writes:
Reply to Bruno,


 - it make invariant/const data change value, if used in a loop.
 
 See the link to the bug report.


 
 Yeah, but that's a bug, it should go away.
 


This is a proposal on how to make it go away. It's a design bug not a compiler 
bug.
Aug 18 2008