www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Function scope arguments

reply "bearophile" <bearophileHUGS lycos.com> writes:
Maybe this thread shows the need for some design work, see the 
comments by Hara:

http://d.puremagic.com/issues/show_bug.cgi?id=8695

What's the meaning of "in" function arguments? What are the 
purposes of "scope" for function arguments that contain 
indirections, and how to implement that?

Bye,
bearophile
Jan 14 2013
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/14/2013 06:52 PM, bearophile wrote:

 Maybe this thread shows the need for some design work, see the comments
 by Hara:

 http://d.puremagic.com/issues/show_bug.cgi?id=8695

 What's the meaning of "in" function arguments? What are the purposes of
 "scope" for function arguments that contain indirections, and how to
 implement that?

 Bye,
 bearophile


'in' is 'const scope'.

'scope' is supposed to restrict escaping references, but it is not 
entirely clear what that means, and how to make it powerful enough. Eg. 
what if part of a structure can be freely escaped, such as the contents 
of an array of Objects, but not the array itself? I'd argue that 
indirections in parameters should not be covered by 'scope', because it 
is not something that is transitive. (the opposite would be)

Furthermore, there is the issue of how to treat 'ref' parameters.

Also, we may want to use 'scope' to annotate struct fields in some way, 
so that structs can capture scope parameters that are ensured at the 
call site to live at least as long as the struct instance.

An implementiation should use flow (and maybe lifetime) analysis.
Jan 14 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-01-15 00:07, Timon Gehr wrote:


 'in' is 'const scope'.

 'scope' is supposed to restrict escaping references, but it is not
 entirely clear what that means, and how to make it powerful enough. Eg.
 what if part of a structure can be freely escaped, such as the contents
 of an array of Objects, but not the array itself? I'd argue that
 indirections in parameters should not be covered by 'scope', because it
 is not something that is transitive. (the opposite would be)

 Furthermore, there is the issue of how to treat 'ref' parameters.

 Also, we may want to use 'scope' to annotate struct fields in some way,
 so that structs can capture scope parameters that are ensured at the
 call site to live at least as long as the struct instance.

 An implementiation should use flow (and maybe lifetime) analysis.


For delegates "scope" indicates the delegate is not a real closure and 
won't capture any variables on the heap.

-- 
/Jacob Carlborg
Jan 14 2013
prev sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 01/15/13 00:07, Timon Gehr wrote:

 'scope' is supposed to restrict escaping references, but it is not entirely
clear what that means, and how to make it powerful enough. Eg. what if part of
a structure can be freely escaped, such as the contents of an array of Objects,
but not the array itself? I'd argue that indirections in parameters should not
be covered by 'scope', because it is not something that is transitive. (the
opposite would be)


A non-transitive scope would not be very useful. Eg. a 'scope' arg could
never refer to a stack object - which is the main advantage of having 'scope'.
Introducing a second kind of top-level-scope would probably complicate the
language too much.


 Furthermore, there is the issue of how to treat 'ref' parameters.


Different problem - lifetime. One approach would be to disallow escaping
them (which in this case includes returning them) unless the compiler is
able to do the right - ie the body of the function is available. Somewhat
unorthodox, but could work. (The problem are not the trivial cases; it's the
ones where the compiler has no idea which ref is escaped/returned at runtime)

artur
Jan 15 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 15 January 2013 at 10:58:17 UTC, Artur Skawina wrote:

 Different problem - lifetime. One approach would be to disallow 
 escaping
 them (which in this case includes returning them) unless the 
 compiler is
 able to do the right - ie the body of the function is 
 available. Somewhat
 unorthodox, but could work. (The problem are not the trivial 
 cases; it's the
 ones where the compiler has no idea which ref is 
 escaped/returned at runtime)


The compiler should assume they may escape unless scope is 
specified.
Jan 15 2013
parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 01/15/13 12:48, deadalnix wrote:

 On Tuesday, 15 January 2013 at 10:58:17 UTC, Artur Skawina wrote:

 Different problem - lifetime. One approach would be to disallow escaping
 them (which in this case includes returning them) unless the compiler is
 able to do the right - ie the body of the function is available. Somewhat
 unorthodox, but could work. (The problem are not the trivial cases; it's the
 ones where the compiler has no idea which ref is escaped/returned at runtime)


 
 The compiler should assume they may escape unless scope is specified.


This is about /avoiding/ "hidden" heap allocations as much as possible. Having
functions with 'ref' and 'auto-ref' args always trigger them is not ideal.

'lazy' args are already problematic enough. (there's currently no way to mark
them as non-escaping, the compiler has to assume that the do -> the result is
that they /always/ cause heap allocations and you have to use explicit scoped
delegates instead, losing the syntax advantages)

artur
Jan 15 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Artur Skawina:


 'lazy' args are already problematic enough. (there's currently 
 no way to mark them as non-escaping,


A future "scope lazy" annotation?

Bye,
bearophile
Jan 15 2013
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/15/2013 01:44 PM, Artur Skawina wrote:

 On 01/15/13 12:48, deadalnix wrote:

 On Tuesday, 15 January 2013 at 10:58:17 UTC, Artur Skawina wrote:

 Different problem - lifetime. One approach would be to disallow escaping
 them (which in this case includes returning them) unless the compiler is
 able to do the right - ie the body of the function is available. Somewhat
 unorthodox, but could work. (The problem are not the trivial cases; it's the
 ones where the compiler has no idea which ref is escaped/returned at runtime)


 The compiler should assume they may escape unless scope is specified.


 This is about /avoiding/ "hidden" heap allocations as much as possible. Having
 functions with 'ref' and 'auto-ref' args always trigger them is not ideal.

 'lazy' args are already problematic enough. (there's currently no way to mark
 them as non-escaping, the compiler has to assume that the do -> the result is
 that they /always/ cause heap allocations and you have to use explicit scoped
 delegates instead, losing the syntax advantages)

 artur


Actually lazy args are implicitly 'scope' and never allocate.
Jan 15 2013
parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 01/15/13 15:09, Timon Gehr wrote:

 On 01/15/2013 01:44 PM, Artur Skawina wrote:

 On 01/15/13 12:48, deadalnix wrote:

 On Tuesday, 15 January 2013 at 10:58:17 UTC, Artur Skawina wrote:

 Different problem - lifetime. One approach would be to disallow escaping
 them (which in this case includes returning them) unless the compiler is
 able to do the right - ie the body of the function is available. Somewhat
 unorthodox, but could work. (The problem are not the trivial cases; it's the
 ones where the compiler has no idea which ref is escaped/returned at runtime)


 The compiler should assume they may escape unless scope is specified.


 This is about /avoiding/ "hidden" heap allocations as much as possible. Having
 functions with 'ref' and 'auto-ref' args always trigger them is not ideal.

 'lazy' args are already problematic enough. (there's currently no way to mark
 them as non-escaping, the compiler has to assume that the do -> the result is
 that they /always/ cause heap allocations and you have to use explicit scoped
 delegates instead, losing the syntax advantages)


 
 Actually lazy args are implicitly 'scope' and never allocate.


I wish. :)

Seriously though, I don't.
They can be escaped and they do allocate. They have to. The problem is just
that there currently is no way to tell the compiler i-know-what-i'm-doing
and avoid the heap allocated closures.

[if the behavior changed in newer (than my old gdc) compiler versions then such
a
change is bogus, as it would mean that stack objects could be escaped]

artur
Jan 15 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/15/2013 04:03 PM, Artur Skawina wrote:

 On 01/15/13 15:09, Timon Gehr wrote:

 On 01/15/2013 01:44 PM, Artur Skawina wrote:

 On 01/15/13 12:48, deadalnix wrote:

 On Tuesday, 15 January 2013 at 10:58:17 UTC, Artur Skawina wrote:

 Different problem - lifetime. One approach would be to disallow escaping
 them (which in this case includes returning them) unless the compiler is
 able to do the right - ie the body of the function is available. Somewhat
 unorthodox, but could work. (The problem are not the trivial cases; it's the
 ones where the compiler has no idea which ref is escaped/returned at runtime)


 The compiler should assume they may escape unless scope is specified.


 This is about /avoiding/ "hidden" heap allocations as much as possible. Having
 functions with 'ref' and 'auto-ref' args always trigger them is not ideal.

 'lazy' args are already problematic enough. (there's currently no way to mark
 them as non-escaping, the compiler has to assume that the do -> the result is
 that they /always/ cause heap allocations and you have to use explicit scoped
 delegates instead, losing the syntax advantages)


 Actually lazy args are implicitly 'scope' and never allocate.


 I wish. :)

 Seriously though, I don't.


Me neither, but that is what happens.


 They can be escaped and they do allocate. They have to. The problem is just
 that there currently is no way to tell the compiler i-know-what-i'm-doing
 and avoid the heap allocated closures.


No, there is no way to get a heap allocated closure from a lazy parameter.


 [if the behavior changed in newer (than my old gdc) compiler versions then
such a
 change is bogus, as it would mean that stack objects could be escaped]

 artur


I think the behaviour has always been the same (at least with DMD).

import std.stdio;

int delegate() foo(lazy int x){
     return ()=>x;
}

int delegate() escape(int x){
     return foo(x);
}

void trash(){
     int[2] x=1337;
}

void main(){
     auto dg = escape(2);
     trash();
     writeln(dg());
}


$ dmd -run tt.d
1337

$ gdmd -run tt.d
-1430461920

If the behaviour was as you suggest, the output would be:
2
Jan 15 2013
next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 01/15/13 17:12, Timon Gehr wrote:

 On 01/15/2013 04:03 PM, Artur Skawina wrote:

 On 01/15/13 15:09, Timon Gehr wrote:

 On 01/15/2013 01:44 PM, Artur Skawina wrote:

 On 01/15/13 12:48, deadalnix wrote:

 On Tuesday, 15 January 2013 at 10:58:17 UTC, Artur Skawina wrote:

 Different problem - lifetime. One approach would be to disallow escaping
 them (which in this case includes returning them) unless the compiler is
 able to do the right - ie the body of the function is available. Somewhat
 unorthodox, but could work. (The problem are not the trivial cases; it's the
 ones where the compiler has no idea which ref is escaped/returned at runtime)


 The compiler should assume they may escape unless scope is specified.


 This is about /avoiding/ "hidden" heap allocations as much as possible. Having
 functions with 'ref' and 'auto-ref' args always trigger them is not ideal.

 'lazy' args are already problematic enough. (there's currently no way to mark
 them as non-escaping, the compiler has to assume that the do -> the result is
 that they /always/ cause heap allocations and you have to use explicit scoped
 delegates instead, losing the syntax advantages)


 Actually lazy args are implicitly 'scope' and never allocate.


 I wish. :)

 Seriously though, I don't.


 
 Me neither, but that is what happens.
 

 They can be escaped and they do allocate. They have to. The problem is just
 that there currently is no way to tell the compiler i-know-what-i'm-doing
 and avoid the heap allocated closures.


 
 No, there is no way to get a heap allocated closure from a lazy parameter.
 

 [if the behavior changed in newer (than my old gdc) compiler versions then
such a
 change is bogus, as it would mean that stack objects could be escaped]

 artur


 
 I think the behaviour has always been the same (at least with DMD).
 
 import std.stdio;
 
 int delegate() foo(lazy int x){
     return ()=>x;
 }
 
 int delegate() escape(int x){
     return foo(x);
 }
 
 void trash(){
     int[2] x=1337;
 }
 
 void main(){
     auto dg = escape(2);
     trash();
     writeln(dg());
 }
 
 
 $ dmd -run tt.d
 1337
 
 $ gdmd -run tt.d
 -1430461920
 
 If the behaviour was as you suggest, the output would be:
 2


The output is actually "2" here.

But after taking a closer look at the generated code I've now realized
that the allocations I am seeing are originating from (the equivalent of)
'foo', not 'main'. Inlining confused me.

Note to myself: never assume the D compiler behaves sanely.

You are right - lazy args currently do *not* cause allocations.
Thanks for the correction.

Unfortunately this makes the situations much worse, as the problem isn't
just a performance issue, but a potential source of nasty bugs.

The fix would be straightforward - make 'lazy' create closures properly
and avoid the unnecessary 'foo' allocations. Then the only thing needed
would be a way to avoid those allocations, as in my real code I was
actually semi-escaping them, at least as far as the compiler could tell.
Oh well, I'll get used to the extra pair of braces. :)

artur
Jan 15 2013
prev sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Tuesday, 15 January 2013 at 16:12:41 UTC, Timon Gehr wrote:

 No, there is no way to get a heap allocated closure from a lazy 
 parameter.

[…]

 I think the behaviour has always been the same (at least with 
 DMD).


This is a hole in SafeD – please file it as such if it isn't in 
Bugzilla already.

David
Jan 15 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 14 January 2013 at 17:52:32 UTC, bearophile wrote:

 Maybe this thread shows the need for some design work, see the 
 comments by Hara:

 http://d.puremagic.com/issues/show_bug.cgi?id=8695

 What's the meaning of "in" function arguments? What are the 
 purposes of "scope" for function arguments that contain 
 indirections, and how to implement that?

 Bye,
 bearophile


Well I think about it a lot, it is unclear what is the solution, 
but here is how I see it :

stack variable as marked as local.

value marked local are considered scope in any inner scope.
values marked local can be passed to function by value or via 
scope argument.
values marked scope can be passed to function by value or via 
scope argument.

void foo() {
     int a; // a is local.

     {
         // a is scope here
     }
}

Everything accessed throw a scope value is scope.
scope are not referable throw any non scope.
local are not referable to scope.

each scope declaration introduce a new scope. different scope can 
never refers each others.

void foo(scope A a, scope B b) {
     // a and b are in different scopes.
     a.memeber; // Is in the same scope as a, but not as b.

     C c; // c is local.
     D d; // ditto.

     {
         // c and d are in the same scope. a and b are in 
different scope.
         c.member = d; // OK
         c.member = a; // Error.
     }
}

I still have some trouble to figure out the proper definition of 
it. And what can't be defined clearly is probably not understood 
enough to create a feature.
Jan 15 2013
parent "bearophile" <bearophileHUGS lycos.com> writes:
deadalnix:


 I still have some trouble to figure out the proper definition 
 of it. And what can't be defined clearly is probably not 
 understood enough to create a feature.


While "scope" is probably not the most important feature to work 
now in D (const/immutable issues, and shared issues are possibly 
more important), it's one of the important unfinished parts of D.

It needs a design, and later a partial implementation. Being a 
core language issue I think "scope" is more important than 
optimization issues, like this one:

https://github.com/D-Programming-Language/dmd/commit/4b139c0a217ccbf3c71a0d993eb6e3556254de60

Bye,
bearophile
Jan 15 2013