www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - closures

reply Jens <jens-theisen-tmp01 gmx.de> writes:
Hello there,

I'm new to D and experimenting with closures. I know the basics of how
compilers translate things, especially in C++. So I appreciate the difficulty
of implementing closures correctly and been wondering if it really works. I
found a place where dmd 2 appears to fail, but I don't know whether that's a
bug or just not supported:

import std.stdio;

struct Foo
{
  int a = 7;
  int bar() { return a; }

  int delegate() makedelegate() {
    int abc() { return a; }
    return &abc;
  }
}

void call(int delegate() dg)
{
  writefln("foo: %d", dg());
}

int delegate() makedelegate1()
{
  int x = 27;
  int abc() { return x; }
  return &abc;
}

int delegate() makedelegate2()
{
  Foo f;
  int abc() { return f.a; }
  return &abc;
}

int delegate() makedelegate3()
{
  Foo f;
  return &f.bar;
}

int delegate() makedelegate4b(ref Foo f)
{
  int abc() { return f.a; }
  return &abc;  
}

int delegate() makedelegate4()
{
  Foo f;
  return makedelegate4b(f);
}


void main(string[] args)
{
  // On dmd v2.029, linux build, this...

  call(makedelegate1()); // ...works: 27
  call(makedelegate2()); // ...works: 7
  call(makedelegate3()); // ...doesn't work: 134518855
  call(makedelegate4()); // ...doesn't work: 134518947

  Foo f;
  call(&f.bar); // ...works: 7
}

In case 4 the reference is explicit, so it's somehow easier to see that
something dangerous is being done, but in case 3, D seems to make it too easy
to shoot yourself in the foot.

Is there a resource discussing these issues?
Jul 16 2009
next sibling parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Thu, Jul 16, 2009 at 8:02 AM, Jens<jens-theisen-tmp01 gmx.de> wrote:

 Hello there,

 I'm new to D and experimenting with closures. I know the basics of how co=


mpilers translate things, especially in C++. So I appreciate the difficulty=
 of implementing closures correctly and been wondering if it really works. =
I found a place where dmd 2 appears to fail, but I don't know whether that'=
s a bug or just not supported:

 import std.stdio;

 struct Foo
 {
 =A0int a =3D 7;
 =A0int bar() { return a; }

 =A0int delegate() makedelegate() {
 =A0 =A0int abc() { return a; }
 =A0 =A0return &abc;
 =A0}
 }

 void call(int delegate() dg)
 {
 =A0writefln("foo: %d", dg());
 }

 int delegate() makedelegate1()
 {
 =A0int x =3D 27;
 =A0int abc() { return x; }
 =A0return &abc;
 }

 int delegate() makedelegate2()
 {
 =A0Foo f;
 =A0int abc() { return f.a; }
 =A0return &abc;
 }

 int delegate() makedelegate3()
 {
 =A0Foo f;
 =A0return &f.bar;
 }

 int delegate() makedelegate4b(ref Foo f)
 {
 =A0int abc() { return f.a; }
 =A0return &abc;
 }

 int delegate() makedelegate4()
 {
 =A0Foo f;
 =A0return makedelegate4b(f);
 }


 void main(string[] args)
 {
 =A0// On dmd v2.029, linux build, this...

 =A0call(makedelegate1()); // ...works: 27
 =A0call(makedelegate2()); // ...works: 7
 =A0call(makedelegate3()); // ...doesn't work: 134518855
 =A0call(makedelegate4()); // ...doesn't work: 134518947

 =A0Foo f;
 =A0call(&f.bar); // ...works: 7
 }

 In case 4 the reference is explicit, so it's somehow easier to see that s=


omething dangerous is being done, but in case 3, D seems to make it too eas=
y to shoot yourself in the foot.

 Is there a resource discussing these issues?


Yeah, you've found it :)

I think what's going on here is that the compiler will *only* allocate
closures for nested functions.  However allocating a closure for a
delegate of a value object would be a nice addition.
Jul 16 2009
parent reply BCS <ao pathlink.com> writes:
Reply to Jarrett,


 I think what's going on here is that the compiler will *only* allocate
 closures for nested functions. However allocating a closure for a
 delegate of a value object would be a nice addition.
 


This will quickly devolve into the general escape analysts problem and here 
there be dragons. I think the correct solution is to say it's unsupported 
by calling this an escaping reference bug in the user code.
Jul 16 2009
parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Thu, Jul 16, 2009 at 1:24 PM, BCS<ao pathlink.com> wrote:

 Reply to Jarrett,


 I think what's going on here is that the compiler will *only* allocate
 closures for nested functions. However allocating a closure for a
 delegate of a value object would be a nice addition.


 This will quickly devolve into the general escape analysts problem and here
 there be dragons. I think the correct solution is to say it's unsupported by
 calling this an escaping reference bug in the user code.


At least for non-ref-param value types, I don't think it's
unreasonable to say that &obj.func should allocate a closure.  I mean,
it's the same as saying

int delegate() dg;
dg.funcptr = &typeof(obj).func;
dg.ptr = &obj;

and the last line there would be illegal to return under normal
circumstances anyway.

But you're probably right that it might just be easier to disallow it
and force people to write { return f.func(); } instead.  :)
Jul 16 2009
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 16 Jul 2009 13:34:27 -0400, Jarrett Billingsley  
<jarrett.billingsley gmail.com> wrote:


 On Thu, Jul 16, 2009 at 1:24 PM, BCS<ao pathlink.com> wrote:

 Reply to Jarrett,


 I think what's going on here is that the compiler will *only* allocate
 closures for nested functions. However allocating a closure for a
 delegate of a value object would be a nice addition.


 This will quickly devolve into the general escape analysts problem and  
 here
 there be dragons. I think the correct solution is to say it's  
 unsupported by
 calling this an escaping reference bug in the user code.


 At least for non-ref-param value types, I don't think it's
 unreasonable to say that &obj.func should allocate a closure.  I mean,
 it's the same as saying

 int delegate() dg;
 dg.funcptr = &typeof(obj).func;
 dg.ptr = &obj;

 and the last line there would be illegal to return under normal
 circumstances anyway.

 But you're probably right that it might just be easier to disallow it
 and force people to write { return f.func(); } instead.  :)


What about syntax to force closure behavior (on or off)?  Not that I have  
any to suggest, but if the compiler is going to remain ignorant about  
escape analysis, then we should be able to supply the intelligence, and  
hacking an extra wrapper function to force behavior seems... well, hackish  
:)

-Steve
Jul 16 2009
parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Thu, Jul 16, 2009 at 2:22 PM, Steven
Schveighoffer<schveiguy yahoo.com> wrote:

 What about syntax to force closure behavior (on or off)? =A0Not that I ha=


ve

 any to suggest, but if the compiler is going to remain ignorant about esc=


ape

 analysis, then we should be able to supply the intelligence, and hacking =


an

 extra wrapper function to force behavior seems... well, hackish :)


There already is, at least for turning closures off.  A common design
pattern in D1 is to pass the address of a nested function as a
callback.  This is convenient and performant:

void putChar(char c)
{
    write(c);
}

std.format.doFormat(&putChar, blahblahblah); // or so

If D2 were to allocate a closure for &putChar, suddenly your
formatting function that's called several hundred times a second
starts eating memory.  The solution is to put 'scope' on the
parameter:

void doFormat(scope void delegate(char) dg, blahblah) { .. }

DMD2 will not allocate a closure for the initial code if 'scope' is
present, and if removed, it will.
Jul 16 2009
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 16 Jul 2009 14:56:28 -0400, Jarrett Billingsley  
<jarrett.billingsley gmail.com> wrote:


 On Thu, Jul 16, 2009 at 2:22 PM, Steven
 Schveighoffer<schveiguy yahoo.com> wrote:

 What about syntax to force closure behavior (on or off)?  Not that I  
 have
 any to suggest, but if the compiler is going to remain ignorant about  
 escape
 analysis, then we should be able to supply the intelligence, and  
 hacking an
 extra wrapper function to force behavior seems... well, hackish :)


 There already is, at least for turning closures off.


Yes I'm aware.   But that has some limitations.

For example, if I truly want my delegate to be a variable, I have to  
decide which delegate to use at the time I'm calling the function.  I  
can't store the delegate and then call the function later.

Also, forcing a closure for the reasons discussed in this thread would be  
useful.

I think the philosophy that D has on say const would be beneficial here:

1. When the compiler can prove the code is correct, allow it.  e.g.  
implicit cast of a const value type to a mutable value type.
2. When the compiler can't prove, do the safest thing.  e.g. e.g. const  
could be a pointer to mutable, but compiler isn't sure, so don't allow  
implicit cast.
3. When the user knows more about the situation than the compiler, allow  
override.  e.g. explicit cast away const is allowed, or explicit cast to  
immutable.

The same thing could be applied to closures:

1. Calling a function with a delegate where the arg is scope is allowed.
2. Assigning a delegate to a variable causes a closure.
3. The user can specify that a delegate does not need a closure.

-Steve
Jul 16 2009
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Jens:

  // On dmd v2.029, linux build, this...
   call(makedelegate1()); // ...works: 27
   call(makedelegate2()); // ...works: 7
   call(makedelegate3()); // ...doesn't work: 134518855
   call(makedelegate4()); // ...doesn't work: 134518947
   Foo f;
   call(&f.bar); // ...works: 7


I have run your code with DMD v2.031 on Windows, and it prints:
foo: 27
foo: 7
foo: 7
foo: 7
foo: 7

That sounds too much good, probably the right values are left in the stack. So
I have added some printf() that return an int and show better the problems:

import std.stdio: writefln, printf;

struct Foo {
    int a = 7;
    int bar() { return a; }

    int delegate() makeDelegate() {
        printf("filler");
        int abc() { return a; }
        printf("filler");
        return &abc;
    }
}

void call(int delegate() dg) {
    writefln("foo: %d", dg());
}

int delegate() makeDelegate1() {
    int x = 27;
    int abc() { return x; }
    return &abc;
}

int delegate() makeDelegate2() {
    Foo f;
    int abc() { return f.a; }
    return &abc;
}

int delegate() makeDelegate3() {
    Foo f;
    return &f.bar;
}

int delegate() makeDelegate4b(ref Foo f) {
    int abc() { return f.a; }
    printf("filler");
    return &abc;
}

int delegate() makeDelegate4() {
    Foo f;
    printf("filler");
    return makeDelegate4b(f);
}


void main() {
    auto d1 = makeDelegate1();
    printf("filler");
    call(d1);

    auto d2 = makeDelegate2();
    printf("filler");
    call(d2);

    auto d3 = makeDelegate3();
    printf("filler");
    call(d3);

    auto d4 = makeDelegate4();
    printf("filler");
    call(d4);

    Foo f;
    call(&f.bar);
}

Now the output is:
fillerfoo: 27
fillerfoo: 7
fillerfoo: 4354187
fillerfillerfillerfoo: 7
foo: 7

It shows some difference still compared to your output. So are some things
improved between 2.029 and 2.031?

I don't know much about where and when to add "scope" to avoid creation of
closures. Maybe Andrei's book will help me understand/learn better :-)

Bye,
bearophile
Jul 16 2009
parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Thu, Jul 16, 2009 at 2:33 PM, bearophile<bearophileHUGS lycos.com> wrote=
:

 Jens:

 =A0// On dmd v2.029, linux build, this...
 =A0 call(makedelegate1()); // ...works: 27
 =A0 call(makedelegate2()); // ...works: 7
 =A0 call(makedelegate3()); // ...doesn't work: 134518855
 =A0 call(makedelegate4()); // ...doesn't work: 134518947
 =A0 Foo f;
 =A0 call(&f.bar); // ...works: 7


 I have run your code with DMD v2.031 on Windows, and it prints:
 foo: 27
 foo: 7
 foo: 7
 foo: 7
 foo: 7


Strange.  I use 2.031 on Windows too, and I get strange values for the
third and fourth items, as expected.
Jul 16 2009