www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Passing a reference to a returned reference

reply Michael <michael toohuman.io> writes:
Hello,

I'm a little confused about what is actually happening when I try 
to pass a reference, returned by a method that produces the 
object (associative array), to a setter method which expects a 
reference. What seems to be happening is that it simply does 
nothing, as if the setter method is never called.

I'll try to provide some code to highlight the signatures.

The setter method being called on the reference returned by the 
method that produces the array:

agent.beliefs = combination( ... ); // should receive a 
reference, and pass this to the setter method.

Combination is a function point to one of two functions, whose 
signatures look like:

static auto ref consensus( ... )

and the agent's setter method looks like the following:

void beliefs(ref double[int] beliefs)

Now obviously if I remove ref in the above signature, everything 
works fine, but I am surprised I wasn't getting any kind of 
warning about the reference because the setter function just 
doesn't run at all. I tried checking if (beliefs is null) but 
perhaps this isn't the correct way to check if an associative 
array's reference is no longer reachable?

Some explanation would be great, thanks!
Jul 06 2018
parent reply Timoses <timosesu gmail.com> writes:
On Friday, 6 July 2018 at 13:13:43 UTC, Michael wrote:

 static auto ref consensus( ... )


`auto ref` infers the return type from the return statement [1]. 
So it's not necessarily returning a ref type.

However, I don't think this matters if the only concern you have 
is that the setter function actually modifies the passed object.


 and the agent's setter method looks like the following:

 void beliefs(ref double[int] beliefs)

 Now obviously if I remove ref in the above signature, 
 everything works fine, but I am surprised I wasn't getting any 
 kind of warning about the reference because the setter function 
 just doesn't run at all. I tried checking if (beliefs is null) 
 but perhaps this isn't the correct way to check if an 
 associative array's reference is no longer reachable?

 Some explanation would be great, thanks!


This works for me:

     auto create()
     {
         string[int] dict;
         dict[2] = "hello";
         return dict;
     }

     void modifyNoRef(string[int] m)
     {
          writeln("Address not ref: ", &m);
          m[0] = "modified";
     }

     void modifyRef(ref string[int] m)
     {
          writeln("Address ref: ", &m);
          m[1] = "modified";
     }

     unittest
     {
         auto r = create();
         writeln("Address r: ", &r);
         assert(r.keys.length == 1);
         modifyNoRef(r);
         assert(r.keys.length == 2);	
         modifyRef(r);
         assert(r.keys.length == 3);	
     }

So either with ref or not as parameter storage class the assoc. 
array is modified. Note the address in the "ref" one is the same 
as "r" in the unittest.

[1]: https://dlang.org/spec/function.html#auto-ref-functions
Jul 06 2018
next sibling parent Michael <michael toohuman.io> writes:
On Friday, 6 July 2018 at 14:11:42 UTC, Timoses wrote:

 On Friday, 6 July 2018 at 13:13:43 UTC, Michael wrote:

 static auto ref consensus( ... )


 `auto ref` infers the return type from the return statement 
 [1]. So it's not necessarily returning a ref type.

 However, I don't think this matters if the only concern you 
 have is that the setter function actually modifies the passed 
 object.


 and the agent's setter method looks like the following:

 void beliefs(ref double[int] beliefs)

 Now obviously if I remove ref in the above signature, 
 everything works fine, but I am surprised I wasn't getting any 
 kind of warning about the reference because the setter 
 function just doesn't run at all. I tried checking if (beliefs 
 is null) but perhaps this isn't the correct way to check if an 
 associative array's reference is no longer reachable?

 Some explanation would be great, thanks!


 This works for me:

     auto create()
     {
         string[int] dict;
         dict[2] = "hello";
         return dict;
     }

     void modifyNoRef(string[int] m)
     {
          writeln("Address not ref: ", &m);
          m[0] = "modified";
     }

     void modifyRef(ref string[int] m)
     {
          writeln("Address ref: ", &m);
          m[1] = "modified";
     }

     unittest
     {
         auto r = create();
         writeln("Address r: ", &r);
         assert(r.keys.length == 1);
         modifyNoRef(r);
         assert(r.keys.length == 2);	
         modifyRef(r);
         assert(r.keys.length == 3);	
     }

 So either with ref or not as parameter storage class the assoc. 
 array is modified. Note the address in the "ref" one is the 
 same as "r" in the unittest.

 [1]: https://dlang.org/spec/function.html#auto-ref-functions


Aah yes I had forgotten that I had set it as auto ref. However, 
even when I set that function to always return a copy of the 
associative array, it didn't change anything -- the issue always 
seems to be with the setter method expecting a reference.

So then, supposing it's returning a copy of the object. Given 
that my setter method was still expecting a reference, is there a 
reason why it doesn't even invoke the setter unless I change the 
parameter from expecting a reference to expecting a copy of the 
object? It's not like the function failed in any way, it just 
wouldn't run the function at all. Not even while I was printing 
something right at the top.
Jul 06 2018
prev sibling parent reply Michael <michael toohuman.io> writes:
On Friday, 6 July 2018 at 14:11:42 UTC, Timoses wrote:

 This works for me:

     auto create()
     {
         string[int] dict;
         dict[2] = "hello";
         return dict;
     }

     void modifyNoRef(string[int] m)
     {
          writeln("Address not ref: ", &m);
          m[0] = "modified";
     }

     void modifyRef(ref string[int] m)
     {
          writeln("Address ref: ", &m);
          m[1] = "modified";
     }

     unittest
     {
         auto r = create();
         writeln("Address r: ", &r);
         assert(r.keys.length == 1);
         modifyNoRef(r);
         assert(r.keys.length == 2);	
         modifyRef(r);
         assert(r.keys.length == 3);	
     }

 So either with ref or not as parameter storage class the assoc. 
 array is modified. Note the address in the "ref" one is the 
 same as "r" in the unittest.

 [1]: https://dlang.org/spec/function.html#auto-ref-functions



I'm wondering, seeing as it works in the example given above, but 
not in my case, if this is a weird edge-case with setter member 
functions?

What I wanted to do was have the "Agent" object set its internal 
variable to point to the newly created associative array but 
instead it just seems to go right past the setter function.
Jul 06 2018
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 07/06/2018 07:36 AM, Michael wrote:

 but not in
 my case, if this is a weird edge-case with setter member functions?


This is all very interesting but I'm dying to see the code. :) Can you 
change Timoses's code to demonstrate your case?

Ali
Jul 06 2018
parent reply Michael <michael toohuman.io> writes:
On Friday, 6 July 2018 at 14:50:39 UTC, Ali Çehreli wrote:

 On 07/06/2018 07:36 AM, Michael wrote:

 but not in
 my case, if this is a weird edge-case with setter member


 functions?

 This is all very interesting but I'm dying to see the code. :) 
 Can you change Timoses's code to demonstrate your case?

 Ali


I'm just trying to do that now.

Here is what I have in terms of code:

class Agent
{
     private
     {
         double[int] mDict;
     }

     // Setter: copy
     void beliefs(ref double[int] dict)
     {
         import std.stdio : writeln;
         writeln("Setter function.");
         this.mDict = dict;
     }

     // Getter
     auto ref beliefs()
     {
         return this.mDict;
     }
}

class Operator
{
     static auto ref create()
     {
         double[int] dict;
         dict[2] = 1.0;
         dict[1] = 0.0;
         return dict;
     }
}

unittest
{
     import std.stdio : writeln;

     Agent a = new Agent();

     a.beliefs = Operator.create();
     assert(a.beliefs.keys.length == 2);

     writeln(a.beliefs);
}


and here is the output WITH "ref" in beliefs's signature before 
double[int] dict:

➜  dempshaf git:(master) ✗ rdmd -I../ -unittest -main bug_test.d
[2:1, 1:0]

and without "ref":

➜  dempshaf git:(master) ✗ rdmd -I../ -unittest -main bug_test.d
Setter function.
[2:1, 1:0]


I'm surprised it's showing the correct value of the associative 
array, but hopefully this shows my point that it's not printing 
for some reason, when the parameter is ref.
Jul 06 2018
next sibling parent reply Michael <michael toohuman.io> writes:
On Friday, 6 July 2018 at 15:14:01 UTC, Michael wrote:

 On Friday, 6 July 2018 at 14:50:39 UTC, Ali Çehreli wrote:

 [...]


 I'm just trying to do that now.

 Here is what I have in terms of code:

 [...]


This is definitely to do with my use of the setter syntax, which 
maybe I am misunderstanding? Because if I change it to a normal 
function call like so:

a.beliefs(Operator.create());

then it complains if I use ref, and doesn't complain if I don't.
Jul 06 2018
parent reply Timoses <timosesu gmail.com> writes:
On Friday, 6 July 2018 at 15:33:18 UTC, Michael wrote:

 This is definitely to do with my use of the setter syntax, 
 which maybe I am misunderstanding? Because if I change it to a 
 normal function call like so:

 a.beliefs(Operator.create());

 then it complains if I use ref, and doesn't complain if I don't.


You can try removing the "auto" from the Operator class method

// Error: returning dict escapes a reference to local variable 
dict
false
         static ref create()
         {
             double[int] dict;
             dict[2] = 1.0;
             dict[1] = 0.0;
             return dict;
         }

That indicates that 'dict' is actually returned by value (at 
least the reference). This sounds a bit confusing.
AFAIK, in above code a reference 'dict' is created pointing 
towards memory that stores that dictionary entries. The reference 
itself is a "pointer"(?) which will "die" when running out of the 
method's scope. This pointer lives on the function's stack. So 
returning this "pointer" (essentially a value type, e.g. int) 
will return it by value.

You could return a reference like so:

     class Operator
     {
         static double[int] hdict;
         static this()
         {
              hdict[2] = 1.0;
              hdict[1] = 0.0;
         }
         static ref create()
         {
             return hdict;
         }
     }

then


 a.beliefs(Operator.create());


should work on the "ref" setter.
Jul 06 2018
parent Michael <michael toohuman.io> writes:
On Friday, 6 July 2018 at 15:57:27 UTC, Timoses wrote:

 On Friday, 6 July 2018 at 15:33:18 UTC, Michael wrote:

 This is definitely to do with my use of the setter syntax, 
 which maybe I am misunderstanding? Because if I change it to a 
 normal function call like so:

 a.beliefs(Operator.create());

 then it complains if I use ref, and doesn't complain if I 
 don't.


 You can try removing the "auto" from the Operator class method

 // Error: returning dict escapes a reference to local variable 
 dict
 false
         static ref create()
         {
             double[int] dict;
             dict[2] = 1.0;
             dict[1] = 0.0;
             return dict;
         }

 That indicates that 'dict' is actually returned by value (at 
 least the reference). This sounds a bit confusing.
 AFAIK, in above code a reference 'dict' is created pointing 
 towards memory that stores that dictionary entries. The 
 reference itself is a "pointer"(?) which will "die" when 
 running out of the method's scope. This pointer lives on the 
 function's stack. So returning this "pointer" (essentially a 
 value type, e.g. int) will return it by value.

 You could return a reference like so:

     class Operator
     {
         static double[int] hdict;
         static this()
         {
              hdict[2] = 1.0;
              hdict[1] = 0.0;
         }
         static ref create()
         {
             return hdict;
         }
     }

 then


 a.beliefs(Operator.create());


 should work on the "ref" setter.


Oh of course, as it's a reference type, the "value" being 
returned by 'auto ref' is the value of the reference to the array 
anyway, so I am already passing the reference. Not sure how I 
managed to get really confused about that. Then, forcing the 
function to return a reference to a local reference to a heap 
alloc. array is obviously a problem. This makes way more sense 
now. Thank you again, Timoses, and I hope you enjoyed the result, 
Ali.

I guess I just need to be careful with the setter/getter 
approach, in case I accidentally invoke the getter to return a 
reference, which I then overwrite with another reference. I don't 
know if that's worthy of adding some kind of compiler warning, or 
if I'm just the stupid one to get caught by that, but I'm glad 
it's resolved.

At least it was, technically, working before by leaking access to 
the private member variable and overwriting it.
Jul 06 2018
prev sibling parent reply Timoses <timosesu gmail.com> writes:
On Friday, 6 July 2018 at 15:14:01 UTC, Michael wrote:

 class Agent
 {
     private
     {
         double[int] mDict;
     }

     // Setter: copy
     void beliefs(ref double[int] dict)
     {
         import std.stdio : writeln;
         writeln("Setter function.");
         this.mDict = dict;
     }

     // Getter
     auto ref beliefs()
     {
         return this.mDict;
     }
 }

 class Operator
 {
     static auto ref create()
     {
         double[int] dict;
         dict[2] = 1.0;
         dict[1] = 0.0;
         return dict;
     }
 }


Throw in a writeln statement in getter and be amazed : D

     class Agent
     {
         private
         {
             double[int] mDict;
         }

         // Setter: copy
         void beliefs(ref double[int] dict)
         {
             import std.stdio : writeln;
             writeln("Setter function.");
             this.mDict = dict;
         }

         // Getter
         auto ref beliefs()
         {
             writeln("getter");
             return this.mDict;
         }
     }

     class Operator
     {
         static auto ref create()
         {
             double[int] dict;
             dict[2] = 1.0;
             dict[1] = 0.0;
             return dict;
         }
     }

     unittest
     {
         import std.stdio : writeln;

         Agent a = new Agent();

         writeln("Statement 1");
         a.beliefs = Operator.create();
         writeln("Statement 2");
         assert(a.beliefs.keys.length == 2);

         writeln("Statement 3");
         writeln(a.beliefs);
     }

If you remove the ref then "Statement 1" will use the setter 
method. Otherwise it calls the "auto ref" function to get a 
reference of 'mDict' and assign it a value. Essentially

         // Getter
         auto ref beliefs()
         {
             return this.mDict;
         }

makes your "private" variable not private any more, since you are 
leaking a reference to it. So any program calling "a.beliefs" can 
get a reference to your private data and modify it.

My guess is that (if ref is removed from the setter) it serves as 
a  property function:
https://dlang.org/spec/function.html#property-functions
So this allows the function to work in a statement like:
a.beliefs = <Whatever>

And apparently it is prioritized over the "ref beliefs()" 
function.


I guess it is nicer to actually use the "setter" function, as it 
allows you to inspect what the private data will be assigned to 
and take appropriate measures, whereas with the "ref beliefs()" 
method the private data is open for any manipulation out of your 
control.
Jul 06 2018
parent reply Michael <michael toohuman.io> writes:
On Friday, 6 July 2018 at 15:37:25 UTC, Timoses wrote:

 On Friday, 6 July 2018 at 15:14:01 UTC, Michael wrote:

 class Agent
 {
     private
     {
         double[int] mDict;
     }

     // Setter: copy
     void beliefs(ref double[int] dict)
     {
         import std.stdio : writeln;
         writeln("Setter function.");
         this.mDict = dict;
     }

     // Getter
     auto ref beliefs()
     {
         return this.mDict;
     }
 }

 class Operator
 {
     static auto ref create()
     {
         double[int] dict;
         dict[2] = 1.0;
         dict[1] = 0.0;
         return dict;
     }
 }


 Throw in a writeln statement in getter and be amazed : D

     class Agent
     {
         private
         {
             double[int] mDict;
         }

         // Setter: copy
         void beliefs(ref double[int] dict)
         {
             import std.stdio : writeln;
             writeln("Setter function.");
             this.mDict = dict;
         }

         // Getter
         auto ref beliefs()
         {
             writeln("getter");
             return this.mDict;
         }
     }

     class Operator
     {
         static auto ref create()
         {
             double[int] dict;
             dict[2] = 1.0;
             dict[1] = 0.0;
             return dict;
         }
     }

     unittest
     {
         import std.stdio : writeln;

         Agent a = new Agent();

         writeln("Statement 1");
         a.beliefs = Operator.create();
         writeln("Statement 2");
         assert(a.beliefs.keys.length == 2);

         writeln("Statement 3");
         writeln(a.beliefs);
     }

 If you remove the ref then "Statement 1" will use the setter 
 method. Otherwise it calls the "auto ref" function to get a 
 reference of 'mDict' and assign it a value. Essentially

         // Getter
         auto ref beliefs()
         {
             return this.mDict;
         }

 makes your "private" variable not private any more, since you 
 are leaking a reference to it. So any program calling 
 "a.beliefs" can get a reference to your private data and modify 
 it.

 My guess is that (if ref is removed from the setter) it serves 
 as a  property function:
 https://dlang.org/spec/function.html#property-functions
 So this allows the function to work in a statement like:
 a.beliefs = <Whatever>

 And apparently it is prioritized over the "ref beliefs()" 
 function.


 I guess it is nicer to actually use the "setter" function, as 
 it allows you to inspect what the private data will be assigned 
 to and take appropriate measures, whereas with the "ref 
 beliefs()" method the private data is open for any manipulation 
 out of your control.


How strange! Nice find though, thanks for your help with this, I 
really appreciate it as this had been bugging me for a while.

So essentially, what is happening is, if the setter method 
expects a reference, then because the create() function isn't 
returning a reference but a copy, it actually does the following:

a.beliefs = Operator.create();

receives an object from create(), while a.beliefs retrieves a 
reference to agent.mBeliefs, and then effectively sets this 
reference to point to the object copied by create()?

Also, yes, I am using the setter method to play around with the 
precision of the double values, and do some normalising. I always 
want it to access the setter method, but I had hoped that, given 
it's an associative array, that the creation of the object in 
create() would simply create it on the heap and I could pass back 
a reference. It seems that I was incorrect, as if I set create() 
to explicitly return a reference, it complains about returning a 
reference to a local variable. Any advice on the best way to pass 
it as a reference?
Jul 06 2018
parent reply Timoses <timosesu gmail.com> writes:
On Friday, 6 July 2018 at 15:51:34 UTC, Michael wrote:

 Also, yes, I am using the setter method to play around with the 
 precision of the double values, and do some normalising.


While writing I realized that the following is even the case 
without the 'ref' parameter:
The caller of the setter will still be able to change the content 
of your private data after you checked it for validity. Take 
following example:

     class A
     {
         private static string[] _disallowed = ["damn"];
     	private string[int] _dict;

         // called before and after every member method call
         invariant
         {
             import std.algorithm : any, canFind;
             // don't allow _dict to contain any from _disallowed
              assert(!this._dict.byValue()
                     .any!((entry) => _disallowed.canFind(entry)));
         }

          property dict(string[int] dict)
         {
              // checks ...
             this._dict = dict;
         }

         void check() {
             import std.stdio : writeln;
             writeln(this._dict);
         }
     }

     unittest
     {
         string[int] loc;
         auto a = new A();
         loc[1] = "hello john";
         a.dict = loc; // using the  property setter
         loc[1] = "damn";
         a.check;
     }

What might be safer is using 'opIndexAssign'

         void opIndexAssign(string value, int key)
         {
             import std.algorithm.searching : canFind;
             if (!_disallowed.canFind(value))
                 this._dict[key] = value;
         }

and also duping the dictionary that is handed to the Agent

          property dict(string[int] dict)
         {
              // checks ...
             this._dict = dict.dup;
         }

So you can call

         string[int] loc;
         auto a = new A();
         loc[1] = "hello john";
         a.dict = loc;
         a[1] = "damn";
         loc[1] = "damn";
         a.check;

without assigning bad values.




 I always want it to access the setter method, but I had hoped 
 that, given it's an associative array, that the creation of the 
 object in create() would simply create it on the heap and I 
 could pass back a reference. It seems that I was incorrect, as 
 if I set create() to explicitly return a reference, it 
 complains about returning a reference to a local variable. Any 
 advice on the best way to pass it as a reference?


I suppose this might already answer your question:
https://forum.dlang.org/post/edrejkakhaylivlqjaqe forum.dlang.org
Jul 06 2018
parent Michael <michael toohuman.io> writes:
On Friday, 6 July 2018 at 16:24:03 UTC, Timoses wrote:

 On Friday, 6 July 2018 at 15:51:34 UTC, Michael wrote:

 [...]


 While writing I realized that the following is even the case 
 without the 'ref' parameter:
 The caller of the setter will still be able to change the 
 content of your private data after you checked it for validity. 
 Take following example:

 [...]


I understand your warning, but as the array is created purely to 
assign to the agent's beliefs, it only appears in the main() 
function as a rhs reference, which is passed to the agent, the 
reference is stored in the member variable, and then the main 
function continues, holding no reference to that array outside of 
agent's agent.beliefs getter function. Appreciate the heads up, 
though.
Jul 06 2018