www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Flaw in DIP1000? Returning a Result Struct in DIP1000

reply Jack Stouffer <jack jackstouffer.com> writes:
Consider this example simplified from this PR 
https://github.com/dlang/phobos/pull/6281

------
struct GetoptResult
{
     Option[] options;
}

struct Option
{
     string optShort;
     string help;
}

GetoptResult getopt(T...)(scope T opts)  safe
{
     GetoptResult res;
     auto o = Option(opts[0], opts[1]);
     res.options ~= o;
     return res;
}

void main()  safe
{
     bool arg;
     getopt("arg", "info", &arg);
}
------

$ dmd -dip1000 -run main.d

------
main.d(16): Error: scope variable o assigned to non-scope res
main.d(23): Error: template instance `onlineapp.getopt!(string, 
string, bool*)` error instantiating
------

The only way I've found to make the code compile and retain the 
pre-dip1000 behavior is to change the Option construction to

------
auto o = Option(opts[0].idup, opts[1].idup);
------

How can we return non-scoped result variables constructed from 
scope variables without copies?
Mar 21 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, March 21, 2018 17:13:40 Jack Stouffer via Digitalmars-d wrote:
 Consider this example simplified from this PR
 https://github.com/dlang/phobos/pull/6281

 ------
 struct GetoptResult
 {
      Option[] options;
 }

 struct Option
 {
      string optShort;
      string help;
 }

 GetoptResult getopt(T...)(scope T opts)  safe
 {
      GetoptResult res;
      auto o = Option(opts[0], opts[1]);
      res.options ~= o;
      return res;
 }

 void main()  safe
 {
      bool arg;
      getopt("arg", "info", &arg);
 }
 ------

 $ dmd -dip1000 -run main.d

 ------
 main.d(16): Error: scope variable o assigned to non-scope res
 main.d(23): Error: template instance `onlineapp.getopt!(string,
 string, bool*)` error instantiating
 ------

 The only way I've found to make the code compile and retain the
 pre-dip1000 behavior is to change the Option construction to

 ------
 auto o = Option(opts[0].idup, opts[1].idup);
 ------

 How can we return non-scoped result variables constructed from
 scope variables without copies?
The struct being returned would need to be marked with scope (or its members marked with scope) such that the compiler treated the result as containing values from the function arguments. I don't know whether that's possible with DIP 1000 as-is, and that could have some pretty nasty consquences when you consider how that then limits what can be done with the return value. Even if we _can_ mark GetoptResult as scope in some manner so that the copying isn't necessary, you're then just pushing the problem a level up. In this case, that's probably not a big deal, since this is stuff that's just going to be used in main and thrown away, but in the general case, having a struct that won't let you escape any of its members means that you're going to either need a struct with the same layout but without scope that you can copy the scope on to, or you're going to need to pull out each of the members individually to copy them or do whatever you need to do to work around scope. My gut reaction is that issues along these lines will either prevent the use of scope when returning user-defined types as opposed to pointers or dynamic arrays, and/or they'll force the kind of copying that you're complaining about. But I don't think that I understand DIP 1000 well enough to know what it's limitations really are in cases like this. Hopefully, Walter has an answer. However, I suspect that we're going to find cases where scope is a blunt enough instrument that we're going to be forced to either drop it or use trusted in places (though in this case, using trusted would be completely unreasonable, because you can't guarantee that it _isn't_ a problem for something from the caller to escape - not without examining the caller code, which would translate to marking the caller as trusted, not the function being called). Actually, I'd be very surprised if we _didn't_ have cases like that. The question is how frequent those cases are and whether issues like this are going to pop up enough that it's going to usually make more sense to simply use pure and manually examine code to mark it as trusted rather than use scope so that the compiler can mark a bunch of stuff as safe for us. As long as the function doesn't return scope, we're probably fine, but once it starts returning scope, things start getting interesting. In this particular case, it may make more sense to just let getopt be safe on its own and just let the caller mark all of the uses of & as trusted. I know that the bug report that sparked this is trying to make using getopt completely safe without requiring the use of trusted at all, but I don't think that being forced to allocate memory is worth that, and I don't think that mucking around with getopt to make it work with ref is worth that. There are advantages to getopt taking pointers, and I'd rather not see getopt's API change in a way that breaks code. If getopt itself is safe, then it's trivial for the caller to determine that their code is safe in spite of the use of & and thus either use trusted appropriately or just not bother, since it's in main, which usually isn't doing much fancy. So, I _really_ don't think that making calling getopt inherently safe is worth code breakage if it comes to that, and I don't think that it's worth allocating memory that we otherwise wouldn't have to allocate. If it's possible to mark GetoptResult with scope such that we can use scope without copying, then great, but if it's not, then I'm inclined to argue that we should just make sure that getopt itself is safe and not worry about whether the caller is doing anything system to call getopt or not. Regardless, this does raise a potential issue with scope and user-defined return types, and we should explore how possible it is for DIP 1000 to solve that problem without forcing copies that wouldn't be necessary in system code. - Jonathan M Davis
Mar 21 2018
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Wednesday, 21 March 2018 at 18:50:59 UTC, Jonathan M Davis 
wrote:
 The struct being returned would need to be marked with scope 
 (or its members marked with scope) such that the compiler 
 treated the result as containing values from the function 
 arguments. I don't know whether that's possible with DIP 1000 
 as-is
Not as far as I can tell. Marking it as scope in the function body means it can't be returned, and marking the array in GetoptResult as scope results in a syntax error.
 In this particular case, it may make more sense to just let 
 getopt be  safe on its own and just let the caller mark all of 
 the uses of & as  trusted.
This is thankfully the case currently.
 If it's possible to mark GetoptResult with scope such that we 
 can use scope without copying, then great, but if it's not, 
 then I'm inclined to argue that we should just make sure that 
 getopt itself is  safe and not worry about whether the caller 
 is doing anything  system to call getopt or not.

 Regardless, this does raise a potential issue with scope and 
 user-defined return types, and we should explore how possible 
 it is for DIP 1000 to solve that problem without forcing copies 
 that wouldn't be necessary in  system code.
My cause for alarm with this limitation is this is one of the issues (taking address of locals safely) that DIP1000 was designed to solve. If, in practice, DIP1000 code can't be used for this case when the code become sufficiently complex, then we have a real problem on our hands.
Mar 21 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, March 21, 2018 19:50:52 Jack Stouffer via Digitalmars-d wrote:
 On Wednesday, 21 March 2018 at 18:50:59 UTC, Jonathan M Davis

 wrote:
 The struct being returned would need to be marked with scope
 (or its members marked with scope) such that the compiler
 treated the result as containing values from the function
 arguments. I don't know whether that's possible with DIP 1000
 as-is
Not as far as I can tell. Marking it as scope in the function body means it can't be returned, and marking the array in GetoptResult as scope results in a syntax error.
 In this particular case, it may make more sense to just let
 getopt be  safe on its own and just let the caller mark all of
 the uses of & as  trusted.
This is thankfully the case currently.
 If it's possible to mark GetoptResult with scope such that we
 can use scope without copying, then great, but if it's not,
 then I'm inclined to argue that we should just make sure that
 getopt itself is  safe and not worry about whether the caller
 is doing anything  system to call getopt or not.

 Regardless, this does raise a potential issue with scope and
 user-defined return types, and we should explore how possible
 it is for DIP 1000 to solve that problem without forcing copies
 that wouldn't be necessary in  system code.
My cause for alarm with this limitation is this is one of the issues (taking address of locals safely) that DIP1000 was designed to solve. If, in practice, DIP1000 code can't be used for this case when the code become sufficiently complex, then we have a real problem on our hands.
Well, it certainly looks like once user-defined types are being returned, DIP 1000 falls short. And even if it were improved to allow for member variables to be marked with scope such that the compiler allowed for arguments to escape via a return value (which may or may not make sense with how DIP 1000 works), we then have the problem with potentially needing to copy the member variables (or the entire struct) in the caller - either that or to cast and have it be trusted. I don't know how a big a problem this is really going to be in practice, but it certainly shows that DIP 1000 isn't a silver bullet. - Jonathan M Davis
Mar 22 2018
prev sibling next sibling parent reply Meta <jared771 gmail.com> writes:
On Wednesday, 21 March 2018 at 17:13:40 UTC, Jack Stouffer wrote:
 Consider this example simplified from this PR 
 https://github.com/dlang/phobos/pull/6281

 ------
 struct GetoptResult
 {
     Option[] options;
 }

 struct Option
 {
     string optShort;
     string help;
 }

 GetoptResult getopt(T...)(scope T opts)  safe
 {
     GetoptResult res;
     auto o = Option(opts[0], opts[1]);
     res.options ~= o;
     return res;
 }

 void main()  safe
 {
     bool arg;
     getopt("arg", "info", &arg);
 }
 ------

 $ dmd -dip1000 -run main.d

 ------
 main.d(16): Error: scope variable o assigned to non-scope res
 main.d(23): Error: template instance `onlineapp.getopt!(string, 
 string, bool*)` error instantiating
 ------

 The only way I've found to make the code compile and retain the 
 pre-dip1000 behavior is to change the Option construction to

 ------
 auto o = Option(opts[0].idup, opts[1].idup);
 ------

 How can we return non-scoped result variables constructed from 
 scope variables without copies?
I thought that maybe adding a function to Option and marking it as `scope` would work: struct GetoptResult { Option[] options; void addOptions(scope Option opt) safe scope { //Error: scope variable opt assigned to non-scope this options ~= opt; } } But the compiler doesn't like that. However, I _did_ get it working by doing this: GetoptResult getopt(T...)(scope T opts) safe { return GetoptResult([Option(opts[0], opts[1])]); } Which is not ideal, obviously, but the notion that some code has to be rewritten to accomodate ownership semantics is not a new one; one of the major complaints I've seen about Rust is that it requires you to adjust your coding style to satisfy the borrow checker.
Mar 21 2018
parent Jack Stouffer <jack jackstouffer.com> writes:
On Wednesday, 21 March 2018 at 19:15:41 UTC, Meta wrote:
 But the compiler doesn't like that. However, I _did_ get it 
 working by doing this:

 GetoptResult getopt(T...)(scope T opts)  safe
 {
     return GetoptResult([Option(opts[0], opts[1])]);
 }

 Which is not ideal, obviously, but the notion that some code 
 has to be rewritten to accomodate ownership semantics is not a 
 new one; one of the major complaints I've seen about Rust is 
 that it requires you to adjust your coding style to satisfy the 
 borrow checker.
The problem here is that it's impossible to apply this to the actual getopt code :/
Mar 21 2018
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 21 March 2018 at 17:13:40 UTC, Jack Stouffer wrote:
 [snip]

 How can we return non-scoped result variables constructed from 
 scope variables without copies?
If you re-wrote this so that it just had pointers, would it be simpler? Below is my attempt, not sure it's the same... struct Foo { int b; } struct Bar { Foo* a; } Bar bar(scope int* a) safe { Bar res; Foo x = Foo(*a); res.a = &x; return res; } void main() safe { int x = 1; bar(&x); }
Mar 21 2018