www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Is safe still a work-in-progress?

reply Peter Alexander <peter.alexander.au gmail.com> writes:
--------
import std.algorithm, std.stdio;

 safe:

auto foo() {
   int[6] xs = [0, 1, 2, 3, 4, 5];
   return xs[].map!(x => x);
}

void main() {
   writeln(foo());
}
--------

https://run.dlang.io/is/qC7HUR

For me this gives:

--------
[0, 0, -2132056872, 22008, 0, 0]
--------

Which looks like its just reading arbitrary memory.

I've filed https://issues.dlang.org/show_bug.cgi?id=19175

My question is: what is the status of  safe? I am quite surprised 
to see such a simple case fail. Is  safe believed to be fully 
implemented (modulo bugs) and this is just an unfortunate corner 
case, or is it known work-in-progress?
Aug 17 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via Digitalmars-d 
wrote:

 --------
 import std.algorithm, std.stdio;

  safe:

 auto foo() {
    int[6] xs = [0, 1, 2, 3, 4, 5];
    return xs[].map!(x => x);
 }

 void main() {
    writeln(foo());
 }
 --------

 https://run.dlang.io/is/qC7HUR

 For me this gives:

 --------
 [0, 0, -2132056872, 22008, 0, 0]
 --------

 Which looks like its just reading arbitrary memory.

 I've filed https://issues.dlang.org/show_bug.cgi?id=19175

 My question is: what is the status of  safe? I am quite surprised
 to see such a simple case fail. Is  safe believed to be fully
 implemented (modulo bugs) and this is just an unfortunate corner
 case, or is it known work-in-progress?


That particular bug is a duplicate of
https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed
based on the fact that -dip1000 fixes the problem by treating marking the
slice of a static array with scope. It's still quite broken without -dip1000
though.

Honestly, the reality of the matter is that  safe is probably always going
to be somewhat broken, because it's implemented via blacklisting rather than
whitelisting. Instead of  safe only allowing stuff that's been proven to be
 safe, it disallows stuff that a programmer decided was  system. The bug you
ran into is a pretty glaring one that arguably should have been fixed ages
ago, but given how hard it is to prove what is and isn't  safe, there are
bound to be corner cases which have been missed. As we find them, they'll be
fixed, but who knows how many are left or whether we'll ever actually get
them all.

- Jonathan M Davis
Aug 17 2018
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote:


 That particular bug is a duplicate of 
 https://issues.dlang.org/show_bug.cgi?id=8838, which was closed 
 as fixed based on the fact that -dip1000 fixes the problem by 
 treating marking the slice of a static array with scope. It's 
 still quite broken without -dip1000 though.


It still appears to be broke even with -dip1000:  
https://run.dlang.io/is/gJi2Fa
Aug 17 2018
prev sibling next sibling parent reply vit <vit vit.vit> writes:
On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote:

 On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via 
 Digitalmars-d wrote:

 [...]


 That particular bug is a duplicate of 
 https://issues.dlang.org/show_bug.cgi?id=8838, which was closed 
 as fixed based on the fact that -dip1000 fixes the problem by 
 treating marking the slice of a static array with scope. It's 
 still quite broken without -dip1000 though.

 [...]


What's the state of -dip1000?
Aug 17 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, August 17, 2018 2:23:20 AM MDT vit via Digitalmars-d wrote:

 On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote:

 On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via

 Digitalmars-d wrote:

 [...]


 That particular bug is a duplicate of
 https://issues.dlang.org/show_bug.cgi?id=8838, which was closed
 as fixed based on the fact that -dip1000 fixes the problem by
 treating marking the slice of a static array with scope. It's
 still quite broken without -dip1000 though.

 [...]


 What's the state of -dip1000?


Well, I _think_ that aside from bugs (which may or may not be a small
number), the compiler implementation is complete. However, there's still a
fair bit of work to do in Phobos to make it fully work with -dip1000. And
AFAIK, we still don't have a real plan as to how we're going to make
-dip1000 the default. But given how much -dip1000 seems to break, I don't
know how we're going to switch to it being the default without being highly
disruptive in the process.

- Jonathan M Davis
Aug 17 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 3:01 AM, Jonathan M Davis wrote:

 we still don't have a real plan as to how we're going to make
 -dip1000 the default.


We do have a plan. Get Phobos to compile with -dip1000. As to why that hasn't 
happened yet, see:

https://github.com/dlang/dmd/pull/8504
Aug 17 2018
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 17 August 2018 at 08:23:20 UTC, vit wrote:

 On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis 
 wrote:

 On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via 
 Digitalmars-d wrote:

 [...]


 That particular bug is a duplicate of 
 https://issues.dlang.org/show_bug.cgi?id=8838, which was 
 closed as fixed based on the fact that -dip1000 fixes the 
 problem by treating marking the slice of a static array with 
 scope. It's still quite broken without -dip1000 though.

 [...]


 What's the state of -dip1000?


I've been using -dip1000 a lot lately. I hit two bugs yesterday. 
When it works, it's great, _except_:

.  safe isn't default
. -dip1000 isn't default
. Good luck figuring out why your template functions aren't  safe 
and coaxing the compiler to tell you why it's inferring the 
attributes it is. Bonus points if it's a Phobos function so you 
can't slap ` safe` on its definition.

Take excel-d for instance. The other day somebody filed a bug 
because Excel was crashing when they wrapped some D code that 
uses std.regex. After investigating I found out that regex 
memoizes the string with the regular expression in it, 
effectively escaping the string passed from Excel to the D 
function. Unfortunately for everyone involved, that string came 
from a region allocator, and after bumping the pointer back, boom.

Telling people to not escape reference parameters doesn't really 
work, because nobody will read that part of the README, and, as 
in the example above, the user didn't even know they were doing 
it!

Ok, so as a good library author, I tried to make it a 
compile-time error to shoot yourself in the foot that way, and I 
did. Great, right? Well, no. I can't force the user to write 
 safe functions, much less to compile with -dip1000. I changed 
the example XLL to use -dip1000 because more likely than not 
users will copy the example and edit it. But I can't force them 
to write  safe functions, and if they don't (the likeliest 
scenario given it's not the default), they'll be able to escape 
whatever it is they want, Excel will crash, and the compiler will 
be ok with it.

And if they _do_ write  safe functions and compile with -dip1000, 
chances are at some point they won't understand the compiler 
error message, make their function  trusted and... crash.
I don't know that because I have a crystal ball, I know that 
because I did it myself this week. Twice.

Did I mention the bugs? I wrote about my interesting experiences 
from this week with -dip1000 here:

https://forum.dlang.org/post/lxavagebcmaxjdzisyfk forum.dlang.org
Aug 17 2018
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 17/08/2018 11:33 PM, Atila Neves wrote:

 . Good luck figuring out why your template functions aren't  safe and 
 coaxing the compiler to tell you why it's inferring the attributes it 
 is. Bonus points if it's a Phobos function so you can't slap ` safe` on 
 its definition.


Sure you can. It's a template so it'll be initialized as its required.

I've done that to fix a bug in druntime before and in my own libraries.
Aug 17 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote:

 On 17/08/2018 11:33 PM, Atila Neves wrote:

 . Good luck figuring out why your template functions aren't 
  safe and coaxing the compiler to tell you why it's inferring 
 the attributes it is. Bonus points if it's a Phobos function 
 so you can't slap ` safe` on its definition.


 Sure you can. It's a template so it'll be initialized as its 
 required.

 I've done that to fix a bug in druntime before and in my own 
 libraries.


It's not easy though. You have to either be building your own 
phobos or sudo editing files in /usr/include. You can, but it's a 
pain. Then there's finding out exactly where in a chain of 10 
template functions that it became  system...
Aug 17 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via Digitalmars-d 
wrote:

 On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote:

 On 17/08/2018 11:33 PM, Atila Neves wrote:

 . Good luck figuring out why your template functions aren't
  safe and coaxing the compiler to tell you why it's inferring
 the attributes it is. Bonus points if it's a Phobos function
 so you can't slap ` safe` on its definition.


 Sure you can. It's a template so it'll be initialized as its
 required.

 I've done that to fix a bug in druntime before and in my own
 libraries.


 It's not easy though. You have to either be building your own
 phobos or sudo editing files in /usr/include. You can, but it's a
 pain. Then there's finding out exactly where in a chain of 10
 template functions that it became  system...


The reality of the matter is that you tend to have to be very motivated to
make your code  safe if you're doing much with templates (the same with
other attributes such as pure). And sadly, it can quickly get to the point
that it just feels like it's not worth it in spite of the fact that there is
definitely value to be had in using them.

I wonder what it would take to make it so that the compiler could actually
tell you where in the function call chain it first becomes  system.

- Jonathan M Davis
Aug 17 2018
parent reply vit <vit vit.vit> writes:
On Friday, 17 August 2018 at 13:02:20 UTC, Jonathan M Davis wrote:

 On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via 
 Digitalmars-d wrote:

 On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole 
 wrote:

 On 17/08/2018 11:33 PM, Atila Neves wrote:

 . Good luck figuring out why your template functions aren't 
  safe and coaxing the compiler to tell you why it's 
 inferring the attributes it is. Bonus points if it's a 
 Phobos function so you can't slap ` safe` on its definition.


 Sure you can. It's a template so it'll be initialized as its 
 required.

 I've done that to fix a bug in druntime before and in my own 
 libraries.


 It's not easy though. You have to either be building your own 
 phobos or sudo editing files in /usr/include. You can, but 
 it's a pain. Then there's finding out exactly where in a chain 
 of 10 template functions that it became  system...


 The reality of the matter is that you tend to have to be very 
 motivated to make your code  safe if you're doing much with 
 templates (the same with other attributes such as pure). And 
 sadly, it can quickly get to the point that it just feels like 
 it's not worth it in spite of the fact that there is definitely 
 value to be had in using them.

 I wonder what it would take to make it so that the compiler 
 could actually tell you where in the function call chain it 
 first becomes  system.

 - Jonathan M Davis


Does -dip1000 affect alias template parameters and closures?
example:

import std.experimental.all;


void main() safe  nogc{
     int i = 2;

     const x = iota(0, 10)
         .filter!((x)scope => x == i)
         .map!((x)scope => x * x)
         .front;	

     assert(x == 4);
}

onlineapp.d(4): Error: function `D main` is  nogc yet allocates 
closures with the GC
onlineapp.d(8):        onlineapp.main.__lambda1 closes over 
variable i at onlineapp.d(5)
Aug 17 2018
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 18/08/2018 1:29 AM, vit wrote:

 On Friday, 17 August 2018 at 13:02:20 UTC, Jonathan M Davis wrote:

 On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via 
 Digitalmars-d wrote:

 On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote:

 On 17/08/2018 11:33 PM, Atila Neves wrote:

 . Good luck figuring out why your template functions aren't >> 




  safe and coaxing the compiler to tell you why it's >> inferring the 
 attributes it is. Bonus points if it's a >> Phobos function so you 
 can't slap ` safe` on its definition.

 Sure you can. It's a template so it'll be initialized as its > 


 required.

 I've done that to fix a bug in druntime before and in my own > 


 libraries.

 It's not easy though. You have to either be building your own phobos 
 or sudo editing files in /usr/include. You can, but it's a pain. Then 
 there's finding out exactly where in a chain of 10 template functions 
 that it became  system...


 The reality of the matter is that you tend to have to be very 
 motivated to make your code  safe if you're doing much with templates 
 (the same with other attributes such as pure). And sadly, it can 
 quickly get to the point that it just feels like it's not worth it in 
 spite of the fact that there is definitely value to be had in using them.

 I wonder what it would take to make it so that the compiler could 
 actually tell you where in the function call chain it first becomes 
  system.

 - Jonathan M Davis


 
 Does -dip1000 affect alias template parameters and closures?
 example:
 
 import std.experimental.all;
 
 
 void main() safe  nogc{
      int i = 2;
 
      const x = iota(0, 10)
          .filter!((x)scope => x == i)
          .map!((x)scope => x * x)
          .front;
 
      assert(x == 4);
 }
 
 onlineapp.d(4): Error: function `D main` is  nogc yet allocates closures 
 with the GC
 onlineapp.d(8):        onlineapp.main.__lambda1 closes over variable i 
 at onlineapp.d(5)


No it shouldn't. You used i there which automatically requires a heap 
allocation for a delegate. Hence it uses GC.
Aug 17 2018
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 5:30 AM, Atila Neves wrote:

 It's not easy though. You have to either be building your own phobos


Not really. You only have to be able to import phobos files. You don't have to 
compile something that will link.

I.e. just copy the phobos source tree, add  safe to it, and point the compiler 
at that one.



 or sudo 
 editing files in /usr/include. You can, but it's a pain. Then there's finding 
 out exactly where in a chain of 10 template functions that it became  system...


I recently fixed the compiler to say which template overload was used. It's a 
huge time saver.

But you're currently doomed on this anyway, as an awful lot of Phobos winds up 
calling put(), which cannot be inferred as  safe without:

https://github.com/dlang/dmd/pull/8504
Aug 17 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 4:33 AM, Atila Neves wrote:

 I've been using -dip1000 a lot lately.


Great news!


 I hit two bugs yesterday.


In D? or in your code?


 When it works, it's great, _except_:
 
 .  safe isn't default
 . -dip1000 isn't default
 . Good luck figuring out why your template functions aren't  safe and coaxing 
 the compiler to tell you why it's inferring the attributes it is.


I empathize with that, having run into the same problem. A couple months ago I 
improved the resulting error messages, which make it a lot easier to track down 
why it is inferring what it does.


 Bonus points 
 if it's a Phobos function so you can't slap ` safe` on its definition.


Sometimes I think it is better to just slap them with  safe and to hell with 
unsafe parameters. :-)


 Did I mention the bugs?


Bugs don't count if they're not in bugzilla!
Aug 17 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 18 August 2018 at 02:17:01 UTC, Walter Bright wrote:

 On 8/17/2018 4:33 AM, Atila Neves wrote:

 I've been using -dip1000 a lot lately.


 Great news!


:)



 I hit two bugs yesterday.


 In D? or in your code?


In dmd.


 Bugs don't count if they're not in bugzilla!


As mentioned in my other thread, I went to file one and it was 
declared not a bug:

https://issues.dlang.org/show_bug.cgi?id=17927

I don't know how that's possible given that supposedly  safe code 
is doing pretty bad things. That disheartened me enough to not 
continue although I did file this one:

https://issues.dlang.org/show_bug.cgi?id=19173
Aug 20 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/20/2018 6:08 AM, Atila Neves wrote:

 As mentioned in my other thread, I went to file one and it was declared not a
bug:
 
 https://issues.dlang.org/show_bug.cgi?id=17927


I'll look at it again.
Aug 20 2018
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 12:50 AM, Jonathan M Davis wrote:

 That particular bug is a duplicate of
 https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed
 based on the fact that -dip1000 fixes the problem by treating marking the
 slice of a static array with scope. It's still quite broken without -dip1000
 though.


D will never be  safe until -dip1000 is the default. However, I cannot get any 
traction with improving this:

https://github.com/dlang/dmd/pull/8504

blocks progress on getting Phobos to compile with -dip1000, and nobody will
help 
me with it.



 Honestly, the reality of the matter is that  safe is probably always going
 to be somewhat broken, because it's implemented via blacklisting rather than
 whitelisting. Instead of  safe only allowing stuff that's been proven to be
  safe, it disallows stuff that a programmer decided was  system. The bug you
 ran into is a pretty glaring one that arguably should have been fixed ages
 ago,


It was fixed with -dip1000 ages ago.



 but given how hard it is to prove what is and isn't  safe, there are
 bound to be corner cases which have been missed. As we find them, they'll be
 fixed, but who knows how many are left or whether we'll ever actually get
 them all.


The whitelisting idea has come up before. I see it, though, as a way to avoid 
dealing with the issues as nobody is willing to do any work at all on this 
approach. Furthermore, I see no evidence that whitelisting will produce
superior 
results.

On the other hand, I see plenty of evidence that  safe bugs that are found and 
posted to bugzilla get fixed.

If people want to see  safe be watertight,

1. post any problems to bugzilla, and tag them with the 'safe' keyword.

2. investigate solutions to any issues tagged with 'safe' keyword.

3. review/approve PR 8504.
Aug 17 2018
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 17, 2018 at 01:50:32AM -0600, Jonathan M Davis via Digitalmars-d
wrote:
[...]

 Honestly, the reality of the matter is that  safe is probably always
 going to be somewhat broken, because it's implemented via blacklisting
 rather than whitelisting. Instead of  safe only allowing stuff that's
 been proven to be  safe, it disallows stuff that a programmer decided
 was  system.


Sigh:

	https://issues.dlang.org/show_bug.cgi?id=12941

This was reported 4 years ago, but was unfortunately closed as invalid.

It will continue to be a problem as long as  safe is implemented via
blacklisting, because every single time there's a new language feature,
there's a chance that a loophole is introduced into  safe. And that's
not counting the combinatorial explosion of existing language features
that might lead to  safe loopholes, that we simply haven't thought of
yet.  It's like allowing anyone to enter your house freely except those
few people whom you've explicitly named. You can't possibly expect *not*
to get robbed that way.



 The bug you ran into is a pretty glaring one that arguably should have
 been fixed ages ago, but given how hard it is to prove what is and
 isn't  safe, there are bound to be corner cases which have been
 missed. As we find them, they'll be fixed, but who knows how many are
 left or whether we'll ever actually get them all.


[...]

And that is exactly why the whole implementation of  safe is currently
rather laughable. By blacklisting rather than whitelisting, we basically
open the door wide open to loopholes -- anything that we haven't thought
of yet could potentially be a  safe-breaking combination, and we
wouldn't know until somebody discovers and reports it.

Sadly, it seems there is little interest in reimplementing  safe to use
whitelisting instead of blacklisting.


T

-- 
A program should be written to model the concepts of the task it performs
rather than the physical world or a process because this maximizes the
potential for it to be applied to tasks that are conceptually similar and, more
important, to tasks that have not yet been conceived. -- Michael B. Allen
Aug 17 2018
next sibling parent David Gileadi <gileadisNOSPM gmail.com> writes:
On 8/17/18 7:26 AM, H. S. Teoh wrote:

 It will continue to be a problem as long as  safe is implemented via
 blacklisting, because every single time there's a new language feature,
 there's a chance that a loophole is introduced into  safe. And that's
 not counting the combinatorial explosion of existing language features
 that might lead to  safe loopholes, that we simply haven't thought of
 yet.  It's like allowing anyone to enter your house freely except those
 few people whom you've explicitly named. You can't possibly expect *not*
 to get robbed that way.


We should tag bugs like these as #safecracking
Aug 17 2018
prev sibling next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote:

 On Fri, Aug 17, 2018 at 01:50:32AM -0600, Jonathan M Davis via 
 Digitalmars-d wrote: [...]

 Honestly, the reality of the matter is that  safe is probably 
 always going to be somewhat broken, because it's implemented 
 via blacklisting rather than whitelisting. Instead of  safe 
 only allowing stuff that's been proven to be  safe, it 
 disallows stuff that a programmer decided was  system.


 Sigh:

 	https://issues.dlang.org/show_bug.cgi?id=12941

 This was reported 4 years ago, but was unfortunately closed as 
 invalid.

 It will continue to be a problem as long as  safe is 
 implemented via blacklisting, because every single time there's 
 a new language feature, there's a chance that a loophole is 
 introduced into  safe. And that's not counting the 
 combinatorial explosion of existing language features that 
 might lead to  safe loopholes, that we simply haven't thought 
 of yet.  It's like allowing anyone to enter your house freely 
 except those few people whom you've explicitly named. You can't 
 possibly expect *not* to get robbed that way.


I knew there was something fundamentally wrong with  safe, but I 
could never put my finger on it.  Now that you and Jonathan 
mention this, it becomes clear.

This makes me exceptionally sad.  D is great in so many ways, but 
then this taints the pool.   I asked if D was ever going to be 
 safe by default at DConf (https://youtu.be/HvqsUO77FGI?t=13242), 
but it didn't elicit a very positive answer.

It seems D is backtracking in some ways ( nogc, -betterC), trying 
to evolve it into something it wasn't originally envisioned to 
be, and now we have another one to add to the list.


 The bug you ran into is a pretty glaring one that arguably 
 should have been fixed ages ago, but given how hard it is to 
 prove what is and isn't  safe, there are bound to be corner 
 cases which have been missed. As we find them, they'll be 
 fixed, but who knows how many are left or whether we'll ever 
 actually get them all.


 [...]

 And that is exactly why the whole implementation of  safe is 
 currently rather laughable. By blacklisting rather than 
 whitelisting, we basically open the door wide open to loopholes 
 -- anything that we haven't thought of yet could potentially be 
 a  safe-breaking combination, and we wouldn't know until 
 somebody discovers and reports it.

 Sadly, it seems there is little interest in reimplementing 
  safe to use whitelisting instead of blacklisting.


I think there is probably some interest, though maybe not from 
the ones with the position or ability to make it happen.  A DIP 
might be the way forward, but it seems like quite a difficult 
task to turn it right-side-up at this point.  I actually started 
writing a DIP for this about a year ago, but I need to pick my 
battles.

Mike
Aug 17 2018
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Friday, 17 August 2018 at 15:27:22 UTC, Mike Franklin wrote:

 I actually started writing a DIP for this about a year ago, but 
 I need to pick my battles.

 Mike


Is it on github?

Alex
Aug 17 2018
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 August 2018 at 16:00:26 UTC, 12345swordy wrote:

 On Friday, 17 August 2018 at 15:27:22 UTC, Mike Franklin wrote:

 I actually started writing a DIP for this about a year ago, 
 but I need to pick my battles.

 Mike


 Is it on github?

 Alex


No,but here are some notes I found in my files.

 safe by default
Before Transition

     Write a program that parses D code and explicitly adds 
 system if no function is not decorated with any safety attribute
     Add a transitional -explicit-safety compiler switch that 
emits a warning if a safety attribute is not explicitly specified.
     Update all dlang repositories and maybe others with explicit 
attributes using aforementioned program.
         Turn on -explicit-safety for all CIs so any new PRs so 
explicit safety is added to all new functions.
     Add transitional  system-by-default compiler switch for 
backward compatibility
         This will be especially useful to keep our CIs running 
with legacy code
         This will also be useful for programs with dub 
dependencies that need to continue running
     Add - system-main compiler switch for backward compatibility
     Add - safe-main and - safe-by-default compiler switch for 
forward-looking D projects
     Update all of D's tutorials, examples, etc… with explicit 
safety attributes.

Transition

     Add changelog entry warning that  safe-by-default is coming 
and how they can use the aforementioned tools eliminate disruption
         Run this change log for 1 year (4 releases)?

After Transition

     Remove  safe attributes in D's example code.
     Remove  safe attributes from phobos/druntime/dmd etc..
     Turn off -explicit-safety for those libraries that have 
updated their code.
     Deprecate  safe-main and  safe-by-default compiler switch 
through the normal deprecation process.
     -explicit-safety,  system-main and  system-by-default remain 
for an indeterminate amount of time. They should eventually be 
deprecated, but only after they have outlived their usefulness.

Difficulties

     The difficulty will be with printed material (books, etc…) 
that will become out-of-date. Recommendation is to add 
information to their published errata, or create a new edition.

Mike
Aug 17 2018
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote:

 [...]

 And that is exactly why the whole implementation of  safe is 
 currently rather laughable. By blacklisting rather than 
 whitelisting, we basically open the door wide open to loopholes 
 -- anything that we haven't thought of yet could potentially be 
 a  safe-breaking combination, and we wouldn't know until 
 somebody discovers and reports it.

 Sadly, it seems there is little interest in reimplementing 
  safe to use whitelisting instead of blacklisting.


 T


Fundamentally, I see it as a good idea. Walter has talked about 
how important memory safety is for D. People thinking their  safe 
code is safe is a big problem when that turns out to not be the 
case. Imagine the black eye D would have if a company was hacked 
because of something like this?

IMO, the problem is that you can't just replace  safe as it is 
now. You could introduce something like  whitelist or 
 safewhitelist and begin implementing it, but it would probably 
be some time before it could replace  safe. Like when  whitelist 
is only breaking unsafe code.
Aug 17 2018
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/17/18 1:26 PM, jmh530 wrote:

 On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote:

 [...]

 And that is exactly why the whole implementation of  safe is currently 
 rather laughable. By blacklisting rather than whitelisting, we 
 basically open the door wide open to loopholes -- anything that we 
 haven't thought of yet could potentially be a  safe-breaking 
 combination, and we wouldn't know until somebody discovers and reports 
 it.

 Sadly, it seems there is little interest in reimplementing  safe to 
 use whitelisting instead of blacklisting.


 T


 
 Fundamentally, I see it as a good idea. Walter has talked about how 
 important memory safety is for D. People thinking their  safe code is 
 safe is a big problem when that turns out to not be the case. Imagine 
 the black eye D would have if a company was hacked because of something 
 like this?


This will always be a possibility thanks to  trusted.


 IMO, the problem is that you can't just replace  safe as it is now. You 
 could introduce something like  whitelist or  safewhitelist and begin 
 implementing it, but it would probably be some time before it could 
 replace  safe. Like when  whitelist is only breaking unsafe code.


I have to say, I don't see how this all helps.

In theory, black-listing and white-listing will get you to the same 
position. Mechanisms to get or use pointers aren't really being added to 
the language any more, so the set of "things" to "list" either black or 
white is finite.

In this thread, we are talking about something that should have been 
black-listed LONG ago, but was not because it was "too useful" (i.e. 
would break too much code). If  safe code was white-listed, nobody would 
use it until it was finished, so it would be theoretical anyway.

Nobody wants a feature that is  safe, but not useful.

However, a bigger problem is that we have a bug that is "fixed" (slicing 
static arrays) but only if you use a feature that doesn't work correctly 
(dip1000). Why? I think the bug should be reopened until dip1000 is the 
default, or it simply gets fixed (i.e. without requiring dip1000).

-Steve
Aug 17 2018
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 17, 2018 at 05:26:48PM +0000, jmh530 via Digitalmars-d wrote:

 On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote:

 [...]
 And that is exactly why the whole implementation of  safe is
 currently rather laughable. By blacklisting rather than
 whitelisting, we basically open the door wide open to loopholes --
 anything that we haven't thought of yet could potentially be a
  safe-breaking combination, and we wouldn't know until somebody
 discovers and reports it.




[...]

 Fundamentally, I see it as a good idea. Walter has talked about how
 important memory safety is for D. People thinking their  safe code is
 safe is a big problem when that turns out to not be the case. Imagine
 the black eye D would have if a company was hacked because of
 something like this?


Indeed.



 IMO, the problem is that you can't just replace  safe as it is now.
 You could introduce something like  whitelist or  safewhitelist and
 begin implementing it, but it would probably be some time before it
 could replace  safe. Like when  whitelist is only breaking unsafe
 code.


The way I envision it is that the implementation would begin as a
separate topic branch in the dmd repo, and gradually brought up to the
point where it begins passing the testsuite.  The good thing is that
nowadays with our CI system put in place, any breakages caused in major
D projects like vibe.d would be detected by the autotester, so we can
immediately fill in any obvious missing whitelist items.

Then we'd merge it into master but controlled by a command-line switch,
say -dipxxxx if we put it under a DIP, and put out for 1 or 2 releases.
People who depend on  safe can then opt in to try it out, and report any
bugs (missing whitelist items) that they find.  We fix those bugs until
no more bugs are reported.  Then we make -dipxxxx the default, perhaps
with a reverse switch for reverting to the old  safe implementation just
in case some projects out there depend on it but haven't gotten around
to reporting bugs in the new implementation or whatever.  After another
release or so, we finally remove the switch and delete the old
implementation.

Then going forward, we will have eliminated  safe loopholes pretty much
completely (the only exception being codegen bugs), and the only likely
bugs are missing items in the whitelist.  Which should be much more
manageable and much less disastrous -- the worst that can happen is that
somebody tried to do something legal and got rejected by a big fat
compiler error.  As opposed to today's situation, where somebody tries
something illegal and the compiler fails to detect it, and nobody even
know there's a problem until it becomes an exploitable bug that causes a
security problem.  Given today's security climate, this would be utterly
disastrous for D's reputation.


T

-- 
It is of the new things that men tire --- of fashions and proposals and
improvements and change. It is the old things that startle and intoxicate. It
is the old things that are young. -- G.K. Chesterton
Aug 17 2018
parent reply bachmeier <no spam.net> writes:
On Friday, 17 August 2018 at 17:59:45 UTC, H. S. Teoh wrote:


 As opposed to today's situation, where somebody tries something 
 illegal and the compiler fails to detect it, and nobody even 
 know there's a problem until it becomes an exploitable bug that 
 causes a security problem.  Given today's security climate, 
 this would be utterly disastrous for D's reputation.


This is a good example of D needing to evolve or fizzle out. I 
don't see evidence that the community has yet figured out how to 
evolve the language. If it had, these problems would not be 
around for so many years.
Aug 17 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 11:17 AM, bachmeier wrote:

 This is a good example of D needing to evolve or fizzle out. I don't see 
 evidence that the community has yet figured out how to evolve the language. If 
 it had, these problems would not be around for so many years.


We deprecate features of D all the time. (Remember the D1 => D2 wrenching
change?)

The reason  safe cannot be default at the moment it because -dip1000 needs
work, 
and nobody is willing to pitch in and review/pull my PRs on it.
Aug 17 2018
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/17/18 11:04 PM, Walter Bright wrote:

 On 8/17/2018 11:17 AM, bachmeier wrote:

 This is a good example of D needing to evolve or fizzle out. I don't 
 see evidence that the community has yet figured out how to evolve the 
 language. If it had, these problems would not be around for so many 
 years.


 
 We deprecate features of D all the time. (Remember the D1 => D2 
 wrenching change?)


Hm... if you are going for "all the time", the example of D1 to D2 
transition is pretty dated.

I'd say more like the addition of UDAs was a big evolution. Or maybe UFCS.


 The reason  safe cannot be default at the moment it because -dip1000 
 needs work, and nobody is willing to pitch in and review/pull my PRs on it.


I would, but I have no idea how dip1000 is supposed to work. I think 
only you understand it. Even looking at the PR that you have been citing 
over and over, I can't make heads or tails of what it does or what it 
allows.

-Steve
Aug 20 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, August 20, 2018 7:46:48 AM MDT Steven Schveighoffer via 
Digitalmars-d wrote:

 I would, but I have no idea how dip1000 is supposed to work. I think
 only you understand it. Even looking at the PR that you have been citing
 over and over, I can't make heads or tails of what it does or what it
 allows.


The basic idea of DIP 1000 seems simple enough, but once you get into the
details, it gets hard to understand fast.

And from what work I've done to make std.datetime work with it better, I'm
very much inclined to think that it's just too intrusive for too little
gain. I mean, it's great that you can know that a pointer isn't going to
escape, but once you have to start marking up member functions all over the
place just so that the type can work with scope? It just seems like it's too
viral. I honestly do not understand why much of anything in
std.datetime.systime needs scope, but I was forced to mark up SysTime all
over the place in order for scope variables of type SysTime to work just
because it contains a class reference - one which cannot posssibly ever be
on the stack, since you can't pass a scope argument to a member function to
have it then stored in a member variable. So, why it has any effect on why
scope is required on member functions, I have no idea.

User-defined types can't contain anything that's scope (since that would
mean that they escaped), and yet, they potentially have to be marked up with
scope all over the place in case someone qualifies a variable of that type
with scope somewhere. And of course, because of how stuff like ranges work,
you're quickly in trouble if they're marked with scope, since then they
can't be wrapped by other ranges, making them useless for pretty much
anything other than foreach. scope seems great for toy examples or stuff
that's extern(C), but once user-defined types get involved, it's a mess.

I've got partially completed changes to std.datetime.interval towards making
it work with scope better that I need to get back to, but to be honest, the
more I've tried to do anything with -dip1000, the less I've like it, and
it's really hard to be motivated to go to the effort of making anything
compatible with it.

- Jonathan M Davis
Aug 20 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/20/2018 6:46 AM, Steven Schveighoffer wrote:

 I would, but I have no idea how dip1000 is supposed to work. I think only you 
 understand it. Even looking at the PR that you have been citing over and over,
I 
 can't make heads or tails of what it does or what it allows.


The way to understand dip1000 is to forget about `this`, arrays, structs, 
classes, delegates, etc. Just think about pointers - functions that take 
pointers as arguments, and return pointers.

That simplifies things enormously.

Once that makes sense, then deconstruct the higher level constructs into 
pointers, and then you'll see how they work.

This is why my presentation does it all in terms of pointers:

     http://dconf.org/2017/talks/bright.html

If you're having trouble understanding a particular example, rewrite it in
terms 
of pointers. If it still is inscrutable, then ask about the pointer version
here.

(When someone gives me some complex thing and says "I don't understand scope 
here", I always first rewrite it in terms of pointers. It's the same thing I do 
with bug reports that use templates - I manually instantiate the templates
first.)
Aug 20 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 21 August 2018 at 00:09:03 UTC, Walter Bright wrote:

 On 8/20/2018 6:46 AM, Steven Schveighoffer wrote:

 I would, but I have no idea how dip1000 is supposed to work. I 
 think only you understand it. Even looking at the PR that you 
 have been citing over and over, I can't make heads or tails of 
 what it does or what it allows.


 The way to understand dip1000 is to forget about `this`, 
 arrays, structs, classes, delegates, etc. Just think about 
 pointers - functions that take pointers as arguments, and 
 return pointers.

 That simplifies things enormously.

 Once that makes sense, then deconstruct the higher level 
 constructs into pointers, and then you'll see how they work.

 This is why my presentation does it all in terms of pointers:

     http://dconf.org/2017/talks/bright.html

 If you're having trouble understanding a particular example, 
 rewrite it in terms of pointers. If it still is inscrutable, 
 then ask about the pointer version here.

 (When someone gives me some complex thing and says "I don't 
 understand scope here", I always first rewrite it in terms of 
 pointers. It's the same thing I do with bug reports that use 
 templates - I manually instantiate the templates first.)


The problem is that the code we write doesn't deal directly with 
pointers - see the recent confusion in this forum over where 
`scope` on the left applies to the `this` pointer or the one 
returned by the member function.

Kagamin just told me I needed to use `return` instead of `scope` 
to get things to work and I'm still not sure why.

Also: destructors? Always `scope`? Sometimes? I just add `scope` 
when the compiler complains at this point.

I think it's interesting that when I played with Rust I didn't 
have problems fighting the borrow checker at all. DIP1000 is 
supposed to have the same safety without the Rust complications 
but currently Rust is much easier to understand for me.

It doesn't help that the current implementation of -dip1000 
doesn't match the document it's supposedly based on.
Aug 21 2018
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/21/2018 7:31 AM, Atila Neves wrote:

 The problem is that the code we write doesn't deal directly with pointers -
see 
 the recent confusion in this forum over where `scope` on the left applies to
the 
 `this` pointer or the one returned by the member function.


That's what I was saying :-)

The way to deal with this is make a copy of the code, then rewrite it so it
does 
the exact same thing, but with pointers and refs only. No member functions, no 
delegates, no dynamic arrays. Then it is MUCH MUCH easier to see where the 
annotations go.



 Kagamin just told me I needed to use `return` instead of `scope` to get things 
 to work and I'm still not sure why.
 
 Also: destructors? Always `scope`? Sometimes? I just add `scope` when the 
 compiler complains at this point.
 
 I think it's interesting that when I played with Rust I didn't have problems 
 fighting the borrow checker at all. DIP1000 is supposed to have the same
safety 
 without the Rust complications but currently Rust is much easier to understand 
 for me.
 
 It doesn't help that the current implementation of -dip1000 doesn't match the 
 document it's supposedly based on.


All good points. But I cannot make any progress when nobody is willing to pull 
my PRs that improve the situation.

https://github.com/dlang/dmd/pull/8504

BTW, the annotations do not break things. The worst that will happen is the 
compiler will complain in  safe code that they are incorrect, and you'll need
to 
fix it or make it  trusted. The compiler is also pretty good about inferring
the 
correct annotations, at least for templates and lambdas, which helps enormously.

However, dip1000 not working with Phobos is a huge impediment to success, and
so 
pulling 8504 is critical.
Aug 21 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 21 August 2018 at 19:36:39 UTC, Walter Bright wrote:

 On 8/21/2018 7:31 AM, Atila Neves wrote:

 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or the 
 one returned by the member function.


 That's what I was saying :-)

 The way to deal with this is make a copy of the code, then 
 rewrite it so it does the exact same thing, but with pointers 
 and refs only. No member functions, no delegates, no dynamic 
 arrays. Then it is MUCH MUCH easier to see where the 
 annotations go.



Well, no. The syntax isn't the same for member functions. The 
examples from the actual DIP don't compile. There it says:

-------
scope can be applied to function return values (even though it is 
not a type qualifier). It must be applied to the left of the 
declaration, in the same way ref is:

scope int* foo();     // applies to return value
--------

Except:

-------
struct MyStruct { scope int* foo() scope; }

foo.d(1): Error: redundant attribute scope
-------

Meaning the first `scope` actually applies to `this`. Writing 
this out as a non-member function won't help me declare member 
functions!

I still don't know how to return a ref/pointer that's scoped. And 
I thought I'd written code that did that. Maybe I did. I'm very 
confused.



 BTW, the annotations do not break things. The worst that will 
 happen is the compiler will complain in  safe code that they 
 are incorrect, and you'll need to fix it or make it  trusted. 
 The compiler is also pretty good about inferring the correct 
 annotations, at least for templates and lambdas, which helps 
 enormously.


In my opinion, the worst that can happen is I successfully 
compile my  safe code with -dip1000 and the resulting binary 
isn't memory-safe, which is what's been happening to me.


 However, dip1000 not working with Phobos is a huge impediment 
 to success, and so pulling 8504 is critical.


I don't have merge rights. I took a look anyway and it mostly 
looks ok, but I'm not familiar enough with that part of the 
codebase.
Aug 21 2018
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Tuesday, 21 August 2018 at 21:17:25 UTC, Atila Neves wrote:


 I don't have merge rights. I took a look anyway and it mostly 
 looks ok, but I'm not familiar enough with that part of the 
 codebase.


It's not the implementation that's preventing it from being 
merged.  It's the idea itself, weak rationale, lack of peer 
review, lack of consideration for alternatives, and lack of 
documentation supporting it.  It reeks of "designed on a whim to 
quickly patch some oversight in the design DIP1000 while trying 
to get Phobos to compile with -dip1000".

With the proposed idea, order of parameters matters.  We'll need 
to establish a convention that return parameters must be declared 
first, which is opposite of 
https://dlang.org/phobos/std_algorithm_mutation.html#copy. Is 
that a good idea?  Maybe it is. We haven't vetted the design yet, 
so I'm not sure.  Why haven't we vetted the design?  Because 
there currently isn't one; there's just an informal 
back-of-the-napkin memo uploaded to a bugzilla entry.

The proposed idea wants to make the first parameter, if it's 
`ref`, special. Why not the first `ref` parameter regardless of 
whether it's the absolute first in the list.  Why not the last 
`ref` parameter?  Why not all `ref` parameters?

But what bothers me the most is I think it's missing the bigger 
picture:  D needs a way to annotate lifetimes.  Maybe `scope` and 
`return` with weird conditions based on the order of parameters 
and their attributes are the way to go.  Maybe there's another 
way that hasn't yet been considered.

Put together a thorough description of the proposal, justify it, 
ask the larger community for comment, vet it, and document it.  
At least that's what it's going to take to get me to take action 
on the PR.

Or maybe someone else is willing to just rubber stamp it in the 
interest of expediency.

Mike
Aug 21 2018
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin wrote:


 But what bothers me the most...


Something else that rubs me the wrong way is that DIP 1000 is 
currently in a status of `DRAFT`:  
https://github.com/dlang/DIPs/blob/master/DIPs/README.md

What the heck is going on here?  We're adding features to the 
compiler and modifying Phobos in production releases based on a 
`DRAFT` proposal?

Furthermore, I find it hypocritical that some of us are put 
through a disproportionately burdensome DIP process requiring 
thorough documentation, multiple peer reviews, excessive delays, 
and judgement that defaults to "no" for some of the most minute 
changes to the language, but a game-changing feature like DIP 
1000 can just be amended on a whim.

Mike
Aug 21 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, August 21, 2018 8:18:15 PM MDT Mike Franklin via Digitalmars-d 
wrote:

 On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin wrote:

 But what bothers me the most...


 Something else that rubs me the wrong way is that DIP 1000 is
 currently in a status of `DRAFT`:
 https://github.com/dlang/DIPs/blob/master/DIPs/README.md

 What the heck is going on here?  We're adding features to the
 compiler and modifying Phobos in production releases based on a
 `DRAFT` proposal?

 Furthermore, I find it hypocritical that some of us are put
 through a disproportionately burdensome DIP process requiring
 thorough documentation, multiple peer reviews, excessive delays,
 and judgement that defaults to "no" for some of the most minute
 changes to the language, but a game-changing feature like DIP
 1000 can just be amended on a whim.


The reality of the matter is that the DIP system is a formal way to propose
language changes in order to convince Walter and Andrei that those changes
should be implemented, whereas if Walter or Andrei writes the DIP, they're
already convinced. This isn't a democracy. Walter is the BDFL, and it's his
call. So, I really don't think that it's hypocritical, but I also do think
that DIP 1000 probably should have gone through more peer review. From what
I can tell, outside of the simple cases, pretty much everyone has a really
hard time understanding it. The situation will likely improve once Phobos
properly supports it, and you can more reasonably use it, but the whole
thing defnitely seems to be overly complicated given what it's supposed to
be doing and what benefits you get from it. Personally, I think that it
seems pretty reasonable as long as user-defined types don't get involved,
but once they do, it's a mess.

- Jonathan M Davis
Aug 21 2018
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 04:23:52 UTC, Jonathan M Davis 
wrote:


 The reality of the matter is that the DIP system is a formal 
 way to propose language changes in order to convince Walter and 
 Andrei that those changes should be implemented, whereas if 
 Walter or Andrei writes the DIP, they're already convinced. 
 This isn't a democracy. Walter is the BDFL, and it's his call. 
 So, I really don't think that it's hypocritical


Walter and Andrei need to have their ideas vetted by the 
community, not in an effort to convince anyone, but for quality 
assurance, to ensure they're not overlooking something.

It is hypocritical an arrogant to believe that only our ideas 
have flaws and require scrutiny.

Mike
Aug 21 2018
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 04:49:15 UTC, Mike Franklin wrote:


 It is hypocritical an arrogant to believe that only our ideas 
 have flaws and require scrutiny.


Sorry, that was poorly stated and conveyed the wrong intent.  It 
should read:

It is hypocritical an arrogant to believe that only our ideas 
should require thorough documentation and scrutiny.

Mike
Aug 21 2018
prev sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Wednesday, 22 August 2018 at 04:49:15 UTC, Mike Franklin wrote:

 On Wednesday, 22 August 2018 at 04:23:52 UTC, Jonathan M Davis 
 wrote:


 The reality of the matter is that the DIP system is a formal 
 way to propose language changes in order to convince Walter 
 and Andrei that those changes should be implemented, whereas 
 if Walter or Andrei writes the DIP, they're already convinced. 
 This isn't a democracy. Walter is the BDFL, and it's his call. 
 So, I really don't think that it's hypocritical


 Walter and Andrei need to have their ideas vetted by the 
 community, not in an effort to convince anyone, but for quality 
 assurance, to ensure they're not overlooking something.

 It is hypocritical an arrogant to believe that only our ideas 
 have flaws and require scrutiny.


The formal DIP process was put in place after DIP1000. I would 
even daresay that the process was put in place because of the 
issue with DIP1000 (the rigorously checked DIP's are all >1000 
for that reason).
Aug 22 2018
prev sibling next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote:


 Furthermore, I find it hypocritical that some of us are put 
 through a disproportionately burdensome DIP process requiring 
 thorough documentation, multiple peer reviews, excessive 
 delays, and judgement that defaults to "no" for some of the 
 most minute changes to the language, but a game-changing 
 feature like DIP 1000 can just be amended on a whim.


DIP 1000 is in a bit of limbo at the moment. When I took over the 
process from Mihails, he told me it was stalled and that the 
proposal did not match the implementation. So I haven't touched 
it, which is why it's still marked as Draft. At some point, 
Walter will revise it to match the implementation and then we'll 
discuss how to handle it.

Whatever the status of DIP 1000, I would point out that that one 
of Walter's DIPs is in Community Review right now after sitting 
in the PR queue in Draft Review for a while. Once this review 
stage is done, it will go back into the queue to await Final 
Review like any other DIP. The only difference between this and 
other DIPs is that there will be no Formal Assessment for it. And 
this is the second DIP from Walter that has gone through the 
process since I've come on board, with DIP 1008 being put on 
pause at his request.
Aug 21 2018
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 05:04:25 UTC, Mike Parker wrote:


 Whatever the status of DIP 1000, I would point out that that 
 one of Walter's DIPs is in Community Review right now after 
 sitting in the PR queue in Draft Review for a while. Once this 
 review stage is done, it will go back into the queue to await 
 Final Review like any other DIP. The only difference between 
 this and other DIPs is that there will be no Formal Assessment 
 for it. And this is the second DIP from Walter that has gone 
 through the process since I've come on board, with DIP 1008 
 being put on pause at his request.


I understand that Walter's DIPs have been put through the process 
just like the others, but with regard to the specific issue in 
this thread (https://issues.dlang.org/show_bug.cgi?id=19097), the 
accompanying PR (https://github.com/dlang/dmd/pull/8504), and 
similarly undocumented PR 
(https://github.com/dlang/dmd/pull/8408), they are amending DIP 
1000 and DIP 25 "under the table".  There is no accompanying PR 
to the specification, no formal rational, no RFC from the 
community, etc...

Yet I and others have to go through the DIP process for much less 
significant changes to the language, and rightly so:

https://github.com/dlang/dmd/pull/7310
https://github.com/dlang/dmd/pull/7079

All I want to see from Walter is:
1) a sufficiently documented proposal for his idea
2) an RFC from the community
3) a PR to the spec documenting the final design

I don't think that's too much to ask.

Mike
Aug 21 2018
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 05:39:05 UTC, Mike Franklin wrote:


 I understand that Walter's DIPs have been put through the 
 process just like the others, but with regard to the specific 
 issue in this thread 
 (https://issues.dlang.org/show_bug.cgi?id=19097), the 
 accompanying PR (https://github.com/dlang/dmd/pull/8504), and 
 similarly undocumented PR 
 (https://github.com/dlang/dmd/pull/8408), they are amending DIP 
 1000 and DIP 25 "under the table".  There is no accompanying PR 
 to the specification, no formal rational, no RFC from the 
 community, etc...


A 3rd example (https://github.com/dlang/dmd/pull/8346) to throw a 
little more salt on the wound.  The DIP25/1000 rabbit hole is 
deepening behind the curtain.

Mike
Aug 21 2018
prev sibling next sibling parent walker <growup_wei qq.com> writes:
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote:

 Something else that rubs me the wrong way is that DIP 1000 is 
 currently in a status of `DRAFT`:  
 https://github.com/dlang/DIPs/blob/master/DIPs/README.md

 What the heck is going on here?  We're adding features to the 
 compiler and modifying Phobos in production releases based on a 
 `DRAFT` proposal?

 Furthermore, I find it hypocritical that some of us are put 
 through a disproportionately burdensome DIP process requiring 
 thorough documentation, multiple peer reviews, excessive 
 delays, and judgment that defaults to "no" for some of the most 
 minute changes to the language, but a game-changing feature 
 like DIP 1000 can just be amended on a whim.


+1,
For valuable contributors like you, it's far from friendly.
Friendly atmosphere increases creativity.
Although, for an industrial-level language, it should be careful 
to add many features (A core engine that can simulate a lot of 
easy to use sugars is important, for those potential hackers. )
Aug 22 2018
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote:

 On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin 
 wrote:


 But what bothers me the most...


 Something else that rubs me the wrong way is that DIP 1000 is 
 currently in a status of `DRAFT`:  
 https://github.com/dlang/DIPs/blob/master/DIPs/README.md

 What the heck is going on here?  We're adding features to the 
 compiler and modifying Phobos in production releases based on a 
 `DRAFT` proposal?


No, it's behind a flag, so you can't really say that we're 
shipping it as "production ready release".
In fact I think we should have a hell of a lot more of such 
experimental flags.
This would allow us to be able to merge things quickly, and gain 
real-world feedback and testing on complicated matters instead of 
PRs stalling to death in the queue.

For reference, Rust has currently 148 opt-in experimental 
language features:

https://doc.rust-lang.org/unstable-book/index.html
Aug 22 2018
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 11:02:00 UTC, Seb wrote:


 No, it's behind a flag, so you can't really say that we're 
 shipping it as "production ready release".


The changes to Phobos are not behind a flag.  We're making 
changes to Phobos in the release branch to accommodate a 
draft/experimental/choose-your-adjective feature.

Mike
Aug 22 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/21/2018 6:07 PM, Mike Franklin wrote:

 The proposed idea wants to make the first parameter, if it's `ref`, special.


This is because Phobos is written with functions of the form:

     void put(sink, parameters...)

which corresponds to:

     sink.put(parameters...)

The two forms are fairly interchangeable, made more so by the Uniform Function 
Call Syntax.


 Why not the first `ref` parameter regardless of whether it's the absolute 


first in the list.  Why not the last `ref` parameter?  Why not all `ref`
parameters?

Good question. If this fairly restricted form solves the problems, then there
is 
no need for the more flexible form. Things can always be made more flexible in 
the future, but tightening things can be pretty disruptive. Hence, unless there 
is an obvious and fairly strong case case for the flexibility, then it should
be 
avoided for now.



 But what bothers me the most is I think it's missing the bigger picture:  D 
 needs a way to annotate lifetimes.  Maybe `scope` and `return` with weird 
 conditions based on the order of parameters and their attributes are the way
to 
 go.  Maybe there's another way that hasn't yet been considered.
 
 Put together a thorough description of the proposal, justify it, ask the
larger 
 community for comment, vet it, and document it. At least that's what it's
going 
 to take to get me to take action on the PR.


dip1000 has been around for two years, and its predecessor dip25 several years 
now. Plenty of time for anyone to comment and/or propose something better.

---

I want to ensure Atila is successful with this. But that means Phobos has to 
compile with dip1000. So I need to make it work.
Aug 22 2018
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote:


 The proposed idea wants to make the first parameter, if it's 
 `ref`, special.


 This is because Phobos is written with functions of the form:

     void put(sink, parameters...)

 which corresponds to:

     sink.put(parameters...)

 The two forms are fairly interchangeable, made more so by the 
 Uniform Function Call Syntax.


Makes perfect sense. This is the kind of stuff I'd like you to 
put in a formal document and present to us as an RFC along with 
your PR.  Then transfer that information to the spec to accompany 
the DMD PR, after you've received feedback.

Mike
Aug 22 2018
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote:


 I want to ensure Atila is successful with this. But that means 
 Phobos has to compile with dip1000. So I need to make it work.


There's a good chance you'll get your PR merged when it's 
documented and vetted, then you'll be able to make it work for 
Atila.  The only think holding it up is you.

I'm not asking for much.  Sufficiently document the idea, open it 
up for comment and Q & A, transfer the documentation to the spec 
to accompany the DMD implementation PR.

Assuming there's no major flaws in the design, it should get 
merged.

Mike

P.S. Actually, I've been trying to document it myself, since you 
don't seem willing to, but it's going to take me a lot longer to 
figure out what's in your head than it would take you.
Aug 22 2018
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote:


 dip1000 has been around for two years, and its predecessor 
 dip25 several years now. Plenty of time for anyone to comment 
 and/or propose something better.


Part of the problem is that the implementation keeps changing 
without keeping the documentation in sync.  For example you're 
implementing all of these inference rules without documenting 
them:

https://github.com/dlang/dmd/pull/8346
https://github.com/dlang/dmd/pull/8408

I asked you about, instead of inferring the attributes, allowing 
users to add such logic themselves:

void foo(T)(T x)
     if (__traits(isPointer, T)) { T = scope T; }
{ }

But, then the PR got rubber-stamped, and now here we are.

Mike
Aug 22 2018
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/22/18 5:23 AM, Walter Bright wrote:

 On 8/21/2018 6:07 PM, Mike Franklin wrote:

 The proposed idea wants to make the first parameter, if it's `ref`, 
 special.


 
 This is because Phobos is written with functions of the form:
 
      void put(sink, parameters...)
 
 which corresponds to:
 
      sink.put(parameters...)
 
 The two forms are fairly interchangeable, made more so by the Uniform 
 Function Call Syntax.




 
  > Why not the first `ref` parameter regardless of whether it's the 
 absolute first in the list.  Why not the last `ref` parameter?  Why not 
 all `ref` parameters?
 
 Good question. If this fairly restricted form solves the problems, then 
 there is no need for the more flexible form. Things can always be made 
 more flexible in the future, but tightening things can be pretty 
 disruptive. Hence, unless there is an obvious and fairly strong case 
 case for the flexibility, then it should be avoided for now.


What about:

size_t put(sink, parameters...)

Does this qualify as the sink being the "return" type? Obviously the 
real return can't contain any references, so it trivially can be ruled 
out as the destination of any escaping parameters.

Or how about a member function that takes a ref parameter? Is `this` the 
"return" or is the ref parameter the "return"?

My problem with the idea is that it is going to seem flaky -- we are 
using convention to dictate what is actually the return parameter, vs. 
what semantically happens inside the function. It's going to confuse 
anyone trying to do it a different way. I've experienced this in the 
past with things like toHash, where if you didn't define it with the 
exact signature, it wouldn't actually be used.

I realize obviously, that `put` is already specified. But as I said in 
the bug report, we should think twice about defining rules based solely 
on how Phobos does things, and calling that the solution.

As for things being made "more flexible in the future" this basically 
translates to code breakage. For example, if you are depending on only 
the first parameter being considered the "return" value, and all of a 
sudden it changes to encompass all your parameters, your existing code 
may fail to compile, even if it's correctly safe and properly annotated.


 
 I want to ensure Atila is successful with this. But that means Phobos 
 has to compile with dip1000. So I need to make it work.
 


I think it's a very worthy goal to make Phobos work, and a great proof 
of concept for dip1000's veracity.

However, one-off rules just to make it work with existing code go 
against that goal IMO. Rules that stand on their own I think will fare 
better than ones that are loopholes to allow existing code to compile.

-Steve
Aug 22 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/22/2018 6:50 AM, Steven Schveighoffer wrote:

 What about:
 
 size_t put(sink, parameters...)
 
 Does this qualify as the sink being the "return" type? Obviously the real
return 
 can't contain any references, so it trivially can be ruled out as the 
 destination of any escaping parameters.


Your reasoning is correct, but currently it only applies with 'void' return
types.



 Or how about a member function that takes a ref parameter? Is `this` the 
 "return" or is the ref parameter the "return"?


`this` is the ref parameter. In particular, consider a constructor:

   struct S {
      int* p;
      this(return scope int* p) { this.p = p; }
   }

   int i;
   S s = S(&i);

This code appears in Phobos, and it is very reasonable to expect it to check as 
safe.



 My problem with the idea is that it is going to seem flaky -- we are using 
 convention to dictate what is actually the return parameter, vs. what 
 semantically happens inside the function. It's going to confuse anyone trying
to 
 do it a different way. I've experienced this in the past with things like 
 toHash, where if you didn't define it with the exact signature, it wouldn't 
 actually be used.
 
 I realize obviously, that `put` is already specified. But as I said in the bug 
 report, we should think twice about defining rules based solely on how Phobos 
 does things, and calling that the solution.


Phobos doesn't do this by accident. It's how constructors work (see above) and 
how pipeline programming works.



 As for things being made "more flexible in the future" this basically
translates 
 to code breakage. For example, if you are depending on only the first
parameter 
 being considered the "return" value, and all of a sudden it changes to
encompass 
 all your parameters, your existing code may fail to compile, even if it's 
 correctly safe and properly annotated.


It's a good point. But I don't see an obvious use case for considering all the 
ref parameters as being returns.


 I want to ensure Atila is successful with this. But that means Phobos has to
 compile with dip1000. So I need to make it work.


 
 I think it's a very worthy goal to make Phobos work, and a great proof of 
 concept for dip1000's veracity.
 
 However, one-off rules just to make it work with existing code go against that 
 goal IMO. Rules that stand on their own I think will fare better than ones
that 
 are loopholes to allow existing code to compile.


I couldn't come up with a better idea than this, and this one works.
Aug 23 2018
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/23/18 4:58 AM, Walter Bright wrote:

 On 8/22/2018 6:50 AM, Steven Schveighoffer wrote:

 What about:

 size_t put(sink, parameters...)

 Does this qualify as the sink being the "return" type? Obviously the 
 real return can't contain any references, so it trivially can be ruled 
 out as the destination of any escaping parameters.


 
 Your reasoning is correct, but currently it only applies with 'void' 
 return types.
 
 

 Or how about a member function that takes a ref parameter? Is `this` 
 the "return" or is the ref parameter the "return"?


 
 `this` is the ref parameter. In particular, consider a constructor:
 
    struct S {
       int* p;
       this(return scope int* p) { this.p = p; }
    }
 
    int i;
    S s = S(&i);
 
 This code appears in Phobos, and it is very reasonable to expect it to 
 check as safe.


What I mean to say is, we have a semantic today -- the return value is 
hooked to any `return` parameters, end of story. This is clear, concise, 
and easy to understand.

You are saying that in some cases, the return value is actually 
deposited in the `this` parameter. In cases where the actual return type 
is void, OK, I see that we can tack on that rule without issues.

Furthermore any member function (or UFCS function for that matter) 
REQUIRES the first parameter to be the aggregate. How do you make a 
member function that stuffs the return into a different parameter 
properly typecheck? What rule do we tack on then? It's going to be 
confusing to anyone who writes their API thinking about how it's call 
syntax reads, not how the compiler wants to do flow analysis.

Not to mention, the keyword is `return`, not `returnorfirstparam`. It's 
still going to be confusing no matter how you look at it.


 My problem with the idea is that it is going to seem flaky -- we are 
 using convention to dictate what is actually the return parameter, vs. 
 what semantically happens inside the function. It's going to confuse 
 anyone trying to do it a different way. I've experienced this in the 
 past with things like toHash, where if you didn't define it with the 
 exact signature, it wouldn't actually be used.

 I realize obviously, that `put` is already specified. But as I said in 
 the bug report, we should think twice about defining rules based 
 solely on how Phobos does things, and calling that the solution.


 
 Phobos doesn't do this by accident. It's how constructors work (see 
 above) and how pipeline programming works.


Constructors I agree are reasonable to consider `this` to be the return 
value. On that point, I would say we should definitely go ahead with 
making that rule, and I think it will lead to no confusion whatsoever.

pipeline programming depends on returning something other than `void`, 
so I don't see how this applies.

It's more when you are setting members via properties where this comes 
into play. We need it -- we need this ability to tell the compiler "this 
parameter connects to this other parameter". I just don't know if the 
proposed rules are a) good enough for the general case, and b) don't 
cause more confusion than they are worth.


 As for things being made "more flexible in the future" this basically 
 translates to code breakage. For example, if you are depending on only 
 the first parameter being considered the "return" value, and all of a 
 sudden it changes to encompass all your parameters, your existing code 
 may fail to compile, even if it's correctly safe and properly annotated.


 
 It's a good point. But I don't see an obvious use case for considering 
 all the ref parameters as being returns.


You would have to consider the shortest liftetime and assume everything 
goes there. It would restrict your legitimate calls. Only mitigating 
factor may be if you take the ones you aren't going to modify as const 
or inout.


 I want to ensure Atila is successful with this. But that means 


 Phobos has to

 compile with dip1000. So I need to make it work.


 I think it's a very worthy goal to make Phobos work, and a great proof 
 of concept for dip1000's veracity.

 However, one-off rules just to make it work with existing code go 
 against that goal IMO. Rules that stand on their own I think will fare 
 better than ones that are loopholes to allow existing code to compile.


 
 I couldn't come up with a better idea than this, and this one works.
 


I want to stress that it may be a valid solution, but we should strive 
to prove the solutions are the best possible rather than just use 
duct-tape methodology.

It should even be considered that perhaps there are better solutions 
even than the approach dip1000 has taken.

I also want to point out that the attitude of 'we could just fix it, but 
nobody will pull my request' is unhelpful. We want to make sure we have 
the best solution possible, don't take criticism as meaningless petty 
bickering. People are genuinely trying to make sure D is improved. 
Hostility towards reviews or debate doesn't foster that.

-Steve
Aug 23 2018
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/23/18 9:32 AM, Steven Schveighoffer wrote:

 On 8/23/18 4:58 AM, Walter Bright wrote:

 On 8/22/2018 6:50 AM, Steven Schveighoffer wrote:

 As for things being made "more flexible in the future" this basically 
 translates to code breakage. For example, if you are depending on 
 only the first parameter being considered the "return" value, and all 
 of a sudden it changes to encompass all your parameters, your 
 existing code may fail to compile, even if it's correctly safe and 
 properly annotated.


 It's a good point. But I don't see an obvious use case for considering 
 all the ref parameters as being returns.


 
 You would have to consider the shortest liftetime and assume everything 
 goes there. It would restrict your legitimate calls. Only mitigating 
 factor may be if you take the ones you aren't going to modify as const 
 or inout.


Actually, thinking about this, the shortest lifetime is dictated by how 
it is called, so there is no valid way to determine which one makes 
sense when compiling the function.

In order for this to work, you'd have to attribute it somehow. I can see 
that is likely going to be way more cumbersome than it's worth.

If I had to design a specific way to allow the common case to be easy, 
but still provide a mechanism for the uncommon cases, I would say:

1. define a compiler-recognized attribute (e.g.  __sink).
2. If  __sink is applied to any parameter, that is effectively the 
return value.
3. In the absence of a  __sink designation on non-void-returning 
functions, it applies to the return value.
4. In the absence of a  __sink designation on void returning functions, 
it applies to the first parameter.
5. Inference of  __sink happens even on non-templates.
6. If  __sink is attributed on multiple parameters, you assume all 
return parameters are assigned to all  __sink parameters for the 
purposes of verifying lifetimes are not exceeded.

Ugly to specify, but might actually be pretty non-intrusive to use.

-Steve
Aug 23 2018
next sibling parent reply Chris M. <chrismohrfeld comcast.net> writes:
On Thursday, 23 August 2018 at 15:14:07 UTC, Steven Schveighoffer 
wrote:

 On 8/23/18 9:32 AM, Steven Schveighoffer wrote:

 [...]


 Actually, thinking about this, the shortest lifetime is 
 dictated by how it is called, so there is no valid way to 
 determine which one makes sense when compiling the function.

 In order for this to work, you'd have to attribute it somehow. 
 I can see that is likely going to be way more cumbersome than 
 it's worth.

 If I had to design a specific way to allow the common case to 
 be easy, but still provide a mechanism for the uncommon cases, 
 I would say:

 1. define a compiler-recognized attribute (e.g.  __sink).
 2. If  __sink is applied to any parameter, that is effectively 
 the return value.
 3. In the absence of a  __sink designation on 
 non-void-returning functions, it applies to the return value.
 4. In the absence of a  __sink designation on void returning 
 functions, it applies to the first parameter.
 5. Inference of  __sink happens even on non-templates.
 6. If  __sink is attributed on multiple parameters, you assume 
 all return parameters are assigned to all  __sink parameters 
 for the purposes of verifying lifetimes are not exceeded.

 Ugly to specify, but might actually be pretty non-intrusive to 
 use.

 -Steve


This is more a general reply to the thread.

If I think I'm getting a good grasp on the issue here, it seems 
like something Rust already solved with lifetime annotations. 
Could they or something similar be leveraged for D as well?

https://doc.rust-lang.org/1.9.0/book/lifetimes.html

Current solution just seems too specific and very restrictive.
Aug 23 2018
parent reply Chris M. <chrismohrfeld comcast.net> writes:
On Thursday, 23 August 2018 at 15:48:00 UTC, Chris M. wrote:

 On Thursday, 23 August 2018 at 15:14:07 UTC, Steven 
 Schveighoffer wrote:

 On 8/23/18 9:32 AM, Steven Schveighoffer wrote:

 [...]


 Actually, thinking about this, the shortest lifetime is 
 dictated by how it is called, so there is no valid way to 
 determine which one makes sense when compiling the function.

 In order for this to work, you'd have to attribute it somehow. 
 I can see that is likely going to be way more cumbersome than 
 it's worth.

 If I had to design a specific way to allow the common case to 
 be easy, but still provide a mechanism for the uncommon cases, 
 I would say:

 1. define a compiler-recognized attribute (e.g.  __sink).
 2. If  __sink is applied to any parameter, that is effectively 
 the return value.
 3. In the absence of a  __sink designation on 
 non-void-returning functions, it applies to the return value.
 4. In the absence of a  __sink designation on void returning 
 functions, it applies to the first parameter.
 5. Inference of  __sink happens even on non-templates.
 6. If  __sink is attributed on multiple parameters, you assume 
 all return parameters are assigned to all  __sink parameters 
 for the purposes of verifying lifetimes are not exceeded.

 Ugly to specify, but might actually be pretty non-intrusive to 
 use.

 -Steve


 This is more a general reply to the thread.

 If I think I'm getting a good grasp on the issue here, it seems 
 like something Rust already solved with lifetime annotations. 
 Could they or something similar be leveraged for D as well?

 https://doc.rust-lang.org/1.9.0/book/lifetimes.html

 Current solution just seems too specific and very restrictive.


Heck, now that I'm looking at it, DIP25 seems like a more 
restricted form of Rust's lifetimes. Let me know if I'm just 
completely wrong about this, but

 safe ref int identity(return ref int x) {
     return x; // fine
}

would basically be like (pseudosyntax)

 safe ref'a int identity(ref'a int x) {
     return x; // fine
}

Maybe the more sane thing would be a syntax that visually ties 
them together as above. Obviously we're looking at possibly 
breaking changes, but how widespread would they be?

void betty(ref'a scope int* r, scope'a int* p); // syntax is not 
so nice since I just arbitrarily stuck them on different 
keywords, but that's besides the point here
Aug 23 2018
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote:

 
 Heck, now that I'm looking at it, DIP25 seems like a more 
 restricted form of Rust's lifetimes. Let me know if I'm just 
 completely wrong about this, but
 [snip]


Check out DIP1000

https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
Aug 23 2018
parent Chris M. <chrismohrfeld comcast.net> writes:
On Thursday, 23 August 2018 at 23:58:00 UTC, jmh530 wrote:

 On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote:

 
 Heck, now that I'm looking at it, DIP25 seems like a more 
 restricted form of Rust's lifetimes. Let me know if I'm just 
 completely wrong about this, but
 [snip]


 Check out DIP1000

 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md


I've read through it already, I'm just throwing out an idea for 
what I think is a more flexible solution to Walter's current 
issue which seems to be stemming more from DIP25 rather than 
DIP1000. More effort but I think it'd be worth it in the long run.
Aug 23 2018
prev sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote:


 Heck, now that I'm looking at it, DIP25 seems like a more 
 restricted form of Rust's lifetimes. Let me know if I'm just 
 completely wrong about this, but



Lifetimes:

http://smallcultfollowing.com/babysteps/blog/2016/04/27/non-lexical-lifetimes-introduction/#problem-case-3-conditional-control-flow-across-functions

http://smallcultfollowing.com/babysteps/blog/2016/05/09/non-lexical-lifetimes-adding-the-outlives-relation/#problem-case-3-revisited


 would basically be like (pseudosyntax)

  safe ref'a int identity(ref'a int x) {
     return x; // fine
 }

 Maybe the more sane thing would be a syntax that visually ties 
 them together as above. Obviously we're looking at possibly 
 breaking changes, but how widespread would they be?

 void betty(ref'a scope int* r, scope'a int* p); // syntax is 
 not so nice since I just arbitrarily stuck them on different 
 keywords, but that's besides the point here


I wish I had been more involved in D when DIP 25 and DIP 1000 
were being proposed, as I don't think the designs were thoroughly 
vetted.  It's taken me at least a year to even begin getting a 
grasp on it.

I think DIP 25 and DIP 1000 should have been combined and thought 
of holistically as simply "annotated lifetimes in D" rather than 
separate things.  I think then it becomes easier to visualize 
what the problem is and see, potentially many, alternatives.

Given the investments that have already been made in DIP 25 and 
DIP 1000, it's going to take an extremely motivated individual to 
fight an uphill battle to change direction now, I'm afraid.  If 
working on D was my full-time job, I'd do it, but who in this 
community has such resources.

Mike
Aug 23 2018
parent reply Chris M. <chrismohrfeld comcast.net> writes:
On Friday, 24 August 2018 at 00:13:48 UTC, Mike Franklin wrote:

 On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote:


 Heck, now that I'm looking at it, DIP25 seems like a more 
 restricted form of Rust's lifetimes. Let me know if I'm just 
 completely wrong about this, but



 Non-Lexical Lifetimes:

 http://smallcultfollowing.com/babysteps/blog/2016/04/27/non-lexical-lifetimes-introduction/#problem-case-3-conditional-control-flow-across-functions

 http://smallcultfollowing.com/babysteps/blog/2016/05/09/non-lexical-lifetimes-adding-the-outlives-relation/#problem-case-3-revisited


Seems to be more of a warning of what issues we may face if 
DIP25/DIP1000 are finally implemented. It would be good to 
consider NLLs as well before D is committed. No point in 
repeating issues that have already been studied.


 would basically be like (pseudosyntax)

  safe ref'a int identity(ref'a int x) {
     return x; // fine
 }

 Maybe the more sane thing would be a syntax that visually ties 
 them together as above. Obviously we're looking at possibly 
 breaking changes, but how widespread would they be?

 void betty(ref'a scope int* r, scope'a int* p); // syntax is 
 not so nice since I just arbitrarily stuck them on different 
 keywords, but that's besides the point here


 I wish I had been more involved in D when DIP 25 and DIP 1000 
 were being proposed, as I don't think the designs were 
 thoroughly vetted.  It's taken me at least a year to even begin 
 getting a grasp on it.

 I think DIP 25 and DIP 1000 should have been combined and 
 thought of holistically as simply "annotated lifetimes in D" 
 rather than separate things.  I think then it becomes easier to 
 visualize what the problem is and see, potentially many, 
 alternatives.

 Given the investments that have already been made in DIP 25 and 
 DIP 1000, it's going to take an extremely motivated individual 
 to fight an uphill battle to change direction now, I'm afraid.  
 If working on D was my full-time job, I'd do it, but who in 
 this community has such resources.


I think the main problem is that these were created before the 
DIP process was fully fleshed out, so who knows how much vetting 
they got. At least they are still compiler switches and probably 
not widely used, so it's not like we're already too far gone. I 
do also agree they should be worked on in conjunction, the hard 
part is finding someone to take ownership and put in the extra 
effort.


 Mike
Aug 23 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2018 5:58 PM, Chris M. wrote:

 Seems to be more of a warning of what issues we may face if DIP25/DIP1000 are 
 finally implemented. It would be good to consider NLLs as well before D is 
 committed. No point in repeating issues that have already been studied.


DIP25 waqs "finally implemented" several years ago, and works well. DIP1000 was 
implemented as well, it works, but it didn't cover the case of returning
through 
a ref parameter.

There's no way to "thoroughly vet" them before implementing. It doesn't happen 
with C++, either, somebody builds an implementation and then people try it out.
Aug 24 2018
parent reply Chris M. <chrismohrfeld comcast.net> writes:
On Saturday, 25 August 2018 at 02:37:00 UTC, Walter Bright wrote:

 On 8/23/2018 5:58 PM, Chris M. wrote:

 Seems to be more of a warning of what issues we may face if 
 DIP25/DIP1000 are finally implemented. It would be good to 
 consider NLLs as well before D is committed. No point in 
 repeating issues that have already been studied.


 DIP25 waqs "finally implemented" several years ago, and works 
 well. DIP1000 was implemented as well, it works, but it didn't 
 cover the case of returning through a ref parameter.

 There's no way to "thoroughly vet" them before implementing. It 
 doesn't happen with C++, either, somebody builds an 
 implementation and then people try it out.


What about my other point then on the syntax? I think something 
similar to what I suggested would be a much more flexible 
solution and is worth considering.
Aug 25 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/25/2018 5:42 AM, Chris M. wrote:

 What about my other point then on the syntax? I think something similar to
what 
 I suggested would be a much more flexible solution and is worth considering.


Much more work would be needed to make that a proposal.
Aug 28 2018
parent aliak <something something.com> writes:
On Wednesday, 29 August 2018 at 05:01:14 UTC, Walter Bright wrote:

 On 8/25/2018 5:42 AM, Chris M. wrote:

 What about my other point then on the syntax? I think 
 something similar to what I suggested would be a much more 
 flexible solution and is worth considering.


 Much more work would be needed to make that a proposal.


Here's some more details that could help with considering it 
incase you didn't see it:

https://forum.dlang.org/post/uxloaftaptfbzhbhquez forum.dlang.org

Seems like it's a super set of return and the current proposal to 
(from what I can understand at least) make some parameters act 
like a sink in some situations when a function's return value is 
of a specific type and the parameters are annotated with some 
convention?

Cheers,
- Ali
Sep 03 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2018 8:14 AM, Steven Schveighoffer wrote:

 If I had to design a specific way to allow the common case to be easy, but
still 
 provide a mechanism for the uncommon cases, I would say:
 
 1. define a compiler-recognized attribute (e.g.  __sink).
 2. If  __sink is applied to any parameter, that is effectively the return
value.
 3. In the absence of a  __sink designation on non-void-returning functions, it 
 applies to the return value.
 4. In the absence of a  __sink designation on void returning functions, it 
 applies to the first parameter.
 5. Inference of  __sink happens even on non-templates.
 6. If  __sink is attributed on multiple parameters, you assume all return 
 parameters are assigned to all  __sink parameters for the purposes of
verifying 
 lifetimes are not exceeded.



'ref' is already  __sink.
Aug 24 2018
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/24/18 10:28 PM, Walter Bright wrote:

 On 8/23/2018 8:14 AM, Steven Schveighoffer wrote:

 If I had to design a specific way to allow the common case to be easy, 
 but still provide a mechanism for the uncommon cases, I would say:

 1. define a compiler-recognized attribute (e.g.  __sink).
 2. If  __sink is applied to any parameter, that is effectively the 
 return value.
 3. In the absence of a  __sink designation on non-void-returning 
 functions, it applies to the return value.
 4. In the absence of a  __sink designation on void returning 
 functions, it applies to the first parameter.
 5. Inference of  __sink happens even on non-templates.
 6. If  __sink is attributed on multiple parameters, you assume all 
 return parameters are assigned to all  __sink parameters for the 
 purposes of verifying lifetimes are not exceeded.


 
 
 'ref' is already  __sink.


No, otherwise we wouldn't need the patch you are pushing.

-Steve
Aug 28 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2018 6:32 AM, Steven Schveighoffer wrote:

 Furthermore any member function (or UFCS function for that matter) REQUIRES
the 
 first parameter to be the aggregate. How do you make a member function that 
 stuffs the return into a different parameter properly typecheck?


What I propose is that the function interface be refactored so it does fit into 
these patterns. Is that an unreasonable requirement? I don't know. But it 
doesn't seem to be, as I haven't run into it yet.



 Phobos doesn't do this by accident. It's how constructors work (see above) and 
 how pipeline programming works.


 
 Constructors I agree are reasonable to consider `this` to be the return value. 
 On that point, I would say we should definitely go ahead with making that
rule, 
 and I think it will lead to no confusion whatsoever.
 
 pipeline programming depends on returning something other than `void`, so I 
 don't see how this applies.


grep Phobos for instances of put() and see its signature. It's part of pipeline 
programming, and it's all over the place.



 You would have to consider the shortest liftetime and assume everything goes 
 there.


That's right.



 It would restrict your legitimate calls.


Maybe that's a good thing. Having multiple simultaneous routes of data out of a 
function is not good practice (note that it is impossible with functional 
programming). If you absolutely must have it, the exit routes can be aggregated 
into a struct, then pass that struct as the first argument.



 I want to stress that it may be a valid solution, but we should strive to
prove 
 the solutions are the best possible rather than just use duct-tape methodology.


I don't know how to prove anything with programming languages.



 It should even be considered that perhaps there are better solutions even than 
 the approach dip1000 has taken.


People have hypothesized that for several years, and so far none have been 
forthcoming beyond a few hand-wavy generalities.



 I also want to point out that the attitude of 'we could just fix it, but
nobody 
 will pull my request' is unhelpful. We want to make sure we have the best 
 solution possible, don't take criticism as meaningless petty bickering. People 
 are genuinely trying to make sure D is improved. Hostility towards reviews or 
 debate doesn't foster that.


I'm not hostile to debate. I just don't care for "this is uncharted territory, 
so let's do nothing" which has been going on for probably 4 years now, 
coincident with "scope is incomplete, D sux".

I.e. lead, follow, or get out of the way :-)
Aug 24 2018
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright wrote:

 I'm not hostile to debate. I just don't care for "this is 
 uncharted territory, so let's do nothing" which has been going 
 on for probably 4 years now, coincident with "scope is 
 incomplete, D sux".

 I.e. lead, follow, or get out of the way :-)


Document it and we will help you get it reviewed and merged. 
Until you do that we will stay in your way. Heck, once we 
understand it we may even lead, but we can't do that if we have 
no clue WTF its supposed to do or why the changes are being made.

At the very least PRs need detailed changelog entries and 
preferably spec updates and revisions to the DIP1000 proposal 
document. Heck even a wiki page would be useful. I'm trying to 
make the spec page on memory safe programming the authoritative 
source see https://github.com/dlang/dlang.org/pull/2453
Aug 25 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/25/2018 4:09 AM, Nicholas Wilson wrote:

 On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright wrote:

 I'm not hostile to debate. I just don't care for "this is uncharted territory, 
 so let's do nothing" which has been going on for probably 4 years now, 
 coincident with "scope is incomplete, D sux".

 I.e. lead, follow, or get out of the way :-)


 
 Document it


Already done: https://issues.dlang.org/show_bug.cgi?id=19097
Aug 28 2018
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 29 August 2018 at 05:04:22 UTC, Walter Bright wrote:

 On 8/25/2018 4:09 AM, Nicholas Wilson wrote:

 On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright 
 wrote:

 I'm not hostile to debate. I just don't care for "this is 
 uncharted territory, so let's do nothing" which has been 
 going on for probably 4 years now, coincident with "scope is 
 incomplete, D sux".

 I.e. lead, follow, or get out of the way :-)


 
 Document it


 Already done: https://issues.dlang.org/show_bug.cgi?id=19097


Bugzilla is not documentation. These are language changes they 
need to be in release notes and the spec.
Aug 28 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/28/2018 10:18 PM, Nicholas Wilson wrote:

 Bugzilla is not documentation. These are language changes they need to be in 
 release notes and the spec.


You asked for a clue: "we have no clue WTF its supposed to do or why the
changes 
are being made" and there it is. There are no barriers to reviewing the idea
nor 
the implementation.
Aug 29 2018
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 29 August 2018 at 08:35:30 UTC, Walter Bright wrote:

 On 8/28/2018 10:18 PM, Nicholas Wilson wrote:

 Bugzilla is not documentation. These are language changes they 
 need to be in release notes and the spec.


 You asked for a clue: "we have no clue WTF its supposed to do 
 or why the changes are being made" and there it is. There are 
 no barriers to reviewing the idea nor the implementation.


I don't know how much more blunt I can be while still being 
professional about it:
_this must be documented properly_. This is not just about us 
reviewing it, this is also about people using it.

Think about it this way: I'm a new user and I hear that D is 
supposed to be memory safe.

Am I going to trawl through bugzilla to find about the features 
of this memory safety?

Suppose I do a search for "dlang memory safety" on how I can use 
this feature. What does a search bring up?

First hit: https://dlang.org/spec/memory-safe-d.html
Gives me a link to https://dlang.org/articles/safed.html, not 
very helpful, and to
   system/ safe/ trusted, that tells me a bunch of things I'm not 
allowed to do in safe code.

Second hit: 
https://dlang.org/blog/2016/09/28/how-to-write-trusted-code-in-d/
More (admittedly better) info on  system/ safe/ trusted

Third Hit: 
https://forum.dlang.org/thread/ofkjuq$or6$1 digitalmars.com welp 
the forum is down, next!

Fourth hit: https://wiki.dlang.org/Memory_Management

Fifth hit: https://dconf.org/2017/talks/bright.pdf
Some stuff about scope, possibly not up to date(!).

Sixth hit is Adam Ruppe's fork of the spec page on memory safety

Seventh is https://news.ycombinator.com/item?id=12391370

Number 8: https://www.meetup.com/en-AU/SeaLang/events/246692611/

9: 
https://www.reddit.com/r/cpp/comments/6b4xrc/walter_bright_believes_memory_safety_will_kill_c/

10: is the Wikipdia page on D.

Searching for "dlang scope" brings up scope(exit) & friends and 
some old forum posts  including 
https://forum.dlang.org/thread/obfftm$2m3j$1 digitalmars.com 
which feels like déjà vu, followed by a bunch of irrelevant links 
to various parts of the spec.

That was on a not anonymous search, I know precisely what I'm 
looking for, and the closest thing I found was 4/5ths the way 
down a 42 slide PDF with no annotations from a year ago.

Even someone relatively familiar is going to look at the spec and 
the changelog, and they're not going to find anything BECAUSE ITS 
NOT THERE! You know where it is? Bugzilla, because that's where 
Walter thinks documentation should go.
Aug 29 2018
parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Wednesday, 29 August 2018 at 13:53:42 UTC, Nicholas Wilson 
wrote:

 On Wednesday, 29 August 2018 at 08:35:30 UTC, Walter Bright 
 wrote:

 On 8/28/2018 10:18 PM, Nicholas Wilson wrote:

 Bugzilla is not documentation. These are language changes 
 they need to be in release notes and the spec.


 You asked for a clue: "we have no clue WTF its supposed to do 
 or why the changes are being made" and there it is. There are 
 no barriers to reviewing the idea nor the implementation.


 I don't know how much more blunt I can be while still being 
 professional about it:
 _this must be documented properly_. This is not just about us 
 reviewing it, this is also about people using it.


I can confirm that, from a user's perspective, (the state of) 
DIP1000 is not clearly defined. I read the forums so I know 
Phobos is not completely compatible yet, but I don't know what 
that means to me. The documentation for -dip1000 [1] simply 
refers to the DIP text, and after reading that I decide I want 
it. Then, possibly a while after switching to it, I start getting 
(mangled) link errors. It is not obvious to me that these relate 
to -dip1000 (which means wasting some time in trouble shooting) 
but they disappear after I take away that option. I would have 
appreciated if [1] would have included or linked to information 
on the situations in which you can successfully use -dip1000, and 
what are the symptoms when you move outside that envelope (link 
errors, presumably). At the very least, [1] could/should stress 
that -dip switches enable experimental functionality and that 
certain things are known to not work.

I do seem to be able to use -dip25 without issues though.

[1] https://dlang.org/dmd-windows.html
Aug 30 2018
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/24/18 10:25 PM, Walter Bright wrote:

 On 8/23/2018 6:32 AM, Steven Schveighoffer wrote:

 Furthermore any member function (or UFCS function for that matter) 
 REQUIRES the first parameter to be the aggregate. How do you make a 
 member function that stuffs the return into a different parameter 
 properly typecheck?


 
 What I propose is that the function interface be refactored so it does 
 fit into these patterns. Is that an unreasonable requirement? I don't 
 know. But it doesn't seem to be, as I haven't run into it yet.


So this would mean a member function would have to be refactored into a 
different function with a different calling syntax. i.e:

x.foo(target);

would have to be refactored to:

target.foo(x);

or foo(target, x);

Aside from the adjustment in name that is necessary to make this read 
correctly, that may cause other problems (sometimes non-member functions 
aren't available if it's a template instantiation).


 Phobos doesn't do this by accident. It's how constructors work (see 
 above) and how pipeline programming works.


 Constructors I agree are reasonable to consider `this` to be the 
 return value. On that point, I would say we should definitely go ahead 
 with making that rule, and I think it will lead to no confusion 
 whatsoever.

 pipeline programming depends on returning something other than `void`, 
 so I don't see how this applies.


 
 grep Phobos for instances of put() and see its signature. It's part of 
 pipeline programming, and it's all over the place.


I stand partly corrected! Indeed you can put a void-returning function 
at the *end* of a pipeline call, I hadn't thought of that.

But in terms of put, strictly speaking, any call of some.pipeline.put(x) 
is wrong. It should be put(some.pipeline, x), to avoid issues with how 
put was designed.


 It would restrict your legitimate calls.


 
 Maybe that's a good thing. Having multiple simultaneous routes of data 
 out of a function is not good practice (note that it is impossible with 
 functional programming). If you absolutely must have it, the exit routes 
 can be aggregated into a struct, then pass that struct as the first 
 argument.


Maybe it's better to designate one sink, and have that be the result. I 
know that after inout was implemented, there were definitely cases where 
one wanted to have multiple inout routes (i.e. independent traces 
between multiple parameters for copying mutability). It may be the same 
for this, I don't know.


 I want to stress that it may be a valid solution, but we should strive 
 to prove the solutions are the best possible rather than just use 
 duct-tape methodology.


 
 I don't know how to prove anything with programming languages.


I don't mean prove like mathematical proof. I mean try to consider how 
this affects all cases instead of just the one case that will make 
phobos compile.

"show" is a better verb than "prove".


 It should even be considered that perhaps there are better solutions 
 even than the approach dip1000 has taken.


 
 People have hypothesized that for several years, and so far none have 
 been forthcoming beyond a few hand-wavy generalities.


I'm just saying if dip1000 cannot fix all the problems, that instead of 
adding weird exceptions, or the classic "you're just doing it wrong", 
maybe we should reconsider the approach.

Another case which was brought up and pretty much ignored was this one: 
https://forum.dlang.org/post/qkrdpmdqaxjadgvsojxr forum.dlang.org


 I also want to point out that the attitude of 'we could just fix it, 
 but nobody will pull my request' is unhelpful. We want to make sure we 
 have the best solution possible, don't take criticism as meaningless 
 petty bickering. People are genuinely trying to make sure D is 
 improved. Hostility towards reviews or debate doesn't foster that.


 
 I'm not hostile to debate. I just don't care for "this is uncharted 
 territory, so let's do nothing" which has been going on for probably 4 
 years now, coincident with "scope is incomplete, D sux".
 
 I.e. lead, follow, or get out of the way :-)


I'm opting for the latter, as the idea of band-aid PRs to get Phobos 
compiling with dip1000 just to see if dip1000 is going to work seems 
like the wrong approach to me.

The consequence of this is that getting out of the way means your PRs 
don't get pulled.

-Steve
Aug 28 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/28/2018 6:12 AM, Steven Schveighoffer wrote:

 So this would mean a member function would have to be refactored into a 
 different function with a different calling syntax. i.e:
 
 x.foo(target);
 
 would have to be refactored to:
 
 target.foo(x);
 
 or foo(target, x);


Maybe it should be anyway.



 But in terms of put, strictly speaking, any call of some.pipeline.put(x) is 
 wrong. It should be put(some.pipeline, x), to avoid issues with how put was 
 designed.


There are some of those in Phobos, too. It's covered by the proposed addition.



 I don't know how to prove anything with programming languages.


 I don't mean prove like mathematical proof. I mean try to consider how this 
 affects all cases instead of just the one case that will make phobos compile.
 
 "show" is a better verb than "prove".


I don't know how to find all cases, either, except by implementing it.
Aug 28 2018
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/21/2018 2:17 PM, Atila Neves wrote:

 Well, no. The syntax isn't the same for member functions. The examples from
the 
 actual DIP don't compile. There it says:
 
 -------
 scope can be applied to function return values (even though it is not a type 
 qualifier). It must be applied to the left of the declaration, in the same way 
 ref is:
 
 scope int* foo();     // applies to return value
 --------
 
 Except:
 
 -------
 struct MyStruct { scope int* foo() scope; }
 
 foo.d(1): Error: redundant attribute scope
 -------
 Meaning the first `scope` actually applies to `this`. Writing this out as a 
 non-member function won't help me declare member functions!
 
 I still don't know how to return a ref/pointer that's scoped. And I thought
I'd 
 written code that did that. Maybe I did. I'm very confused.


Here's how you make it work:

---
 safe:

struct MyStruct {
     ref int foo() return;
     int* bar() return;
}

ref int sun() {
     MyStruct s;
     return s.foo(); // returning s.foo() escapes a reference to local variable
s
}

int* moon() {
     MyStruct s;
     return s.bar(); // returning s.bar() escapes a reference to local variable
s
}
---
In effect, the 'return' on 'foo()' says:

     The return value of foo() contains the address of 'this', and if the
return 
escapes the scope of what 'this' is a ref to, then it's an error.
Aug 22 2018
prev sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:

 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or the 
 one returned by the member function.

 Kagamin just told me I needed to use `return` instead of 
 `scope` to get things to work and I'm still not sure why.


The way I think about it is if you have a function that takes a 
pointer, any pointer, and either returns it or a pointer derived 
from it (dereferencing or indexing) that argument must be marked 
`return`. In your case it was a pointer derived from `this` so 
`return` must be applied to `this`.
Aug 21 2018
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 22 August 2018 at 03:58:42 UTC, Nicholas Wilson 
wrote:

 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:

 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or the 
 one returned by the member function.

 Kagamin just told me I needed to use `return` instead of 
 `scope` to get things to work and I'm still not sure why.


 The way I think about it is if you have a function that takes a 
 pointer, any pointer, and either returns it or a pointer 
 derived from it (dereferencing or indexing) that argument must 
 be marked `return`. In your case it was a pointer derived from 
 `this` so `return` must be applied to `this`.


I guess my problem is that DIP1000 talks about returning scope 
values and they don't seem to actually exist in the 
implementation.
Aug 22 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/21/2018 8:58 PM, Nicholas Wilson wrote:

 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:

 The problem is that the code we write doesn't deal directly with pointers - 
 see the recent confusion in this forum over where `scope` on the left applies 
 to the `this` pointer or the one returned by the member function.

 Kagamin just told me I needed to use `return` instead of `scope` to get things 
 to work and I'm still not sure why.


 
 The way I think about it is if you have a function that takes a pointer, any 
 pointer, and either returns it or a pointer derived from it (dereferencing or 
 indexing) that argument must be marked `return`. In your case it was a pointer 
 derived from `this` so `return` must be applied to `this`.



Another way to think about it is this:

    S s;
    return &s;

We all know that is an error. The idea is to have a way to express that for:

     S s;
     return s.foo();

and:

     S s;
     return foo(&s);

so that the compiler knows that the return value of foo() is attached to the 
lifetime of s. Pretty much everything flows from that.
Aug 22 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright wrote:

 On 8/21/2018 8:58 PM, Nicholas Wilson wrote:

 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:

 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or 
 the one returned by the member function.

 Kagamin just told me I needed to use `return` instead of 
 `scope` to get things to work and I'm still not sure why.


 
 The way I think about it is if you have a function that takes 
 a pointer, any pointer, and either returns it or a pointer 
 derived from it (dereferencing or indexing) that argument must 
 be marked `return`. In your case it was a pointer derived from 
 `this` so `return` must be applied to `this`.



 Another way to think about it is this:

    S s;
    return &s;

 We all know that is an error. The idea is to have a way to 
 express that for:

     S s;
     return s.foo();

 and:

     S s;
     return foo(&s);

 so that the compiler knows that the return value of foo() is 
 attached to the lifetime of s. Pretty much everything flows 
 from that.


Would the guideline below be correct?

"Add scope to every non-template member function that isn't meant 
to escape this and add return to every non-template member 
function that returns all or part of `this` by pointer or ref if 
you want the compiler to check that nothing gets escaped in  safe 
code."
Aug 22 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/22/2018 3:52 AM, Atila Neves wrote:

 On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright wrote:

 On 8/21/2018 8:58 PM, Nicholas Wilson wrote:

 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:

 The problem is that the code we write doesn't deal directly with pointers - 
 see the recent confusion in this forum over where `scope` on the left 
 applies to the `this` pointer or the one returned by the member function.

 Kagamin just told me I needed to use `return` instead of `scope` to get 
 things to work and I'm still not sure why.


 The way I think about it is if you have a function that takes a pointer, any 
 pointer, and either returns it or a pointer derived from it (dereferencing or 
 indexing) that argument must be marked `return`. In your case it was a 
 pointer derived from `this` so `return` must be applied to `this`.



 Another way to think about it is this:

    S s;
    return &s;

 We all know that is an error. The idea is to have a way to express that for:

     S s;
     return s.foo();

 and:

     S s;
     return foo(&s);

 so that the compiler knows that the return value of foo() is attached to the 
 lifetime of s. Pretty much everything flows from that.


 
 Would the guideline below be correct?
 
 "Add scope to every non-template member function that isn't meant to escape
this 
 and add return to every non-template member function that returns all or part
of 
 `this` by pointer or ref if you want the compiler to check that nothing gets 
 escaped in  safe code."


Being a template doesn't make any difference, except that it will helpfully 
infer these things.

Also, since 'this' is passed by 'ref' to struct member functions, it cannot 
escape anyway with dip1000:

   struct S {
     int x;
      safe ref int foo() { return x; }
   }

   dmd test -dip1000
   test.d(4): Error: returning this.x escapes a reference to parameter this, 
perhaps annotate with return

`scope` is for pointers, `ref` does not need further annotation.
Aug 23 2018
next sibling parent ag0aep6g <anonymous example.com> writes:
On Thursday, 23 August 2018 at 08:48:15 UTC, Walter Bright wrote:

 Being a template doesn't make any difference, except that it 
 will helpfully infer these things.


Including the `return` attribute.


 Also, since 'this' is passed by 'ref' to struct member 
 functions, it cannot escape anyway with dip1000:

   struct S {
     int x;
      safe ref int foo() { return x; }
   }

   dmd test -dip1000
   test.d(4): Error: returning this.x escapes a reference to 
 parameter this, perhaps annotate with return


If we add the `return` attribute, we can escape a pointer to 
`this`. And since attributes are often inferred, it might come as 
a surprise to the programmer.

However, as far as I understand DIP 1000, it should make sure 
that the returned pointer won't outlive the struct instance. And 
I guess that's why it's ok to infer the `return` attribute?

That depends on DMD not over-estimating the lifetimes of things, 
though. Unfortunately, it does just that. When in doubt, DMD goes 
with infinite (e.g., `malloc`). I think that's a problem.

But I've already demonstrated my ignorance regarding DIP 1000 in 
a long, not very fruitful discussion with Atila on Bugzilla [1], 
so maybe I just don't get it.


[1] https://issues.dlang.org/show_bug.cgi?id=19183
Aug 23 2018
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 23 August 2018 at 08:48:15 UTC, Walter Bright wrote:

 On 8/22/2018 3:52 AM, Atila Neves wrote:

 On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright 
 wrote:

 On 8/21/2018 8:58 PM, Nicholas Wilson wrote:

 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves 
 wrote:

 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or 
 the one returned by the member function.

 Kagamin just told me I needed to use `return` instead of 
 `scope` to get things to work and I'm still not sure why.


 The way I think about it is if you have a function that 
 takes a pointer, any pointer, and either returns it or a 
 pointer derived from it (dereferencing or indexing) that 
 argument must be marked `return`. In your case it was a 
 pointer derived from `this` so `return` must be applied to 
 `this`.



 Another way to think about it is this:

    S s;
    return &s;

 We all know that is an error. The idea is to have a way to 
 express that for:

     S s;
     return s.foo();

 and:

     S s;
     return foo(&s);

 so that the compiler knows that the return value of foo() is 
 attached to the lifetime of s. Pretty much everything flows 
 from that.


 
 Would the guideline below be correct?
 
 "Add scope to every non-template member function that isn't 
 meant to escape this and add return to every non-template 
 member function that returns all or part of `this` by pointer 
 or ref if you want the compiler to check that nothing gets 
 escaped in  safe code."


 Being a template doesn't make any difference, except that it 
 will helpfully infer these things.


The reason I wrote "non-template" is precisely because attributes 
get inferred for templates and therefore it would be at best 
redundant to annotate.

What about the guideline being correct or not?


 Also, since 'this' is passed by 'ref' to struct member 
 functions, it cannot escape anyway with dip1000:

   struct S {
     int x;
      safe ref int foo() { return x; }
   }

   dmd test -dip1000
   test.d(4): Error: returning this.x escapes a reference to 
 parameter this, perhaps annotate with return


Returning by ref is good, since it's a lot harder to escape it. 
Since variables can't be declared ref, I can't pass it to a ref 
global. However:

----------
struct S {
     int x;
      safe int* foo() { return &x; }
}
----------

% dmd -o- -dip1000 foo.d
% echo $?
0

Oops:

----------
int* gPtr;
void main() {
     auto s = S(42);
     gPtr = s.foo;
}
----------

"Don't use pointers then!". Ok:

----------
int[] gSlice;
void main() {
     auto s = S([42]);
     gSlice = s.foo;
}

struct S {
     int[] x;
      safe int[] foo() return { return x; }
}
----------

% dmd -o- -dip1000 bar.d
% echo $?
0

Oops. I can't say `int[] x;` since it doesn't apply to fields.


 `scope` is for pointers, `ref` does not need further annotation.


I always forget this, it's confusing. It doesn't help that slices 
have pointers, so I guess `scope` is for them too?

And in a struct, `this` is a `ref`, yet `scope` on a member 
function applies to `this`.

Atila
Aug 23 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2018 6:10 AM, Atila Neves wrote:

 ----------
 struct S {
      int x;
       safe int* foo() { return &x; }
 }
 ----------
 
 % dmd -o- -dip1000 foo.d
 % echo $?
 0


struct S {
     int x;
      safe int* foo() { return &x; }
}

int* bar() {
     S s;
     return s.foo();
}

dmd test -dip1000
test.d(3): Error: returning &this.x escapes a reference to parameter this, 
perhaps annotate with return



 
 Oops:
 
 ----------
 int* gPtr;
 void main() {
      auto s = S(42);
      gPtr = s.foo;
 }
 ----------


I get:

test.d(3): Error: returning &this.x escapes a reference to parameter this, 
perhaps annotate with return



 ----------
 int[] gSlice;
 void main() {
      auto s = S([42]);
      gSlice = s.foo;
 }
 
 struct S {
      int[] x;
       safe int[] foo() return { return x; }
 }
 ----------


T[] is treated like T* as far as scope, etc., are concerned. You should see 
identical results.


 And in a struct, `this` is a `ref`, yet `scope` on a member function applies
to 
 `this`.


Actually, it applies to fields of `this`.
Aug 28 2018
prev sibling parent reply Maksim Fomin <mxfm protonmail.com> writes:
On Friday, 17 August 2018 at 07:19:25 UTC, Peter Alexander wrote:

 My question is: what is the status of  safe? I am quite 
 surprised to see such a simple case fail. Is  safe believed to 
 be fully implemented (modulo bugs) and this is just an 
 unfortunate corner case, or is it known work-in-progress?


Years ago (2012-2013 
https://forum.dlang.org/post/xibbzslaunogifsomapl forum.dlang.org) I was
arguing that  safe is crippled: 1) there is high probability that some
loopholes are missing (several of them were posted to bugzilla and still not
fixed since then) 2) fixing loopholes will make using  safe inconvenient.

My point is somewhat different from expressed already here by 
Jonathan. He speaks about whitelisting/blacklisting and that the 
former model will always contain loopholes. In my view, the 
reason is that memory safety in (essentially good old C) memory 
model depends on runtime memory type which is unrelated to static 
type. Taking random variable - it can be allocated on stack, 
heap, GC-heap, thread-local and there is no way at CT to 
determine this in general case. In other words, static type rules 
is bad approach to determine memory safety. It can be used to 
detect some obvious bugs, but does not work across compilation 
boundaries. The better approach in my view is to insert runtime
Aug 30 2018
parent Peter Alexander <peter.alexander.au gmail.com> writes:
On Thursday, 30 August 2018 at 16:57:05 UTC, Maksim Fomin wrote:

 My point is somewhat different from expressed already here by 
 Jonathan. He speaks about whitelisting/blacklisting and that 
 the former model will always contain loopholes. In my view, the 
 reason is that memory safety in (essentially good old C) memory 
 model depends on runtime memory type which is unrelated to 
 static type. Taking random variable - it can be allocated on 
 stack, heap, GC-heap, thread-local and there is no way at CT to 
 determine this in general case. In other words, static type 
 rules is bad approach to determine memory safety. It can be 
 used to detect some obvious bugs, but does not work across 
 compilation boundaries. The better approach in my view is to 
 insert runtime code which performs some tests (at least this 



I think it is understood that any compile time checking must be 
conservative in the general case to be sound. The same is true 
for compile time type checking, which must reject type safe code 
that can only be determined type safe at runtime.

Adding runtime checks does sound reasonable, and D already does 
this for array bounds.
Aug 30 2018