www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Is safe still a work-in-progress?

reply Peter Alexander <peter.alexander.au gmail.com> writes:
--------
import std.algorithm, std.stdio;

 safe:

auto foo() {
   int[6] xs = [0, 1, 2, 3, 4, 5];
   return xs[].map!(x => x);
}

void main() {
   writeln(foo());
}
--------

https://run.dlang.io/is/qC7HUR

For me this gives:

--------
[0, 0, -2132056872, 22008, 0, 0]
--------

Which looks like its just reading arbitrary memory.

I've filed https://issues.dlang.org/show_bug.cgi?id=19175

My question is: what is the status of  safe? I am quite surprised 
to see such a simple case fail. Is  safe believed to be fully 
implemented (modulo bugs) and this is just an unfortunate corner 
case, or is it known work-in-progress?
Aug 17 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via Digitalmars-d 
wrote:
 --------
 import std.algorithm, std.stdio;

  safe:

 auto foo() {
    int[6] xs = [0, 1, 2, 3, 4, 5];
    return xs[].map!(x => x);
 }

 void main() {
    writeln(foo());
 }
 --------

 https://run.dlang.io/is/qC7HUR

 For me this gives:

 --------
 [0, 0, -2132056872, 22008, 0, 0]
 --------

 Which looks like its just reading arbitrary memory.

 I've filed https://issues.dlang.org/show_bug.cgi?id=19175

 My question is: what is the status of  safe? I am quite surprised
 to see such a simple case fail. Is  safe believed to be fully
 implemented (modulo bugs) and this is just an unfortunate corner
 case, or is it known work-in-progress?
That particular bug is a duplicate of https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed based on the fact that -dip1000 fixes the problem by treating marking the slice of a static array with scope. It's still quite broken without -dip1000 though. Honestly, the reality of the matter is that safe is probably always going to be somewhat broken, because it's implemented via blacklisting rather than whitelisting. Instead of safe only allowing stuff that's been proven to be safe, it disallows stuff that a programmer decided was system. The bug you ran into is a pretty glaring one that arguably should have been fixed ages ago, but given how hard it is to prove what is and isn't safe, there are bound to be corner cases which have been missed. As we find them, they'll be fixed, but who knows how many are left or whether we'll ever actually get them all. - Jonathan M Davis
Aug 17 2018
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote:

 That particular bug is a duplicate of 
 https://issues.dlang.org/show_bug.cgi?id=8838, which was closed 
 as fixed based on the fact that -dip1000 fixes the problem by 
 treating marking the slice of a static array with scope. It's 
 still quite broken without -dip1000 though.
It still appears to be broke even with -dip1000: https://run.dlang.io/is/gJi2Fa
Aug 17 2018
prev sibling next sibling parent reply vit <vit vit.vit> writes:
On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote:
 On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via 
 Digitalmars-d wrote:
 [...]
That particular bug is a duplicate of https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed based on the fact that -dip1000 fixes the problem by treating marking the slice of a static array with scope. It's still quite broken without -dip1000 though. [...]
What's the state of -dip1000?
Aug 17 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, August 17, 2018 2:23:20 AM MDT vit via Digitalmars-d wrote:
 On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote:
 On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via

 Digitalmars-d wrote:
 [...]
That particular bug is a duplicate of https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed based on the fact that -dip1000 fixes the problem by treating marking the slice of a static array with scope. It's still quite broken without -dip1000 though. [...]
What's the state of -dip1000?
Well, I _think_ that aside from bugs (which may or may not be a small number), the compiler implementation is complete. However, there's still a fair bit of work to do in Phobos to make it fully work with -dip1000. And AFAIK, we still don't have a real plan as to how we're going to make -dip1000 the default. But given how much -dip1000 seems to break, I don't know how we're going to switch to it being the default without being highly disruptive in the process. - Jonathan M Davis
Aug 17 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 3:01 AM, Jonathan M Davis wrote:
 we still don't have a real plan as to how we're going to make
 -dip1000 the default.
We do have a plan. Get Phobos to compile with -dip1000. As to why that hasn't happened yet, see: https://github.com/dlang/dmd/pull/8504
Aug 17 2018
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 17 August 2018 at 08:23:20 UTC, vit wrote:
 On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis 
 wrote:
 On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via 
 Digitalmars-d wrote:
 [...]
That particular bug is a duplicate of https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed based on the fact that -dip1000 fixes the problem by treating marking the slice of a static array with scope. It's still quite broken without -dip1000 though. [...]
What's the state of -dip1000?
I've been using -dip1000 a lot lately. I hit two bugs yesterday. When it works, it's great, _except_: . safe isn't default . -dip1000 isn't default . Good luck figuring out why your template functions aren't safe and coaxing the compiler to tell you why it's inferring the attributes it is. Bonus points if it's a Phobos function so you can't slap ` safe` on its definition. Take excel-d for instance. The other day somebody filed a bug because Excel was crashing when they wrapped some D code that uses std.regex. After investigating I found out that regex memoizes the string with the regular expression in it, effectively escaping the string passed from Excel to the D function. Unfortunately for everyone involved, that string came from a region allocator, and after bumping the pointer back, boom. Telling people to not escape reference parameters doesn't really work, because nobody will read that part of the README, and, as in the example above, the user didn't even know they were doing it! Ok, so as a good library author, I tried to make it a compile-time error to shoot yourself in the foot that way, and I did. Great, right? Well, no. I can't force the user to write safe functions, much less to compile with -dip1000. I changed the example XLL to use -dip1000 because more likely than not users will copy the example and edit it. But I can't force them to write safe functions, and if they don't (the likeliest scenario given it's not the default), they'll be able to escape whatever it is they want, Excel will crash, and the compiler will be ok with it. And if they _do_ write safe functions and compile with -dip1000, chances are at some point they won't understand the compiler error message, make their function trusted and... crash. I don't know that because I have a crystal ball, I know that because I did it myself this week. Twice. Did I mention the bugs? I wrote about my interesting experiences from this week with -dip1000 here: https://forum.dlang.org/post/lxavagebcmaxjdzisyfk forum.dlang.org
Aug 17 2018
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 17/08/2018 11:33 PM, Atila Neves wrote:
 . Good luck figuring out why your template functions aren't  safe and 
 coaxing the compiler to tell you why it's inferring the attributes it 
 is. Bonus points if it's a Phobos function so you can't slap ` safe` on 
 its definition.
Sure you can. It's a template so it'll be initialized as its required. I've done that to fix a bug in druntime before and in my own libraries.
Aug 17 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote:
 On 17/08/2018 11:33 PM, Atila Neves wrote:
 . Good luck figuring out why your template functions aren't 
  safe and coaxing the compiler to tell you why it's inferring 
 the attributes it is. Bonus points if it's a Phobos function 
 so you can't slap ` safe` on its definition.
Sure you can. It's a template so it'll be initialized as its required. I've done that to fix a bug in druntime before and in my own libraries.
It's not easy though. You have to either be building your own phobos or sudo editing files in /usr/include. You can, but it's a pain. Then there's finding out exactly where in a chain of 10 template functions that it became system...
Aug 17 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via Digitalmars-d 
wrote:
 On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote:
 On 17/08/2018 11:33 PM, Atila Neves wrote:
 . Good luck figuring out why your template functions aren't
  safe and coaxing the compiler to tell you why it's inferring
 the attributes it is. Bonus points if it's a Phobos function
 so you can't slap ` safe` on its definition.
Sure you can. It's a template so it'll be initialized as its required. I've done that to fix a bug in druntime before and in my own libraries.
It's not easy though. You have to either be building your own phobos or sudo editing files in /usr/include. You can, but it's a pain. Then there's finding out exactly where in a chain of 10 template functions that it became system...
The reality of the matter is that you tend to have to be very motivated to make your code safe if you're doing much with templates (the same with other attributes such as pure). And sadly, it can quickly get to the point that it just feels like it's not worth it in spite of the fact that there is definitely value to be had in using them. I wonder what it would take to make it so that the compiler could actually tell you where in the function call chain it first becomes system. - Jonathan M Davis
Aug 17 2018
parent reply vit <vit vit.vit> writes:
On Friday, 17 August 2018 at 13:02:20 UTC, Jonathan M Davis wrote:
 On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via 
 Digitalmars-d wrote:
 On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole 
 wrote:
 On 17/08/2018 11:33 PM, Atila Neves wrote:
 . Good luck figuring out why your template functions aren't 
  safe and coaxing the compiler to tell you why it's 
 inferring the attributes it is. Bonus points if it's a 
 Phobos function so you can't slap ` safe` on its definition.
Sure you can. It's a template so it'll be initialized as its required. I've done that to fix a bug in druntime before and in my own libraries.
It's not easy though. You have to either be building your own phobos or sudo editing files in /usr/include. You can, but it's a pain. Then there's finding out exactly where in a chain of 10 template functions that it became system...
The reality of the matter is that you tend to have to be very motivated to make your code safe if you're doing much with templates (the same with other attributes such as pure). And sadly, it can quickly get to the point that it just feels like it's not worth it in spite of the fact that there is definitely value to be had in using them. I wonder what it would take to make it so that the compiler could actually tell you where in the function call chain it first becomes system. - Jonathan M Davis
Does -dip1000 affect alias template parameters and closures? example: import std.experimental.all; void main() safe nogc{ int i = 2; const x = iota(0, 10) .filter!((x)scope => x == i) .map!((x)scope => x * x) .front; assert(x == 4); } onlineapp.d(4): Error: function `D main` is nogc yet allocates closures with the GC onlineapp.d(8): onlineapp.main.__lambda1 closes over variable i at onlineapp.d(5)
Aug 17 2018
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 18/08/2018 1:29 AM, vit wrote:
 On Friday, 17 August 2018 at 13:02:20 UTC, Jonathan M Davis wrote:
 On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via 
 Digitalmars-d wrote:
 On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote:
 On 17/08/2018 11:33 PM, Atila Neves wrote:
 . Good luck figuring out why your template functions aren't >> 
safe and coaxing the compiler to tell you why it's >> inferring the attributes it is. Bonus points if it's a >> Phobos function so you can't slap ` safe` on its definition.
 Sure you can. It's a template so it'll be initialized as its > 
required.
 I've done that to fix a bug in druntime before and in my own > 
libraries. It's not easy though. You have to either be building your own phobos or sudo editing files in /usr/include. You can, but it's a pain. Then there's finding out exactly where in a chain of 10 template functions that it became system...
The reality of the matter is that you tend to have to be very motivated to make your code safe if you're doing much with templates (the same with other attributes such as pure). And sadly, it can quickly get to the point that it just feels like it's not worth it in spite of the fact that there is definitely value to be had in using them. I wonder what it would take to make it so that the compiler could actually tell you where in the function call chain it first becomes system. - Jonathan M Davis
Does -dip1000 affect alias template parameters and closures? example: import std.experimental.all; void main() safe nogc{     int i = 2;     const x = iota(0, 10)         .filter!((x)scope => x == i)         .map!((x)scope => x * x)         .front;     assert(x == 4); } onlineapp.d(4): Error: function `D main` is nogc yet allocates closures with the GC onlineapp.d(8):        onlineapp.main.__lambda1 closes over variable i at onlineapp.d(5)
No it shouldn't. You used i there which automatically requires a heap allocation for a delegate. Hence it uses GC.
Aug 17 2018
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 5:30 AM, Atila Neves wrote:
 It's not easy though. You have to either be building your own phobos
Not really. You only have to be able to import phobos files. You don't have to compile something that will link. I.e. just copy the phobos source tree, add safe to it, and point the compiler at that one.
 or sudo 
 editing files in /usr/include. You can, but it's a pain. Then there's finding 
 out exactly where in a chain of 10 template functions that it became  system...
I recently fixed the compiler to say which template overload was used. It's a huge time saver. But you're currently doomed on this anyway, as an awful lot of Phobos winds up calling put(), which cannot be inferred as safe without: https://github.com/dlang/dmd/pull/8504
Aug 17 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 4:33 AM, Atila Neves wrote:
 I've been using -dip1000 a lot lately.
Great news!
 I hit two bugs yesterday.
In D? or in your code?
 When it works, it's great, _except_:
 
 .  safe isn't default
 . -dip1000 isn't default
 . Good luck figuring out why your template functions aren't  safe and coaxing 
 the compiler to tell you why it's inferring the attributes it is.
I empathize with that, having run into the same problem. A couple months ago I improved the resulting error messages, which make it a lot easier to track down why it is inferring what it does.
 Bonus points 
 if it's a Phobos function so you can't slap ` safe` on its definition.
Sometimes I think it is better to just slap them with safe and to hell with unsafe parameters. :-)
 Did I mention the bugs?
Bugs don't count if they're not in bugzilla!
Aug 17 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 18 August 2018 at 02:17:01 UTC, Walter Bright wrote:
 On 8/17/2018 4:33 AM, Atila Neves wrote:
 I've been using -dip1000 a lot lately.
Great news!
:)
 I hit two bugs yesterday.
In D? or in your code?
In dmd.
 Bugs don't count if they're not in bugzilla!
As mentioned in my other thread, I went to file one and it was declared not a bug: https://issues.dlang.org/show_bug.cgi?id=17927 I don't know how that's possible given that supposedly safe code is doing pretty bad things. That disheartened me enough to not continue although I did file this one: https://issues.dlang.org/show_bug.cgi?id=19173
Aug 20 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/20/2018 6:08 AM, Atila Neves wrote:
 As mentioned in my other thread, I went to file one and it was declared not a
bug:
 
 https://issues.dlang.org/show_bug.cgi?id=17927
I'll look at it again.
Aug 20 2018
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 12:50 AM, Jonathan M Davis wrote:
 That particular bug is a duplicate of
 https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed
 based on the fact that -dip1000 fixes the problem by treating marking the
 slice of a static array with scope. It's still quite broken without -dip1000
 though.
D will never be safe until -dip1000 is the default. However, I cannot get any traction with improving this: https://github.com/dlang/dmd/pull/8504 blocks progress on getting Phobos to compile with -dip1000, and nobody will help me with it.
 Honestly, the reality of the matter is that  safe is probably always going
 to be somewhat broken, because it's implemented via blacklisting rather than
 whitelisting. Instead of  safe only allowing stuff that's been proven to be
  safe, it disallows stuff that a programmer decided was  system. The bug you
 ran into is a pretty glaring one that arguably should have been fixed ages
 ago,
It was fixed with -dip1000 ages ago.
 but given how hard it is to prove what is and isn't  safe, there are
 bound to be corner cases which have been missed. As we find them, they'll be
 fixed, but who knows how many are left or whether we'll ever actually get
 them all.
The whitelisting idea has come up before. I see it, though, as a way to avoid dealing with the issues as nobody is willing to do any work at all on this approach. Furthermore, I see no evidence that whitelisting will produce superior results. On the other hand, I see plenty of evidence that safe bugs that are found and posted to bugzilla get fixed. If people want to see safe be watertight, 1. post any problems to bugzilla, and tag them with the 'safe' keyword. 2. investigate solutions to any issues tagged with 'safe' keyword. 3. review/approve PR 8504.
Aug 17 2018
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 17, 2018 at 01:50:32AM -0600, Jonathan M Davis via Digitalmars-d
wrote:
[...]
 Honestly, the reality of the matter is that  safe is probably always
 going to be somewhat broken, because it's implemented via blacklisting
 rather than whitelisting. Instead of  safe only allowing stuff that's
 been proven to be  safe, it disallows stuff that a programmer decided
 was  system.
Sigh: https://issues.dlang.org/show_bug.cgi?id=12941 This was reported 4 years ago, but was unfortunately closed as invalid. It will continue to be a problem as long as safe is implemented via blacklisting, because every single time there's a new language feature, there's a chance that a loophole is introduced into safe. And that's not counting the combinatorial explosion of existing language features that might lead to safe loopholes, that we simply haven't thought of yet. It's like allowing anyone to enter your house freely except those few people whom you've explicitly named. You can't possibly expect *not* to get robbed that way.
 The bug you ran into is a pretty glaring one that arguably should have
 been fixed ages ago, but given how hard it is to prove what is and
 isn't  safe, there are bound to be corner cases which have been
 missed. As we find them, they'll be fixed, but who knows how many are
 left or whether we'll ever actually get them all.
[...] And that is exactly why the whole implementation of safe is currently rather laughable. By blacklisting rather than whitelisting, we basically open the door wide open to loopholes -- anything that we haven't thought of yet could potentially be a safe-breaking combination, and we wouldn't know until somebody discovers and reports it. Sadly, it seems there is little interest in reimplementing safe to use whitelisting instead of blacklisting. T -- A program should be written to model the concepts of the task it performs rather than the physical world or a process because this maximizes the potential for it to be applied to tasks that are conceptually similar and, more important, to tasks that have not yet been conceived. -- Michael B. Allen
Aug 17 2018
next sibling parent David Gileadi <gileadisNOSPM gmail.com> writes:
On 8/17/18 7:26 AM, H. S. Teoh wrote:
 It will continue to be a problem as long as  safe is implemented via
 blacklisting, because every single time there's a new language feature,
 there's a chance that a loophole is introduced into  safe. And that's
 not counting the combinatorial explosion of existing language features
 that might lead to  safe loopholes, that we simply haven't thought of
 yet.  It's like allowing anyone to enter your house freely except those
 few people whom you've explicitly named. You can't possibly expect *not*
 to get robbed that way.
We should tag bugs like these as #safecracking
Aug 17 2018
prev sibling next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote:
 On Fri, Aug 17, 2018 at 01:50:32AM -0600, Jonathan M Davis via 
 Digitalmars-d wrote: [...]
 Honestly, the reality of the matter is that  safe is probably 
 always going to be somewhat broken, because it's implemented 
 via blacklisting rather than whitelisting. Instead of  safe 
 only allowing stuff that's been proven to be  safe, it 
 disallows stuff that a programmer decided was  system.
Sigh: https://issues.dlang.org/show_bug.cgi?id=12941 This was reported 4 years ago, but was unfortunately closed as invalid. It will continue to be a problem as long as safe is implemented via blacklisting, because every single time there's a new language feature, there's a chance that a loophole is introduced into safe. And that's not counting the combinatorial explosion of existing language features that might lead to safe loopholes, that we simply haven't thought of yet. It's like allowing anyone to enter your house freely except those few people whom you've explicitly named. You can't possibly expect *not* to get robbed that way.
I knew there was something fundamentally wrong with safe, but I could never put my finger on it. Now that you and Jonathan mention this, it becomes clear. This makes me exceptionally sad. D is great in so many ways, but then this taints the pool. I asked if D was ever going to be safe by default at DConf (https://youtu.be/HvqsUO77FGI?t=13242), but it didn't elicit a very positive answer. It seems D is backtracking in some ways ( nogc, -betterC), trying to evolve it into something it wasn't originally envisioned to be, and now we have another one to add to the list.
 The bug you ran into is a pretty glaring one that arguably 
 should have been fixed ages ago, but given how hard it is to 
 prove what is and isn't  safe, there are bound to be corner 
 cases which have been missed. As we find them, they'll be 
 fixed, but who knows how many are left or whether we'll ever 
 actually get them all.
[...] And that is exactly why the whole implementation of safe is currently rather laughable. By blacklisting rather than whitelisting, we basically open the door wide open to loopholes -- anything that we haven't thought of yet could potentially be a safe-breaking combination, and we wouldn't know until somebody discovers and reports it. Sadly, it seems there is little interest in reimplementing safe to use whitelisting instead of blacklisting.
I think there is probably some interest, though maybe not from the ones with the position or ability to make it happen. A DIP might be the way forward, but it seems like quite a difficult task to turn it right-side-up at this point. I actually started writing a DIP for this about a year ago, but I need to pick my battles. Mike
Aug 17 2018
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Friday, 17 August 2018 at 15:27:22 UTC, Mike Franklin wrote:
 I actually started writing a DIP for this about a year ago, but 
 I need to pick my battles.

 Mike
Is it on github? Alex
Aug 17 2018
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 17 August 2018 at 16:00:26 UTC, 12345swordy wrote:
 On Friday, 17 August 2018 at 15:27:22 UTC, Mike Franklin wrote:
 I actually started writing a DIP for this about a year ago, 
 but I need to pick my battles.

 Mike
Is it on github? Alex
No,but here are some notes I found in my files. safe by default Before Transition Write a program that parses D code and explicitly adds system if no function is not decorated with any safety attribute Add a transitional -explicit-safety compiler switch that emits a warning if a safety attribute is not explicitly specified. Update all dlang repositories and maybe others with explicit attributes using aforementioned program. Turn on -explicit-safety for all CIs so any new PRs so explicit safety is added to all new functions. Add transitional system-by-default compiler switch for backward compatibility This will be especially useful to keep our CIs running with legacy code This will also be useful for programs with dub dependencies that need to continue running Add - system-main compiler switch for backward compatibility Add - safe-main and - safe-by-default compiler switch for forward-looking D projects Update all of D's tutorials, examples, etc… with explicit safety attributes. Transition Add changelog entry warning that safe-by-default is coming and how they can use the aforementioned tools eliminate disruption Run this change log for 1 year (4 releases)? After Transition Remove safe attributes in D's example code. Remove safe attributes from phobos/druntime/dmd etc.. Turn off -explicit-safety for those libraries that have updated their code. Deprecate safe-main and safe-by-default compiler switch through the normal deprecation process. -explicit-safety, system-main and system-by-default remain for an indeterminate amount of time. They should eventually be deprecated, but only after they have outlived their usefulness. Difficulties The difficulty will be with printed material (books, etc…) that will become out-of-date. Recommendation is to add information to their published errata, or create a new edition. Mike
Aug 17 2018
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote:
 [...]

 And that is exactly why the whole implementation of  safe is 
 currently rather laughable. By blacklisting rather than 
 whitelisting, we basically open the door wide open to loopholes 
 -- anything that we haven't thought of yet could potentially be 
 a  safe-breaking combination, and we wouldn't know until 
 somebody discovers and reports it.

 Sadly, it seems there is little interest in reimplementing 
  safe to use whitelisting instead of blacklisting.


 T
Fundamentally, I see it as a good idea. Walter has talked about how important memory safety is for D. People thinking their safe code is safe is a big problem when that turns out to not be the case. Imagine the black eye D would have if a company was hacked because of something like this? IMO, the problem is that you can't just replace safe as it is now. You could introduce something like whitelist or safewhitelist and begin implementing it, but it would probably be some time before it could replace safe. Like when whitelist is only breaking unsafe code.
Aug 17 2018
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/17/18 1:26 PM, jmh530 wrote:
 On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote:
 [...]

 And that is exactly why the whole implementation of  safe is currently 
 rather laughable. By blacklisting rather than whitelisting, we 
 basically open the door wide open to loopholes -- anything that we 
 haven't thought of yet could potentially be a  safe-breaking 
 combination, and we wouldn't know until somebody discovers and reports 
 it.

 Sadly, it seems there is little interest in reimplementing  safe to 
 use whitelisting instead of blacklisting.


 T
Fundamentally, I see it as a good idea. Walter has talked about how important memory safety is for D. People thinking their safe code is safe is a big problem when that turns out to not be the case. Imagine the black eye D would have if a company was hacked because of something like this?
This will always be a possibility thanks to trusted.
 IMO, the problem is that you can't just replace  safe as it is now. You 
 could introduce something like  whitelist or  safewhitelist and begin 
 implementing it, but it would probably be some time before it could 
 replace  safe. Like when  whitelist is only breaking unsafe code.
I have to say, I don't see how this all helps. In theory, black-listing and white-listing will get you to the same position. Mechanisms to get or use pointers aren't really being added to the language any more, so the set of "things" to "list" either black or white is finite. In this thread, we are talking about something that should have been black-listed LONG ago, but was not because it was "too useful" (i.e. would break too much code). If safe code was white-listed, nobody would use it until it was finished, so it would be theoretical anyway. Nobody wants a feature that is safe, but not useful. However, a bigger problem is that we have a bug that is "fixed" (slicing static arrays) but only if you use a feature that doesn't work correctly (dip1000). Why? I think the bug should be reopened until dip1000 is the default, or it simply gets fixed (i.e. without requiring dip1000). -Steve
Aug 17 2018
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 17, 2018 at 05:26:48PM +0000, jmh530 via Digitalmars-d wrote:
 On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote:
 [...]
 And that is exactly why the whole implementation of  safe is
 currently rather laughable. By blacklisting rather than
 whitelisting, we basically open the door wide open to loopholes --
 anything that we haven't thought of yet could potentially be a
  safe-breaking combination, and we wouldn't know until somebody
 discovers and reports it.
[...]
 Fundamentally, I see it as a good idea. Walter has talked about how
 important memory safety is for D. People thinking their  safe code is
 safe is a big problem when that turns out to not be the case. Imagine
 the black eye D would have if a company was hacked because of
 something like this?
Indeed.
 IMO, the problem is that you can't just replace  safe as it is now.
 You could introduce something like  whitelist or  safewhitelist and
 begin implementing it, but it would probably be some time before it
 could replace  safe. Like when  whitelist is only breaking unsafe
 code.
The way I envision it is that the implementation would begin as a separate topic branch in the dmd repo, and gradually brought up to the point where it begins passing the testsuite. The good thing is that nowadays with our CI system put in place, any breakages caused in major D projects like vibe.d would be detected by the autotester, so we can immediately fill in any obvious missing whitelist items. Then we'd merge it into master but controlled by a command-line switch, say -dipxxxx if we put it under a DIP, and put out for 1 or 2 releases. People who depend on safe can then opt in to try it out, and report any bugs (missing whitelist items) that they find. We fix those bugs until no more bugs are reported. Then we make -dipxxxx the default, perhaps with a reverse switch for reverting to the old safe implementation just in case some projects out there depend on it but haven't gotten around to reporting bugs in the new implementation or whatever. After another release or so, we finally remove the switch and delete the old implementation. Then going forward, we will have eliminated safe loopholes pretty much completely (the only exception being codegen bugs), and the only likely bugs are missing items in the whitelist. Which should be much more manageable and much less disastrous -- the worst that can happen is that somebody tried to do something legal and got rejected by a big fat compiler error. As opposed to today's situation, where somebody tries something illegal and the compiler fails to detect it, and nobody even know there's a problem until it becomes an exploitable bug that causes a security problem. Given today's security climate, this would be utterly disastrous for D's reputation. T -- It is of the new things that men tire --- of fashions and proposals and improvements and change. It is the old things that startle and intoxicate. It is the old things that are young. -- G.K. Chesterton
Aug 17 2018
parent reply bachmeier <no spam.net> writes:
On Friday, 17 August 2018 at 17:59:45 UTC, H. S. Teoh wrote:

 As opposed to today's situation, where somebody tries something 
 illegal and the compiler fails to detect it, and nobody even 
 know there's a problem until it becomes an exploitable bug that 
 causes a security problem.  Given today's security climate, 
 this would be utterly disastrous for D's reputation.
This is a good example of D needing to evolve or fizzle out. I don't see evidence that the community has yet figured out how to evolve the language. If it had, these problems would not be around for so many years.
Aug 17 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/17/2018 11:17 AM, bachmeier wrote:
 This is a good example of D needing to evolve or fizzle out. I don't see 
 evidence that the community has yet figured out how to evolve the language. If 
 it had, these problems would not be around for so many years.
We deprecate features of D all the time. (Remember the D1 => D2 wrenching change?) The reason safe cannot be default at the moment it because -dip1000 needs work, and nobody is willing to pitch in and review/pull my PRs on it.
Aug 17 2018
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/17/18 11:04 PM, Walter Bright wrote:
 On 8/17/2018 11:17 AM, bachmeier wrote:
 This is a good example of D needing to evolve or fizzle out. I don't 
 see evidence that the community has yet figured out how to evolve the 
 language. If it had, these problems would not be around for so many 
 years.
We deprecate features of D all the time. (Remember the D1 => D2 wrenching change?)
Hm... if you are going for "all the time", the example of D1 to D2 transition is pretty dated. I'd say more like the addition of UDAs was a big evolution. Or maybe UFCS.
 The reason  safe cannot be default at the moment it because -dip1000 
 needs work, and nobody is willing to pitch in and review/pull my PRs on it.
I would, but I have no idea how dip1000 is supposed to work. I think only you understand it. Even looking at the PR that you have been citing over and over, I can't make heads or tails of what it does or what it allows. -Steve
Aug 20 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, August 20, 2018 7:46:48 AM MDT Steven Schveighoffer via 
Digitalmars-d wrote:
 I would, but I have no idea how dip1000 is supposed to work. I think
 only you understand it. Even looking at the PR that you have been citing
 over and over, I can't make heads or tails of what it does or what it
 allows.
The basic idea of DIP 1000 seems simple enough, but once you get into the details, it gets hard to understand fast. And from what work I've done to make std.datetime work with it better, I'm very much inclined to think that it's just too intrusive for too little gain. I mean, it's great that you can know that a pointer isn't going to escape, but once you have to start marking up member functions all over the place just so that the type can work with scope? It just seems like it's too viral. I honestly do not understand why much of anything in std.datetime.systime needs scope, but I was forced to mark up SysTime all over the place in order for scope variables of type SysTime to work just because it contains a class reference - one which cannot posssibly ever be on the stack, since you can't pass a scope argument to a member function to have it then stored in a member variable. So, why it has any effect on why scope is required on member functions, I have no idea. User-defined types can't contain anything that's scope (since that would mean that they escaped), and yet, they potentially have to be marked up with scope all over the place in case someone qualifies a variable of that type with scope somewhere. And of course, because of how stuff like ranges work, you're quickly in trouble if they're marked with scope, since then they can't be wrapped by other ranges, making them useless for pretty much anything other than foreach. scope seems great for toy examples or stuff that's extern(C), but once user-defined types get involved, it's a mess. I've got partially completed changes to std.datetime.interval towards making it work with scope better that I need to get back to, but to be honest, the more I've tried to do anything with -dip1000, the less I've like it, and it's really hard to be motivated to go to the effort of making anything compatible with it. - Jonathan M Davis
Aug 20 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/20/2018 6:46 AM, Steven Schveighoffer wrote:
 I would, but I have no idea how dip1000 is supposed to work. I think only you 
 understand it. Even looking at the PR that you have been citing over and over,
I 
 can't make heads or tails of what it does or what it allows.
The way to understand dip1000 is to forget about `this`, arrays, structs, classes, delegates, etc. Just think about pointers - functions that take pointers as arguments, and return pointers. That simplifies things enormously. Once that makes sense, then deconstruct the higher level constructs into pointers, and then you'll see how they work. This is why my presentation does it all in terms of pointers: http://dconf.org/2017/talks/bright.html If you're having trouble understanding a particular example, rewrite it in terms of pointers. If it still is inscrutable, then ask about the pointer version here. (When someone gives me some complex thing and says "I don't understand scope here", I always first rewrite it in terms of pointers. It's the same thing I do with bug reports that use templates - I manually instantiate the templates first.)
Aug 20 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 21 August 2018 at 00:09:03 UTC, Walter Bright wrote:
 On 8/20/2018 6:46 AM, Steven Schveighoffer wrote:
 I would, but I have no idea how dip1000 is supposed to work. I 
 think only you understand it. Even looking at the PR that you 
 have been citing over and over, I can't make heads or tails of 
 what it does or what it allows.
The way to understand dip1000 is to forget about `this`, arrays, structs, classes, delegates, etc. Just think about pointers - functions that take pointers as arguments, and return pointers. That simplifies things enormously. Once that makes sense, then deconstruct the higher level constructs into pointers, and then you'll see how they work. This is why my presentation does it all in terms of pointers: http://dconf.org/2017/talks/bright.html If you're having trouble understanding a particular example, rewrite it in terms of pointers. If it still is inscrutable, then ask about the pointer version here. (When someone gives me some complex thing and says "I don't understand scope here", I always first rewrite it in terms of pointers. It's the same thing I do with bug reports that use templates - I manually instantiate the templates first.)
The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. Also: destructors? Always `scope`? Sometimes? I just add `scope` when the compiler complains at this point. I think it's interesting that when I played with Rust I didn't have problems fighting the borrow checker at all. DIP1000 is supposed to have the same safety without the Rust complications but currently Rust is much easier to understand for me. It doesn't help that the current implementation of -dip1000 doesn't match the document it's supposedly based on.
Aug 21 2018
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/21/2018 7:31 AM, Atila Neves wrote:
 The problem is that the code we write doesn't deal directly with pointers -
see 
 the recent confusion in this forum over where `scope` on the left applies to
the 
 `this` pointer or the one returned by the member function.
That's what I was saying :-) The way to deal with this is make a copy of the code, then rewrite it so it does the exact same thing, but with pointers and refs only. No member functions, no delegates, no dynamic arrays. Then it is MUCH MUCH easier to see where the annotations go.
 Kagamin just told me I needed to use `return` instead of `scope` to get things 
 to work and I'm still not sure why.
 
 Also: destructors? Always `scope`? Sometimes? I just add `scope` when the 
 compiler complains at this point.
 
 I think it's interesting that when I played with Rust I didn't have problems 
 fighting the borrow checker at all. DIP1000 is supposed to have the same
safety 
 without the Rust complications but currently Rust is much easier to understand 
 for me.
 
 It doesn't help that the current implementation of -dip1000 doesn't match the 
 document it's supposedly based on.
All good points. But I cannot make any progress when nobody is willing to pull my PRs that improve the situation. https://github.com/dlang/dmd/pull/8504 BTW, the annotations do not break things. The worst that will happen is the compiler will complain in safe code that they are incorrect, and you'll need to fix it or make it trusted. The compiler is also pretty good about inferring the correct annotations, at least for templates and lambdas, which helps enormously. However, dip1000 not working with Phobos is a huge impediment to success, and so pulling 8504 is critical.
Aug 21 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 21 August 2018 at 19:36:39 UTC, Walter Bright wrote:
 On 8/21/2018 7:31 AM, Atila Neves wrote:
 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or the 
 one returned by the member function.
That's what I was saying :-) The way to deal with this is make a copy of the code, then rewrite it so it does the exact same thing, but with pointers and refs only. No member functions, no delegates, no dynamic arrays. Then it is MUCH MUCH easier to see where the annotations go.
Well, no. The syntax isn't the same for member functions. The examples from the actual DIP don't compile. There it says: ------- scope can be applied to function return values (even though it is not a type qualifier). It must be applied to the left of the declaration, in the same way ref is: scope int* foo(); // applies to return value -------- Except: ------- struct MyStruct { scope int* foo() scope; } foo.d(1): Error: redundant attribute scope ------- Meaning the first `scope` actually applies to `this`. Writing this out as a non-member function won't help me declare member functions! I still don't know how to return a ref/pointer that's scoped. And I thought I'd written code that did that. Maybe I did. I'm very confused.
 BTW, the annotations do not break things. The worst that will 
 happen is the compiler will complain in  safe code that they 
 are incorrect, and you'll need to fix it or make it  trusted. 
 The compiler is also pretty good about inferring the correct 
 annotations, at least for templates and lambdas, which helps 
 enormously.
In my opinion, the worst that can happen is I successfully compile my safe code with -dip1000 and the resulting binary isn't memory-safe, which is what's been happening to me.
 However, dip1000 not working with Phobos is a huge impediment 
 to success, and so pulling 8504 is critical.
I don't have merge rights. I took a look anyway and it mostly looks ok, but I'm not familiar enough with that part of the codebase.
Aug 21 2018
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Tuesday, 21 August 2018 at 21:17:25 UTC, Atila Neves wrote:

 I don't have merge rights. I took a look anyway and it mostly 
 looks ok, but I'm not familiar enough with that part of the 
 codebase.
It's not the implementation that's preventing it from being merged. It's the idea itself, weak rationale, lack of peer review, lack of consideration for alternatives, and lack of documentation supporting it. It reeks of "designed on a whim to quickly patch some oversight in the design DIP1000 while trying to get Phobos to compile with -dip1000". With the proposed idea, order of parameters matters. We'll need to establish a convention that return parameters must be declared first, which is opposite of https://dlang.org/phobos/std_algorithm_mutation.html#copy. Is that a good idea? Maybe it is. We haven't vetted the design yet, so I'm not sure. Why haven't we vetted the design? Because there currently isn't one; there's just an informal back-of-the-napkin memo uploaded to a bugzilla entry. The proposed idea wants to make the first parameter, if it's `ref`, special. Why not the first `ref` parameter regardless of whether it's the absolute first in the list. Why not the last `ref` parameter? Why not all `ref` parameters? But what bothers me the most is I think it's missing the bigger picture: D needs a way to annotate lifetimes. Maybe `scope` and `return` with weird conditions based on the order of parameters and their attributes are the way to go. Maybe there's another way that hasn't yet been considered. Put together a thorough description of the proposal, justify it, ask the larger community for comment, vet it, and document it. At least that's what it's going to take to get me to take action on the PR. Or maybe someone else is willing to just rubber stamp it in the interest of expediency. Mike
Aug 21 2018
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin wrote:

 But what bothers me the most...
Something else that rubs me the wrong way is that DIP 1000 is currently in a status of `DRAFT`: https://github.com/dlang/DIPs/blob/master/DIPs/README.md What the heck is going on here? We're adding features to the compiler and modifying Phobos in production releases based on a `DRAFT` proposal? Furthermore, I find it hypocritical that some of us are put through a disproportionately burdensome DIP process requiring thorough documentation, multiple peer reviews, excessive delays, and judgement that defaults to "no" for some of the most minute changes to the language, but a game-changing feature like DIP 1000 can just be amended on a whim. Mike
Aug 21 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, August 21, 2018 8:18:15 PM MDT Mike Franklin via Digitalmars-d 
wrote:
 On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin wrote:
 But what bothers me the most...
Something else that rubs me the wrong way is that DIP 1000 is currently in a status of `DRAFT`: https://github.com/dlang/DIPs/blob/master/DIPs/README.md What the heck is going on here? We're adding features to the compiler and modifying Phobos in production releases based on a `DRAFT` proposal? Furthermore, I find it hypocritical that some of us are put through a disproportionately burdensome DIP process requiring thorough documentation, multiple peer reviews, excessive delays, and judgement that defaults to "no" for some of the most minute changes to the language, but a game-changing feature like DIP 1000 can just be amended on a whim.
The reality of the matter is that the DIP system is a formal way to propose language changes in order to convince Walter and Andrei that those changes should be implemented, whereas if Walter or Andrei writes the DIP, they're already convinced. This isn't a democracy. Walter is the BDFL, and it's his call. So, I really don't think that it's hypocritical, but I also do think that DIP 1000 probably should have gone through more peer review. From what I can tell, outside of the simple cases, pretty much everyone has a really hard time understanding it. The situation will likely improve once Phobos properly supports it, and you can more reasonably use it, but the whole thing defnitely seems to be overly complicated given what it's supposed to be doing and what benefits you get from it. Personally, I think that it seems pretty reasonable as long as user-defined types don't get involved, but once they do, it's a mess. - Jonathan M Davis
Aug 21 2018
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 04:23:52 UTC, Jonathan M Davis 
wrote:

 The reality of the matter is that the DIP system is a formal 
 way to propose language changes in order to convince Walter and 
 Andrei that those changes should be implemented, whereas if 
 Walter or Andrei writes the DIP, they're already convinced. 
 This isn't a democracy. Walter is the BDFL, and it's his call. 
 So, I really don't think that it's hypocritical
Walter and Andrei need to have their ideas vetted by the community, not in an effort to convince anyone, but for quality assurance, to ensure they're not overlooking something. It is hypocritical an arrogant to believe that only our ideas have flaws and require scrutiny. Mike
Aug 21 2018
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 04:49:15 UTC, Mike Franklin wrote:

 It is hypocritical an arrogant to believe that only our ideas 
 have flaws and require scrutiny.
Sorry, that was poorly stated and conveyed the wrong intent. It should read: It is hypocritical an arrogant to believe that only our ideas should require thorough documentation and scrutiny. Mike
Aug 21 2018
prev sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Wednesday, 22 August 2018 at 04:49:15 UTC, Mike Franklin wrote:
 On Wednesday, 22 August 2018 at 04:23:52 UTC, Jonathan M Davis 
 wrote:

 The reality of the matter is that the DIP system is a formal 
 way to propose language changes in order to convince Walter 
 and Andrei that those changes should be implemented, whereas 
 if Walter or Andrei writes the DIP, they're already convinced. 
 This isn't a democracy. Walter is the BDFL, and it's his call. 
 So, I really don't think that it's hypocritical
Walter and Andrei need to have their ideas vetted by the community, not in an effort to convince anyone, but for quality assurance, to ensure they're not overlooking something. It is hypocritical an arrogant to believe that only our ideas have flaws and require scrutiny.
The formal DIP process was put in place after DIP1000. I would even daresay that the process was put in place because of the issue with DIP1000 (the rigorously checked DIP's are all >1000 for that reason).
Aug 22 2018
prev sibling next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote:

 Furthermore, I find it hypocritical that some of us are put 
 through a disproportionately burdensome DIP process requiring 
 thorough documentation, multiple peer reviews, excessive 
 delays, and judgement that defaults to "no" for some of the 
 most minute changes to the language, but a game-changing 
 feature like DIP 1000 can just be amended on a whim.
DIP 1000 is in a bit of limbo at the moment. When I took over the process from Mihails, he told me it was stalled and that the proposal did not match the implementation. So I haven't touched it, which is why it's still marked as Draft. At some point, Walter will revise it to match the implementation and then we'll discuss how to handle it. Whatever the status of DIP 1000, I would point out that that one of Walter's DIPs is in Community Review right now after sitting in the PR queue in Draft Review for a while. Once this review stage is done, it will go back into the queue to await Final Review like any other DIP. The only difference between this and other DIPs is that there will be no Formal Assessment for it. And this is the second DIP from Walter that has gone through the process since I've come on board, with DIP 1008 being put on pause at his request.
Aug 21 2018
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 05:04:25 UTC, Mike Parker wrote:

 Whatever the status of DIP 1000, I would point out that that 
 one of Walter's DIPs is in Community Review right now after 
 sitting in the PR queue in Draft Review for a while. Once this 
 review stage is done, it will go back into the queue to await 
 Final Review like any other DIP. The only difference between 
 this and other DIPs is that there will be no Formal Assessment 
 for it. And this is the second DIP from Walter that has gone 
 through the process since I've come on board, with DIP 1008 
 being put on pause at his request.
I understand that Walter's DIPs have been put through the process just like the others, but with regard to the specific issue in this thread (https://issues.dlang.org/show_bug.cgi?id=19097), the accompanying PR (https://github.com/dlang/dmd/pull/8504), and similarly undocumented PR (https://github.com/dlang/dmd/pull/8408), they are amending DIP 1000 and DIP 25 "under the table". There is no accompanying PR to the specification, no formal rational, no RFC from the community, etc... Yet I and others have to go through the DIP process for much less significant changes to the language, and rightly so: https://github.com/dlang/dmd/pull/7310 https://github.com/dlang/dmd/pull/7079 All I want to see from Walter is: 1) a sufficiently documented proposal for his idea 2) an RFC from the community 3) a PR to the spec documenting the final design I don't think that's too much to ask. Mike
Aug 21 2018
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 05:39:05 UTC, Mike Franklin wrote:

 I understand that Walter's DIPs have been put through the 
 process just like the others, but with regard to the specific 
 issue in this thread 
 (https://issues.dlang.org/show_bug.cgi?id=19097), the 
 accompanying PR (https://github.com/dlang/dmd/pull/8504), and 
 similarly undocumented PR 
 (https://github.com/dlang/dmd/pull/8408), they are amending DIP 
 1000 and DIP 25 "under the table".  There is no accompanying PR 
 to the specification, no formal rational, no RFC from the 
 community, etc...
A 3rd example (https://github.com/dlang/dmd/pull/8346) to throw a little more salt on the wound. The DIP25/1000 rabbit hole is deepening behind the curtain. Mike
Aug 21 2018
prev sibling next sibling parent walker <growup_wei qq.com> writes:
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote:
 Something else that rubs me the wrong way is that DIP 1000 is 
 currently in a status of `DRAFT`:  
 https://github.com/dlang/DIPs/blob/master/DIPs/README.md

 What the heck is going on here?  We're adding features to the 
 compiler and modifying Phobos in production releases based on a 
 `DRAFT` proposal?

 Furthermore, I find it hypocritical that some of us are put 
 through a disproportionately burdensome DIP process requiring 
 thorough documentation, multiple peer reviews, excessive 
 delays, and judgment that defaults to "no" for some of the most 
 minute changes to the language, but a game-changing feature 
 like DIP 1000 can just be amended on a whim.
+1, For valuable contributors like you, it's far from friendly. Friendly atmosphere increases creativity. Although, for an industrial-level language, it should be careful to add many features (A core engine that can simulate a lot of easy to use sugars is important, for those potential hackers. )
Aug 22 2018
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote:
 On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin 
 wrote:

 But what bothers me the most...
Something else that rubs me the wrong way is that DIP 1000 is currently in a status of `DRAFT`: https://github.com/dlang/DIPs/blob/master/DIPs/README.md What the heck is going on here? We're adding features to the compiler and modifying Phobos in production releases based on a `DRAFT` proposal?
No, it's behind a flag, so you can't really say that we're shipping it as "production ready release". In fact I think we should have a hell of a lot more of such experimental flags. This would allow us to be able to merge things quickly, and gain real-world feedback and testing on complicated matters instead of PRs stalling to death in the queue. For reference, Rust has currently 148 opt-in experimental language features: https://doc.rust-lang.org/unstable-book/index.html
Aug 22 2018
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 11:02:00 UTC, Seb wrote:

 No, it's behind a flag, so you can't really say that we're 
 shipping it as "production ready release".
The changes to Phobos are not behind a flag. We're making changes to Phobos in the release branch to accommodate a draft/experimental/choose-your-adjective feature. Mike
Aug 22 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/21/2018 6:07 PM, Mike Franklin wrote:
 The proposed idea wants to make the first parameter, if it's `ref`, special.
This is because Phobos is written with functions of the form: void put(sink, parameters...) which corresponds to: sink.put(parameters...) The two forms are fairly interchangeable, made more so by the Uniform Function Call Syntax.
 Why not the first `ref` parameter regardless of whether it's the absolute 
first in the list. Why not the last `ref` parameter? Why not all `ref` parameters? Good question. If this fairly restricted form solves the problems, then there is no need for the more flexible form. Things can always be made more flexible in the future, but tightening things can be pretty disruptive. Hence, unless there is an obvious and fairly strong case case for the flexibility, then it should be avoided for now.
 But what bothers me the most is I think it's missing the bigger picture:  D 
 needs a way to annotate lifetimes.  Maybe `scope` and `return` with weird 
 conditions based on the order of parameters and their attributes are the way
to 
 go.  Maybe there's another way that hasn't yet been considered.
 
 Put together a thorough description of the proposal, justify it, ask the
larger 
 community for comment, vet it, and document it. At least that's what it's
going 
 to take to get me to take action on the PR.
dip1000 has been around for two years, and its predecessor dip25 several years now. Plenty of time for anyone to comment and/or propose something better. --- I want to ensure Atila is successful with this. But that means Phobos has to compile with dip1000. So I need to make it work.
Aug 22 2018
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote:

 The proposed idea wants to make the first parameter, if it's 
 `ref`, special.
This is because Phobos is written with functions of the form: void put(sink, parameters...) which corresponds to: sink.put(parameters...) The two forms are fairly interchangeable, made more so by the Uniform Function Call Syntax.
Makes perfect sense. This is the kind of stuff I'd like you to put in a formal document and present to us as an RFC along with your PR. Then transfer that information to the spec to accompany the DMD PR, after you've received feedback. Mike
Aug 22 2018
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote:

 I want to ensure Atila is successful with this. But that means 
 Phobos has to compile with dip1000. So I need to make it work.
There's a good chance you'll get your PR merged when it's documented and vetted, then you'll be able to make it work for Atila. The only think holding it up is you. I'm not asking for much. Sufficiently document the idea, open it up for comment and Q & A, transfer the documentation to the spec to accompany the DMD implementation PR. Assuming there's no major flaws in the design, it should get merged. Mike P.S. Actually, I've been trying to document it myself, since you don't seem willing to, but it's going to take me a lot longer to figure out what's in your head than it would take you.
Aug 22 2018
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote:

 dip1000 has been around for two years, and its predecessor 
 dip25 several years now. Plenty of time for anyone to comment 
 and/or propose something better.
Part of the problem is that the implementation keeps changing without keeping the documentation in sync. For example you're implementing all of these inference rules without documenting them: https://github.com/dlang/dmd/pull/8346 https://github.com/dlang/dmd/pull/8408 I asked you about, instead of inferring the attributes, allowing users to add such logic themselves: void foo(T)(T x) if (__traits(isPointer, T)) { T = scope T; } { } But, then the PR got rubber-stamped, and now here we are. Mike
Aug 22 2018
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/22/18 5:23 AM, Walter Bright wrote:
 On 8/21/2018 6:07 PM, Mike Franklin wrote:
 The proposed idea wants to make the first parameter, if it's `ref`, 
 special.
This is because Phobos is written with functions of the form:     void put(sink, parameters...) which corresponds to:     sink.put(parameters...) The two forms are fairly interchangeable, made more so by the Uniform Function Call Syntax.
 
  > Why not the first `ref` parameter regardless of whether it's the 
 absolute first in the list.  Why not the last `ref` parameter?  Why not 
 all `ref` parameters?
 
 Good question. If this fairly restricted form solves the problems, then 
 there is no need for the more flexible form. Things can always be made 
 more flexible in the future, but tightening things can be pretty 
 disruptive. Hence, unless there is an obvious and fairly strong case 
 case for the flexibility, then it should be avoided for now.
What about: size_t put(sink, parameters...) Does this qualify as the sink being the "return" type? Obviously the real return can't contain any references, so it trivially can be ruled out as the destination of any escaping parameters. Or how about a member function that takes a ref parameter? Is `this` the "return" or is the ref parameter the "return"? My problem with the idea is that it is going to seem flaky -- we are using convention to dictate what is actually the return parameter, vs. what semantically happens inside the function. It's going to confuse anyone trying to do it a different way. I've experienced this in the past with things like toHash, where if you didn't define it with the exact signature, it wouldn't actually be used. I realize obviously, that `put` is already specified. But as I said in the bug report, we should think twice about defining rules based solely on how Phobos does things, and calling that the solution. As for things being made "more flexible in the future" this basically translates to code breakage. For example, if you are depending on only the first parameter being considered the "return" value, and all of a sudden it changes to encompass all your parameters, your existing code may fail to compile, even if it's correctly safe and properly annotated.
 
 I want to ensure Atila is successful with this. But that means Phobos 
 has to compile with dip1000. So I need to make it work.
 
I think it's a very worthy goal to make Phobos work, and a great proof of concept for dip1000's veracity. However, one-off rules just to make it work with existing code go against that goal IMO. Rules that stand on their own I think will fare better than ones that are loopholes to allow existing code to compile. -Steve
Aug 22 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/22/2018 6:50 AM, Steven Schveighoffer wrote:
 What about:
 
 size_t put(sink, parameters...)
 
 Does this qualify as the sink being the "return" type? Obviously the real
return 
 can't contain any references, so it trivially can be ruled out as the 
 destination of any escaping parameters.
Your reasoning is correct, but currently it only applies with 'void' return types.
 Or how about a member function that takes a ref parameter? Is `this` the 
 "return" or is the ref parameter the "return"?
`this` is the ref parameter. In particular, consider a constructor: struct S { int* p; this(return scope int* p) { this.p = p; } } int i; S s = S(&i); This code appears in Phobos, and it is very reasonable to expect it to check as safe.
 My problem with the idea is that it is going to seem flaky -- we are using 
 convention to dictate what is actually the return parameter, vs. what 
 semantically happens inside the function. It's going to confuse anyone trying
to 
 do it a different way. I've experienced this in the past with things like 
 toHash, where if you didn't define it with the exact signature, it wouldn't 
 actually be used.
 
 I realize obviously, that `put` is already specified. But as I said in the bug 
 report, we should think twice about defining rules based solely on how Phobos 
 does things, and calling that the solution.
Phobos doesn't do this by accident. It's how constructors work (see above) and how pipeline programming works.
 As for things being made "more flexible in the future" this basically
translates 
 to code breakage. For example, if you are depending on only the first
parameter 
 being considered the "return" value, and all of a sudden it changes to
encompass 
 all your parameters, your existing code may fail to compile, even if it's 
 correctly safe and properly annotated.
It's a good point. But I don't see an obvious use case for considering all the ref parameters as being returns.
 I want to ensure Atila is successful with this. But that means Phobos has to
 compile with dip1000. So I need to make it work.
I think it's a very worthy goal to make Phobos work, and a great proof of concept for dip1000's veracity. However, one-off rules just to make it work with existing code go against that goal IMO. Rules that stand on their own I think will fare better than ones that are loopholes to allow existing code to compile.
I couldn't come up with a better idea than this, and this one works.
Aug 23 2018
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/23/18 4:58 AM, Walter Bright wrote:
 On 8/22/2018 6:50 AM, Steven Schveighoffer wrote:
 What about:

 size_t put(sink, parameters...)

 Does this qualify as the sink being the "return" type? Obviously the 
 real return can't contain any references, so it trivially can be ruled 
 out as the destination of any escaping parameters.
Your reasoning is correct, but currently it only applies with 'void' return types.
 Or how about a member function that takes a ref parameter? Is `this` 
 the "return" or is the ref parameter the "return"?
`this` is the ref parameter. In particular, consider a constructor:   struct S {      int* p;      this(return scope int* p) { this.p = p; }   }   int i;   S s = S(&i); This code appears in Phobos, and it is very reasonable to expect it to check as safe.
What I mean to say is, we have a semantic today -- the return value is hooked to any `return` parameters, end of story. This is clear, concise, and easy to understand. You are saying that in some cases, the return value is actually deposited in the `this` parameter. In cases where the actual return type is void, OK, I see that we can tack on that rule without issues. Furthermore any member function (or UFCS function for that matter) REQUIRES the first parameter to be the aggregate. How do you make a member function that stuffs the return into a different parameter properly typecheck? What rule do we tack on then? It's going to be confusing to anyone who writes their API thinking about how it's call syntax reads, not how the compiler wants to do flow analysis. Not to mention, the keyword is `return`, not `returnorfirstparam`. It's still going to be confusing no matter how you look at it.
 My problem with the idea is that it is going to seem flaky -- we are 
 using convention to dictate what is actually the return parameter, vs. 
 what semantically happens inside the function. It's going to confuse 
 anyone trying to do it a different way. I've experienced this in the 
 past with things like toHash, where if you didn't define it with the 
 exact signature, it wouldn't actually be used.

 I realize obviously, that `put` is already specified. But as I said in 
 the bug report, we should think twice about defining rules based 
 solely on how Phobos does things, and calling that the solution.
Phobos doesn't do this by accident. It's how constructors work (see above) and how pipeline programming works.
Constructors I agree are reasonable to consider `this` to be the return value. On that point, I would say we should definitely go ahead with making that rule, and I think it will lead to no confusion whatsoever. pipeline programming depends on returning something other than `void`, so I don't see how this applies. It's more when you are setting members via properties where this comes into play. We need it -- we need this ability to tell the compiler "this parameter connects to this other parameter". I just don't know if the proposed rules are a) good enough for the general case, and b) don't cause more confusion than they are worth.
 As for things being made "more flexible in the future" this basically 
 translates to code breakage. For example, if you are depending on only 
 the first parameter being considered the "return" value, and all of a 
 sudden it changes to encompass all your parameters, your existing code 
 may fail to compile, even if it's correctly safe and properly annotated.
It's a good point. But I don't see an obvious use case for considering all the ref parameters as being returns.
You would have to consider the shortest liftetime and assume everything goes there. It would restrict your legitimate calls. Only mitigating factor may be if you take the ones you aren't going to modify as const or inout.
 I want to ensure Atila is successful with this. But that means 
Phobos has to
 compile with dip1000. So I need to make it work.
I think it's a very worthy goal to make Phobos work, and a great proof of concept for dip1000's veracity. However, one-off rules just to make it work with existing code go against that goal IMO. Rules that stand on their own I think will fare better than ones that are loopholes to allow existing code to compile.
I couldn't come up with a better idea than this, and this one works.
I want to stress that it may be a valid solution, but we should strive to prove the solutions are the best possible rather than just use duct-tape methodology. It should even be considered that perhaps there are better solutions even than the approach dip1000 has taken. I also want to point out that the attitude of 'we could just fix it, but nobody will pull my request' is unhelpful. We want to make sure we have the best solution possible, don't take criticism as meaningless petty bickering. People are genuinely trying to make sure D is improved. Hostility towards reviews or debate doesn't foster that. -Steve
Aug 23 2018
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/23/18 9:32 AM, Steven Schveighoffer wrote:
 On 8/23/18 4:58 AM, Walter Bright wrote:
 On 8/22/2018 6:50 AM, Steven Schveighoffer wrote:
 As for things being made "more flexible in the future" this basically 
 translates to code breakage. For example, if you are depending on 
 only the first parameter being considered the "return" value, and all 
 of a sudden it changes to encompass all your parameters, your 
 existing code may fail to compile, even if it's correctly safe and 
 properly annotated.
It's a good point. But I don't see an obvious use case for considering all the ref parameters as being returns.
You would have to consider the shortest liftetime and assume everything goes there. It would restrict your legitimate calls. Only mitigating factor may be if you take the ones you aren't going to modify as const or inout.
Actually, thinking about this, the shortest lifetime is dictated by how it is called, so there is no valid way to determine which one makes sense when compiling the function. In order for this to work, you'd have to attribute it somehow. I can see that is likely going to be way more cumbersome than it's worth. If I had to design a specific way to allow the common case to be easy, but still provide a mechanism for the uncommon cases, I would say: 1. define a compiler-recognized attribute (e.g. __sink). 2. If __sink is applied to any parameter, that is effectively the return value. 3. In the absence of a __sink designation on non-void-returning functions, it applies to the return value. 4. In the absence of a __sink designation on void returning functions, it applies to the first parameter. 5. Inference of __sink happens even on non-templates. 6. If __sink is attributed on multiple parameters, you assume all return parameters are assigned to all __sink parameters for the purposes of verifying lifetimes are not exceeded. Ugly to specify, but might actually be pretty non-intrusive to use. -Steve
Aug 23 2018
next sibling parent reply Chris M. <chrismohrfeld comcast.net> writes:
On Thursday, 23 August 2018 at 15:14:07 UTC, Steven Schveighoffer 
wrote:
 On 8/23/18 9:32 AM, Steven Schveighoffer wrote:
 [...]
Actually, thinking about this, the shortest lifetime is dictated by how it is called, so there is no valid way to determine which one makes sense when compiling the function. In order for this to work, you'd have to attribute it somehow. I can see that is likely going to be way more cumbersome than it's worth. If I had to design a specific way to allow the common case to be easy, but still provide a mechanism for the uncommon cases, I would say: 1. define a compiler-recognized attribute (e.g. __sink). 2. If __sink is applied to any parameter, that is effectively the return value. 3. In the absence of a __sink designation on non-void-returning functions, it applies to the return value. 4. In the absence of a __sink designation on void returning functions, it applies to the first parameter. 5. Inference of __sink happens even on non-templates. 6. If __sink is attributed on multiple parameters, you assume all return parameters are assigned to all __sink parameters for the purposes of verifying lifetimes are not exceeded. Ugly to specify, but might actually be pretty non-intrusive to use. -Steve
This is more a general reply to the thread. If I think I'm getting a good grasp on the issue here, it seems like something Rust already solved with lifetime annotations. Could they or something similar be leveraged for D as well? https://doc.rust-lang.org/1.9.0/book/lifetimes.html Current solution just seems too specific and very restrictive.
Aug 23 2018
parent reply Chris M. <chrismohrfeld comcast.net> writes:
On Thursday, 23 August 2018 at 15:48:00 UTC, Chris M. wrote:
 On Thursday, 23 August 2018 at 15:14:07 UTC, Steven 
 Schveighoffer wrote:
 On 8/23/18 9:32 AM, Steven Schveighoffer wrote:
 [...]
Actually, thinking about this, the shortest lifetime is dictated by how it is called, so there is no valid way to determine which one makes sense when compiling the function. In order for this to work, you'd have to attribute it somehow. I can see that is likely going to be way more cumbersome than it's worth. If I had to design a specific way to allow the common case to be easy, but still provide a mechanism for the uncommon cases, I would say: 1. define a compiler-recognized attribute (e.g. __sink). 2. If __sink is applied to any parameter, that is effectively the return value. 3. In the absence of a __sink designation on non-void-returning functions, it applies to the return value. 4. In the absence of a __sink designation on void returning functions, it applies to the first parameter. 5. Inference of __sink happens even on non-templates. 6. If __sink is attributed on multiple parameters, you assume all return parameters are assigned to all __sink parameters for the purposes of verifying lifetimes are not exceeded. Ugly to specify, but might actually be pretty non-intrusive to use. -Steve
This is more a general reply to the thread. If I think I'm getting a good grasp on the issue here, it seems like something Rust already solved with lifetime annotations. Could they or something similar be leveraged for D as well? https://doc.rust-lang.org/1.9.0/book/lifetimes.html Current solution just seems too specific and very restrictive.
Heck, now that I'm looking at it, DIP25 seems like a more restricted form of Rust's lifetimes. Let me know if I'm just completely wrong about this, but safe ref int identity(return ref int x) { return x; // fine } would basically be like (pseudosyntax) safe ref'a int identity(ref'a int x) { return x; // fine } Maybe the more sane thing would be a syntax that visually ties them together as above. Obviously we're looking at possibly breaking changes, but how widespread would they be? void betty(ref'a scope int* r, scope'a int* p); // syntax is not so nice since I just arbitrarily stuck them on different keywords, but that's besides the point here
Aug 23 2018
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote:
 
 Heck, now that I'm looking at it, DIP25 seems like a more 
 restricted form of Rust's lifetimes. Let me know if I'm just 
 completely wrong about this, but
 [snip]
Check out DIP1000 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
Aug 23 2018
parent Chris M. <chrismohrfeld comcast.net> writes:
On Thursday, 23 August 2018 at 23:58:00 UTC, jmh530 wrote:
 On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote:
 
 Heck, now that I'm looking at it, DIP25 seems like a more 
 restricted form of Rust's lifetimes. Let me know if I'm just 
 completely wrong about this, but
 [snip]
Check out DIP1000 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
I've read through it already, I'm just throwing out an idea for what I think is a more flexible solution to Walter's current issue which seems to be stemming more from DIP25 rather than DIP1000. More effort but I think it'd be worth it in the long run.
Aug 23 2018
prev sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote:

 Heck, now that I'm looking at it, DIP25 seems like a more 
 restricted form of Rust's lifetimes. Let me know if I'm just 
 completely wrong about this, but
Lifetimes: http://smallcultfollowing.com/babysteps/blog/2016/04/27/non-lexical-lifetimes-introduction/#problem-case-3-conditional-control-flow-across-functions http://smallcultfollowing.com/babysteps/blog/2016/05/09/non-lexical-lifetimes-adding-the-outlives-relation/#problem-case-3-revisited
 would basically be like (pseudosyntax)

  safe ref'a int identity(ref'a int x) {
     return x; // fine
 }

 Maybe the more sane thing would be a syntax that visually ties 
 them together as above. Obviously we're looking at possibly 
 breaking changes, but how widespread would they be?

 void betty(ref'a scope int* r, scope'a int* p); // syntax is 
 not so nice since I just arbitrarily stuck them on different 
 keywords, but that's besides the point here
I wish I had been more involved in D when DIP 25 and DIP 1000 were being proposed, as I don't think the designs were thoroughly vetted. It's taken me at least a year to even begin getting a grasp on it. I think DIP 25 and DIP 1000 should have been combined and thought of holistically as simply "annotated lifetimes in D" rather than separate things. I think then it becomes easier to visualize what the problem is and see, potentially many, alternatives. Given the investments that have already been made in DIP 25 and DIP 1000, it's going to take an extremely motivated individual to fight an uphill battle to change direction now, I'm afraid. If working on D was my full-time job, I'd do it, but who in this community has such resources. Mike
Aug 23 2018
parent reply Chris M. <chrismohrfeld comcast.net> writes:
On Friday, 24 August 2018 at 00:13:48 UTC, Mike Franklin wrote:
 On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote:

 Heck, now that I'm looking at it, DIP25 seems like a more 
 restricted form of Rust's lifetimes. Let me know if I'm just 
 completely wrong about this, but
Non-Lexical Lifetimes: http://smallcultfollowing.com/babysteps/blog/2016/04/27/non-lexical-lifetimes-introduction/#problem-case-3-conditional-control-flow-across-functions http://smallcultfollowing.com/babysteps/blog/2016/05/09/non-lexical-lifetimes-adding-the-outlives-relation/#problem-case-3-revisited
Seems to be more of a warning of what issues we may face if DIP25/DIP1000 are finally implemented. It would be good to consider NLLs as well before D is committed. No point in repeating issues that have already been studied.
 would basically be like (pseudosyntax)

  safe ref'a int identity(ref'a int x) {
     return x; // fine
 }

 Maybe the more sane thing would be a syntax that visually ties 
 them together as above. Obviously we're looking at possibly 
 breaking changes, but how widespread would they be?

 void betty(ref'a scope int* r, scope'a int* p); // syntax is 
 not so nice since I just arbitrarily stuck them on different 
 keywords, but that's besides the point here
I wish I had been more involved in D when DIP 25 and DIP 1000 were being proposed, as I don't think the designs were thoroughly vetted. It's taken me at least a year to even begin getting a grasp on it. I think DIP 25 and DIP 1000 should have been combined and thought of holistically as simply "annotated lifetimes in D" rather than separate things. I think then it becomes easier to visualize what the problem is and see, potentially many, alternatives. Given the investments that have already been made in DIP 25 and DIP 1000, it's going to take an extremely motivated individual to fight an uphill battle to change direction now, I'm afraid. If working on D was my full-time job, I'd do it, but who in this community has such resources.
I think the main problem is that these were created before the DIP process was fully fleshed out, so who knows how much vetting they got. At least they are still compiler switches and probably not widely used, so it's not like we're already too far gone. I do also agree they should be worked on in conjunction, the hard part is finding someone to take ownership and put in the extra effort.
 Mike
Aug 23 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2018 5:58 PM, Chris M. wrote:
 Seems to be more of a warning of what issues we may face if DIP25/DIP1000 are 
 finally implemented. It would be good to consider NLLs as well before D is 
 committed. No point in repeating issues that have already been studied.
DIP25 waqs "finally implemented" several years ago, and works well. DIP1000 was implemented as well, it works, but it didn't cover the case of returning through a ref parameter. There's no way to "thoroughly vet" them before implementing. It doesn't happen with C++, either, somebody builds an implementation and then people try it out.
Aug 24 2018
parent reply Chris M. <chrismohrfeld comcast.net> writes:
On Saturday, 25 August 2018 at 02:37:00 UTC, Walter Bright wrote:
 On 8/23/2018 5:58 PM, Chris M. wrote:
 Seems to be more of a warning of what issues we may face if 
 DIP25/DIP1000 are finally implemented. It would be good to 
 consider NLLs as well before D is committed. No point in 
 repeating issues that have already been studied.
DIP25 waqs "finally implemented" several years ago, and works well. DIP1000 was implemented as well, it works, but it didn't cover the case of returning through a ref parameter. There's no way to "thoroughly vet" them before implementing. It doesn't happen with C++, either, somebody builds an implementation and then people try it out.
What about my other point then on the syntax? I think something similar to what I suggested would be a much more flexible solution and is worth considering.
Aug 25 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/25/2018 5:42 AM, Chris M. wrote:
 What about my other point then on the syntax? I think something similar to
what 
 I suggested would be a much more flexible solution and is worth considering.
Much more work would be needed to make that a proposal.
Aug 28 2018
parent aliak <something something.com> writes:
On Wednesday, 29 August 2018 at 05:01:14 UTC, Walter Bright wrote:
 On 8/25/2018 5:42 AM, Chris M. wrote:
 What about my other point then on the syntax? I think 
 something similar to what I suggested would be a much more 
 flexible solution and is worth considering.
Much more work would be needed to make that a proposal.
Here's some more details that could help with considering it incase you didn't see it: https://forum.dlang.org/post/uxloaftaptfbzhbhquez forum.dlang.org Seems like it's a super set of return and the current proposal to (from what I can understand at least) make some parameters act like a sink in some situations when a function's return value is of a specific type and the parameters are annotated with some convention? Cheers, - Ali
Sep 03 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2018 8:14 AM, Steven Schveighoffer wrote:
 If I had to design a specific way to allow the common case to be easy, but
still 
 provide a mechanism for the uncommon cases, I would say:
 
 1. define a compiler-recognized attribute (e.g.  __sink).
 2. If  __sink is applied to any parameter, that is effectively the return
value.
 3. In the absence of a  __sink designation on non-void-returning functions, it 
 applies to the return value.
 4. In the absence of a  __sink designation on void returning functions, it 
 applies to the first parameter.
 5. Inference of  __sink happens even on non-templates.
 6. If  __sink is attributed on multiple parameters, you assume all return 
 parameters are assigned to all  __sink parameters for the purposes of
verifying 
 lifetimes are not exceeded.
'ref' is already __sink.
Aug 24 2018
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/24/18 10:28 PM, Walter Bright wrote:
 On 8/23/2018 8:14 AM, Steven Schveighoffer wrote:
 If I had to design a specific way to allow the common case to be easy, 
 but still provide a mechanism for the uncommon cases, I would say:

 1. define a compiler-recognized attribute (e.g.  __sink).
 2. If  __sink is applied to any parameter, that is effectively the 
 return value.
 3. In the absence of a  __sink designation on non-void-returning 
 functions, it applies to the return value.
 4. In the absence of a  __sink designation on void returning 
 functions, it applies to the first parameter.
 5. Inference of  __sink happens even on non-templates.
 6. If  __sink is attributed on multiple parameters, you assume all 
 return parameters are assigned to all  __sink parameters for the 
 purposes of verifying lifetimes are not exceeded.
'ref' is already __sink.
No, otherwise we wouldn't need the patch you are pushing. -Steve
Aug 28 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2018 6:32 AM, Steven Schveighoffer wrote:
 Furthermore any member function (or UFCS function for that matter) REQUIRES
the 
 first parameter to be the aggregate. How do you make a member function that 
 stuffs the return into a different parameter properly typecheck?
What I propose is that the function interface be refactored so it does fit into these patterns. Is that an unreasonable requirement? I don't know. But it doesn't seem to be, as I haven't run into it yet.
 Phobos doesn't do this by accident. It's how constructors work (see above) and 
 how pipeline programming works.
Constructors I agree are reasonable to consider `this` to be the return value. On that point, I would say we should definitely go ahead with making that rule, and I think it will lead to no confusion whatsoever. pipeline programming depends on returning something other than `void`, so I don't see how this applies.
grep Phobos for instances of put() and see its signature. It's part of pipeline programming, and it's all over the place.
 You would have to consider the shortest liftetime and assume everything goes 
 there.
That's right.
 It would restrict your legitimate calls.
Maybe that's a good thing. Having multiple simultaneous routes of data out of a function is not good practice (note that it is impossible with functional programming). If you absolutely must have it, the exit routes can be aggregated into a struct, then pass that struct as the first argument.
 I want to stress that it may be a valid solution, but we should strive to
prove 
 the solutions are the best possible rather than just use duct-tape methodology.
I don't know how to prove anything with programming languages.
 It should even be considered that perhaps there are better solutions even than 
 the approach dip1000 has taken.
People have hypothesized that for several years, and so far none have been forthcoming beyond a few hand-wavy generalities.
 I also want to point out that the attitude of 'we could just fix it, but
nobody 
 will pull my request' is unhelpful. We want to make sure we have the best 
 solution possible, don't take criticism as meaningless petty bickering. People 
 are genuinely trying to make sure D is improved. Hostility towards reviews or 
 debate doesn't foster that.
I'm not hostile to debate. I just don't care for "this is uncharted territory, so let's do nothing" which has been going on for probably 4 years now, coincident with "scope is incomplete, D sux". I.e. lead, follow, or get out of the way :-)
Aug 24 2018
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright wrote:
 I'm not hostile to debate. I just don't care for "this is 
 uncharted territory, so let's do nothing" which has been going 
 on for probably 4 years now, coincident with "scope is 
 incomplete, D sux".

 I.e. lead, follow, or get out of the way :-)
Document it and we will help you get it reviewed and merged. Until you do that we will stay in your way. Heck, once we understand it we may even lead, but we can't do that if we have no clue WTF its supposed to do or why the changes are being made. At the very least PRs need detailed changelog entries and preferably spec updates and revisions to the DIP1000 proposal document. Heck even a wiki page would be useful. I'm trying to make the spec page on memory safe programming the authoritative source see https://github.com/dlang/dlang.org/pull/2453
Aug 25 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/25/2018 4:09 AM, Nicholas Wilson wrote:
 On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright wrote:
 I'm not hostile to debate. I just don't care for "this is uncharted territory, 
 so let's do nothing" which has been going on for probably 4 years now, 
 coincident with "scope is incomplete, D sux".

 I.e. lead, follow, or get out of the way :-)
Document it
Already done: https://issues.dlang.org/show_bug.cgi?id=19097
Aug 28 2018
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 29 August 2018 at 05:04:22 UTC, Walter Bright wrote:
 On 8/25/2018 4:09 AM, Nicholas Wilson wrote:
 On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright 
 wrote:
 I'm not hostile to debate. I just don't care for "this is 
 uncharted territory, so let's do nothing" which has been 
 going on for probably 4 years now, coincident with "scope is 
 incomplete, D sux".

 I.e. lead, follow, or get out of the way :-)
Document it
Already done: https://issues.dlang.org/show_bug.cgi?id=19097
Bugzilla is not documentation. These are language changes they need to be in release notes and the spec.
Aug 28 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/28/2018 10:18 PM, Nicholas Wilson wrote:
 Bugzilla is not documentation. These are language changes they need to be in 
 release notes and the spec.
You asked for a clue: "we have no clue WTF its supposed to do or why the changes are being made" and there it is. There are no barriers to reviewing the idea nor the implementation.
Aug 29 2018
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 29 August 2018 at 08:35:30 UTC, Walter Bright wrote:
 On 8/28/2018 10:18 PM, Nicholas Wilson wrote:
 Bugzilla is not documentation. These are language changes they 
 need to be in release notes and the spec.
You asked for a clue: "we have no clue WTF its supposed to do or why the changes are being made" and there it is. There are no barriers to reviewing the idea nor the implementation.
I don't know how much more blunt I can be while still being professional about it: _this must be documented properly_. This is not just about us reviewing it, this is also about people using it. Think about it this way: I'm a new user and I hear that D is supposed to be memory safe. Am I going to trawl through bugzilla to find about the features of this memory safety? Suppose I do a search for "dlang memory safety" on how I can use this feature. What does a search bring up? First hit: https://dlang.org/spec/memory-safe-d.html Gives me a link to https://dlang.org/articles/safed.html, not very helpful, and to system/ safe/ trusted, that tells me a bunch of things I'm not allowed to do in safe code. Second hit: https://dlang.org/blog/2016/09/28/how-to-write-trusted-code-in-d/ More (admittedly better) info on system/ safe/ trusted Third Hit: https://forum.dlang.org/thread/ofkjuq$or6$1 digitalmars.com welp the forum is down, next! Fourth hit: https://wiki.dlang.org/Memory_Management Fifth hit: https://dconf.org/2017/talks/bright.pdf Some stuff about scope, possibly not up to date(!). Sixth hit is Adam Ruppe's fork of the spec page on memory safety Seventh is https://news.ycombinator.com/item?id=12391370 Number 8: https://www.meetup.com/en-AU/SeaLang/events/246692611/ 9: https://www.reddit.com/r/cpp/comments/6b4xrc/walter_bright_believes_memory_safety_will_kill_c/ 10: is the Wikipdia page on D. Searching for "dlang scope" brings up scope(exit) & friends and some old forum posts including https://forum.dlang.org/thread/obfftm$2m3j$1 digitalmars.com which feels like déjà vu, followed by a bunch of irrelevant links to various parts of the spec. That was on a not anonymous search, I know precisely what I'm looking for, and the closest thing I found was 4/5ths the way down a 42 slide PDF with no annotations from a year ago. Even someone relatively familiar is going to look at the spec and the changelog, and they're not going to find anything BECAUSE ITS NOT THERE! You know where it is? Bugzilla, because that's where Walter thinks documentation should go.
Aug 29 2018
parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Wednesday, 29 August 2018 at 13:53:42 UTC, Nicholas Wilson 
wrote:
 On Wednesday, 29 August 2018 at 08:35:30 UTC, Walter Bright 
 wrote:
 On 8/28/2018 10:18 PM, Nicholas Wilson wrote:
 Bugzilla is not documentation. These are language changes 
 they need to be in release notes and the spec.
You asked for a clue: "we have no clue WTF its supposed to do or why the changes are being made" and there it is. There are no barriers to reviewing the idea nor the implementation.
I don't know how much more blunt I can be while still being professional about it: _this must be documented properly_. This is not just about us reviewing it, this is also about people using it.
I can confirm that, from a user's perspective, (the state of) DIP1000 is not clearly defined. I read the forums so I know Phobos is not completely compatible yet, but I don't know what that means to me. The documentation for -dip1000 [1] simply refers to the DIP text, and after reading that I decide I want it. Then, possibly a while after switching to it, I start getting (mangled) link errors. It is not obvious to me that these relate to -dip1000 (which means wasting some time in trouble shooting) but they disappear after I take away that option. I would have appreciated if [1] would have included or linked to information on the situations in which you can successfully use -dip1000, and what are the symptoms when you move outside that envelope (link errors, presumably). At the very least, [1] could/should stress that -dip switches enable experimental functionality and that certain things are known to not work. I do seem to be able to use -dip25 without issues though. [1] https://dlang.org/dmd-windows.html
Aug 30 2018
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/24/18 10:25 PM, Walter Bright wrote:
 On 8/23/2018 6:32 AM, Steven Schveighoffer wrote:
 Furthermore any member function (or UFCS function for that matter) 
 REQUIRES the first parameter to be the aggregate. How do you make a 
 member function that stuffs the return into a different parameter 
 properly typecheck?
What I propose is that the function interface be refactored so it does fit into these patterns. Is that an unreasonable requirement? I don't know. But it doesn't seem to be, as I haven't run into it yet.
So this would mean a member function would have to be refactored into a different function with a different calling syntax. i.e: x.foo(target); would have to be refactored to: target.foo(x); or foo(target, x); Aside from the adjustment in name that is necessary to make this read correctly, that may cause other problems (sometimes non-member functions aren't available if it's a template instantiation).
 Phobos doesn't do this by accident. It's how constructors work (see 
 above) and how pipeline programming works.
Constructors I agree are reasonable to consider `this` to be the return value. On that point, I would say we should definitely go ahead with making that rule, and I think it will lead to no confusion whatsoever. pipeline programming depends on returning something other than `void`, so I don't see how this applies.
grep Phobos for instances of put() and see its signature. It's part of pipeline programming, and it's all over the place.
I stand partly corrected! Indeed you can put a void-returning function at the *end* of a pipeline call, I hadn't thought of that. But in terms of put, strictly speaking, any call of some.pipeline.put(x) is wrong. It should be put(some.pipeline, x), to avoid issues with how put was designed.
 It would restrict your legitimate calls.
Maybe that's a good thing. Having multiple simultaneous routes of data out of a function is not good practice (note that it is impossible with functional programming). If you absolutely must have it, the exit routes can be aggregated into a struct, then pass that struct as the first argument.
Maybe it's better to designate one sink, and have that be the result. I know that after inout was implemented, there were definitely cases where one wanted to have multiple inout routes (i.e. independent traces between multiple parameters for copying mutability). It may be the same for this, I don't know.
 I want to stress that it may be a valid solution, but we should strive 
 to prove the solutions are the best possible rather than just use 
 duct-tape methodology.
I don't know how to prove anything with programming languages.
I don't mean prove like mathematical proof. I mean try to consider how this affects all cases instead of just the one case that will make phobos compile. "show" is a better verb than "prove".
 It should even be considered that perhaps there are better solutions 
 even than the approach dip1000 has taken.
People have hypothesized that for several years, and so far none have been forthcoming beyond a few hand-wavy generalities.
I'm just saying if dip1000 cannot fix all the problems, that instead of adding weird exceptions, or the classic "you're just doing it wrong", maybe we should reconsider the approach. Another case which was brought up and pretty much ignored was this one: https://forum.dlang.org/post/qkrdpmdqaxjadgvsojxr forum.dlang.org
 I also want to point out that the attitude of 'we could just fix it, 
 but nobody will pull my request' is unhelpful. We want to make sure we 
 have the best solution possible, don't take criticism as meaningless 
 petty bickering. People are genuinely trying to make sure D is 
 improved. Hostility towards reviews or debate doesn't foster that.
I'm not hostile to debate. I just don't care for "this is uncharted territory, so let's do nothing" which has been going on for probably 4 years now, coincident with "scope is incomplete, D sux". I.e. lead, follow, or get out of the way :-)
I'm opting for the latter, as the idea of band-aid PRs to get Phobos compiling with dip1000 just to see if dip1000 is going to work seems like the wrong approach to me. The consequence of this is that getting out of the way means your PRs don't get pulled. -Steve
Aug 28 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/28/2018 6:12 AM, Steven Schveighoffer wrote:
 So this would mean a member function would have to be refactored into a 
 different function with a different calling syntax. i.e:
 
 x.foo(target);
 
 would have to be refactored to:
 
 target.foo(x);
 
 or foo(target, x);
Maybe it should be anyway.
 But in terms of put, strictly speaking, any call of some.pipeline.put(x) is 
 wrong. It should be put(some.pipeline, x), to avoid issues with how put was 
 designed.
There are some of those in Phobos, too. It's covered by the proposed addition.
 I don't know how to prove anything with programming languages.
I don't mean prove like mathematical proof. I mean try to consider how this affects all cases instead of just the one case that will make phobos compile. "show" is a better verb than "prove".
I don't know how to find all cases, either, except by implementing it.
Aug 28 2018
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/21/2018 2:17 PM, Atila Neves wrote:
 Well, no. The syntax isn't the same for member functions. The examples from
the 
 actual DIP don't compile. There it says:
 
 -------
 scope can be applied to function return values (even though it is not a type 
 qualifier). It must be applied to the left of the declaration, in the same way 
 ref is:
 
 scope int* foo();     // applies to return value
 --------
 
 Except:
 
 -------
 struct MyStruct { scope int* foo() scope; }
 
 foo.d(1): Error: redundant attribute scope
 -------
 Meaning the first `scope` actually applies to `this`. Writing this out as a 
 non-member function won't help me declare member functions!
 
 I still don't know how to return a ref/pointer that's scoped. And I thought
I'd 
 written code that did that. Maybe I did. I'm very confused.
Here's how you make it work: --- safe: struct MyStruct { ref int foo() return; int* bar() return; } ref int sun() { MyStruct s; return s.foo(); // returning s.foo() escapes a reference to local variable s } int* moon() { MyStruct s; return s.bar(); // returning s.bar() escapes a reference to local variable s } --- In effect, the 'return' on 'foo()' says: The return value of foo() contains the address of 'this', and if the return escapes the scope of what 'this' is a ref to, then it's an error.
Aug 22 2018
prev sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:
 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or the 
 one returned by the member function.

 Kagamin just told me I needed to use `return` instead of 
 `scope` to get things to work and I'm still not sure why.
The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`.
Aug 21 2018
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 22 August 2018 at 03:58:42 UTC, Nicholas Wilson 
wrote:
 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:
 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or the 
 one returned by the member function.

 Kagamin just told me I needed to use `return` instead of 
 `scope` to get things to work and I'm still not sure why.
The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`.
I guess my problem is that DIP1000 talks about returning scope values and they don't seem to actually exist in the implementation.
Aug 22 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/21/2018 8:58 PM, Nicholas Wilson wrote:
 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:
 The problem is that the code we write doesn't deal directly with pointers - 
 see the recent confusion in this forum over where `scope` on the left applies 
 to the `this` pointer or the one returned by the member function.

 Kagamin just told me I needed to use `return` instead of `scope` to get things 
 to work and I'm still not sure why.
The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`.
Another way to think about it is this: S s; return &s; We all know that is an error. The idea is to have a way to express that for: S s; return s.foo(); and: S s; return foo(&s); so that the compiler knows that the return value of foo() is attached to the lifetime of s. Pretty much everything flows from that.
Aug 22 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright wrote:
 On 8/21/2018 8:58 PM, Nicholas Wilson wrote:
 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:
 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or 
 the one returned by the member function.

 Kagamin just told me I needed to use `return` instead of 
 `scope` to get things to work and I'm still not sure why.
The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`.
Another way to think about it is this: S s; return &s; We all know that is an error. The idea is to have a way to express that for: S s; return s.foo(); and: S s; return foo(&s); so that the compiler knows that the return value of foo() is attached to the lifetime of s. Pretty much everything flows from that.
Would the guideline below be correct? "Add scope to every non-template member function that isn't meant to escape this and add return to every non-template member function that returns all or part of `this` by pointer or ref if you want the compiler to check that nothing gets escaped in safe code."
Aug 22 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/22/2018 3:52 AM, Atila Neves wrote:
 On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright wrote:
 On 8/21/2018 8:58 PM, Nicholas Wilson wrote:
 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote:
 The problem is that the code we write doesn't deal directly with pointers - 
 see the recent confusion in this forum over where `scope` on the left 
 applies to the `this` pointer or the one returned by the member function.

 Kagamin just told me I needed to use `return` instead of `scope` to get 
 things to work and I'm still not sure why.
The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`.
Another way to think about it is this:    S s;    return &s; We all know that is an error. The idea is to have a way to express that for:     S s;     return s.foo(); and:     S s;     return foo(&s); so that the compiler knows that the return value of foo() is attached to the lifetime of s. Pretty much everything flows from that.
Would the guideline below be correct? "Add scope to every non-template member function that isn't meant to escape this and add return to every non-template member function that returns all or part of `this` by pointer or ref if you want the compiler to check that nothing gets escaped in safe code."
Being a template doesn't make any difference, except that it will helpfully infer these things. Also, since 'this' is passed by 'ref' to struct member functions, it cannot escape anyway with dip1000: struct S { int x; safe ref int foo() { return x; } } dmd test -dip1000 test.d(4): Error: returning this.x escapes a reference to parameter this, perhaps annotate with return `scope` is for pointers, `ref` does not need further annotation.
Aug 23 2018
next sibling parent ag0aep6g <anonymous example.com> writes:
On Thursday, 23 August 2018 at 08:48:15 UTC, Walter Bright wrote:
 Being a template doesn't make any difference, except that it 
 will helpfully infer these things.
Including the `return` attribute.
 Also, since 'this' is passed by 'ref' to struct member 
 functions, it cannot escape anyway with dip1000:

   struct S {
     int x;
      safe ref int foo() { return x; }
   }

   dmd test -dip1000
   test.d(4): Error: returning this.x escapes a reference to 
 parameter this, perhaps annotate with return
If we add the `return` attribute, we can escape a pointer to `this`. And since attributes are often inferred, it might come as a surprise to the programmer. However, as far as I understand DIP 1000, it should make sure that the returned pointer won't outlive the struct instance. And I guess that's why it's ok to infer the `return` attribute? That depends on DMD not over-estimating the lifetimes of things, though. Unfortunately, it does just that. When in doubt, DMD goes with infinite (e.g., `malloc`). I think that's a problem. But I've already demonstrated my ignorance regarding DIP 1000 in a long, not very fruitful discussion with Atila on Bugzilla [1], so maybe I just don't get it. [1] https://issues.dlang.org/show_bug.cgi?id=19183
Aug 23 2018
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 23 August 2018 at 08:48:15 UTC, Walter Bright wrote:
 On 8/22/2018 3:52 AM, Atila Neves wrote:
 On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright 
 wrote:
 On 8/21/2018 8:58 PM, Nicholas Wilson wrote:
 On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves 
 wrote:
 The problem is that the code we write doesn't deal directly 
 with pointers - see the recent confusion in this forum over 
 where `scope` on the left applies to the `this` pointer or 
 the one returned by the member function.

 Kagamin just told me I needed to use `return` instead of 
 `scope` to get things to work and I'm still not sure why.
The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`.
Another way to think about it is this:    S s;    return &s; We all know that is an error. The idea is to have a way to express that for:     S s;     return s.foo(); and:     S s;     return foo(&s); so that the compiler knows that the return value of foo() is attached to the lifetime of s. Pretty much everything flows from that.
Would the guideline below be correct? "Add scope to every non-template member function that isn't meant to escape this and add return to every non-template member function that returns all or part of `this` by pointer or ref if you want the compiler to check that nothing gets escaped in safe code."
Being a template doesn't make any difference, except that it will helpfully infer these things.
The reason I wrote "non-template" is precisely because attributes get inferred for templates and therefore it would be at best redundant to annotate. What about the guideline being correct or not?
 Also, since 'this' is passed by 'ref' to struct member 
 functions, it cannot escape anyway with dip1000:

   struct S {
     int x;
      safe ref int foo() { return x; }
   }

   dmd test -dip1000
   test.d(4): Error: returning this.x escapes a reference to 
 parameter this, perhaps annotate with return
Returning by ref is good, since it's a lot harder to escape it. Since variables can't be declared ref, I can't pass it to a ref global. However: ---------- struct S { int x; safe int* foo() { return &x; } } ---------- % dmd -o- -dip1000 foo.d % echo $? 0 Oops: ---------- int* gPtr; void main() { auto s = S(42); gPtr = s.foo; } ---------- "Don't use pointers then!". Ok: ---------- int[] gSlice; void main() { auto s = S([42]); gSlice = s.foo; } struct S { int[] x; safe int[] foo() return { return x; } } ---------- % dmd -o- -dip1000 bar.d % echo $? 0 Oops. I can't say `int[] x;` since it doesn't apply to fields.
 `scope` is for pointers, `ref` does not need further annotation.
I always forget this, it's confusing. It doesn't help that slices have pointers, so I guess `scope` is for them too? And in a struct, `this` is a `ref`, yet `scope` on a member function applies to `this`. Atila
Aug 23 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2018 6:10 AM, Atila Neves wrote:
 ----------
 struct S {
      int x;
       safe int* foo() { return &x; }
 }
 ----------
 
 % dmd -o- -dip1000 foo.d
 % echo $?
 0
struct S { int x; safe int* foo() { return &x; } } int* bar() { S s; return s.foo(); } dmd test -dip1000 test.d(3): Error: returning &this.x escapes a reference to parameter this, perhaps annotate with return
 
 Oops:
 
 ----------
 int* gPtr;
 void main() {
      auto s = S(42);
      gPtr = s.foo;
 }
 ----------
I get: test.d(3): Error: returning &this.x escapes a reference to parameter this, perhaps annotate with return
 ----------
 int[] gSlice;
 void main() {
      auto s = S([42]);
      gSlice = s.foo;
 }
 
 struct S {
      int[] x;
       safe int[] foo() return { return x; }
 }
 ----------
T[] is treated like T* as far as scope, etc., are concerned. You should see identical results.
 And in a struct, `this` is a `ref`, yet `scope` on a member function applies
to 
 `this`.
Actually, it applies to fields of `this`.
Aug 28 2018
prev sibling parent reply Maksim Fomin <mxfm protonmail.com> writes:
On Friday, 17 August 2018 at 07:19:25 UTC, Peter Alexander wrote:
 My question is: what is the status of  safe? I am quite 
 surprised to see such a simple case fail. Is  safe believed to 
 be fully implemented (modulo bugs) and this is just an 
 unfortunate corner case, or is it known work-in-progress?
Years ago (2012-2013 https://forum.dlang.org/post/xibbzslaunogifsomapl forum.dlang.org) I was arguing that safe is crippled: 1) there is high probability that some loopholes are missing (several of them were posted to bugzilla and still not fixed since then) 2) fixing loopholes will make using safe inconvenient. My point is somewhat different from expressed already here by Jonathan. He speaks about whitelisting/blacklisting and that the former model will always contain loopholes. In my view, the reason is that memory safety in (essentially good old C) memory model depends on runtime memory type which is unrelated to static type. Taking random variable - it can be allocated on stack, heap, GC-heap, thread-local and there is no way at CT to determine this in general case. In other words, static type rules is bad approach to determine memory safety. It can be used to detect some obvious bugs, but does not work across compilation boundaries. The better approach in my view is to insert runtime
Aug 30 2018
parent Peter Alexander <peter.alexander.au gmail.com> writes:
On Thursday, 30 August 2018 at 16:57:05 UTC, Maksim Fomin wrote:
 My point is somewhat different from expressed already here by 
 Jonathan. He speaks about whitelisting/blacklisting and that 
 the former model will always contain loopholes. In my view, the 
 reason is that memory safety in (essentially good old C) memory 
 model depends on runtime memory type which is unrelated to 
 static type. Taking random variable - it can be allocated on 
 stack, heap, GC-heap, thread-local and there is no way at CT to 
 determine this in general case. In other words, static type 
 rules is bad approach to determine memory safety. It can be 
 used to detect some obvious bugs, but does not work across 
 compilation boundaries. The better approach in my view is to 
 insert runtime code which performs some tests (at least this 

I think it is understood that any compile time checking must be conservative in the general case to be sound. The same is true for compile time type checking, which must reject type safe code that can only be determined type safe at runtime. Adding runtime checks does sound reasonable, and D already does this for array bounds.
Aug 30 2018