www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - memory safety checks and trust

reply Adam D. Ruppe <destructionator gmail.com> writes:
```
void main() {
         int a;
         b ~= &a;
}

int*[] b;
```

trust.d(3): Error: copying & a into allocated memory escapes a 
reference to local variable a


(Interestingly, `b = [&a]` instead of ~= passes muster. What's 
the difference? Just another bug in this?)


But the inconsistency isn't why I'm posting right now, it is my 
fear that D is losing faith in me. There seems to be no way to 
say "trust me" to the compiler. Note that I'm not even using any 
switches to dmd, and adding  trusted had no effect.

One of the nice things about D is that I very rarely feel like I 
am fighting the language (unless I'm working under constraints 
like -w and  safe pure nothrow  nogc stuff, but that's why I just 
don't use those!). From low-level bit twiddling to high level 
"just make it work", the D language usually works with me, a few 
exceptions excluded - but when those happen, you can cast or 
whatever to tell it to trust me.

But this new thing... can I tell it to trust me? Is that just a 
bug too? Or will I have to trick it with extern(C) or asm or 
something?

(PS I actually changed the code somewhat and used a pointer to a 
static instance which worked for my specific case. But I'm a bit 
concerned about the future.)
Apr 10 2020
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:

 ```
 void main() {
          int a;
          b ~= &a;
 }
 
 int*[] b;
 ```
 
 trust.d(3): Error: copying & a into allocated memory escapes a reference to 
 local variable a
 
 
 (Interestingly, `b = [&a]` instead of ~= passes muster. What's the difference? 
 Just another bug in this?)


You will get the error with -preview=dip1000. Since that will eventually be the 
default, it's not a bug.

You can get it to pass without error with the following:

    system int* foo(int* p) { return p; }

    system void test() {
         int a;
         b ~= &a;
         b ~= [foo(&a)];
   }

   int*[] b;

The compiler will inline foo(). I highly recommend annotating such code with 
&system.



 But the inconsistency isn't why I'm posting right now, it is my fear that D is 
 losing faith in me. There seems to be no way to say "trust me" to the
compiler. 
 Note that I'm not even using any switches to dmd, and adding  trusted had no 
 effect.
 
 One of the nice things about D is that I very rarely feel like I am fighting
the 
 language (unless I'm working under constraints like -w and  safe pure nothrow 
  nogc stuff, but that's why I just don't use those!). From low-level bit 
 twiddling to high level "just make it work", the D language usually works with 
 me, a few exceptions excluded - but when those happen, you can cast or
whatever 
 to tell it to trust me.


If  system isn't letting you do what you need to do, let me know.
Apr 10 2020
next sibling parent reply Johan <j j.nl> writes:
On Saturday, 11 April 2020 at 02:57:03 UTC, Walter Bright wrote:

 On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:

 ```
 void main() {
          int a;
          b ~= &a;
 }
 
 int*[] b;
 ```
 
 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a
 
 
 (Interestingly, `b = [&a]` instead of ~= passes muster. What's 
 the difference? Just another bug in this?)


 You will get the error with -preview=dip1000. Since that will 
 eventually be the default, it's not a bug.

 You can get it to pass without error with the following:

    system int* foo(int* p) { return p; }

    system void test() {
         int a;
         b ~= &a;
         b ~= [foo(&a)];
   }

   int*[] b;


The OP's point was that exactly this does not compile. Trivial to 
test online:
https://d.godbolt.org/z/i8WFcs

-Johan
Apr 11 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/11/20 6:01 AM, Johan wrote:

 On Saturday, 11 April 2020 at 02:57:03 UTC, Walter Bright wrote:

 On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:

 ```
 void main() {
          int a;
          b ~= &a;
 }

 int*[] b;
 ```

 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a


 (Interestingly, `b = [&a]` instead of ~= passes muster. What's the 
 difference? Just another bug in this?)


 You will get the error with -preview=dip1000. Since that will 
 eventually be the default, it's not a bug.

 You can get it to pass without error with the following:

    system int* foo(int* p) { return p; }

    system void test() {
         int a;
         b ~= &a;
         b ~= [foo(&a)];
   }

   int*[] b;


 
 The OP's point was that exactly this does not compile. Trivial to test 
 online:
 https://d.godbolt.org/z/i8WFcs


It does if you write it correctly:

b ~= foo(&a);

That was Walter's point. Once you get out of one expression, the checks 
stop.

You can do this too:

auto p = &;
b ~= p;

Note that a more robust argument for the OP's point is that you can 
easily make sure the allocation isn't used outside the function. What if 
you need scratch space to deal with things?

e.g.:

 system void test() {
    int *[] buf;
    int a;
    buf ~= &a; // Same error
}

How is &a escaping here?

-Steve
Apr 11 2020
prev sibling next sibling parent jeckel <jeckel12381236 gmail.com> writes:
On Saturday, 11 April 2020 at 02:57:03 UTC, Walter Bright wrote:

 On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:

 ```
 void main() {
          int a;
          b ~= &a;
 }
 
 int*[] b;
 ```
 
 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a
 
 
 (Interestingly, `b = [&a]` instead of ~= passes muster. What's 
 the difference? Just another bug in this?)


 You will get the error with -preview=dip1000. Since that will 
 eventually be the default, it's not a bug.

 You can get it to pass without error with the following:

    system int* foo(int* p) { return p; }

    system void test() {
         int a;
         b ~= &a;
         b ~= [foo(&a)];
   }

   int*[] b;

 The compiler will inline foo(). I highly recommend annotating 
 such code with &system.


FYI, you don't have to put  system. You can save yourself some 
time as  system is the default. This is equivalent:

    int* foo(int* p) { return p; }

    void test() {
          int a;
          b ~= &a;
          b ~= [foo(&a)];
    }

    int*[] b;
Apr 11 2020
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 11.04.20 04:57, Walter Bright wrote:

 On 4/10/2020 6:21 PM, Adam D. Ruppe wrote:

 ```
 void main() {
          int a;
          b ~= &a;
 }

 int*[] b;
 ```

 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a


 (Interestingly, `b = [&a]` instead of ~= passes muster. What's the 
 difference? Just another bug in this?)


 
 You will get the error with -preview=dip1000. Since that will eventually 
 be the default, it's not a bug.
 ...


Clearly there is a bug or bad design if the address of a`` escaping in 
`b ~= &a` and in `b = [&a]` are not treated the same. But like Adam I 
don't see why there should be such a check in  system/ trusted code at 
all. (I understand that there is a workaround, but that should not be 
required.)

Can we please settle on making  safe actually memory safe and 
 system/ trusted actually trust the programmer?
Apr 11 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2020 7:43 PM, Timon Gehr wrote:

 Clearly there is a bug or bad design if the address of a`` escaping in `b ~=
&a` 
 and in `b = [&a]` are not treated the same.


They are treated the same with dip1000.


 But like Adam I don't see why there 
 should be such a check in  system/ trusted code at all. (I understand that
there 
 is a workaround, but that should not be required.)
 
 Can we please settle on making  safe actually memory safe and  system/ trusted 
 actually trust the programmer?


Consider:

    system int* pumpkin(int i) { return &i);

Should that give an error or not?

I.e. where does one draw the line?
Apr 13 2020
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 13.04.20 09:50, Walter Bright wrote:

 

 But like Adam I don't see why there should be such a check in 
  system/ trusted code at all. (I understand that there is a 
 workaround, but that should not be required.)

 Can we please settle on making  safe actually memory safe and 
  system/ trusted actually trust the programmer?


 
 Consider:
 
     system int* pumpkin(int i) { return &i; }
 
 Should that give an error or not?
 ...


I don't see why not.


 I.e. where does one draw the line?


I think D has been in a pretty good place for a quite long time. Beyond 
straightforward parser/type system diagnostics that you can disable 
using explicit parentheses/type casts, only diagnose cases that have no 
legitimate use. You might not get this right, so it's probably a good 
idea to be open to the idea of removing/restricting diagnostics when 
people complain on the forums that their perfectly valid 
 system/ trusted code was rejected.

In  system/ trusted code, if you diagnose, the code should usually be 
wrong and if it is not, the workaround should have the same behavior as 
if the diagnostic had not been there in the first place (e.g., don't 
slow down debug builds).
Apr 13 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2020 2:03 AM, Timon Gehr wrote:

 Consider:

     system int* pumpkin(int i) { return &i; }

 Should that give an error or not?
 ...


 
 I don't see why not.


I've written code like that to get the stack pointer value.

As to the other case, to my mind putting the address of a stack local into a GC 
allocated object is highly suspicious:

1. it is very very easy to do it unintentionally

2. GC objects tend to be intended to outlast a function, yet the stack address 
will not, so why is this a GC allocated object, as opposed to, say, an RAII 
object or even a stack allocated object?

I don't know what Adam's actual use case for this is, but I would reject such 
code because of (1) and (2) even for  system use in any project I have control
over.

Therefore, whether you agree with where I drew the line or not, this is a 
subjective decision not an objective one.
Apr 14 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/14/20 5:56 AM, Walter Bright wrote:

 On 4/13/2020 2:03 AM, Timon Gehr wrote:

 Consider:

     system int* pumpkin(int i) { return &i; }

 Should that give an error or not?
 ...


 I don't see why not.


 
 I've written code like that to get the stack pointer value.


Can't you use inline assembly?


 
 As to the other case, to my mind putting the address of a stack local 
 into a GC allocated object is highly suspicious:
 
 1. it is very very easy to do it unintentionally
 
 2. GC objects tend to be intended to outlast a function, yet the stack 
 address will not, so why is this a GC allocated object, as opposed to, 
 say, an RAII object or even a stack allocated object?


Because the stack doesn't provide a high limit of data space, and RAII 
may not be what you want to use. In  system code, you should be free to 
shoot yourself in the foot, and also to correctly write memory safe code 
knowing where memory will stop being accessed.

-Steve
Apr 14 2020
parent aliak <something something.com> writes:
On Tuesday, 14 April 2020 at 12:32:18 UTC, Steven Schveighoffer 
wrote:

 On 4/14/20 5:56 AM, Walter Bright wrote:

 On 4/13/2020 2:03 AM, Timon Gehr wrote:

 Consider:

     system int* pumpkin(int i) { return &i; }

 Should that give an error or not?
 ...


 I don't see why not.


 
 I've written code like that to get the stack pointer value.


 Can't you use inline assembly?


You can cast at least:

ptrdiff_t f(int i) { return cast(ptrdiff_t)&i; }

Also,

I tried annotating that with  safe but it didn't work. What's the 
danger of casting the address of a variable to an integer type 
thing?

(I know that I can use  trusted as a work around)


 
 As to the other case, to my mind putting the address of a 
 stack local into a GC allocated object is highly suspicious:
 
 1. it is very very easy to do it unintentionally
 
 2. GC objects tend to be intended to outlast a function, yet




GC objects are a matter of convenience and safety and are using 
within function scopes all the time.


 the stack address will not, so why is this a GC allocated 
 object, as opposed to, say, an RAII object or even a stack 
 allocated object?


 Because the stack doesn't provide a high limit of data space, 
 and RAII may not be what you want to use. In  system code, you 
 should be free to shoot yourself in the foot, and also to 
 correctly write memory safe code knowing where memory will stop 
 being accessed.

 -Steve


Another reason, you may not know how big your array is and need 
to rely on the GC (or manual memory management but the GC is just 
much more easier and safer to use)
Apr 14 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 14 April 2020 at 09:56:39 UTC, Walter Bright wrote:

 As to the other case, to my mind putting the address of a stack 
 local into a GC allocated object is highly suspicious:

 1. it is very very easy to do it unintentionally


Indeed, but isn't that what  safe is for? There should be a way 
do it intentionally, to tell the compiler "trust me". That's my 
main point with this thread - D is being overbearing now.

"Copy the pointer to that array please, D."

"I'm afraid I can't do that, Adam."

https://www.youtube.com/watch?v=ARJ8cAGm6JE
Apr 14 2020
next sibling parent bachmeier <no spam.net> writes:
On Tuesday, 14 April 2020 at 13:34:00 UTC, Adam D. Ruppe wrote:

 On Tuesday, 14 April 2020 at 09:56:39 UTC, Walter Bright wrote:

 As to the other case, to my mind putting the address of a 
 stack local into a GC allocated object is highly suspicious:

 1. it is very very easy to do it unintentionally


 Indeed, but isn't that what  safe is for? There should be a way 
 do it intentionally, to tell the compiler "trust me". That's my 
 main point with this thread - D is being overbearing now.

 "Copy the pointer to that array please, D."

 "I'm afraid I can't do that, Adam."

 https://www.youtube.com/watch?v=ARJ8cAGm6JE


+1 Most of us aren't writing web browsers and probably none of us 
are working on airplanes. Give us enough rope to shoot ourselves 
in the foot. If I wanted a language to tell me how to program, 
I'd still be using Go.
Apr 14 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/14/2020 6:34 AM, Adam D. Ruppe wrote:

 Indeed, but isn't that what  safe is for? There should be a way do it 
 intentionally, to tell the compiler "trust me". That's my main point with this 
 thread - D is being overbearing now.
 
 "Copy the pointer to that array please, D."



So, Adam, how do you feel about:

    system int* pumpkin(int i) { return &i);

Timon says that should be an error. What is your opinion?
Apr 14 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 14 April 2020 at 20:10:20 UTC, Walter Bright wrote:

 So, Adam, how do you feel about:

    system int* pumpkin(int i) { return &i);

 Timon says that should be an error. What is your opinion?


I could go either way on it. That does catch a real problem in 
real world Phobos use and it is very rarely what you actually 
want:

string s() {
         return hexDigest!MD5("test");
}

So I see the value in that error.

But the array example is different anyway because the above thing 
is almost always wrong while the array is far more ambiguous. Is 
it temporary scratch? Is it being passed to a different thread 
but you then wait until the other thread is done before 
returning? (That's what my crazy code that prompted this did btw.)

If it is an error 99% of the time, the error seems reasonable. 
But even then, what if you are one of the 1% of exceptions?

Or with the array, I'd guess it is an error 60% of the time. 
Maybe still fair to have it an error there... but what if you are 
in the other 40%? What do you do?

How do I tell the compiler "I know this looks wrong at first 
glance, but trust me"?

Again, that's why my subject line for this thread was "and 
trust". I don't mind this error most the time, we can see from 
experience that this kind of mistake can be made often enough to 
take note of it.

I just want some way to tell the compiler to trust me. OK, I 
confess, it'd probably still annoy me, like the annoying integral 
promotion casts kinda drive me nuts. But I can live with it, 
since at least there's a way to cast it away.

But here,  trusted didn't work. cast didn't work. So it just 
stopped me. I changed the code to use a static TLS instance 
instead of a stack item and got it compiling again, but I'm also 
concerned that D is going down an ugly path here. What if my 
workaround hack now gets plugged later? I want something, akin to 
`cast`, that is formally defined to function as an escape hatch.


(Maybe a double cast through void* might work, there seems to be 
potential in tricking the compiler with that, but those can be 
even harder to get right than the original lifetime problem, so 
we do need to make sure the cure isn't worse than the disease in 
allowing the exceptions.)
Apr 14 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/14/2020 3:58 PM, Adam D. Ruppe wrote:

 but what if you are in the other 40%? What do you do?


I proposed a direct way you can proceed with your method, it works.

I also proposed a more meta  safe rewrite that produced smaller and faster code 
than the  system example rejected by the compiler.

This makes it hard to discern the value in loosening up that particular
restriction.
Apr 14 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 15 April 2020 at 02:19:51 UTC, Walter Bright wrote:

 I proposed a direct way you can proceed with your method, it 
 works.


Is that formally defined and guaranteed to continue working in 
the future? Or is it just exploiting a temporary hole in the 
system that could be patched at any time without notice?

Maybe document it something like "assumeUnique" with 
"allowEscape" to give that peace of mind and I'll be happy with 
it.

Note that my original problem was already solved before I wrote 
this opening message. I'm not asking for a solution to this 
particular case; I have plenty of those (for now at least, who 
knows until you arbitrarily break them too).

I'm asking for confidence in D's future.
Apr 15 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/15/2020 5:01 AM, Adam D. Ruppe wrote:

 I'm asking for confidence in D's future.


If you've got an issue in the future that you need help with, that's what we're 
here for.

We're not going to make D  system useless.
Apr 16 2020
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 14.04.20 22:10, Walter Bright wrote:

 On 4/14/2020 6:34 AM, Adam D. Ruppe wrote:

 Indeed, but isn't that what  safe is for? There should be a way do it 
 intentionally, to tell the compiler "trust me". That's my main point 
 with this thread - D is being overbearing now.

 "Copy the pointer to that array please, D."


 
 
 So, Adam, how do you feel about:
 
     system int* pumpkin(int i) { return &i);
 
 Timon says that should be an error.


That's actually not what I said. I said I would not be opposed to it 
being an error.
Apr 15 2020
prev sibling next sibling parent Claude <claudemr live.fr> writes:
On Monday, 13 April 2020 at 07:50:43 UTC, Walter Bright wrote:

 Consider:

    system int* pumpkin(int i) { return &i);

 Should that give an error or not?

 I.e. where does one draw the line?


I think it should not return an error. It is  system code, the 
programmer should be able to hack around as he wishes.

If he chooses safety,  safe is here.

For me the line is:  system code code should behave like plain 
old C. D just gives some extra expressiveness.

As a system-programmer, I wish to do with D what I can already do 
with C, faster, more elegantly.
Apr 13 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/13/20 3:50 AM, Walter Bright wrote:

 On 4/11/2020 7:43 PM, Timon Gehr wrote:

 Clearly there is a bug or bad design if the address of a`` escaping in 
 `b ~= &a` and in `b = [&a]` are not treated the same.


 
 They are treated the same with dip1000.
 

 But like Adam I don't see why there should be such a check in 
  system/ trusted code at all. (I understand that there is a 
 workaround, but that should not be required.)

 Can we please settle on making  safe actually memory safe and 
  system/ trusted actually trust the programmer?


 
 Consider:
 
     system int* pumpkin(int i) { return &i);
 
 Should that give an error or not?


Yes. This one directly exposes dangling references, and is easily caught 
by the compiler.


 
 I.e. where does one draw the line?
 


Where it's possible to write valid code that is memory safe even though 
it cannot be proven.

For instance:

void foo()
{
	int*[] b;
	int a;
	b ~= &a;
         // use b but don't expose it outside foo
}

There are no memory safety violations there.

This one is also memory safe:

int*[] b;
void foo()
{
    int a;
    b ~= &a;
    // use b, but don't allow it to be exposed in a way that can result 
in corruption.
    b = b[0 .. $-1];
}

The first case is highly useful, as one often needs scratch space to 
perform complex calculations or graph algorithms. I see no reason to 
disallow it.

The second case is more questionable, because it's very easy for someone 
to keep a copy of b at some point, and then you have lost track of who 
has access to a's memory. But it still is possible for it to be safe. 
 system is supposed to mean "I know what I'm doing".

-Steve
Apr 13 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/13/2020 5:25 AM, Steven Schveighoffer wrote:

 For instance:
 
 void foo()
 {
      int*[] b;
      int a;
      b ~= &a;
          // use b but don't expose it outside foo
 }
 
 There are no memory safety violations there.
 The first case is highly useful, as one often needs scratch space to perform 
 complex calculations or graph algorithms. I see no reason to disallow it.


I see a reason:

  void foo()
  {
     int*[1] b = void;
     int a;
     b[0] = &a;
  }

It's faster, too. And if it is written as:

  &safe void foo()
  {
     int a;
     int*[1] b;
     b[0] = &a;
  }

it's even  safe (with -preview=dip1000).

I know, I know, this isn't the real use case. But I've done plenty of "use 
scratch data structures on the stack" programming for speed and I know how to 
make it work without needing to store addresses in GC objects.
Apr 14 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/14/20 6:08 AM, Walter Bright wrote:

 On 4/13/2020 5:25 AM, Steven Schveighoffer wrote:

 For instance:

 void foo()
 {
      int*[] b;
      int a;
      b ~= &a;
          // use b but don't expose it outside foo
 }

 There are no memory safety violations there.
 The first case is highly useful, as one often needs scratch space to 
 perform complex calculations or graph algorithms. I see no reason to 
 disallow it.


 
 I see a reason:
 
   void foo()
   {
      int*[1] b = void;
      int a;
      b[0] = &a;
   }


The toy cases aren't real cases, they just show the error and that the 
usage is safe.

We are not trying to solve the toy. Of course, adding one element to an 
array and doing nothing is not the real case. I'm talking about maybe an 
array of millions of pointers, some of which are on the stack (consider 
like a linked list where you store the head on the stack), and ensuring 
the array doesn't escape the function.


 
 It's faster, too. And if it is written as:
 
   &safe void foo()
   {
      int a;
      int*[1] b;
      b[0] = &a;
   }
 
 it's even  safe (with -preview=dip1000).
 
 I know, I know, this isn't the real use case. But I've done plenty of 
 "use scratch data structures on the stack" programming for speed and I 
 know how to make it work without needing to store addresses in GC objects.


It's not about knowing how to do it other ways, or doing it the fastest 
way. It's about knowing how to do it correctly *this* way. Maybe I don't 
want to deal with having to ensure things are freed properly. What if my 
graph algorithm has cycles, will RAII (e.g. RefCounted) take care of 
that? If you do it right with the GC, then there are no memory issues, 
why is the compiler stopping that in  system code?  system is supposed 
to be "I know what I am doing".

The answer to "why won't  system let me write this memory safe code?" 
shouldn't be "you're doing it wrong". That's factually incorrect, and 
the compiler shouldn't bug me about it. Especially when it's trivially 
circumvented.  safe is for compiler checks for safety.

That being said, I agree with the simple case of returning a pointer 
from a stack variable directly from a function being disallowed. That 
also can be easily worked around, which should probably be required, but 
is never correct anyway.

-Steve
Apr 14 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/14/2020 10:16 AM, Steven Schveighoffer wrote:

 That being said, I agree with the simple case of returning a pointer from a 
 stack variable directly from a function being disallowed. That also can be 
 easily worked around, which should probably be required, but is never correct 
 anyway.


Oh, it can be correct, when one wants to examine the stack pointer value. I've 
used it for that purpose myself. (Examining the stack pointer is valuable when 
determining whether other pointers are pointing into the stack or not.)

As I replied to Timon, you're drawing a subjective (not objective) line at what 
is acceptable or not.
Apr 14 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/14/20 4:14 PM, Walter Bright wrote:

 On 4/14/2020 10:16 AM, Steven Schveighoffer wrote:

 That being said, I agree with the simple case of returning a pointer 
 from a stack variable directly from a function being disallowed. That 
 also can be easily worked around, which should probably be required, 
 but is never correct anyway.


 
 Oh, it can be correct, when one wants to examine the stack pointer 
 value. I've used it for that purpose myself. (Examining the stack 
 pointer is valuable when determining whether other pointers are pointing 
 into the stack or not.)


Do you need to call a function to do that? Won't just &somevariable work?

Also, this compiles just fine, and gives you the stack pointer value:

size_t stackPtr()  system
{
   int i;
   return cast(size_t)&i;
}


 
 As I replied to Timon, you're drawing a subjective (not objective) line 
 at what is acceptable or not.
 


Yes, it's subjective. But so is common sense.

The objective line is that all should be allowed. I'm fine with that too 
if that means you can write useful memory safe  system code. It's an 
entirely valid and *consistent* proposal to say " system just allows 
anything, including obvious memory errors. Use  safe if you want 
compiler restrictions."

Or, we could say, for the 0.00000001% of the time you want to get a 
stack pointer value, use a cast, which means the other 99.9999999% of 
the cases which are actually memory problems are helpfully caught. You 
must be able to see that there is a difference in common utility between 
needing to capture the stack pointer value, and wanting to utilize heap 
space.

It doesn't even make sense that array appending is somehow "the one 
thing" that causes problems, when you can pass stack pointers to 
functions which you don't control, slice static arrays, create stack 
pointers at will and *then* do whatever you want, etc. If there is a 
subjective story for this "feature" it has quite a few holes in it. It's 
like putting up a fence post instead of a fence to protect a space. It 
just ends up being more annoying than functional.

I can do this too, apparently since it's a TLS array and not a heap 
array, totes ok!

int*[1] arr;

void bar()
{
     int i;
     arr[0] = &i;
}

-Steve
Apr 14 2020
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/14/20 4:40 PM, Steven Schveighoffer wrote:

 It's an entirely valid and *consistent* proposal to say " system just 
 allows anything, including obvious memory errors. Use  safe if you want 
 compiler restrictions."


Note that this is especially palatable if  safe is the default and you 
have to opt-in to  system.

-Steve
Apr 14 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/14/2020 1:40 PM, Steven Schveighoffer wrote:

 Also, this compiles just fine, and gives you the stack pointer value:
 
 size_t stackPtr()  system
 {
    int i;
    return cast(size_t)&i;
 }


Yup. I also gave workarounds to Adam's example.



 I can do this too, apparently since it's a TLS array and not a heap array,
totes 
 ok!
 
 int*[1] arr;
 
 void bar()
 {
      int i;
      arr[0] = &i;
 }


Ironically, I argue for consistency with "safe by default" and you argue for 
special cases, and here the reverse.

But be careful, you may talk me into disallowing these cases, too :-/
Apr 14 2020
next sibling parent reply Max Samukha <maxsamukha gmail.com> writes:
On Wednesday, 15 April 2020 at 06:12:22 UTC, Walter Bright wrote:


 Ironically, I argue for consistency with "safe by default" and 
 you argue for special cases, and here the reverse.

 But be careful, you may talk me into disallowing these cases, 
 too :-/


A good reference point would be C. If it is allowed in C, it 
should be allowed in  system D. If I put  system on a D function, 
i tell the compiler not to bug me with incorrect assumptions 
about my code's safety.
Apr 15 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/15/2020 1:14 AM, Max Samukha wrote:

 A good reference point would be C. If it is allowed in C, it should be allowed 
 in  system D. If I put  system on a D function, i tell the compiler not to bug 
 me with incorrect assumptions about my code's safety.


What matters is having a way to get things done, not any way.
Apr 15 2020
parent Max Samukha <maxsamukha gmail.com> writes:
On Thursday, 16 April 2020 at 04:49:08 UTC, Walter Bright wrote:

 On 4/15/2020 1:14 AM, Max Samukha wrote:

 A good reference point would be C. If it is allowed in C, it 
 should be allowed in  system D. If I put  system on a D 
 function, i tell the compiler not to bug me with incorrect 
 assumptions about my code's safety.


 What matters is having a way to get things done, not any way.


Did I propose "any way"? You said that there were no objective 
criteria for what checks to perform. I proposed C as a reference, 
which basically means "unsafe enough to rule out complaints such 
as Adam's". If you are going to make  safe the default, it makes 
sense to reduce the restrictions on  system, not the other way 
round.
Apr 16 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/15/20 2:12 AM, Walter Bright wrote:

 On 4/14/2020 1:40 PM, Steven Schveighoffer wrote:

 Also, this compiles just fine, and gives you the stack pointer value:

 size_t stackPtr()  system
 {
    int i;
    return cast(size_t)&i;
 }


 
 Yup. I also gave workarounds to Adam's example.


Of course. There are workarounds to all these issues. The question then 
becomes, is the inconsistency for this particular restriction worth the 
annoyance it causes to work around it.

The answer is subjective, as always. IMO, fetching the stack pointer is 
something that just isn't ever a requirement for code. It's such a 
specialized need that the annoyance it causes is worth all the memory 
problems it prevents when people accidentally return stack references. 
Maybe that's just my code, maybe everyone else needs to fetch the stack 
pointer in their code. But I didn't think that was the case.

On the other hand, using heap data is pretty common in my code, and 
occasionally I might stick a reference to stack data in there. If I am 
doing it responsibly, there is no harm. The compiler might make me jump 
through some annoyance hoops, but it's possible to do.

So we have to draw a line, an arbitrary line, that says "everything like 
this is going to require some workaround, everything else, well it's 
 system code, you got what you signed up for". My line is that returning 
stack pointers, if provable, is more often than not an error, and I 
would like to be reminded of it from the compiler. Putting stack data 
into heap blocks, is already not consistently flagged, so I don't think 
it's saving much to have this annoyance available.

But if the compiler is going to be annoying, be annoying according to a 
plan, not just random annoyance.


 I can do this too, apparently since it's a TLS array and not a heap 
 array, totes ok!

 int*[1] arr;

 void bar()
 {
      int i;
      arr[0] = &i;
 }


 
 Ironically, I argue for consistency with "safe by default" and you argue 
 for special cases, and here the reverse.


There are different levels of consistency. If you have a reasonable 
rule, then be consistent in that rule. It might make you have to do some 
extra workarounds to use the language, but the result is that you should 
be a better programmer, and you have a rationale to point at for why 
things are the way they are.

I'll take for example the rule that if(cond); is disallowed. It's a 
wonderful rule, but totally not consistent with the language (I can 
write empty statements in most other places). BUT it's such a common 
mistake that it's worth the annoyance. It's caught bugs for me many many 
times.

However, what if while(cond); was allowed? That kind of inconsistency 
doesn't make sense. It's inconsistent to disallow empty statements for 
an if block, but allow them for while blocks. Arguing that people would 
want consistency with while statements but not if statements is hard to 
justify.

In the safe debate, the problem is simply that code that compiles today 
and is  system will compile tomorrow and be  safe, without ANY checking 
or any deprecation. This destroys everything that  safe has been built upon.

Ironically, what I'm asking for is consistency with existing code, and 
you are arguing to make an exception that extern(C)  system functions 
compile should silently convert to  safe functions without warning. We 
just cannot change the meaning of all code without warning people or 
causing an error.

If we didn't have existing code that's extern(C) then the debate would 
be entirely different. It would be a new thing, and it would be a 
legitimate position to have extern(C) prototypes safe by default.

I'll note that I proposed an alteration that has received zero responses 
that I think would solve all those concerns:

https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com


 
 But be careful, you may talk me into disallowing these cases, too :-/
 


If you did, it would at LEAST be reasonable! At least the rule would be 
consistent. So I would be fine with it! As Adam says, there is no rhyme 
or reason for this one rule vs. the allowances for other code, so 
there's no predictability as to when D might somehow decide what you did 
was not correct.

Ruling by whim is not as understandable as providing clear and 
consistent rules to base changes on.

-Steve
Apr 15 2020
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.20 17:11, Steven Schveighoffer wrote:

 
 If we didn't have existing code that's extern(C) then the debate would 
 be entirely different. It would be a new thing, and it would be a 
 legitimate position to have extern(C) prototypes safe by default.


No.
Apr 15 2020
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.20 17:11, Steven Schveighoffer wrote:

 
 I'll note that I proposed an alteration that has received zero responses 
 that I think would solve all those concerns:
 
 https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com


I had answered to that with the example of  safe functions with 
inconsistent return values. I think you'd need to do some more mangling 
to make this acceptable.
Apr 15 2020
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.04.20 08:12, Walter Bright wrote:


 
 Ironically, I argue for consistency with "safe by default"


You really don't. Allowing memory corruption to be annotated  safe is 
inconsistent with the meaning of  safe.



 Apr 15 2020



prev sibling  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 14.04.20 22:14, Walter Bright wrote:

 On 4/14/2020 10:16 AM, Steven Schveighoffer wrote:

 That being said, I agree with the simple case of returning a pointer 
 from a stack variable directly from a function being disallowed. That 
 also can be easily worked around, which should probably be required, 
 but is never correct anyway.


 
 Oh, it can be correct, when one wants to examine the stack pointer 
 value. I've used it for that purpose myself. (Examining the stack 
 pointer is valuable when determining whether other pointers are pointing 
 into the stack or not.)
 
 As I replied to Timon, you're drawing a subjective (not objective) line 
 at what is acceptable or not.
 


It depends on the language being consistent. If the language says what 
you are doing is guaranteed to lead to UB, a diagnostic is fine.

If the spec says this is a supported use case that is guaranteed to work 
across all D compilers, the story is a bit different.

The spec is not formal enough to figure out which of those is the case. 
Does the spec really mandate an implementation to put the program stack 
at a continuous range of addresses? Are dangling pointers guaranteed to 
compare correctly to valid pointers? etc.



 Apr 15 2020



prev sibling next sibling parent  Kagamin <spam here.lot>  writes:


On Saturday, 11 April 2020 at 01:21:56 UTC, Adam D. Ruppe wrote:

 One of the nice things about D is that I very rarely feel like 
 I am fighting the language (unless I'm working under 
 constraints like -w and  safe pure nothrow  nogc stuff, but 
 that's why I just don't use those!). From low-level bit 
 twiddling to high level "just make it work", the D language 
 usually works with me, a few exceptions excluded - but when 
 those happen, you can cast or whatever to tell it to trust me.


Walter said there's now a third kind of safety for not annotated 
functions or something like that.



 Apr 10 2020



prev sibling  parent  mipri <mipri minimaltype.com>  writes:


On Saturday, 11 April 2020 at 01:21:56 UTC, Adam D. Ruppe wrote:

 ```
 void main() {
         int a;
         b ~= &a;
 }

 int*[] b;
 ```

 trust.d(3): Error: copying & a into allocated memory escapes a 
 reference to local variable a


 (Interestingly, `b = [&a]` instead of ~= passes muster. What's 
 the difference? Just another bug in this?)


 But the inconsistency isn't why I'm posting right now, it is my 
 fear that D is losing faith in me. There seems to be no way to 
 say "trust me" to the compiler. Note that I'm not even using 
 any switches to dmd, and adding  trusted had no effect.


Meanwhile, in Rust:

static mut b: Vec<*mut i32> = Vec::new();

fn bad() {
     let mut a = 0;
     unsafe { b.push(&mut a as *mut i32) };
}

fn main() {
     bad();
     println!("{:?}", unsafe { &b });
}

which outputs (for a particular run): [0x7ffc6e4b23b8]

So you have to say "trust me" a couple of times (or more than
twice, if you count the muts and the cast), but you *can* say
that. You can faithfully implement this bug in Rust.

Part of why you can do that, is that this code is using
pointers. The same code with only references fails despite the
unsafe{} blocks:

5 |     unsafe { b.push(&mut a) };
   |              -------^^^^^^-
   |              |      |
   |              |      borrowed value does not live long enough
   |              argument requires that `a` is borrowed for 
`'static`
6 | }
   | - `a` dropped here while still borrowed

Pointers in Rust lack safety and liveness guarantees. That
stuff is for references. D has references too...

| Rust     | D             |
|----------+---------------|
| &i32     | const ref int |
| &mut i32 | ref int       |
| *i32     | const(int)*   |
| *mut i32 | int*          |

... but not really. You can't declare them ("only parameters or
foreach declarations can be ref"), and you can't take a
reference of a variable, and you can't cast a pointer to a ref.

So it's not enough to say "Rust has normal vs. unsafe{} code,
and D has  safe vs. normal code, and the difference is only a
matter of defaults." It's also the case that Rust has unsafe
pointers and safe references, but D only has pointers, so any
memory protection that D gets must fall on its pointers.

With  live the protection only applies to pointers within
 safe code, and with  safe-as-default it might be even easier
to do things that way, or even to remove protections that
formerly applied to  system code. So the trend is not
necessarily towards it one day being more appropriate to say
that Rust has pointers and references, but D only has
references which it calls pointers.



 Apr 14 2020