www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Struct should be invalid after move

reply Sebastiaan Koppe <mail skoppe.eu> writes:
I have a non-copyable struct and I really want a compiler error 
whenever I access it after it has been moved.

---
struct Handle {
     ...
      disable this(this);
     ...
}

void main() {
     import std.algorithm : move;
     auto handle = getOne();
     auto second = handle.move;  /// line 14
     auto third = handle.move;    ///  <- compiler error, variable 
handle is invalid after line 14
}
---

I believe this would prevent some nasty bugs when dealing with 
these structs.

What do you think?
Nov 27 2018
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 27 November 2018 at 08:00:22 UTC, Sebastiaan Koppe 
wrote:

 I have a non-copyable struct and I really want a compiler error 
 whenever I access it after it has been moved.

 ---
 struct Handle {
     ...
      disable this(this);
     ...
 }

 void main() {
     import std.algorithm : move;
     auto handle = getOne();
     auto second = handle.move;  /// line 14
     auto third = handle.move;    ///  <- compiler error, 
 variable handle is invalid after line 14
 }
 ---

 I believe this would prevent some nasty bugs when dealing with 
 these structs.

 What do you think?


we need an ` invalidate`s attribute for that, would apply to 
pointers passed to free etc. Probably would need a DIP though. I 
like it.
Nov 27 2018
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 27 November 2018 at 08:37:32 UTC, Nicholas Wilson 
wrote:

 On Tuesday, 27 November 2018 at 08:00:22 UTC, Sebastiaan Koppe 
 wrote:

 I have a non-copyable struct and I really want a compiler 
 error whenever I access it after it has been moved.

 ---
 struct Handle {
     ...
      disable this(this);
     ...
 }

 void main() {
     import std.algorithm : move;
     auto handle = getOne();
     auto second = handle.move;  /// line 14
     auto third = handle.move;    ///  <- compiler error, 
 variable handle is invalid after line 14
 }
 ---

 I believe this would prevent some nasty bugs when dealing with 
 these structs.

 What do you think?


 we need an ` invalidate`s attribute for that, would apply to 
 pointers passed to free etc. Probably would need a DIP though. 
 I like it.


Yeah, that could be an awesome addition. However, it's not as 
simple, because this should be legal too:

```
auto handle = getOne();
auto second = handle.move; // `handle` becomes invalid
// ...
handle = getOne(); // `handle` is valid again
```
Nov 27 2018
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Tue, Nov 27, 2018 at 1:08 AM Stanislav Blinov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 27 November 2018 at 08:37:32 UTC, Nicholas Wilson
 wrote:

 On Tuesday, 27 November 2018 at 08:00:22 UTC, Sebastiaan Koppe
 wrote:

 I have a non-copyable struct and I really want a compiler
 error whenever I access it after it has been moved.

 ---
 struct Handle {
     ...
      disable this(this);
     ...
 }

 void main() {
     import std.algorithm : move;
     auto handle = getOne();
     auto second = handle.move;  /// line 14
     auto third = handle.move;    ///  <- compiler error,
 variable handle is invalid after line 14
 }
 ---

 I believe this would prevent some nasty bugs when dealing with
 these structs.

 What do you think?


 we need an ` invalidate`s attribute for that, would apply to
 pointers passed to free etc. Probably would need a DIP though.
 I like it.


 Yeah, that could be an awesome addition. However, it's not as
 simple, because this should be legal too:

 ```
 auto handle = getOne();
 auto second = handle.move; // `handle` becomes invalid
 // ...
 handle = getOne(); // `handle` is valid again
 ```


The language goes to great effort to return everything to it's `init`
state after being moved or destroyed. The whole point of that is to
cover the cases you are concerned about here.
If it was invalid to access a thing after it's moved (or after
destruction), then there's no reason for any of the work that resets
thing to their `init` to happen... that's quite a different set of
language semantics.
Nov 27 2018
parent reply kinke <noone nowhere.com> writes:
On Tuesday, 27 November 2018 at 19:51:12 UTC, Manu wrote:

 The language goes to great effort to return everything to it's 
 `init`
 state after being moved or destroyed. The whole point of that 
 is to
 cover the cases you are concerned about here.
 If it was invalid to access a thing after it's moved (or after
 destruction), then there's no reason for any of the work that 
 resets
 thing to their `init` to happen... that's quite a different set 
 of
 language semantics.


Yeah, I also totally fail to see a reason why a moved-from 
instance should be treated any different than a default-allocated 
one. Ownership taken away, reset to init, memory obviously still 
claimed and accessible as long as in scope.
Nov 27 2018
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 27 November 2018 at 22:49:44 UTC, kinke wrote:

 Yeah, I also totally fail to see a reason why a moved-from 
 instance should be treated any different than a 
 default-allocated one.


You are right, it shouldn't. A Handle.init or one after a move 
are both invalid. Invalid from the standpoint that a Handle 
should always refer to some underlying resource. And when it 
provably doesn't, I want to compiler to help me.
Nov 27 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, November 27, 2018 12:51:12 PM MST Manu via Digitalmars-d wrote:

 The language goes to great effort to return everything to it's `init`
 state after being moved or destroyed. The whole point of that is to
 cover the cases you are concerned about here.
 If it was invalid to access a thing after it's moved (or after
 destruction), then there's no reason for any of the work that resets
 thing to their `init` to happen... that's quite a different set of
 language semantics.


I'd forgotten that move reset the variable to its init value. But given that
it does, that pretty much does eliminate this entire problem. I can
understand it if someone would still consider it a bug in their code to
reuse the variable given that it then doesn't hold the value that it did
before, but it's perfectly  safe to access it at that point, and memory
safety issues would have been the biggest reason to be concerned about
touching the variable again after it's been moved. As such, having the
compiler flag a variable being used after a move would be pretty much
equivalent to having it flag it if you used a variable that was simply
default-initialized rather than having been given an explicit value.

- Jonathan M Davis
Nov 27 2018
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 28 November 2018 at 00:28:35 UTC, Jonathan M Davis 
wrote:

 As such, having the compiler flag a variable being used after a 
 move would be pretty much equivalent to having it flag it if 
 you used a variable that was simply default-initialized rather 
 than having been given an explicit value.


Yep. Exactly.

I would love to opt-in such a feature via e.g.:

struct Handle {
    ...
    disable Handle.init;
   ...
}
Nov 27 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, November 28, 2018 12:48:54 AM MST Sebastiaan Koppe via 
Digitalmars-d wrote:

 On Wednesday, 28 November 2018 at 00:28:35 UTC, Jonathan M Davis

 wrote:

 As such, having the compiler flag a variable being used after a
 move would be pretty much equivalent to having it flag it if
 you used a variable that was simply default-initialized rather
 than having been given an explicit value.


 Yep. Exactly.

 I would love to opt-in such a feature via e.g.:

 struct Handle {
     ...
     disable Handle.init;
    ...
 }


If you have

 disable this();

in your struct, then that makes it illegal to default-initialize that type.
However, it still has an init value (since _all_ types in D have init
values), and so something like move would still use the init value, and any
type introspection using init would continue to work. It's just the default
initialization which is then illegal, forcing you to explicitly initialize
all variables of that type.

That being said, I wouldn't advise disabling default initialization unless
you really need to, because a number of things rely on it (e.g. I'm not sure
if you can stick a type which can't be default-initialized into an array,
because certain array operations depend on default initialization). So,
while it's occasionally useful, it's one of those things that tends to cause
problems when you do it.

- Jonathan M Davis
Nov 28 2018
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, November 27, 2018 1:00:22 AM MST Sebastiaan Koppe via 
Digitalmars-d wrote:

 I have a non-copyable struct and I really want a compiler error
 whenever I access it after it has been moved.

 ---
 struct Handle {
      ...
       disable this(this);
      ...
 }

 void main() {
      import std.algorithm : move;
      auto handle = getOne();
      auto second = handle.move;  /// line 14
      auto third = handle.move;    ///  <- compiler error, variable
 handle is invalid after line 14
 }
 ---

 I believe this would prevent some nasty bugs when dealing with
 these structs.

 What do you think?


Well, the DIP to add opPostMove has been approved, and if opPostMove is then
 disabled, then that should disable moving entirely (making it a compile
error for it to happen at all, not to access the struct after it's been
moved), which is probably a better approach. However, that will require that
the DIP actually be implemented, and AFAIK, there's no ETA on that.

- Jonathan M Davis
Nov 27 2018
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 27 November 2018 at 09:39:13 UTC, Jonathan M Davis 
wrote:

 Well, the DIP to add opPostMove has been approved, and if 
 opPostMove is then  disabled, then that should disable moving 
 entirely (making it a compile error for it to happen at all, 
 not to access the struct after it's been moved), which is 
 probably a better approach. However, that will require that the 
 DIP actually be implemented, and AFAIK, there's no ETA on that.

 - Jonathan M Davis


But I actually want to move it. Just so that the variable moved 
from is considered invalid afterwards.
Nov 27 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, November 27, 2018 3:32:40 AM MST Sebastiaan Koppe via 
Digitalmars-d wrote:

 On Tuesday, 27 November 2018 at 09:39:13 UTC, Jonathan M Davis

 wrote:

 Well, the DIP to add opPostMove has been approved, and if
 opPostMove is then  disabled, then that should disable moving
 entirely (making it a compile error for it to happen at all,
 not to access the struct after it's been moved), which is
 probably a better approach. However, that will require that the
 DIP actually be implemented, and AFAIK, there's no ETA on that.

 - Jonathan M Davis


 But I actually want to move it. Just so that the variable moved
 from is considered invalid afterwards.


Well, if the compiler were doing the move, then there would be no move if
there were any possibility of the variable being used after it was moved.
So, basically, you're looking at the case where an  system function that the
compiler almost ceratinly does not understand is doing any moves has moved
an object, and you want the compiler to then complain if the variable is
used after the move. That would require some way for the compiler to be told
that a move has occurred, which is likely possible, but I don't know how it
could be reasonably done - especially since the move could be happening
inside a function that the compiler has no visibility into. We'd probably
have to add yet another attribute to indicate that a function is doing a
move, and I think that most everyone would consider that to be a terrible
idea.

The other issue that comes to mind is that there are cases where you
actually _do_ want to use a variable after a move has occurred - usually
because you want to move something else into that variable (e.g. with swap).
So, even if the compiler could perfectly detect when a move had occurred, it
would then have to somehow know when using the variable again was then okay
and when it wasn't. And I'm not sure that that's really solvable. If it
already could detect all moves, then it could know whether another move was
being attempted and consider that case safe, but I doubt that it could be
made to understand when using opAssign would be safe. In many cases, it
would not be, but for some types it would be (in particular when opAssign
doesn't care about the previous value). So, it seems like trying to detect
what you're trying to detect could easily require flagging cases which are
actually valid as being illegal. All in all, at best, this just sounds like
it would be adding a lot of extra complexity for detecting problems with an
operation that is  system by its very nature and which most programs aren't
going to do.

If you can come up with a simple way for the kind of problem that you want
detected to be detected while not having it complain about valid code, then
maybe it could be implemented, but thinking about it, I'm pretty sure that a
new function attribute would be required, and I don't see that flying at
this point. As it is, most folks think that D has too many attributes. So,
any new ones have to really be worth it, whereas this just seems like it
would be dealing with an uncommon edge case. It would certainly be valuable
in that context, but I seriously question that it would be worth the extra
language complexity. Also, remember that Walter is likely to shoot down most
solutions (to pretty much any problem) that require code flow analysis, and
this seems like the kind of problem that would require it (at least if it
isn't going to have false positives or miss cases which it should flag).

But if you want to solve this problem, then it looks to me like the first
thing that you need to solve is how the compiler knows when a move has even
occurred. And simply flagging common functions that do moves isn't going to
be enough to catch all cases - especially when all it takes to hide the fact
that a move occurred is to wrap the move call in another function that the
compiler doesn't have visibility into thanks to separate compilation.

- Jonathan M Davis
Nov 27 2018
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 27 November 2018 at 12:14:17 UTC, Jonathan M Davis 
wrote:


 Well, if the compiler were doing the move...


[wall snipped]


 And simply flagging common functions that do moves isn't going 
 to be enough to catch all cases - especially when all it takes 
 to hide the fact that a move occurred is to wrap the move call 
 in another function that the compiler doesn't have visibility 
 into thanks to separate compilation.


"Easy" enough. The move and emplace families should be compiler 
intrinsics, not library functions.
Nov 27 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, November 27, 2018 5:37:29 AM MST Stanislav Blinov via 
Digitalmars-d wrote:

 On Tuesday, 27 November 2018 at 12:14:17 UTC, Jonathan M Davis

 wrote:

 Well, if the compiler were doing the move...


 [wall snipped]


 And simply flagging common functions that do moves isn't going
 to be enough to catch all cases - especially when all it takes
 to hide the fact that a move occurred is to wrap the move call
 in another function that the compiler doesn't have visibility
 into thanks to separate compilation.


 "Easy" enough. The move and emplace families should be compiler
 intrinsics, not library functions.


Well, they're not currently. And even if they were, that doesn't solve the
problem of detecting when functions are called that do moves that the
compiler does not have the source code for due to separate compilation or
because the move is actually being done in C code that gets called from D
code. Without that, at best, you're catching the really simple cases, and
even that likely requires code flow analysis. It also has the problem that
it gives the impression that the compiler catches a certain class of problem
for you in general when in fact it only catches it in a few cases. It's
basically getting into the same territory as detecting when a pointer or
reference is never initialized with anything other null and making that an
error - something that Walter has already refused to do, because only the
most basic cases would be caught (especially if you want to avoid false
positives), and he doesn't want to add features that require code flow
analysis.

Honestly, this seems like the sort of thing that's better suited to a linter
that detects some cases where it thinks you might be accessing a variable in
an invalid manner after it's been moved (allowing you to examine the code to
see whether it's right or not), whereas the compiler has to get it right
100% of the time - especially if it's an error.

- Jonathan M Davis
Nov 27 2018
prev sibling next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 27 November 2018 at 08:00:22 UTC, Sebastiaan Koppe 
wrote:

 I have a non-copyable struct and I really want a compiler error 
 whenever I access it after it has been moved.

 ---
 struct Handle {
     ...
      disable this(this);
     ...
 }

 void main() {
     import std.algorithm : move;
     auto handle = getOne();
     auto second = handle.move;  /// line 14
     auto third = handle.move;    ///  <- compiler error, 
 variable handle is invalid after line 14
 }
 ---

 I believe this would prevent some nasty bugs when dealing with 
 these structs.

 What do you think?


void foo(int a)
{
     import std.algorithm : move;
     auto handle = getOne();
     if (a > 0)
         auto second = handle.move;
     auto third = handle.move; // compile error?
}

Not a trivial problem in the general case.
Nov 27 2018
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 27 November 2018 at 09:50:25 UTC, John Colvin wrote:

 void foo(int a)
 {
     import std.algorithm : move;
     auto handle = getOne();
     if (a > 0)
         auto second = handle.move;
     auto third = handle.move; // compile error?
 }

 Not a trivial problem in the general case.


Not a trivial problem indeed. But I really believe we need these 
things (or similar) for safety's sake.

One other issue I can see is if the handle is passed by ref into 
opaque function. At this point we can't reason about whether the 
struct is still valid after the call to bar:

---
void bar(ref Handle);

void main() {
   auto handle = getOne();
   bar(handle);
   auto second = handle.move(); // compile error? don't know
}
---

Possible solution is to always disallow move after passing as ref.

And maybe move could be disallowed on a scope ref param, then we 
could safely pass Handle to functions that take by ref while 
provably keeping the caller's Handle valid.
Nov 27 2018
parent reply Alex <sascha.orlov gmail.com> writes:
On Tuesday, 27 November 2018 at 10:53:21 UTC, Sebastiaan Koppe 
wrote:

 Not a trivial problem indeed. But I really believe we need 
 these things (or similar) for safety's sake.

 One other issue I can see is if the handle is passed by ref 
 into opaque function. At this point we can't reason about 
 whether the struct is still valid after the call to bar:

 ---
 void bar(ref Handle);

 void main() {
   auto handle = getOne();
   bar(handle);
   auto second = handle.move(); // compile error? don't know
 }
 ---

 Possible solution is to always disallow move after passing as 
 ref.

 And maybe move could be disallowed on a scope ref param, then 
 we could safely pass Handle to functions that take by ref while 
 provably keeping the caller's Handle valid.


There exist

auto val = Handle.init;

1. How do you treat this?
2. Why do you don't want to treat the handle after movement the 
same way?
Nov 27 2018
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 27 November 2018 at 10:59:03 UTC, Alex wrote:

 There exist

 auto val = Handle.init;

 1. How do you treat this?


I have no idea. Logically I would say that - in my case - the val 
is invalid.


 2. Why do you don't want to treat the handle after movement the 
 same way?


Because the handle refers to an underlying resource, and any 
access to that resource through that handle is invalid after a 
move. Much like one doesn't want to call .release() twice on an 
unique(T) wrapper type.

Sure, I could put in a runtime check, but then I get runtime 
errors and I rather have compile time errors.
Nov 27 2018
next sibling parent reply Alex <sascha.orlov gmail.com> writes:
On Tuesday, 27 November 2018 at 12:03:20 UTC, Sebastiaan Koppe 
wrote:

 On Tuesday, 27 November 2018 at 10:59:03 UTC, Alex wrote:

 There exist

 auto val = Handle.init;

 1. How do you treat this?


 I have no idea. Logically I would say that - in my case - the 
 val is invalid.


This is a source of bugs. The T.init value is the one and only 
value, which exists, independently from the definition (of 
Handle). Therefore, your own functions on Handle have to consider 
this case and have to behave properly. I.e. define some behavior.

I mean, what would happen, if you take an existent function and 
pass Handle.init to it? Would it assert? Would it yield 42? Some 
reproducible behavior is defined well enough.


 2. Why do you don't want to treat the handle after movement 
 the same way?


 Because the handle refers to an underlying resource, and any 
 access to that resource through that handle is invalid after a 
 move. Much like one doesn't want to call .release() twice on an 
 unique(T) wrapper type.

 Sure, I could put in a runtime check, but then I get runtime 
 errors and I rather have compile time errors.


This is not an answer to my question. The question was: why you 
would like to have a different behavior w.r.t. some other 
behavior, which exists, independent of your actions.
Nov 27 2018
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 27 November 2018 at 12:42:43 UTC, Alex wrote:

 On Tuesday, 27 November 2018 at 12:03:20 UTC, Sebastiaan Koppe 
 wrote:



 auto val = Handle.init;

 1. How do you treat this?


 I have no idea. Logically I would say that - in my case - the 
 val is invalid.


 This is a source of bugs. The T.init value is the one and only 
 value, which exists, independently from the definition (of 
 Handle). Therefore, your own functions on Handle have to 
 consider this case and have to behave properly. I.e. define 
 some behavior.


No. The only thing that must be valid to do with a value in .init 
state is destruct it. Standard library move et al. assume as 
much. In fact, what they do is exactly that - wipe out the moved 
from value, emplacing the initializer there. What also *may* be 
valid is reinitialization/assignment.


 Sure, I could put in a runtime check, but then I get runtime 
 errors and I rather have compile time errors.


 This is not an answer to my question. The question was: why you 
 would like to have a different behavior w.r.t. some other 
 behavior, which exists, independent of your actions.


Because .init does not generally exert any useful behavior.
Nov 27 2018
parent reply Alex <sascha.orlov gmail.com> writes:
On Tuesday, 27 November 2018 at 12:51:58 UTC, Stanislav Blinov 
wrote:

 On Tuesday, 27 November 2018 at 12:42:43 UTC, Alex wrote:

 On Tuesday, 27 November 2018 at 12:03:20 UTC, Sebastiaan Koppe 
 wrote:



 auto val = Handle.init;

 1. How do you treat this?


 I have no idea. Logically I would say that - in my case - the 
 val is invalid.


 This is a source of bugs. The T.init value is the one and only 
 value, which exists, independently from the definition (of 
 Handle). Therefore, your own functions on Handle have to 
 consider this case and have to behave properly. I.e. define 
 some behavior.


 No. The only thing that must be valid to do with a value in 
 .init state is destruct it.


I have something completely different in mind...
std.traits is full of examples, where T.init is used beyond of 
destruction.
And the behavior of a function for a .init value is also defined 
at all times. Independently from how the function is written. It 
can be easily tested by plugging the .init value as the input 
parameter.


 Standard library move et al. assume as much. In fact, what they 
 do is exactly that - wipe out the moved from value, emplacing 
 the initializer there. What also *may* be valid is 
 reinitialization/assignment.


Sure. And I don't question this. I just would like to know, how 
the custom functions of OP handle the case of .init.
I mean... If they don't, then they assume the value is not in the 
.init state and perform some actions with it, which can lead to 
some assert. Which would be perfectly ok.


 Sure, I could put in a runtime check, but then I get runtime 
 errors and I rather have compile time errors.


 This is not an answer to my question. The question was: why 
 you would like to have a different behavior w.r.t. some other 
 behavior, which exists, independent of your actions.


 Because .init does not generally exert any useful behavior.


Hmm... sure. But some behavior exists. And my question is: why 
(not even how) the behavior should differ between the .init case 
and the case after a move.
Nov 27 2018
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 27 November 2018 at 13:17:24 UTC, Alex wrote:


 I have something completely different in mind...
 std.traits is full of examples, where T.init is used beyond of 
 destruction.
 And the behavior of a function for a .init value is also 
 defined at all times. Independently from how the function is 
 written. It can be easily tested by plugging the .init value as 
 the input parameter.


For compile-time checks, sure, as it is a convenient way to get 
at the underlying type. But that's all that is.


 Standard library move et al. assume as much. In fact, what 
 they do is exactly that - wipe out the moved from value, 
 emplacing the initializer there. What also *may* be valid is 
 reinitialization/assignment.


 Sure. And I don't question this. I just would like to know, how 
 the custom functions of OP handle the case of .init.
 I mean... If they don't, then they assume the value is not in 
 the .init state and perform some actions with it, which can 
 lead to some assert. Which would be perfectly ok.


Right now we have to resort to runtime checks. Which is hilarious 
considering we *can* disable default construction, thereby 
asserting that .init is indeed an invalid state.


 This is not an answer to my question. The question was: why 
 you would like to have a different behavior w.r.t. some other 
 behavior, which exists, independent of your actions.


 Because .init does not generally exert any useful behavior.


 Hmm... sure. But some behavior exists. And my question is: why 
 (not even how) the behavior should differ between the .init 
 case and the case after a move.


What Sebastiaan proposes is for there to be *no* behavior at all, 
as using an invalid value would become a compile-time error. 
Which would mean one wouldn't need extraneous run-time checks.
Nov 27 2018
next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 27 November 2018 at 14:37:40 UTC, Stanislav Blinov 
wrote:

 What Sebastiaan proposes is for there to be *no* behavior at 
 all, as using an invalid value would become a compile-time 
 error. Which would mean one wouldn't need extraneous run-time 
 checks.


Yes. The only valid thing to do would be to move an existing 
Handle into it (explicitly or as a result of a function).
Nov 27 2018
prev sibling parent reply Alex <sascha.orlov gmail.com> writes:
On Tuesday, 27 November 2018 at 14:37:40 UTC, Stanislav Blinov 
wrote:

 On Tuesday, 27 November 2018 at 13:17:24 UTC, Alex wrote:

 For compile-time checks, sure, as it is a convenient way to get 
 at the underlying type. But that's all that is.

 Right now we have to resort to runtime checks. Which is 
 hilarious considering we *can* disable default construction, 
 thereby asserting that .init is indeed an invalid state.


Ok... so. Without default construction,

int i;
and
int i = void;

become the same. But you won't be able to prevent it from being 
an int, right?
How would you detect an invalid state in this case?


 What Sebastiaan proposes is for there to be *no* behavior at 
 all, as using an invalid value would become a compile-time 
 error. Which would mean one wouldn't need extraneous run-time 
 checks.


I assume, that detecting invalid states becomes harder as types 
become simpler. And, it depends strongly on the application, 
whether a value is considered invalid... Doesn't Typedef already 
allow this functionality?
Nov 27 2018
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 27 November 2018 at 15:14:13 UTC, Alex wrote:

 On Tuesday, 27 November 2018 at 14:37:40 UTC, Stanislav Blinov 
 wrote:

 On Tuesday, 27 November 2018 at 13:17:24 UTC, Alex wrote:

 For compile-time checks, sure, as it is a convenient way to 
 get at the underlying type. But that's all that is.

 Right now we have to resort to runtime checks. Which is 
 hilarious considering we *can* disable default construction, 
 thereby asserting that .init is indeed an invalid state.


 Ok... so. Without default construction,

 int i;
 and
 int i = void;

 become the same. But you won't be able to prevent it from being 
 an int, right?
 How would you detect an invalid state in this case?


You wouldn't. int doesn't have an invalid state. That's not 
what's in question here. This is:

```
struct S {
     private X* ptr;
      disable this();
      disable this(this);
     this(X* p) { ptr = p; }
     ref get() return  safe { return *p; }
     alias get this;
}
```

...or pointers in general.


 What Sebastiaan proposes is for there to be *no* behavior at 
 all, as using an invalid value would become a compile-time 
 error. Which would mean one wouldn't need extraneous run-time 
 checks.


 I assume, that detecting invalid states becomes harder as types 
 become simpler.


There's no difference. If the compiler would know what an invalid 
state is and what operations result in it, it doesn't matter what 
the type is.


 And, it depends strongly on the application, whether a value is 
 considered invalid... Doesn't Typedef already allow this 
 functionality?


No, it does not. We are talking about complie-time checking, not 
runtime checking.
Nov 27 2018
prev sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 27 November 2018 at 12:42:43 UTC, Alex wrote:

 On Tuesday, 27 November 2018 at 12:03:20 UTC, Sebastiaan Koppe

 I have no idea. Logically I would say that - in my case - the 
 val is invalid.


 This is a source of bugs. The T.init value is the one and only 
 value, which exists, independently from the definition (of 
 Handle). Therefore, your own functions on Handle have to 
 consider this case and have to behave properly. I.e. define 
 some behavior.

 I mean, what would happen, if you take an existent function and 
 pass Handle.init to it? Would it assert? Would it yield 42? 
 Some reproducible behavior is defined well enough.


You are right, Handle.init is in the same state as it were after 
a move, so you would expect similar behaviour.
Nov 27 2018
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 27 November 2018 at 12:03:20 UTC, Sebastiaan Koppe 
wrote:

 On Tuesday, 27 November 2018 at 10:59:03 UTC, Alex wrote:

 There exist

 auto val = Handle.init;

 1. How do you treat this?


 I have no idea. Logically I would say that - in my case - the 
 val is invalid.


 2. Why do you don't want to treat the handle after movement 
 the same way?


 Because the handle refers to an underlying resource, and any 
 access to that resource through that handle is invalid after a 
 move. Much like one doesn't want to call .release() twice on an 
 unique(T) wrapper type.

 Sure, I could put in a runtime check, but then I get runtime 
 errors and I rather have compile time errors.


So ... what you want is this:

"Compiler, ensure that there is no possible execution-flow path 
in which this variable is ever set to 0."

We can actually attempt to prove that, using SMT or other logic 
solvers, though it will most likely say that there is set up 
inputs and flow-decisions which will result in the variable set 
to 0, and it can even give you an example.

You can then take that example and remove the invalid case, for 
the input provided.

This is a massive amount of work to get right.
Nov 28 2018
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 28 November 2018 at 09:11:48 UTC, Stefan Koch wrote:

 So ... what you want is this:

 "Compiler, ensure that there is no possible execution-flow path 
 in which this variable is ever set to 0."


Almost. I want to prevent usage *after* it is set to 0. Similar 
to how the compiler already recognises this error:

---
void main() {
   int* p = null;
   (*p) = 5;  // <- Error: null dereference in function _Dmain
}
---

Which more and more languages already provide. Swift, Kotlin etc.
Nov 28 2018
parent John Colvin <john.loughran.colvin gmail.com> writes:
On Wednesday, 28 November 2018 at 09:30:21 UTC, Sebastiaan Koppe 
wrote:

 On Wednesday, 28 November 2018 at 09:11:48 UTC, Stefan Koch 
 wrote:

 So ... what you want is this:

 "Compiler, ensure that there is no possible execution-flow 
 path in which this variable is ever set to 0."


 Almost. I want to prevent usage *after* it is set to 0. Similar 
 to how the compiler already recognises this error:

 ---
 void main() {
   int* p = null;
   (*p) = 5;  // <- Error: null dereference in function _Dmain
 }
 ---

 Which more and more languages already provide. Swift, Kotlin 
 etc.


Sure, catching some simple cases is definitely doable, if someone 
has the compiler chops (or is prepared to learn) and some time to 
dedicate to it.

I suspect if it was a small code change and had a zero 
false-positive rate (I.e. only caught things that *definitely* 
were wrong) then it would be accepted. Large code change -> 
uncertain. Any false positives that forbid valid programs -> 
unlikely to be accepted.

That's just my feeling though, who knows....
Nov 28 2018
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/27/18 3:00 AM, Sebastiaan Koppe wrote:

 I have a non-copyable struct and I really want a compiler error whenever 
 I access it after it has been moved.
 
 ---
 struct Handle {
      ...
       disable this(this);
      ...
 }
 
 void main() {
      import std.algorithm : move;
      auto handle = getOne();
      auto second = handle.move;  /// line 14
      auto third = handle.move;    ///  <- compiler error, variable 
 handle is invalid after line 14
 }
 ---
 
 I believe this would prevent some nasty bugs when dealing with these 
 structs.
 
 What do you think?


You can do a runtime error (with opPostMove included), but I think a 
compiler error would require full flow analysis, and would have to 
default to allowing it, as otherwise you will get false errors that are 
more annoying than the original problem.

-Steve
Nov 27 2018
parent reply burjui <bytefu gmail.com> writes:
On Tuesday, 27 November 2018 at 15:27:19 UTC, Steven 
Schveighoffer wrote:

 You can do a runtime error (with opPostMove included), but I 
 think a compiler error would require full flow analysis, and 
 would have to default to allowing it, as otherwise you will get 
 false errors that are more annoying than the original problem.


A good modern compiler of a statically compiled language has to 
have flow analysis enabled unconditionally. I don't see any 
reason not to enable it in D frontend, except Walter's 
stubbornness. The current situation with D doesn't even make 
sense: with optimizations enabled, the compiler can reject code 
that compiles just fine without them. I find it sad and 
ridiculous at the same time. This happens exactly because 
dataflow analysis is only enabled when optimizations are enabled:

void main()
{
     int* x;
     *x = 0;
}

This compiles just fine with `dmd x.d`, but emits an error with 
`dmd -O x.d`:
Error: null dereference in function _Dmain
Nov 27 2018
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/27/18 11:49 AM, burjui wrote:

 On Tuesday, 27 November 2018 at 15:27:19 UTC, Steven Schveighoffer wrote:

 You can do a runtime error (with opPostMove included), but I think a 
 compiler error would require full flow analysis, and would have to 
 default to allowing it, as otherwise you will get false errors that 
 are more annoying than the original problem.


 
 A good modern compiler of a statically compiled language has to have 
 flow analysis enabled unconditionally.


Does g++ count as a modern compiler? It doesn't do flow analysis on the 
code you have below.


 I 
 find it sad and ridiculous at the same time. This happens exactly 
 because dataflow analysis is only enabled when optimizations are enabled:
 
 void main()
 {
      int* x;
      *x = 0;
 }
 
 This compiles just fine with `dmd x.d`, but emits an error with `dmd -O 
 x.d`:
 Error: null dereference in function _Dmain


Note that complete flow analysis is equivalent to the halting problem. 
So really, you can never get it perfect, and therefore, there are always 
going to be some things you can't catch. So it's impossible to fulfill 
that promise in any compiler.

In any case, I can't imagine how to do it without some handshaking 
between std.algorithm.move and the compiler, which is likely to be met 
with resistance.

-Steve
Nov 27 2018
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Tuesday, 27 November 2018 at 19:28:13 UTC, Steven 
Schveighoffer wrote:

 void main()
 {
      int* x;
      *x = 0;
 }
 
 This compiles just fine with `dmd x.d`, but emits an error 
 with `dmd -O x.d`:
 Error: null dereference in function _Dmain


 Note that complete flow analysis is equivalent to the halting 
 problem. So really, you can never get it perfect, and 
 therefore, there are always going to be some things you can't 
 catch. So it's impossible to fulfill that promise in any 
 compiler.


It's possible if you're willing to make the rules strict enough 
that some otherwise-valid programs get rejected--that's what Rust 
does, after all. But for D, that isn't really an option.
Nov 27 2018
prev sibling parent reply burjui <bytefu gmail.com> writes:
On Tuesday, 27 November 2018 at 19:28:13 UTC, Steven 
Schveighoffer wrote:

 Note that complete flow analysis is equivalent to the halting 
 problem. So really, you can never get it perfect, and 
 therefore, there are always going to be some things you can't 
 catch. So it's impossible to fulfill that promise in any 
 compiler.


That's a straw man. I was not talking about complete flow 
analysis and getting it perfect. Some form of flow analysis is 
already build into DMD, so I suggest to:
1. At least enable it unconditionally
2. Improve it to account for more complex scenarios than simple 
null dereferencing

Anyway, the point I am trying to make here is that it's much 
better to make one compiler smarter, than to rely on many users' 
not-even-close-to-perfect discipline, and I think C++ proves it. 
Software development these days is too complex to comprehend 
without the help of tools, which must become smarter to really 
have an impact and stay in use.
Nov 30 2018
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/30/18 10:01 AM, burjui wrote:

 On Tuesday, 27 November 2018 at 19:28:13 UTC, Steven Schveighoffer wrote:

 Note that complete flow analysis is equivalent to the halting problem. 
 So really, you can never get it perfect, and therefore, there are 
 always going to be some things you can't catch. So it's impossible to 
 fulfill that promise in any compiler.


 
 That's a straw man. I was not talking about complete flow analysis and 
 getting it perfect. Some form of flow analysis is already build into 
 DMD, so I suggest to:
 1. At least enable it unconditionally


I'm not a compiler writer, and don't know the first thing about the 
requirements, but I would hazard a guess that the reason this is the 
case is if optimization isn't enabled, some critical pieces needed to do 
flow analysis aren't present. I don't think it's orthogonal.


 2. Improve it to account for more complex scenarios than simple null 
 dereferencing


I think any improvements to the flow analysis would be welcome! The more 
the compiler can flag as an obvious error, the better.


 Anyway, the point I am trying to make here is that it's much better to 
 make one compiler smarter, than to rely on many users' 
 not-even-close-to-perfect discipline, and I think C++ proves it. 
 Software development these days is too complex to comprehend without the 
 help of tools, which must become smarter to really have an impact and 
 stay in use.


Best effort to detect problems is generally what we have with D. My 
point was simply that any time you solve "how come it doesn't detect 
this", someone will highlight another problem that is harder or 
impossible to solve.

-Steve
Nov 30 2018
prev sibling parent reply sanjayss <dummy dummy.dummy> writes:
On Tuesday, 27 November 2018 at 08:00:22 UTC, Sebastiaan Koppe 
wrote:

 I have a non-copyable struct and I really want a compiler error 
 whenever I access it after it has been moved.

 ---
 struct Handle {
     ...
      disable this(this);
     ...
 }

 void main() {
     import std.algorithm : move;
     auto handle = getOne();
     auto second = handle.move;  /// line 14
     auto third = handle.move;    ///  <- compiler error, 
 variable handle is invalid after line 14
 }
 ---

 I believe this would prevent some nasty bugs when dealing with 
 these structs.

 What do you think?


I have always wanted a feature in C that would let me explicitly 
tell the compiler that a variable is no longer in scope (some 
sort of unset of a variable). This would be useful to do 
defensive programming against use-after-free of pointers to 
allocated memory and such.

Though it would be nice to have the compiler auto-detect these 
kinds of things, maybe having the programmer explicitly request 
this might be way easier to implement.
Nov 28 2018
next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 28 November 2018 at 09:17:39 UTC, sanjayss wrote:

 I have always wanted a feature in C that would let me 
 explicitly tell the compiler that a variable is no longer in 
 scope (some sort of unset of a variable). This would be useful 
 to do defensive programming against use-after-free of pointers 
 to allocated memory and such.

 Though it would be nice to have the compiler auto-detect these 
 kinds of things, maybe having the programmer explicitly request 
 this might be way easier to implement.


Yeah. It would be awesome if the programmer could 'taint' a 
variable as invalid. Even if that doesn't propagate in all cases.
Nov 28 2018
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 28 November 2018 at 09:17:39 UTC, sanjayss wrote:

 On Tuesday, 27 November 2018 at 08:00:22 UTC, Sebastiaan Koppe 
 wrote:

 [...]


 I have always wanted a feature in C that would let me 
 explicitly tell the compiler that a variable is no longer in 
 scope (some sort of unset of a variable). This would be useful 
 to do defensive programming against use-after-free of pointers 
 to allocated memory and such.



{
     void* ptr = malloc(5);
}

// ptr no longer in scope
Nov 28 2018
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 28, 2018 at 04:49:02PM +0000, Atila Neves via Digitalmars-d wrote:

 On Wednesday, 28 November 2018 at 09:17:39 UTC, sanjayss wrote:

 On Tuesday, 27 November 2018 at 08:00:22 UTC, Sebastiaan Koppe wrote:

 [...]


 
 I have always wanted a feature in C that would let me explicitly
 tell the compiler that a variable is no longer in scope (some sort
 of unset of a variable). This would be useful to do defensive
 programming against use-after-free of pointers to allocated memory
 and such.


 
 
 {
     void* ptr = malloc(5);
 }
 
 // ptr no longer in scope


Yeah, this seems to be a not-very-well-known aspect of syntax common
across C, C++, Java, and D:

	ReturnType myFunc(Args args) {
		int var1;
		someCode();
		{
			int var2;
			someOtherCode();
		}
		// var2 no longer in scope, but var1 still is.
		{
			// Can reuse identifier 'var2' without conflict
			float var2;
			yetMoreCode();
		}
		etcetera();
	}

It's a very useful construct in ensuring that temporaries don't last
longer than they ought to. (Syntactically anyway... the compiler may or
may not actually translate that directly in the executable. But the
point is to avoid programmer slip-ups.)


T

-- 
In theory, there is no difference between theory and practice.
Nov 28 2018
parent sanjayss <dummy dummy.dummy> writes:
On Wednesday, 28 November 2018 at 16:58:32 UTC, H. S. Teoh wrote:

 Yeah, this seems to be a not-very-well-known aspect of syntax 
 common across C, C++, Java, and D:

 	ReturnType myFunc(Args args) {
 		int var1;
 		someCode();
 		{
 			int var2;
 			someOtherCode();
 		}
 		// var2 no longer in scope, but var1 still is.
 		{
 			// Can reuse identifier 'var2' without conflict
 			float var2;
 			yetMoreCode();
 		}
 		etcetera();
 	}

 It's a very useful construct in ensuring that temporaries don't 
 last longer than they ought to. (Syntactically anyway... the 
 compiler may or may not actually translate that directly in the 
 executable. But the point is to avoid programmer slip-ups.)


 T


Yes, you can get what I asked for using blocks, but there are use 
cases where you can't always use blocks effectively. Also there 
is the aesthetic aspect -- all the curly braces and indentation 
makes code annoying to read.
Nov 28 2018
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/28/18 11:49 AM, Atila Neves wrote:

 On Wednesday, 28 November 2018 at 09:17:39 UTC, sanjayss wrote:

 On Tuesday, 27 November 2018 at 08:00:22 UTC, Sebastiaan Koppe wrote:

 [...]


 I have always wanted a feature in C that would let me explicitly tell 
 the compiler that a variable is no longer in scope (some sort of unset 
 of a variable). This would be useful to do defensive programming 
 against use-after-free of pointers to allocated memory and such.


 
 
 {
      void* ptr = malloc(5);
 }
 
 // ptr no longer in scope
 


I think what the request is that you have variables with overlapping scope.

For example:

void *ptr1 = malloc(5);
void *ptr2 = malloc(5);
...
free(ptr1); // end ptr1 scope
...
free(ptr2); // end ptr2 scope

Which isn't possible by adding a nested scope.

But in any case, this doesn't fix all the problems anyway, you could 
have another alias to the same data, free that alias, and then ptr1 is 
still "valid".

-Steve
Nov 28 2018
parent reply Neia Neutuladh <neia ikeran.org> writes:
On Wed, 28 Nov 2018 12:47:07 -0500, Steven Schveighoffer wrote:

 But in any case, this doesn't fix all the problems anyway, you could
 have another alias to the same data, free that alias, and then ptr1 is
 still "valid".


Move constructors and init'ing out the moved variable address that, no?
Nov 28 2018
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/28/18 1:59 PM, Neia Neutuladh wrote:

 On Wed, 28 Nov 2018 12:47:07 -0500, Steven Schveighoffer wrote:

 But in any case, this doesn't fix all the problems anyway, you could
 have another alias to the same data, free that alias, and then ptr1 is
 still "valid".


 
 Move constructors and init'ing out the moved variable address that, no?
 


Not sure what you mean. What I'm talking about is this:

int *ptr1 = (int *)malloc(sizeof(int));
int *ptr2 = ptr1;

free(ptr2);

*ptr1 = 5;

If we have a hypothetical C compiler that prevents use of ptr2 after 
free, it probably doesn't know about the link to ptr1, to make that also 
unusable.

Indeed the runtime solution is much more possible.

-Steve
Nov 28 2018
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 28 November 2018 at 19:11:14 UTC, Steven 
Schveighoffer wrote:


 int *ptr1 = (int *)malloc(sizeof(int));
 int *ptr2 = ptr1;

 free(ptr2);

 *ptr1 = 5;

 If we have a hypothetical C compiler that prevents use of ptr2 
 after free, it probably doesn't know about the link to ptr1, to 
 make that also unusable.

 Indeed the runtime solution is much more possible.



For that (or rather, to prevent that) we have non-copyable types. 
Like the one in the OP.
Nov 28 2018
prev sibling parent Neia Neutuladh <neia ikeran.org> writes:
On Wed, 28 Nov 2018 14:11:14 -0500, Steven Schveighoffer wrote:

 Not sure what you mean. What I'm talking about is this:
 
 int *ptr1 = (int *)malloc(sizeof(int));
 int *ptr2 = ptr1;
 
 free(ptr2);
 
 *ptr1 = 5;


Yes, and the compiler can't detect this very often (at least not without 
an ownership / borrowing system like Rust or Vala).

Instead, you can eliminate aliasing, introduce reference counting or a 
garbage collector to avoid manually freeing, or use indirection to detect 
when the value is freed.

There are a bunch of solutions that work some of the time. The end result 
is a bit of code that doesn't fit with any of it and needs special caution.
Nov 28 2018