www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Is the following well defined and allowed?

reply Shachar Shemesh <shachar weka.io> writes:
a[0..10][] = a[5..15][];

I.e. - is it well defined to copy between overlapping slices? Does it 
matter if, in the copy, the source is overlapping the end or the 
beginning of the destination?
Mar 01 2018
parent reply David Nadlinger <code klickverbot.at> writes:
On Thursday, 1 March 2018 at 14:54:41 UTC, Shachar Shemesh wrote:

 I.e. - is it well defined to copy between overlapping slices?


No: https://dlang.org/spec/arrays.html#overlapping-copying

  —David
Mar 01 2018
parent reply ag0aep6g <anonymous example.com> writes:
On 03/01/2018 04:34 PM, David Nadlinger wrote:

 On Thursday, 1 March 2018 at 14:54:41 UTC, Shachar Shemesh wrote:

 I.e. - is it well defined to copy between overlapping slices?


 
 No: https://dlang.org/spec/arrays.html#overlapping-copying


Does that mean it has undefined behavior and should not be allowed in 
 safe code?
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 11:48 AM, ag0aep6g wrote:

 On 03/01/2018 04:34 PM, David Nadlinger wrote:

 On Thursday, 1 March 2018 at 14:54:41 UTC, Shachar Shemesh wrote:

 I.e. - is it well defined to copy between overlapping slices?


 No: https://dlang.org/spec/arrays.html#overlapping-copying


 
 Does that mean it has undefined behavior and should not be allowed in 
  safe code?


No, it means it's a runtime error.

https://run.dlang.io/is/xdiKxx

-Steve
Mar 01 2018
parent reply ag0aep6g <anonymous example.com> writes:
On Thursday, 1 March 2018 at 17:06:48 UTC, Steven Schveighoffer 
wrote:

 On 3/1/18 11:48 AM, ag0aep6g wrote:


[...]

 Does that mean it has undefined behavior and should not be 
 allowed in  safe code?


 No, it means it's a runtime error.


But then it's well-defined, like going beyond array bounds, no?
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 12:31 PM, ag0aep6g wrote:

 On Thursday, 1 March 2018 at 17:06:48 UTC, Steven Schveighoffer wrote:

 On 3/1/18 11:48 AM, ag0aep6g wrote:


 [...]

 Does that mean it has undefined behavior and should not be allowed in 
  safe code?


 No, it means it's a runtime error.


 
 But then it's well-defined, like going beyond array bounds, no?


Yes it behaves just like array bounds. No it's not well-defined if you 
disable asserts.

-Steve
Mar 01 2018
parent reply ag0aep6g <anonymous example.com> writes:
On Thursday, 1 March 2018 at 19:05:26 UTC, Steven Schveighoffer 
wrote:

 Yes it behaves just like array bounds. No it's not well-defined 
 if you disable asserts.


Right. So it's defined to throw an Error in  safe code, and it 
has undefined behavior in  system code. The spec should say this.
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 3:06 PM, ag0aep6g wrote:

 On Thursday, 1 March 2018 at 19:05:26 UTC, Steven Schveighoffer wrote:

 Yes it behaves just like array bounds. No it's not well-defined if you 
 disable asserts.


 
 Right. So it's defined to throw an Error in  safe code, and it has 
 undefined behavior in  system code. The spec should say this.


No.

----
version(dosafe)
{
     void main()  safe
     {
         int[4] arr;
         arr[0 .. 3] = arr[1 .. 4];
     }
}
else
{
     void main()
     {
         int[4] arr;
         arr[0 .. 3] = arr[1 .. 4];
     }
}
---

dmd -version=dosafe -noboundscheck -run testarrayoverlap.d => no error, 
undefined behavior
dmd -run testarrayoverlap.d => error

 safe has nothing to do with it.

-Steve
Mar 01 2018
parent reply ag0aep6g <anonymous example.com> writes:
On Thursday, 1 March 2018 at 20:14:07 UTC, Steven Schveighoffer 
wrote:

 dmd -version=dosafe -noboundscheck -run testarrayoverlap.d => 
 no error, undefined behavior
 dmd -run testarrayoverlap.d => error

  safe has nothing to do with it.


 safe has everything to do with.  safe guarantees that there's no 
undefined behavior. If you can trigger UB in  safe code, that's a 
bug. The guarantee must hold whether asserts are enabled or not. 
-noboundscheck does not just disable asserts, it undermines 
 safe, rendering it meaningless.

For just disabling asserts (without affecting  safe), there's 
-release. With your same code, `dmd -release` => UB, but `dmd 
-release -version=dosafe` => Error.
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 3:24 PM, ag0aep6g wrote:

 On Thursday, 1 March 2018 at 20:14:07 UTC, Steven Schveighoffer wrote:

 dmd -version=dosafe -noboundscheck -run testarrayoverlap.d => no 
 error, undefined behavior
 dmd -run testarrayoverlap.d => error

  safe has nothing to do with it.


 
  safe has everything to do with.  safe guarantees that there's no 
 undefined behavior. If you can trigger UB in  safe code, that's a bug. 
 The guarantee must hold whether asserts are enabled or not. 
 -noboundscheck does not just disable asserts, it undermines  safe, 
 rendering it meaningless.


Yeah, it seems like -noboundscheck should never be used.

It's undefined behavior if the check is disabled. How you get the check 
disabled may be affected by  safe, but whether it's UB or not has 
nothing to do with  safe. It has to do with "I turned off the checks". 
Basically, in D, if your program *would have* thrown an Error, but 
didn't because you turned it off, the compiler is free to assume UB.


 
 For just disabling asserts (without affecting  safe), there's -release. 
 With your same code, `dmd -release` => UB, but `dmd -release 
 -version=dosafe` => Error.


I was unaware that it would work this way. I thought it would be the 
same whether you used  safe or not with -release.

-Steve
Mar 01 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 01, 2018 16:01:08 Steven Schveighoffer via Digitalmars-d 
wrote:

 On 3/1/18 3:24 PM, ag0aep6g wrote:

 On Thursday, 1 March 2018 at 20:14:07 UTC, Steven Schveighoffer wrote:

 dmd -version=dosafe -noboundscheck -run testarrayoverlap.d => no
 error, undefined behavior
 dmd -run testarrayoverlap.d => error

  safe has nothing to do with it.


  safe has everything to do with.  safe guarantees that there's no
 undefined behavior. If you can trigger UB in  safe code, that's a bug.
 The guarantee must hold whether asserts are enabled or not.
 -noboundscheck does not just disable asserts, it undermines  safe,
 rendering it meaningless.


 Yeah, it seems like -noboundscheck should never be used.


IMHO, it was a mistake to ever add it.


 For just disabling asserts (without affecting  safe), there's -release.
 With your same code, `dmd -release` => UB, but `dmd -release
 -version=dosafe` => Error.


 I was unaware that it would work this way. I thought it would be the
 same whether you used  safe or not with -release.


That depends on the feature. It's true for array bounds checking, but it's
not true for stuff like final switch statements - or apparently, this
situation. I don't know if that should be changed or not. Probably.

- Jonathan M Davis
Mar 01 2018
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On Thursday, 1 March 2018 at 21:01:08 UTC, Steven Schveighoffer 
wrote:

 Yeah, it seems like -noboundscheck should never be used.


Agreed.


 It's undefined behavior if the check is disabled. How you get 
 the check disabled may be affected by  safe, but whether it's 
 UB or not has nothing to do with  safe. It has to do with "I 
 turned off the checks". Basically, in D, if your program *would 
 have* thrown an Error, but didn't because you turned it off, 
 the compiler is free to assume UB.


You're looking at the behavior of the compiled executable. Then 
it makes sense to say that a program compiled with the checks has 
defined behavior (throwing Errors) and a program without the 
checks does something undefined (because the compiler manual 
doesn't say that anything in particular is supposed to happen).

That's not how I understand/use "undefined behavior". To me, it's 
the D source code that can have "undefined behavior". When it 
does, the program (as given in source code) is invalid and the 
compiler can do whatever it wants with it. It may:

1) reject the program statically, or it may
2) generate code that throws an Error at run time, or it may
3) generate code that does something arbitrary.

With that meaning of UB, out-of-bounds accesses and overlapping 
copies have undefined behavior in  system code, but they have 
defined behavior in  safe code.


whether you give -release or not. DMD is free to do this, because 
the source code has UB.


spec demands it.


 system and  safe code. The spec allows/encourages that.

In this interpetation, -noboundscheck switches DMD to a different 
dialect of D. In that dialect, out-of-bounds accesses (and 
overlapping copies, apparently) always have UB, in both  system 
and  safe code. That defeats the purpose of  safe. Which is why I 
don't really care for that dialect.
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 5:27 PM, ag0aep6g wrote:


 You're looking at the behavior of the compiled executable. Then it makes 
 sense to say that a program compiled with the checks has defined 
 behavior (throwing Errors) and a program without the checks does 
 something undefined (because the compiler manual doesn't say that 
 anything in particular is supposed to happen).


No, I'm looking at the source code.

At the very basic level, you have this:

assert(foo == 0);

Or whatever other condition you have. What this does is gives the 
compiler leeway to ASSUME foo is 0 at this point. It can make any number 
of optimizations assuming this. If foo is NOT equal to 0 at this point, 
then it is a program error, and the assumptions may cause bad things to 
happen.

When you compile in normal mode, this assert causes an Error to be 
thrown, before your code can do any damage.

When you compile without asserts, this causes undefined behavior.

The various switches and modes can say whether asserts or other checks 
are present or not, and if they are not, the compiler is going to 
execute your UB code with those wrong assumptions. But ONLY if the 
assert would have been false.

I extend this same treatment and logic to all checks: contracts, bounds 
checks, overlapping slice assign checks, etc. To me, they aren't any 
different than asserts. They aren't any different if they are in  safe 
code or  system code.


 That's not how I understand/use "undefined behavior". To me, it's the D 
 source code that can have "undefined behavior". When it does, the 
 program (as given in source code) is invalid and the compiler can do 
 whatever it wants with it. It may:
 
 1) reject the program statically, or it may
 2) generate code that throws an Error at run time, or it may
 3) generate code that does something arbitrary.
 
 With that meaning of UB, out-of-bounds accesses and overlapping copies 
 have undefined behavior in  system code, but they have defined behavior 
 in  safe code.


By default, both  system and  safe code have bounds checks. So by 
default, it is defined behavior (an Error is thrown) regardless of the 
safety.

If you turn the checks off (through any means) then it becomes UB.


 In this interpetation, -noboundscheck switches DMD to a different 
 dialect of D. In that dialect, out-of-bounds accesses (and overlapping 
 copies, apparently) always have UB, in both  system and  safe code. That 
 defeats the purpose of  safe. Which is why I don't really care for that 
 dialect.


I agree, I think we should remove the option to disable bounds checks on 
 safe code, in any way. It's too dangerous. If you want performance that 
comes without bounds checks, use a trusted escape, or write system code.

-Steve
Mar 02 2018
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 02.03.2018 15:39, Steven Schveighoffer wrote:

 
 

 In this interpetation, -noboundscheck switches DMD to a different 
 dialect of D. In that dialect, out-of-bounds accesses (and overlapping 
 copies, apparently) always have UB, in both  system and  safe code. 
 That defeats the purpose of  safe. Which is why I don't really care 
 for that dialect.


 
 I agree, I think we should remove the option to disable bounds checks on 
  safe code, in any way. It's too dangerous. If you want performance that 
 comes without bounds checks, use a trusted escape, or write system code.


I.e., the -release flag should not remove assertions in  safe code, or 
at the very least it should not turn them into sources of UB.
Mar 02 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/2/18 10:00 AM, Timon Gehr wrote:

 On 02.03.2018 15:39, Steven Schveighoffer wrote:

 In this interpetation, -noboundscheck switches DMD to a different 
 dialect of D. In that dialect, out-of-bounds accesses (and 
 overlapping copies, apparently) always have UB, in both  system and 
  safe code. That defeats the purpose of  safe. Which is why I don't 
 really care for that dialect.


 I agree, I think we should remove the option to disable bounds checks 
 on  safe code, in any way. It's too dangerous. If you want performance 
 that comes without bounds checks, use a trusted escape, or write 
 system code.


 
 I.e., the -release flag should not remove assertions in  safe code, or 
 at the very least it should not turn them into sources of UB.


-release flag already operates this way. It's the -noboundscheck or 
-boundscheck=off that causes problems.

-Steve
Mar 02 2018
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 02.03.2018 16:05, Steven Schveighoffer wrote:

 On 3/2/18 10:00 AM, Timon Gehr wrote:

 On 02.03.2018 15:39, Steven Schveighoffer wrote:

 In this interpetation, -noboundscheck switches DMD to a different 
 dialect of D. In that dialect, out-of-bounds accesses (and 
 overlapping copies, apparently) always have UB, in both  system and 
  safe code. That defeats the purpose of  safe. Which is why I don't 
 really care for that dialect.


 I agree, I think we should remove the option to disable bounds checks 
 on  safe code, in any way. It's too dangerous. If you want 
 performance that comes without bounds checks, use a trusted escape, 
 or write system code.


 I.e., the -release flag should not remove assertions in  safe code, or 
 at the very least it should not turn them into sources of UB.


 
 -release flag already operates this way.


That's not what the spec says:

https://dlang.org/dmd-linux.html#switch-release

"Compile release version, which means not emitting run-time checks for 
contracts and asserts. Array bounds checking is not done for system and 
trusted functions, and assertion failures are undefined behaviour."

Note that this is the only way to stop checking assertions. There is no 
option to just ignore them.


 It's the -noboundscheck or -boundscheck=off that causes problems.
 
 -Steve


At least the specification advises to use caution:

"off: Bounds checks are disabled completely (even in  safe code). This 
option should be used with caution and as a last resort to improve 
performance. Confirm turning off  safe bounds checks is worthwhile by 
benchmarking."
Mar 02 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/2/18 10:26 AM, Timon Gehr wrote:

 On 02.03.2018 16:05, Steven Schveighoffer wrote:

 On 3/2/18 10:00 AM, Timon Gehr wrote:

 On 02.03.2018 15:39, Steven Schveighoffer wrote:

 In this interpetation, -noboundscheck switches DMD to a different 
 dialect of D. In that dialect, out-of-bounds accesses (and 
 overlapping copies, apparently) always have UB, in both  system and 
  safe code. That defeats the purpose of  safe. Which is why I don't 
 really care for that dialect.


 I agree, I think we should remove the option to disable bounds 
 checks on  safe code, in any way. It's too dangerous. If you want 
 performance that comes without bounds checks, use a trusted escape, 
 or write system code.


 I.e., the -release flag should not remove assertions in  safe code, 
 or at the very least it should not turn them into sources of UB.


 -release flag already operates this way.


 
 That's not what the spec says:
 
 https://dlang.org/dmd-linux.html#switch-release
 
 "Compile release version, which means not emitting run-time checks for 
 contracts and asserts. Array bounds checking is not done for system and 
 trusted functions, and assertion failures are undefined behaviour."


I misunderstood, yes, assertions are turned off in  safe code, but not 
array bounds checking (which is what I thought you meant).

Yes, I think assertions should be kept in  safe code. It's weird to have 
array bounds checks kept, but not assertions (which is how you would do 
equivalent bounds checks in a custom type).


 Note that this is the only way to stop checking assertions. There is no 
 option to just ignore them.


I'm not sure what the purpose of just ignoring assertions and not 
contracts (which are made up of assertions anyway).

The boundscheck switch can control the bounds checking independently.


 

 It's the -noboundscheck or -boundscheck=off that causes problems.


 
 At least the specification advises to use caution:
 
 "off: Bounds checks are disabled completely (even in  safe code). This 
 option should be used with caution and as a last resort to improve 
 performance. Confirm turning off  safe bounds checks is worthwhile by 
 benchmarking."


I suppose the one reason to use -boundscheck=off is to check if it does 
save any performance, and then go in and manually use trusted escapes 
for those where it counts.

I'd rather have it named, however, -boundscheck=unsafe.

-Steve
Mar 02 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, March 02, 2018 11:25:00 Steven Schveighoffer via Digitalmars-d 
wrote:

 Yes, I think assertions should be kept in  safe code. It's weird to have
 array bounds checks kept, but not assertions (which is how you would do
 equivalent bounds checks in a custom type).


Then just don't compile with -release. I'd be a _lot_ less likely to use
assertions if I knew that the compiler was going to leave them in in release
mode. Assertions help find bugs, but they aren't doing anything to make the
code any more  safe unless some aspect of the compiler assuming that the
assertion is true when it's compiled out allows for the compiler to then
function in a manner that isn't  safe, which I doubt, but if it does, then
that needs to be fixed, not leaving in a bunch of extra checks intended for
catching bugs in development.

- Jonathan M Davis
Mar 02 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/2/18 1:21 PM, Jonathan M Davis wrote:

 On Friday, March 02, 2018 11:25:00 Steven Schveighoffer via Digitalmars-d
 wrote:

 Yes, I think assertions should be kept in  safe code. It's weird to have
 array bounds checks kept, but not assertions (which is how you would do
 equivalent bounds checks in a custom type).


 
 Then just don't compile with -release. I'd be a _lot_ less likely to use
 assertions if I knew that the compiler was going to leave them in in release
 mode. Assertions help find bugs, but they aren't doing anything to make the
 code any more  safe unless some aspect of the compiler assuming that the
 assertion is true when it's compiled out allows for the compiler to then
 function in a manner that isn't  safe, which I doubt, but if it does, then
 that needs to be fixed, not leaving in a bunch of extra checks intended for
 catching bugs in development.


As ag says,  safe code is supposed to be free of undefined behavior. If 
-release means it has undefined behavior, that is a problem. I'd much 
rather have assertions in release mode than undefined behavior in 
release mode.

The other option is to make sure the compiler cannot make any 
assumptions about the asserts when they are compiled out in  safe code.

-Steve
Mar 02 2018
prev sibling parent ag0aep6g <anonymous example.com> writes:
On 03/02/2018 03:39 PM, Steven Schveighoffer wrote:

 On 3/1/18 5:27 PM, ag0aep6g wrote:


[...]

 No, I'm looking at the source code.
 
 At the very basic level, you have this:
 
 assert(foo == 0);
 
 Or whatever other condition you have. What this does is gives the 
 compiler leeway to ASSUME foo is 0 at this point. It can make any number 
 of optimizations assuming this. If foo is NOT equal to 0 at this point, 
 then it is a program error, and the assumptions may cause bad things to 
 happen.
 
 When you compile in normal mode, this assert causes an Error to be 
 thrown, before your code can do any damage.
 
 When you compile without asserts, this causes undefined behavior.


Now you're looking beyond the source code. There's no "normal mode" or 
"without asserts mode" in D source code. What you've got in the D code 
is  safe and  system.

Looking at it your way makes sense when the question is how the actual 
generated executable will behave. But I think it's also valuable to look 
at the language in isolation, without compiler settings.

I'd put it this way:

1)  system code may have UB. For example, out-of-bounds accesses have UB 
in  system code.

2) A compiler may give guarantees on what it does with specific 
instances of UB. For example, DMD guarantees that it throws an Error on 
out-of-bounds accesses, unless you compile with -release. This makes it 
effectively implementation defined behavior. You can rely on this. But 
when considering the source code in isolation, it still has UB.

3)  safe code cannot have UB. This is a strong guarantee. This cannot be 
affected by compiler switches without violating the spec.

[...]

 I extend this same treatment and logic to all checks: contracts, bounds 
 checks, overlapping slice assign checks, etc. To me, they aren't any 
 different than asserts. They aren't any different if they are in  safe 
 code or  system code.


I don't agree with the  safe/ system part.  safe code must not exhibit 
UB. That's the point of the attribute.

Checks that would fail may cause UB in  system code, but they can't be 
allowed to cause UB in  safe code. Because nothing can be allowed to 
cause UB in  safe code.

[...]

 I agree, I think we should remove the option to disable bounds checks on 
  safe code, in any way. It's too dangerous. If you want performance that 
 comes without bounds checks, use a trusted escape, or write system code.


I'm not going to argue against that, obviously.
Mar 02 2018
prev sibling parent reply Nathan S. <no.public.email example.com> writes:
On Thursday, 1 March 2018 at 21:01:08 UTC, Steven Schveighoffer 
wrote:

 Yeah, it seems like -noboundscheck should never be used.


How good is DMD at omitting redundant bounds checks? I assume not 
much engineering effort has been put towards that due to 
"-boundscheck=off" being available.
Mar 01 2018
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
I do not know, but from my experience it is good at it. I have done many
benchmarks for plenty of code, and in recent D compilers -boundscheck=off
does not improve speed. To be fair using -boundscheck=off make D code
slower in many cases, which is wierd but true.

On Fri, Mar 2, 2018 at 8:48 AM, Nathan S. via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Thursday, 1 March 2018 at 21:01:08 UTC, Steven Schveighoffer wrote:


 Yeah, it seems like -noboundscheck should never be used.


 How good is DMD at omitting redundant bounds checks? I assume not much
 engineering effort has been put towards that due to "-boundscheck=off"
 being available.
Mar 02 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/2/18 3:36 AM, Daniel Kozak wrote:

 I do not know, but from my experience it is good at it. I have done many 
 benchmarks for plenty of code, and in recent D 
 compilers -boundscheck=off does not improve speed. To be fair 
 using -boundscheck=off make D code slower in many cases, which is wierd 
 but true.


If you are doing -release, then -boundscheck=off is redundant except for 
 safe code.

-Steve
Mar 02 2018