www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Typical security issues in C++: why the GC isn't your enemy

reply "H. S. Teoh" <hsteoh qfbox.info> writes:
In the past, I've posted about my impressions of coding issues
encountered in a large C codebase (approx 2M LOC), as found by Coverity.
My impression was that there was a large predominance of bugs related to
memory management and raw pointers.  However, I didn't have actual data
to back up my memory.  So I decided to do a slightly more evidence-based
analysis by doing a little informal analysis of the following list of
CVE issues in the Chromium browser, a commonly-used browser, from the
Debian/Linux security tracker page:

	https://security-tracker.debian.org/tracker/source-package/chromium

There are 1239 issues listed on this page, with several frequently-
recurring keywords / key phrases, that one could argue represents the
most commonly encountered issues in a typical large C++ codebase. Here
are some of keywords of interest with their counts:

Use after free:				423	(34%)
Insufficient policy enforcement:	159	(13%)
Inappropriate implementation:		149	(12%)
Buffer overflow:			98	(8%)
Data validation:			91	(7%)
Out of bounds access/read/write/etc:	71	(6%)
Type confusion (JS):			48	(4%)
Incorrect security UI:			33	(3%)
Integer overflow:			22	(2%)
Object lifecycle/lifetime:		15	(1%)
Uninitialized data/use:			14	(1%)
Information leak:			14	(1%)
Side-channel:				7	(<1%)
Handling of confusable characters:	7	(<1%)
Data race:				7	(<1%)
Double free:				3	(<1%)


Most interesting point here is that the largest category of bugs is
use-after-free bugs, constituting 34% of the reported issues.  (Arguably
we should include "object lifecycle/lifetime" in this category, but I
think those refer to bugs in the JS implementation. In any case, it
doesn't change the conclusion.)  This is strong evidence that memory
management is a major source of bugs, and a strong argument for GC use
in application code.

The next largest categories are insufficient policy enforcement and
inappropriate implementation (it's unclear what exactly the latter
means, at a glance it looks like various issues with JS and various
browser features).  I contend that these two categories could be lumped
together as application / business logic bugs.  Tellingly, these add up
to 25%, overshadowed by use-after-free bugs.

D's bounds checks are often touted as a major feature to prevent issues
with buffer overflow and out-of-bounds accesses.  Interestingly, "buffer
overflow" and "out of bounds..." add up only to about 14% of the total
issues.  Nothing to sneeze at, but nonetheless not as big an issue as
use-after-free bugs.

Integer overflow is also sometimes brought up as something important;
but at least according to the above categorization it only accounts for
2% of issues.  So not as big a deal as some may have made it sound.

Similarly, D's initialized-by-default variables are often touted as a
big thing, but overall issues with uninitialized variables only
constitute about 1% of the total issues.

I included also a few categories with small counts that are nonetheless
interesting: side-channel attacks, which in recent years have been
making noise in security circles, seem not as common as one might think
(<1% of total issues).  Also interesting is the "confusable character"
category: apparently this is increasingly being recognized as an issue
in today's climate of spoofers and online swindling. But it's still only
a rather minor category of issues.  Data races are also only a small
category, at least as far as Chromium is concerned.

Most interestingly, "double free" only has 3 counts of the total, less
than 1%, compared with "use after free", which constitute the largest
category of issues.  This seems to suggest that it's not memory
management in general that's necessarily problematic, but it's keeping
track of the *lifetime* of allocated memory.  One could say that this is
proof that lifetime is a complex problem. But again it's a strong
argument that the GC brings a major benefit: it relieves the programmer
from having to worry about lifetime issues.  You can instantly be freed
from 34% of security issues, if the above numbers are anything to go by.
:-P


T

-- 
If Java had true garbage collection, most programs would delete themselves upon
execution. -- Robert Sewell
Dec 05 2022
next sibling parent reply Sergey <kornburn yandex.ru> writes:
On Monday, 5 December 2022 at 19:57:39 UTC, H. S. Teoh wrote:

 but it's keeping track of the *lifetime* of allocated memory.  
 One could say that this is proof that lifetime is a complex 
 problem. But again it's a strong argument that the GC brings a 
 major benefit: it relieves


Thank you for interesting statistics and analysis. Unfortunately 
it seems that the IT community (biggest companies with largest 
code bases) stick with borrow-checker lifetime approach to solve 
those issues and not on GC.
Dec 05 2022
next sibling parent Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Monday, 5 December 2022 at 20:28:42 UTC, Sergey wrote:

 On Monday, 5 December 2022 at 19:57:39 UTC, H. S. Teoh wrote:

 but it's keeping track of the *lifetime* of allocated memory.  
 One could say that this is proof that lifetime is a complex 
 problem. But again it's a strong argument that the GC brings a 
 major benefit: it relieves


 Thank you for interesting statistics and analysis. 
 Unfortunately it seems that the IT community (biggest companies 
 with largest code bases) stick with borrow-checker lifetime 
 approach to solve those issues and not on GC.


I think you haven't explored quite entire domain IT is. Just 

find employment positions for them, and quite a lot.

Best regards,
Alexandru.
Dec 05 2022
prev sibling parent Sergey <kornburn yandex.ru> writes:
On Monday, 5 December 2022 at 20:28:42 UTC, Sergey wrote:

 On Monday, 5 December 2022 at 19:57:39 UTC, H. S. Teoh wrote:

 but it's keeping track of the *lifetime* of allocated memory.  
 One could say that this is proof that lifetime is a complex 
 problem. But again it's a strong argument that the GC brings a 
 major benefit: it relieves


 Thank you for interesting statistics and analysis. 
 Unfortunately it seems that the IT community (biggest companies 
 with largest code bases) stick with borrow-checker lifetime 
 approach to solve those issues and not on GC.


The story with recommendations from NSA about “safe” languages is 
continued..
Answer from Bjarne 
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2739r0.pdf
Jan 22 2023
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/5/22 20:57, H. S. Teoh wrote:

 Similarly, D's initialized-by-default variables are often touted as a
 big thing, but overall issues with uninitialized variables only
 constitute about 1% of the total issues.


Default initialization does not even fix all initialization issues, it 
just makes them reproducible. Anyway, I think neither default 
initialization nor uninitialized variables are the right solution, but 
you kind of have to do it this way given how scoping works in C++ and in D.
Dec 05 2022
next sibling parent drug007 <drug2004 bk.ru> writes:
06.12.2022 02:58, Timon Gehr пишет:

 Anyway, I think neither default initialization nor uninitialized variables are
the right solution


What is the right solution in your opinion?
Dec 06 2022
prev sibling next sibling parent reply Arjan <arjan ask.me.to> writes:
On Monday, 5 December 2022 at 23:58:58 UTC, Timon Gehr wrote:

 On 12/5/22 20:57, H. S. Teoh wrote:
 Default initialization does not even fix all initialization 
 issues, it just makes them reproducible. Anyway, I think 
 neither default initialization nor uninitialized variables are 
 the right solution, but you kind of have to do it this way 
 given how scoping works in C++ and in D.


Now I'm curious, what, in you opinion, would be best for 
initialization?
How is C++/D scoping limiting in this?
Dec 06 2022
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Tuesday, 6 December 2022 at 10:03:24 UTC, Arjan wrote:

 On Monday, 5 December 2022 at 23:58:58 UTC, Timon Gehr wrote:

 On 12/5/22 20:57, H. S. Teoh wrote:
 Default initialization does not even fix all initialization 
 issues, it just makes them reproducible. Anyway, I think 
 neither default initialization nor uninitialized variables are 
 the right solution, but you kind of have to do it this way 
 given how scoping works in C++ and in D.


 Now I'm curious, what, in you opinion, would be best for 
 initialization?
 How is C++/D scoping limiting in this?


Compiler can do control flow analysis, so they can force you to 
initialize things before you use them. This is the right solution 
to that problem.
Dec 06 2022
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2022 5:33 AM, deadalnix wrote:

 Compiler can do control flow analysis, so they can force you to initialize 
 things before you use them. This is the right solution to that problem.


Your are correct, but this isn't done because it makes compiles 10x or more
slower.
Dec 08 2022
next sibling parent reply Zealot <no2 no.no> writes:
On Thursday, 8 December 2022 at 20:52:26 UTC, Walter Bright wrote:

 On 12/6/2022 5:33 AM, deadalnix wrote:

 Compiler can do control flow analysis, so they can force you 
 to initialize things before you use them. This is the right 
 solution to that problem.


 Your are correct, but this isn't done because it makes compiles 
 10x or more slower.


would it be possible to have it behind a compiler switch if it's 
a useful tool to catch bugs. it's enough to run extra checks one 
before release.
Dec 08 2022
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2022 1:02 PM, Zealot wrote:

 would it be possible to have it behind a compiler switch if it's a useful tool 
 to catch bugs. it's enough to run extra checks one before release.


I used to have it when the optimizer was run, but people just didn't like 
sometimes seeing the error and sometimes not.
Dec 08 2022
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 8 December 2022 at 20:52:26 UTC, Walter Bright wrote:

 On 12/6/2022 5:33 AM, deadalnix wrote:

 Compiler can do control flow analysis, so they can force you 
 to initialize things before you use them. This is the right 
 solution to that problem.


 Your are correct, but this isn't done because it makes compiles 
 10x or more slower.


I do not expect this to be that expensive. It is simpler than the 
escape analysis that is being baked in the language right now, 
and roughly on par with keeping track of constructors/destructors.
Dec 08 2022
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2022 4:24 PM, deadalnix wrote:

 I do not expect this to be that expensive. It is simpler than the escape 
 analysis that is being baked in the language right now, and roughly on par
with 
 keeping track of constructors/destructors.


I've done it. It's expensive. It is not on par with ctors/dtors which rely on 
FILO scope analysis, not data flow.

D's borrow checker does do DFA, and it's slow, and one of the reasons why it's 
active only with  live annotations. Rust uses DFA, and is notoriously slow.

D always initializes variables, and then the (optional) optimization pass 
removes the unneeded ones. This is a reasonable approach.
Dec 08 2022
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 9 December 2022 at 01:55:22 UTC, Walter Bright wrote:

 On 12/8/2022 4:24 PM, deadalnix wrote:

 I do not expect this to be that expensive. It is simpler than 
 the escape analysis that is being baked in the language right 
 now, and roughly on par with keeping track of 
 constructors/destructors.


 I've done it. It's expensive. It is not on par with ctors/dtors 
 which rely on FILO scope analysis, not data flow.

 D's borrow checker does do DFA, and it's slow, and one of the 
 reasons why it's active only with  live annotations. Rust uses 
 DFA, and is notoriously slow.

 D always initializes variables, and then the (optional) 
 optimization pass removes the unneeded ones. This is a 
 reasonable approach.


I would suggest that the way to was done probably wasn't the 
right way. Knowing how DMD works, I suspect this is done on an 
AST representation, and this is definitively hard to do it on 
such representation. This is much easier to do on an SSA or Sea 
of Node style representation.

It's fairly easy to convince oneself this is fast, any modern JIT 
does it.
Dec 09 2022
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/9/2022 2:49 PM, deadalnix wrote:

 I would suggest that the way to was done probably wasn't the right way.
Knowing 
 how DMD works, I suspect this is done on an AST representation, and this is 
 definitively hard to do it on such representation. This is much easier to do
on 
 an SSA or Sea of Node style representation.
 
 It's fairly easy to convince oneself this is fast, any modern JIT does it.


D's optimizer runs on the intermediate code, not the AST. It's as fast as it's 
going to get, but is still slow enough that the optimizer is enabled only with
a 
switch.

(The optimizer was built in the DOS days, and you can bet I spent a great deal 
of time trying to speed it up. It had to work in reasonable time for 16 bit 
computers. It uses the bit vector approach, recommended by Hennessy&Ullman for 
optimization algorithms.)

Modern jits work on demand (that's why they're called "Just In Time"), one 
function at a time, which spreads the time spent optimizing out to the point 
where one doesn't notice it so much. A native compiler does the *entire
program* 
at the same time.
Dec 11 2022
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/6/22 11:03, Arjan wrote:

 On Monday, 5 December 2022 at 23:58:58 UTC, Timon Gehr wrote:

 On 12/5/22 20:57, H. S. Teoh wrote:
 Default initialization does not even fix all initialization issues, it 
 just makes them reproducible. Anyway, I think neither default 
 initialization nor uninitialized variables are the right solution, but 
 you kind of have to do it this way given how scoping works in C++ and 
 in D.


 
 Now I'm curious, what, in you opinion, would be best for initialization?


Ideally you just eliminate those cases where a programmer feels like 
they have to leave a variable uninitialized. The whole concept of 
"uninitialized variable" does not make a whole lot of sense from the 
perspective of a safe high-level programming language.


 How is C++/D scoping limiting in this?
 


Variables are scoped within the innermost block that they are declared 
in. Languages like Python that don't have block-local scoping just don't 
have this particular problem (there's plenty of things to dislike about 
Python, but this is something it got right I think):

```python

if cond:
     x = f()
else:
     x = g()
print(x)
```

A particularly egregious case is the do-while loop:

```d
do{
     int x=4;
     if(condition){
         ...
         x++;
     }
     ...
}while(x<10); // error
```

Just... why? x)
Dec 06 2022
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2022 3:58 PM, Timon Gehr wrote:

 Variables are scoped within the innermost block that they are declared in. 
 Languages like Python that don't have block-local scoping just don't have this 
 particular problem (there's plenty of things to dislike about Python, but this 
 is something it got right I think):
 
 ```python

 if cond:
      x = f()
 else:
      x = g()
 print(x)
 ```


That looks like a matter of taste rather than righteousness.



 A particularly egregious case is the do-while loop:
 
 ```d
 do{
      int x=4;
      if(condition){
          ...
          x++;
      }
      ...
 }while(x<10); // error
 ```
 
 Just... why? x)


Because we love to annoy people.
Dec 08 2022
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/8/22 21:57, Walter Bright wrote:

 On 12/6/2022 3:58 PM, Timon Gehr wrote:

 Variables are scoped within the innermost block that they are declared 
 in. Languages like Python that don't have block-local scoping just 
 don't have this particular problem (there's plenty of things to 
 dislike about Python, but this is something it got right I think):

 ```python

 if cond:
      x = f()
 else:
      x = g()
 print(x)
 ```


 
 That looks like a matter of taste rather than righteousness.
 ...


I am quite confident that C scoping is not how it should be done, then 
people asked what I think is the right solution, so I provided that.

Almost everything is a trade-off, but this fixes the issue without 
introducing spurious behavior, limitations or syntactic incantations.

D's solution of "just put a null state into every type" is probably 
close to my least favorite part of the language. Yes, that's a 
preference, but it's backed up by more reasoning than "YMMV".


 

 A particularly egregious case is the do-while loop:

 do{
      int x=4;
      if(condition){
          ...
          x++;
      }
      ...
 }while(x<10); // error
 ```

 Just... why? x)


 
 Because we love to annoy people.
 


Well, for D specifically, I hope the answer is just that the reason is 
that C does it this way.
Dec 08 2022
next sibling parent reply Dom DiSc <dominikus scherkl.de> writes:
On Thursday, 8 December 2022 at 23:30:56 UTC, Timon Gehr wrote:

 A particularly egregious case is the do-while loop:

 do{
      int x=4;
      if(condition){
          ...
          x++;
      }
      ...
 }while(x<10); // error
 ```

 Just... why? x)


 
 Because we love to annoy people.
 


 Well, for D specifically, I hope the answer is just that the 
 reason is that C does it this way.


Well, declaring variables within a loop is just awful style, 
don't do that!
And it will also not work with for-loops, but may be somewhat 
more obvious for humans:

for(int i = 0; i < x; ++i) // error: x undeclared
{
    int x = 10;
    ...
}
Dec 09 2022
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12/9/22 10:43, Dom DiSc wrote:

 On Thursday, 8 December 2022 at 23:30:56 UTC, Timon Gehr wrote:

 A particularly egregious case is the do-while loop:

 do{
      int x=4;
      if(condition){
          ...
          x++;
      }
      ...
 }while(x<10); // error
 ```

 Just... why? x)


 Because we love to annoy people.


 Well, for D specifically, I hope the answer is just that the reason is 
 that C does it this way.


 
 Well, declaring variables within a loop is just awful style, don't do that!


Nonsense.


 And it will also not work with for-loops, but may be somewhat more 
 obvious for humans:
 
 for(int i = 0; i < x; ++i) // error: x undeclared
 {
     int x = 10;
     ...
 }
 
 


What is your point? That's a completely different case, you are 
accessing `x` before it is even declared.
Dec 09 2022
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2022 3:30 PM, Timon Gehr wrote:

 I hope the answer is just that the reason is that C 
 does it this way.


Yes.
Dec 09 2022
prev sibling parent Quirin Schroll <qs.il.paperinik gmail.com> writes:
On Tuesday, 6 December 2022 at 23:58:08 UTC, Timon Gehr wrote:

 On 12/6/22 11:03, Arjan wrote:

 On Monday, 5 December 2022 at 23:58:58 UTC, Timon Gehr wrote:

 On 12/5/22 20:57, H. S. Teoh wrote:
 Default initialization does not even fix all initialization 
 issues, it just makes them reproducible. Anyway, I think 
 neither default initialization nor uninitialized variables 
 are the right solution, but you kind of have to do it this 
 way given how scoping works in C++ and in D.


 
 Now I'm curious, what, in you opinion, would be best for 
 initialization?


 Ideally you just eliminate those cases where a programmer feels 
 like they have to leave a variable uninitialized. The whole 
 concept of "uninitialized variable" does not make a whole lot 
 of sense from the perspective of a safe high-level programming 
 language.


 How is C++/D scoping limiting in this?
 


 Variables are scoped within the innermost block that they are 
 declared in. Languages like Python that don't have block-local 
 scoping just don't have this particular problem (there's plenty 
 of things to dislike about Python, but this is something it got 
 right I think):

 ```python

 if cond:
     x = f()
 else:
     x = g()
 print(x)
 ```

 A particularly egregious case is the do-while loop:

 ```d
 do{
     int x=4;
     if(condition){
         ...
         x++;
     }
     ...
 }while(x<10); // error
 ```

 Just... why? x)


Maybe this could be addressed by giving `do` an optional 
initializer like `for`. Probably, we’d want this:
```d
do (int x = 4)
{
     if (…) { … x++; }
}
while (x < 10);
```
But it’s likely an issue for parsing. A more parseable and more 
general construct would be
```d
do (int x = 4; x++)
{
     if (…) { … x++; }
}
while (x < 10);
```
Which is effectively a `for` loop, but the condition `x < 10` is 
checked first after the first iteration and put lexically at the 
end.

I found myself wishing for a construct like that a few times.
Feb 13 2023
prev sibling next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Monday, 5 December 2022 at 23:58:58 UTC, Timon Gehr wrote:

 On 12/5/22 20:57, H. S. Teoh wrote:

 Similarly, D's initialized-by-default variables are often 
 touted as a
 big thing, but overall issues with uninitialized variables only
 constitute about 1% of the total issues.


 Default initialization does not even fix all initialization 
 issues, it just makes them reproducible. Anyway, I think 
 neither default initialization nor uninitialized variables are 
 the right solution, but you kind of have to do it this way 
 given how scoping works in C++ and in D.


I wouldn't see lack of default initialization as a source of 
bugs, rather an attack vector.  It isn't a concern that there are 
uninitialized data pointing to garbage causing your program to do 
something wild/unexpected.  The concern is when it might point to 
useful information.
Dec 06 2022
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12/6/22 19:24, Iain Buclaw wrote:

 On Monday, 5 December 2022 at 23:58:58 UTC, Timon Gehr wrote:

 On 12/5/22 20:57, H. S. Teoh wrote:

 Similarly, D's initialized-by-default variables are often touted as a
 big thing, but overall issues with uninitialized variables only
 constitute about 1% of the total issues.


 Default initialization does not even fix all initialization issues, it 
 just makes them reproducible. Anyway, I think neither default 
 initialization nor uninitialized variables are the right solution, but 
 you kind of have to do it this way given how scoping works in C++ and 
 in D.


 
 I wouldn't see lack of default initialization as a source of bugs, 
 rather an attack vector.  It isn't a concern that there are 
 uninitialized data pointing to garbage causing your program to do 
 something wild/unexpected.  The concern is when it might point to useful 
 information.


True, that's a concern (default initialization does fix _some_ issues 
around initialization).
Dec 06 2022
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/6/2022 10:24 AM, Iain Buclaw wrote:

 I wouldn't see lack of default initialization as a source of bugs, rather an 
 attack vector.  It isn't a concern that there are uninitialized data pointing
to 
 garbage causing your program to do something wild/unexpected.  The concern is 
 when it might point to useful information.


Default initializing it to null doesn't present an attack vector.
Dec 08 2022
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2022 3:58 PM, Timon Gehr wrote:

 Default initialization does not even fix all initialization issues, it just 
 makes them reproducible.


As I mentioned in another post, making them reproducible is a huge deal.
Leaving 
variables uninitialized is Heisenbug City. Testing finds the reproducible ones, 
not the Heisenbugs.
Dec 08 2022
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12/8/22 21:47, Walter Bright wrote:

 On 12/5/2022 3:58 PM, Timon Gehr wrote:

 Default initialization does not even fix all initialization issues, it 
 just makes them reproducible.


 
 As I mentioned in another post, making them reproducible is a huge deal.


Absolutely, I agree with that. It's just that I think eliminating them 
statically makes even more sense.


 Leaving variables uninitialized is Heisenbug City. Testing finds the 
 reproducible ones, not the Heisenbugs.
 


Yes. (This is why I don't want UB in general.)
Dec 08 2022
prev sibling next sibling parent reply Tejas <notrealemail gmail.com> writes:
On Monday, 5 December 2022 at 19:57:39 UTC, H. S. Teoh wrote:

 In the past, I've posted about my impressions of coding issues 
 encountered in a large C codebase (approx 2M LOC), as found by 
 Coverity. My impression was that there was a large predominance 
 of bugs related to memory management and raw pointers.  
 However, I didn't have actual data to back up my memory.  So I 
 decided to do a slightly more evidence-based analysis by doing 
 a little informal analysis of the following list of CVE issues 
 in the Chromium browser, a commonly-used browser, from the 
 Debian/Linux security tracker page:

 [...]



People will resort have to resort to ` nogc` stuff when writing 
performance critical code anyways, so the protection from `D`'s 
GC will go away in those cases; and Chrome engineers aren't the 
kind of folk who will find the Rust learning curve to be steep.

It will most likely be a competition between Rust and Carbon, as 
far as Chrome's future is concerned, and that is even assuming 
that they're interested in changing languages
Dec 05 2022
next sibling parent Commander Zot <no no.no> writes:
On Tuesday, 6 December 2022 at 03:13:41 UTC, Tejas wrote:

 On Monday, 5 December 2022 at 19:57:39 UTC, H. S. Teoh wrote:



 People will resort have to resort to ` nogc` stuff when writing 
 performance critical code anyways, so the protection from `D`'s 
 GC will go away in those cases; and Chrome engineers aren't the 
 kind of folk who will find the Rust learning curve to be steep.


This is just wrong. if you have performance critical code, you 
won't allocate in it. so you can just slap  nogc on it without 
problems.
the rest of your code can still use the gc just fine.
Dec 06 2022
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2022 7:13 PM, Tejas wrote:

 People will resort have to resort to ` nogc` stuff when writing performance 
 critical code anyways, so the protection from `D`'s GC will go away in those 
 cases;


The myths about D's GC never seem to die.

D's GC is NEVER, EVER run unless you allocate with the GC. You don't need  nogc 
to disable the GC. Just don't use it. Performance critical code (well-written 
code) does not allocate within the critical sections anyway, so the GC will
have 
ZERO effect on it.
Dec 08 2022
prev sibling next sibling parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Monday, 5 December 2022 at 19:57:39 UTC, H. S. Teoh wrote:

 D's bounds checks are often touted as a major feature to 
 prevent issues with buffer overflow and out-of-bounds accesses. 
  Interestingly, "buffer overflow" and "out of bounds..." add up 
 only to about 14% of the total issues.  Nothing to sneeze at, 
 but nonetheless not as big an issue as use-after-free bugs.


This doesn't mean that there are fewer "out of bounds" bugs in 
general. But they are less likely to become exploitable security 
vulnerabilities. The "use after free" bugs are relatively easily 
to exploit:
   1. The old buffer is freed.
   2. This part of memory is later allocated for storing some 
security sensitive information.
   3. Access via the old pointer results in leaking the sensitive 
information.

GC surely helps to mitigate the security aspect of this. But the 
bug in the application logic may still be there and only manifest 
itself as a memory leak (if the application still keeps a pointer 
to the data that is not supposed to be used anymore). A good 
coding practice in C/C++ is to explicitly set a pointer to NULL 
immediately after deallocating memory.


 Integer overflow is also sometimes brought up as something 
 important; but at least according to the above categorization 
 it only accounts for 2% of issues.  So not as big a deal as 
 some may have made it sound.


First, not every bug easily results in an exploitable security 
vulnerability. It's the same as with the "buffer overflow" vs. 
"use after free" case. And more importantly, C/C++ compilers are 
just safer than D compilers when it comes to integer overflows 
detection. Chromium developers are actively using 
[UBSAN](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html), which
even allows detecting unsigned integer overflows:

     -fsanitize=unsigned-integer-overflow: Unsigned integer 
overflow, where the result
     of an unsigned integer computation cannot be represented in 
its type. Unlike signed
     integer overflow, this is not undefined behavior, but it is 
often unintentional.

You can grep Chromium source code for `no_sanitize` or even 
`__attribute__((no_sanitize("unsigned-integer-overflow")))` if 
you don't believe me.

Many of the integer overflow bugs are caught by the C++ compiler 
via UBSAN during the development and never reach the end users. 
While D compilers don't offer any reasonable protection. Except 
for GDC, which supports `-ftrapv` option as an undocumented 
"Easter egg".
Dec 05 2022
next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On Tuesday, 6 December 2022 at 04:35:18 UTC, Siarhei Siamashka 
wrote:

 Many of the integer overflow bugs are caught by the C++ 
 compiler via UBSAN during the development and never reach the 
 end users. While D compilers don't offer any reasonable 
 protection. Except for GDC, which supports `-ftrapv` option as 
 an undocumented "Easter egg".


It isn't undocumented, and it certainly isn't an easter egg.  All 
GCC optimization and code generation options are common to all 
front-end languages (C/C++/D/Fortran/Go/Rust/...)

It's mentioned right there in the [first 
sentence](https://gcc.gnu.org/onlinedocs/gdc/Invoking-gdc.html), 
and the [documentation for 
-ftrapv](https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#index-ftrapv)
is just two page links away.

The default behaviour is `-fwrapv` because that is what [the D 
spec asks for](https://dlang.org/spec/intro.html#arithmetic) 
(also mentioned in the spec for [binary+, 
binary-](https://dlang.org/spec/expression.html#add_expressions) 
and 
[binary*](https://dlang.org/spec/expression.html#mul_expressions) 
operators specifically).
Dec 06 2022
prev sibling next sibling parent reply Sergey <kornburn yandex.ru> writes:
On Tuesday, 6 December 2022 at 04:35:18 UTC, Siarhei Siamashka 
wrote:

 end users. While D compilers don't offer any reasonable 
 protection. Except for GDC, which supports `-ftrapv` option as 
 an undocumented "Easter egg".


What about these modules?
https://dlang.org/phobos/std_checkedint.html
https://dlang.org/phobos/core_checkedint.html
Dec 06 2022
parent Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Tuesday, 6 December 2022 at 22:26:17 UTC, Sergey wrote:

 On Tuesday, 6 December 2022 at 04:35:18 UTC, Siarhei Siamashka 
 wrote:

 end users. While D compilers don't offer any reasonable 
 protection. Except for GDC, which supports `-ftrapv` option as 
 an undocumented "Easter egg".


 What about these modules?
 https://dlang.org/phobos/std_checkedint.html
 https://dlang.org/phobos/core_checkedint.html


Imagine that you have several millions of D code (a big popular 
browser) and you want to do something to safeguard against 
integer overflow bugs and security issues (or at least mitigate 
them). How would these modules help?

In my opinion, the `std.checkedint` module is completely useless 
and there are no practical scenarios in the real world where it 
can help in any meaningful way. I see no real alternative to 
`-ftrapv` or UBSan for integer overflows diagnostics in large 
software projects. But the `core.checkedint` module is surely 
useful *after* you already know the exact part of the code where 
the overflow happens and needs to be patched up.

My reply would be incomplete without mentioning that D compilers 
do have some limited static analysis at compile time, intended to 
improve integer overflows safety: 
https://dlang.org/spec/type.html#vrp
But this analysis doesn't catch everything and it also sometimes 
unnecessarily gets in the way by forcing type casts. So it's not 
good enough. Want to see it in action? Here's one example:

```D
import std.stdio;
void main() {
   long bigsum = 0;
   int min_a = int.max - 50, max_a = int.max - 1;
   int min_b = 50, max_b = 100;
   foreach (a ; min_a .. max_a + 1) {
     foreach (b ; min_b .. max_b + 1) {
       // Compiles fine, but overflows at runtime
       int a_plus_b = a + b;
       bigsum += a_plus_b;
     }
   }
   byte min_c = 1, max_c = 50;
   byte min_d = 1, max_d = 50;
   foreach (c ; min_c .. max_c + 1) {
     foreach (d ; min_d .. max_d + 1) {
       // Won't compile without this explicit cast
       byte c_plus_d = cast(byte)(c + d);
       bigsum += c_plus_d;
     }
   }
   writeln(bigsum);
}
```

```
$ gdc -ftrapv test.d && ./a.out
Aborted

$ gdc test.d && ./a.out
-5471788083929
```
Dec 06 2022
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2022 8:35 PM, Siarhei Siamashka wrote:

 Many of the integer overflow bugs are caught by the C++ compiler via UBSAN 
 during the development and never reach the end users.


While that is a good option to have on the compiler, it will only never reach 
the end users if there is a test case that would trigger an overflow.
Dec 08 2022
parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Friday, 9 December 2022 at 02:11:13 UTC, Walter Bright wrote:

 On 12/5/2022 8:35 PM, Siarhei Siamashka wrote:

 Many of the integer overflow bugs are caught by the C++ 
 compiler via UBSAN during the development and never reach the 
 end users.


 While that is a good option to have on the compiler, it will 
 only never reach the end users if there is a test case that 
 would trigger an overflow.


https://en.wikipedia.org/wiki/Fuzzing is typically how you get 
these testcases in the real world. Human beta testers running 
debug builds with a bunch of extra runtime checks enabled may 
catch something too.

All of this indeed doesn't guarantee absolute 100% safety and 
that's the reason why we still see integer overflow security 
issues showing up in the stats. But without UBSAN actually 
existing and being actively used, the share of integer overflow 
issues could have been larger than 2%.

Now imagine some users in a C++ forum discussing the list of 
security issues in some large D application. Just like H. S. Teoh 
in the first message of this thread, somebody in this C++ forum 
could come up with the following statement: *"Bounds checking is 
also sometimes brought up as something important; but at least 
according to the above categorization it only accounts for X% of 
issues.  So not as big a deal as some may have made it sound."* 
;-)
Dec 08 2022
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/8/2022 7:07 PM, Siarhei Siamashka wrote:

 Now imagine some users in a C++ forum discussing the list of security issues
in 
 some large D application. Just like H. S. Teoh in the first message of this 
 thread, somebody in this C++ forum could come up with the following statement: 
 *"Bounds checking is also sometimes brought up as something important; but at 
 least according to the above categorization it only accounts for X% of issues.
 
 So not as big a deal as some may have made it sound."* ;-)


Bounds checking is always done, so is not dependent on test coverage to find
the 
bounds check bugs.

But I do agree that an option to insert arithmetic overflow checking would be a 
good thing.

What's also a good thing is an ability to mark certain calculations as "always 
check for overflow". You can see that in the D compiler source code in various 
places.
Dec 08 2022
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 9 December 2022 at 03:40:56 UTC, Walter Bright wrote:

 Bounds checking is always done


As long as you never use -release!
Dec 09 2022
parent reply bauss <jacobbauss gmail.com> writes:
On Friday, 9 December 2022 at 12:57:48 UTC, Adam D Ruppe wrote:

 On Friday, 9 December 2022 at 03:40:56 UTC, Walter Bright wrote:

 Bounds checking is always done


 As long as you never use -release!


Should never be used regardless of what you do! It's broken and 
misleading.
Dec 09 2022
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/9/2022 5:03 AM, bauss wrote:

 On Friday, 9 December 2022 at 12:57:48 UTC, Adam D Ruppe wrote:

 On Friday, 9 December 2022 at 03:40:56 UTC, Walter Bright wrote:

 Bounds checking is always done


 As long as you never use -release!


 
 Should never be used regardless of what you do! It's broken and misleading.


Bounds checking is not disabled with -release
Dec 09 2022
next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Fri, Dec 09, 2022 at 02:22:36PM -0800, Walter Bright via Digitalmars-d wrote:

 On 12/9/2022 5:03 AM, bauss wrote:

 On Friday, 9 December 2022 at 12:57:48 UTC, Adam D Ruppe wrote:

 On Friday, 9 December 2022 at 03:40:56 UTC, Walter Bright wrote:

 Bounds checking is always done


 
 As long as you never use -release!


 
 Should never be used regardless of what you do! It's broken and
 misleading.


 
 Bounds checking is not disabled with -release


According to `dmd -h`, they are disabled everywhere except  safe code.
That's still a pretty wide area where things could go wrong.  And
arguably,  system code is precisely where you WANT to have bounds
checks.


T

-- 
A program should be written to model the concepts of the task it performs
rather than the physical world or a process because this maximizes the
potential for it to be applied to tasks that are conceptually similar and, more
important, to tasks that have not yet been conceived. -- Michael B. Allen
Dec 09 2022
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/9/2022 2:39 PM, H. S. Teoh wrote:

 According to `dmd -h`, they are disabled everywhere except  safe code.


Safe code is where bounds checking is needed.


 And arguably,  system code is precisely where you WANT to have bounds
 checks.


Use of  system code should be minimized and in it you're allowed to do whatever 
you want.
Dec 11 2022
next sibling parent youSureAboutThat <youSureAboutThat gmail.com> writes:
On Monday, 12 December 2022 at 03:48:26 UTC, Walter Bright wrote:

 On 12/9/2022 2:39 PM, H. S. Teoh wrote:

 According to `dmd -h`, they are disabled everywhere except 
  safe code.


 Safe code is where bounds checking is needed.


True, it's needed in  safe, but possibly warranted nonetheless in 
 system as well.

Of course bounds checking at runtime will always degrade 
performance.

So the question for  system code - which presumably is being 
written and optimised for performance (over safety), is whether 
the developer(s) can be bothered measuring the performace impact 
of run-time bounds checks, and then, if they can be bothered, 
subsequently deciding whether that performace impact is 
sufficient to do away with the memory related safety guarantees 
provided by those run-time bounds checks.

My guess is developers writing  system code will not even bother 
to measure - and thus the cycle goes: on and on and on..
Dec 11 2022
prev sibling next sibling parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Monday, 12 December 2022 at 03:48:26 UTC, Walter Bright wrote:

 On 12/9/2022 2:39 PM, H. S. Teoh wrote:

 According to `dmd -h`, they are disabled everywhere except 
  safe code.


 Safe code is where bounds checking is needed.


 And arguably,  system code is precisely where you WANT to have 
 bounds
 checks.


 Use of  system code should be minimized and in it you're 
 allowed to do whatever you want.


You have my 100% support here. But the tricky thing is that the 
majority of the current D language community seems to disagree 
with us, as evidenced by the apparent rejection of DIP 1028. And 
we should do a much better job convincing them to change their 
mind.

What kind of dangers are threatening the users of  system code? 
This needs to be explained better. Especially considering that 
every beginner pretty much starts using  system code by default 
even before knowing that there's a choice.

=== use after free ===

Right from the top of the Chromium list, the "use after free" 
security problem. Have a look at [this forum 
thread](https://forum.dlang.org/post/wssohnwydjgsouywbitk forum.dlang.org).
Presumably a beginner is asking about how to deal with a c-string returned by
the sqlite3 library. And many of the forum regulars are very happily suggesting
to use `std.conv.fromStringz` or some other ways of turning the pointer into a
slice while avoiding a GC allocation. One person even believed that
`std.conv.fromStringz` is doing a GC allocation. Now if this string is saved
somewhere instead of just printing it, then we potentially have a "use after
free" problem, because this memory is managed by sqlite3 and will get a
`realloc` treatment by the follow-up sqlite3 API calls (I may be wrong, but
that's what it looks like after quickly skimming through the sqlite3 source
code). This is a potentially dangerous stuff and it mostly affects the  system
code because the compiler remains silent about a lot of fishy things.

Also see the https://github.com/dlang/dmd/pull/14639 pull 
request. Some of the functions used to be incorrectly annotated 
as  trusted and this was fixed later.

The users (and especially beginners!) need to annotate their code 
as  safe before receiving any help from the compiler related to 
this problem.

=== references to stack variables escaping the scope ===

Here's a good example 
https://forum.dlang.org/post/duwrxnkjaafnzpfgnted forum.dlang.org 
posted by someone, who is not a complete beginner (I can see 
their post 
https://forum.dlang.org/post/pdjvwvzysetndrtkxmea forum.dlang.org 
from many months earlier). Looks like this person was not aware 
of the  safe attribute all this time. And possibly started 
experimenting and asking questions only after running into 
troubles and having to debug some real code.

Another interesting observation is that despite my recommendation 
to add " safe:" at the top of the source file, this person still 
plastered the  safe attribute all over the place in the next code 
snippet. Also not recognizing this syntax doesn't seem uncommon: 
https://forum.dlang.org/post/ddhxlvprhdpqrhkbxuyb forum.dlang.org

=== bounds checking ===

Bounds checking is not done in  safe and  trusted code in 
"-release" builds. How long does it typically take for a beginner 
to learn about this? And how many of them learn it proactively by 
reading the tutorials/documentation rather than getting 
surprised/ambushed by an unexpected segfault in their release 
build? This reminds me about 
https://forum.dlang.org/post/couzhdooeskwppuklasf forum.dlang.org

Now let's look at the DUB ecosystem and what kind of options are 
used for compiling the existing DUB packages. Running `dub build 
--help` shows:

     -b  --build=VALUE     Specifies the type of build to perform. 
Note that
                           setting the DFLAGS environment variable 
will override
                           the build type with custom flags.
                           Possible names:
                             debug, plain, release, release-debug,
                             release-nobounds, unittest, profile, 
profile-gc,
                             docs, ddox, cov, unittest-cov, syntax 
and custom
                             types

Trying these build types in verbose mode reveals that the 
following command line options are supplied to DMD:

     debug            = "-debug -g -w ..."
     plain            = "-w ..."
     release          = "-release -inline -O -w ..."
     release-debug    = "-release -g -inline -O -w ..."
     release-nobounds = "-release -inline -noboundscheck -O -w ..."

Basically `-O` is always bundled together with `-release` and DUB 
packages built with optimizations won't have bounds checking in 
 system code. Of course, unless the packages explicitly override 
this in their sdl/json.


So we are in a rather weird situation. At least in the DUB 
ecosystem a lot of applications and libraries de-facto may have 
unnecessary security issues caused by having too much  system 
code. The community doesn't want to change the default from 
 system to  safe. And at the same time the same community wants 
to promote D as a safe language. People do get upset when, let's 
say, NSA is not listing D as an example of a safe language.

=== how to fix all of this ===

My suggestion is still the same: the compiler should start making 
noise whenever a function gets the  system attribute assigned to 
it by default. Similar to deprecation warnings. This message does 
not have to abort compilation and the existing DUB packages 
should still build successfully (despite the '-w' option). The 
text of the message may look like this:

     "The function 'foobar' got  system attribute by default. If 
this is really what you want, then please add ' system:' at the 
top of the source file or read this <URL> for detailed 
explanations."

I think that this will provide a gentle push/reminder for the 
maintainers of the existing packages. Also beginners will learn 
about the  safe attribute much faster.
Dec 12 2022
next sibling parent reply Nick Treleaven <nick geany.org> writes:
On Monday, 12 December 2022 at 11:07:00 UTC, Siarhei Siamashka 
wrote:

 You have my 100% support here. But the tricky thing is that the 
 majority of the current D language community seems to disagree 
 with us, as evidenced by the apparent rejection of DIP 1028.


That was unpopular because extern(C) functions silently defaulted 
to  trusted, a major security hole. However I am concerned about 
breakage if the default changes, and the lack of a  trusted 
block. The latter means you can't encapsulate  trusted code 
easily if it contains a return statement because you have to wrap 
it in a lambda (which repurposes that return). It could be 
abused, but we need to give programmers practical tools and not 
make things unnecessarily difficult and noisy. Rust has `unsafe` 
blocks.


 What kind of dangers are threatening the users of  system code? 
 This needs to be explained better. Especially considering that 
 every beginner pretty much starts using  system code by default 
 even before knowing that there's a choice.


What about deprecating defining `main` without a 
 system/ trusted/ safe attribute? Then users have to make a 
choice. If they choose  safe then anything main calls has to have 
a safe interface.


 Another interesting observation is that despite my 
 recommendation to add " safe:" at the top of the source file, 
 this person still plastered the  safe attribute all over the 
 place in the next code snippet. Also not recognizing this 
 syntax doesn't seem uncommon: 
 https://forum.dlang.org/post/ddhxlvprhdpqrhkbxuyb forum.dlang.org


I almost never use ` safe:` because it prevents  safe attribute 
inference.


 And at the same time the same community wants to promote D as a 
 safe language. People do get upset when, let's say, NSA is not 
 listing D as an example of a safe language.


Even if D was safe by default, would the NSA really list it? They 
seem to only list some examples of better-known languages:


Rust®, and Swift®."
https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI_SOFTWARE_MEMORY_SAFETY.PDF

Of course, if D can encourage/default to using  safe that may 
make the language more popular by helping market it.


 === how to fix all of this ===

 My suggestion is still the same: the compiler should start 
 making noise whenever a function gets the  system attribute 
 assigned to it by default. Similar to deprecation warnings. 
 This message does not have to abort compilation and the 
 existing DUB packages should still build successfully (despite 
 the '-w' option). The text of the message may look like this:

     "The function 'foobar' got  system attribute by default. If 
 this is really what you want, then please add ' system:' at the 
 top of the source file or read this <URL> for detailed 
 explanations."

 I think that this will provide a gentle push/reminder for the 
 maintainers of the existing packages. Also beginners will learn 
 about the  safe attribute much faster.


Sounds more appropriate for a linter.
Dec 12 2022
parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Monday, 12 December 2022 at 12:53:42 UTC, Nick Treleaven wrote:

 What about deprecating defining `main` without a 
  system/ trusted/ safe attribute? Then users have to make a 
 choice. If they choose  safe then anything main calls has to 
 have a safe interface.


This would be a step in the right direction. But modules don't 
have `main`. And it's not ergonomic:
```D
void bar() {
}
void foo() {
   bar();
}
void main()  safe {
   foo();
}
```
The user will first get a complaint about annotating "foo". And 
after this is done, there will be a complaint about "bar" and so 
on. Many people will be very much annoyed after a few minutes of 
such activity. This can be alternatively resolved by adding 
` safe:` at the top of the file, but the compiler needs to give a 
clear hint about this. If this is not done, then many beginners 
won't be happy and some of them will quit right from the start.


 I almost never use ` safe:` because it prevents  safe attribute 
 inference.


Why is this a problem?


 Of course, if D can encourage/default to using  safe that may 
 make the language more popular by helping market it.


D has very effective marketing. It's very good at attracting 
attention because the advertisements are spread everywhere in 
wikipedia, hackernews and other places. I think that the majority 
of software developers (excluding the youngsters) already know 
that D language exists and already gave it a try at some point in 
the past. But making a good first impression and then managing 
not to betray expectations is another story.

Default  safe can prevent beginners from feeling stabbed in the 
back if they came looking for a safe language, only to be 
surprised by an unexpected out of bounds bug in a -release build. 
It's like "LOL, are you kidding?" discovery. And not everyone is 
particularly enthusiastic about registering in the forum to ask 
questions.
Dec 12 2022
parent reply Nick Treleaven <nick geany.org> writes:
On Monday, 12 December 2022 at 14:33:21 UTC, Siarhei Siamashka 
wrote:

 On Monday, 12 December 2022 at 12:53:42 UTC, Nick Treleaven 
 wrote:

 I almost never use ` safe:` because it prevents  safe 
 attribute inference.


 Why is this a problem?


For example, template functions that are marked  safe are often 
overly restrictive. The safety can be inferred (and  safe 
unittests used to ensure they are inferred  safe). Template 
functions often take a type parameter that could be a type with 
 system operations, or an alias parameter that could take a 
 system variable or a  system delegate say. In those cases the 
template function will fail to instantiate (assuming the delegate 
is called). I would say  safe annotated template functions are an 
anti-pattern, certainly in a public API.

This is why ` safe module foo;` would be better than ` safe:` - 
it wouldn't override inference of  safe/ system. The only 
downside is then you have to have a module declaration even for 
simple D files.
Dec 15 2022
parent areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Thursday, 15 December 2022 at 18:48:56 UTC, Nick Treleaven 
wrote:

 This is why ` safe module foo;` would be better than ` safe:` - 
 it wouldn't override inference of  safe/ system. The only 
 downside is then you have to have a module declaration even for 
 simple D files.


One really should compile dmd from source, **after** altering the 
dmd source in this way:

Required a module to always be annotated with  safe.

And if it is not annotated as such, the compiler provides a 
warning, saying -> If you want to compile an unsafe module (i.e. 
one not annotated as  safe), then you MUST pass -unsafe as a 
parameter to the compiler.

I suspect D3 (in 2045) will be safe by default though.

But one does not need to wait that long just to make  safe 
default .. its pretty easy to do it ;-)
Dec 15 2022
prev sibling next sibling parent reply Nick Treleaven <nick geany.org> writes:
On Monday, 12 December 2022 at 11:07:00 UTC, Siarhei Siamashka 
wrote:

 Bounds checking is not done in  safe and  trusted code in 
 "-release" builds.


False. Compile this with `-release`:

```d
 safe int main()
{
	auto a = [1];
	return a[1];
}
```

core.exception.ArrayIndexError safestd.d(4): index [1] is out of 
bounds for array of length 1
----------------
??:? _d_arraybounds_indexp [0x561bc98fb805]
??:? _Dmain [0x561bc98fb75d]
Dec 12 2022
parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Monday, 12 December 2022 at 12:57:54 UTC, Nick Treleaven wrote:

 On Monday, 12 December 2022 at 11:07:00 UTC, Siarhei Siamashka 
 wrote:

 Bounds checking is not done in  safe and  trusted code in 
 "-release" builds.


 False. Compile this with `-release`:


Is it really difficult to see that this was a typo given the 
context? You need to read it as  system and  trusted code.
Dec 12 2022
parent Nick Treleaven <nick geany.org> writes:
On Monday, 12 December 2022 at 13:08:10 UTC, Siarhei Siamashka 
wrote:

 On Monday, 12 December 2022 at 12:57:54 UTC, Nick Treleaven 
 wrote:

 On Monday, 12 December 2022 at 11:07:00 UTC, Siarhei Siamashka 
 wrote:

 Bounds checking is not done in  safe and  trusted code in 
 "-release" builds.


 False. Compile this with `-release`:


 Is it really difficult to see that this was a typo given the 
 context? You need to read it as  system and  trusted code.


OK, I realized you may have meant that after posting. (Also I 
wasn't expecting markdown to make the first line of the backtrace 
so big!).
Dec 12 2022
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/12/22 12:07, Siarhei Siamashka wrote:

 
 You have my 100% support here. But the tricky thing is that the majority 
 of the current D language community seems to disagree with us, as 
 evidenced by the apparent rejection of DIP 1028. And we should do a much 
 better job convincing them to change their mind.


Sorry, but this is a really bad take. DIP 1028 wanted to make extern(C) 
prototypes ` safe` by default, which is essentially the only criticism 
it attracted. [1] Walter then decided to pull the DIP entirely. It's not 
on the community.


[1] ` safe` on an `extern(C)` prototype is currently equivalent to 
` trusted`. DIP 1028 was therefore " trusted by default". A really bad 
default and one that would make existing _unsafe_ code ` safe` implicitly.
Dec 12 2022
parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Monday, 12 December 2022 at 23:18:13 UTC, Timon Gehr wrote:

 On 12/12/22 12:07, Siarhei Siamashka wrote:

 
 You have my 100% support here. But the tricky thing is that 
 the majority of the current D language community seems to 
 disagree with us, as evidenced by the apparent rejection of 
 DIP 1028. And we should do a much better job convincing them 
 to change their mind.


 Sorry, but this is a really bad take. DIP 1028 wanted to make 
 extern(C) prototypes ` safe` by default, which is essentially 
 the only criticism it attracted. [1] Walter then decided to 
 pull the DIP entirely. It's not on the community.


 [1] ` safe` on an `extern(C)` prototype is currently equivalent 
 to ` trusted`. DIP 1028 was therefore " trusted by default". A 
 really bad default and one that would make existing _unsafe_ 
 code ` safe` implicitly.


The problem with D (which has always been its problem, and likely 
always will be its problem), is that it 'defaults' to allowing 
the programmer to shoot themselves in the foot, at any time, and 
typically without warning (i.e. it defaults to 'ease of use' over 
safety).

That's what 'ease of use' means in D (the same as it means in C - 
trust the programmer).

This is why it's not mentioned by NSA I suspect (assuming they've 
even heard of it). NSA do NOT want you to use languages that 
'implicately trust' the programmer to get memory safety correct.

'ease of use' (I suspect) is what attracts some to D. But for 
others, it's  safe that attract them to D. Both are at opposite 
ends of the spectrum.

So how to please people that are at opposite ends of the spectrum?

You give them 'the option', to opt-in or opt-out. It's that 
simple.

So how about a compile time, commandline switch, that (warns or 
fails) when  safe code is calling  trusted code?

Basically, I'm, saying to the compiler -> "if it's not  safe, I 
do not 'trust' it."
Dec 12 2022
parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Monday, 12 December 2022 at 23:48:58 UTC, areYouSureAboutThat 
wrote:

 So how about a compile time, commandline switch, that (warns or 
 fails) when  safe code is calling  trusted code?

 Basically, I'm, saying to the compiler -> "if it's not  safe, I 
 do not 'trust' it."


Then you will have this warning in pretty much every application. 
Because any  safe code eventually ends up doing OS kernel API 
calls to show stuff on your screen or save it to a hard drive. 
The bridge from  safe to  system has to be crossed somewhere and 
the code annotated as  trusted works as such bridge.

If your OS kernel is buggy, then it compromises the safety of 
your program and you can't do anything about this (other than 
patching up your OS kernel). If your  trusted function is buggy, 
then it compromises the safety of your program and you can't do 
anything about this (other than fixing bugs in the  trusted 
code). Of course, assuming that it was actually necessary to 
annotate the function as  trusted in the first place and it could 
not be  safe for a very good reason.

I don't see any problem with this model. It takes a deliberate 
effort to annotate a function as  trusted. You are not getting 
this attribute by default or via 
https://dlang.org/spec/function.html#function-attribute-inference
Dec 12 2022
next sibling parent areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Tuesday, 13 December 2022 at 00:56:12 UTC, Siarhei Siamashka 
wrote:

 On Monday, 12 December 2022 at 23:48:58 UTC, 
 areYouSureAboutThat wrote:

 So how about a compile time, commandline switch, that (warns 
 or fails) when  safe code is calling  trusted code?

 Basically, I'm, saying to the compiler -> "if it's not  safe, 
 I do not 'trust' it."


 Then you will have this warning in pretty much every 
 application. Because any  safe code eventually ends up doing OS 
 kernel API calls to show stuff on your screen or save it to a 
 hard drive. The bridge from  safe to  system has to be crossed 
 somewhere and the code annotated as  trusted works as such 
 bridge.


Fair enough. But in that case,  unsafe would be a much better 
attribute than  trusted.

If you  trust code, you are presumably implicately trusting that 
it is memory safe.

Whereas,  unsafe, you're saying, look... I know it's not  safe, 
but I wanna use it anyway, even if that results in memory 
corruption.

D should move towards deprecating  trusted, and replacing it with 
 unsafe.
Dec 12 2022
prev sibling parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Tuesday, 13 December 2022 at 00:56:12 UTC, Siarhei Siamashka 
wrote:

 On Monday, 12 December 2022 at 23:48:58 UTC, 
 areYouSureAboutThat wrote:

 So how about a compile time, commandline switch, that (warns 
 or fails) when  safe code is calling  trusted code?

 Basically, I'm, saying to the compiler -> "if it's not  safe, 
 I do not 'trust' it."


 Then you will have this warning in pretty much every 
 application. Because any  safe code eventually ends up doing OS 
 kernel API calls to show stuff on your screen or save it to a 
 hard drive. The bridge from  safe to  system has to be crossed 
 somewhere and the code annotated as  trusted works as such 
 bridge.


you mean like this:

https://learn.microsoft.com/en-us/dotnet/csharp/misc/cs0227
Dec 12 2022
next sibling parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Tuesday, 13 December 2022 at 01:35:48 UTC, areYouSureAboutThat 
wrote:

 you mean like this:

 https://learn.microsoft.com/en-us/dotnet/csharp/misc/cs0227


and this:

https://doc.rust-lang.org/error_codes/E0133.html
Dec 12 2022
parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 13 December 2022 at 01:50:26 UTC, areYouSureAboutThat 
wrote:

 and this:

 https://doc.rust-lang.org/error_codes/E0133.html


This isn't any different from D.

```D
 system void f() { return; } // This is the unsafe code

 safe void main() {
     // error: ` safe` function `D main` cannot call ` system` 
function `app.f`
     f();
}

 safe void main() {
     ()  trusted { f(); }(); // ok!
}
```

The difference is that D lets you also write

```D
 trusted void main() {
     f(); // ok!
}
```

This is really just a nice shorthand for the ` safe` main with 
` trusted` lambda inside. It's also a better practice, since 
` trusted` in a function signature is easier to spot for a code 
reviewer than the lambda inside the function.
Dec 14 2022
next sibling parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Wednesday, 14 December 2022 at 11:45:02 UTC, Dukc wrote:

 This is really just a nice shorthand for the ` safe` main with 
 ` trusted` lambda inside. It's also a better practice, since 
 ` trusted` in a function signature is easier to spot for a code 
 reviewer than the lambda inside the function.


The point of my referencing that link, is that you cannot compile 
in unsafe code into your library in Rust without (1) the compiler 
telling you, you can't do it unless.. or (2) you do what the 
compiler tells you.

That is, you cannot unknowingly compile in unsafe code into your 
library.

In D you can, since  trusted is no different from  system. It's 
just that  safe trusts  trusted. Therefore you have no way of 
knowing your getting  trusted (unsafe) in your library, as you do 
not need to annotate anything to get it. It just gets in there 
without you ever knowing.

Put simply, I would like to know when  trusted code is being 
compiled into my library..  hence my suggestion about an 'optin' 
compiler switch that tells you just that (as it currently does 
for  system,  nogc ...
Dec 14 2022
parent Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Wednesday, 14 December 2022 at 20:54:44 UTC, 
areYouSureAboutThat wrote:

 On Wednesday, 14 December 2022 at 11:45:02 UTC, Dukc wrote:

 This is really just a nice shorthand for the ` safe` main with 
 ` trusted` lambda inside. It's also a better practice, since 
 ` trusted` in a function signature is easier to spot for a 
 code reviewer than the lambda inside the function.


 The point of my referencing that link, is that you cannot 
 compile in unsafe code into your library in Rust without (1) 
 the compiler telling you, you can't do it unless.. or (2) you 
 do what the compiler tells you.

 That is, you cannot unknowingly compile in unsafe code into 
 your library.


Dukc already explained it, but let me show some practical 
examples. Here's a buggy program in Rust:

```Rust
use std::slice;

unsafe fn f(i: usize) -> i32 {
     let mut arr = [1, 2, 3];
     let s = slice::from_raw_parts_mut(arr.as_mut_ptr(), 1000);
     return s[i];
}

fn main() {
     unsafe { println!("{}", f(3)); } // ok!
}
```

And here's exactly the same buggy program converted to D:

```D
 safe: import std;

 system int f(size_t i) {
     auto arr = [1, 2, 3];
     return arr.ptr[i];
}

void main() {
     ()  trusted { writeln(f(3)); }(); // ok!
}
```

Both Rust and D compilers accept this code without complaining. 
Such code can be unknowingly compiled into your library. It may 
be a part of the standard library, or it may be a part of some 
third-party library that you want to use. And if the bug is 
difficult to reproduce, then you even won't be aware of it.

Now if you review the code, then D code is much more readable 
than Rust. If you want to find the boundary between safe and 
unsafe code in D, then you can just search for the " trusted" 
attribute. But in Rust you can't easily find the boundary between 
safe and unsafe code, because the same "unsafe" keyword is reused 
for different purposes.


  trusted is no different from  system


It's is very different. If you look at a function as a black box 
without knowing what is inside, then:

   * ` trusted` function is typically safe to use. But this safety 
is guaranteed by human reviewers, because the compiler is not 
smart enough to do such analysis. It's important that the way how 
such functions are handling their input/output data is reasonably 
foolproof.

   * ` system` function may have some dangerous quirks and can be 
easily misused to shoot yourself in the foot (leak resources, 
segfault on incorrect input, deadlock when used from multiple 
threads, ...). It can be used correctly if the user is very 
careful, but allowing to call such function directly from the 
` safe` code would be reckless.
Dec 14 2022
prev sibling parent reply Dom DiSc <dominikus scherkl.de> writes:
On Wednesday, 14 December 2022 at 11:45:02 UTC, Dukc wrote:

 ```D
  safe void main() {
     ()  trusted { f(); }(); // ok!
 }
 ```

 The difference is that D lets you also write

 ```D
  trusted void main() {
     f(); // ok!
 }
 ```

 This is really just a nice shorthand for the ` safe` main with 
 ` trusted` lambda inside. It's also a better practice, since 
 ` trusted` in a function signature is easier to spot for a code 
 reviewer than the lambda inside the function.


I don't think so. A search for  trusted finds both.
But the lambda reduces the amount of  trusted code (e.g. all the 
rest in main() can be  safe, so you don't need to check it).
On the contrary: I would allow  trusted blocks (so that the ugly 
lambda construct wouldn't be necessary anymore) and forbid 
 trusted functions altogether - they are simply  system.

This allows to reduce the trusted parts step by step by declaring 
 system functions as  save and instead mark the few lines in them 
that do unsafe things as  trusted - until there are no  system 
functions anymore and the  trusted parts don't contain any calls 
(except for external calls to parts out of your control because 
they are from unsafe languages or the source is not available - 
which should be avoided, unless someone guarantees* them to be 
save).

Best practice should be to use  trusted around as few lines of 
code as possible (and of course avoid  system as far as possible).
This is because everything  trusted need to be checked manually 
and this is very time consuming.

*Meaning: Pays for any damage, if this fails.
Dec 15 2022
next sibling parent areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Thursday, 15 December 2022 at 08:01:23 UTC, Dom DiSc wrote:

 This is because everything  trusted need to be checked manually 
 and this is very time consuming.


manually? that will **never** work.

https://www.federalregister.gov/documents/2021/05/17/2021-10460/improving-the-nations-cybersecurity#p-68
Dec 15 2022
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/15/22 09:01, Dom DiSc wrote:

 
 I don't think so. A search for  trusted finds both.
 But the lambda reduces the amount of  trusted code (e.g. all the rest in 
 main() can be  safe, so you don't need to check it).
 On the contrary: I would allow  trusted blocks (so that the ugly lambda 
 construct wouldn't be necessary anymore) and forbid  trusted functions 
 altogether - they are simply  system.
 
 This allows to reduce the trusted parts step by step by declaring 
  system functions as  save and instead mark the few lines in them that 
 do unsafe things as  trusted - until there are no  system functions 
 anymore and the  trusted parts don't contain any calls (except for 
 external calls to parts out of your control because they are from unsafe 
 languages or the source is not available - which should be avoided, 
 unless someone guarantees* them to be save).
 
 Best practice should be to use  trusted around as few lines of code as 
 possible (and of course avoid  system as far as possible).
 This is because everything  trusted need to be checked manually and this 
 is very time consuming.


This is a fallacy. All it achieves is:

- Some incomplete  safe-ty checks in code that should _actually_ be 
` system/ trusted` (this part can be useful, but OTOH you will have 
` safe` annotations that are fundamentally a lie).

- Unknown scope of _actual_ ` trusted`, you will have to manually check 
_` safe`_ code as well, and a priori you will have no idea how much of 
it you have to check (this part is terrible).

It certainly does not help you minimize the number of lines you have to 
check.
Jan 22 2023
next sibling parent reply Dom Disc <dominikus scherkl.de> writes:
On Sunday, 22 January 2023 at 10:08:55 UTC, Timon Gehr wrote:

 On 12/15/22 09:01, Dom DiSc wrote:

 Best practice should be to use  trusted around as few lines of 
 code as possible (and of course avoid  system as far as 
 possible).
 This is because everything  trusted need to be checked 
 manually and this is very time consuming.


 This is a fallacy. All it achieves is:

 - Some incomplete  safe-ty checks in code that should 
 _actually_ be ` system/ trusted` (this part can be useful, but 
 OTOH you will have ` safe` annotations that are fundamentally a 
 lie).


If you want, you can take any  safe function that contains a 
 trusted block as something to be completely manually checked - 
than this system won't make any difference to the current one.

But I'm pretty sure most  trusted blocks didn't invalidate the 
safety of the whole function, because they ARE only a wrapper 
around a  system function call, and it is sufficient to check 
manually, that the system function is called in a safe way, e.g. 
with valid safe parameters.

Most prominent are wrappers around C string functions, that can 
be considered safe if the parameters are checked to have valid 
length, don't point to zero and are indeed null-terminated.
This kind of  trusted blocks are (at least in my code) the vast 
majority.
Only very seldom I do something complicated and unsafe that I 
really do trust - and then I simply mark the whole function with 
a  trusted block.
The difference is only

 trusted fn() {

};

vs.

 safe fn() {  trusted {

} };

But what we gain is that  save/ system is then a binary attribute 
and not a cumbersome tri-state anymore. Everything is either safe 
or not.


 - Unknown scope of _actual_ ` trusted`, you will have to 
 manually check _` safe`_ code as well, and a priori you will 
 have no idea how much of it you have to check (this part is 
 terrible).


Why is this terrible? In worst case you have to check exactly as 
much code manually as you do today. But in most cases you have to 
check much less.


 It certainly does not help you minimize the number of lines you 
 have to check.


Again, why? Most  trusted blocks are isolated one-liners.
Jan 22 2023
next sibling parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Sunday, 22 January 2023 at 21:02:06 UTC, Dom Disc wrote:

 The difference is only

  trusted fn() {

 };

 vs.

  safe fn() {  trusted {

 } };


The second variant looks more verbose and more ugly to me. Also 
wouldn't it require an extra level of indentation if properly 
formatted?


 But what we gain is that  save/ system is then a binary 
 attribute and not a cumbersome tri-state anymore. Everything is 
 either safe or not.


Dropping support for the  trusted function attribute even after 
some deprecation period would be a rather annoying compatibility 
breaking change. That's a high price for something that is just 
cosmetics.

Also how do you imagine writing future code that is compatible 
with both the most recent versions of D compilers and also with 
GDC 12 (frontend version 2.100)? GDC 12 is included in Ubuntu 
22.04 LTS and will be relevant for a very long time.
Jan 22 2023
parent reply Dom DiSc <dominikus scherkl.de> writes:
On Monday, 23 January 2023 at 03:19:07 UTC, Siarhei Siamashka 
wrote:

 On Sunday, 22 January 2023 at 21:02:06 UTC, Dom Disc wrote:

 The difference is only

  trusted fn() {

 };

 vs.

  safe fn() {  trusted {

 } };


 The second variant looks more verbose and more ugly to me.


That's so because a whole trusted function is an abomination.
Most of the time it should look more like this:

```d
 safe fn()
{
    // lot of safe stuff

     trusted {
         assert(/*systemFunc is safe to be used with param1*/);
         assert(/*systemFunc is safe to be used with param2*/);
        systemFunc(param1, param2);
    }

    // more safe stuff

}
```

And I like that much more than

```d
 trusted fn()
{
    // lot of safe stuff - why do I need to check that manually?

    // only the two asserts have to be checked manually:
     assert(/*systemFunc is safe to be used with param1*/);
     assert(/*systemFunc is safe to be used with param2*/);
    systemFunc(param1, param2);

    // more safe stuff - why do I need to check that manually?

}
```


 But what we gain is that  save/ system is then a binary 
 attribute and not a cumbersome tri-state anymore. Everything 
 is either safe or not.


 Dropping support for the  trusted function attribute even after 
 some deprecation period would be a rather annoying 
 compatibility breaking change.


Of course - it should not have been introduced to begin with.
Fixing things very late is always a pain in the ass.


 That's a high price for something that is just cosmetics.


I don't consider this pure esthetic.
It's also much easier to understand for newcommers.

But ok, I can live with the state we have now. This is more 
something for D3
Jan 23 2023
parent RTM <riven baryonides.ru> writes:
On Monday, 23 January 2023 at 16:31:01 UTC, Dom DiSc wrote:

 That's so because a whole trusted function is an abomination.
 Most of the time it should look more like this:

 ```d
  safe fn()
 {
    // lot of safe stuff

     trusted {
         assert(/*systemFunc is safe to be used with param1*/);
         assert(/*systemFunc is safe to be used with param2*/);
        systemFunc(param1, param2);
    }

    // more safe stuff

 }
 ```


Should be:

```d
 safe fn()
{
    // lot of safe stuff

    fghn(param1, param2);

    // more safe stuff
}

 trusted fghn(p1, p2)
{
    assert(...);
    assert(...);
    systemFunc(p1, p2);
}
```
Jan 23 2023
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 1/22/23 22:02, Dom Disc wrote:

 

 - Unknown scope of _actual_ ` trusted`, you will have to manually 
 check _` safe`_ code as well, and a priori you will have no idea how 
 much of it you have to check (this part is terrible).


 
 Why is this terrible? In worst case you have to check exactly as much 
 code manually as you do today. But in most cases you have to check much 
 less.


It's terrible for the reason I pointed out. It's not modular. A priori 
_you don't know how much_ code you have to check, you just know that you 
have to check _some_ ` safe` code, because if you check the ` trusted` 
lambdas on their own they cannot pass muster.

Officially you only have to check ` trusted` code. However, if you do 
that, typically all you learn is that the ` trusted` code is bad because 
it has an unsafe interface...
Jan 23 2023
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
I tend to agree with Timon. We've been using the  trusted lambda thing for
years 
now, and it just looks like a code smell. For example:

 safe void merry()
{
      trusted int* p = malloc(10);
     ...
      trusted free(p);
}

Has this improved anything? I doubt it.

Wouldn't it be far better to encapsulate the malloc/free pair as the allocation 
for an object with certain behaviors, and thus encapsulate the object rather 
than the operations?
Jan 22 2023
prev sibling parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Tuesday, 13 December 2022 at 01:35:48 UTC, areYouSureAboutThat 
wrote:

 On Tuesday, 13 December 2022 at 00:56:12 UTC, Siarhei Siamashka 
 wrote:

 On Monday, 12 December 2022 at 23:48:58 UTC, 
 areYouSureAboutThat wrote:

 So how about a compile time, commandline switch, that (warns 
 or fails) when  safe code is calling  trusted code?

 Basically, I'm, saying to the compiler -> "if it's not  safe, 
 I do not 'trust' it."


 Then you will have this warning in pretty much every 
 application. Because any  safe code eventually ends up doing 
 OS kernel API calls to show stuff on your screen or save it to 
 a hard drive. The bridge from  safe to  system has to be 
 crossed somewhere and the code annotated as  trusted works as 
 such bridge.


 you mean like this:

 https://learn.microsoft.com/en-us/dotnet/csharp/misc/cs0227


I mean like this:

```D
import std.stdio;
void main()  safe {
   writeln("Hello, world!");
}
```

```
$ dmd -g hello.d
$ gdb ./hello
(gdb) b write
(gdb) r
Breakpoint 1, 0x00007ffff7714bb0 in write () from /lib64/libc.so.6

(gdb) bt


/lib64/libc.so.6


/lib64/libc.so.6

/lib64/libc.so.6

std.stdio.File.LockingTextWriter.put!(char).put(char)

std.stdio.writeln!(immutable(char)[]).writeln(immutable(char)[])


(gdb) disassemble
Dump of assembler code for function write:
=> 0x00007ffff7714bb0 <+0>:     mov    %fs:0x18,%eax
    0x00007ffff7714bb8 <+8>:     test   %eax,%eax
    0x00007ffff7714bba <+10>:    jne    0x7ffff7714bd0 <write+32>
    0x00007ffff7714bbc <+12>:    mov    $0x1,%eax
    0x00007ffff7714bc1 <+17>:    syscall
    0x00007ffff7714bc3 <+19>:    cmp    $0xfffffffffffff000,%rax
    0x00007ffff7714bc9 <+25>:    ja     0x7ffff7714c20 <write+112>
    0x00007ffff7714bcb <+27>:    ret
```

That's how you go from a ` safe` function main() down to a 
`syscall` instruction (then the rest is happening in the Linux 
kernel). Somewhere on this way a ` trusted` attribute needs to be 
used, which means *"please  trust me, these particular 
potentially unsafe things are used correctly here and won't cause 
troubles"*: 
https://github.com/dlang/phobos/blob/v2.101.0/std/stdio.d#L3187

As the end users, we expect that Phobos interacts with a 
dangerous system library (libc) in a correct way. Phobos itself 
annotates [trustedFPUTC/trustedFPUTWC 
functions](https://github.com/dlang/phobos/blob/v2.101.0/std/stdio.d#L354-L362)
imported from glibc as ` trusted` and expects that they are interacting with
the OS kernel correctly without causing troubles.

It's impossible to do a syscall by using just ` safe` code and 
nothing else.
Dec 12 2022
parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Tuesday, 13 December 2022 at 02:45:29 UTC, Siarhei Siamashka 
wrote:

 It's impossible to do a syscall by using just ` safe` code and 
 nothing else.


oh. so you mean syscalls and integrating with C is not safe? ;-)

My concern really is:

How do I know my  safe code is calling  safe code and not 
 trusted code?

 safe interfaces are nice - but all of Phobos could just as well 
be marked as  trusted, since  safe implicately trusts  trusted, 
and my  safe code would still compile just fine.

But when I -release it, I discover something that is not so 
welcome...cause all of that  trusted code is no longer subject to 
runtime memory safety checks.

I would like the compiler to tell me I'm calling code that has 
been marked as  trusted, and that if I want to proceed, then I 
need to compile with: -enableTrusted

That way, I've opted in to unsafety (i.e.  safe being allowed to 
use  trusted), rather than being defaulted in to it.
Dec 12 2022
parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Tuesday, 13 December 2022 at 05:26:23 UTC, areYouSureAboutThat 
wrote:

 My concern really is:

 How do I know my  safe code is calling  safe code and not 
  trusted code?


You can be sure that your  safe code is always calling at least 
some small amount of  trusted code inside of Phobos.


  safe interfaces are nice - but all of Phobos could just as 
 well be marked as  trusted, since  safe implicately trusts 
  trusted, and my  safe code would still compile just fine.


Yes, it could be all marked as  trusted, but it isn't. Also 
Phobos or any other third-party library theoretically could do a 
lot of various bad things with or without  safe annotations. Some 
libraries have better code quality than the others and that's 
just how it is.


 But when I -release it, I discover something that is not so 
 welcome...cause all of that  trusted code is no longer subject 
 to runtime memory safety checks.


If these  trusted parts of Phobos are bug free, then everything 
is fine. Your code shouldn't be able to trigger memory safety 
bugs in Phobos by feeding incorrect input to it.


 I would like the compiler to tell me I'm calling code that has 
 been marked as  trusted, and that if I want to proceed, then I 
 need to compile with: -enableTrusted


As I already said before, you will effectively have to always use 
this switch for compiling each and every application.

I guess, you probably want the  trusted parts of Phobos to be 
annotated as  supertrusted and ignored by this switch, because 
it's the standard library deserving special privileges? And only 
complain about the  trusted attribute usage in your own code or 
in third-party libraries written by plebeians ;-)
Dec 14 2022
parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Wednesday, 14 December 2022 at 13:00:51 UTC, Siarhei Siamashka 
wrote:

 I guess, you probably want the  trusted parts of Phobos to be 
 annotated as  supertrusted and ignored by this switch, because 
 it's the standard library deserving special privileges? And 
 only complain about the  trusted attribute usage in your own 
 code or in third-party libraries written by plebeians ;-)


No. I do not 'trust' the standard library to be 'safe'. Why 
should I?

This is where a compiler switch would come in handy, as it could 
tell me when my  safe is calling  trusted, in the same way it 
does with  system and  nogc ....

That is, the compiler would help me discover that I might be 
relying on  trusted code somewhere down the chain, and that I 
should know about it.

As it is, I do not know unless I go examine the source code of 
Phobos.

The compiler switch could provide the assistance i need to go 
discover those parts mark as trusted.

Of course it could be optin, and as you point out, probably 
should be.

I guess I just need to work out how the compiler does it with 
 system and  nogc .. and adapt it to 'my needs', so I know 
 trusted code is being compiled into my library as well.
Dec 14 2022
parent reply Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Wednesday, 14 December 2022 at 20:36:39 UTC, 
areYouSureAboutThat wrote:

 On Wednesday, 14 December 2022 at 13:00:51 UTC, Siarhei 
 Siamashka wrote:

 I guess, you probably want the  trusted parts of Phobos to be 
 annotated as  supertrusted and ignored by this switch, because 
 it's the standard library deserving special privileges? And 
 only complain about the  trusted attribute usage in your own 
 code or in third-party libraries written by plebeians ;-)


 No. I do not 'trust' the standard library to be 'safe'. Why 
 should I?


Earlier you posted this link: 
https://learn.microsoft.com/en-us/dotnet/csharp/misc/cs0227
Dec 14 2022
parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Wednesday, 14 December 2022 at 21:23:16 UTC, Siarhei Siamashka 
wrote:

 On Wednesday, 14 December 2022 at 20:36:39 UTC, 
 areYouSureAboutThat wrote:

 On Wednesday, 14 December 2022 at 13:00:51 UTC, Siarhei 
 Siamashka wrote:

 I guess, you probably want the  trusted parts of Phobos to be 
 annotated as  supertrusted and ignored by this switch, 
 because it's the standard library deserving special 
 privileges? And only complain about the  trusted attribute 
 usage in your own code or in third-party libraries written by 
 plebeians ;-)


 No. I do not 'trust' the standard library to be 'safe'. Why 
 should I?


 Earlier you posted this link: 
 https://learn.microsoft.com/en-us/dotnet/csharp/misc/cs0227

 'safe'?


That wasn't posted in the context of trusting code, but rather in 
the context of a thread where someone mentioned that OOP is not 
suited to building containers. I just posted that link to MS 
containers to refute that assertion.

In the context of trust, however, no, I do not trust those 
containers either. Why should I?

I rely (not trust) on the CLR to do it's job properly though.


marking code as unsafe, and **additionally**, supplying to the 
compiler an option that explicately tells the compiler you are ok 
with compiling in that unsafe code.

But in D, you implicately tell the compiler you completely trust 
unsafe system code (that has essentially just been annoted as 
 trusted so that  safe can use it), but is nonetheless unsafe 
system code) - and you do this just by marking your module as 
 safe??

No. To me,  safe does not mean I'm completely ok with 
incorporating unsafe system code into my library (even if someone 
has annotated it with  trusted).

It's not that I don't ever want it, but I would like to know when 
it's occuring, and not find out in -release mode :-(

It would be a great auditing tool, if the compiler had this 
opt-in switch.

I expect its not that hard to do, since the compiler can already 
do it with  system.. nogc..etc.. I guess I'll just have to work 
it out :-(

If I do, I'll submit a PR... maybe..
Dec 14 2022
next sibling parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Wednesday, 14 December 2022 at 23:29:00 UTC, 
areYouSureAboutThat wrote:

 That wasn't posted in the context of trusting code, but rather 
 in the context of a thread where someone mentioned that OOP is 
 not suited to building containers. I just posted that link to 
 MS containers to refute that assertion.


Oops. A vorrection to my response above, as I mis-read that link 


the correct answer to the question posed to me:

"Earlier you posted this link: 
https://learn.microsoft.com/en-us/dotnet/csharp/misc/cs0227

'safe'?"

I don't. But that is not the point I was trying to make.


code into my code without my explicately telling the compiler I 

*also* need to compile with the unsafe option - it cannot just 
sneak in just cause someone marked it as unsafe.

This is not the case in D, as per my previous post.

Anyway....
Dec 14 2022
parent reply norm <norm.rowtree gamil.com> writes:
On Wednesday, 14 December 2022 at 23:56:53 UTC, 
areYouSureAboutThat wrote:

 On Wednesday, 14 December 2022 at 23:29:00 UTC, 
 areYouSureAboutThat wrote:

 That wasn't posted in the context of trusting code, but rather 
 in the context of a thread where someone mentioned that OOP is 
 not suited to building containers. I just posted that link to 
 MS containers to refute that assertion.


 Oops. A vorrection to my response above, as I mis-read that 


 the correct answer to the question posed to me:

 "Earlier you posted this link: 
 https://learn.microsoft.com/en-us/dotnet/csharp/misc/cs0227

 'safe'?"

 I don't. But that is not the point I was trying to make.


 code into my code without my explicately telling the compiler I 

 *also* need to compile with the unsafe option - it cannot just 
 sneak in just cause someone marked it as unsafe.

 This is not the case in D, as per my previous post.

 Anyway....


At the very least D should be  safe by default. Slapping a  safe 
at the top of main isn't good enough, I know it is easy but 
programmers forget or simply do not know. Out of box experience 
is vital,  safe by default is part of that IMO if you really want 
to market as a Rust alternative.

Having said that, I personally don't think Rust is a good 
approach to SW engineering. It solves one problem well, but that 
solution negatively impacts other aspects of SW development, some 
of which are not related to the memory problem. Rust does other 
things really well too but they are often masked by this 
everything is a nail approach for average developers.

Most security hacks (that I know of) are not due to C array 
overruns. Those headliners that usually come from C/C++ code 
written >10yrs ago. The majority of security breaches are due to 
human activities, people clicking on fake email links, sharing 
passwords, leaving computers unlocked, incorrect server and 
database configuration etc. Rust and  safe will not solve these 
real world problems.
Dec 14 2022
next sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, Dec 15, 2022 at 01:12:28AM +0000, norm via Digitalmars-d wrote:
[...]

 Most security hacks (that I know of) are not due to C array overruns.
 Those headliners that usually come from C/C++ code written >10yrs ago.
 The majority of security breaches are due to human activities, people
 clicking on fake email links, sharing passwords, leaving computers
 unlocked, incorrect server and database configuration etc. Rust and
  safe will not solve these real world problems.


No technology will solve those problems either.  It's a social problem.


T

-- 
IBM = I'll Buy Microsoft!
Dec 14 2022
prev sibling parent reply areYouSureAboutThat <areYouSureAboutThat gmail.com> writes:
On Thursday, 15 December 2022 at 01:12:28 UTC, norm wrote:



 Most security hacks (that I know of) are not due to C array 
 overruns. Those headliners that usually come from C/C++ code 
 written >10yrs ago. The majority of security breaches are due 
 to human activities, people clicking on fake email links, 
 sharing passwords, leaving computers unlocked, incorrect server 
 and database configuration etc. Rust and  safe will not solve 
 these real world problems.


you forget to mention.. 'and programmers blindly using 
third-party packages'

'An Empirical Study on Production Dependencies in NPM' - 2022

"The problem is that developers struggle to identify what 
vulnerabilities may affect their software application.."

https://arxiv.org/pdf/2207.14711.pdf



 Dec 15 2022



  parent reply "H. S. Teoh" <hsteoh qfbox.info>  writes:


On Thu, Dec 15, 2022 at 09:41:08AM +0000, areYouSureAboutThat via Digitalmars-d
wrote:

 On Thursday, 15 December 2022 at 01:12:28 UTC, norm wrote:


[...]

 Most security hacks (that I know of) are not due to C array
 overruns.  Those headliners that usually come from C/C++ code
 written >10yrs ago.  The majority of security breaches are due to
 human activities, people clicking on fake email links, sharing
 passwords, leaving computers unlocked, incorrect server and database
 configuration etc. Rust and  safe will not solve these real world
 problems.


 
 you forget to mention.. 'and programmers blindly using third-party
 packages'
 
 'An Empirical Study on Production Dependencies in NPM' - 2022
 
 "The problem is that developers struggle to identify what
 vulnerabilities may affect their software application.."
 
 https://arxiv.org/pdf/2207.14711.pdf


+1, I have always been skeptical about the contemporary trend of using
fancy package managers with hairball external dependencies that make
your builds dependent on some opaque remote server somewhere out there
on the 'Net that you have no control over.  Some time ago somebody also
posted another article about how easy it is to conduct MITM attacks on
these external package repositories to insert a malicious package /
substitute a legitimate package with a malicious version.

Not only it's a security hazard, it's also a logistic time-bomb: you
never know if the remote server won't suddenly vanish off the face of
the internet right at the moment when you're about to release your
product and need to make the final production build. (It may come back
tomorrow or the day after, but deadlines would be missed, heads would
roll.)  You're basically putting yourself at the mercy (and/or
(in)competence) of some independent external entity you have zero
control or influence over.


T

-- 
In a world without fences, who needs Windows and Gates? -- Christian Surchi



 Dec 15 2022



 next sibling parent reply Siarhei Siamashka <siarhei.siamashka gmail.com>  writes:


On Thursday, 15 December 2022 at 14:36:18 UTC, H. S. Teoh wrote:

 +1, I have always been skeptical about the contemporary trend 
 of using fancy package managers with hairball external 
 dependencies that make your builds dependent on some opaque 
 remote server somewhere out there on the 'Net that you have no 
 control over.  Some time ago somebody also posted another 
 article about how easy it is to conduct MITM attacks on these 
 external package repositories to insert a malicious package / 
 substitute a legitimate package with a malicious version.

 Not only it's a security hazard, it's also a logistic 
 time-bomb: you never know if the remote server won't suddenly 
 vanish off the face of the internet right at the moment when 
 you're about to release your product and need to make the final 
 production build. (It may come back tomorrow or the day after, 
 but deadlines would be missed, heads would roll.)  You're 
 basically putting yourself at the mercy (and/or (in)competence) 
 of some independent external entity you have zero control or 
 influence over.


Any serious organization relying on DUB packages would have their 
own local mirror of this stuff and also pin all dependencies to 
specific commit hashes rather than version tags (after reviewing 
the code, checking digital signatures, etc.).

It's also possible to use the compilers and libraries packaged in 
some reputable Linux distribution. Albeit the compiler versions 
will be somewhat stale and the variety of the available 
third-party libraries will be somewhat limited.



 Dec 15 2022



 next sibling parent  rikki cattermole <rikki cattermole.co.nz>  writes:


On 16/12/2022 4:19 AM, Siarhei Siamashka wrote:

 Any serious organization relying on DUB packages would have their own 
 local mirror of this stuff and also pin all dependencies to specific 
 commit hashes rather than version tags (after reviewing the code, 
 checking digital signatures, etc.).


We should have our own artifact repository as part of dub-registry.

That way we can pin versions to artifacts and not let people move things 
out from under other peoples feet.

But yeah, lots of work that can be done to mitigate any potential issues.



 Dec 15 2022



prev sibling  parent  "H. S. Teoh" <hsteoh qfbox.info>  writes:


On Thu, Dec 15, 2022 at 03:19:12PM +0000, Siarhei Siamashka via Digitalmars-d
wrote:
[...]

 Any serious organization relying on DUB packages would have their own
 local mirror of this stuff


How easy is it to create a DUB mirror?  Perhaps that's one area of
improvement. Ideally it should be as easy as `dub clone-repo`, though in
practice of course that would be much more involved than that.



 and also pin all dependencies to specific commit hashes rather than
 version tags (after reviewing the code, checking digital signatures,
 etc.).


[...]

Tying dependencies to commit hashes instead of version tags is a good
step forward.  It's not 100% foolproof, though, no thanks to the SHA-1
collision attack. (It's extremely unlikely, of course. But in theory
it's possible that a malicious actor could have generated a repo
containing the innocent half of a colliding pair and substituted it with
the malicious half afterwards. Though probably it's not worth thinking
about, unless your project happens to be a sensitive security-related
one where the risk of being targeted by a sophisticated attacker is
high. In which case you probably wouldn't be using random packages from
a DUB repo in the first place. :-P)

Having said that, though, if you're going to be tying dependencies to
specific hashes, might as well just graft the dependencies into your
source tree instead.  Then you could just `git checkout` some branch or
tag and be 100% assured that it will build successfully, and that the
build products will be 100% identical to the last time you built that
branch or tag.  Reproducibility is a critical factor in debugging --
nothing like trying to chase down a bug only to have it vanish from
under you because some random upstream repo decided to upgrade their
package in the middle of your debug session.


T

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it.
-- Donald Knuth



 Dec 15 2022



prev sibling  parent  Was <cryptobitmann gmail.com>  writes:


On Thursday, 15 December 2022 at 14:36:18 UTC, H. S. Teoh wrote:

 On Thu, Dec 15, 2022 at 09:41:08AM +0000, areYouSureAboutThat 
 via Digitalmars-d wrote:

 [...]


 [...]

 [...]


 +1, I have always been skeptical about the contemporary trend 
 of using fancy package managers with hairball external 
 dependencies that make your builds dependent on some opaque 
 remote server somewhere out there on the 'Net that you have no 
 control over.  Some time ago somebody also posted another 
 article about how easy it is to conduct MITM attacks on these 
 external package repositories to insert a malicious package / 
 substitute a legitimate package with a malicious version.

 [...]


agree



 Apr 28 2023



prev sibling  parent reply Dom DiSc <dominikus scherkl.de>  writes:


On Wednesday, 14 December 2022 at 23:29:00 UTC, 
areYouSureAboutThat wrote:

 No. To me,  safe does not mean I'm completely ok with 
 incorporating unsafe system code into my library (even if 
 someone has annotated it with  trusted).


 trusted is not there to make anything  save.
It is there to reduce the amount of code you need to check for 
correctness.


 It's not that I don't ever want it, but I would like to know 
 when it's occuring,


Then simply search for  trusted. This is really easy and very 
unlikely to find any false positives. The fewer you find it, the 
more trustworthy I would consider the code to be.
But to really consider the program  save you need to check these 
parts yourself - or find someone YOU trust to have checked it.

At least in D you can do that check yourself. In other languages 
you have no chance other than checking EVERYTHING you include - 
which tend to be millions of lines of code, so is completely 
unfeasible.


 It would be a great auditing tool, if the compiler had this 
 opt-in switch.


Don't know how much easier this would be than simply searching 
for  trusted. As this is already so easy, can't get us much. But 
at least the compiler can further reduce the search space by 
indicating which  trusted code is actually used by a program.

But I prefer to do the full search anyway for a new library, to 
decide if I want to use it or not.



 Dec 14 2022



  parent  Dom DiSc <dominikus scherkl.de>  writes:


On Thursday, 15 December 2022 at 07:19:14 UTC, Dom DiSc wrote:

 In other languages you have no chance other than checking 
 EVERYTHING
 you include - which tend to be millions of lines of code,


If you are even able to get the source code, that is.



 Dec 14 2022



prev sibling  parent  Adam D Ruppe <destructionator gmail.com>  writes:


On Monday, 12 December 2022 at 03:48:26 UTC, Walter Bright wrote:

 Safe code is where bounds checking is needed.


Most code needs bounds checking. The exceptions can use the 
`.ptr` property to *locally* bypass it. (Which is correctly 
prohibited in ` safe` code, since it is trusting the programmer 
to do it right.)

The global switches should *never* be used in *any* real world 
code. If you think you can trust every last bit of the code, you 
have no business handling user data.



 Dec 12 2022



prev sibling  parent reply youSureAboutThat <youSureAboutThat gmail.com>  writes:


On Friday, 9 December 2022 at 22:22:36 UTC, Walter Bright wrote:

 Bounds checking is not disabled with -release


//compile with: -release (and see what happens)

module test;

import std.stdio;

void main()
{

     int[5] array = [0, 1, 2, 3, 4];

     for (size_t i = 0; i < array.length + 1; i++)
         writeln("Element ", i, " = ", array[i]);

}



 Dec 11 2022



  parent reply youSureAboutThat <youSureAboutThat gmail.com>  writes:


On Monday, 12 December 2022 at 05:06:06 UTC, youSureAboutThat 
wrote:


just to clarify -> when  safe isn't safe:


module test;

import std.stdio;

// so -boundscheck=off will always disable runtime boundscheck,
// regardless of whether code is annotated as  safe,  system, or 
 trusted.

// -release will disable runtime boundscheck for code annotated 
as  trusted and  system
// however, code annotated with  safe will still have runtime 
boundscheck enabled.

// here is where it's a little tricky...
// calling code annotated with  trusted from code annotated with 
 safe (see below).

 safe void main() { foo(); } // even though this is marked as 
 safe
                              // when you compile with -release
                              // boundscheck is disabled for foo()
                              // since foo() is marked as  trusted
                              // and -release removes boundscheck
                              // for  trusted and  system..(but 
not  safe)

 trusted foo()
{
     int[5] array = [0, 1, 2, 3, 4];

     for (size_t i = 0; i < array.length + 1; i++)
         writeln("Element ", i, " = ", array[i]);
}



 Dec 11 2022



 next sibling parent reply youSureAboutThat <youSureAboutThat gmail.com>  writes:


On Monday, 12 December 2022 at 06:13:20 UTC, youSureAboutThat 
wrote:

 // -release will disable runtime boundscheck for code annotated 
 as  trusted and  system
 // however, code annotated with  safe will still have runtime 
 boundscheck enabled.


further clarifying the above:

// -release will disable runtime boundscheck for code annotated 
as  trusted and  system
// and also code not annotated at all (as it defaults to  system)
// However, code annotated with  safe will still have runtime 
boundscheck enabled.



 Dec 11 2022



  parent  Nick Treleaven <nick geany.org>  writes:


On Monday, 12 December 2022 at 06:23:09 UTC, youSureAboutThat 
wrote:

 further clarifying the above:

 // -release will disable runtime boundscheck for code annotated 
 as  trusted and  system


Unless you override the default bounds checks option.


 // and also code not annotated at all (as it defaults to 
  system)


Except certain functions can be inferred as safe:
https://dlang.org/spec/function.html#function-attribute-inference



 Dec 12 2022



prev sibling  parent reply Basile B. <b2.temp gmx.com>  writes:


On Monday, 12 December 2022 at 06:13:20 UTC, youSureAboutThat 
wrote:

 On Monday, 12 December 2022 at 05:06:06 UTC, youSureAboutThat 
 wrote:


 just to clarify -> when  safe isn't safe:


 module test;

 import std.stdio;

 // so -boundscheck=off will always disable runtime boundscheck,
 // regardless of whether code is annotated as  safe,  system, 
 or  trusted.

 // -release will disable runtime boundscheck for code annotated 
 as  trusted and  system
 // however, code annotated with  safe will still have runtime 
 boundscheck enabled.

 // here is where it's a little tricky...
 // calling code annotated with  trusted from code annotated 
 with  safe (see below).

  safe void main() { foo(); } // even though this is marked as 
  safe
                              // when you compile with -release
                              // boundscheck is disabled for 
 foo()
                              // since foo() is marked as 
  trusted
                              // and -release removes boundscheck
                              // for  trusted and  system..(but 
 not  safe)

  trusted foo()
 {
     int[5] array = [0, 1, 2, 3, 4];

     for (size_t i = 0; i < array.length + 1; i++)
         writeln("Element ", i, " = ", array[i]);
 }


You're supposed to catch this bug during development and with 
your tests but here the real problem is "-release" (that 
problem's been mentioned earlier).

That switch is not well designed. Bound checks could have their 
own front-end optimizations so we could have

     -boundscheck={on|optimized|off}

instead of the current system, tied to the language safety system.



 Dec 12 2022



 next sibling parent reply Basile B. <b2.temp gmx.com>  writes:


On Monday, 12 December 2022 at 11:21:29 UTC, Basile B. wrote:

 On Monday, 12 December 2022 at 06:13:20 UTC, youSureAboutThat 
 wrote:

 [...]


 You're supposed to catch this bug during development and with 
 your tests but here the real problem is "-release" (that 
 problem's been mentioned earlier).

 That switch is not well designed. Bound checks could have their 
 own front-end optimizations so we could have

     -boundscheck={on|optimized|off}

 instead of the current system, tied to the language safety 
 system.


to extend the idea. The problem would be then that there's no way 
to disable assertions, so we also need

     -assertion={on|off}

And then we can drop -release. Actually release is more like a 
set of switches that's not always ideal. It would be better to 
have the oppostunity to control each single particular code 
instrumentation.



 Dec 12 2022



 next sibling parent  Basile B. <b2.temp gmx.com>  writes:


On Monday, 12 December 2022 at 12:23:08 UTC, Basile B. wrote:

 On Monday, 12 December 2022 at 11:21:29 UTC, Basile B. wrote:

 On Monday, 12 December 2022 at 06:13:20 UTC, youSureAboutThat 
 wrote:

 [...]


 You're supposed to catch this bug during development and with 
 your tests but here the real problem is "-release" (that 
 problem's been mentioned earlier).

 That switch is not well designed. Bound checks could have 
 their own front-end optimizations so we could have

     -boundscheck={on|optimized|off}

 instead of the current system, tied to the language safety 
 system.


 to extend the idea. The problem would be then that there's no 
 way to disable assertions, so we also need

     -assertion={on|off}

 And then we can drop -release. Actually release is more like a 
 set of switches that's not always ideal. It would be better to 
 have the oppostunity to control each single particular code 
 instrumentation.


well TIL we have

     -check=[assert|bounds|in|invariant|out|switch][=[on|off]]

so why bother why -release at all ?
You can control finely each instrumentation already 😁😁



 Dec 12 2022



prev sibling  parent reply Adam D Ruppe <destructionator gmail.com>  writes:


On Monday, 12 December 2022 at 12:23:08 UTC, Basile B. wrote:

 The problem would be then that there's no way to disable 
 assertions, so we also need

     -assertion={on|off}


Already there:

   -check=[assert|bounds|in|invariant|out|switch][=[on|off]]
                     enable or disable specific checks
   -checkaction=[D|C|halt|context]
                     behavior on assert/boundscheck/finalswitch


 And then we can drop -release.


-release should never have existed. It is absolutely unacceptable 
to ever use is in real world code under any circumstances.



 Dec 12 2022



 next sibling parent  Siarhei Siamashka <siarhei.siamashka gmail.com>  writes:


On Monday, 12 December 2022 at 13:22:56 UTC, Adam D Ruppe wrote:

 -release should never have existed. It is absolutely 
 unacceptable to ever use is in real world code under any 
 circumstances.


Well, somebody may claim that it is absolutely unacceptable to 
have real world D code without annotating most of it as  safe 
under any circumstances. If you don't annotate it as  safe, then 
you may accidentally use functions like 
https://dlang.org/library/std/string/from_stringz.html in an 
inappropriate way or have stack variable references escaping the 
scope. Memory safety is more than just bounds checking.

And then -release won't be a problem after your code is  safe.



 Dec 12 2022



prev sibling  parent  Dukc <ajieskola gmail.com>  writes:


On Monday, 12 December 2022 at 13:22:56 UTC, Adam D Ruppe wrote:

 On Monday, 12 December 2022 at 12:23:08 UTC, Basile B. wrote:

 The problem would be then that there's no way to disable 
 assertions, so we also need

     -assertion={on|off}


 Already there:

   -check=[assert|bounds|in|invariant|out|switch][=[on|off]]
                     enable or disable specific checks
   -checkaction=[D|C|halt|context]
                     behavior on assert/boundscheck/finalswitch


Plus, you should not have much more desire to disable assertions 
than bounds checking. Failing a disabled assert is undefined 
behaviour, meaning memory safety is off if they are disabled. If 
you want an assertion you can disable for performance, write 
`debug assert`.



 Dec 12 2022



prev sibling  parent  Siarhei Siamashka <siarhei.siamashka gmail.com>  writes:


On Monday, 12 December 2022 at 11:21:29 UTC, Basile B. wrote:

  trusted foo()
 {
     int[5] array = [0, 1, 2, 3, 4];

     for (size_t i = 0; i < array.length + 1; i++)
         writeln("Element ", i, " = ", array[i]);
 }


 You're supposed to catch this bug during development and with 
 your tests


The amount of  trusted and  system code is supposed to be 
relatively small in any application (this is possible thanks to 
the Pareto principle), so it can be reviewed with extra scrutiny 
and carefully tested. And this code is the primary suspect when 
troubleshooting in case of troubles.

Testing and reviewing efforts can be allocated more efficiently 
when  safe and  trusted/ system code is segregated.


 but here the real problem is "-release" (that problem's been 
 mentioned earlier).


The real problem is not "-release". But the fact that way too 
much code gets annotated as  system for no reason. And for 
beginners even effectively without their intention/consent.


 That switch is not well designed. Bound checks could have their 
 own front-end optimizations so we could have

     -boundscheck={on|optimized|off}


Such switch already exists. And I'm using it during development 
for very narrow specific purposes:

   1. To do a test build with "-boundscheck=off" and run 
benchmarks. If the performance difference is not measurable, then 
everything is fine. If there's some difference, then identify the 
part of code responsible for it and consider the risks/benefits 
of changing specifically this part to  trusted/ system.

   2. When debugging, "-boundscheck=on" for everything may assist 
in catching an unexpected bug in  trusted/ system code.


 instead of the current system, tied to the language safety 
 system.


Why instead? The two extreme options ("full checks everywhere" 
vs. "no checks at all") offered by the command line switches are 
not flexible and both have major disadvantages.



 Dec 12 2022



prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


Very good post!

On 12/5/2022 11:57 AM, H. S. Teoh wrote:

 Similarly, D's initialized-by-default variables are often touted as a
 big thing, but overall issues with uninitialized variables only
 constitute about 1% of the total issues.


True, I did not encounter this bug that often. But, and this is a big but, they 
cost me a *lot* of time to find, sometimes days. This is because when you'd 
close in on where the bug was, it would dance away. The way it exhibits is 
totally dependent on everything else in the program.

That's why it's a very serious problem.



 Dec 08 2022



  parent reply Max Samukha <maxsamukha gmail.com>  writes:


On Thursday, 8 December 2022 at 17:55:21 UTC, Walter Bright wrote:


 That's why it's a very serious problem.


There seem to be no data to support the claim that default 
initialization aleviates the problem of subtle bugs.



 Dec 08 2022



 next sibling parent reply Max Samukha <maxsamukha gmail.com>  writes:


On Friday, 9 December 2022 at 06:38:36 UTC, Max Samukha wrote:

 On Thursday, 8 December 2022 at 17:55:21 UTC, Walter Bright 
 wrote:


 That's why it's a very serious problem.


 There seem to be no data to support the claim that default 
 initialization aleviates the problem of subtle bugs.


*alleviates



 Dec 08 2022



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 12/8/2022 10:48 PM, Max Samukha wrote:

 *alleviates


No need to worry about non-substantive typos, as nitpicking them is off topic.



 Dec 09 2022



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 12/8/2022 10:38 PM, Max Samukha wrote:

 There seem to be no data to support the claim that default initialization 
 aleviates the problem of subtle bugs.


Plenty of personal experience resolving heisenbugs from lack of default 
initialization.



 Dec 09 2022



prev sibling next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 12/5/2022 11:57 AM, H. S. Teoh wrote:

 Most interesting point here is that the largest category of bugs is
 use-after-free bugs, constituting 34% of the reported issues.  (Arguably
 we should include "object lifecycle/lifetime" in this category, but I
 think those refer to bugs in the JS implementation. In any case, it
 doesn't change the conclusion.)  This is strong evidence that memory
 management is a major source of bugs, and a strong argument for GC use
 in application code.


I'm a bit surprised at this, but maybe I shouldn't. C++ doesn't have a good 
feature set to prevent use-after-free.



 D's bounds checks are often touted as a major feature to prevent issues
 with buffer overflow and out-of-bounds accesses.  Interestingly, "buffer
 overflow" and "out of bounds..." add up only to about 14% of the total
 issues.  Nothing to sneeze at, but nonetheless not as big an issue as
 use-after-free bugs.


The language here is C++, and C++ has touted that if you use the latest C++ 
features, you'll have fewer bounds problems. I suspect that is the cause of the 
reduction. With C code, the percent is a lot higher.



 Dec 08 2022



prev sibling next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 12/5/2022 11:57 AM, H. S. Teoh wrote:

 Most interestingly, "double free" only has 3 counts of the total, less
 than 1%, compared with "use after free", which constitute the largest
 category of issues.  This seems to suggest that it's not memory
 management in general that's necessarily problematic, but it's keeping
 track of the *lifetime* of allocated memory.  One could say that this is
 proof that lifetime is a complex problem. But again it's a strong
 argument that the GC brings a major benefit: it relieves the programmer
 from having to worry about lifetime issues.  You can instantly be freed
 from 34% of security issues, if the above numbers are anything to go by.


This is a good interpretation of the results.



 Dec 08 2022



prev sibling  parent  areYouSureAboutThat <areYouSureAboutThat gmail.com>  writes:


On Monday, 5 December 2022 at 19:57:39 UTC, H. S. Teoh wrote:


well.. for while today, the GC was indeed my enemy...
I was trying to exhaust my heap with this:

module test;

void main()
{
   int *p = new int();
   while (p != null) p = new int();
}

but wtf!
.. rattling my brain...
.. why can I do this in C++ and not in D...wtf..
..wtf... god damn it! why won't you work!

Took me a while to work out I needed to do this first:

import core.memory;
GC.disable;

Nonetheless, I'm glad the GC stepped in, and stopped my heap from 
overflowing, and preventing my pc from coming to a complete halt 
;-)



 Dec 15 2022