Mike Parker <aldacron gmail.com> writes:

DIP 1028, "Make  safe the Default", has been accepted without 
comment.

https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md

ag0aep6g <anonymous example.com> writes:

On 21.05.20 15:51, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without comment.

just another brick in the wall

12345swordy <alexanderheistermann gmail.com> writes:

On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md

I guess they be more open to dips that fixes holes in the "safe 
by default" feature then.

Seb <seb wilzba.ch> writes:

On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md

"without comment" - even though there were a lot of unaddressed 
problems :/

Great! So what's the entire point of this process?
To give people the illusion of progress and participation?

Why we can't we have a technical board where the community can 
vote in experts and potentially companies could even buy a seat 
for $$$ which would mean a lot more for them than the current 
very vague sponsorship options.
I'm aware that Walter doesn't like the idea of giving up 
ownership, but it makes all the other people question why they 
should still bother with this process and not simply fork and 
move to an open, transparent development...

Adam D. Ruppe <destructionator gmail.com> writes:

On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a seat 
 for $$$ which would mean a lot more for them than the current 
 very vague sponsorship options.

ditto, I think we should have like a seven person elected DIP 
committee who pass/fail things by majority vote. It is obvious to 
me that the current process is totally useless.

SashaGreat <s g.com> writes:

On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a 
 seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.

 ditto, I think we should have like a seven person elected DIP 
 committee who pass/fail things by majority vote. It is obvious 
 to me that the current process is totally useless.

Exactly and I even remember there was even a discussion about 
this on reddit (/r/programming) sometime ago where someone was 
pointing the flaws on D, and someone said the DIP process was one 
of them, and there were some known people around here replying 
that comment saying it was not the case.

The discussion topic about this DIP 
(https://forum.dlang.org/thread/wkdpnzarkbtqryighzpx forum.dlang.org) had 210
replies, some concerns and in the end was ACCEPT WITHOUT ANY COMMENT.

This is what I call a waste of everybody's time.

SG.

Les De Ridder <les lesderid.net> writes:

On Thursday, 21 May 2020 at 16:48:22 UTC, SashaGreat wrote:
 On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community 
 can vote in experts and potentially companies could even buy 
 a seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.

 ditto, I think we should have like a seven person elected DIP 
 committee who pass/fail things by majority vote. It is obvious 
 to me that the current process is totally useless.

 Exactly and I even remember there was even a discussion about 
 this on reddit (/r/programming) sometime ago where someone was 
 pointing the flaws on D, and someone said the DIP process was 
 one of them, and there were some known people around here 
 replying that comment saying it was not the case.

 The discussion topic about this DIP 
 (https://forum.dlang.org/thread/wkdpnzarkbtqryighzpx forum.dlang.org) had 210
replies, some concerns and in the end was ACCEPT WITHOUT ANY COMMENT.

 This is what I call a waste of everybody's time.

Completely agree. The way DIPs are being handled is quite 
disappointing
and unproductive. If the authors aren't even intending to address 
(or
even acknowledge) difficult comments, why even write a DIP and 
have
the community spend time on it at all?

It also is not very encouraging to potential DIP authors who 
aren't
Walter (or Átila or Andrei).

Bruce Carneal <bcarneal gmail.com> writes:

On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a 
 seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.

 ditto, I think we should have like a seven person elected DIP 
 committee who pass/fail things by majority vote. It is obvious 
 to me that the current process is totally useless.

As noted earlier, I'm with Steve, Seb, Adam, and "everyone but 
Walter" as far as I can see.  If this stands we'll have gone from 
 safe meaning "the compiler is responsible" to "the compiler 
can't guarantee anything unless you're pure D all the way down".

Atila, what's your take on all this?  Is it fork time?

Atila Neves <atila.neves gmail.com> writes:

On Thursday, 21 May 2020 at 23:49:22 UTC, Bruce Carneal wrote:
 On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 [...]

 ditto, I think we should have like a seven person elected DIP 
 committee who pass/fail things by majority vote. It is obvious 
 to me that the current process is totally useless.

 As noted earlier, I'm with Steve, Seb, Adam, and "everyone but 
 Walter" as far as I can see.  If this stands we'll have gone 
 from  safe meaning "the compiler is responsible" to "the 
 compiler can't guarantee anything unless you're pure D all the 
 way down".

 Atila, what's your take on all this?  Is it fork time?

No.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/21/20 7:49 PM, Bruce Carneal wrote:
 On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community can vote 
 in experts and potentially companies could even buy a seat for $$$ 
 which would mean a lot more for them than the current very vague 
 sponsorship options.

 ditto, I think we should have like a seven person elected DIP 
 committee who pass/fail things by majority vote. It is obvious to me 
 that the current process is totally useless.

 
 As noted earlier, I'm with Steve, Seb, Adam, and "everyone but Walter" 
 as far as I can see.  If this stands we'll have gone from  safe meaning 
 "the compiler is responsible" to "the compiler can't guarantee anything 
 unless you're pure D all the way down".
 
 Atila, what's your take on all this?  Is it fork time?

A fork does exist. As expected it went nowhere. 
https://bitbucket.org/larsivi/amber/wiki/Home. There is a paradox about 
forking the language - anyone good enough to lead a successful fork 
would also be wise enough to work on the D language instead.

A committee would work if the community were at least one order of 
magnitude larger. It's just big numbers. Walter is world-class, another 
way of saying there's only a few of comparable strength in the world. A 
larger community would mean a larger likelihood of there being other 
people of comparable strength in it. Those could form a committee. As 
things are, Walter is so much stronger than every one of us, the 
relationship is highly asymmetric. His strength comes with equally 
unique ability to explain and debate, which makes many of the 
discussions in forums very frustrating.

ag0aep6g <anonymous example.com> writes:

On 23.05.20 17:07, Andrei Alexandrescu wrote:
 A fork does exist. As expected it went nowhere. 
 https://bitbucket.org/larsivi/amber/wiki/Home. There is a paradox about 
 forking the language - anyone good enough to lead a successful fork 
 would also be wise enough to work on the D language instead.

I can only read that as you calling larsivi not "good enough to lead a 
successful fork" and not "wise enough to work on the D language instead".

Walter says: "Belittling [others] [...] is unprofessional behavior."

He was refering to other people on the forum, of course. But I would 
expand that to authors of other projects as well, particularly ones that 
are (even) smaller than D. Punching down is not cool.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/23/20 6:49 PM, ag0aep6g wrote:
 On 23.05.20 17:07, Andrei Alexandrescu wrote:
 A fork does exist. As expected it went nowhere. 
 https://bitbucket.org/larsivi/amber/wiki/Home. There is a paradox 
 about forking the language - anyone good enough to lead a successful 
 fork would also be wise enough to work on the D language instead.

 
 I can only read that as you calling larsivi not "good enough to lead a 
 successful fork" and not "wise enough to work on the D language instead".
 
 Walter says: "Belittling [others] [...] is unprofessional behavior."
 
 He was refering to other people on the forum, of course. But I would 
 expand that to authors of other projects as well, particularly ones that 
 are (even) smaller than D. Punching down is not cool.

The point is on the other side - the barrier is very high. Carrying a 
programming language design and implementation is an extremely difficult 
task.

Arine <arine1283798123 gmail.com> writes:

On Saturday, 23 May 2020 at 15:07:16 UTC, Andrei Alexandrescu 
wrote:
 On 5/21/20 7:49 PM, Bruce Carneal wrote:
 On Thursday, 21 May 2020 at 16:32:32 UTC, Adam D. Ruppe wrote:
 On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 Why we can't we have a technical board where the community 
 can vote in experts and potentially companies could even buy 
 a seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.

 ditto, I think we should have like a seven person elected DIP 
 committee who pass/fail things by majority vote. It is 
 obvious to me that the current process is totally useless.

 
 As noted earlier, I'm with Steve, Seb, Adam, and "everyone but 
 Walter" as far as I can see.  If this stands we'll have gone 
 from  safe meaning "the compiler is responsible" to "the 
 compiler can't guarantee anything unless you're pure D all the 
 way down".
 
 Atila, what's your take on all this?  Is it fork time?

 A fork does exist. As expected it went nowhere. 
 https://bitbucket.org/larsivi/amber/wiki/Home. There is a 
 paradox about forking the language - anyone good enough to lead 
 a successful fork would also be wise enough to work on the D 
 language instead.

Arguably they would then be wise enough to stay away from D (and 
have).

 His strength comes with equally unique ability to explain and 
 debate, which makes many of the discussions in forums very 
 frustrating.

It is unique that's for sure. That's a good way to put it without 
being rude.

bachmeier <no spam.net> writes:

On Thursday, 21 May 2020 at 16:14:02 UTC, Seb wrote:
 On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md

 "without comment" - even though there were a lot of unaddressed 
 problems :/

 Great! So what's the entire point of this process?
 To give people the illusion of progress and participation?

 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a seat 
 for $$$ which would mean a lot more for them than the current 
 very vague sponsorship options.
 I'm aware that Walter doesn't like the idea of giving up 
 ownership, but it makes all the other people question why they 
 should still bother with this process and not simply fork and 
 move to an open, transparent development...

I honestly don't know if that would help. We'd be moving from a 
system where Walter makes decisions based on his mood on a 
particular day to one where others make decisions based on their 
moods on a particular day. The only thing worse than letting one 
person choose what to implement is having a group of people 
choose what to implement.

Everyone has their own view of what is important. In my case, 
it's beginners and appealing to less technical users. Others view 
20-year C++ programmers that specialize in performance 
optimizations as the only ones that matter. Needless to say, 
there's not a lot of overlap in the set of changes we think make 
sense. No matter who is making the decisions, the tradeoff 
between ease of use and technical awesomeness will continue to 
exist.

The problem as I see it is someone making a decision on his own 
DIP. That just doesn't make any sense to me, and I've stated that 
numerous times. Walter has a tendency to throw gas on the fire by 
ignoring much of the feedback and not spending time to understand 
the points others are making when he does respond. I really think 
you should have to convince *someone else* that your proposal is 
reasonable.

Paul Backus <snarwin gmail.com> writes:

On Thursday, 21 May 2020 at 17:03:49 UTC, bachmeier wrote:
 The problem as I see it is someone making a decision on his own 
 DIP. That just doesn't make any sense to me, and I've stated 
 that numerous times. Walter has a tendency to throw gas on the 
 fire by ignoring much of the feedback and not spending time to 
 understand the points others are making when he does respond. I 
 really think you should have to convince *someone else* that 
 your proposal is reasonable.

In principle, at least, this is why we have two "language 
maintainers," Walter and Atila (previously, Walter and Andrei).

I think the real problem here is the lack of communication. As it 
stands, we have no way to tell whether feedback was considered or 
ignored, or what the ultimate rationale behind this decision 
was--we can only speculate. Even if we suppose for the sake of 
argument that the decision is sound on a technical level, this is 
poor leadership, and bodes ill for the future of the D language 
and its community.

bachmeier <no spam.net> writes:

On Thursday, 21 May 2020 at 17:49:27 UTC, Paul Backus wrote:
 On Thursday, 21 May 2020 at 17:03:49 UTC, bachmeier wrote:
 The problem as I see it is someone making a decision on his 
 own DIP. That just doesn't make any sense to me, and I've 
 stated that numerous times. Walter has a tendency to throw gas 
 on the fire by ignoring much of the feedback and not spending 
 time to understand the points others are making when he does 
 respond. I really think you should have to convince *someone 
 else* that your proposal is reasonable.

 In principle, at least, this is why we have two "language 
 maintainers," Walter and Atila (previously, Walter and Andrei).

There's a big difference between being part of a three-person 
committee discussing a proposal by Walter and working together 
with Walter to make a decision on his own proposal. It's 
certainly doesn't pass the smell test.

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Thu, May 21, 2020 at 05:49:27PM +0000, Paul Backus via
Digitalmars-d-announce wrote:
[...]
 I think the real problem here is the lack of communication. As it
 stands, we have no way to tell whether feedback was considered or
 ignored, or what the ultimate rationale behind this decision was--we
 can only speculate.

I've said pretty much the same thing before, but it seems to have fallen
on deaf ears. :-/  What's frustrating to the community isn't primarily
the technical aspects (though of course, that's also a factor), but the
human aspect of communication. Or rather, the lack thereof.

I'd even grant that Walter, being BFDL, can make whatever arbitrary
decisions he wants, but at the very least acknowledge the existence of
the rest of us.  "Accepted without comment" amounts to denial that we
even exist, considering how much feedback was given on this DIP.  Even a
"accepted in spite of community feedback" is better than "without
comment".

(Maybe our virtual non-existence might ultimately become actual
non-existence as this community shrinks to zero. :-P)


 Even if we suppose for the sake of argument that the decision is sound
 on a technical level, this is poor leadership, and bodes ill for the
 future of the D language and its community.

D excels at the technical aspects, but time and time again has shown
that it lacks proper management / leadership.  Contrary to my own
inclinations I'm compelled to suggest hiring a *non-technical* person to
take up the management/leadership roles (emphatically non-technical,
because let's face it, we techies just don't have the people skillz it
takes to do this properly), because the current situation clearly isn't
working, and is quite detrimental to D and its future.


T

-- 
Computers shouldn't beep through the keyhole.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 11:36 AM, H. S. Teoh wrote:
 I'd even grant that Walter, being BFDL, can make whatever arbitrary
 decisions he wants, but at the very least acknowledge the existence of
 the rest of us.  "Accepted without comment" amounts to denial that we
 even exist, considering how much feedback was given on this DIP.  Even a
 "accepted in spite of community feedback" is better than "without
 comment".

I've discussed it at length in the n.g. with y'all.

aberba <karabutaworld gmail.com> writes:

On Thursday, 21 May 2020 at 18:36:51 UTC, H. S. Teoh wrote:
 On Thu, May 21, 2020 at 05:49:27PM +0000, Paul Backus via 
 Digitalmars-d-announce wrote: [...]

 Even if we suppose for the sake of argument that the decision 
 is sound on a technical level, this is poor leadership, and 
 bodes ill for the future of the D language and its community.

 D excels at the technical aspects, but time and time again has 
 shown that it lacks proper management / leadership.  Contrary 
 to my own inclinations I'm compelled to suggest hiring a 
 *non-technical* person to take up the management/leadership 
 roles (emphatically non-technical, because let's face it, we 
 techies just don't have the people skillz it takes to do this 
 properly), because the current situation clearly isn't working, 
 and is quite detrimental to D and its future.


 T

Ha ha. I tried suggesting something like this some yrs ago as I 
saw it happen and deadalnix bashed me like crazy. Didn't 
articulate it as well as you just did though. This needs to 
happen.

rikki cattermole <rikki cattermole.co.nz> writes:

On 28/05/2020 12:33 AM, aberba wrote:
 On Thursday, 21 May 2020 at 18:36:51 UTC, H. S. Teoh wrote:
 On Thu, May 21, 2020 at 05:49:27PM +0000, Paul Backus via 
 Digitalmars-d-announce wrote: [...]

 
 Even if we suppose for the sake of argument that the decision is 
 sound on a technical level, this is poor leadership, and bodes ill 
 for the future of the D language and its community.

 D excels at the technical aspects, but time and time again has shown 
 that it lacks proper management / leadership.  Contrary to my own 
 inclinations I'm compelled to suggest hiring a *non-technical* person 
 to take up the management/leadership roles (emphatically 
 non-technical, because let's face it, we techies just don't have the 
 people skillz it takes to do this properly), because the current 
 situation clearly isn't working, and is quite detrimental to D and its 
 future.


 T

 
 Ha ha. I tried suggesting something like this some yrs ago as I saw it 
 happen and deadalnix bashed me like crazy. Didn't articulate it as well 
 as you just did though. This needs to happen.

I've been saying since 2012 that we need a project manager to help with 
communication.

Mike has taken on parts of this role and has done quite a good job of it.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 10:03 AM, bachmeier wrote:
 Walter makes decisions based on his mood on a particular day

That's uncalled for.

bachmeier <no spam.net> writes:

On Thursday, 21 May 2020 at 20:48:18 UTC, Walter Bright wrote:
 On 5/21/2020 10:03 AM, bachmeier wrote:
 Walter makes decisions based on his mood on a particular day

 That's uncalled for.

Regional variation in English? Translation: You make your 
decisions based on how you feel about the situation at a point in 
time. "Mood" is used because there's some subjectivity and "gut 
feel" involved in the decision. It's not intended to be negative, 
it's simply a description of how any human makes any decision. 
That's why I said the same about how a committee would make 
decisions.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 2:45 PM, bachmeier wrote:
 On Thursday, 21 May 2020 at 20:48:18 UTC, Walter Bright wrote:
 On 5/21/2020 10:03 AM, bachmeier wrote:
 Walter makes decisions based on his mood on a particular day

 That's uncalled for.

 
 Regional variation in English? Translation: You make your decisions based on
how 
 you feel about the situation at a point in time. "Mood" is used because
there's 
 some subjectivity and "gut feel" involved in the decision. It's not intended
to 
 be negative, it's simply a description of how any human makes any decision. 
 That's why I said the same about how a committee would make decisions.

As a native English speaker, telling someone they are making decisions based on 
moods is telling them they are making irrational decisions.

I accept that you did not intend it in this manner, but you should be aware
that 
many people would be offended by it.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/21/20 12:14 PM, Seb wrote:
 On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md

 
 "without comment" - even though there were a lot of unaddressed problems :/
 
 Great! So what's the entire point of this process?
 To give people the illusion of progress and participation?

Agree. I will not be participating in the DIP process from now on. It is 
a complete waste of time. Walter should just make the changes he wants 
and not bother with the facade of discussion.

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 10:26 AM, Steven Schveighoffer wrote:
 Agree. I will not be participating in the DIP process from now on. It is a 
 complete waste of time. Walter should just make the changes he wants and not 
 bother with the facade of discussion.

Many replies to you, Steven:

https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Final_Review_336354.html

I did not ignore you. I just didn't agree.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/21/20 4:59 PM, Walter Bright wrote:
 On 5/21/2020 10:26 AM, Steven Schveighoffer wrote:
 Agree. I will not be participating in the DIP process from now on. It 
 is a complete waste of time. Walter should just make the changes he 
 wants and not bother with the facade of discussion.

 
 Many replies to you, Steven:
 
 https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Fina
_Review_336354.html 

Many unsatisfactory replies. Eventually my attempts to explain my 
position were ignored. It's the equivalent of walking off stage because 
you lost the debate, only to claim you won because you stopped talking.

My points were straightforward, easy to understand, vastly agreed upon 
(and echoed) by all of the experienced people here (except you), and 
your rebuttals were extremely dubious, depending on hypothetical 
non-sequiturs with an imagined developer who doesn't understand how 
prototypes might work. Short story is, you made poor arguments and 
considered the matter closed "without comment". This is extremely 
unsatisfactory.

 I did not ignore you. I just didn't agree.

You didn't agree, and that is your choice. But you also didn't address 
any of the points I made, instead choosing to focus on confusion from an 
imaginary person who has no clue why his safe code cannot call unsafe 
functions even though the attributes are not explicit (which means he 
should probably spend the 10 seconds to learn and become a better 
programmer).

The unfortunate end result of this change is that safety will be gutted 
with all C functions being trusted by default, ironically by the person 
who is claiming that memory safety is of utmost importance, and how C 
got it wrong. All without delivering a single convincing argument as to 
why he is right.

I even put forth a completely ignored compromise solution that would 
have solved the problem and allowed extern(C) functions to be considered 
 safe by default: 
https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Final_Review_336354.html#N336968

This is why I consider the process a waste of time, and why I'm done 
participating.

-Steve

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On Thursday, 21 May 2020 at 21:41:53 UTC, Steven Schveighoffer 
wrote:
 The unfortunate end result of this change is that safety will 
 be gutted with all C functions being trusted by default

I'm really sorry, Walter, but I have to agree with Steve on this 
point.  This was the one aspect of the DIP that I really wanted 
to get modified, I also proposed a number of options for how to 
do so, and I'm really worried about the consequences of this 
decision.

I recognize and respect your right to make decisions as you see 
fit for the language, but I really wish I could persuade you to 
change your mind about this one :-)

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 2:41 PM, Steven Schveighoffer wrote:
 I even put forth a completely ignored compromise solution that would have
solved 
 the problem and allowed extern(C) functions to be considered  safe by default: 
 https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Final_Review
336354.html#N336968 

I did see it. I can't even get across the notion of what "undefined symbol
xxxx" 
messages from the linker mean to a lot of programmers. I'll never get this one 
across. The linking process is already as complex as I dare.

BTW, two symbols resolving to the same address can be a bit awkward in some 
object formats and isn't really supported, like with COMDATs. In my experience, 
unless C or C++ compilers generate certain constructs, linkers (and debuggers) 
don't work for those constructs, whether they are documented to or not.


 This is why I consider the process a waste of time, and why I'm done
participating.

The level of negativity in that thread was what caused me to stop responding, 
though I continued reading. Every reply I made produced 10 responses, an 
exponential explosion, and yet I was just repeating myself. Two sides to every 
story.

FWIW, I am going to try again with another post here, for those who want a 
convenient summary of the rationale.

Faux Amis <faux amis.com> writes:

On 2020-05-22 03:16, Walter Bright wrote:

 
 The level of negativity in that thread was what caused me to stop 
 responding, though I continued reading. Every reply I made produced 10 
 responses, an exponential explosion, and yet I was just repeating 
 myself. Two sides to every story.
 
 FWIW, I am going to try again with another post here, for those who want 
 a convenient summary of the rationale.

Just a suggestion, but sometimes matters are best discussed over 
audio/video. Would having a public teams/zoom/.. meeting be helpful?

I would definitely listen/watch; even if I were muted and could only 
chat maybe.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/23/2020 4:26 AM, Faux Amis wrote:
 Just a suggestion, but sometimes matters are best discussed over audio/video. 
 Would having a public teams/zoom/.. meeting be helpful?
 
 I would definitely listen/watch; even if I were muted and could only chat
maybe.

You're right, and that is the whole purpose behind DConf. It's amazing how our 
differences melt away when discussing with a beer in hand :-)

Faux Amis <faux amis.com> writes:

On 2020-05-24 00:15, Walter Bright wrote:
 On 5/23/2020 4:26 AM, Faux Amis wrote:
 Just a suggestion, but sometimes matters are best discussed over 
 audio/video. Would having a public teams/zoom/.. meeting be helpful?

 I would definitely listen/watch; even if I were muted and could only 
 chat maybe.

 
 You're right, and that is the whole purpose behind DConf. It's amazing 
 how our differences melt away when discussing with a beer in hand :-)

I meant it could be something being done more often, Dconf is (normally) 
only once a year. Maybe discussions like these need to be discussed more 
frequently.. quarterly, or even a quick official Dtalk every month :)

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On Thursday, 21 May 2020 at 20:59:08 UTC, Walter Bright wrote:
 Many replies to you, Steven:

 https://digitalmars.com/d/archives/digitalmars/D/Discussion_Thread_DIP_1028--Make_safe_the_Default--Final_Review_336354.html

 I did not ignore you. I just didn't agree.

One concern here is that these responses are scattered across 
different parts of a long discussion thread.  So it probably 
would be a good idea for the acceptance to be accompanied by an 
explanation of what the major objections were to the DIP, and why 
they were discounted.

It's not so different from the way that, at the end of a talk or 
presentation, it's a good idea to recap the key points that have 
already been covered.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 2:44 PM, Joseph Rushton Wakeling wrote:
 One concern here is that these responses are scattered across different parts
of 
 a long discussion thread.  So it probably would be a good idea for the 
 acceptance to be accompanied by an explanation of what the major objections
were 
 to the DIP, and why they were discounted.
 
 It's not so different from the way that, at the end of a talk or presentation, 
 it's a good idea to recap the key points that have already been covered.

Fair enough. I'll do so. Stay tuned.

12345swordy <alexanderheistermann gmail.com> writes:

On Friday, 22 May 2020 at 00:50:00 UTC, Walter Bright wrote:
 On 5/21/2020 2:44 PM, Joseph Rushton Wakeling wrote:
 One concern here is that these responses are scattered across 
 different parts of a long discussion thread.  So it probably 
 would be a good idea for the acceptance to be accompanied by 
 an explanation of what the major objections were to the DIP, 
 and why they were discounted.
 
 It's not so different from the way that, at the end of a talk 
 or presentation, it's a good idea to recap the key points that 
 have already been covered.

 Fair enough. I'll do so. Stay tuned.

Thank you.

Having an explanation on why the DIP is accepted despite being 
highly controversial would be sufficient.

-Alex

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 9:14 AM, Seb wrote:
 On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md

 
 "without comment" - even though there were a lot of unaddressed problems :/

I did address, several times over, the problems in the discussion thread.


 Great! So what's the entire point of this process?
 To give people the illusion of progress and participation?

Consider that several of my own DIPs were rejected due to community response.


 Why we can't we have a technical board where the community can vote in experts 
 and potentially companies could even buy a seat for $$$ which would mean a lot 
 more for them than the current very vague sponsorship options.
 I'm aware that Walter doesn't like the idea of giving up ownership, but it
makes 
 all the other people question why they should still bother with this process
and 
 not simply fork and move to an open, transparent development...

I expected flak from this decision. I'm prepared to take the flak because this 
is the right decision. I did not make it lightly.

Please keep in mind that I've made other unpopular decisions that have proven 
their worth over time. I hope you'll reserve judgement until we all see how
this 
change plays out.

If it does turn out badly, it's on me and I'll take my lumps.

Arine <arine1283798123 gmail.com> writes:

On Thursday, 21 May 2020 at 20:46:09 UTC, Walter Bright wrote:
 On 5/21/2020 9:14 AM, Seb wrote:
 Why we can't we have a technical board where the community can 
 vote in experts and potentially companies could even buy a 
 seat for $$$ which would mean a lot more for them than the 
 current very vague sponsorship options.
 I'm aware that Walter doesn't like the idea of giving up 
 ownership, but it makes all the other people question why they 
 should still bother with this process and not simply fork and 
 move to an open, transparent development...

 I expected flak from this decision. I'm prepared to take the 
 flak because this is the right decision. I did not make it 
 lightly.

That's interesting that you are saying it is the "right" 
decision. Let's analyze that for a second:

What this boils down to is

     Simplicity Vs. Memory Safety

Is the **correct** code actually simpler? A beginner that doesn't 
have the knowledge of D's safety system will ultimately create C 
bindings that are implicitly  safe without realizing the 
consequences of their actions.

This change introduces a risk to memory safety, extern(C) 
declarations that were  system will now be flipped to  safe. No, 
putting  system: at the top isn't a solution, extern(C) functions 
aren't all nicely put in their own modules. Druntime and Phobos 
both have extern(C) declarations throughout their source; I 
guarantee you missed them with your PR's trying to illustrate how 
simple it is. As others will most definitely miss them as well.

Reasons for Memory Safety:
- Less error prone and simpler for beginners, the default is what 
the *correct* behavior should be. They organically discover  safe 
writing C bindings.
- Less error prone for advanced users, won't accidentally call a 
C function that isn't explicitly verified as being  trusted.
- Switch to  safe by default won't introduce potential security 
bugs from extern(C) functions not being annotated  system in the 
switch.
- Proven implementation by other safe languages (Rust).

Reasons for Simplicity:
- simpler logic for the compiler
- simpler rules for the user that default to *incorrect* code
- Fear of the unknown, unforeseen complexity
- trust Walter


This is where the divide is, from my perspective. There's this 
fear of the unknown, that making it slightly more complicated 
will introduce these unforeseen complications that will far 
outweigh any benefit that is added from it.

I don't see what that can be. There are other languages that have 
safe by default, and then have C declarations be unsafe by 
default. They don't have any of these devastation complications 
you seem to think that there will be.

This is what doesn't sit right with me. If you could give an 
example of a complication that could occur, then I think you 
could get more people onboard. But it would have to be 
justifiable against the increased security risk of having 
extern(C) declarations be  safe by default. For a DIP that is 
trying to increase safety, it is a bit ironic that it will also 
be reducing safety at the same time.

Otherwise right now, as it stands, this basically boils down to 
Logic Vs. Fear.

 Please keep in mind that I've made other unpopular decisions 
 that have proven their worth over time. I hope you'll reserve 
 judgement until we all see how this change plays out.

 If it does turn out badly, it's on me and I'll take my lumps.

You've also done the opposite, you can't justify it with such a 
statement unless you've never made a mistake. This whole process 
is suppose to avoid exactly this; a single person taking a gamble.

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Thu, May 21, 2020 at 01:46:09PM -0700, Walter Bright via
Digitalmars-d-announce wrote:
[...]
 I expected flak from this decision. I'm prepared to take the flak
 because this is the right decision. I did not make it lightly.

This makes it sound like you think that those who disagree with you
disagree with  safe by default.  That is not the case.  I think in
general, (almost?) everyone here agrees that safety by default is the
way to go.  We all agree that  safe by default ought to be in the
language, in one form or another. That's not the dispute here.

The dispute concerns the specific implementation of  safe by default as
set out in this DIP. Specifically, the weakening of the promise of  safe
by implicitly trusting extern(C) declarations to be  safe, which is
equivalent to programming by convention ("trust the programmer to
remember to write  system on extern(C) prototypes"). Which, ironically,
undermines the whole purpose of this DIP. (That is, unless I
misunderstood, and the whole purpose of this DIP is to undermine  safety
and thereby make it even more of a joke than it currently is right now.)


 Please keep in mind that I've made other unpopular decisions that have
 proven their worth over time. I hope you'll reserve judgement until we
 all see how this change plays out.

[...]

Oh I'm waiting to see this one plays out, you bet!  What will come of
trusting the programmer to do the right thing when it comes to
annotating extern(C) prototypes with  system?  Well, I'm sure it will
all work out well in the end, since, after all, it isn't as if we didn't
have ~50 years or so of experience with programming by convention,
y'know, esp. in C code (that we should implicitly trust when calling
from D, btw), which has led to beautiful security holes and exploits
caused by memory corruption and other such wonderful things.

Or, since you like airplane analogies, there's absolutely nothing wrong
with designing your cockpit controls such that the default behaviour,
when the pilot does not indicate otherwise, is to nose-dive into the
ground.  After all, if he didn't mean to do that, he should've
remembered to override the default behaviour explicitly, right? That's
what he was trained to do, anyway.  And besides, it's easier to
implement the controls this way, since doing otherwise would add
excessive complication to the design.


T

-- 
The early bird gets the worm. Moral: ewww...

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 4:45 PM, H. S. Teoh wrote:
 This makes it sound like you think that those who disagree with you
 disagree with  safe by default.  That is not the case.

I'm sure you all know what I'm talking about.

Johannes Loher <johannes.loher fg4f.de> writes:

On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md

As others have mentioned, this really is a farce. I understand 
that not everybody will be happy with every decision but the fact 
that there is not even a comment is just disrespectful towards 
all the people that put in a lot of effort to review the DIP. How 
can you seriously expect them to continue to do that if you treat 
them like this? Don't you think their work is valuable? If not, 
why bother with the DIP process at all? It just seems like a 
total waste of everybody’s time. I am really disappointed about 
this...

Walter Bright <newshound2 digitalmars.com> writes:

I have made these points before, but I'll summarize them here
for convenient referral.

----

The proposed amendment to Safe by Default is:

"Extern C and C++ functions without bodies should be  system
by default."

The rationale is that since the compiler cannot check them for  safe,
they should be assumed unsafe.

This is Obviously A Good Idea. Why would I oppose it?

1. I've been hittin' the crack pipe again.
2. I was secretly convinced, but wanted to save face.
3. I make decisions based on consultation with my astrologer.
4. I am evil.

I'm going to play the experience card. Many Obviously Good Ideas
appear again and again. Why they aren't so hot only becomes
clear after (sometimes considerable) experience. Some are:

1. Implicit declaration of variables.
2. AST macros (debated in the n.g. multiple times).
3. Exception specifications.

I'll expound on Exception Specifications, as they are more similar
to this case than the others.

Exception Specifications appeared with Java back in the 90's. With it,
each function must list which exceptions it throws, such as (in D):

     T myFunction() throws Alpha, Gamma;

If myFunction() throws any exception that is not derived from Alpha
or Gamma, the compiler issues a compile-time error.

This is Obviously A Good Idea. Smart, experienced, well-meaning programmers
endorsed it wholeheartedly. (Even C++98 adopted the idea.) One such person,
Bruce Eckel (who incidentally authored the early Zortech C++ manual and
is a friend of mine) authored the well-regarded "Thinking In ..." series of
books, one of which was "Thinking in Java". He bought the idea, and preached
and promoted it heavily.

Then came the hangover. Bruce wrote a brilliant mea culpa (that has since
sadly vanished from the internet) about what went wrong. The troubles were:

1. It was tedious to go through every function called by myFunction() and
figure out what all the possible Exceptions thrown might be.

2. If a low level function started throwing a new type, every single caller
all the way up the call tree had to add it to its exception specification.

This, of course, was incredibly tedious and annoying. So what happened?

1. Functions would just do a try-catch(Throwable) and ignore the exceptions.
Oops.

2. The Exception Specifications became simply `throws Throwable`. Oops.

What was most damning was the Bruce Eckel himself was horrified to find that
this was his behavior when writing his own code. Other advocates of it also
admitted they did that too. Exception specifications became a farce, and
Java was quietly softened to not require them. Exception specifications were
just ignored by the C++ community.

How does this relate to safe by default?

Consider the common (because that's how D started out) case of:

----- clibrary.d --------

     T massage_data(... many parameters ...);
     ... 200 more such declarations ...

----- app.d ----------

     import clibrary;

     void useClibrary( ... parameters ...) {
         massage_data(parameters);
     }

---------------------

This code, today, does not use annotations and it works. It's been working
for a long time. Now, we implement the amendment, and it stops compiling
because useClibrary is  safe and massage_data is  system. The user is faced
with the following alternatives:

1. Go through 200 functions in clibrary.d and determine which are  safe
and which are  system. This is what we want them to do. We try to motivate
this with compiler error messages. Unfortunately, this is both tedious and
thoroughly impractical, as our poor user Will Not Know which are safe and
which are system. We can correctly annotate core.stdc.stdio because I know
those functions intimately. This is not true for other system C APIs, and
even less true for some third party C library we're trying to interface to.

2. Annotate useClibrary() as  trusted or  system. While easier, this causes
all benefits to  safe by default to be lost.

3. Wrap the call to massage_data() with:

     ()  trusted { massage_data(parameters); } ();

If there are a lot of calls to clibrary, this is going to look pretty awful.
Nobody likes writing or reading such ugly code. It's ok here and there, but
not as a general thing.

4. Edit clibrary.d and make the first line:

      safe:

I submit that, just like with Java, Option 4 is what people will reach for,
nearly every time. I've had some private conversations where people admitted
this was what they'd do. People who knew it was wrong to do that.

If it's  safe by default, and then someone chooses to annotate it with  system
here and there, I'd feel a lot more confident about the accuracy of the code
annotations than if it just had  safe: at the top. At least they tried.

What is actually accomplished with this amendment if it was implemented?

1. Adds a funky, special case rule. It's better to have simple, easily
understood rules than ones with special cases offering little improvement.

2. Existing, working code breaks.

3. The most likely code fixes are to just make it compile, absolutely
nothing safety-wise is improved. The added annotations will be a fraud.

D should not encourage "greenwashing" practices like the Java
exception specification engendered. The compiler cannot vet the accuracy
of bodyless C functions, and we'll just have to live with that. The proposed
amendment does not fix that.

And so, I did not incorporate the proposed amendment to the Safe by Default
DIP.

Paul Backus <snarwin gmail.com> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.

 ----

 [...]

First, thank you for taking the time to summarize your position. 
I understand
your objection to the proposed amendment much better now than I 
did after
only reviewing the discussion thread.

Second, I'd like to present a potential failure mode of DIP 1028 
that I think
still remains unaddressed. Consider the following scenario:

Someone has created bindings for a C library and published them on
code.dlang.org. Because they were written prior to DIP 1028, the 
author assumed
that  system was the default, and didn't bother to add explicit 
annotations to
 system functions. Their code looks like this:

--- clibrary.d

     void monkey_around(...); // assumed  system-by-default

---

Months or years later, I decide to write a D program that makes 
use of these
bindings. By then,  safe-by-default has been fully implemented. I 
add
`clibrary` as a dependency to my Dub project and write the 
following code:

--- app.d

     import clibrary;

     void main() //  safe-by-default
     {
         /* ... code ... */

         monkey_around(...);

         /* ... more code ... */
     }

---

My program compiles with no errors...and then corrupts memory at 
run-time, even
though every line of code I've written is  safe. Oops.

This is the nightmare scenario that people are worried about: 
safety violations
being introduced *silently* into existing, correct D code.

Of course, the author of clibrary.d *should* update their code to 
add explicit
 system annotations, but how will they know there's a problem if 
the compiler
never complains about it? Most likely, by the time they or one of 
their users
finally track down the root of the problem, the damage will 
already have been
done.

Something ought to be done to prevent this. It doesn't have to be 
the exact
proposal from the discussion thread, but doing nothing and 
allowing widespread
silent breakage cannot possibly be the best solution.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/21/2020 8:36 PM, Paul Backus wrote:
 Something ought to be done to prevent this. It doesn't have to be the exact
 proposal from the discussion thread, but doing nothing and allowing widespread
 silent breakage cannot possibly be the best solution.

I can see that happening. A simple example would be:

     extern (C) void free(void* p);
     ...
     free(p);
     free(p);

The thing is, you are no worse off than now. If free() can be misused by
calling 
it from system code, it can be misused by calling it from safe code.

There's no way the compiler can detect this. If you annotate it, you'll just 
have to annotate it correctly. Forcing an annotation just means slapping  safe: 
at the beginning of the file and moving on - it's not going to help.

Dukc <ajieskola gmail.com> writes:

 I can see that happening. A simple example would be:

     extern (C) void free(void* p);
     ...
     free(p);
     free(p);

 The thing is, you are no worse off than now. If free() can be 
 misused by calling it from system code, it can be misused by 
 calling it from safe code.

Wrong :-(. The scenario is this:

```
 safe void foo(int* p)
{   import customfreefunction.noannotations;
     p.free;
     p.free;
}
```

Now, this will not compile, because `free` is ` system`. But if 
`free` is in unannotated module, this will compile after the DIP 
implementation. Obviously not with the standard library `free` 
because it'll be annotated ` system` anyway, but with some custom 
`free` function this is an issue.

If the situation in using new features is indeed as dire as you 
fear, a better alternative would have been to just reject the 
DIP. Then there would be less ` safe` code, but at least you 
could trust ` safe` much more, due to the above phenomenon. I do 
think there would have been a way to have ` safe` by default even 
with the assumptions you made about it's abuse, but the DIP 
reviews are over so I don't think it's worth explaining anymore.

But at least you gave the reasonable rationale we wanted. Thank 
you.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/22/20 6:13 AM, Dukc wrote:
 Wrong :-(. The scenario is this:
 
 ```
  safe void foo(int* p)
 {   import customfreefunction.noannotations;
      p.free;
      p.free;
 }
 ```
 
 Now, this will not compile, because `free` is ` system`. But if `free` 
 is in unannotated module, this will compile after the DIP 
 implementation. Obviously not with the standard library `free` because 
 it'll be annotated ` system` anyway, but with some custom `free` 
 function this is an issue.

I want to interject slightly to say that this isn't the exact problem. 
The exact problem is that today `foo` is NOT marked  safe (or else it 
wouldn't compile). But after the DIP it will be IMPLICITLY marked safe, 
and so will `free`. Thereby silently migrating all system code that 
should actually be (and was INTENDED to be) system code straight to safe 
code without any warnings.

This in itself isn't a problem, code that compiled yesterday compiles 
today, and is just as bad.

The true problem is that after the DIP is implemented, new users will 
use foo within safe code they INTEND to be safe, and will be none the 
wiser that it's not.

The scenario I imagine is that unadorned code will exist for years as 
 safe (implicitly) and only after years of being depended on as  safe 
code, is discovered to be completely unsafe, and the only way to fix is 
to break existing usage. Or worse, it's not discovered and then D's 
already shaky reputation for  safe code is destroyed when a hacker 
exploits the memory corruption in fully-marked  safe code.

The actual bug may be extremely subtle. We have code in phobos that does 
stuff like:

static if(is(typeof(()  safe { obj.doSomething(); }))) obj.doSomething();
else doSomethingSafely(obj);

which will silently switch to using unsafe (but marked safe) mechanisms 
(for optimization).

-Steve

jmh530 <john.michael.hall gmail.com> writes:

On Friday, 22 May 2020 at 16:03:00 UTC, Steven Schveighoffer 
wrote:
 [snip]

Fortunately, the above point can be more easily fixed by making 
`free`  system, which will then require annotating every 
subsequent piece of code that touches it. It's annoying 
transition, but it's do-able. The problem is when you aren't sure 
what needs to be marked  system. For instance, if you have some 
function now that is  system by default and does pointer 
arithmetic, then the switch to  safe by default will result in 
that no longer compiling. That's good. The emphasis on prototypes 
is that the compiler will not be able to notify you that the 
function should not be compiled. If the `free` function comes 
from a prototype (and thus cannot be manually verified by the 
compiler), then that is much worse for safety than a `free` 
function with the full body available.

This comes back to a point I had made on one of the original DIP 
discussion threads that one issue is that  safe is a blacklist of 
things you can't do rather than a whitelist of allowed things. If 
 safe were designed as a whitelist, then only things that are 
compiled verified could be marked  safe. That means that a 
function without a body should not be inferred to be  safe.

That `static if` thing just looks like a nightmare, but I think 
the point still applies that you will get error messages if the 
body is available and you try to do something that isn't  safe.

Adam D. Ruppe <destructionator gmail.com> writes:

On Friday, 22 May 2020 at 16:39:42 UTC, jmh530 wrote:
 Fortunately, the above point can be more easily fixed by making 
 `free`  system

With the o/b system `free` might actually work out OK....

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.05.20 18:43, Adam D. Ruppe wrote:
 On Friday, 22 May 2020 at 16:39:42 UTC, jmh530 wrote:
 Fortunately, the above point can be more easily fixed by making `free` 
  system

 
 With the o/b system `free` might actually work out OK....

free(new int);

Walter Bright <newshound2 digitalmars.com> writes:

On 5/22/2020 10:25 AM, Timon Gehr wrote:
 With the o/b system `free` might actually work out OK....

 
 free(new int);

I did mention in the documentation that when using different memory allocators, 
mixing them up will not be detected. You'd have to declare the allocators using 
specific pointer types, rather than void*.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/22/20 12:39 PM, jmh530 wrote:
 On Friday, 22 May 2020 at 16:03:00 UTC, Steven Schveighoffer wrote:
 [snip]

 
 Fortunately, the above point can be more easily fixed by making `free` 
  system, which will then require annotating every subsequent piece of 
 code that touches it. It's annoying transition, but it's do-able.

You can't, you don't control that code, someone else does (this is 
important, you can declare extern(C) functions anywhere, even locally).

And from the experience of .save for forward ranges, people just aren't 
going to do this. They are going to do the easiest thing that works 
without complaint -- i.e. don't bother marking.

 That `static if` thing just looks like a nightmare, but I think the 
 point still applies that you will get error messages if the body is 
 available and you try to do something that isn't  safe.

It essentially is a "best effort" at calling the right thing. The code 
compiles either way, and be  safe either way, but will do the unsafe 
thing when it really should do the safe thing. It's ACTIVELY trying to 
ensure safety, but the compiler will thwart it.

-Steve

jmh530 <john.michael.hall gmail.com> writes:

On Friday, 22 May 2020 at 16:47:34 UTC, Steven Schveighoffer 
wrote:
 [snip]

 You can't, you don't control that code, someone else does (this 
 is important, you can declare extern(C) functions anywhere, 
 even locally).

You can make a separate module with one function that just calls 
the `free` function and has the  system attribute. Then, just 
change the import for the free function to that new module. 
Annoying, but potentially fix-able.

The bigger issue is that we know that we should check the `free` 
function, but what about all the other functions out there that 
might corrupt memory that we don't really know about. It's not 
scale-able.

 And from the experience of .save for forward ranges, people 
 just aren't going to do this. They are going to do the easiest 
 thing that works without complaint -- i.e. don't bother marking.

Probably true.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/22/20 1:52 PM, jmh530 wrote:
 On Friday, 22 May 2020 at 16:47:34 UTC, Steven Schveighoffer wrote:
 [snip]

 You can't, you don't control that code, someone else does (this is 
 important, you can declare extern(C) functions anywhere, even locally).

 
 You can make a separate module with one function that just calls the 
 `free` function and has the  system attribute. Then, just change the 
 import for the free function to that new module. Annoying, but 
 potentially fix-able.

You are using library fubar.

fubar defines a function foo:

void foo(int *p)
{
    import fubar.cfunctions : free;
    free(p);
}

After this DIP, it now becomes completely assumed  safe. The author of 
fubar intended it to be  system. But now all safe code can now call it, 
and there's nothing you can do except ask them to fix both modules to be 
 system. You can just stop using the library, sure. But you have no 
control over whether fubar's code is properly written. That is the point.

If the C prototype was instead considered  system, then foo stops 
compiling (for good reason), and you can either motivate the author to 
fix it, or use a different library. But at least you can't ACCIDENTALLY 
use it in  safe code.

Essentially all code written before this DIP is switched on is 
completely suspect, and needs manual verification -- if you care about 
safety at all.

-Steve

jmh530 <john.michael.hall gmail.com> writes:

On Friday, 22 May 2020 at 18:44:09 UTC, Steven Schveighoffer 
wrote:
 [snip]
 You are using library fubar.

 fubar defines a function foo:

I agree with most of what you said.

I was trying to make the point that this is a contrived example 
where we know for sure that one function should be  system and is 
actually  safe. For one function where you know it should be 
 system, then you can create a fix. However, it's unreasonable to 
require that for all of the code from outside modules without 
attributes (even if it could be possible to write something in D 
that makes it relatively less painful to do this).

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Fri, May 22, 2020 at 04:39:42PM +0000, jmh530 via Digitalmars-d-announce
wrote:
[...]
 This comes back to a point I had made on one of the original DIP
 discussion threads that one issue is that  safe is a blacklist of
 things you can't do rather than a whitelist of allowed things.

[...]

Yes, and that's why  safe has been on a shaky foundation since the
beginning.  In theory, a perfect, bug-free blacklist is equivalent to a
perfect, bug-free whitelist.  But given the rate at which D is adopting
new features, and the combinatorial explosion of feature combinations,
any of which might lead to violation of  safe, my confidence level on
the completeness of the blacklist is on the low side.  Had it been a
whitelist, it would have been much better, because when people ran into
a combination of features that ought to be allowed, but isn't, they
would file a bug, and then that case could be added to the whitelist
after careful vetting. You always err on the safe side. With a
blacklist, you're playing catch-a-mole with missed cases, overlooked
cases, and you never have complete certainty that you've weeded out all
dangerous constructs.


T

-- 
"Computer Science is no more about computers than astronomy is about
telescopes." -- E.W. Dijkstra

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Fri, May 22, 2020 at 12:03:00PM -0400, Steven Schveighoffer via
Digitalmars-d-announce wrote:
[...]
 Or worse, it's not discovered and then D's already shaky reputation
 for  safe code is destroyed when a hacker exploits the memory
 corruption in fully-marked  safe code.

[...]

TBH, this whole fiasco just confirms my initial skepticism about  safe.
The concept of mechanical verifiability is cool, I'm on board with that.
But the execution of it in  safe up to before this DIP already left a
lot to be desired (the beginning of which is the fact that  safe is
based on a blacklist rather than a whitelist, but let's not go there
now).  Now with this DIP, it's basically another nail in the  safe
coffin.

Essentially what it means is that  safe is a nice tool to have for
casual use, but don't depend on it for real-world, mission critical use,
because it doesn't have the strong guarantees you might imagine it has.
It promises a lot in bold letters, but the fine print contains a lot of
disclaimers.

Which means, in my case, it's not worth bothering with. The amount of
effort required to make code 100%  safe is just not worth the leaky
"guarantees" it brings.


T

-- 
First Rule of History: History doesn't repeat itself -- historians merely
repeat each other.

Paul Backus <snarwin gmail.com> writes:

On Friday, 22 May 2020 at 05:37:55 UTC, Walter Bright wrote:
 On 5/21/2020 8:36 PM, Paul Backus wrote:
 Something ought to be done to prevent this. It doesn't have to 
 be the exact
 proposal from the discussion thread, but doing nothing and 
 allowing widespread
 silent breakage cannot possibly be the best solution.

 I can see that happening. A simple example would be:

     extern (C) void free(void* p);
     ...
     free(p);
     free(p);

 The thing is, you are no worse off than now. If free() can be 
 misused by calling it from system code, it can be misused by 
 calling it from safe code.

 There's no way the compiler can detect this. If you annotate 
 it, you'll just have to annotate it correctly. Forcing an 
 annotation just means slapping  safe: at the beginning of the 
 file and moving on - it's not going to help.

Currently, if I want to use  safe correctly, all I need to do is 
annotate my code with  safe and fix any compile errors that arise 
as a result.

With -preview=safedefault enabled, if I want to use  safe 
correctly, I am now required to *manually* review every extern 
function declaration in every module of every dependency of my 
program to make sure that they are explicitly annotated. If I do 
not do this, I risk introducing silent safety violations to my 
program.

You are correct that in both cases, I can run into trouble if 
 safe is misused. Nonetheless, I submit to you that I am worse 
off in the second case than in the first.

The compiler cannot be expected to guard against deliberate 
misuse, but it *can* be expected to give us the necessary tools 
to use  safe correctly. The current implementation of DIP 1028 
does not do so.

bachmeier <no spam.net> writes:

On Friday, 22 May 2020 at 03:36:03 UTC, Paul Backus wrote:
 Someone has created bindings for a C library and published them 
 on
 code.dlang.org. Because they were written prior to DIP 1028, 
 the author assumed
 that  system was the default, and didn't bother to add explicit 
 annotations to
  system functions. Their code looks like this:

 --- clibrary.d

     void monkey_around(...); // assumed  system-by-default

 ---

 Months or years later, I decide to write a D program that makes 
 use of these
 bindings. By then,  safe-by-default has been fully implemented. 
 I add
 `clibrary` as a dependency to my Dub project and write the 
 following code:

 --- app.d

     import clibrary;

     void main() //  safe-by-default
     {
         /* ... code ... */

         monkey_around(...);

         /* ... more code ... */
     }

 ---

 My program compiles with no errors...and then corrupts memory 
 at run-time, even
 though every line of code I've written is  safe. Oops.

 This is the nightmare scenario that people are worried about: 
 safety violations
 being introduced *silently* into existing, correct D code.

Honest question: What is the use case for an 
absolutely-positively-has-to-be-safe program that calls C code? 
Why would anyone ever do that? C is not and will never be a safe 
language. "Someone looked at that blob of horrendous C code and 
thinks it's safe" does not inspire confidence. Why not rewrite 
the code in D (or Rust or Haskell or whatever) if safety is that 
critical?

Paul Backus <snarwin gmail.com> writes:

On Friday, 22 May 2020 at 13:58:14 UTC, bachmeier wrote:
 On Friday, 22 May 2020 at 03:36:03 UTC, Paul Backus wrote:
 This is the nightmare scenario that people are worried about: 
 safety violations
 being introduced *silently* into existing, correct D code.

 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C code? 
 Why would anyone ever do that? C is not and will never be a 
 safe language. "Someone looked at that blob of horrendous C 
 code and thinks it's safe" does not inspire confidence. Why not 
 rewrite the code in D (or Rust or Haskell or whatever) if 
 safety is that critical?

The problem isn't that safety is critical, it's that the D 
compiler is lying to me about the safety of my code.

If the compiler was honest and told me that my code was unsafe, 
I'd be able to make an informed decision to either (a) accept the 
lack of safety, or (b) do the additional work needed to make it 
safe. As-is, DIP 1028 takes that choice away from me, and I am 
forced to accept a lack of safety whether I want to or not. At 
that point, why not use C?

bachmeier <no spam.net> writes:

On Friday, 22 May 2020 at 14:13:20 UTC, Paul Backus wrote:
 On Friday, 22 May 2020 at 13:58:14 UTC, bachmeier wrote:
 On Friday, 22 May 2020 at 03:36:03 UTC, Paul Backus wrote:
 This is the nightmare scenario that people are worried about: 
 safety violations
 being introduced *silently* into existing, correct D code.

 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C 
 code? Why would anyone ever do that? C is not and will never 
 be a safe language. "Someone looked at that blob of horrendous 
 C code and thinks it's safe" does not inspire confidence. Why 
 not rewrite the code in D (or Rust or Haskell or whatever) if 
 safety is that critical?

 The problem isn't that safety is critical, it's that the D 
 compiler is lying to me about the safety of my code.

 If the compiler was honest and told me that my code was unsafe, 
 I'd be able to make an informed decision to either (a) accept 
 the lack of safety, or (b) do the additional work needed to 
 make it safe. As-is, DIP 1028 takes that choice away from me, 
 and I am forced to accept a lack of safety whether I want to or 
 not. At that point, why not use C?

If you're compiling a program that calls into C, you know that's 
what you're doing, so you know you've given up any guarantees of 
safety. Back in the early days of Rust, someone made a comment 
along the lines of Rust being a language to rewrite C code in, 
not to use to call C code. D has, to this point, been a language 
designed to interoperate easily with C code. I personally don't 
care about safety guarantees at all. It's not obvious to me that 
guaranteed safety at all costs is the way to go - especially 
considering the degree to which that breaks from current behavior.

Paul Backus <snarwin gmail.com> writes:

On Friday, 22 May 2020 at 14:43:27 UTC, bachmeier wrote:
 If you're compiling a program that calls into C, you know 
 that's what you're doing, so you know you've given up any 
 guarantees of safety.

The entire problem is that with DIP 1028, I *don't* know what I'm 
doing, because the compiler will silently allow me to write  safe 
D code that calls into unsafe C code.

Keep in mind that none of this requires *me* to write the call to 
C directly. It could be buried several layers deep in a 
dependency, and merely cause a D function I call to be 
incorrectly inferred as  safe.

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 14:13:20 UTC, Paul Backus wrote:
 On Friday, 22 May 2020 at 13:58:14 UTC, bachmeier wrote:
 On Friday, 22 May 2020 at 03:36:03 UTC, Paul Backus wrote:
 This is the nightmare scenario that people are worried about: 
 safety violations
 being introduced *silently* into existing, correct D code.

 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C 
 code? Why would anyone ever do that? C is not and will never 
 be a safe language. "Someone looked at that blob of horrendous 
 C code and thinks it's safe" does not inspire confidence. Why 
 not rewrite the code in D (or Rust or Haskell or whatever) if 
 safety is that critical?

 The problem isn't that safety is critical, it's that the D 
 compiler is lying to me about the safety of my code.

I understand your argument, especially since I wrote nearly 
identical words weeks ago.

For me, what it boils down to is this: currently the compiler 
isn't doing what it could because of the  system default. I work 
on a D codebase that can't be called from a  safe unittest and we 
could (probably are) be hiding bugs due to this.

Flipping the default will cause more incorrect code to fail to 
compile. Yes, there's a cost, which is carefully vetting 
extern(C) and extern(C++) declarations. The decision came down to 
finding this an acceptable trade-off.

ag0aep6g <anonymous example.com> writes:

On 22.05.20 19:12, Atila Neves wrote:
 Flipping the default will cause more incorrect code to fail to compile. 
 Yes, there's a cost, which is carefully vetting extern(C) and 
 extern(C++) declarations. The decision came down to finding this an 
 acceptable trade-off.

You can still carefully vet the extern declarations if DMD requires an 
explicit safety attribute on them. You can do it better even, because 
DMD will tell you where they are.

Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:

On Friday, 22 May 2020 at 17:12:47 UTC, Atila Neves wrote:
 [..]
 Yes, there's a cost, which is carefully vetting extern(C) and 
 extern(C++) declarations. The decision came down to finding 
 this an acceptable trade-off.

How would you feel about a DIP that the only thing it did was 
assume all non-extern (D) code was implciti  trusted? And how 
about if all such code suddenly became  safe without any vetting 
by developers and not even a compiler switch to revert to the old 
behavior?

As evidenced by the community outrage, no one but Walter and 
(maybe) you are convinced that safe-by-default on function bodies 
should imply  safe on non-extern(D) function declarations.

So how about a compromise? We accept a modified version of 
DIP1028 in which safe-by-default applies only to function 
definitions and extern(D) function declarations. And then there's 
a separate compiler switch that changes non-extern (D) function 
declarations to be  trusted? If you like this "feature" you're 
free to use it on your personal projects, but please don't force 
it on everyone who wants  safe to mean anything meaningful.

Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:

On Friday, 22 May 2020 at 17:12:47 UTC, Atila Neves wrote:
 [..]
 Yes, there's a cost, which is carefully vetting extern(C) and 
 extern(C++) declarations. The decision came down to finding 
 this an acceptable trade-off.

How would you feel about a DIP that the only thing it did was 
make all non-extern (D) code implicitly  trusted? And how about 
if all such code suddenly became  safe without any vetting by 
developers and not even a compiler switch to revert to the old 
behavior?
Until DIP1028, putting  trusted: at the start of a module has 
always been considered bad practice and rightfully forbidden in 
libraries with a higher quality bar, such as phobos. But at least 
doing so is honest: you as an author are admitting that you don't 
have time to take care of  safe-ty for now, but it is likely ok 
to assume that the rest of your project is safe (well at least 
the  safe functions) *modulo* the  trusted parts. That way, later 
you can come back and search for  trusted and address those 
issues one by one.

As evidenced by the community outrage, no one but Walter and 
(maybe) you are convinced that safe-by-default on function bodies 
should imply  safe on non-extern(D) function declarations.

---

So how about a compromise?
1. We accept a modified version of DIP1028 in which 
safe-by-default applies only to function definitions and 
extern(D) function declarations.
2. We add a separate compiler switch that changes non-extern (D) 
function declarations to be  trusted.
If you enable 1. and 2. you get the current version of DIP1028, 
but at least corrected, so that extern(C) code  trusted, and not 
 safe.
If you like this "feature" (2.) you're free to use it on your 
personal projects, but please don't force it on everyone who 
wants  safe to be something meaningful.

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.05.20 15:58, bachmeier wrote:
 ...
 
 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C code? Why 
 would anyone ever do that? C is not and will never be a safe language. 
 "Someone looked at that blob of horrendous C code and thinks it's safe" 
 does not inspire confidence. Why not rewrite the code in D (or Rust or 
 Haskell or whatever) if safety is that critical?

Honesty is what's critical. The annotations should mean what they are 
advertised to mean and making those meanings as simple as possible makes 
them easier to explain. As things stand,  safe can mean that someone 
accidentally or historically did not annotate an extern(C) prototype and 
an unsafe API a few calls up was ultimately exposed with the  safe 
attribute because the compiler never complained.

How would you feel if you never intended to expose a  safe interface, 
but someone imported your library after determining it contained no 
 trusted annotations (following the advice of articles on  safe), relied 
on the  safe annotation and then had weird sporadic memory corruption 
issues in production that took them months to ultimately trace back to 
your library? Would you feel responsible or would you rather put the 
blame on Walter?

bachmeier <no spam.net> writes:

On Friday, 22 May 2020 at 14:38:09 UTC, Timon Gehr wrote:
 On 22.05.20 15:58, bachmeier wrote:
 ...
 
 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C 
 code? Why would anyone ever do that? C is not and will never 
 be a safe language. "Someone looked at that blob of horrendous 
 C code and thinks it's safe" does not inspire confidence. Why 
 not rewrite the code in D (or Rust or Haskell or whatever) if 
 safety is that critical?

 Honesty is what's critical. The annotations should mean what 
 they are advertised to mean and making those meanings as simple 
 as possible makes them easier to explain. As things stand, 
  safe can mean that someone accidentally or historically did 
 not annotate an extern(C) prototype and an unsafe API a few 
 calls up was ultimately exposed with the  safe attribute 
 because the compiler never complained.

In my opinion, the only advantage of  safe is that the compiler 
has checked the code and determines that it only does things 
considered safe. I don't see that marking an extern(C) function 
 trusted buys you anything, at least not until you can provide a 
compiler guarantee for arbitrary C code.

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.05.20 16:49, bachmeier wrote:
 On Friday, 22 May 2020 at 14:38:09 UTC, Timon Gehr wrote:
 On 22.05.20 15:58, bachmeier wrote:
 ...

 Honest question: What is the use case for an 
 absolutely-positively-has-to-be-safe program that calls C code? Why 
 would anyone ever do that? C is not and will never be a safe 
 language. "Someone looked at that blob of horrendous C code and 
 thinks it's safe" does not inspire confidence. Why not rewrite the 
 code in D (or Rust or Haskell or whatever) if safety is that critical?

 Honesty is what's critical. The annotations should mean what they are 
 advertised to mean and making those meanings as simple as possible 
 makes them easier to explain. As things stand,  safe can mean that 
 someone accidentally or historically did not annotate an extern(C) 
 prototype and an unsafe API a few calls up was ultimately exposed with 
 the  safe attribute because the compiler never complained.

 
 In my opinion, the only advantage of  safe is that the compiler has 
 checked the code and determines that it only does things considered 
 safe.

Wrong, but let's roll with that.

 I don't see that marking an extern(C) function  trusted buys you 
 anything, at least not until you can provide a compiler guarantee for 
 arbitrary C code.

It buys you the ability to call that function from  safe code. Clearly 
you can't mark it  safe because the compiler has not checked it.

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Friday, May 22, 2020 9:29:03 AM MDT Timon Gehr via Digitalmars-d-announce 
wrote:
 On 22.05.20 16:49, bachmeier wrote:
 I don't see that marking an extern(C) function  trusted buys you
 anything, at least not until you can provide a compiler guarantee for
 arbitrary C code.

 It buys you the ability to call that function from  safe code. Clearly
 you can't mark it  safe because the compiler has not checked it.

Yes, and  trusted in general buys you the ability to segregate and find code
that is doing stuff that the compiler was unable to prove was  safe. Then
when there's a bug related to memory safety, you know where to look.

C functions cannot be  safe, because they haven't been vetted by the
compiler. So, either what they're doing is actually memory safe (and it's
fine to mark whem with  trusted), or what it's doing is memory safe if used
correctly, and the calling code needs to be marked  trusted when using it
correctly. Either way, the code with potential memory safety issues is
segregated, and so when inevitably you do run into a memory safety bug, you
know that it's either the  trusted code or the  system code that it calls
which needs to be fixed.

Of course, with this DIP,  safe no longer truly means that the code was
vetted by  safety by the compiler but rather that all of the code that the
compiler can see has been vetted for  safety, meaning that when there is an
 safety bug, you must look not only for  trusted code, but you must look for
non-extern(D) declarations which have been implictly treated as  trusted by
the compiler.  safe still has value (and may even provide more value in that
it will be used more often), but it provides much weaker guarantees in the
process.

- Jonathan M Davis

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Thu, May 21, 2020 at 06:22:19PM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.

[...]

Thank you.  This makes your position clear, even if I don't entirely
agree with it.  On that note, though: this really should have been part
of the DIP acceptance announcement, not added as an afterthought.
Otherwise people will perceive (correctly or not) the DIP as being
accepted without acknowledging any of the issues raised.  This summary
is exactly what would have defused a lot of the frustrations raised
here.  For the future may I recommend making it a point to include such
a summary with the announcement of each DIP accepted/rejected.


T

-- 
Nobody is perfect.  I am Nobody. -- pepoluan, GKC forum

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 03:57:22 UTC, H. S. Teoh wrote:
 On Thu, May 21, 2020 at 06:22:19PM -0700, Walter Bright via 
 Digitalmars-d-announce wrote:
 I have made these points before, but I'll summarize them here 
 for convenient referral.

 [...]

 Thank you.  This makes your position clear, even if I don't 
 entirely agree with it.  On that note, though: this really 
 should have been part of the DIP acceptance announcement, not 
 added as an afterthought. Otherwise people will perceive 
 (correctly or not) the DIP as being accepted without 
 acknowledging any of the issues raised.  This summary is 
 exactly what would have defused a lot of the frustrations 
 raised here.  For the future may I recommend making it a point 
 to include such a summary with the announcement of each DIP 
 accepted/rejected.


 T

This is a very good point, and I think that in the future Walter 
and I should attach an explanation of why a DIP got accepted or 
not for transparency and communication reasons.

Paolo Invernizzi <paolo.invernizzi gmail.com> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.

 [...]

Thank's for the reasoning, that should be added to the DIP 
acceptance since the beginning.

Stated that we need to live with that, I'm asking you a simple 
question:
What's the "official" definition of the  safe advantages that D 
offers as a language?

Sometime we need to explain that to customers asking for details 
about the code we wrote for them.

I would love to have a good insight detailing the promises that 
the language is committed to hold, maybe trying to give me a 
vision on the future ... since from now on I will stop using 
"mechanical verification" at all as selling argument.

Bruce Carneal <bcarneal gmail.com> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.

<big snip> of material indicating, among other things, that even 
really good programmers can screw up when it comes to language 
design now and then.  Ahem.
Continuing...

 How does this relate to safe by default?

 Consider the common (because that's how D started out) case of:

 ----- clibrary.d --------

     T massage_data(... many parameters ...);
     ... 200 more such declarations ...

 ----- app.d ----------

     import clibrary;

     void useClibrary( ... parameters ...) {
         massage_data(parameters);
     }

 ---------------------

 This code, today, does not use annotations and it works. It's 
 been working
 for a long time. Now, we implement the amendment, and it stops 
 compiling
 because useClibrary is  safe and massage_data is  system. The 
 user is faced
 with the following alternatives:

 1. Go through 200 functions in clibrary.d and determine which 
 are  safe
 and which are  system. This is what we want them to do. We try 
 to motivate
 this with compiler error messages. Unfortunately, this is both 
 tedious and
 thoroughly impractical, as our poor user Will Not Know which 
 are safe and
 which are system. We can correctly annotate core.stdc.stdio 
 because I know
 those functions intimately. This is not true for other system C 
 APIs, and
 even less true for some third party C library we're trying to 
 interface to.

Agree completely.   Annotating C code correctly can be very 
difficult.  This argues against defaulting such code to  safe.

 2. Annotate useClibrary() as  trusted or  system. While easier, 
 this causes
 all benefits to  safe by default to be lost.

No.  The benefit of  safe is actually having machine checkable 
 safety.

 3. Wrap the call to massage_data() with:

     ()  trusted { massage_data(parameters); } ();

 If there are a lot of calls to clibrary, this is going to look 
 pretty awful.
 Nobody likes writing or reading such ugly code. It's ok here 
 and there, but
 not as a general thing.

Yeah, this is ugly.

 4. Edit clibrary.d and make the first line:

      safe:

 I submit that, just like with Java, Option 4 is what people 
 will reach for,
 nearly every time. I've had some private conversations where 
 people admitted
 this was what they'd do. People who knew it was wrong to do 
 that.

The DIP gives us 4), the worst case, by default.  Anywhere and 
everywhere a C lib shows up it gives us 4), automatically.

 If it's  safe by default, and then someone chooses to annotate 
 it with  system
 here and there, I'd feel a lot more confident about the 
 accuracy of the code
 annotations than if it just had  safe: at the top. At least 
 they tried.

So  safe is to be based on a feeling?  At least they *tried*?  
This is a key point of disagreement.  Machine checked  safety is 
something.  Human hand waving  safety is not.

It is bizarre that no one has been able to get you to see the 
contradiction with your earlier writings on safety (big fan there 
btw). C's biggest blunder?  Yeah, well, all the bugs engendered 
by that blunder and many many more are now to be considered  safe 
by default. What the heck?

Finally, Atila, can you shed any light on this?  Where do you 
stand?  Why did it seem like a good idea to withhold your veto?

ag0aep6g <anonymous example.com> writes:

On 22.05.20 03:22, Walter Bright wrote:
 This is Obviously A Good Idea. Why would I oppose it?
 
 1. I've been hittin' the crack pipe again.
 2. I was secretly convinced, but wanted to save face.
 3. I make decisions based on consultation with my astrologer.
 4. I am evil.

The sarcasm is not appreciated.

[...]> Now, we implement the amendment, and it stops compiling
 because useClibrary is  safe and massage_data is  system. The user is faced
 with the following alternatives:
 
 1. Go through 200 functions in clibrary.d and determine which are  safe
 and which are  system. This is what we want them to do. We try to motivate
 this with compiler error messages. Unfortunately, this is both tedious and
 thoroughly impractical, as our poor user Will Not Know which are safe and
 which are system. We can correctly annotate core.stdc.stdio because I know
 those functions intimately. This is not true for other system C APIs, and
 even less true for some third party C library we're trying to interface to.
 
 2. Annotate useClibrary() as  trusted or  system. While easier, this causes
 all benefits to  safe by default to be lost.

In the past you have said repeatedly that the point of  safe is to be 
"100% mechanically checkable, not correct by convention" [1]. In other 
words, it's supposed to be watertight.

Now you're saying that  safe can have "benefits" without being 
watertight. You're even widening an existing hole in  safe to make those 
benefits more accessible.

If you have actually given up on a watertight  safe, that's 
disappointing for those of us who were with you on a watertight  safe.

 3. Wrap the call to massage_data() with:
 
      ()  trusted { massage_data(parameters); } ();
 
 If there are a lot of calls to clibrary, this is going to look pretty 
 awful.
 Nobody likes writing or reading such ugly code. It's ok here and there, but
 not as a general thing.

Let me get this straight. As a general rule we have: Unsafe stuff is 
"supposed to be inconvenient" [2]. But unsafe calls to C functions must 
be convenient, because they're so common. So we add an exception to the 
rule. And that exception is cool. But making an exception in 
 safe-by-default for `extern (C)` prototypes is not cool. Because 
exceptions are bad.

Makes sense ... not.

 4. Edit clibrary.d and make the first line:
 
       safe:
 
 I submit that, just like with Java, Option 4 is what people will reach for,
 nearly every time. I've had some private conversations where people 
 admitted
 this was what they'd do. People who knew it was wrong to do that.

Unlike the catch-and-ignore of your Java analogy, ` safe:` does no harm. 
Or rather, it does the same amount of harm that your  safe-by-default 
`extern (C)` declarations do. But with the amendment, the idiot 
programmer at least has to pull the trigger to shoot their foot off. 
You're letting the compiler do it for them.

That  safe even applies to `extern (C)` prototypes is its own issue.

 If it's  safe by default, and then someone chooses to annotate it with 
  system
 here and there, I'd feel a lot more confident about the accuracy of the 
 code
 annotations than if it just had  safe: at the top. At least they tried.

The morale of your Java analogy is that people will find and use the 
easy way out. No one is going to add  system to the declarations if the 
code already compiles with  safe-by-default.

 What is actually accomplished with this amendment if it was implemented?
 
 1. Adds a funky, special case rule. It's better to have simple, easily
 understood rules than ones with special cases offering little improvement.
 
 2. Existing, working code breaks.

You can't forgo breakage when making  safe the default. Any template 
that is inferred as  system will throw a wrench in your suddenly- safe 
`main`.

 3. The most likely code fixes are to just make it compile, absolutely
 nothing safety-wise is improved. The added annotations will be a fraud.

And your solution is to let the compiler add the frauds.

 D should not encourage "greenwashing" practices like the Java
 exception specification engendered. The compiler cannot vet the accuracy
 of bodyless C functions, and we'll just have to live with that. The 
 proposed
 amendment does not fix that.

The proposed amendment at least makes the greenwashing visible. With the 
DIP as it is, the greenwashing is still there. The compiler just does it 
for you. Imagine Java doing catch-and-ignore by itself, just so that the 
idiot programmer doesn't have to do it.

It would be better if the greenwashing wasn't even possible. I.e., don't 
allow  safe on `extern (C)` prototypes. If that's what you're going for, 
great. But you're not, are you?

By the way, I don't think you have acknowledged Steven's idea [3] at 
all, yet. Maybe you've missed it. It's completely different from the 
"Obviously A Good Idea" amendment you've discussed here.




[1] https://issues.dlang.org/show_bug.cgi?id=14125#c26
[2] https://forum.dlang.org/post/r5hnjo$1j40$1 digitalmars.com
[3] https://forum.dlang.org/post/r6kvm4$1vq5$1 digitalmars.com

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/22/20 4:24 AM, ag0aep6g wrote:
 By the way, I don't think you have acknowledged Steven's idea [3] at 
 all, yet.

Yes, he did. With more nitpicking as a rebuttal, but I'm not really 
interested in having more discussions about this DIP or that proposal.

https://forum.dlang.org/post/ra7958$2r8v$1 digitalmars.com

-Steve

aliak <something something.com> writes:

First, thank you for the explanation! I have a few observations 
though, mainly that the exception analogy feels like more an 
argument against the DIP than for it. And I also have an 
alternative proposal that might be considered?

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 This, of course, was incredibly tedious and annoying. So what 
 happened?

 1. Functions would just do a try-catch(Throwable) and ignore 
 the exceptions.
 Oops.

 2. The Exception Specifications became simply `throws 
 Throwable`. Oops.

This is what happens in swift these days:

func g() /*throws*/ {
   do {
     try f()
   } catch {}
}

But, it's very rare. And passing something like that in code 
review is very difficult and, almost always, the reviewer will 
ask for comments so other people will know why the laziness (if 
it even really was laziness, sometimes it's the right thing to 
do).

Of course the one big difference between exception specifications 
and  safe is the implied importance;  safe is compiler-enforced 
and has the user's trust, exception specifications are human 
guess work of what are the important exceptions and how varied 
they need to be - it's API design - very hard to get right.

The exception specification becoming "throws Exception", while 
highlights the failure of exception specifications, does so with 
a non-dangerous resolution. I.e. you still need to mark your 
function as throwing, and you still need to at least "try" to 
catch the exception. Even a forced "catch(Exception)" is better 
than nothing and shows where problems can be and encourages 
scrutiny.

 How does this relate to safe by default?

That people will use the most convenient solution to get their 
work to compile if the juice is not worth the squeeze?

 1. Go through...

 2. Annotate...safe by default to be lost.

 3. Wrap the call...

 4. Edit clibrary.d and make the first line:

      safe:

5. Edit clibrary.d and make the first line  trusted: ?

Which brings me to the proposal of making  safe illegal on 
function prototypes. Reason being that  safe stays compiler 
checked.  trusted maintains it's path and value in hunting down 
memory corruption. extern(Anything) functions cannot lie anymore 
about being compiler-checked.

 I submit that, just like with Java, Option 4 is what people 
 will reach for,
 nearly every time. I've had some private conversations where 
 people admitted
 this was what they'd do. People who knew it was wrong to do 
 that.

 If it's  safe by default, and then someone chooses to annotate 
 it with  system

Which is the rare case in this scenario right? And 1) there's no 
way to tell if someone made an effort in the case where none are 
marked  system. 2) this is also only for the case of C library 
bindings (reason being that most of my code, and also others' 
code I've seen, has extern(C) functions splattered around various 
D files and not in a dedicated clibrary.d file - is this just me?)

 here and there, I'd feel a lot more confident about the 
 accuracy of the code
 annotations than if it just had  safe: at the top. At least 
 they tried.

 What is actually accomplished with this amendment if it was 
 implemented?

 1. Adds a funky, special case rule. It's better to have simple, 
 easily
 understood rules than ones with special cases offering little 
 improvement.

 2. Existing, working code breaks.

 3. The most likely code fixes are to just make it compile, 
 absolutely
 nothing safety-wise is improved. The added annotations will be 
 a fraud.

 D should not encourage "greenwashing" practices like the Java
 exception specification engendered. The compiler cannot vet the 
 accuracy
 of bodyless C functions, and we'll just have to live with that. 
 The proposed
 amendment does not fix that.

 And so, I did not incorporate the proposed amendment to the 
 Safe by Default
 DIP.

I think there's a failure to understand why explicit greenwashing 
via a deliberate (possibly code reviewed) addition of  safe: at 
the top is worse than a global implicit greenwashing by the 
compiler, ignoring the rare "feel-good-but-can't-be-sure" 
scenario above where some functions are marked as  system, and 
especially given the premise that programmers are lazy.

Also ... error on  safe prototypes?

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.

Thanks Walter.  I really appreciate you taking the time to do 
this, as it's obviously no fun to be getting a big tide of 
negativity in this way.

 This is Obviously A Good Idea. Why would I oppose it?

 1. I've been hittin' the crack pipe again.
 2. I was secretly convinced, but wanted to save face.
 3. I make decisions based on consultation with my astrologer.
 4. I am evil.

... wait, we don't ALL make decision like that? :-)

In seriousness: I really, really wish that everyone throwing 
around personal attacks would take a moment to stop and think 
about the consequences of pursuing this kind of discourse: it 
distracts from the technical, practical and organizational issues 
at hand, it creates a worse mood than exists already, and 
generally makes it harder to have a constructive conversation 
around the core disagreements here.

It is perfectly possible to critique the decision taken, or the 
way in which it was taken, or to voice personal frustrations, 
without casting aspersions on the decision-maker.

With the rationale laid out clearly as it is here, I do have some 
responses in mind.  But before sharing them, I'd like to know 
whether that would be useful right now: I've no wish to just 
press for a re-hashing of conversations that have already been 
had.

rikki cattermole <rikki cattermole.co.nz> writes:

On 23/05/2020 12:19 AM, Joseph Rushton Wakeling wrote:
 With the rationale laid out clearly as it is here, I do have some 
 responses in mind.  But before sharing them, I'd like to know whether 
 that would be useful right now: I've no wish to just press for a 
 re-hashing of conversations that have already been had.

No.

This wasn't the first and certainly won't be the last time we as a 
community have been very unhappy with how Walter conducts himself with 
his DIP's.

Although it seems an improvement has been made to how he needs to 
respond to the DIP assessment. It should also include a statement from 
Atila as well given his position.

matheus <matheus gmail.com> writes:

On Friday, 22 May 2020 at 12:28:56 UTC, rikki cattermole wrote:
 Although it seems an improvement has been made to how he needs 
 to respond to the DIP assessment. It should also include a 
 statement from Atila as well given his position.

One thing that need to be clear and for I read it was not 
remotely answered from Walter is why this DIP process and 
discussion exists if in the end like it or not it will be 
incorporated.

As an end user, I'd like to know if this language will be guided 
by community or one person, because it seems the "democracy" is 
very shallow right now.

And again why waste time with this process plus 2 rounds of 
discussion?

I mean just do it and tell in this announcement section about the 
feature.

Matheus.

Johannes Loher <johannes.loher fg4f.de> writes:

On Friday, 22 May 2020 at 12:47:04 UTC, matheus wrote:
 On Friday, 22 May 2020 at 12:28:56 UTC, rikki cattermole wrote:
 Although it seems an improvement has been made to how he needs 
 to respond to the DIP assessment. It should also include a 
 statement from Atila as well given his position.

 One thing that need to be clear and for I read it was not 
 remotely answered from Walter is why this DIP process and 
 discussion exists if in the end like it or not it will be 
 incorporated.

 As an end user, I'd like to know if this language will be 
 guided by community or one person, because it seems the 
 "democracy" is very shallow right now.

 And again why waste time with this process plus 2 rounds of 
 discussion?

 I mean just do it and tell in this announcement section about 
 the feature.

 Matheus.

There is no democracy. The decisions are simply made by Walter 
and Atila.

 From my understanding, the purpose of the DIP process is to 
improve DIPs and find any potential flaws before a final decision 
ist made by the two language maintainers. However, how much of 
the review the maintainers take into account is their own 
decision.

In theory, this results in much higher quality DIPs and the 
chance of a DIP that actually makes it to the formal assessment 
stage to be accepted is a lot higher. It also seems to work quite 
well when the DIP author is not one of the language maintainers. 
 From my experience, the quality of those DIPs really has gone up 
by a lot. There is also a lively discussion about each DIP and 
the authors are actively participating. It is in their own 
interest to do so because getting feedback and addressing it 
increases the quality of the proposal which in turn increases the 
chance of the DIP being accepted.

If the author is one the language maintainers, this doesn't seem 
to work that well though. There is simply less motivation for 
them to do all of that because in the end, they can make the 
decision by themselves anyways. I am not saying that the language 
maintainers do this on purpose, it is simply a psychological 
effect. And the result can be seen very clearly when comparing 
review threads for DIPs from the language authors with review 
threads for DIPs from others.

In defense of the language authors, it has to be stated that the 
situation has improved quite a bit over time. Examples of this 
are how DIP1017 was handled or the fact that Walter now gave a 
reasoning for his decision regarding DIP1028 (even if only after 
being asked to do so). But unfortunately, sometimes it still 
really feels like DIPs from the language maintainers and DIPs 
from others are handled quite differently by the maintainers.

matheus <matheus gmail.com> writes:

On Friday, 22 May 2020 at 13:16:50 UTC, Johannes Loher wrote:
 ... But unfortunately, sometimes it still really feels like 
 DIPs from the language maintainers and DIPs from others are 
 handled quite differently by the maintainers.

That's was my point about "democracy", maybe not a right word.

In fact I really think that in the end someone will need to 
decide (for good or bad), but it's not the first time I read 
about this nuances between DIPs created by maintainers and others 
like you said.

I think for such small community, there should be a vote system 
for those who are actively contributing (I am NOT in this niche 
by the way) to guide the language.

It would be better than a rounds of discussion without any 
meaning.

Matheus.

Mike Parker <aldacron gmail.com> writes:

On Friday, 22 May 2020 at 12:47:04 UTC, matheus wrote:
 As an end user, I'd like to know if this language will be 
 guided by community or one person, because it seems the 
 "democracy" is very shallow right now.

 And again why waste time with this process plus 2 rounds of 
 discussion?

 I mean just do it and tell in this announcement section about 
 the feature.

The DIP review process is not intended for community approval or 
rejection of DIPs. It's not a democratic voting process. It's 
intended to elicit community feedback to enhance the DIP under 
review (the Feedback Threead) and to allow the airing of opinions 
(the Discussion Thread). All DIP authors have the freedom to 
incorporate suggestions into their DIP or not, and Walter and 
Atila make the decision to accept or reject. If you look at the 
history of Walter's DIPs, they *do* take the opinions into 
consideration even when he is the author. Several of his previous 
DIPs have been withdrawn or rejected.

If a popular DIP is rejected, it means neither of them were 
convinced by opinion to accept it. And, as in the case for this 
DIP, if an unpopular DIP is accepted, it means they were not 
persuaded by the arguments against it.

 From my perspective, the process is working as intended, despite 
the comments to the contrary in this thread. You either convince 
a DIP author to modify his DIP, or you don't. You either persuade 
Walter and Atila to accept or reject it, or you don't.

bachmeier <no spam.net> writes:

On Friday, 22 May 2020 at 13:57:27 UTC, Mike Parker wrote:
 On Friday, 22 May 2020 at 12:47:04 UTC, matheus wrote:
 As an end user, I'd like to know if this language will be 
 guided by community or one person, because it seems the 
 "democracy" is very shallow right now.

 And again why waste time with this process plus 2 rounds of 
 discussion?

 I mean just do it and tell in this announcement section about 
 the feature.

 The DIP review process is not intended for community approval 
 or rejection of DIPs. It's not a democratic voting process. 
 It's intended to elicit community feedback to enhance the DIP 
 under review (the Feedback Threead) and to allow the airing of 
 opinions (the Discussion Thread). All DIP authors have the 
 freedom to incorporate suggestions into their DIP or not, and 
 Walter and Atila make the decision to accept or reject. If you 
 look at the history of Walter's DIPs, they *do* take the 
 opinions into consideration even when he is the author. Several 
 of his previous DIPs have been withdrawn or rejected.

 If a popular DIP is rejected, it means neither of them were 
 convinced by opinion to accept it. And, as in the case for this 
 DIP, if an unpopular DIP is accepted, it means they were not 
 persuaded by the arguments against it.

 From my perspective, the process is working as intended, 
 despite the comments to the contrary in this thread. You either 
 convince a DIP author to modify his DIP, or you don't. You 
 either persuade Walter and Atila to accept or reject it, or you 
 don't.

I think the source of the problem is that Walter's DIPs require 
the community to prove that Walter's proposal is so bad that he 
needs to reject it. Anyone else's proposal has to prove that it's 
worthy of being added to the language. There's a big perceived 
gap between those two. As I've said many times, it's odd for 
someone to judge his own DIPs, and as someone that is an academic 
administrator and runs a high-profile journal, I can say this 
type of practice is not the norm in those areas because it 
doesn't lead to good decisions.

Bruce Carneal <bcarneal gmail.com> writes:

On Friday, 22 May 2020 at 14:37:04 UTC, bachmeier wrote:
 I think the source of the problem is that Walter's DIPs require 
 the community to prove that Walter's proposal is so bad that he 
 needs to reject it. Anyone else's proposal has to prove that 
 it's worthy of being added to the language. There's a big 
 perceived gap between those two. As I've said many times, it's 
 odd for someone to judge his own DIPs, ...

When either Walter or Atila author a DIP, their spot in the 
"convince both or face rejection" duopoly could be taken by 
another person.

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 14:37:04 UTC, bachmeier wrote:
 On Friday, 22 May 2020 at 13:57:27 UTC, Mike Parker wrote:
 [...]

 I think the source of the problem is that Walter's DIPs require 
 the community to prove that Walter's proposal is so bad that he 
 needs to reject it. Anyone else's proposal has to prove that 
 it's worthy of being added to the language. There's a big 
 perceived gap between those two. As I've said many times, it's 
 odd for someone to judge his own DIPs, and as someone that is 
 an academic administrator and runs a high-profile journal, I 
 can say this type of practice is not the norm in those areas 
 because it doesn't lead to good decisions.

I could have rejected it, and nearly did.

Bruce Carneal <bcarneal gmail.com> writes:

On Friday, 22 May 2020 at 13:57:27 UTC, Mike Parker wrote:
 On Friday, 22 May 2020 at 12:47:04 UTC, matheus wrote:
 As an end user, I'd like to know if this language will be 
 guided by community or one person, because it seems the 
 "democracy" is very shallow right now.

 And again why waste time with this process plus 2 rounds of 
 discussion?

 I mean just do it and tell in this announcement section about 
 the feature.

 The DIP review process is not intended for community approval 
 or rejection of DIPs. It's not a democratic voting process. 
 It's intended to elicit community feedback to enhance the DIP 
 under review (the Feedback Threead) and to allow the airing of 
 opinions (the Discussion Thread). All DIP authors have the 
 freedom to incorporate suggestions into their DIP or not, and 
 Walter and Atila make the decision to accept or reject. If you 
 look at the history of Walter's DIPs, they *do* take the 
 opinions into consideration even when he is the author. Several 
 of his previous DIPs have been withdrawn or rejected.

No dispute here.

 If a popular DIP is rejected, it means neither of them were 
 convinced by opinion to accept it. And, as in the case for this 
 DIP, if an unpopular DIP is accepted, it means they were not 
 persuaded by the arguments against it.

Not exactly.  IIUC logical "and" is required for approval.  One 
of the two might have been persuaded to accept.  "not and", a 
single disapproval, suffices for rejection.  I believe this is a 
useful weighting.

 From my perspective, the process is working as intended, 
 despite the comments to the contrary in this thread. You either 
 convince a DIP author to modify his DIP, or you don't. You 
 either persuade Walter and Atila to accept or reject it, or you 
 don't.

Again, IIUC, you either persuade Walter *or* Atila to reject or 
you don't.  Apropos of which, what does Atila think of this DIP?

Adam D. Ruppe <destructionator gmail.com> writes:

On Friday, 22 May 2020 at 13:57:27 UTC, Mike Parker wrote:
 The DIP review process is not intended for community approval 
 or rejection of DIPs. It's not a democratic voting process.

The community needs to unite and fix this. It is an ineffectual 
process that leads to worse results for the language and further 
poisons the relationship between us.

A small voting review committee solves these problems.

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 12:28:56 UTC, rikki cattermole wrote:
 On 23/05/2020 12:19 AM, Joseph Rushton Wakeling wrote:
 With the rationale laid out clearly as it is here, I do have 
 some responses in mind.  But before sharing them, I'd like to 
 know whether that would be useful right now: I've no wish to 
 just press for a re-hashing of conversations that have already 
 been had.

 No.

 This wasn't the first and certainly won't be the last time we 
 as a community have been very unhappy with how Walter conducts 
 himself with his DIP's.

 Although it seems an improvement has been made to how he needs 
 to respond to the DIP assessment. It should also include a 
 statement from Atila as well given his position.

I'm going through posts in order, so apologies if I'm "ignoring" 
something that shows up later in the discussion.

Personally and initially, I would have preferred it if 
non-extern(D) declarations were implicitly  system. I understood 
Walter's argument about special cases and how they're bad, but 
the thought of them being  safe made me feel, for lack of a 
better word, "icky".

Then I talked to Walter and he pointed out that if those 
declarations were  system, users would be prevented from calling 
them from now  safe code. A regular user would probably just slap 
` safe:` at the top of the bindings module anyway. Then I 
realised that I did exactly that with my libclang bindings:

https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/clang/c/index.d#L254

"Worse", I made all functions `pure` and all parameters `in` as 
well for good measure. Why? I wanted to call them from my  safe 
pure code with `const` arguments. My reasoning at the time was "I 
trust libclang".

And so I was convinced that everything being  safe is actually 
ok, especially because in real life, most C/C++ APIs aren't going 
to secretly corrupt your code.

Then there's the fact that, today, there's no safety anyway 
calling anything because everything is  system by default. And D 
source code won't be able to lie.

Does that clear it up?

Adam D. Ruppe <destructionator gmail.com> writes:

On Friday, 22 May 2020 at 17:07:37 UTC, Atila Neves wrote:
 And so I was convinced that everything being  safe is actually 
 ok, especially because in real life, most C/C++ APIs aren't 
 going to secretly corrupt your code.

I wrote up a thing about this earlier then deleted it because I 
didn't want to yell into the void, but now I kinda wish I kept it.

Short version of my thoughts is yes, most C functions are 
implicitly trusted, but this depends heavily on using correct 
arguments and that's what  trusted wrappers are supposed to 
ensure.

Consider:

char[16] buffer;
snprintf(&buffer[0], buffer.length, "%s", data);

btw fix this https://issues.dlang.org/show_bug.cgi?id=20853


But it is fair to say that usage is safe/trusted. However, you 
can just as well write:

snprintf(&buffer[12], buffer.length......

or

snprintf(&buffer[0], 156, ....); // oops my finger was misaligned 
on the keyboard and hit 56 without realizing it. or the size 
changed and this didn't etc



This is what a  trusted wrapper is supposed to be about - 
providing an interface that cannot be misused.

int snprintf(char[] buf, const char* fmt, ...) {
    return c.snprintf(buf.ptr, buf.length, fmt, ...);
}

Well, lol, that's not exactly *correct* thanks to the possibility 
of format string mismatch, but it bounds checks both sides of the 
slice and becomes more appropriately trusted due to the 
parameters being checked too.

We now actually have a trusted implementation with a safe 
interface - which is what  trusted is SUPPOSED to mean.

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Fri, May 22, 2020 at 05:26:03PM +0000, Adam D. Ruppe via
Digitalmars-d-announce wrote:
[...]
 int snprintf(char[] buf, const char* fmt, ...) {
    return c.snprintf(buf.ptr, buf.length, fmt, ...);
 }
 
 Well, lol, that's not exactly *correct* thanks to the possibility of
 format string mismatch, but it bounds checks both sides of the slice
 and becomes more appropriately trusted due to the parameters being
 checked too.
 
 We now actually have a trusted implementation with a safe interface -
 which is what  trusted is SUPPOSED to mean.

Ironically, it was Walter himself who was a staunch proponent of this.
And now we're basically saying, every C API is  safe by default unless
you take the pains to realize that, oh noes it takes a pointer and a
size and therefore passing the wrong size will cause buffer overflow
yikes we better mark it  system.  Seems totally backwards to me.

But anyway, thinking about this a little more, it occurs to me that what
we're trying to do here is really not 100% watertight  safe-ty, rather,
what we're trying to do here is to maximize the surface area of compiler
mechnical verification.  So  safe doesn't *really* mean "no memory
corruption, guaranteed", it means "compiler will do its best to verify
what it can, but what it can't, we just have to take on faith."  Or, to
put it more nicely, "this code is  safe modulo the safety of calls to
unverifiable C APIs".

Which leads to the idea of having some way of marking a function as
 safe up to the safety of {a delegate parameter, a C interface, etc.}. A
kind of conditional  safe-ty.  It has been a recurring need that keeps
coming up, e.g., in relation to the recent DIP on lazy parameters. For
example:

	// N.B.: hypothetical syntax
	int myFunc(int delegate(SomeObj obj) cb)  conditionalSafe {
		SomeObj obj = ...;
		// &obj++; // error: pointer arithmetic not allowed in  safe
		// systemFunc(); // error: cannot call systemFunc from  safe

		return cb(obj); // OK: we are  safe modulo safety of cb
	}
	void systemFunc()  system {...}

Then myFunc would become callable from  safe code, provided the
passed-in argument is also  safe.

The crucial point here is that while compiling myFunc, the compiler
doesn't (need to) know the  safe-ty of `cb`, it can just treat it as an
opaque object that it assumes the safety of, while it verifies the rest
of the function body.

This is parallel to C functions being of unverifiable safety, so if
extern(C) functions were somehow marked and treated as opaque objects of
unknown safety, then the compiler can still verify the rest of the code
and produce a certificate of safety (modulo the C APIs used).

If we had a way of expressing conditional safety, it could be a way to
salvage  safe from this current situation.


T

-- 
Doubtless it is a good thing to have an open mind, but a truly open mind should
be open at both ends, like the food-pipe, with the capacity for excretion as
well as absorption. -- Northrop Frye

Q. Schroll <qs.il.paperinik gmail.com> writes:

On Friday, 22 May 2020 at 18:14:12 UTC, H. S. Teoh wrote:
 Then myFunc would become callable from  safe code, provided the 
 passed-in argument is also  safe.

 The crucial point here is that while compiling myFunc, the 
 compiler doesn't (need to) know the  safe-ty of `cb`, it can 
 just treat it as an opaque object that it assumes the safety 
 of, while it verifies the rest of the function body.

 This is parallel to C functions being of unverifiable safety, 
 so if extern(C) functions were somehow marked and treated as 
 opaque objects of unknown safety, then the compiler can still 
 verify the rest of the code and produce a certificate of safety 
 (modulo the C APIs used).

 If we had a way of expressing conditional safety, it could be a 
 way to salvage  safe from this current situation.

I had this thought a million times, tried to explain it at two 
DConfs but no-one (as it appeared to me) got the point.

The main difference is, the function calling `myFunc` "knows" 
whether the delegate/function pointer it used as an argument is 
 safe or not. It's perfectly decidable.

Personally, I'd even go that far and say, that's how all the 
function attributes (also pure, nothrow,  nogc) are intended to 
work.

rikki cattermole <rikki cattermole.co.nz> writes:

On 23/05/2020 5:07 AM, Atila Neves wrote:
 On Friday, 22 May 2020 at 12:28:56 UTC, rikki cattermole wrote:
 On 23/05/2020 12:19 AM, Joseph Rushton Wakeling wrote:
 With the rationale laid out clearly as it is here, I do have some 
 responses in mind.  But before sharing them, I'd like to know whether 
 that would be useful right now: I've no wish to just press for a 
 re-hashing of conversations that have already been had.

 No.

 This wasn't the first and certainly won't be the last time we as a 
 community have been very unhappy with how Walter conducts himself with 
 his DIP's.

 Although it seems an improvement has been made to how he needs to 
 respond to the DIP assessment. It should also include a statement from 
 Atila as well given his position.

 
 I'm going through posts in order, so apologies if I'm "ignoring" 
 something that shows up later in the discussion.
 
 Personally and initially, I would have preferred it if non-extern(D) 
 declarations were implicitly  system. I understood Walter's argument 
 about special cases and how they're bad, but the thought of them being 
  safe made me feel, for lack of a better word, "icky".

This is one of the issues I had a problem with.

It is not about the linkage.

The problem is solely does the compiler have the source to the function 
body to verify it?

 Then I talked to Walter and he pointed out that if those declarations 
 were  system, users would be prevented from calling them from now  safe 
 code. A regular user would probably just slap ` safe:` at the top of the 
 bindings module anyway. Then I realised that I did exactly that with my 
 libclang bindings:
 
 https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/
lang/c/index.d#L254 

That is a failure of the language that should be resolved.

One of the arguments that has been brought up (although I don't remember 
if it made its way to the N.G.) is that if you don't have the body, can 
a function /even/ be  safe?

 "Worse", I made all functions `pure` and all parameters `in` as well for 
 good measure. Why? I wanted to call them from my  safe pure code with 
 `const` arguments. My reasoning at the time was "I trust libclang".

You might, but that doesn't give the compiler the right to do so by 
default. This a decision for a skilled programmer to make.

 And so I was convinced that everything being  safe is actually ok, 
 especially because in real life, most C/C++ APIs aren't going to 
 secretly corrupt your code.

What you did was give up some security in favor of freedom and now you 
can't get certified to work on top secret projects (analogy).

 Then there's the fact that, today, there's no safety anyway calling 
 anything because everything is  system by default. And D source code 
 won't be able to lie.
 
 Does that clear it up?

No.

We simply do not agree, nor do I expect for us to come to terms on it 
anytime soon.

To me at least, this butchers  safe/trusted/system into a system that is 
near useless for guarantees for an entire program.

Sure its convenient for a single function, but absolutely useless in 
trying to solve the actual problem it was trying to solve in the first 
place.

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 17:33:11 UTC, rikki cattermole wrote:
 On 23/05/2020 5:07 AM, Atila Neves wrote:
 [...]

 It is not about the linkage.

 The problem is solely does the compiler have the source to the 
 function body to verify it?

That's what I meant, sorry for not making it clearer.


 [...]

 That is a failure of the language that should be resolved.

And how do you suggest we fix it?

 One of the arguments that has been brought up (although I don't 
 remember if it made its way to the N.G.) is that if you don't 
 have the body, can a function /even/ be  safe?

Yes.

 [...]

 You might, but that doesn't give the compiler the right to do 
 so by default. This a decision for a skilled programmer to make.

They still can (and should). The DIP is about changing the 
default.

 [...]

 No.

 We simply do not agree, nor do I expect for us to come to terms 
 on it anytime soon.

I meant "did I explain myself well enough that now you understand 
where I'm coming from, even though you might not ultimately 
agree?".

rikki cattermole <rikki cattermole.co.nz> writes:

On 23/05/2020 5:40 AM, Atila Neves wrote:
 On Friday, 22 May 2020 at 17:33:11 UTC, rikki cattermole wrote:
 On 23/05/2020 5:07 AM, Atila Neves wrote:
 [...]

 It is not about the linkage.

 The problem is solely does the compiler have the source to the 
 function body to verify it?

 
 That's what I meant, sorry for not making it clearer.

It kept being swapped about in the discussion thread, so I have been a 
little on edge over people using non-extern(D). Because linkage doesn't 
mean anything to anything but to cpu and linker in this discussion.

 [...]

 That is a failure of the language that should be resolved.

 
 And how do you suggest we fix it?

I don't know.

It is a problem that needs to be explored in more detail.

Adam had one idea that I convinced him to write up, but that didn't get 
very far when commented relating to propagation of attributes.

Major changes like this, should have been discussed and RFC'd instead of 
going to straight to a DIP. That is effectively what happened.

 [...]

 No.

 We simply do not agree, nor do I expect for us to come to terms on it 
 anytime soon.

 
 I meant "did I explain myself well enough that now you understand where 
 I'm coming from, even though you might not ultimately agree?".

Yeah.

-------------

I didn't look at the binding, since you described it well previously, 
but of note is that you used  safe. What you should have used is 
 trusted instead.

If the programmer wants to slap  trusted: at the top of the file, I have 
no problems with that. You checked it (i.e. its LLVM, so of course its 
fine!).

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via Digitalmars-d-
announce wrote:
 It kept being swapped about in the discussion thread, so I have been a
 little on edge over people using non-extern(D). Because linkage doesn't
 mean anything to anything but to cpu and linker in this discussion.

Except that the linker matters a great deal in this discussion with regards
to extern(D) functions, because  safe and  trusted are part of the name
mangling for extern(D) functions. That means that if an extern(D) function
declaration's attributes do not match its definition, then you'll get a
linker error. So, treating non-extern(D) function declarations as  safe by
default isn't necessarily a problem (though it would certainly work to just
treat all function declarations as  system by default rather than treating
extern(D) function declarations differently). The cases where non-extern(D)
function declarations weren't actually  safe would be caught during the
linking process. Sure, it would be nice if it were caught sooner, but you
don't end up with them being invisibly treated  safe when they're not like
we're going to get with DIP 1028.

However, of course, with non-extern(D) declarations, even if the function
definition is actually written in D, the fact that the function body was
checked is not transmitted via the linking process, because it doesn't end
up in the name mangling. So, the linkage used has a huge impact on whether
you can rely on the  safe attribute on the function declaration actually
meaning anything about whether the body was verified by the compiler.

- Jonathan M Davis

Bruce Carneal <bcarneal gmail.com> writes:

On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis wrote:
 On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via 
 Digitalmars-d- announce wrote:
 [...]

 Except that the linker matters a great deal in this discussion 
 with regards to extern(D) functions, because  safe and  trusted 
 are part of the name mangling for extern(D) functions. That 
 means that if an extern(D) function declaration's attributes do 
 not match its definition, then you'll get a linker error. So, 
 treating non-extern(D) function declarations as  safe by 
 default isn't necessarily a problem (though it would certainly 
 work to just treat all function declarations as  system by 
 default rather than treating extern(D) function declarations 
 differently). The cases where non-extern(D) function 
 declarations weren't actually  safe would be caught during the 
 linking process. Sure, it would be nice if it were caught 
 sooner, but you don't end up with them being invisibly treated 
  safe when they're not like we're going to get with DIP 1028.

 [...]

I remember reading a suggestion that additional linker symbols be 
emitted to carry the attribute and possibly type information 
while leaving the ABI untouched.  Was this found to be 
impractical?

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Wednesday, May 27, 2020 8:13:52 PM MDT Bruce Carneal via Digitalmars-d-
announce wrote:
 On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis wrote:
 On Friday, May 22, 2020 12:09:16 PM MDT rikki cattermole via

 Digitalmars-d- announce wrote:
 [...]

 Except that the linker matters a great deal in this discussion
 with regards to extern(D) functions, because  safe and  trusted
 are part of the name mangling for extern(D) functions. That
 means that if an extern(D) function declaration's attributes do
 not match its definition, then you'll get a linker error. So,
 treating non-extern(D) function declarations as  safe by
 default isn't necessarily a problem (though it would certainly
 work to just treat all function declarations as  system by
 default rather than treating extern(D) function declarations
 differently). The cases where non-extern(D) function
 declarations weren't actually  safe would be caught during the
 linking process. Sure, it would be nice if it were caught
 sooner, but you don't end up with them being invisibly treated
  safe when they're not like we're going to get with DIP 1028.

 [...]

 I remember reading a suggestion that additional linker symbols be
 emitted to carry the attribute and possibly type information
 while leaving the ABI untouched.  Was this found to be
 impractical?

Steven suggested something along those lines. I don't know how practical it
would or wouldn't be, but I don't think that Walter even responded to the
idea.

Regardless, even if it worked perfectly, all such a solution would deal with
would be the cases where you have a non-extern(D) function which is actually
written in D and had its function body marked as  safe (and thus was
verified to be  safe). Such functions definitely exist, but the main problem
is really non-extern(D) functions which are actually written in C or C++ or
whatnot. They won't have been checked for  safety.

Right now, such declarations are treated by the compiler as  system by
default, because it cannot check them for  safety, whereas with DIP 1028, it
will then treat them as  safe in spite of the fact that it can't check them
for  safety simply because Walter thinks that it would be too exceptional to
treat them differently from the general rule of  safe by default and that if
they're left as  system, too many people will just slap  trusted on their
code as a whole to get the compiler to shut up.

But regardless of whether DIP 1028 is the correct decision, the problem
remains that your typical extern(C) function cannot be checked for  safety
by the compiler, because it was compiled by a C compiler. The question then
is just how the D compiler should treat them, and that's the main point of
contention. There may be solutions like Steven suggested which would deal
with edge cases where the implementation is in D but the linkage isn't, but
ultimately, they're just edge cases.

- Jonathan M Davis

Bruce Carneal <bcarneal gmail.com> writes:

On Thursday, 28 May 2020 at 02:39:40 UTC, Jonathan M Davis wrote:
 On Wednesday, May 27, 2020 8:13:52 PM MDT Bruce Carneal via 
 Digitalmars-d- announce wrote:
 On Thursday, 28 May 2020 at 01:14:43 UTC, Jonathan M Davis 
 wrote:
 [...]

 I remember reading a suggestion that additional linker symbols 
 be emitted to carry the attribute and possibly type 
 information while leaving the ABI untouched.  Was this found 
 to be impractical?

 Steven suggested something along those lines. I don't know how 
 practical it would or wouldn't be, but I don't think that 
 Walter even responded to the idea.

Changing the DIP process to be 2 of 3 LMs required for acceptance 
could get us past all the "this is useless because Walter xyz..." 
road blocks.  I've a proposal in to Mike for that now.  We'll see 
how it goes.

[snip]

 But regardless of whether DIP 1028 is the correct decision, the 
 problem remains that your typical extern(C) function cannot be 
 checked for  safety by the compiler, because it was compiled by 
 a C compiler. The question then is just how the D compiler 
 should treat them, and that's the main point of contention. 
 There may be solutions like Steven suggested which would deal 
 with edge cases where the implementation is in D but the 
 linkage isn't, but ultimately, they're just edge cases.

 - Jonathan M Davis

Yes.  I understand the practical limits of machine checking and 
the 1028 issues.  The edge case that I had in mind was piece wise 
replacement of C libraries with dlang reworkings and LCD FFI 
libraries written in dlang for Python and other languages.  With 
the symbol additions, ABI could be decoupled from D-specific 
capabilities.  Python and other C FFIs just work.  Dlang programs 
get broader guarantees.  Still, as you say, probably not a big 
concern.

jmh530 <john.michael.hall gmail.com> writes:

On Friday, 22 May 2020 at 17:40:59 UTC, Atila Neves wrote:
 [snip]
 That is a failure of the language that should be resolved.

 And how do you suggest we fix it?

 safe as whitelist instead of blacklist.

When the compiler cannot verify that it is  safe, then it cannot 
be  safe.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/22/2020 10:33 AM, rikki cattermole wrote:
 To me at least, this butchers  safe/trusted/system into a system that is near 
 useless for guarantees for an entire program.

It never attempted to guarantee safety in code that was never compiled with a D 
compiler. It's impossible to do that. No language does that.

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Fri, May 22, 2020 at 10:50:02PM -0700, Walter Bright via
Digitalmars-d-announce wrote:
 On 5/22/2020 10:33 AM, rikki cattermole wrote:
 To me at least, this butchers  safe/trusted/system into a system
 that is near useless for guarantees for an entire program.

 
 It never attempted to guarantee safety in code that was never compiled
 with a D compiler. It's impossible to do that. No language does that.

And therefore what we need is a way of indicating verifiability up to
things outside of our control. E.g., some kind of way to express that
the safety of a piece of code is keyed upon some external function or
delegate, thus enabling  safe checks for all code except calls into said
external function/delegate.

This would work out to be practically where we're at now, except that we
don't implicitly pretend external code is  safe where there is no
verification at all.


T

-- 
Designer clothes: how to cover less by paying more.

Dukc <ajieskola gmail.com> writes:

The more I think of Atila's and Walter's responses, the more they 
are starting to make sense.

When I look my own code that uses the Nuklear GUI library, 
written in C, it's all ` system`. I have not had the time to make 
` trusted` wrappers over the BindBC-nuklear API, so I did what 
tends to occur to us as the next best thing: resign and make the 
whole client code ` system`. Just making ` trusted` wrappers over 
BindBC-nuklear seemed to me as inresponsible use of the 
attribute. And reading this theard, it would seem like most of 
you would agree.

But when I think it, what I have accomplised from avoiding that 
antipattern? The only difference is, that if my D code does 
something ` system`, it'll remain under the radar. So I'm worse 
off than had I submitted to the antipattern!

Now, were I designing a library instead of an application, of 
course I should not pass such client code as ` safe`, regardless 
of which of those two ways I choose. But I think the correct way 
would be to mark the API functions as  system. Then it's be best 
of both cheap tricks: The compiler will verify my D code from 
mistakes, but I won't pretend that my code is truly ` safe`.

Dukc <ajieskola gmail.com> writes:

On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote:
 Just making ` trusted` wrappers over BindBC-nuklear seemed to 
 me as inresponsible use of the attribute.

Meant: blindly making the wrappers, without thinking whether 
calling the wrapper would always be ` safe`

Dukc <ajieskola gmail.com> writes:

On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote:
 The more I think of Atila's and Walter's responses, the more 
 they are starting to make sense.
 [snip]

In fact this former antipattern means that it'd make sense to 
have an inverse of ` trusted` attribute, lets say ` suspect`. It 
would mean that the body of the function is verified just like 
` safe`, but calling it is allowed only within ` system` or 
` trusted` code.

Paolo Invernizzi <paolo.invernizzi gmail.com> writes:

On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote:

 When I look my own code that uses the Nuklear GUI library, 
 written in C, it's all ` system`. I have not had the time to 
 make ` trusted` wrappers over the BindBC-nuklear API, so I did 
 what tends to occur to us as the next best thing: resign and 
 make the whole client code ` system`.

I really don't understand, there's no "maybe" memory safety: if 
there's no time to spend for memory safety in a project, why care?

 Just making ` trusted` wrappers over BindBC-nuklear seemed to 
 me as inresponsible use of the attribute. And reading this 
 theard, it would seem like most of you would agree.

I disagree: if you *really* want to use  safe in the rest of the 
codebase, just mark the binding as trusted, raising your hand 
towards reviewers (or everybody is interested in checking the 
memory safety of the project codebase), writing in the comment 
that "you" have decided to spend no time in a proper wrapping, 
and the motivations.

Let the reviewers and the other guys out there just decide what 
to do with your codebase.

 But when I think it, what I have accomplised from avoiding that 
 antipattern? The only difference is, that if my D code does 
 something ` system`, it'll remain under the radar. So I'm worse 
 off than had I submitted to the antipattern!

It's not an anti pattern, it clearly show your motivation, that's 
all about  trusted. May also write a big *disclaimer* in the 
README pointing to the pitfalls

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/23/20 6:55 AM, Dukc wrote:
 The more I think of Atila's and Walter's responses, the more they are 
 starting to make sense.
 
 When I look my own code that uses the Nuklear GUI library, written in C, 
 it's all ` system`. I have not had the time to make ` trusted` wrappers 
 over the BindBC-nuklear API, so I did what tends to occur to us as the 
 next best thing: resign and make the whole client code ` system`. Just 
 making ` trusted` wrappers over BindBC-nuklear seemed to me as 
 inresponsible use of the attribute. And reading this theard, it would 
 seem like most of you would agree.

This is fine, the code *is*  system. There's nothing wrong with  system 
code in D.

What is wrong is blanket assumption of  safety. In this case, people who 
actually care about memory safety want to be sure that  safe functions 
are actually  safe. If you lie about that, then the whole system breaks 
down.  safe just becomes a convention, and really everything just has to 
be manually checked.

 
 But when I think it, what I have accomplised from avoiding that 
 antipattern? The only difference is, that if my D code does something 
 ` system`, it'll remain under the radar. So I'm worse off than had I 
 submitted to the antipattern!

I understand what you are saying, you want the  safety checking in your 
code, but do not want to deal with the hassle of making special wrappers.

And let's be honest here, if you are OK with it, putting  trusted: at 
the top of your extern(C) functions is fine with me. At least that's not 
a lie.

What is not fine is having the compiler do it for you so nary a  trusted 
marking is in sight. I don't really understand the draw of that.

-Steve

JN <666total wp.pl> writes:

On Saturday, 23 May 2020 at 12:13:45 UTC, Steven Schveighoffer 
wrote:
 What is not fine is having the compiler do it for you so nary a 
  trusted marking is in sight. I don't really understand the 
 draw of that.

 -Steve

I think not only about how  safe mechanically verifiable is an 
advantage, but also  trusted blocks are very greppable so it's 
easy to find "danger zones" in any project.

I briefly skimmed through the discussion and I don't really see 
the issue with  trusted blocks other than the slightly awkward 
syntax. Yes, it's additional effort, but it should be an 
additional effort, so that people try to avoid unsafe code.


blocks whenever interacting with C code, or people build safe 
wrappers over unsafe APIs, which is exactly what would be 
expected in D after these changes.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/23/2020 5:13 AM, Steven Schveighoffer wrote:
 And let's be honest here, if you are OK with it, putting  trusted: at the top
of 
 your extern(C) functions is fine with me. At least that's not a lie.

 trusted says the interface to the function is safe. If the programmer did not 
actually check the interface, it is greenwashing. For example, memcpy() does
not 
have a safe interface and marking it  trusted doesn't improve matters.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/25/20 5:17 AM, Walter Bright wrote:
 On 5/23/2020 5:13 AM, Steven Schveighoffer wrote:
 And let's be honest here, if you are OK with it, putting  trusted: at 
 the top of your extern(C) functions is fine with me. At least that's 
 not a lie.

 
  trusted says the interface to the function is safe. If the programmer 
 did not actually check the interface, it is greenwashing. For example, 
 memcpy() does not have a safe interface and marking it  trusted doesn't 
 improve matters.

Yes, and the programmer did this on purpose. To quote someone else `My 
reasoning at the time was "I trust libclang".` In that sense, it's not a 
lie, or "incorrect," you are just trusting that whomever wrote that C 
library (maybe it was the author of the prototype) followed strictly the 
D safety rules. It's possible to write a function that takes 2 pointers 
and a size_t that is  safe, just like memcpy (it won't do anything, but 
there is nothing inherently wrong with that). If it were possible to 
determine directly from the types of parameters that a function wasn't 
safe, we would not be having this discussion.

So "trusting" a library regardless of interface is what  trusted: does. 
And while I wouldn't do that, or use a library that does that, at least 
I can see where the problem is!  trusted code is hard to write 
correctly, and there's going to be greenwashing abound when this change 
comes about. I can even envision someone *recommending* "Don't import 
core.stdc.x, just declare the prototype yourself, and you can then call 
them from safe functions!".

Greenwashing by the compiler essentially adding  trusted to every C 
porototype doesn't make it any better. I'd much rather have a clear 
indication that someone made this conscious decision, correct or not, 
and here is where you should look to find the problems. If I am 
interested in ENSURING that all code I use is  safe or correctly 
 trusted, then I know how to find those points of contention if the 
 trusted attribute is required where it should be.

The biggest problem with the magic greenwashing of this DIP is that 
there is 10-20 years of existing D code with extern(C) function 
prototypes that are specifically unmarked BECAUSE the author knew that 
meant  system. I don't know how this DIP can be approved just on that 
problem alone.

I hope something better comes out of this discussion, I'm going to mute 
it now because I don't have the time to keep arguing against people who 
don't seem to understand either memory safety or human nature.

-Steve

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Sat, May 23, 2020 at 10:55:40AM +0000, Dukc via Digitalmars-d-announce wrote:
[...]
 When I look my own code that uses the Nuklear GUI library, written in
 C, it's all ` system`. I have not had the time to make ` trusted`
 wrappers over the BindBC-nuklear API, so I did what tends to occur to
 us as the next best thing: resign and make the whole client code
 ` system`. Just making ` trusted` wrappers over BindBC-nuklear seemed
 to me as inresponsible use of the attribute. And reading this theard,
 it would seem like most of you would agree.
 
 But when I think it, what I have accomplised from avoiding that
 antipattern?  The only difference is, that if my D code does something
 ` system`, it'll remain under the radar. So I'm worse off than had I
 submitted to the antipattern!

[...]

And this is precisely why I proposed that what we need is a way for the
compiler to mechanically check all code *except* certain specified
blackboxes that are skipped over.  Then you can have your calls to
unvetted C functions and still have the mechanical checks enabled for
the rest of your code.

This is also related to  trusted blocks inside a function, the intention
of which is to limit the  system code to as small a surface area as
possible while enabling  safe checks for the rest of the function.


T

-- 
Gone Chopin. Bach in a minuet.

Paul Backus <snarwin gmail.com> writes:

On Saturday, 23 May 2020 at 15:33:05 UTC, H. S. Teoh wrote:
 On Sat, May 23, 2020 at 10:55:40AM +0000

 And this is precisely why I proposed that what we need is a way 
 for the compiler to mechanically check all code *except* 
 certain specified blackboxes that are skipped over.  Then you 
 can have your calls to unvetted C functions and still have the 
 mechanical checks enabled for the rest of your code.

 This is also related to  trusted blocks inside a function, the 
 intention of which is to limit the  system code to as small a 
 surface area as possible while enabling  safe checks for the 
 rest of the function.


 T

 system void foo(...) {
     ()  safe {
         // mechanically checked
     }();
     // not checked
}

Atila Neves <atila.neves gmail.com> writes:

On Saturday, 23 May 2020 at 10:55:40 UTC, Dukc wrote:
 The more I think of Atila's and Walter's responses, the more 
 they are starting to make sense.

 [...]

Thank you for the anecdote, especially since it captures the 
spirit of what I've been trying to convey here.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/22/20 1:07 PM, Atila Neves wrote:
 And so I was convinced that everything being  safe is actually ok, 
 especially because in real life, most C/C++ APIs aren't going to 
 secretly corrupt your code.

Yes, it can, but not secretly. Just obviously and easily. Note this 
function:

https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/clang/c/index.d#L861

And so, you are free to pepper your  safe code with dangling pointers. 
Sure, you can claim that the C++ library didn't "corrupt your code", 
which is the case for ALL libraries if you use them properly. You did 
it, you created a dangling pointer, not the library.

The point of  safe is to protect you from corrupting your own code based 
on the guarantees that  safe provides. If we don't care about the 
guarantees of  safe as long as you are using C libraries, why are we 
bothering at all with any of this? I'd rather see a DIP that just 
removes safety completely if the language maintainers just trust all C 
libraries "in real life".

BTW, you should fix that invalid attribute, freeing a pointer is never 
 safe unless you can guarantee nobody else has a copy of that pointer 
(and considering it's passed by value, the CALLER still has that pointer!)

Will that break your code? It sure might! It just means YOUR code needs 
to be marked  trusted or  system as appropriate.

-Steve

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 17:41:38 UTC, Steven Schveighoffer 
wrote:
 On 5/22/20 1:07 PM, Atila Neves wrote:
 And so I was convinced that everything being  safe is actually 
 ok, especially because in real life, most C/C++ APIs aren't 
 going to secretly corrupt your code.

 Yes, it can, but not secretly. Just obviously and easily. Note 
 this function:

 https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/clang/c/index.d#L861

 And so, you are free to pepper your  safe code with dangling 
 pointers. Sure, you can claim that the C++ library didn't 
 "corrupt your code", which is the case for ALL libraries if you 
 use them properly. You did it, you created a dangling pointer, 
 not the library.

Right. And the point I was trying to make wasn't "look at what I 
did, it's cool". No, what I did was dumb. So dumb it took you no 
time at all to point out one of my mistakes. My point is that the 
result of making declarations implicity  system instead of  safe 
would make people just slap  safe on them without really thinking 
about it to get their code to compile. Like I did.

 The point of  safe is to protect you from corrupting your own 
 code based on the guarantees that  safe provides.

I agree.

 If we don't care about the guarantees of  safe as long as you 
 are using C libraries, why are we bothering at all with any of 
 this?

We care. Annotations become explicit. Do I think this is ideal? 
No.

 BTW, you should fix that invalid attribute, freeing a pointer 
 is never  safe unless you can guarantee nobody else has a copy 
 of that pointer (and considering it's passed by value, the 
 CALLER still has that pointer!)

You're completely right.

ag0aep6g <anonymous example.com> writes:

On 22.05.20 19:54, Atila Neves wrote:
 Right. And the point I was trying to make wasn't "look at what I did, 
 it's cool". No, what I did was dumb. So dumb it took you no time at all 
 to point out one of my mistakes. My point is that the result of making 
 declarations implicity  system instead of  safe would make people just 
 slap  safe on them without really thinking about it to get their code to 
 compile. Like I did.

Now you have accepted a DIP that does the dumb thing automatically. How 
is that any better?

[...]
 If we don't care about the guarantees of  safe as long as you are 
 using C libraries, why are we bothering at all with any of this?

 
 We care. Annotations become explicit. Do I think this is ideal? No.

"Annotations become explicit." - What now? I probably misunderstand that 
sentence, but DIP 1028 does not require explicit annotations. That's why 
everyone is upset.

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 18:17:29 UTC, ag0aep6g wrote:
 On 22.05.20 19:54, Atila Neves wrote:

 We care. Annotations become explicit. Do I think this is 
 ideal? No.

 "Annotations become explicit." - What now? I probably 
 misunderstand that sentence, but DIP 1028 does not require 
 explicit annotations. That's why everyone is upset.

Sorry, I didn't express myself well. I meant that the user can 
still mark functions as  system, they just have to do it 
explicitly.

Dukc <ajieskola gmail.com> writes:

On Friday, 22 May 2020 at 18:27:42 UTC, Atila Neves wrote:
 Sorry, I didn't express myself well. I meant that the user can 
 still mark functions as  system, they just have to do it 
 explicitly.

Hm, DPP might be of help here. Becuse I quess you are going to 
make sure it'll mark everything ` system`?

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 20:08:37 UTC, Dukc wrote:
 On Friday, 22 May 2020 at 18:27:42 UTC, Atila Neves wrote:
 Sorry, I didn't express myself well. I meant that the user can 
 still mark functions as  system, they just have to do it 
 explicitly.

 Hm, DPP might be of help here. Becuse I quess you are going to 
 make sure it'll mark everything ` system`?

My plan is probably to do that, yes. I've been thinking about it 
and I'll likely need to give users options to opt-in to declaring 
functions  safe, otherwise none of them every would be with this:

 system {
     #include "header.h"
}

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/22/20 1:54 PM, Atila Neves wrote:
 My point is that the result of making declarations implicity  system 
 instead of  safe would make people just slap  safe on them without 
 really thinking about it to get their code to compile. Like I did.

So the solution is -- make the compiler be dumb for you? If you are 
going to incorrectly put  safe on it, I want you to have to do it 
because YOU made a conscious decision to be careless, not have it done 
for you because you forgot.

How about we change the law so if you don't sign a contract, we assumed 
you signed it anyway. In order to reject the contract, you have to X out 
every paragraph and write "I don't agree to this" and sign that. Any 
paragraphs you forget are enforceable.

-Steve

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On Friday, 22 May 2020 at 18:32:59 UTC, Steven Schveighoffer 
wrote:
 So the solution is -- make the compiler be dumb for you? If you 
 are going to incorrectly put  safe on it, I want you to have to 
 do it because YOU made a conscious decision to be careless, not 
 have it done for you because you forgot.

Yea, agreed.  If a library works around an extern(C) API by 
slapping ` safe:` on it, this is a visibly problematic thing to 
do, and people can file bug reports and patches about it.  And we 
can have well documented warnings about this being a bad way to 
do things.

If a library just gets extern(C) APIs as  safe by default, then 
it's not visibly obvious that this is problematic.

I would much rather have the problematic thing as something 
visibly done by the developer, than automatically done by the 
compiler.

Paolo Invernizzi <paolo.invernizzi gmail.com> writes:

On Friday, 22 May 2020 at 17:54:26 UTC, Atila Neves wrote:
 On Friday, 22 May 2020 at 17:41:38 UTC, Steven Schveighoffer

 And so, you are free to pepper your  safe code with dangling 
 pointers. Sure, you can claim that the C++ library didn't 
 "corrupt your code", which is the case for ALL libraries if 
 you use them properly. You did it, you created a dangling 
 pointer, not the library.

 Right. And the point I was trying to make wasn't "look at what 
 I did, it's cool". No, what I did was dumb. So dumb it took you 
 no time at all to point out one of my mistakes. My point is 
 that the result of making declarations implicity  system 
 instead of  safe would make people just slap  safe on them 
 without really thinking about it to get their code to compile. 
 Like I did.

So force people to slap  trusted instead, via compiler complains, 
not  safe, and reviewers will catch the laziness: why this is 
worst that what you picture?

Walter Bright <newshound2 digitalmars.com> writes:

On 5/22/2020 10:54 AM, Atila Neves wrote:
 BTW, you should fix that invalid attribute, freeing a pointer is never  safe 
 unless you can guarantee nobody else has a copy of that pointer (and 
 considering it's passed by value, the CALLER still has that pointer!)

 
 You're completely right.

 live is intended to fix that!

To get ahead of the curve, annotate pointer parameters with `scope` if you
don't 
intend to transfer ownership to the callee. Better yet, use `ref` instead of 
pointer parameters where possible.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/22/20 8:05 PM, Walter Bright wrote:
 On 5/22/2020 10:54 AM, Atila Neves wrote:
 BTW, you should fix that invalid attribute, freeing a pointer is 
 never  safe unless you can guarantee nobody else has a copy of that 
 pointer (and considering it's passed by value, the CALLER still has 
 that pointer!)

 You're completely right.

 
  live is intended to fix that!
 
 To get ahead of the curve, annotate pointer parameters with `scope` if 
 you don't intend to transfer ownership to the callee. Better yet, use 
 `ref` instead of pointer parameters where possible.

These are extern(C) functions written in C. How does one apply  live to 
that? You can't change it to ref in the prototype.

Besides, a deallocation function shouldn't use  live, it's about to 
destroy the pointer.

-Steve

Paolo Invernizzi <paolo.invernizzi gmail.com> writes:

On Friday, 22 May 2020 at 17:07:37 UTC, Atila Neves wrote:

 And so I was convinced that everything being  safe is actually 
 ok, especially because in real life, most C/C++ APIs aren't 
 going to secretly corrupt your code.

Uh? There's plenty of C/C++ code out there with api that when 
"used in the wrong way" will corrupt memory.

That's the _essence_ of  trusted code, the _programmer_ need to 
_correctly_ handle the usage, because the compiler can't do that 
job alone.

ag0aep6g <anonymous example.com> writes:

On 22.05.20 19:07, Atila Neves wrote:
 Personally and initially, I would have preferred it if non-extern(D) 
 declarations were implicitly  system. I understood Walter's argument 
 about special cases and how they're bad, but the thought of them being 
  safe made me feel, for lack of a better word, "icky".

So the DIP itself wasn't good enough to convince you. Should have 
rejected it then. Walter could have made another, better argued DIP. 
That's how this is supposed to work, isn't it?

 Then I talked to Walter and he pointed out that if those declarations 
 were  system, users would be prevented from calling them from now  safe 
 code. A regular user would probably just slap ` safe:` at the top of the 
 bindings module anyway. Then I realised that I did exactly that with my 
 libclang bindings:
 
 https://github.com/atilaneves/libclang/blob/5415707fa6072700bdbf21f827567ffa2fd5c424/source/
lang/c/index.d#L254 
 
 
 "Worse", I made all functions `pure` and all parameters `in` as well for 
 good measure. Why? I wanted to call them from my  safe pure code with 
 `const` arguments. My reasoning at the time was "I trust libclang".
 
 And so I was convinced that everything being  safe is actually ok, 
 especially because in real life, most C/C++ APIs aren't going to 
 secretly corrupt your code.

Your reasoning is fine when you're dealing with a function that has a 
safe interface. I.e., it can only corrupt your code when it's buggy. 
Then it makes sense so say "I trust the authors". D users do it with 
Phobos. Phobos authors do it with the C standard library. This kind of 
trust is perfectly ok.

Your reasoning is completely wrong when you're thinking about a function 
that doesn't have a safe interface. E.g., C's `memcpy` just isn't safe. 
It doesn't matter whether you trust the authors or not. The user has to 
make sure that the call is correct. Otherwise, `memcpy` will corrupt 
your memory. And `memcpy` is far from the only such function.

I'm positive that Walter is aware of the difference. Judging from this 
post, I'm afraid you might not be.

Atila Neves <atila.neves gmail.com> writes:

On Friday, 22 May 2020 at 18:11:28 UTC, ag0aep6g wrote:
 So the DIP itself wasn't good enough to convince you.

Had that been the case, I would have rejected it.

 Your reasoning is fine when you're dealing with a function that 
 has a safe interface. I.e., it can only corrupt your code when 
 it's buggy. Then it makes sense so say "I trust the authors". D 
 users do it with Phobos. Phobos authors do it with the C 
 standard library. This kind of trust is perfectly ok.
 Your reasoning is completely wrong when you're thinking about a 
 function that doesn't have a safe interface. E.g., C's `memcpy` 
 just isn't safe.

memcpy isn't a good example since it's explicitly  system:

https://dlang.org/phobos/core_stdc_string.html#.memcpy

 It doesn't matter whether you trust the authors or not. The 
 user has to make sure that the call is correct. Otherwise, 
 `memcpy` will corrupt your memory. And `memcpy` is far from the 
 only such function.

Yes. But most of them aren't like memcpy. Most D code calls other 
D code, not C.

Am I saying nothing bad can happen if we implicitly trust 
extern(C) declarations? No. I'm saying we'll be no worse off if 
they're all implicitly  system.

This compiles with no warnings right *now*:

void main() {
     import core.stdc.stdlib: free;
     free(cast(void*) 42);
     free(new int);
     free(&main);
}

ag0aep6g <anonymous example.com> writes:

On 22.05.20 20:24, Atila Neves wrote:
 On Friday, 22 May 2020 at 18:11:28 UTC, ag0aep6g wrote:
 So the DIP itself wasn't good enough to convince you.

 
 Had that been the case, I would have rejected it.

You said the DIP as written felt "icky". And only after a chat with 
Walter did that feeling go away. You're saying you would have accepted a 
DIP that feels icky? DIPs are supposed to be convincing, not barely 
tolerable.

 memcpy isn't a good example since it's explicitly  system:
 
 https://dlang.org/phobos/core_stdc_string.html#.memcpy

core.stdc.string.memcpy is a specific binding to C's memcpy. Any other 
declaration of it will be implicitly  safe.

"But why would anyone declare `memcpy` themselves, when they can just 
import the one from `core`?" I hear you ask. And I answer:

1) Shouldn't matter, if we're talking about a principled safety system. 
But I realize that the leadership isn't interested in that anymore.

2) I have quickly typed out C declarations before, because it was more 
convenient than looking up where it is exactly in the standard library. 
And we're all about catering to convenience now, aren't we?

 Yes. But most of them aren't like memcpy. Most D code calls other D 
 code, not C.

Most D code isn't behind an `extern (C)` prototype. Virtually no one is 
(strongly) against making D functions  safe by default.

 Am I saying nothing bad can happen if we implicitly trust extern(C) 
 declarations? No. I'm saying we'll be no worse off if they're all 
 implicitly  system.

"No worse off" should not be good enough for a DIP.

 This compiles with no warnings right *now*:
 
 void main() {
      import core.stdc.stdlib: free;
      free(cast(void*) 42);
      free(new int);
      free(&main);
 }

And this doesn't compile right now, but it will with DIP 1028:

----
extern (C) void free(void*);
void main()  safe {
      free(cast(void*) 42);
      free(new int);
      free(&main);
}
----

Yes, yes, I know. That's soo much less common in the real world. But 
before the great  safe schism that is happening right now,  safe wasn't 
about catching more bugs than not on average. It was about being "100% 
mechanically checkable" (Walter's words).

I for one liked it better when it had that aspiration. And I was 
contributing towards that goal (latest PR was merged just yesterday, 
coincidentally [1]). I have no interest in a version of  safe that is 
deliberately fuzzy, where I have to defend safety fixes against 
convenience arguments.


[1] https://github.com/dlang/dlang.org/pull/2773

Paul Backus <snarwin gmail.com> writes:

On Friday, 22 May 2020 at 19:16:26 UTC, ag0aep6g wrote:
 I for one liked it better when it had that aspiration. And I 
 was contributing towards that goal (latest PR was merged just 
 yesterday, coincidentally [1]).

 [1] https://github.com/dlang/dlang.org/pull/2773

Thank you for that PR, by the way. The new version of the spec is 
*much* clearer, and actually helped me find a subtle safety issue 
in my own code that I'd previously overlooked. I sincerely hope 
the D leadership appreciates how valuable this kind of work is.

Mathias LANG <geod24 gmail.com> writes:

On Friday, 22 May 2020 at 18:24:39 UTC, Atila Neves wrote:
 Am I saying nothing bad can happen if we implicitly trust 
 extern(C) declarations? No. I'm saying we'll be no worse off if 
 they're all implicitly  system.

 This compiles with no warnings right *now*:

 void main() {
     import core.stdc.stdlib: free;
     free(cast(void*) 42);
     free(new int);
     free(&main);
 }

And this code won't compile at all when you make ` safe` the 
default:

```
void main ()
{
     int[] myArray = (cast(int*) malloc(42 * int.sizeof))[0 .. 42];
}
```

So, we changed the default.

Now those people that want to easily prototype something, or 
write applications that extensively use system libraries, will be 
extremely inconvenienced, but we satisfied the people that want 
more ` safe`ty, which we all known and agree is the future.

Except that in that process we also destroyed the meaning of 
` safe` by shooting a hole into it, for the convenience of the 
people that want to easily prototype something and write 
applications that interact extensively with system libraries, 
because that's what half of the D users do.

Clearly, an improvement.

Dominikus Dittes Scherkl <dominikus scherkl.de> writes:

On Friday, 22 May 2020 at 18:24:39 UTC, Atila Neves wrote:
 memcpy isn't a good example since it's explicitly  system

Yes, it's a good example. Because if you include any C function, 
you don't know if it uses memcpy internally - except if you have 
the code. And as memcpy is used heavily within C libraries, it's 
not unlikely that your "oh so safe" function is using it too. 
Very bad advice.

It should be simply forbidden to slap  safe at anything with no 
code available, because  save should be reserved for mechanically 
verified parts of code.

Ok, you can slap  trusted on it, and it will compile (the easy 
way), but anyone using it is at least warned. So it's possible to 
check - and if no body is availabe (e.g. an object library) I 
would refrain from trusting it.

But I don't want the compiler to greenwash it for me, no thanks. 
There should be at least someone on the D side to blame, 
destroying his reputation by slapping  trusted at whole modules.

jmh530 <john.michael.hall gmail.com> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 [snip]

Thank you for your reply.

How about some time before this DIP is fully in the language, a 
compiler flag is added that will produce warnings for when extern 
prototypes without explicit  safe/ trusted/ system are used? Or 
something like that. I have a good sense of when my own code uses 
extern(C), but not always other people's.

Francesco Mecca <me francescomecca.eu> writes:

On Friday, 22 May 2020 at 12:25:23 UTC, jmh530 wrote:
 On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 [snip]

 Thank you for your reply.

 How about some time before this DIP is fully in the language, a 
 compiler flag is added that will produce warnings for when 
 extern prototypes without explicit  safe/ trusted/ system are 
 used? Or something like that. I have a good sense of when my 
 own code uses extern(C), but not always other people's.

There is no reason to make the compiler do that. I think it would 
be more appropriate for D-scanner to do that or more practically 
for dub

https://github.com/dlang-community/D-Scanner

Timon Gehr <timon.gehr gmx.ch> writes:

On 22.05.20 03:22, Walter Bright wrote:
 
 This is Obviously A Good Idea. Why would I oppose it?
 
 1. I've been hittin' the crack pipe again.
 2. I was secretly convinced, but wanted to save face.
 3. I make decisions based on consultation with my astrologer.
 4. I am evil.

5. You are backwards-rationalizing a wrong intuition that is based on 
experiences that are not actually analogous. You are ignoring feedback 
given by many people around you because that worked out well for you in 
the past.

I know that you had many interactions with large groups of ignorant 
people who thought that you would never be able to pull off a certain 
thing. This is not one of those cases. I understand the appeal, but the 
backlash really should not encourage you to soldier on this time.

 
 1. Go through 200 functions in clibrary.d and determine which are  safe
 and which are  system. This is what we want them to do. We try to motivate
 this with compiler error messages. Unfortunately, this is both tedious and
 thoroughly impractical, as our poor user Will Not Know which are safe and
 which are system. We can correctly annotate core.stdc.stdio because I know
 those functions intimately. This is not true for other system C APIs, and
 even less true for some third party C library we're trying to interface to.
 
 2. Annotate useClibrary() as  trusted or  system. While easier,

First do 2, then, over time, do 1. If having the  safe tag and no 
 trusted code is important to you, aim to replace the C code with 
something you can automatically verify, by slowly porting it over to D.

 this causes all benefits to  safe by default to be lost. 

Absolutely not. Greenwashing causes the benefits of certification to be 
lost. Honesty does not. The value of  safe code is what it is because 
there is code that can't be  safe.

 
 4. Edit clibrary.d and make the first line:
 
       safe:
 
 I submit that, just like with Java, Option 4 is what people will reach for,
 nearly every time. I've had some private conversations where people 
 admitted
 this was what they'd do. People who knew it was wrong to do that.
 ...

They should know to put  trusted instead of  safe, and the compiler 
should enforce it. Also, why do those people speak for everyone else? 
They don't speak for me.

 If it's  safe by default, and then someone chooses to annotate it with 
  system
 here and there, I'd feel a lot more confident about the accuracy of the 
 code
 annotations than if it just had  safe: at the top. At least they tried.
 ...

If it has  safe:/ trusted: at the top at least you know they were aware 
what they were doing.
Also, what about if it has  trusted: at the top and some  system 
annotations here and there? Did they not try?

 What is actually accomplished with this amendment if it was implemented?
 
 1. Adds a funky, special case rule. It's better to have simple, easily
 understood rules than ones with special cases offering little improvement.
 ...

What about the funky special case rule that the compiler is responsible 
for memory safety of  safe code except in this one weird special case?

 2. Existing, working code breaks.
 ...

Making  safe the default is bound to break code. It's bad enough that 
code will break. Avoiding part of that code breakage is no justification 
for breaking  safe.

 3. The most likely code fixes are to just make it compile, absolutely
 nothing safety-wise is improved. The added annotations will be a fraud.
 ...

It's vastly better to have some fraudulent annotations in some projects 
than a fraudulent compiler compiling all projects. Do you really want to 
put the responsibility for the memory safety of random C libraries on 
the compiler developers?

 D should not encourage "greenwashing" practices like the Java
 exception specification engendered.

So your argument is that you don't want D programmers to do have to do 
the dirty work of greenwashing. Therefore the compiler will implicitly 
greenwash for them? What about the programmers who actually want to do 
the right thing and don't want the compiler to implicitly greenwash C 
libraries for them?

 The compiler cannot vet the accuracy
 of bodyless C functions, and we'll just have to live with that. The 
 proposed
 amendment does not fix that.
 ...

 trusted is the fix.

 And so, I did not incorporate the proposed amendment to the Safe by Default
 DIP.

Which (so far) is a harmless mistake with an easy fix.

Adam D. Ruppe <destructionator gmail.com> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 3. Wrap the call to massage_data() with:

     ()  trusted { massage_data(parameters); } ();

The correct solution is to encapsulate the C functions as-needed 
with a higher level API - and this is somewhat commonly done 
already and - as i understand it anyway - the whole purpose of 
 trusted being function level instead of statement level.

 2. Existing, working code breaks.

 safe by default is probably the biggest breaking change D has 
ever seen as of this point. It eclipses anything in the D2 
transition.

It is D3 in all by name. My preference, as I've said many times, 
is to fix the propagation of attributes into child scopes and 
leave the defaults alone, but failing that, I'd actually say 
embrace this: if we're doing big breakage anyway, go big and do 
it right.

Gregory <g.thompson.1892 gmall.com> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 Consider the common (because that's how D started out) case of:

 ----- clibrary.d --------

     T massage_data(... many parameters ...);
     ... 200 more such declarations ...

 ----- app.d ----------

     import clibrary;

     void useClibrary( ... parameters ...) {
         massage_data(parameters);
     }

 ---------------------

 This code, today, does not use annotations and it works. It's 
 been working
 for a long time. Now, we implement the amendment, and it stops 
 compiling
 because useClibrary is  safe and massage_data is  system. The 
 user is faced
 with the following alternatives:

Why are you assuming that the only thing wrong with useClibrary() 
is that it would use a C declaration that is  system? All  system 
code I've written is  system and wouldn't compile to  safe even 
if extern(C) declarations are wrongly annotated as  safe.

What if app uses core.stdc? All those functions have already been 
annotated as  system. Any code that uses the C stdlib is going to 
break right now anyways.

What's stopping someone from just as easily slapping  trusted: at 
the top of app.d? Since there are going to be more issues than 
simply calling extern(C) functions.

I find it odd also that you are using one of the rationales 
against making  safe the default (that code will break and lazy 
solutions will be employed to make them work again), as a means 
to make  safe, less memory safe.

jmh530 <john.michael.hall gmail.com> writes:

On Friday, 22 May 2020 at 18:34:32 UTC, Gregory wrote:
 [snip]

 Why are you assuming that the only thing wrong with 
 useClibrary() is that it would use a C declaration that is 
  system? All  system code I've written is  system and wouldn't 
 compile to  safe even if extern(C) declarations are wrongly 
 annotated as  safe.

 What if app uses core.stdc? All those functions have already 
 been annotated as  system. Any code that uses the C stdlib is 
 going to break right now anyways.

 [snip]

The breakage would produce error messages. Annoying to fix, but 
at least obvious. The worry is more about not receiving any error 
messages. In other words, some code would compile as  system 
previously, but now would compile as  safe with nothing else 
changing. It's  safe, but is it really safe?

Johannes T <isrvoid gmail.com> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.
 [..]

Thank you for the detailed and insightful explanation.
Would it be feasible to have a follow-up DIP that enables 
implicit  system extern as an opt-in (e.g. -nosafeextern). The 
frustration most people seem to have with  safe extern is that it 
lessens the promise of  safe without a recourse.

Arafel <er.krali gmail.com> writes:

On 23/5/20 20:40, Johannes T wrote:
 On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.
 [..]

 
 Thank you for the detailed and insightful explanation.
 Would it be feasible to have a follow-up DIP that enables implicit 
  system extern as an opt-in (e.g. -nosafeextern). The frustration most 
 people seem to have with  safe extern is that it lessens the promise of 
  safe without a recourse.

Also there will need to be an answer to this probably inthe future not 
unusual question:

My code is 100%  safe, there's no  trusted block anywhere, why am I 
getting memory corruption?

To me the expectation / promise was that  safe code was safe from memory 
corruption modulo bugs in the implementation.

The only source for issues would be then  trusted code that could be 
audited as needed.

You could say "well, you always had to check for those", but the 
difference is that until now the compiler would shout at me if I didn't, 
so I wouldn't forget about them.

Now it has to be clearly explained that you should check for  trusted 
code AND unmarked external C functions.

ag0aep6g <anonymous example.com> writes:

On 23.05.20 21:46, Arafel wrote:
 Now it has to be clearly explained that you should check for  trusted 
 code AND unmarked external C functions.

... and  system static constructors and `--boundscheck=off` and 
initializers of globals

Arafel <er.krali gmail.com> writes:

On 24/5/20 0:02, ag0aep6g wrote:
 
 ... and  system static constructors and `--boundscheck=off` and 
 initializers of globals

Other than `--boundscheck=off`, that is presumably actively chosen by 
the user (as is  trust), would the others be allowed without ` trusted` 
in otherwise 100%  safe code?

I would find concerning that any  system code is allowed, but I guess 
initializers of globals should be ok as long as they are  safe themselves?

ag0aep6g <anonymous example.com> writes:

On 24.05.20 00:17, Arafel wrote:
 On 24/5/20 0:02, ag0aep6g wrote:
 ... and  system static constructors and `--boundscheck=off` and 
 initializers of globals

 
 Other than `--boundscheck=off`, that is presumably actively chosen by 
 the user (as is  trust), would the others be allowed without ` trusted` 
 in otherwise 100%  safe code?

Yup. Today they can be unmarked, defaulting to  system. With DIP 1028, 
they can be explicitly marked  system. Either way, they don't show up 
when you only look for " trusted".

 I would find concerning that any  system code is allowed, but I guess 
 initializers of globals should be ok as long as they are  safe themselves?

As long as they're  safe, sure. But they can also be  system.

An example:
----
const int x = 42;
const int y = 43;

void main()  safe
{
     import std.stdio;
     writeln(x, " ", y); /* Prints "42 43" as expected. */
     auto px = &x;
     auto py = &y;
     writeln(*px, " ", *py); /* Prints "13 14". Wat? */
}

int* p = cast(int*) &x;
static this()  system { *p = 13; *++p = 14; }
----

Arafel <er.krali gmail.com> writes:

On 24/5/20 0:38, ag0aep6g wrote:
 On 24.05.20 00:17, Arafel wrote:
 On 24/5/20 0:02, ag0aep6g wrote:
 ... and  system static constructors and `--boundscheck=off` and 
 initializers of globals

 Other than `--boundscheck=off`, that is presumably actively chosen by 
 the user (as is  trust), would the others be allowed without 
 ` trusted` in otherwise 100%  safe code?

 
 Yup. Today they can be unmarked, defaulting to  system. With DIP 1028, 
 they can be explicitly marked  system. Either way, they don't show up 
 when you only look for " trusted".
 
 I would find concerning that any  system code is allowed, but I guess 
 initializers of globals should be ok as long as they are  safe 
 themselves?

 
 As long as they're  safe, sure. But they can also be  system.
 
 An example:
 ----
 const int x = 42;
 const int y = 43;
 
 void main()  safe
 {
      import std.stdio;
      writeln(x, " ", y); /* Prints "42 43" as expected. */
      auto px = &x;
      auto py = &y;
      writeln(*px, " ", *py); /* Prints "13 14". Wat? */
 }
 
 int* p = cast(int*) &x;
 static this()  system { *p = 13; *++p = 14; }
 ----
 

I find this... disturbing... what is worse, it also happens with 
`immutable`.

Then there should most definitely be a list of features to look (in 
addition to ` trusted`) if you want to make sure your ` safe` code is 
actually safe.

Arine <arine1283798123 gmail.com> writes:

On Saturday, 23 May 2020 at 22:38:58 UTC, ag0aep6g wrote:
 On 24.05.20 00:17, Arafel wrote:
 On 24/5/20 0:02, ag0aep6g wrote:
 ... and  system static constructors and `--boundscheck=off` 
 and initializers of globals

 
 Other than `--boundscheck=off`, that is presumably actively 
 chosen by the user (as is  trust), would the others be allowed 
 without ` trusted` in otherwise 100%  safe code?

 Yup. Today they can be unmarked, defaulting to  system. With 
 DIP 1028, they can be explicitly marked  system. Either way, 
 they don't show up when you only look for " trusted".

 I would find concerning that any  system code is allowed, but 
 I guess initializers of globals should be ok as long as they 
 are  safe themselves?

 As long as they're  safe, sure. But they can also be  system.

 An example:
 ----
 const int x = 42;
 const int y = 43;

 void main()  safe
 {
     import std.stdio;
     writeln(x, " ", y); /* Prints "42 43" as expected. */
     auto px = &x;
     auto py = &y;
     writeln(*px, " ", *py); /* Prints "13 14". Wat? */
 }

 int* p = cast(int*) &x;
 static this()  system { *p = 13; *++p = 14; }
 ----

That works even if you make the static this()  safe, and remove 
the pointer incrementation.

You'd have to make the p initialization  safe.

      safe:
         int* p = cast(int*) &x; // error

But note this doesn't work:

      safe int* p = cast(int*) &x; // compiles

Having the default become  safe will help detect this, as I don't 
imagine that is a whole lot of usage of  safe: to begin with.

ag0aep6g <anonymous example.com> writes:

On 24.05.20 02:55, Arine wrote:
 That works even if you make the static this()  safe, and remove the 
 pointer incrementation.

Sure. `*p = 13;` is perfectly  safe. The static constructor isn't needed 
for that part. You can just as well do the assignment in `main`. The 
static constructor is another feature that can smuggle unsafe code (the 
increment) into your program without the  trusted warning label.

 You'd have to make the p initialization  safe.
 
       safe:
          int* p = cast(int*) &x; // error
 
 But note this doesn't work:
 
       safe int* p = cast(int*) &x; // compiles
 
 Having the default become  safe will help detect this, as I don't 
 imagine that is a whole lot of usage of  safe: to begin with.

The example compiles with `-preview=safedefault`. And even if that gets 
changed, it will probably still compile when marked  system. So we still 
won't find it when looking for " trusted".

Arine <arine1283798123 gmail.com> writes:

On Sunday, 24 May 2020 at 01:26:02 UTC, ag0aep6g wrote:
 On 24.05.20 02:55, Arine wrote:
 That works even if you make the static this()  safe, and 
 remove the pointer incrementation.

 Sure. `*p = 13;` is perfectly  safe. The static constructor 
 isn't needed for that part. You can just as well do the 
 assignment in `main`. The static constructor is another feature 
 that can smuggle unsafe code (the increment) into your program 
 without the  trusted warning label.

It'd be no different than passing the pointer into  safe code as 
a parameter from  system code. Ultimately the error occurs in 
 system code and directly as a result of  system code. It is 
undefined behavior as well. No amount of safe code can save you 
from that.

 You'd have to make the p initialization  safe.
 
       safe:
          int* p = cast(int*) &x; // error
 
 But note this doesn't work:
 
       safe int* p = cast(int*) &x; // compiles
 
 Having the default become  safe will help detect this, as I 
 don't imagine that is a whole lot of usage of  safe: to begin 
 with.

 The example compiles with `-preview=safedefault`. And even if 
 that gets changed, it will probably still compile when marked 
  system. So we still won't find it when looking for " trusted".

Then that is definitely a bug if that's the case. Someone should 
probably make a bug report, Walter? If you are still using 
 system with  safe, then that would still be somewhere you have 
to look for not memory safe code.  trusted should just mean that 
someone verified it.  system then would mean no one's verified it 
to be safe, that doesn't mean you don't have to check it.

ag0aep6g <anonymous example.com> writes:

On Sunday, 24 May 2020 at 14:39:50 UTC, Arine wrote:
[...]
 It'd be no different than passing the pointer into  safe code 
 as a parameter from  system code. Ultimately the error occurs 
 in  system code and directly as a result of  system code. It is 
 undefined behavior as well. No amount of safe code can save you 
 from that.

I think you're arguing against a point that wasn't made. I'm not 
saying that there's anything fundamentally unsound about an 
 system static constructor. As you say, it's the same any other 
 system function.

I'm just saying that it's another thing you have to check when 
you want to verify that a program is actually safe.

[...]
 Then that is definitely a bug if that's the case. Someone 
 should probably make a bug report, Walter? If you are still 
 using  system with  safe, then that would still be somewhere 
 you have to look for not memory safe code.  trusted should just 
 mean that someone verified it.  system then would mean no one's 
 verified it to be safe, that doesn't mean you don't have to 
 check it.

 system does indicate that you don't have to check a function. 
But its trumped by other indicators:

*  system entry points (`main`, static constructors, static 
initializers) - have to check those.

* Foreign prototypes (`extern (C)` and friends) - have to check 
those, whether they're  system or  safe or  trusted.

*  system functions that are being called by  trusted ones - have 
to check those. But I would say that's part of verifying  trusted 
functions.

Other than that (and maybe other special cases that I've missed), 
you can safely ignore  system functions, because your  safe 
program cannot possibly be calling them.

Arine <arine1283798123 gmail.com> writes:

On Sunday, 24 May 2020 at 15:42:54 UTC, ag0aep6g wrote:
 On Sunday, 24 May 2020 at 14:39:50 UTC, Arine wrote:
 Then that is definitely a bug if that's the case. Someone 
 should probably make a bug report, Walter? If you are still 
 using  system with  safe, then that would still be somewhere 
 you have to look for not memory safe code.  trusted should 
 just mean that someone verified it.  system then would mean no 
 one's verified it to be safe, that doesn't mean you don't have 
 to check it.

  system does indicate that you don't have to check a function. 
 But its trumped by other indicators:

 *  system entry points (`main`, static constructors, static 
 initializers) - have to check those.

 * Foreign prototypes (`extern (C)` and friends) - have to check 
 those, whether they're  system or  safe or  trusted.

 *  system functions that are being called by  trusted ones - 
 have to check those. But I would say that's part of verifying 
  trusted functions.

 Other than that (and maybe other special cases that I've 
 missed), you can safely ignore  system functions, because your 
  safe program cannot possibly be calling them.

You *have* to check  system code. That's where you are 
guarantee'd to have memory safety issues. If you are ignoring 
 system code because you think  safe code doesn't interact with 
it at all, then that's a problem you are creating for yourself. 
 system code can still call  safe code, and that  system code 
that is calling the  safe code can pass invalid information that 
causes the  safe code to misbehave. You have to check  system for 
memory safety issues. It seems Walter's comments about only have 
to review  trusted are being taken too literally.

ag0aep6g <anonymous example.com> writes:

On 24.05.20 19:44, Arine wrote:
 On Sunday, 24 May 2020 at 15:42:54 UTC, ag0aep6g wrote:

[...]
  system does indicate that you don't have to check a function. But its 
 trumped by other indicators:


[...]
 You *have* to check  system code. That's where you are guarantee'd to 
 have memory safety issues. If you are ignoring  system code because you 
 think  safe code doesn't interact with it at all, then that's a problem 
 you are creating for yourself.  system code can still call  safe code, 
 and that  system code that is calling the  safe code can pass invalid 
 information that causes the  safe code to misbehave. You have to check 
  system for memory safety issues.

You're right; it's not accurate that " system does indicate that you 
don't have to check a function". That's only true under particular 
conditions:

When your entry points are  safe and you have already verified all 
 trusted functions (including their call graphs which might include 
 system functions), then you can ignore any other  system functions, 
because your program doesn't call them anyway. But that's true for any 
function. If your program doesn't call it, you don't need to check it.

So it's not a particularly meaningful thing to say about  system, and 
that's on me.

Walter Bright <newshound2 digitalmars.com> writes:

I'd like to emphasize:

1. It is not possible for the compiler to check any declarations where the 
implementation is not available. Not in D, not in any language. Declaring a 
declaration safe does not make it safe.

2. If un-annotated declarations cause a compile time error, it is highly likely 
the programmer will resort to "greenwashing" - just slapping  safe on it. I've 
greenwashed code. Atila has. Bruce Eckel has. We've all done it. Sometimes even 
for good reasons.

3. Un-annotated declarations are easily detectable in a code review.

4. Greenwashing is not easily detectable in a code review.

5. Greenwashing doesn't fix anything. The code is not safer. It's an illusion, 
not a guarantee.

6. If someone cares to annotate declarations, it means he has at least thought 
about it, because he doesn't need to. Hence it's more likely to be correct than 
when greenwashed.

7. D should *not* make it worthwhile for people to greenwash code.

It is, in a not-at-all obvious way, safer for C declarations to default to
being 
safe.

Timon Gehr <timon.gehr gmx.ch> writes:

On 24.05.20 05:28, Walter Bright wrote:
 I'd like to emphasize:
 

I understand all of those points and most of them are true, and obvious.

The issue is that they are not a justification for the decision. You 
seem to think that greenwashing is not greenwashing when it is done by 
the compiler without user interaction. Why is that?

 1. It is not possible for the compiler to check any declarations where 
 the implementation is not available. Not in D, not in any language. 
 Declaring a declaration safe does not make it safe.
 ...

Which is exactly why it should not be possible to declare it  safe.

 2. If un-annotated declarations cause a compile time error, it is highly 
 likely the programmer will resort to "greenwashing" - just slapping 
  safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've 
 all done it. Sometimes even for good reasons.
 ...

Slapping  safe on it should not even compile. You should slap either 
 system or  trusted on it.

 3. Un-annotated declarations are easily detectable in a code review.
 ...

It's easier to find something that is there than something that is not 
there. Greenwashing is not easier to detect if the compiler did it 
implicitly.

 4. Greenwashing is not easily detectable in a code review.
 ...

Even though it is easy to miss in a code review, it's easy to detect 
automatically. Any extern(C) prototype that is annotated  safe 
(explicitly or implicitly) is greenwashed.

 5. Greenwashing doesn't fix anything. The code is not safer.

Actually further down you say that it makes the code safer in a 
"not-at-all obvious way". Which is it?

 It's an illusion, not a guarantee.
 ...

Yes. On the other hand,  trusted is not an illusion, it is a way to 
clarify responsibilities.

 6. If someone cares to annotate declarations, it means he has at least 
 thought about it, because he doesn't need to.

True, but this is an argument against restrictive defaults in general, 
in particular  safe by default. Also note that if someone cares to 
annotate declarations, the compiler pointing out missing annotations 
that would otherwise cause implicit greenwashing is _useful_.

 Hence it's more likely to be correct than when greenwashed.
 ...

This is true whether or not the compiler does the greenwashing 
implicitly. Annotating with  safe is a lie, whether the compiler does it 
or the programmer. It should be rejected and force  system or  trusted. 
You can still quickly see a difference in applied care by checking 
whether it's a single  trusted: or each prototype is annotated individually.

 7. D should *not* make it worthwhile for people to greenwash code.
 ...

Greenwashing automatically is not a solution, it's admitting defeat. Why 
can't the compiler just reject greenwashing with  safe?
Slapping  trusted on prototypes is not greenwashing, it's saying "I take 
responsibility for the memory safety of this external C code".

 It is, in a not-at-all obvious way, safer for C declarations to default 
 to being safe.

 safe is advertised to give mechanical guarantees, where  trusted is a 
way for programmers to take responsibility for parts of the code. It is 
not advertised to be an unsound linter with pseudo-pragmatic trade-offs 
and implicit false negatives.

Paolo Invernizzi <paolo.invernizzi gmail.com> writes:

On Sunday, 24 May 2020 at 05:43:45 UTC, Timon Gehr wrote:

  safe is advertised to give mechanical guarantees, where 
  trusted is a way for programmers to take responsibility for 
 parts of the code. It is not advertised to be an unsound linter 
 with pseudo-pragmatic trade-offs and implicit false negatives.

And turns back to my previous question, that Walter (or Atila) 
never replied: what I need to reply back to customers asking us 
about  safe.

 safe is for mechanical check or not?

An official and public declaration please.

/P

Walter Bright <newshound2 digitalmars.com> writes:

I infer your position is the idea that putting  trusted on the declarations 
isn't greenwashing, while  safe is.

I can't see a practical difference between:

 safe extern (C) void whatevs(parameters);
 trusted extern (C) void whatevs(parameters);

Both require that whatevs() provide a safe interface. The difference between 
them is in the implementation of those functions, not the interface. Since the
D 
compiler cannot see those implementations, they are immaterial to the compiler 
and user.

Panke <tobias pankrath.net> writes:

On Sunday, 24 May 2020 at 08:55:32 UTC, Walter Bright wrote:
 I infer your position is the idea that putting  trusted on the 
 declarations isn't greenwashing, while  safe is.

 I can't see a practical difference between:

  safe extern (C) void whatevs(parameters);
  trusted extern (C) void whatevs(parameters);

 Both require that whatevs() provide a safe interface. The 
 difference between them is in the implementation of those 
 functions, not the interface. Since the D compiler cannot see 
 those implementations, they are immaterial to the compiler and 
 user.

I've always understood that the  safe, trusted, system machinery 
provides the following guarantee once all holes are fixed:

If I have a memory corruption in my code than I need to only look 
at the  trusted and  system parts to find it.

Marking whatevs  safe violates this, marking it  trusted does not.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/24/2020 2:29 AM, Panke wrote:
 I've always understood that the  safe, trusted, system machinery provides the 
 following guarantee once all holes are fixed:
 
 If I have a memory corruption in my code than I need to only look at the 
  trusted and  system parts to find it.
 
 Marking whatevs  safe violates this, marking it  trusted does not.

It's a fair point, but without the source code the distinction is meaningless.

Stefan Koch <uplink.coder googlemail.com> writes:

On Sunday, 24 May 2020 at 09:47:37 UTC, Walter Bright wrote:
 On 5/24/2020 2:29 AM, Panke wrote:
 I've always understood that the  safe, trusted, system 
 machinery provides the following guarantee once all holes are 
 fixed:
 
 If I have a memory corruption in my code than I need to only 
 look at the  trusted and  system parts to find it.
 
 Marking whatevs  safe violates this, marking it  trusted does 
 not.

 It's a fair point, but without the source code the distinction 
 is meaningless.

The distinction is that you can find a slapped on trusted with a 
grep.
Rather than valgrind.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/24/2020 3:40 AM, Stefan Koch wrote:
 The distinction is that you can find a slapped on trusted with a grep.

It's just as easy to use grep to *not* find  trusted.

Johannes Loher <johannes.loher fg4f.de> writes:

Am 27.05.20 um 11:25 schrieb Walter Bright:
 On 5/24/2020 3:40 AM, Stefan Koch wrote:
 The distinction is that you can find a slapped on trusted with a grep.

 
 It's just as easy to use grep to *not* find  trusted.

But that's not enough. You need a regexp that searches for extern
(C(++)) declarations that do not have any of  safe,  trusted,  system.
The attributes can also be either before the return type + name +
parameters or after it. They can also be mixed with any other
attributes. Sure, you can probably write a regex that matches all of
this but it is a _lot_ more complicated than simply searching for  trusted.

rikki cattermole <rikki cattermole.co.nz> writes:

On 27/05/2020 10:12 PM, Johannes Loher wrote:
 Am 27.05.20 um 11:25 schrieb Walter Bright:
 On 5/24/2020 3:40 AM, Stefan Koch wrote:
 The distinction is that you can find a slapped on trusted with a grep.

 It's just as easy to use grep to *not* find  trusted.

 
 But that's not enough. You need a regexp that searches for extern
 (C(++)) declarations that do not have any of  safe,  trusted,  system.
 The attributes can also be either before the return type + name +
 parameters or after it. They can also be mixed with any other
 attributes. Sure, you can probably write a regex that matches all of
 this but it is a _lot_ more complicated than simply searching for  trusted.

extern(Windows)
extern(System)

COM

Few more things there.

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On Sunday, 24 May 2020 at 09:47:37 UTC, Walter Bright wrote:
 It's a fair point, but without the source code the distinction 
 is meaningless.

It's meaningless in terms of what the compiler can check, but 
it's not meaningless in terms of documenting the assumptions and 
promises the developer is making.

Johannes Loher <johannes.loher fg4f.de> writes:

On Sunday, 24 May 2020 at 08:55:32 UTC, Walter Bright wrote:
 I infer your position is the idea that putting  trusted on the 
 declarations isn't greenwashing, while  safe is.

 I can't see a practical difference between:

  safe extern (C) void whatevs(parameters);
  trusted extern (C) void whatevs(parameters);

 Both require that whatevs() provide a safe interface. The 
 difference between them is in the implementation of those 
 functions, not the interface. Since the D compiler cannot see 
 those implementations, they are immaterial to the compiler and 
 user.

 From my perspective, there is a clear difference in meaning 
between them:  safe means verified by the compiler (which as you 
mentioned can never be true for declarations where the code is 
not available, hence it always is a lie in that case) and 
 trusted means verified by the developer.

Whenever  trusted is slapped on _anything_, it is clear that 
extra caution is needed (in codereviews etc.) and as others have 
mentioned, it is easily searchable.

When I put  safe on a piece of code, my expectation is that it is 
actually verified by the compiler. If that is not possible, it 
should not compile.

The same reasoning also applies to the case where the annotations 
are not added by the developer explicitly and the defaults are 
used instead:

If the default is  safe and the code for an unannotated 
declaration is not available, it should not compile.

Making  trusted the default is not an option because as mentioned 
earlier, one of the points of  trusted is for it to be actually 
seen, which is not the case if it is the default, so I will not 
elaborate on that.

If  sytem is the default, there is no issue but of course the 
whole point of this DIP is to move away from that.

I understand that today it is possible to slap  safe on 
declarations without code without the compiler complaining but it 
really _should_ be an error in order to have a consistent meaning 
of  safe,  trusted and  system ( safe = compiler verified, 
 trusted = user verified,  system = not verified) also in the 
case of declarations where no code is available.

Let's assume this is indeed the way to go, then there are 2 
possible solutions for this DIP regarding how to handle this:

1. Make declarations without code default to  system instead of 
 safe. This is what many here have been arguing for but as you 
mentioned several times, it has the drawback of adding special 
cases to the language. As a user, it does not seem that 
problematic though because the compiler can easily tell you what 
is going on when there is an issue (i.e. calling an unannotated 
declaration without code from a  safe function could result in 
compiler error message that explains why this does not work). But 
I admit that it is still a bit weird to have 2 different 
defaults. Additionally, explicitly annotating such declarations 
with  safe should be a compile error.

2. Make  safe the default for _all_ declarations but still make 
 safe on declarations without code a compile error (because the 
compiler cannot verify it). This means that annotating function 
declarations without code with  system or  trusted is now 
mandatory. This is what Timon has been arguing for if I 
understood him correctly.

3. There may be a third option if we introduce something like 
conditional safety but I do not completely understand that yet. 
This is what H.S. Theo has been suggesting.

Option 1 and 2 both have the „issue“ that people might 
„greenwash“ things by simply slapping  trusted: at the top of a 
file. But that can always be done anyways and at least it is 
explicit and searchable. As mentioned, using  trusted now has the 
meaning that it is user verified so it always needs extra caution.

Personally I prefer option 2 because it is a lot more consistent: 
There are no special cases.

Until now, I did not discuss whether or not all of this should 
apply only to extern(C) function declarations without code or all 
function declarations without code. This is because it is a 
separate point. Both are possible and have pros and cons:

If it applies to extern(C) function declarations without code 
only, then we can still have  safe extern(D) declarations. They 
are not verified by the compiler but you get linker errors 
instead because  safe is part of the mangling. The benefit is 
that you get more  safe code by default, the drawback is that it 
relies on linker errors instead of compiler errors and that it 
introduces a special case.

If it applies to all function declarations without code, also 
extern(D) function declarations without bodies need to be 
annotated with  trusted explicitly if they are  safe or  trusted. 
If they are  system, we still get linker errors due to name 
mangling. The benefit is that it is a very consistent solution, 
no special cases are needed. The drawback is that it requires 
more manual effort to add the additional annotations (but then, 
don't we want to encourage people to explicitly annotate things 
anyways...?).

Personally, I prefer the second option because of its simplicity. 
Also you mentioned somewhere that linker errors are not that 
helpful to the user and I couldn’t agree more. Also I personally 
don't use extern(D) function declarations without bodies a lot, 
so the drawback is negligible for me. Others' experiences might 
be different though.

Steven actually made a proposal regarding creating 2 different 
manglings for extern(C) functions that are implemented in D. 
Regardless of which of the solutions  is taken, this could 
provide the same benefits that we have for extern(D) functions 
(linker errors if  safety does not match in the mangling). 
However, it sounds like a complicated solution (in an answer to 
him, you already mentioned that there might be technical 
difficulties regarding some object formats, debugging symbols, 
etc.) and I am not sure it's worth it. It also makes swapping out 
the libraries a bit weird: if you use an actual C library, it 
will always link but if swap to a library implemented in D, it 
only links if the  saftey mangling matches.


Coming back to the original point: I think it is really important 
that we give  safe,  trusted and  sytem their clear meaning (as 
explained above) also for declarations without bodies. In my 
opinion, we should choose the second option in both of the 
questions I presented.

aliak <something something.com> writes:

On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 does not work). But I admit that it is still a bit weird to 
 have 2 different defaults.

Is that any more or less weirder than having functions inferred 
with different attributes based on context?

Johannes Loher <johannes.loher fg4f.de> writes:

On Sunday, 24 May 2020 at 11:25:06 UTC, aliak wrote:
 On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 does not work). But I admit that it is still a bit weird to 
 have 2 different defaults.

 Is that any more or less weirder than having functions inferred 
 with different attributes based on context?

What exactly are you referring to?

aliak <something something.com> writes:

On Sunday, 24 May 2020 at 11:30:53 UTC, Johannes Loher wrote:
 On Sunday, 24 May 2020 at 11:25:06 UTC, aliak wrote:
 On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 does not work). But I admit that it is still a bit weird to 
 have 2 different defaults.

 Is that any more or less weirder than having functions 
 inferred with different attributes based on context?

 What exactly are you referring to?

Attribute inference by D, specifically template functions. The 
attributes are inferred based on context (I don't know the exact 
algorithm). So a function f(T)(T) when called can maybe be pure, 
maybe safe, maybe not?

Johannes Loher <johannes.loher fg4f.de> writes:

On Sunday, 24 May 2020 at 12:14:13 UTC, aliak wrote:
 On Sunday, 24 May 2020 at 11:30:53 UTC, Johannes Loher wrote:
 On Sunday, 24 May 2020 at 11:25:06 UTC, aliak wrote:
 On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 does not work). But I admit that it is still a bit weird to 
 have 2 different defaults.

 Is that any more or less weirder than having functions 
 inferred with different attributes based on context?

 What exactly are you referring to?

 Attribute inference by D, specifically template functions. The 
 attributes are inferred based on context (I don't know the 
 exact algorithm). So a function f(T)(T) when called can maybe 
 be pure, maybe safe, maybe not?

 From what I understand, it does not depend on the context but on 
the template parameters you pass to the template. I agree that it 
might be a bit confusing at first, but it makes sense if you 
realize that templates themselves are not functions but something 
that can generate functions (and other constructs) from compile 
time parameters. Why shouldn't the attributes of a generated 
function be able to depend on the parameters being passed to the 
template? Basically everything else can depend on them, too. 
Automatically inferring the attributes is just a very convenient 
way to do that. But yeah, it's not 100% consistent that they are 
not also inferred for regular functions (some people have been 
arguing for that).

However, at least for templates, there is a very good reason for 
the difference (or at least the fact that attributes are inferred 
for templates): if that was not the case, it would basically be 
impossible to write generic code that works with all the 
attribute combinations. However, the very purpose of templates is 
to enable writing generic code. They wouldn't make that much 
sense if that capability was strongly limited.

On the other hand, having different  safety defaults for bodiless 
function declarations and regular faction declarations does not 
have such a big benefit, especially when considering the fact, 
that we can have the same defaults but make  safe (default or 
explicit) bodiless function declarations a compiler error. If we 
ignore that option for some reason, it would only be dort of a 
necessity in order to prevent people from shooting them selves in 
the foot without even noticing. But there is no inherent value in 
the difference, it doesn't enable anything.

That said, I'd still prefer this variant over what DIP1028 does 
currently. It's just that I think the other option is even better 
because it is more consistent.

Steven Schveighoffer <schveiguy gmail.com> writes:

On 5/24/20 6:40 AM, Johannes Loher wrote:
 Steven actually made a proposal regarding creating 2 different manglings 
 for extern(C) functions that are implemented in D. Regardless of which 
 of the solutions  is taken, this could provide the same benefits that we 
 have for extern(D) functions (linker errors if  safety does not match in 
 the mangling). However, it sounds like a complicated solution (in an 
 answer to him, you already mentioned that there might be technical 
 difficulties regarding some object formats, debugging symbols, etc.) and 
 I am not sure it's worth it. It also makes swapping out the libraries a 
 bit weird: if you use an actual C library, it will always link but if 
 swap to a library implemented in D, it only links if the  saftey 
 mangling matches.

I don't think the technical difficulties make it impossible. If you 
can't point at the same address, point at a noop/jmp before the real 
address. This isn't hard.

In terms of actual C libraries, only the unsafe symbol will be defined 
(naturally), so marking the extern(C) function  safe (or assuming it's 
safe by default) is going to fail to link.

Note that "linker errors if  safety does not match" is not entirely 
accurate. This is ONLY the case if the prototype is  safe and the 
implementation is not. In all other cases, the program will link.

Next, I'd say that linker errors are reasonable for C functions -- they 
are not mangled, so the name is obvious (for extern(C) safe functions I 
would propose a really simple mangling like _d_safe_functionname). Plus, 
users who declare extern(C) prototypes already have to deal with linker 
errors because they have to name their prototypes correctly.

But I don't anticipate anyone taking up this idea (I lack the skills), 
it already seems DOA based on Walter's position.

-Steve

Juraj Mojzis <junk vec4.xyz> writes:

On Sunday, 24 May 2020 at 10:40:11 UTC, Johannes Loher wrote:
 2. Make  safe the default for _all_ declarations but still make 
  safe on declarations without code a compile error (because the 
 compiler cannot verify it). This means that annotating function 
 declarations without code with  system or  trusted is now 
 mandatory. This is what Timon has been arguing for if I 
 understood him correctly.

If this DIP is going to be implemented "as is", this should be 
added in the future.
It could also be a warning, but that is not something that D 
often does.
With this addition, not having a special case for extern(), is a 
positive.

ag0aep6g <anonymous example.com> writes:

On 24.05.20 10:55, Walter Bright wrote:
 I infer your position is the idea that putting  trusted on the 
 declarations isn't greenwashing, while  safe is.

That's just arguing semantics. Your intended meaning of "greenwashing" 
is apparently slightly different from Timon's.

You can call this greenwashing:

      trusted extern (C) void whatevs(parameters);

And this:

      safe extern (C) void whatevs(parameters);

And then you must also call this greenwashing when compiled with 
`-preview=safedefault`:

     extern (C) void whatevs(parameters);

They all do the same thing.

But the first one is the most preferable form, because it shows up when 
you search for " trusted" which is our "here be dragons" sign.

The second one should be deprecated in favor of the first one. That 
would move us one baby step closer to the ideal: " safe code is memory 
safe, unless someone made a mistake with  trusted."

The third one moves us one step further away from that ideal. That's the 
wrong direction. It's greenwashing by default. And since greenwashing is 
bad, greenwashing by default means bad by default.

Adam D. Ruppe <destructionator gmail.com> writes:

On Sunday, 24 May 2020 at 08:55:32 UTC, Walter Bright wrote:
 I can't see a practical difference between:

  safe extern (C) void whatevs(parameters);
  trusted extern (C) void whatevs(parameters);

 Both require that whatevs() provide a safe interface.

Remember that D has reflection. If we ever do a reflection check 
on these,  trusted stands out better there than  safe. Of course 
it could also just return null in reflection meaning "default 
applied" and that could be detected as well.

Otherwise it is the same, yes, but this reflection thing is a 
small nice benefit.

Timon Gehr <timon.gehr gmx.ch> writes:

On 24.05.20 10:55, Walter Bright wrote:
 I infer your position is the idea that putting  trusted on the 
 declarations isn't greenwashing, while  safe is.
 ...

It's only greenwashing if it's misleading. Putting  safe is a lie, 
putting  trusted is honest.

 I can't see a practical difference between:
 
  safe extern (C) void whatevs(parameters);
  trusted extern (C) void whatevs(parameters);
 
 Both require that whatevs() provide a safe interface. The difference 
 between them is in the implementation of those functions, not the 
 interface. Since the D compiler cannot see those implementations, they 
 are immaterial to the compiler and user.

Sure, that's the point. Your  safe by default DIP in practice makes 
certain declarations  trusted by default.  safe is a fine default. 
 trusted is a horrible default. That's why your DIP claims it is for 
 safe by default (and not  trusted by default). Except in this one weird 
special case, where it introduces  trusted by default.

Johannes T <isrvoid gmail.com> writes:

On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote:
 [..]

After thinking about it, Walter ultimately made the right 
decision, leading to overall higher safety and code quality.
We all agree that making extern C  safe is incorrect. It's also 
meaningless. Even if you were to verify the safety of a specific 
version of your binding, it can't be known what's loaded at 
runtime. It's not the compiler's concern.  safe extern shall be 
an error. We might additionally make an exception and make all 
extern C  system. It would be correct for the declarations, but 
inconsistent in regard to default safety. It doesn't affect the 
outcome. Let's say we'll go with  system, it gives us a bit more 
freedom.
We hit compile, our now safe wrappers are errors. We are most 
likely to do one of two things, depending on the module failing 
to compile. If it mostly wrappers, we slap  trusted: at the top. 
If there are just a few functions calling C, we only annotate 
those with  trusted. Let's be real, we probably won't begin 
checking and annotating the trustworthiness of the C functions. 
An individual programmer might, but not on average.
We are formally correct, but  trusted can't be trusted at this 
point. It has lost its meaning. We now have to check all foreign 
 trusted code, which we probably won't.
We could have moved the problem one level down and slapped 
 trusted: on top of the  system declarations. Now there is a 
bunch of safe code using them. It doesn't change much.  trusted: 
is easier to grep, but we won't put everything down and begin 
trustworthily annotate extern C. So yeah, I do believe Walter was 
right.  safe on extern is formally incorrect but leads to safer 
code because  trusted still has its power.

Johannes Loher <johannes.loher fg4f.de> writes:

On Monday, 25 May 2020 at 09:25:52 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote:
 [..]

 After thinking about it, Walter ultimately made the right 
 decision, leading to overall higher safety and code quality.
 We all agree that making extern C  safe is incorrect. It's also 
 meaningless. Even if you were to verify the safety of a 
 specific version of your binding, it can't be known what's 
 loaded at runtime. It's not the compiler's concern.  safe 
 extern shall be an error. We might additionally make an 
 exception and make all extern C  system. It would be correct 
 for the declarations, but inconsistent in regard to default 
 safety. It doesn't affect the outcome. Let's say we'll go with 
  system, it gives us a bit more freedom.
 We hit compile, our now safe wrappers are errors. We are most 
 likely to do one of two things, depending on the module failing 
 to compile. If it mostly wrappers, we slap  trusted: at the 
 top. If there are just a few functions calling C, we only 
 annotate those with  trusted. Let's be real, we probably won't 
 begin checking and annotating the trustworthiness of the C 
 functions. An individual programmer might, but not on average.
 We are formally correct, but  trusted can't be trusted at this 
 point. It has lost its meaning. We now have to check all 
 foreign  trusted code, which we probably won't.
 We could have moved the problem one level down and slapped 
  trusted: on top of the  system declarations. Now there is a 
 bunch of safe code using them. It doesn't change much. 
  trusted: is easier to grep, but we won't put everything down 
 and begin trustworthily annotate extern C. So yeah, I do 
 believe Walter was right.  safe on extern is formally incorrect 
 but leads to safer code because  trusted still has its power.

So basically you are saying we should do it the way it is 
described in the DIP because otherwise it will lead developers to 
incorrect usage of  trusted (just slapping it on declarations 
whiteout actually checking the implementation) thus making it 
loose its „power“ which leads to less safe code.

But with the DIP in its current form, we make  safe lose its 
meaning and power, which is much worse in my opinion. It makes 
the statement „ safe code cannot create memory corruptions except 
for  trusted code“ wrong (it already is wrong now but it really 
shouldn’t be!).

What you are describing is really just an admit of defeat. Then 
why still bother with  safe at all if it doesn't give us any 
meaningful guarantees? I don't think we are at that point (yet).

Johannes T <isrvoid gmail.com> writes:

On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote:
 [..]
 But with the DIP in its current form, we make  safe lose its 
 meaning and power, which is much worse in my opinion.
 [..]

The alternative, not making extern  safe, would result in more 
untrustworthy  trusted code we have to worry about. It's a 
vicious circle.
I try to relax my view on extern annotations. They are  system. 
We *should* go ahead and diligently mark with  trusted. From 
experience, it doesn't normally happen.
I don't like  safe extern, but it seems like the lesser evil. 
Walter got a lot of flak. I tried to retrace his thoughts and see 
the merits.

Zoadian <no no.no> writes:

On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote:
 [..]
 But with the DIP in its current form, we make  safe lose its 
 meaning and power, which is much worse in my opinion.
 [..]

 The alternative, not making extern  safe, would result in more 
 untrustworthy  trusted code we have to worry about. It's a 
 vicious circle.
 I try to relax my view on extern annotations. They are  system. 
 We *should* go ahead and diligently mark with  trusted. From 
 experience, it doesn't normally happen.
 I don't like  safe extern, but it seems like the lesser evil. 
 Walter got a lot of flak. I tried to retrace his thoughts and 
 see the merits.

there is no such thing as a trustworthy  trusted. not with how 
the whole safety system work now.
you can break previously verified  trusted code by just writing 
 safe code today.

Panke <tobias pankrath.net> writes:

On Monday, 25 May 2020 at 12:22:25 UTC, Zoadian wrote:
 there is no such thing as a trustworthy  trusted. not with how 
 the whole safety system work now.
 you can break previously verified  trusted code by just writing 
  safe code today.

Do you have an example of code that was correctly labeled 
 trusted in DMD version X, which may not be labeled  trusted in 
DMD version Y because of changes to  safe/ trusted / system?

Johannes T <isrvoid gmail.com> writes:

On Monday, 25 May 2020 at 12:22:25 UTC, Zoadian wrote:
 [..]
 there is no such thing as a trustworthy  trusted.
 [..]

Sorry, my bad. I meant the quality of  trusted annotations would 
decline.

Johannes Loher <johannes.loher fg4f.de> writes:

On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote:
 [..]
 But with the DIP in its current form, we make  safe lose its 
 meaning and power, which is much worse in my opinion.
 [..]

 The alternative, not making extern  safe, would result in more 
 untrustworthy  trusted code we have to worry about. It's a 
 vicious circle.
 I try to relax my view on extern annotations. They are  system. 
 We *should* go ahead and diligently mark with  trusted. From 
 experience, it doesn't normally happen.
 I don't like  safe extern, but it seems like the lesser evil. 
 Walter got a lot of flak. I tried to retrace his thoughts and 
 see the merits.

 From my perspective it is really simple: Either we have a strict 
 safety system (with exactly one escape hatch) or we don't. At 
the moment we don't because you can have  safe extern(C) 
declarations but with this DIP it becomes even worse as it is 
more likely that  safe promises are broken without anybody 
noticing.

If you and more importantly Walter and Atila really think this is 
the lesser evil and that it is worth dropping the strict  safety 
system with exactly one controlled escape hatch for this, then we 
also need to clarify what  safe means after that. We can 
definitely not claim that it means machine verified except for 
 trusted because extern(C) functions are another escape hatch. 
The question has been asked in this thread a few times already: 
What does  safe actually mean? How to „sell“ / „advertise“ it?

Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:

On Monday, 25 May 2020 at 11:40:46 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 10:19:22 UTC, Johannes Loher wrote:
 [..]
 But with the DIP in its current form, we make  safe lose its 
 meaning and power, which is much worse in my opinion.
 [..]

 The alternative, not making extern  safe, would result in more 
 untrustworthy  trusted code we have to worry about. It's a 
 vicious circle.

Wrong. The quantity of untrustworthy code remains the same, but 
with DIP1028 (at least in the current form) the compiler sweeps 
the previously  system code under the rug and makes it harder for 
those who care about safety to trust  safe.

 safe must mean only one thing: compiler verified. Or otherwise 
needing less manual review.  system and  trusted means that code 
review should be prioritized.  safe non-extern (D) marked either 
by the programmer, or the implicitly by the compiler should be 
disallowed as it *is* greenwashing.

 I try to relax my view on extern annotations. They are  system. 
 We *should* go ahead and diligently mark with  trusted. From 
 experience, it doesn't normally happen.

It didn't happen, because it didn't need to. Naturally, most 
things go through the path of least resistance. Most developers 
are coming from other languages where they have never had the 
requirement to write  safe code [external pressure]. Also 
previously,  safe wasn't the default for D function definitions, 
so there was less [internal pressure] to do so. With  safe being 
the default of function definitions, it's more difficult to leave 
code as  system (of course, modulo  trusted).

 I don't like  safe extern, but it seems like the lesser evil.

No,  safe extern is the worst possible option! It basically makes 
 safe meaningless.

 Walter got a lot of flak. I tried to retrace his thoughts and 
 see the merits.

On several occasions (e.g. on Reddit) I have defended Walter from 
unfair accusations, however, in this case, he's rightfully 
criticized.
He seems to think that he's taking an unpopular decision for the 
greater good, but that's not the case.  safe-by-default on D 
function definitions could be considered an unpopular decision 
for the greater good. Implicitly  safe non-extern(D) functions is 
greenwashing, where the responsibility for the action is removed 
from the developer by a compiler switch. That's basically 
negating all the benefits of  safe-by-default on function 
definitions.

Zoadian <no no.no> writes:

On Monday, 25 May 2020 at 09:25:52 UTC, Johannes T wrote:
 On Monday, 25 May 2020 at 00:56:04 UTC, Timon Gehr wrote:
 [..]

 After thinking about it, Walter ultimately made the right 
 decision, leading to overall higher safety and code quality.
 We all agree that making extern C  safe is incorrect. It's also 
 meaningless. Even if you were to verify the safety of a 
 specific version of your binding, it can't be known what's 
 loaded at runtime. It's not the compiler's concern.  safe 
 extern shall be an error. We might additionally make an 
 exception and make all extern C  system. It would be correct 
 for the declarations, but inconsistent in regard to default 
 safety. It doesn't affect the outcome. Let's say we'll go with 
  system, it gives us a bit more freedom.
 We hit compile, our now safe wrappers are errors. We are most 
 likely to do one of two things, depending on the module failing 
 to compile. If it mostly wrappers, we slap  trusted: at the 
 top. If there are just a few functions calling C, we only 
 annotate those with  trusted. Let's be real, we probably won't 
 begin checking and annotating the trustworthiness of the C 
 functions. An individual programmer might, but not on average.
 We are formally correct, but  trusted can't be trusted at this 
 point. It has lost its meaning. We now have to check all 
 foreign  trusted code, which we probably won't.
 We could have moved the problem one level down and slapped 
  trusted: on top of the  system declarations. Now there is a 
 bunch of safe code using them. It doesn't change much. 
  trusted: is easier to grep, but we won't put everything down 
 and begin trustworthily annotate extern C. So yeah, I do 
 believe Walter was right.  safe on extern is formally incorrect 
 but leads to safer code because  trusted still has its power.

you complain about  trusted losing it's meaning, but  safe was 
ment to mean "mechanically verified memory safety". it should be 
forbidden to add  safe to any function that can not be verified 
by the compiler.

the compiler should do it's best to alert you of any mistakes you 
might make.
if the compiler does silently add  safe or  trusted to extern(C) 
functions you are are to miss annotating it. that is a bad thing 
for everyone that actually cares about safety.

you can't prevent programmers from doing stupid things. but you 
can at least warn them.

rikki cattermole <rikki cattermole.co.nz> writes:

On 25/05/2020 10:29 PM, Zoadian wrote:
 you complain about  trusted losing it's meaning, but  safe was ment to 
 mean "mechanically verified memory safety". it should be forbidden to 
 add  safe to any function that can not be verified by the compiler.

It is meant to mean that at some point it has been mechanically checked 
by the compiler.

Either during current compilation or a prior one.

Which means it has to be valid on function declarations without bodies 
so i.e. .di file generation works correctly which is just a generated D 
file, nothing special syntax of semantics wise.

Zoadian <no no.no> writes:

On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote:
 On 25/05/2020 10:29 PM, Zoadian wrote:
 you complain about  trusted losing it's meaning, but  safe was 
 ment to mean "mechanically verified memory safety". it should 
 be forbidden to add  safe to any function that can not be 
 verified by the compiler.

 It is meant to mean that at some point it has been mechanically 
 checked by the compiler.

 Either during current compilation or a prior one.

 Which means it has to be valid on function declarations without 
 bodies so i.e. .di file generation works correctly which is 
 just a generated D file, nothing special syntax of semantics 
 wise.

.di files _could_ just use  trusted instead of  safe. but for 
extern(D) we could at least add it to the name mangling. it's 
still not 100% safe, but at least you'd have to work hard to get 
it wrong.

Paul Backus <snarwin gmail.com> writes:

On Monday, 25 May 2020 at 12:30:11 UTC, Zoadian wrote:
 On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote:
 It is meant to mean that at some point it has been 
 mechanically checked by the compiler.

 Either during current compilation or a prior one.

 Which means it has to be valid on function declarations 
 without bodies so i.e. .di file generation works correctly 
 which is just a generated D file, nothing special syntax of 
 semantics wise.

 .di files _could_ just use  trusted instead of  safe. but for 
 extern(D) we could at least add it to the name mangling. it's 
 still not 100% safe, but at least you'd have to work hard to 
 get it wrong.

It's been proposed before that  safe and  trusted should have the 
same mangling, since there's no difference between them from the 
calling code's perspective.

Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:

On Monday, 25 May 2020 at 12:41:01 UTC, Paul Backus wrote:
 On Monday, 25 May 2020 at 12:30:11 UTC, Zoadian wrote:
 On Monday, 25 May 2020 at 10:41:43 UTC, rikki cattermole wrote:
 It is meant to mean that at some point it has been 
 mechanically checked by the compiler.

 Either during current compilation or a prior one.

 Which means it has to be valid on function declarations 
 without bodies so i.e. .di file generation works correctly 
 which is just a generated D file, nothing special syntax of 
 semantics wise.

 .di files _could_ just use  trusted instead of  safe. but for 
 extern(D) we could at least add it to the name mangling. it's 
 still not 100% safe, but at least you'd have to work hard to 
 get it wrong.

 It's been proposed before that  safe and  trusted should have 
 the same mangling, since there's no difference between them 
 from the calling code's perspective.

It may be true (of course modulo meta-programming) that it 
doesn't make a difference for the calling code, but I personally 
want have the guarantees that a function that I'm calling is 
truly  safe (it doesn't contain or call any  trusted code, 
transitively, nor it calls any  safe code, which access global 
variables initialized by  system static/module constructors).

In my line work (blockchain smart contracts) some of the ways of 
how this is typically achieved include:
* having a very minimal smart contract code size
* having either no third-party dependencies or using one or two 
which are open-source and more importantly verified by multiple 
teams and having very high reputation
* extensive code auditing by third-party teams. Depending on the 
circumstances, we may end up paying more for the auditing of the 
code, than the actual development.

That said, there is no "strong"- safe today and even if there 
was, it would account for a tiny subset of all attack vectors 
that I have to care about (basically all possible logical bugs 
allowed in type-safe and memory-safe code), but I'm not sure how 
erasing the difference between  safe and  trusted on the 
interface level would help.

Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:

On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov [ZombineDev] 
wrote:
 It may be true (of course modulo meta-programming) that it 
 doesn't make a difference for the calling code, but I 
 personally want have the guarantees that a function that I'm

doesn't make a difference for the calling code, but personally
I want [to] have the guarantees that a function that I'm

 calling is truly  safe (it doesn't contain or call any  trusted 
 code, transitively, nor it calls any  safe code, which access 
 global variables initialized by  system static/module 
 constructors).

This is very far from a rigorous definition of "strong  safe-ty" 
- but I hope it's just enough for the casual reader to understand 
my intention.

 In my line work (blockchain smart contracts) some of the ways 
 of how this is typically achieved include:
 * having a very minimal smart contract code size
 * having either no third-party dependencies or using one or two 
 which are open-source and more importantly verified by multiple 
 teams and having very high reputation
 * extensive code auditing by third-party teams. Depending on 
 the circumstances, we may end up paying more for the auditing 
 of the code, than the actual development.

 That said, there is no "strong"- safe today and even if there

That said, there is no "strong- safe" [in D] today and even if 
there

 was, it would account for a tiny subset of all attack vectors 
 that I have to care about (basically all possible logical bugs 
 allowed in type-safe and memory-safe code), but I'm not sure 
 how erasing the difference between  safe and  trusted on the 
 interface level would help.

Paul Backus <snarwin gmail.com> writes:

On Monday, 25 May 2020 at 13:22:36 UTC, Petar Kirov [ZombineDev] 
wrote:
 On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov 
 [ZombineDev] wrote:
 It may be true (of course modulo meta-programming) that it 
 doesn't make a difference for the calling code, but I 
 personally want have the guarantees that a function that I'm

 doesn't make a difference for the calling code, but personally
 I want [to] have the guarantees that a function that I'm

 calling is truly  safe (it doesn't contain or call any 
  trusted code, transitively, nor it calls any  safe code, 
 which access global variables initialized by  system 
 static/module constructors).

 This is very far from a rigorous definition of "strong 
  safe-ty" - but I hope it's just enough for the casual reader 
 to understand my intention.

I'm sure this is reasonable for your use-case, but I hope you can 
recognize that this definition of safety is far too narrow to be 
suitable for a general-purpose programming language (which D 
purports to be). Most people would like their  safe code to be 
able to do I/O, for example, despite the fact that it necessarily 
involves calling  system code under the hood.

Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:

On Monday, 25 May 2020 at 13:43:07 UTC, Paul Backus wrote:
 On Monday, 25 May 2020 at 13:22:36 UTC, Petar Kirov 
 [ZombineDev] wrote:
 On Monday, 25 May 2020 at 13:14:51 UTC, Petar Kirov 
 [ZombineDev] wrote:
 It may be true (of course modulo meta-programming) that it 
 doesn't make a difference for the calling code, but I 
 personally want have the guarantees that a function that I'm

 doesn't make a difference for the calling code, but personally
 I want [to] have the guarantees that a function that I'm

 calling is truly  safe (it doesn't contain or call any 
  trusted code, transitively, nor it calls any  safe code, 
 which access global variables initialized by  system 
 static/module constructors).

 This is very far from a rigorous definition of "strong 
  safe-ty" - but I hope it's just enough for the casual reader 
 to understand my intention.

 I'm sure this is reasonable for your use-case, but I hope you 
 can recognize that this definition of safety is far too narrow 
 to be suitable for a general-purpose programming language 
 (which D purports to be). Most people would like their  safe 
 code to be able to do I/O, for example, despite the fact that 
 it necessarily involves calling  system code under the hood.

I don't want to change the definition of  safe in D, but would 
rather like if D supported  strongSafe, that interested people 
like me could opt into.
I know that worded like this it may sound like too narrow feature 
to add to the language (or at least not having favorable 
complexity/use cases ratio).
So instead, I'd like to have transitive UDAs [1], a feature that 
has been requested by many, for various use cases ;)

[1]: Basically I want to be able to implement function attributes 
like  nogc or nothrow in user-space, but that's a long way from 
now, as first, we need to be able to introspect function bodies.

Paul Backus <snarwin gmail.com> writes:

On Monday, 25 May 2020 at 15:07:19 UTC, Petar Kirov [ZombineDev] 
wrote:
 I don't want to change the definition of  safe in D, but would 
 rather like if D supported  strongSafe, that interested people 
 like me could opt into.
 I know that worded like this it may sound like too narrow 
 feature to add to the language (or at least not having 
 favorable complexity/use cases ratio).
 So instead, I'd like to have transitive UDAs [1], a feature 
 that has been requested by many, for various use cases ;)

 [1]: Basically I want to be able to implement function 
 attributes like  nogc or nothrow in user-space, but that's a 
 long way from now, as first, we need to be able to introspect 
 function bodies.

Sounds to me like a great use-case for the 
DMD-frontend-as-a-library project.

Timon Gehr <timon.gehr gmx.ch> writes:

On 25.05.20 11:25, Johannes T wrote:
  trusted can't be trusted

That's the point of  trusted. ._.

The code is trusted by the programmer, not the annotation by the compiler.

Johannes T <isrvoid gmail.com> writes:

On Monday, 25 May 2020 at 11:52:27 UTC, Timon Gehr wrote:
 On 25.05.20 11:25, Johannes T wrote:
  trusted can't be trusted

 That's the point of  trusted. ._.

 The code is trusted by the programmer, not the annotation by 
 the compiler.

Sorry, I phrased it poorly. I meant  trusted would be used more 
frequently. The focus would spread and lead to less rigorous 
checks.

Timon Gehr <timon.gehr gmx.ch> writes:

On 25.05.20 14:22, Johannes T wrote:
 On Monday, 25 May 2020 at 11:52:27 UTC, Timon Gehr wrote:
 On 25.05.20 11:25, Johannes T wrote:
  trusted can't be trusted

 That's the point of  trusted. ._.

 The code is trusted by the programmer, not the annotation by the 
 compiler.

 
 Sorry, I phrased it poorly. I meant  trusted would be used more 
 frequently. The focus would spread and lead to less rigorous checks.

This is just not true. If the compiler forces you to decide to put 
either  trusted or  system, then you will not get more usage of  trusted 
than if it implicitly decides for you that what you want is implicit 
 trusted.

ag0aep6g <anonymous example.com> writes:

On 25.05.20 14:22, Johannes T wrote:
 Sorry, I phrased it poorly. I meant  trusted would be used more 
 frequently. The focus would spread and lead to less rigorous checks.

The focus is as wide as it's ever been. If you want to verify a program, 
you have to check those prototypes, whether they're explicitly  trusted 
or implicitly  safe.  safe-by-default only makes them harder to find.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/24/2020 5:56 PM, Timon Gehr wrote:
 It's only greenwashing if it's misleading. Putting  safe is a lie, putting 
  trusted is honest.

It is not honest unless the programmer actually carefully examined the
interface 
and the documentation to determine if it is a safe interface or not. For 
example, labeling memcpy() with  trusted is not honest.

Forcing people to add uncheckable annotations is a path to convenience, not
honesty.

Johannes Loher <johannes.loher fg4f.de> writes:

On Tuesday, 26 May 2020 at 01:16:49 UTC, Walter Bright wrote:
 On 5/24/2020 5:56 PM, Timon Gehr wrote:
 It's only greenwashing if it's misleading. Putting  safe is a 
 lie, putting  trusted is honest.

 It is not honest unless the programmer actually carefully 
 examined the interface and the documentation to determine if it 
 is a safe interface or not. For example, labeling memcpy() with 
  trusted is not honest.

 Forcing people to add uncheckable annotations is a path to 
 convenience, not honesty.

This has already been repeated over and over again but I'll do it 
once more:

If people cannot check the code (or get enough confidence in 
another way), they really shouldn’t annotate it with  trusted but 
with  system instead. If they annotate it with  trusted anyways, 
it is their own fault. And at least in that case they are lying 
and not the compiler. Also if something bad actually happens, 
they can at least search for „high risk areas“ ( trusted 
declarations).

Just consider the following situation:

A (maybe inexperienced) programmer needs to call an existing C 
function from D code. Because no bindings exist yet, he creates 
them himself. He does not add any annotations regarding safety. 
Now he uses this function in  safe code and he uses it in a way 
that actually creates a memory corruption (the function is not  
safe, but the developer does not know about that).

Now let's compare the two different options:

1. With DIP1028 in its current form, the code will compile and a 
memory corruption will actually happen. The problem might be 
extremely difficult to track down for the developer because he 
has no clues whatsoever where to start looking.

2. With one of the solutions that were presented, the code will 
not compile as it is. According to your argument of 
„convenience“, the developer will probably just mark the function 
incorrectly as  trusted which makes the code compile. The memory 
corruption will happen. However, even if the developer did not 
think much about potential safety issues when adding  trusted to 
the function, he now still remembers that he did that (it was a 
conscious decision, even if it was a careless and lazy one). He 
has a clear point to start looking for the reason of the memory 
corruption.

Do you honestly think option 1 is better?

Walter Bright <newshound2 digitalmars.com> writes:

On 5/25/2020 7:04 PM, Johannes Loher wrote:
 Now let's compare the two different options:
 
 1. With DIP1028 in its current form, the code will compile and a memory 
 corruption will actually happen. The problem might be extremely difficult to 
 track down for the developer because he has no clues whatsoever where to start 
 looking.
 
 2. With one of the solutions that were presented, the code will not compile as 
 it is. According to your argument of „convenience“, the developer will
probably 
 just mark the function incorrectly as  trusted which makes the code compile.
The 
 memory corruption will happen. However, even if the developer did not think
much 
 about potential safety issues when adding  trusted to the function, he now
still 
 remembers that he did that (it was a conscious decision, even if it was a 
 careless and lazy one). He has a clear point to start looking for the reason
of 
 the memory corruption.
 
 Do you honestly think option 1 is better?

Yes, for reasons I carefully laid out.

 no clues whatsoever

He can look at unattributed declarations.

The whole debate boils down to "is greenwashing better, more honest, more 
debuggable than leaving things unattributed?" No on all three accounts.

Patrick Schluter <Patrick.Schluter bbox.fr> writes:

On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 On 5/25/2020 7:04 PM, Johannes Loher wrote:

[..]
 
 Do you honestly think option 1 is better?

 Yes, for reasons I carefully laid out.

which fails to convince anyone because the reasoning is flawed.

 no clues whatsoever

 He can look at unattributed declarations.

 The whole debate boils down to "is greenwashing better, more 
 honest, more debuggable than leaving things unattributed?" No 
 on all three accounts.

Unattended automatic greenwashing by the compiler is WORSE!

Bruce Carneal <bcarneal gmail.com> writes:

On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 On 5/25/2020 7:04 PM, Johannes Loher wrote:
 Now let's compare the two different options:
 
 1. With DIP1028 in its current form, the code will compile and 
 a memory corruption will actually happen. The problem might be 
 extremely difficult to track down for the developer because he 
 has no clues whatsoever where to start looking.
 
 2. With one of the solutions that were presented, the code 
 will not compile as it is. According to your argument of 
 „convenience“, the developer will probably just mark the 
 function incorrectly as  trusted which makes the code compile. 
 The memory corruption will happen. However, even if the 
 developer did not think much about potential safety issues 
 when adding  trusted to the function, he now still remembers 
 that he did that (it was a conscious decision, even if it was 
 a careless and lazy one). He has a clear point to start 
 looking for the reason of the memory corruption.
 
 Do you honestly think option 1 is better?

 Yes, for reasons I carefully laid out.

A direct response to Andre's thorough critique of your reasoning 
would be appreciated.

 no clues whatsoever

 He can look at unattributed declarations.

Again, as many have noted, putting a coverage problem like this 
on the programmer is problematic.

 The whole debate boils down to "is greenwashing better, more 
 honest, more debuggable than leaving things unattributed?" No 
 on all three accounts.

Greenwashing is bad.  En masse greenwashing by the compiler, as 
mandated by the DIP currently, is really bad.

Johannes Loher <johannes.loher fg4f.de> writes:

On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 He can look at unattributed declarations.

The issue is, you have to know about that beforehand. In the 
other situation, you explicitly get warned by the compiler (by 
the compile error) that that something dangerous is going on.

Also, this means, there are now 2 different things to look for: 
 trusted function declarations and unannotated extern(C) 
declarations. The latter is much harder to search for (as Andrei 
also noted in his reply) and it's just much simpler to only have 
one escape hatch to look out for. It is also a very weird special 
case.  safe means no memory corruption is possible except for any 
 trusted declarations and any unannotated extern(C) declarations, 
wrongfully  safe extern(C) declarations and wrongfully  trusted 
extern(C) declarations. This just doesn’t sound right.

 The whole debate boils down to "is greenwashing better, more 
 honest, more debuggable than leaving things unattributed?" No 
 on all three accounts.

The question should be rephrased: „Is explicit greenwashing by 
the developer better, more honest, more debuggable than implicit 
greenwashing by the compiler?“ Yes for better and more debuggable 
for the reasons that have already been mentioned so many times. 
Regarding honesty: In your variant, the compiler is dishonest,  
in the other variant, the developer is dishonest.

Just think about what the developer's reaction would be in the 
situation I described in my last post, when he actually finds the 
issue.

In your variant, the developer will be questioning why the 
compiler did not help him at all with realizing that there might 
be a problem. It is very likely that he blames the issue on the 
compiler.

In the other variant, he will realize that this something he had 
been warned about by the compiler and that he made a careless 
wrong decision and that it is his own fault. In the future he 
will be likely more careful when it comes to annotating stuff 
with  trusted. So this even has an educational effect.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/26/2020 12:18 AM, Johannes Loher wrote:
 Just think about what the developer's reaction would be in the situation I 
 described in my last post, when he actually finds the issue.
 
 In your variant, the developer will be questioning why the compiler did not
help 
 him at all with realizing that there might be a problem. It is very likely
that 
 he blames the issue on the compiler.
 
 In the other variant, he will realize that this something he had been warned 
 about by the compiler and that he made a careless wrong decision and that it
is 
 his own fault. In the future he will be likely more careful when it comes to 
 annotating stuff with  trusted. So this even has an educational effect.

It is a fair point. But I am looking a bit farther than that - the team that is 
responsible for QAing software (sometimes it is a separate team). The QA team 
cannot tell the difference between correctly annotated  trusted code and 
greenwashed code.

But they can tell when code is not annotated (no, it is not harder to do, the 
annotations are designed to not be hidden - annotations cannot be hidden by the 
preprocessor nor are they propagated from imports. They have to be there, and
if 
grep doesn't find them, they are not there. I've never had any difficulty 
finding the annotations belonging to a declaration).

Un-annotated C declarations should be a red flag to any competent QA team. 
Recognizing a false  trusted is a whole lot harder.

BTW, one good thing that has come out of this issue is people are strongly in 
favor of what  safe does. That bodes well for DIP1000 and the  live code, for a 
long time I seemed to be the only one who cared about it.

I also have no problem if the D style checker checked for un-annotated C 
declarations. That's the kind of thing it's for.

rikki cattermole <rikki cattermole.co.nz> writes:

On 27/05/2020 9:50 PM, Walter Bright wrote:
 BTW, one good thing that has come out of this issue is people are 
 strongly in favor of what  safe does. That bodes well for DIP1000 and 
 the  live code, for a long time I seemed to be the only one who cared 
 about it.

Most of the arguments against  safe by default I have seen over the 
years (including from me) originate in /how/ to make it default, not in 
it becoming the default.

For me at least, I will not ever want to use  live, pointers with 
multiple behaviors depending on how it is used which don't change the 
syntax? No thanks, I thought we had learned that was a bad idea.

But a head const storage class that handles lifetimes with no extra 
syntax thanks to DIP25/1000, yes please! I will use that.

Johannes Loher <johannes.loher fg4f.de> writes:

Am 27.05.20 um 11:50 schrieb Walter Bright:
 It is a fair point. But I am looking a bit farther than that - the team
 that is responsible for QAing software (sometimes it is a separate
 team). The QA team cannot tell the difference between correctly
 annotated  trusted code and greenwashed code.
 
 But they can tell when code is not annotated (no, it is not harder to
 do, the annotations are designed to not be hidden - annotations cannot
 be hidden by the preprocessor nor are they propagated from imports. They
 have to be there, and if grep doesn't find them, they are not there.
 I've never had any difficulty finding the annotations belonging to a
 declaration).
 
 Un-annotated C declarations should be a red flag to any competent QA
 team. Recognizing a false  trusted is a whole lot harder.

This is a very specific situation. There are a lot of teams / developers
that do not work in this manner. I don't know the numbers so I will make
no statement about what is more common but my personal experience is a
different one.

Also what is the difference between your QA department telling you to
correctly annotate C declarations and the compiler telling you that? If
you expect people to ignore what the compiler is telling them, why do
you expect them to listen to the QA department? In my opinion, the
compiler actually _is_ one of the best QA departments.

Also in my opinion, a competent QA department should carefully look at
any  trusted code /declarations. Maybe it is not a "red flag" but it is
definitely something that needs to be checked with extra care.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/27/2020 3:07 AM, Johannes Loher wrote:
 This is a very specific situation. There are a lot of teams / developers
 that do not work in this manner. I don't know the numbers so I will make
 no statement about what is more common but my personal experience is a
 different one.

I've seen larger companies operate this way. Smaller ones cannot afford to.


 Also what is the difference between your QA department telling you to
 correctly annotate C declarations and the compiler telling you that?
 If you expect people to ignore what the compiler is telling them, why do
 you expect them to listen to the QA department?

The QA dept is motivated to not be taken in by greenwashing.

(Back in my days at Boeing, I worked in the design department. There was an 
entirely separate organization call the Stress Group. Their job was to sign off 
on every nitpicky detail. If they signed off on something that turned out to be 
wrong, it was bad for their careers. Your designs did not move forward without 
Stress signoff. If you tried to trick Stress, that was the end of your career
at 
Boeing.)

So yeah, there's a huge difference between tricking the compiler and tricking 
the QA department.


 In my opinion, the
 compiler actually _is_ one of the best QA departments.

Indeed it is, and that's the whole point to  safe. My motivation here is make 
suspicious code stand out.  trusted code does not stand out so much, because it 
is required to exist.


 Also in my opinion, a competent QA department should carefully look at
 any  trusted code /declarations. Maybe it is not a "red flag" but it is
 definitely something that needs to be checked with extra care.

Looking at un-annotated declarations should be the first step. If it is 
annotated with  trusted, at least there's the coffee stain on the drawing 
indicating that somebody looked at it.

Johannes Loher <johannes.loher fg4f.de> writes:

Am 27.05.20 um 12:40 schrieb Walter Bright:
 
 Indeed it is, and that's the whole point to  safe. My motivation here is
 make suspicious code stand out.  trusted code does not stand out so
 much, because it is required to exist.

By that logic, wouldn't it also make more sense to implicitly default to
 trusted instead of  safe for any unannotated D code?

Following that logic, if  safe is the default and somebody writes
non- safe code without annotating it, it won't compile they might just
slap  trusted on it without actually checking it. By that they make it
harder for the QA department to spot.
If  trusted was the default, it would compile, there would be no
annotation on it an the QA department would see it is a red flag.

What's the difference here?

RazvanN <razvan.nitu1305 gmail.com> writes:

On Wednesday, 27 May 2020 at 10:40:18 UTC, Walter Bright wrote:
 On 5/27/2020 3:07 AM, Johannes Loher wrote:
 This is a very specific situation. There are a lot of teams / 
 developers
 that do not work in this manner. I don't know the numbers so I 
 will make
 no statement about what is more common but my personal 
 experience is a
 different one.

 I've seen larger companies operate this way. Smaller ones 
 cannot afford to.


 Also what is the difference between your QA department telling 
 you to
 correctly annotate C declarations and the compiler telling you 
 that?
 If you expect people to ignore what the compiler is telling

 them, why do
 you expect them to listen to the QA department?

 The QA dept is motivated to not be taken in by greenwashing.

 (Back in my days at Boeing, I worked in the design department. 
 There was an entirely separate organization call the Stress 
 Group. Their job was to sign off on every nitpicky detail. If 
 they signed off on something that turned out to be wrong, it 
 was bad for their careers. Your designs did not move forward 
 without Stress signoff. If you tried to trick Stress, that was 
 the end of your career at Boeing.)

 So yeah, there's a huge difference between tricking the 
 compiler and tricking the QA department.


 In my opinion, the
 compiler actually _is_ one of the best QA departments.

 Indeed it is, and that's the whole point to  safe. My 
 motivation here is make suspicious code stand out.  trusted 
 code does not stand out so much, because it is required to 
 exist.

I'm utterly confused by this.

Trusted code is always suspicious and always stands out because 
it is not formally verified whereas  safe code should be 
bulletproof. Why would anyone bother to manually check code that 
is supposed to be  safe? Trusted code is the villain here; 
wherever you see  trusted you know that the code might contain 
vulnerabilities because it was audited by a human and not by a 
machine.

If you have  safe code that has memory safety issues (from 
callinc C functions) then what's the point of  safe?

 Also in my opinion, a competent QA department should carefully 
 look at
 any  trusted code /declarations. Maybe it is not a "red flag" 
 but it is
 definitely something that needs to be checked with extra care.

 Looking at un-annotated declarations should be the first step. 
 If it is annotated with  trusted, at least there's the coffee 
 stain on the drawing indicating that somebody looked at it.

WebFreak001 <d.forum webfreak.org> writes:

On Wednesday, 27 May 2020 at 10:07:48 UTC, Johannes Loher wrote:
 Am 27.05.20 um 11:50 schrieb Walter Bright:
 [...]
 
 Un-annotated C declarations should be a red flag to any 
 competent QA team. Recognizing a false  trusted is a whole lot 
 harder.

 [...]

 Also in my opinion, a competent QA department should carefully 
 look at any  trusted code /declarations. Maybe it is not a "red 
 flag" but it is definitely something that needs to be checked 
 with extra care.

I think additionally we shouldn't make D code review harder to do 
than it needs to be.

For a reviewer "something correct which is missing" is harder to 
pick up than "something wrong which is there" ( trusted/ system 
attributes)

When I, as an incompetent but trying QA department, come across a 
code change introducing a new module wrapping a C library and all 
tests pass, I would most likely not notice that the author did 
not type ` system:`. If there was an ` trusted:` at the top of 
the file it would be picked up by D-Scanner and help in code 
review. And well if someone slapped ` safe:` at the top of the 
file... read on:

Someone in this thread wrote something like " safe" should be 
forbidden on functions without method body and instead only use 
 trusted or  system. I think it might even be you who brought it 
up? I love the idea of forbidding  safe when the compiler can't 
actually check the content of a function.  safe,  trusted and 
 system could then have perfect definitions without edge cases. 
And you would change the mangling of  safe and  trusted to be the 
same which was also brought up in this thread.

Johannes Loher <johannes.loher fg4f.de> writes:

Am 27.05.20 um 11:50 schrieb Walter Bright:
 
 It is a fair point.

By the way, thank you for acknowledging that. I appreciate it.

Arafel <er.krali gmail.com> writes:

On 27/5/20 11:50, Walter Bright wrote:
 
 It is a fair point. But I am looking a bit farther than that - the team 
 that is responsible for QAing software (sometimes it is a separate 
 team). The QA team cannot tell the difference between correctly 
 annotated  trusted code and greenwashed code.
 

This is a most unsubstantiated assertion.

If this were the case, I would have to wonder what the QA team is there 
for, if they can just be replaced by an automatic lint tool.

*AND* even if that were so, see below:

 But they can tell when code is not annotated (no, it is not harder to 
 do, the annotations are designed to not be hidden - annotations cannot 
 be hidden by the preprocessor nor are they propagated from imports. They 
 have to be there, and if grep doesn't find them, they are not there. 
 I've never had any difficulty finding the annotations belonging to a 
 declaration).
 

Then the coder just slaps ` safe:`, ` trusted:`, or whatever 
greenwashing you were trying to avoid, and we're just back to square one 
since, per your previous assertion, QA can't notice the difference.

Heck, in such an environment I might slap ` trusted:` just in case at 
the beginning of each and every file and live in a land of bliss thereafter!

 Un-annotated C declarations should be a red flag to any competent QA 
 team. Recognizing a false  trusted is a whole lot harder.
 

Exactly, which is why they should be disallowed by the compiler. 
Considering them ` system` by default is a way to do it. Another 
(perhaps even better) already proposed one would be to actually consider 
them ` safe`, but disallow ` safe` declarations where the code can't be 
analyzed by the compiler (it would be debatable if the mangling should 
be enough or not).

 BTW, one good thing that has come out of this issue is people are 
 strongly in favor of what  safe does. That bodes well for DIP1000 and 
 the  live code, for a long time I seemed to be the only one who cared 
 about it.
 

Being in favour of what  safe purports to do and in favour of what it 
actually does are two very different things. The whole selling point of 
 safe used to be "you're only responsible for ` trusted` code, now 
that's no longer true (or at least even less, if we account for the 
already existing holes).

The direction of travel is opposite to the announced destination.

 I also have no problem if the D style checker checked for un-annotated C 
 declarations. That's the kind of thing it's for.
 

So, if you leave them for the style checker that means that you envision 
cases where this could be the desired behaviour, otherwise you should 
just let the compiler error.

What would these cases look like? Because I can't imagine any.

Andrej Mitrovic <andrej.mitrovich gmail.com> writes:

On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
 Un-annotated C declarations should be a red flag to any 
 competent QA team. Recognizing a false  trusted is a whole lot 
 harder.

Is the actual problem those ` trusted:` declarations at the top 
of C headers?

There could be a simple solution to that:

Ban ` trusted:` and ` trusted { }` which apply to multiple 
symbols. Only allow ` trusted` to apply to a single symbol. For 
example:

---
 trusted:

extern(C) void memcpy (void*, void*, size_t);
extern(C) void write42 (void*);
---

Error.

---
 trusted extern(C) void memcpy (void*, void*, size_t);
 trusted extern(C) void write42 (void*);
---

OK, compiles. The bindings author has clearly added  trusted 
manually to each symbol.

Obviously there are escape hatches like mixins, code generation, 
etc.

Claude <claudemr live.fr> writes:

On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic wrote:
 Is the actual problem those ` trusted:` declarations at the top 
 of C headers?

 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple 
 symbols. Only allow ` trusted` to apply to a single symbol. For

IMO, it makes things worse. Because the careless programmer will 
slap  trusted to every declaration (maybe with a script or 
find/replace macro of his editor). So now, we don't know if the 
annotation is greenwashing or careful examination of the 
definition.

At least with " trusted:" and " trusted { }", the greenwashing is 
obvious.

Andrej Mitrovic <andrej.mitrovich gmail.com> writes:

On Wednesday, 27 May 2020 at 13:47:46 UTC, Claude wrote:
 On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic 
 wrote:
 Is the actual problem those ` trusted:` declarations at the 
 top of C headers?

 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple 
 symbols. Only allow ` trusted` to apply to a single symbol. For

 IMO, it makes things worse. Because the careless programmer 
 will slap  trusted to every declaration (maybe with a script or 
 find/replace macro of his editor). So now, we don't know if the 
 annotation is greenwashing or careful examination of the 
 definition.

The difference is when adding new symbols. If version control is 
used, it would be very obvious in a review whether a new  trusted 
symbol was added or not. A diff typically shows several lines of 
context, and a toplevel ` trusted:` is easy to miss.

Zoadian <no no.no> writes:

On Wednesday, 27 May 2020 at 13:47:46 UTC, Claude wrote:
 On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic 
 wrote:
 Is the actual problem those ` trusted:` declarations at the 
 top of C headers?

 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple 
 symbols. Only allow ` trusted` to apply to a single symbol. For

 IMO, it makes things worse. Because the careless programmer 
 will slap  trusted to every declaration (maybe with a script or 
 find/replace macro of his editor). So now, we don't know if the 
 annotation is greenwashing or careful examination of the 
 definition.

 At least with " trusted:" and " trusted { }", the greenwashing 
 is obvious.

the problem here ist: we are talking about careless programmers.
for those of us who actually care about safety it will be like:
1. ok lets write an extern(c) function.
2. forget to annotate it by accident.
3. hit compile
4. a) get an compiler error the hints that we missed that 
annotation
or b) no error at all and we now have silently added unsafe code 
to our codebase.

also, you can never trust that an  trusted function is actually 
memory safe.
even if it was evaluated in the past. the addition of new code 
(and even completely  safe code) can break any assumptions that 
 trusted fun was relying on.

so a competent QA team _has_ to either check all  trusted code 
fragments.
or with this dip check all  trusted and all extern functions.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/27/20 9:42 AM, Andrej Mitrovic wrote:
 On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
 Un-annotated C declarations should be a red flag to any competent QA 
 team. Recognizing a false  trusted is a whole lot harder.

 
 Is the actual problem those ` trusted:` declarations at the top of C 
 headers?
 
 There could be a simple solution to that:
 
 Ban ` trusted:` and ` trusted { }` which apply to multiple symbols. Only 
 allow ` trusted` to apply to a single symbol. For example:

Oh wow what an interesting idea. Thanks.

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Wednesday, May 27, 2020 6:24:25 PM MDT Andrei Alexandrescu via Digitalmars-
d-announce wrote:
 On 5/27/20 9:42 AM, Andrej Mitrovic wrote:
 On Wednesday, 27 May 2020 at 09:50:50 UTC, Walter Bright wrote:
 Un-annotated C declarations should be a red flag to any competent QA
 team. Recognizing a false  trusted is a whole lot harder.

 Is the actual problem those ` trusted:` declarations at the top of C
 headers?

 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple symbols. Only

 allow ` trusted` to apply to a single symbol. For example:

 Oh wow what an interesting idea. Thanks.

I've argued for years that mass-applying any attribute is bad practice. It
makes it way too easy to accidentally apply an attribute to a function, and
experience has shown that it makes it _far_ easier to not realize that a
funtion already has a particular attribute (it's definitely happened in
Phobos PRs that an attribute has been mass-applied and then someone comes
along later and applies it to a specific function, because they didn't think
that that function had that attribute). Unfortunately, plenty of people seem
to love to mass apply attributes rather than marking each function
individually (presumably, because they hate having to use attributes all
over the place). And regardless of whether DIP 1028 is a good idea as-is,
the sad reality of the matter is that it's not uncommon even in druntime for
someone to slap  trusted: at the top of a module. It implies that the
declarations in question were not actually checked, and it's incredibly
error-prone when new declarations are added.

Personally, I normally only use : with access-level modifiers, and while I
like doing that (especially since I think that public and private functions
should normally be segregated anyway), I'm increasingly coming closer to the
conclusion that it's actually a bad idea in practice, because it's not
uncommon for people to not know which access-level modifier applies -
especially when looking at diffs in PRs.

I would _love_ to see it become illegal to mass-apply  trusted (or even
attributes in general), but I have no clue how easy it would be to get such
a DIP accepted or how much screaming there would be over it if it were
actually accepted.

- Jonathan M Davis

Q. Schroll <qs.il.paperinik gmail.com> writes:

On Wednesday, 27 May 2020 at 13:42:08 UTC, Andrej Mitrovic wrote:
 There could be a simple solution to that:

 Ban ` trusted:` and ` trusted { }` which apply to multiple 
 symbols. Only allow ` trusted` to apply to a single symbol. For 
 example:

That came to my mind immediately. I'm not entirely sure about 
` trusted { }`, because it might be handy at times (I don't use 
` attribute { }` often). But ` trusted:` is such that it is 
almost guaranteed that sooner or later it's applied to something 
it wasn't intended to. I have no idea if that's the case with 
braces.
Since ` trusted` is intended to be special, I'd go the full route 
and ban it for any non-direct use. Mass-` trusted` makes no sense 
logically: If the programmer is required to investigate a 
function to determine it's ` safe`-ty, the time required for the 
investigation clearly outweighs the time marking the function 
` trusted` (if the investigation has that result, of course), 
even if the judgment is fast.

If people go through a whole lot of `extern(C)` declarations and 
marks them  trusted individually because neither ` trusted:` nor 
` trusted { }` did the job, it may occur to them it's probably 
not the right thing.

I really hope this makes it through.

Gregory <g.thompson.1892 gmall.com> writes:

On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 On 5/25/2020 7:04 PM, Johannes Loher wrote:
 Now let's compare the two different options:
 
 1. With DIP1028 in its current form, the code will compile and 
 a memory corruption will actually happen. The problem might be 
 extremely difficult to track down for the developer because he 
 has no clues whatsoever where to start looking.
 
 2. With one of the solutions that were presented, the code 
 will not compile as it is. According to your argument of 
 „convenience“, the developer will probably just mark the 
 function incorrectly as  trusted which makes the code compile. 
 The memory corruption will happen. However, even if the 
 developer did not think much about potential safety issues 
 when adding  trusted to the function, he now still remembers 
 that he did that (it was a conscious decision, even if it was 
 a careless and lazy one). He has a clear point to start 
 looking for the reason of the memory corruption.
 
 Do you honestly think option 1 is better?

 Yes, for reasons I carefully laid out.

 no clues whatsoever

 He can look at unattributed declarations.

 The whole debate boils down to "is greenwashing better, more 
 honest, more debuggable than leaving things unattributed?" No 
 on all three accounts.

I think this is what's the most frustrating part. You aren't 
actually taking criticism if you cherry-pick what you reply to.

Johannes T <isrvoid gmail.com> writes:

On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]

Thank you very much for your patience with all the negative 
feedback. I get your decision to not annotate extern C with 
 system by default. The biggest issue with extern  system is that 
 trusted would become less useful when dealing with declarations 
and bindings.  trusted would appear more frequently. We wouldn't 
be able to assume that the author put in effort in assessing the 
trustworthiness of a declaration. More greenwashing, less safety. 
Those are severe drawbacks, I agree.
However, as Andrei pointed out, PR is a huge problem. We need to 
be able to sell it with a straight face.
I think this might be done by formally redefining the meaning of 
safety attributes when applied to extern:
The safety of extern can't be assessed by the compiler. Implicit 
" safe" extern within  safe code is accepted for the convenience 
of the average user. Explicit  system or  trusted can be added 
for desired behavior.
I think this can be explained.
It's not an acceptable solution for people who require 
correctness. ZombineDev can't tell an auditor that he relies on 
D-Scanner to find the problematic spots. I believe it's 
unavoidable to provide an option to change the default. It could 
make extern  system or refuse to compile safe code calling  safe 
extern.

Johannes Loher <johannes.loher fg4f.de> writes:

Am 26.05.20 um 14:20 schrieb Johannes T:
 Thank you very much for your patience with all the negative feedback. I
 get your decision to not annotate extern C with  system by default.

As much as i disagree with the decision, I am still very glad that we
are at least having a discussion. I also want to thank Walter for
participating here. I understand this can be difficult but it is the
right thing to do.

 The biggest issue with extern  system is that  trusted would become less
 useful when dealing with declarations and bindings.  trusted would
 appear more frequently. We wouldn't be able to assume that the author
 put in effort in assessing the trustworthiness of a declaration. More
 greenwashing, less safety. Those are severe drawbacks, I agree.
 However, as Andrei pointed out, PR is a huge problem. We need to be able
 to sell it with a straight face.
 I think this might be done by formally redefining the meaning of safety
 attributes when applied to extern:
 The safety of extern can't be assessed by the compiler. Implicit " safe"
 extern within  safe code is accepted for the convenience of the average
 user. Explicit  system or  trusted can be added for desired behavior.
 I think this can be explained.

Well, the spec actually already mentions that:

 Safe External Functions

 External functions don't have a function body visible to the compiler:

  safe extern (C) void play();

 and so safety cannot be verified automatically.
 Best Practices: Explicitly set an attribute for external functions

rather than relying on default settings.

I still don't think this is the correct thing to do. Instead, annotating
external (at least extern(C)) functions with  safe (implicitly as the
default or explicitly) should be a compile error in my opinion.

 It's not an acceptable solution for people who require correctness.
 ZombineDev can't tell an auditor that he relies on D-Scanner to find the
 problematic spots. I believe it's unavoidable to provide an option to
 change the default. It could make extern  system or refuse to compile
 safe code calling  safe extern.

A switch might help but the whole purpose of this DIP is to get the
defaults right. In my opinion, such a switch should be enabled by
default. There is also the danger of creating 2 versions of the language
that are incompatible with each other. Only giving this a quick thought,
I believe it could work (you can still link libraries compiled with and
without the switch) but I'm not completely sure.

Panke <tobias pankrath.net> writes:

On Tuesday, 26 May 2020 at 12:38:54 UTC, Johannes Loher wrote:
 A switch might help but the whole purpose of this DIP is to get 
 the defaults right. In my opinion, such a switch should be 
 enabled by default. There is also the danger of creating 2 
 versions of the language that are incompatible with each other. 
 Only giving this a quick thought, I believe it could work (you 
 can still link libraries compiled with and without the switch) 
 but I'm not completely sure.

The bazel community has lots of such switches. Basically every 
new behaviour get's introduced with a --preview switch, that will 
turn into a --revert after some time.

For each switch there is a github issue, explaining the change it 
detail. Why it was necessary, what the changed behaviour is, how 
to migrate and the timeline for this particular switch.

Johannes Loher <johannes.loher fg4f.de> writes:

Am 26.05.20 um 15:10 schrieb Panke:
 
 The bazel community has lots of such switches. Basically every new
 behaviour get's introduced with a --preview switch, that will turn into
 a --revert after some time.
 
 For each switch there is a github issue, explaining the change it
 detail. Why it was necessary, what the changed behaviour is, how to
 migrate and the timeline for this particular switch.

D has the same thing (e.g. -preview=safedefault will enable this DIP).
The main difference to Johannes T's suggestion is that these preview and
revert switches are only a temporary measure to ease the transition.
They do not create different versions of the language permanently.

Panke <tobias pankrath.net> writes:

On Tuesday, 26 May 2020 at 13:21:08 UTC, Johannes Loher wrote:
 Am 26.05.20 um 15:10 schrieb Panke:
 
 The bazel community has lots of such switches. Basically every 
 new behaviour get's introduced with a --preview switch, that 
 will turn into a --revert after some time.
 
 For each switch there is a github issue, explaining the change 
 it detail. Why it was necessary, what the changed behaviour 
 is, how to migrate and the timeline for this particular switch.

 D has the same thing (e.g. -preview=safedefault will enable 
 this DIP). The main difference to Johannes T's suggestion is 
 that these preview and revert switches are only a temporary 
 measure to ease the transition. They do not create different 
 versions of the language permanently.

What we don't have is good documentation and a clear timeline for 
the switches.

Panke <tobias pankrath.net> writes:

On Tuesday, 26 May 2020 at 12:20:31 UTC, Johannes T wrote:
 On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]

 Thank you very much for your patience with all the negative 
 feedback.

Yes, good think to stop once in a while and appreciate it. "To 
not complain is sufficient praise", does not always cut it.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/26/2020 6:07 AM, Panke wrote:
 On Tuesday, 26 May 2020 at 12:20:31 UTC, Johannes T wrote:
 On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]

 Thank you very much for your patience with all the negative feedback.

 
 Yes, good think to stop once in a while and appreciate it. "To not complain is 
 sufficient praise", does not always cut it.


Thank you, it is nice to see this.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/26/2020 5:20 AM, Johannes T wrote:
 On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]

 
 Thank you very much for your patience with all the negative feedback. I get
your 
 decision to not annotate extern C with  system by default. The biggest issue 
 with extern  system is that  trusted would become less useful when dealing
with 
 declarations and bindings.  trusted would appear more frequently. We wouldn't
be 
 able to assume that the author put in effort in assessing the trustworthiness
of 
 a declaration. More greenwashing, less safety. Those are severe drawbacks, I
agree.
 However, as Andrei pointed out, PR is a huge problem. We need to be able to
sell 
 it with a straight face.

Frankly, I feel that if I could sit down with you folks, I can get the idea 
across what I'm trying to accomplish.

 I think this might be done by formally redefining the meaning of safety 
 attributes when applied to extern:
 The safety of extern can't be assessed by the compiler. Implicit " safe"
extern 
 within  safe code is accepted for the convenience of the average user.
Explicit 
  system or  trusted can be added for desired behavior.
 I think this can be explained.

I do, too. But apparently not by typing.

 It's not an acceptable solution for people who require correctness. ZombineDev 
 can't tell an auditor that he relies on D-Scanner to find the problematic
spots. 
 I believe it's unavoidable to provide an option to change the default. It
could 
 make extern  system or refuse to compile safe code calling  safe extern.

People who require correctness are going to have to manually audit each and 
every C declaration regardless. There's no way around it. A less strict auditor 
would focus his attention on un-annotated declarations and assume that
annotated 
ones are annotated correctly.

rikki cattermole <rikki cattermole.co.nz> writes:

On 27/05/2020 10:03 PM, Walter Bright wrote:
 Frankly, I feel that if I could sit down with you folks, I can get the 
 idea across what I'm trying to accomplish.

Okay, how is your camera and mic situation?

Lets do a Twitch stream, make sure there are some moderators in place as 
well.

Open to everybody and it can be recorded by Twitch.

Stefan Koch <uplink.coder googlemail.com> writes:

On Wednesday, 27 May 2020 at 10:06:41 UTC, rikki cattermole wrote:
 On 27/05/2020 10:03 PM, Walter Bright wrote:
 Frankly, I feel that if I could sit down with you folks, I can 
 get the idea across what I'm trying to accomplish.

 Okay, how is your camera and mic situation?

 Lets do a Twitch stream, make sure there are some moderators in 
 place as well.

 Open to everybody and it can be recorded by Twitch.

Youtube streams work better in my experience.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/27/2020 3:06 AM, rikki cattermole wrote:
 Open to everybody and it can be recorded by Twitch.

I've done the video conferencing many times over the decades. I just don't find 
it to be productive. Maybe it's some defect in me.

They usually go something like this:

"Who else is here?"
"Hello ... Hello?"
"Can anybody hear me? Is my mike working?"
"Where's Bob? Is he connected?"
"I just got my connection working again. What did I miss?"
"Bob? Bob?"
"Can someone go find Bob?"
"Can you say that again? The audio cut out."
"Bob here!"
"Hmm. We lost Bob again?"
"Fred, can you please turn off your motorcycle?"
"Bob here again! Sorry!"
"Fred? Now where's Fred? Fred?"

What has always worked best for me is a face to face over coffee and/or beer.

Which is why I'm missing DConf this year terribly.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/27/20 6:22 AM, Walter Bright wrote:
 On 5/27/2020 3:06 AM, rikki cattermole wrote:
 Open to everybody and it can be recorded by Twitch.

 
 I've done the video conferencing many times over the decades. I just 
 don't find it to be productive. Maybe it's some defect in me.
 
 They usually go something like this:

[attempt at joke snipped]

It's safe to assume things have improved over time.

matheus <matheus gmail.com> writes:

On Wednesday, 27 May 2020 at 11:46:40 UTC, Andrei Alexandrescu 
wrote:
 On 5/27/20 6:22 AM, Walter Bright wrote:
 On 5/27/2020 3:06 AM, rikki cattermole wrote:
 Open to everybody and it can be recorded by Twitch.

 
 I've done the video conferencing many times over the decades. 
 I just don't find it to be productive. Maybe it's some defect 
 in me.
 
 They usually go something like this:

 [attempt at joke snipped]

 It's safe to assume things have improved over time.

Currently because COVID-19 we have (At least where I live) 
Politicians having meetings online and open to the public and it 
goes without any problems.

I've been saying this since last year, why not go for online 
meetings to fix things instead of only using this Forum which 
sometimes posts get neglected and ideas misunderstood.

Ouch this is about Technology and people don't use it the right 
way.

Matheus.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/27/20 6:03 AM, Walter Bright wrote:
 On 5/26/2020 5:20 AM, Johannes T wrote:
 On Tuesday, 26 May 2020 at 03:37:29 UTC, Walter Bright wrote:
 [..]

 Thank you very much for your patience with all the negative feedback. 
 I get your decision to not annotate extern C with  system by default. 
 The biggest issue with extern  system is that  trusted would become 
 less useful when dealing with declarations and bindings.  trusted 
 would appear more frequently. We wouldn't be able to assume that the 
 author put in effort in assessing the trustworthiness of a 
 declaration. More greenwashing, less safety. Those are severe 
 drawbacks, I agree.
 However, as Andrei pointed out, PR is a huge problem. We need to be 
 able to sell it with a straight face.

 
 Frankly, I feel that if I could sit down with you folks, I can get the 
 idea across what I'm trying to accomplish.

Didn't work with me.

Gregory <g.thompson.1892 gmall.com> writes:

On Wednesday, 27 May 2020 at 10:03:21 UTC, Walter Bright wrote:
 It's not an acceptable solution for people who require 
 correctness. ZombineDev can't tell an auditor that he relies 
 on D-Scanner to find the problematic spots. I believe it's 
 unavoidable to provide an option to change the default. It 
 could make extern  system or refuse to compile safe code 
 calling  safe extern.

 People who require correctness are going to have to manually 
 audit each and every C declaration regardless. There's no way 
 around it. A less strict auditor would focus his attention on 
 un-annotated declarations and assume that annotated ones are 
 annotated correctly.

A less strict auditor wouldn't have to look at un-annotated 
extern(C) functions if they were  system by default. It's more 
work for them to ensure they are actually  safe. The majority 
will be un-annotated if you leave it as  safe.

JN <666total wp.pl> writes:

On Tuesday, 26 May 2020 at 02:04:02 UTC, Johannes Loher wrote:
 According to your argument of „convenience“, the developer will 
 probably just mark the function incorrectly as  trusted which 
 makes the code compile. The memory corruption will happen. 
 However, even if the developer did not think much about 
 potential safety issues when adding  trusted to the function, 
 he now still remembers that he did that (it was a conscious 
 decision, even if it was a careless and lazy one). He has a 
 clear point to start looking for the reason of the memory 
 corruption.

Or he'll do the right thing, and keep the function  system but 
call it from a  trusted block. Then when he's stuck debugging, he 
can show it to his experienced buddy and his experienced buddy 
will instantly look in the  trusted block because it's suspicious.

M.M. <matus email.cz> writes:

On Tuesday, 26 May 2020 at 01:16:49 UTC, Walter Bright wrote:
 On 5/24/2020 5:56 PM, Timon Gehr wrote:
 It's only greenwashing if it's misleading. Putting  safe is a 
 lie, putting  trusted is honest.

 It is not honest unless the programmer actually carefully 
 examined the interface and the documentation to determine if it 
 is a safe interface or not. For example, labeling memcpy() with 
  trusted is not honest.

 Forcing people to add uncheckable annotations is a path to 
 convenience, not honesty.

What is the difference of  safe to  trusted in that respect? Does 
the compiler "carefully examines" any interface or documentation?

Why not simply introducing new label as a solution, something in 
the realm  extern_safe_dont_know?

Bruce Carneal <bcarneal gmail.com> writes:

On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.

Agree completely.  Not in dispute that I've seen.  In the same 
vein, defaulting a declaration to  safe doesn't make it safe.

For the ultra paranoid, even the name mangling in D libraries is 
not to be trusted because "the implementation is not available".

 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" - 
 just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.

I don't believe that you or any other competent programmer 
greenwashes safety critical code.  Regardless, the safety 
conscious must review their dependencies whatever default applies.

 3. Un-annotated declarations are easily detectable in a code 
 review.

Automating this for the transitive closure of defaulted  safe 
functions would help.  Maybe that capability is there already and 
I missed it?

 [snip]
 It is, in a not-at-all obvious way, safer for C declarations to 
 default to being safe.

I agree that it is not-at-all obvious.

On a positive note, the DIP discussion/clarification should 
encourage the safety conscious to rebase code to a machine 
checkable form whenever feasible.

Bruce Carneal <bcarneal gmail.com> writes:

On Sunday, 24 May 2020 at 06:26:56 UTC, Bruce Carneal wrote:
 On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 [snip]
 3. Un-annotated declarations are easily detectable in a code 
 review.

 Automating this for the transitive closure of defaulted  safe 
 functions would help.  Maybe that capability is there already 
 and I missed it?

This tooling up to try and mitigate the damage caused by the DIP 
would not be necessary were Timon's model followed.

After reading Timon's latest I've regained hope that this could 
still work out well.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/23/2020 11:26 PM, Bruce Carneal wrote:
 I don't believe that you or any other competent programmer greenwashes safety 
 critical code.  Regardless, the safety conscious must review their
dependencies 
 whatever default applies.

That's the theory. But we do, for various reasons. I've seen it a lot over the 
years, at all levels of programming ability. It particularly happens when 
someone needs to get the code compiling and running, and the error message is 
perceived as a nuisance getting in the way.

We should be very careful about adding nuisances to the language that make it 
easier to greenwash than to do the job correctly.

Timon Gehr <timon.gehr gmx.ch> writes:

On 24.05.20 11:10, Walter Bright wrote:
 On 5/23/2020 11:26 PM, Bruce Carneal wrote:
 I don't believe that you or any other competent programmer greenwashes 
 safety critical code.  Regardless, the safety conscious must review 
 their dependencies whatever default applies.

 
 That's the theory. But we do, for various reasons. I've seen it a lot 
 over the years, at all levels of programming ability. It particularly 
 happens when someone needs to get the code compiling and running, and 
 the error message is perceived as a nuisance getting in the way.
 
 We should be very careful about adding nuisances to the language that 
 make it easier to greenwash than to do the job correctly.

Implicit greenwashing by the compiler is a nuisance that makes it harder 
to do the job correctly and easier to do the wrong thing.

Bruce Carneal <bcarneal gmail.com> writes:

On Monday, 25 May 2020 at 01:04:24 UTC, Timon Gehr wrote:
 On 24.05.20 11:10, Walter Bright wrote:
 On 5/23/2020 11:26 PM, Bruce Carneal wrote:
 I don't believe that you or any other competent programmer 
 greenwashes safety critical code.  Regardless, the safety 
 conscious must review their dependencies whatever default 
 applies.

 
 That's the theory. But we do, for various reasons. I've seen 
 it a lot over the years, at all levels of programming ability. 
 It particularly happens when someone needs to get the code 
 compiling and running, and the error message is perceived as a 
 nuisance getting in the way.
 
 We should be very careful about adding nuisances to the 
 language that make it easier to greenwash than to do the job 
 correctly.

 Implicit greenwashing by the compiler is a nuisance that makes 
 it harder to do the job correctly and easier to do the wrong 
 thing.

Yes, it would be a big nuisance.  Absent a change in the DIP the 
safety conscious who want to continue with D will try to back out 
the compiler lies as best they can: additional tooling, 
additional code review strictures, selective rewrites, ...

Not sure how that unfortunate future would play out exactly but 
it would not be pretty.  Much much better to fix the DIP.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/24/2020 6:04 PM, Timon Gehr wrote:
 Implicit greenwashing by the compiler is a nuisance that makes it harder to do 
 the job correctly and easier to do the wrong thing.

You and I are just going to disagree about that.

Aliak <something something.com> writes:

On Wednesday, 27 May 2020 at 02:42:24 UTC, Walter Bright wrote:
 On 5/24/2020 6:04 PM, Timon Gehr wrote:
 Implicit greenwashing by the compiler is a nuisance that makes 
 it harder to do the job correctly and easier to do the wrong 
 thing.

 You and I are just going to disagree about that.

Unfortunately science and research Doesn't agree with you. 
https://en.m.wikipedia.org/wiki/Default_effect

Arafel <er.krali gmail.com> writes:

On 24/5/20 5:28, Walter Bright wrote:
 I'd like to emphasize:
 
 1. It is not possible for the compiler to check any declarations where 
 the implementation is not available. Not in D, not in any language. 
 Declaring a declaration safe does not make it safe.
 

Here, I agree with Timon: only  system and  trusted should be possible, 
and in my personal view it should be mandatory to annotate external 
declarations. If not, at least default to ` system`.

 2. If un-annotated declarations cause a compile time error, it is highly 
 likely the programmer will resort to "greenwashing" - just slapping 
  safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've 
 all done it. Sometimes even for good reasons.
 

So the compiler helpfully does the "greenwashing" for us. How is that an 
improvement in any way?

 3. Un-annotated declarations are easily detectable in a code review.
 

No, they are not. If somebody slaps ` trusted:` at the beginning of a 
file, that's a conscious decision, there will be a commit for that. 
Fingers can be pointed and questions asked.

If the compiler silently does it's highly likely that many people won't 
notice, and at the very least there will be plausible deniability 
("sure, boss, I didn't know you had to slap ` system` to our C 
declarations, also, I was using  safe and the compiler didn't complain").

 4. Greenwashing is not easily detectable in a code review.
 

I would say that greping for ` trusted:` is no that hard. I'm pretty 
sure tools like d-scanner are more than able to catch the more advanced 
cases.

 5. Greenwashing doesn't fix anything. The code is not safer. It's an 
 illusion, not a guarantee.
 

Yet the compiler will just do it, breaking all the promises for ` safe` 
code in the process. Now the whole ` safe` concept becomes the same 
"illusion".

 6. If someone cares to annotate declarations, it means he has at least 
 thought about it, because he doesn't need to. Hence it's more likely to 
 be correct than when greenwashed.
 

Q.E.D.

How come this doesn't apply to the compiler?

 7. D should *not* make it worthwhile for people to greenwash code.
 

Then just disallow blanket greenwashing at all. You're just making the 
case for forbidding ` trusted:`, and that's something I'd totally support.

Or make it so that function declarations need

 It is, in a not-at-all obvious way, safer for C declarations to default 
 to being safe.

It's definitely one of most not-obvious things I've ever seen here.

Johannes T <isrvoid gmail.com> writes:

On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:
 [..]

Thank you. I do see the picture now. When taking safety 
seriously, you would find unannotated declarations and 
audit/annotate them. It is not worthy of a compile option. An 
average user is, from experience, worse off with extern being 
implicitly  system.

Patrick Schluter <Patrick.Schluter bbox.fr> writes:

On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.

 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" - 
 just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.

 3. Un-annotated declarations are easily detectable in a code 
 review.

 4. Greenwashing is not easily detectable in a code review.

 5. Greenwashing doesn't fix anything. The code is not safer. 
 It's an illusion, not a guarantee.

 6. If someone cares to annotate declarations, it means he has 
 at least thought about it, because he doesn't need to. Hence 
 it's more likely to be correct than when greenwashed.

 7. D should *not* make it worthwhile for people to greenwash 
 code.

 It is, in a not-at-all obvious way, safer for C declarations to 
 default to being safe.

Apparently, you're of the opinion it's better the compiler does 
the greenwashing. Got it!

Arine <arine1283798123 gmail.com> writes:

On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.

That's why Rust has gotten rid of declarations all together, 
except for C, which are implicitly marked unsafe, and they cannot 
be marked "safe" (there is no such feature).

 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" - 
 just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.

Having extern(C) be  safe by default doesn't stop greenwashing. 
As others have pointed out, you are doing the greenwashing for 
the programmer. That's not better.

 3. Un-annotated declarations are easily detectable in a code 
 review.

Declarations annotated with  trusted are even easier to detect. 
And arguably declarations that aren't annotated aren't that easy 
to detect.

 4. Greenwashing is not easily detectable in a code review.

 5. Greenwashing doesn't fix anything. The code is not safer. 
 It's an illusion, not a guarantee.

 6. If someone cares to annotate declarations, it means he has 
 at least thought about it, because he doesn't need to. Hence 
 it's more likely to be correct than when greenwashed.

 7. D should *not* make it worthwhile for people to greenwash 
 code.

 It is, in a not-at-all obvious way, safer for C declarations to 
 default to being safe.

This whole situation just makes it more obvious to me, you 
shouldn't be able to mark extern(C) declarations as  safe or 
 trusted at all. If that truly is your fear, it should just not 
be possible to mark them as  safe/ trusted.

8. D should make incorrect code *more* difficult, not easier. Or 
are you conveniently ignoring this ideology?

Code is going to break no matter what. That's the cost you are 
going to pay for making  safe default. You can't fix that. The 
difference is what the aftermath will look like.

Paul Backus <snarwin gmail.com> writes:

On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.

 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" - 
 just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.

 3. Un-annotated declarations are easily detectable in a code 
 review.

 [...]

If we were designing a new language from scratch, I would agree 
100% with your reasoning.

The problem is that there are un-annotated declarations in 
existing code that have already been reviewed, committed, and 
published under the assumption of  system-by-default. Those 
declarations need to be flagged for re-review in order to avoid 
introducing silent safety violations to existing D projects.

To address this issue, I'm working on a PR to have dmd emit a 
diagnostic message when it encounters an un-annotated external 
function declaration. Feel free to drop by and comment:

https://github.com/dlang/dmd/pull/11176

Atila Neves <atila.neves gmail.com> writes:

On Sunday, 24 May 2020 at 16:44:01 UTC, Paul Backus wrote:
 On Sunday, 24 May 2020 at 03:28:25 UTC, Walter Bright wrote:
 I'd like to emphasize:

 1. It is not possible for the compiler to check any 
 declarations where the implementation is not available. Not in 
 D, not in any language. Declaring a declaration safe does not 
 make it safe.

 2. If un-annotated declarations cause a compile time error, it 
 is highly likely the programmer will resort to "greenwashing" 
 - just slapping  safe on it. I've greenwashed code. Atila has. 
 Bruce Eckel has. We've all done it. Sometimes even for good 
 reasons.

 3. Un-annotated declarations are easily detectable in a code 
 review.

 [...]

 If we were designing a new language from scratch, I would agree 
 100% with your reasoning.

 The problem is that there are un-annotated declarations in 
 existing code that have already been reviewed, committed, and 
 published under the assumption of  system-by-default. Those 
 declarations need to be flagged for re-review in order to avoid 
 introducing silent safety violations to existing D projects.

I share your concerns on this, but disagree on the likelihood of 
reviews having gone by under the assumption of  system by 
default. I doubt most people even thought about 
 safe/ trusted/ system, and that's assuming anyone reviewed the 
code in the first place.

A few years ago I submitted several PRs to Phobos to mark all 
unittests that could with  safe explicitly. I'd say that was a 
good example of nobody reviewing them for their  systemness.

Paul Backus <snarwin gmail.com> writes:

On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 On Sunday, 24 May 2020 at 16:44:01 UTC, Paul Backus wrote:
 If we were designing a new language from scratch, I would 
 agree 100% with your reasoning.

 The problem is that there are un-annotated declarations in 
 existing code that have already been reviewed, committed, and 
 published under the assumption of  system-by-default. Those 
 declarations need to be flagged for re-review in order to 
 avoid introducing silent safety violations to existing D 
 projects.

 I share your concerns on this, but disagree on the likelihood 
 of reviews having gone by under the assumption of  system by 
 default. I doubt most people even thought about 
  safe/ trusted/ system, and that's assuming anyone reviewed the 
 code in the first place.

 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a 
 good example of nobody reviewing them for their  systemness.

Walter's claim was that "un-annotated declarations are easily 
detectable in code review." The intent of my response was simply 
to point out that for existing D code, the opportunity for such 
review, and therefore for the detection of such declarations, is 
already in the past.

In any case, your response does not change my conclusion: the 
compiler must warn D programmers that their declarations need to 
be reviewed for compatibility with  safe-by-default. Whether they 
were reviewed before that or not makes no difference.

Clarice <cl ar.ice> writes:

On Monday, 25 May 2020 at 16:47:50 UTC, Paul Backus wrote:
 the compiler must warn D programmers that their declarations 
 need to be reviewed for compatibility with  safe-by-default. 
 Whether they were reviewed before that or not makes no 
 difference.

I'm not an engineer; I'm not learned in CS. So some clarification 
on this would be nice: if you're pulling in C code, or any code 
written in an unsafe language, wouldn't the developers doing such 
a thing already be aware they need to do their due diligence? 
Foreign code isn't just going to magically appear in your 
codebase, right? Maybe through dependencies, but one should check 
those too, eh? (And if the dependency tree is needlessly large, 
then the language has another, maybe bigger, problem à la the NPM 
ecosystem.)

Paul Backus <snarwin gmail.com> writes:

On Monday, 25 May 2020 at 17:09:50 UTC, Clarice wrote:
 On Monday, 25 May 2020 at 16:47:50 UTC, Paul Backus wrote:
 the compiler must warn D programmers that their declarations 
 need to be reviewed for compatibility with  safe-by-default. 
 Whether they were reviewed before that or not makes no 
 difference.

 I'm not an engineer; I'm not learned in CS. So some 
 clarification on this would be nice: if you're pulling in C 
 code, or any code written in an unsafe language, wouldn't the 
 developers doing such a thing already be aware they need to do 
 their due diligence? Foreign code isn't just going to magically 
 appear in your codebase, right? Maybe through dependencies, but 
 one should check those too, eh? (And if the dependency tree is 
 needlessly large, then the language has another, maybe bigger, 
 problem à la the NPM ecosystem.)

Even if you did review your code *and* all of your dependencies, 
if you did it before DIP 1028 was accepted (i.e., any time in the 
last 10 years or so), you may have deliberately left external 
 system functions un-annotated, because  system was the default.

In fact, I *know* that people have done this, because I've been 
running my patched version of the compiler [1] against the dmd, 
druntime, and phobos codebases, and I have found an enormous 
number of such un-annotated declarations. Here are just a few of 
them in druntime:

https://github.com/dlang/druntime/pull/3117

Lots and lots of projects in the D ecosystem are going to need 
similar patches to be compatible with DIP 1028. And I don't think 
it's realistic to expect people to make these changes without 
some kind of help from the tooling.

[1] https://github.com/dlang/dmd/pull/11176

mw <mingwu gmail.com> writes:

On Monday, 25 May 2020 at 17:31:29 UTC, Paul Backus wrote:
 Even if you did review your code *and* all of your 
 dependencies, if you did it before DIP 1028 was accepted (i.e., 
 any time in the last 10 years or so), you may have deliberately 
 left external  system functions un-annotated, because  system 
 was the default.

 In fact, I *know* that people have done this, because I've been 
 running my patched version of the compiler [1] against the dmd, 
 druntime, and phobos codebases, and I have found an enormous 
 number of such un-annotated declarations. Here are just a few 
 of them in druntime:

 https://github.com/dlang/druntime/pull/3117

 Lots and lots of projects in the D ecosystem are going to need 
 similar patches to be compatible with DIP 1028. And I don't 
 think it's realistic to expect people to make these changes 
 without some kind of help from the tooling.

 [1] https://github.com/dlang/dmd/pull/11176


Can we declare the new D compiler version 3.0 for 
 safe-as-default? and keep version 2.x compiler for the old 
 system-as-default?

In industries, it is a big-NO-NO to break exiting *working* code 
base.

I've read some past threads talking about the instability of 
compilers (due to new feature introduced), e.g. in this 
on_leaving_d thread:

https://forum.dlang.org/thread/ncbawciyybdksecurmsc forum.dlang.org?page=1

https://gitlab.com/mihails.strasuns/blog/blob/master/articles/on_leaving_d.md

Let me quote what D's *industry users* (whose trademark is on the 
front page of dlang.org) have said 2 years ago:

"""
But when it comes to be boring and predictable, D fails terribly. 
You can't
assume that released features will become 100% usable (no bugs, 
sufficient docs
and tests, well-defined integration with other language features) 
even few
years after initial release. You can't assume that next compiler 
upgrade won't
suddenly break your project or any of its transitive 
dependencies. You can't
assume that any compiler version will be maintained for more than 
few months.
You can't assume there is any control over how declared vision
documents get executed in practice. You
can't trust any promises from language authors because they don't 
keep any track
of those. It is anarchy driven development in all its glory.
"""


I understand the desire of Walter to continue innovation and I 
appreciate it, and I also think this change from  system to 
 safe-as default is an important step for D's future direction, 
as I commented in my thread yesterday:

https://forum.dlang.org/post/brvexgjuqkbebizdmfjj forum.dlang.org

"""
-- the recent change  safe as default opt-in is good thing: 
without a  nogc phobos, nobody is going to write  system level 
software using D. I think this  safe change means the shift of 
D's focus from system software to application software. And I'm 
glad to see the the template project that dub generate starts 
with "app.d", which is the right direction for D.
"""

But on the other hand, we also need to take care of our existing 
D users, esp industry users, are they going to be happy about 
breaking their working code base with a new compiler version?

For example, we can define a metric: say, how many percent of the 
packages on https://code.dlang.org/ will continue work as-it-is 
*without* to make any change with the new  safe-as-default 
compiler. If > 95% of these packages (the 68–95–99.7 three-sigma 
rule of thumb 
https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule) 
require not any code change, then go ahead, release the new 
 safe-as-default compiler as 2.093.

Otherwise, I'd suggest to keep 2 versions of compiler:

-- version 2.x as stable compiler for (industry) users to use in 
production, and will only provide bug fixes, but no new language 
feature (changes) to the compiler. (And I believe the 2.x version 
D language has already be *feature rich enough* to beat all those 


-- version 3.x for Walter to experiment and explore new ideas, 
and warn all the users that version 3 is unstable compiler 
feature-wise, use-it-at-your-own-risk, your code base may need 
major change in the future.

Thoughts?

Paul Backus <snarwin gmail.com> writes:

On Monday, 25 May 2020 at 18:39:16 UTC, mw wrote:
 Can we declare the new D compiler version 3.0 for 
  safe-as-default? and keep version 2.x compiler for the old 
  system-as-default?

 safe-by-default is currently hidden behind a -preview switch. I 
think it's safe to assume there will be a long deprecation period 
before it's made the default, and even then, there will be a 
-revert switch to turn it off.

Clarice <cl ar.ice> writes:

On Monday, 25 May 2020 at 17:31:29 UTC, Paul Backus wrote:
 Even if you did review your code *and* all of your 
 dependencies, if you did it before DIP 1028 was accepted (i.e., 
 any time in the last 10 years or so), you may have deliberately 
 left external  system functions un-annotated, because  system 
 was the default.

 In fact, I *know* that people have done this, because I've been 
 running my patched version of the compiler [1] against the dmd, 
 druntime, and phobos codebases, and I have found an enormous 
 number of such un-annotated declarations. Here are just a few 
 of them in druntime:

 https://github.com/dlang/druntime/pull/3117

 Lots and lots of projects in the D ecosystem are going to need 
 similar patches to be compatible with DIP 1028. And I don't 
 think it's realistic to expect people to make these changes 
 without some kind of help from the tooling.

 [1] https://github.com/dlang/dmd/pull/11176

I really appreciate your contributions. (The latter of which will 
be quite handy.) But isn't the whole point of Walter's 
implementation of  safe-as-default is that FFIs are going to be 
assumed  safe? So they should still compile; that is unless 
you're referring to the need for re-evaluation.

Paul Backus <snarwin gmail.com> writes:

On Monday, 25 May 2020 at 19:35:04 UTC, Clarice wrote:
 I really appreciate your contributions. (The latter of which 
 will be quite handy.) But isn't the whole point of Walter's 
 implementation of  safe-as-default is that FFIs are going to be 
 assumed  safe? So they should still compile; that is unless 
 you're referring to the need for re-evaluation.

Yes, that's Walter's current plan. I am hoping that he'll change 
his mind about allowing un-annotated FFIs to be assumed  safe, 
but even if he doesn't, having the compiler print a warning will 
allow those of us who care about safety to find them and handle 
them properly.

Clarice <cl ar.ice> writes:

On Monday, 25 May 2020 at 19:44:00 UTC, Paul Backus wrote:
 On Monday, 25 May 2020 at 19:35:04 UTC, Clarice wrote:
 I really appreciate your contributions. (The latter of which 
 will be quite handy.) But isn't the whole point of Walter's 
 implementation of  safe-as-default is that FFIs are going to 
 be assumed  safe? So they should still compile; that is unless 
 you're referring to the need for re-evaluation.

 Yes, that's Walter's current plan. I am hoping that he'll 
 change his mind about allowing un-annotated FFIs to be assumed 
  safe, but even if he doesn't, having the compiler print a 
 warning will allow those of us who care about safety to find 
 them and handle them properly.

I'm still unsure which implementation is the right one, but I 
think this middle-ground you're attempting to get merged is a 
good compromise. So, again, thank you!

Panke <tobias pankrath.net> writes:

On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a 
 good example of nobody reviewing them for their  systemness.

Ideally you should be able to blindly mark every function 
definition with  safe, because the compiler will catch you if you 
fall. Only if you type  trusted you should need to be careful.

Atila Neves <atila.neves gmail.com> writes:

On Monday, 25 May 2020 at 17:01:24 UTC, Panke wrote:
 On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a 
 good example of nobody reviewing them for their  systemness.

 Ideally you should be able to blindly mark every function 
 definition with  safe, because the compiler will catch you if 
 you fall. Only if you type  trusted you should need to be 
 careful.

Doesn't work for templated functions since their  safety might 
depend on the the particular instantiation (consider 
std.algorithm.map).

Timon Gehr <timon.gehr gmx.ch> writes:

On 26.05.20 13:09, Atila Neves wrote:
 On Monday, 25 May 2020 at 17:01:24 UTC, Panke wrote:
 On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a good 
 example of nobody reviewing them for their  systemness.

 Ideally you should be able to blindly mark every function definition 
 with  safe, because the compiler will catch you if you fall. Only if 
 you type  trusted you should need to be careful.

 
 Doesn't work for templated functions since their  safety might depend on 
 the the particular instantiation (consider std.algorithm.map).

I think the point was that annotating with  safe liberally should not 
enable memory corruption.

Arine <arine1283798123 gmail.com> writes:

On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
 I share your concerns on this, but disagree on the likelihood 
 of reviews having gone by under the assumption of  system by 
 default. I doubt most people even thought about 
  safe/ trusted/ system, and that's assuming anyone reviewed the 
 code in the first place.

 A few years ago I submitted several PRs to Phobos to mark all 
 unittests that could with  safe explicitly. I'd say that was a 
 good example of nobody reviewing them for their  systemness.

I don't think a community run project without a lead or team 
behind it is a very good example. No one is going to spend their 
free time reviewing code ensuring it is memory safe. I've seen it 
such that unit tests in phobos/druntime weren't being run on some 
platforms, and there were tests that failed when run.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

The DIP is trying to accomplish (copied from its Rationale):

A. Costs of unsafe code have become ever more apparent and expensive, 
and  safe has grown more capable. Users expect safety to be opt-out, not 
opt-in.

B. Most code should be naturally safe, and system code should be 
relatively rare. It makes sense that the common case - safe code - 
should be the default and not require an annotation.

Any judgment on the regime of extern(C) functions should be judged as 
follows: "To what extent does the chosen regime of C functions push the 
rationale forward?"

The decision fails to support (A) because it keeps safety of C functions 
opt-in. It also fails to support (B) because it keeps code that does not 
require an annotation yet is not safe.

A good process would simply go back to the drawing board once the matter 
is exposed. It is very easy for reasonable people to agree that there is 
no part of rationale that justifies the decision. Either redefine the 
rationale (adding e.g. "keep some applications compilable without 
change" as an important goal), or change the handing of C functions.

The core argument recently detailed is:

A. Define "greenwashing" as a quick workaround to get code to work.
B. Describe in excruciating detail the experience with Java exceptions.
C. Establish a vague analogy with people placing  trusted on C declarations.
D. Assume without stating that the user would do "greenwashing" but the 
compiler doing it surreptitiously is somehow exempt.
E. Conclude that extern(C) code must be trusted.

This argument is already a smorgasbord of fallacies. It iterates a good 
subset of https://en.wikipedia.org/wiki/List_of_fallacies, and I am not 
kidding when I say a few edits to those pages with backreferences to 
this thread would be appropriate.

To start:

False analogy: making extern(C) function system is analogous with the 
Java exeception specification debacle.

https://en.wikipedia.org/wiki/Argument_from_analogy#False_analogy

This is a false analogy belying what seems to be a misunderstanding of 
the problem with Java exceptions: it was a maintenance problem, not an 
introduction problem.

The matter was, during maintenance, routinely code would change the set 
of exceptions they threw. So maintainers would need to touch code that 
didn't belong to them. There was also no "shrinking" of exception 
specification lists because the compiler does not enforce 
larger-than-necessary lists. So these lists would just grow forever 
during maintenance, with no real information attached and no usefulness.

None of these apply to  safe. Functions don't change their safety with 
ease, and when they do you definitely want to be informed. There's no 
growth of attributes for obvious reasons.

To the extent the analogy applies, the burden of proof is entirely on 
the DIP, and in this case the differences are so many and large, it's 
not worth making the argument by analogy in the first place.

The excruciatingly long discussion of Java exception specifications is 
an instance of another fallacy, misleading vividness:

http://nizkor.com/features/fallacies/misleading-vividness.html

It was expected that any of the bad things about exceptions would count 
as yet another good argument for the DIP. This constructs a textbook 
straw man:

https://en.wikipedia.org/wiki/Straw_man

So the argument constructs a straw man (Java exception specifications), 
discusses it at length on a long tangent (misleading vividness), to then 
conclude (by false analogy) that we don't want extern(C) functions to be 
 system.

What's missing is, of course, an explanation on how the decision 
supports the very rationale of the DIP. Which it doesn't - it factually 
works straight against it.

I decided to make this public only once the second boot fell (read on):

On 5/23/20 11:28 PM, Walter Bright wrote:
 I'd like to emphasize:
 
 1. It is not possible for the compiler to check any declarations where 
 the implementation is not available. Not in D, not in any language. 
 Declaring a declaration safe does not make it safe.

Strawman: https://en.wikipedia.org/wiki/Straw_man

Nobody asked for that, and listing it as an argument is pointing at a 
strawman.

 2. If un-annotated declarations cause a compile time error, it is highly 
 likely the programmer will resort to "greenwashing" - just slapping 
  safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've 
 all done it. Sometimes even for good reasons.

Nirvana fallacy: https://en.wikipedia.org/wiki/Nirvana_fallacy

"Electric cars still use polluting materials and processes, so let's 
continue making gasoline cars."

"People will add workarounds to code, so let's accept the dangerous code 
anyway."

 3. Un-annotated declarations are easily detectable in a code review.

Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion

(Um... there's no grep for "find me all un-annnotated declarations")

 4. Greenwashing is not easily detectable in a code review.

Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion

(Um... grep ' safe| trusted')

 5. Greenwashing doesn't fix anything. The code is not safer. It's an 
 illusion, not a guarantee.

Shifting the burden of proof from the DIP to the reviewer: 
https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy)

The DIP needs to prove it does not perform greenwashing itself. Which it 
factually does - it paints all C code as safe with no user intervention.
"Quod licet bovis non licet Iovis" much? That smacks of special pleading:

https://en.wikipedia.org/wiki/Special_pleading

Of course we have the strawman: https://en.wikipedia.org/wiki/Straw_man. 
Nobody asked for greenwashing.

 6. If someone cares to annotate declarations, it means he has at least 
 thought about it, because he doesn't need to. Hence it's more likely to 
 be correct than when greenwashed.

So... requiring adding annotations to extern(C) is good?

 7. D should *not* make it worthwhile for people to greenwash code.

Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion

No accounting for the fact that the very built-in rule does exactly 
greenwashing as the post seems to define it.

 It is, in a not-at-all obvious way, safer for C declarations to default 
 to being safe.

In ensemble, all this adds up to a good Kettle logic fallacy:

https://en.wikipedia.org/wiki/Kettle_logic

For the record, I think adoption of this rule works against the very 
thing that the DIP is trying to accomplish and will also cause a serious 
PR blow.

Arine <arine1283798123 gmail.com> writes:

Wowza, wow. The voice of reason returns!

Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:

On Monday, 25 May 2020 at 23:39:33 UTC, Andrei Alexandrescu wrote:
 [..]

Thank you, Andrei, you've put this quite eloquently. With more 
than 200+ replies, unfortunately, this whole discussion looks 
like an excessively inefficient use of the community's time.

One way to resolve this stalemate (which I've proposed in another 
post) is to split off the point of contention from DIP1028. In 
other words, amend the newly proposed compiler switch 
`-preview=safedefault` to not change the meaning of non-extern(D) 
function declarations and introduce a new compiler switch 
`-implicitly-{safe,trusted}-extern-fn-decls` (*) that adds the 
automatic "greenwashing" functionality that Walter desires so 
much.

(*) I think that `-implicitly-safe-extern-fn-decls` would be lie, 
but `-implicitly-trusted-extern-fn-decls` I can tolerate.

NaN <divide by.zero> writes:

On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov [ZombineDev] 
wrote:
 On Monday, 25 May 2020 at 23:39:33 UTC, Andrei Alexandrescu 
 wrote:
 [..]

 Thank you, Andrei, you've put this quite eloquently. With more 
 than 200+ replies, unfortunately, this whole discussion looks 
 like an excessively inefficient use of the community's time.

 One way to resolve this stalemate (which I've proposed in 
 another post) is to split off the point of contention from 
 DIP1028. In other words, amend the newly proposed compiler 
 switch `-preview=safedefault` to not change the meaning of 
 non-extern(D) function declarations and introduce a new 
 compiler switch `-implicitly-{safe,trusted}-extern-fn-decls` 
 (*) that adds the automatic "greenwashing" functionality that 
 Walter desires so much.

If the greenwashing part was separated and delayed it would give 
time to find out if Walters hypothesis about people just doing it 
themselves is true.

Atila Neves <atila.neves gmail.com> writes:

On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote:
 On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov 
 [ZombineDev] wrote:
 [...]

 If the greenwashing part was separated and delayed it would 
 give time to find out if Walters hypothesis about people just 
 doing it themselves is true.

-preview=safe now
-revert=safe "tomorrow"

NaN <divide by.zero> writes:

On Tuesday, 26 May 2020 at 12:51:59 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote:
 On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov 
 [ZombineDev] wrote:
 [...]

 If the greenwashing part was separated and delayed it would 
 give time to find out if Walters hypothesis about people just 
 doing it themselves is true.

 -preview=safe now
 -revert=safe "tomorrow"

Does that separate "safe by default" from "c functions are 
assumed safe"?

Bruce Carneal <bcarneal gmail.com> writes:

On Tuesday, 26 May 2020 at 12:51:59 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 12:28:06 UTC, NaN wrote:
 On Tuesday, 26 May 2020 at 06:55:31 UTC, Petar Kirov 
 [ZombineDev] wrote:
 [...]

 If the greenwashing part was separated and delayed it would 
 give time to find out if Walters hypothesis about people just 
 doing it themselves is true.

 -preview=safe now
 -revert=safe "tomorrow"

If these are permanent, an ugly fork will have occurred in all 
but name.

If these are temporary, we'll have gained little since the damage 
caused by assuming human checked routines are  safe will remain 
after the switches are removed.

Bruce Carneal <bcarneal gmail.com> writes:

On Friday, 22 May 2020 at 01:22:19 UTC, Walter Bright wrote:
 I have made these points before, but I'll summarize them here
 for convenient referral.

  [big snip of very long and arguably tangential Java screw-up and 
other]

 How does this relate to safe by default?

 Consider the common (because that's how D started out) case of:

 ----- clibrary.d --------

     T massage_data(... many parameters ...);
     ... 200 more such declarations ...

 ----- app.d ----------

     import clibrary;

     void useClibrary( ... parameters ...) {
         massage_data(parameters);
     }

 ---------------------

 This code, today, does not use annotations and it works. It's 
 been working
 for a long time. Now, we implement the amendment, and it stops 
 compiling
 because useClibrary is  safe and massage_data is  system. The 
 user is faced
 with the following alternatives:

 1. Go through 200 functions in clibrary.d and determine which 
 are  safe
 and which are  system. This is what we want them to do. We try 
 to motivate
 this with compiler error messages. Unfortunately, this is both 
 tedious and
 thoroughly impractical, as our poor user Will Not Know which 
 are safe and
 which are system. We can correctly annotate core.stdc.stdio 
 because I know
 those functions intimately. This is not true for other system C 
 APIs, and
 even less true for some third party C library we're trying to 
 interface to.

So, to condense and hopefully clarify: we can not use option 1) 
and  system by default because "our poor user Will Not Know which 
are safe and which are system"?
IOW, because someone unable to comment on safety would be 
inconvenienced we enable them to inject more code into the 
ecosphere by making  safe the default.

What am I missing?

 2. Annotate useClibrary() as  trusted or  system. While easier, 
 this causes
 all benefits to  safe by default to be lost.

 3. Wrap the call to massage_data() with:

     ()  trusted { massage_data(parameters); } ();

 If there are a lot of calls to clibrary, this is going to look 
 pretty awful.
 Nobody likes writing or reading such ugly code. It's ok here 
 and there, but
 not as a general thing.

 4. Edit clibrary.d and make the first line:

      safe:

[snip regarding greenwashing and other]

Paul Backus <snarwin gmail.com> writes:

On Thursday, 21 May 2020 at 13:51:34 UTC, Mike Parker wrote:
 DIP 1028, "Make  safe the Default", has been accepted without 
 comment.

 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1028.md

I've made a draft PR to try and address the potential safety 
issue discussed in this thread. Any and all constructive feedback 
is greatly appreciated, so feel free to comment:

https://github.com/dlang/dmd/pull/11176

Rivet <ri v.et> writes:

Perhaps not the ideal solution, but would a compiler flag, e.g. 
--strict-safe, that ensures the compiler errors on un- trusted 
non-D function prototypes be appropriate? This way, those who do 
work on mission-critical stuff can feel a tad better.

Bastiaan Veelo <Bastiaan Veelo.net> writes:

I think there is an exit to the current impasse that has been 
overlooked thus far.

The key question is, during a safety audit, how to find the 
unsafe sections of code that need to be vetted thoroughly. If DIP 
1028 is kept as is, these sections can be found by `grep 
 trusted|extern\(C`. If DIP 1028 would be amended as many argue 
for, it would be `grep  trusted`. But isn't grep a rather crude 
tool, and wouldn't the compiler be able to create more useful 
reports either way?

The compiler already has options for listing gc allocations 
(-vgc) and variables that go into thread local storage (-vtls). A 
new option could be implemented to list all  trusted functions 
and all calls into extern(C[++]) functions from within safe 
sections.

It is the calls into extern(C[++]) functions that cannot be 
guaranteed to be safe, irrespective of whether the declarations 
are marked  trusted or  safe. So listing these calls is much more 
valuable than grepping for  trusted, which can be misused as 
" trusted extern(...". If a C library has a proper D wrapper that 
defines a safe interface, only the  trusted functions in the 
wrapper would appear in the list, not the extern(C) calls that 
are made within its  trusted functions. The list could possibly 
be reduced by excluding functions that are never called, making 
it even more valuable.

 trusted is not a seal of approval. It bears no signature, nor a 
hash of the implementation. It is simply a bridge from safe 
territory into unsafe territory, and is a checkpoint that cannot 
be skipped in any certification process.

Considering this, the compiler can actually produce a hash of the 
implementation of trusted functions as part of this list. A 
certification authority can then actually use this list to 
maintain a table of sections, hashes and stamps. If a new release 
needs to renew its certification, the earlier table can be 
compared against the current list, and functions whose hash 
didn't change need not be reviewed again. Extrapolating even 
further, this could be the basis of a repository of approvals 
from various individuals and authorities representing statements 
like "I have looked at this function carefully and I believe it 
can indeed be trusted".

I think this would be a tool that adds real practical value and 
helps to reduce the cost of audits. And not the least, regarding 
the current discussion, it diminishes the importance of whether 
extern(C[++]) declarations are actually  system or  safe.

-- Bastiaan.

Bruce Carneal <bcarneal gmail.com> writes:

On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:
 [snipped an outline of tooling to mitigate 1028 damage]

 I think this would be a tool that adds real practical value and 
 helps to reduce the cost of audits. And not the least, 
 regarding the current discussion, it diminishes the importance 
 of whether extern(C[++]) declarations are actually  system or 
  safe.

Yes.  Tooling is good and will be much appreciated if 1028 
stands.  Reducing the need for tooling is even better.

 safe: the compiler checks
 safe post 1028: the compiler checks, sometimes, just not in the 
scary parts

Bastiaan Veelo <Bastiaan Veelo.net> writes:

On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:
 [snipped an outline of tooling to mitigate 1028 damage]

 I think this would be a tool that adds real practical value 
 and helps to reduce the cost of audits. And not the least, 
 regarding the current discussion, it diminishes the importance 
 of whether extern(C[++]) declarations are actually  system or 
  safe.

 Yes.  Tooling is good and will be much appreciated if 1028 
 stands.  Reducing the need for tooling is even better.

  safe: the compiler checks

The compiler does not and cannot check inside  trusted. Whether 
or not one requires extern(C[++]) to be behind or within  trusted 
does not change what the compiler can or cannot check.

  safe post 1028: the compiler checks, sometimes, just not in 
 the scary parts

The amount of code that requires human auditing remains the same. 
What matters is how to find that code, and how to maintain the 
validity of the audits.

-- Bastiaan.

Bruce Carneal <bcarneal gmail.com> writes:

On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:

  safe: the compiler checks

 The compiler does not and cannot check inside  trusted. Whether 
 or not one requires extern(C[++]) to be behind or within 
  trusted does not change what the compiler can or cannot check.j

Completely agree but my above says nothing about  trusted.

  safe post 1028: the compiler checks, sometimes, just not in 
 the scary parts

 The amount of code that requires human auditing remains the 
 same. What matters is how to find that code, and how to 
 maintain the validity of the audits.

Another distinction: pre 1028 your compilation will error out.  
Post 1028 it will not.

Atila Neves <atila.neves gmail.com> writes:

On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:

  safe: the compiler checks

 The compiler does not and cannot check inside  trusted. 
 Whether or not one requires extern(C[++]) to be behind or 
 within  trusted does not change what the compiler can or 
 cannot check.j

 Completely agree but my above says nothing about  trusted.

  safe post 1028: the compiler checks, sometimes, just not in 
 the scary parts

 The amount of code that requires human auditing remains the 
 same. What matters is how to find that code, and how to 
 maintain the validity of the audits.

 Another distinction: pre 1028 your compilation will error out.  
 Post 1028 it will not.

Quite the opposite. Most code out there isn't marked as 
 safe/ trusted/ system. If I add a dub dependency and don't 
bother with  safe, I can call anything, in any way. Post 1028... 
nope. Unmarked  system function definitions themselves won't 
compile.

Bruce Carneal <bcarneal gmail.com> writes:

On Tuesday, 26 May 2020 at 16:20:23 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 [...]

 Completely agree but my above says nothing about  trusted.

 [...]

 Another distinction: pre 1028 your compilation will error out.
  Post 1028 it will not.

 Quite the opposite. Most code out there isn't marked as 
  safe/ trusted/ system. If I add a dub dependency and don't 
 bother with  safe, I can call anything, in any way. Post 
 1028... nope. Unmarked  system function definitions themselves 
 won't compile.

Currently a machine checked  safe function calling an unannotated 
extern C routine will error out during compilation. This is great 
as the C routine was not machine checked, and generally can not 
be checked.  Post 1028, IIUC, the compilation will go through 
without complaint.  This seems quite clear.  What am I missing?

Bastiaan Veelo <Bastiaan Veelo.net> writes:

On Tuesday, 26 May 2020 at 16:31:57 UTC, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during compilation. 
 This is great as the C routine was not machine checked, and 
 generally can not be checked.  Post 1028, IIUC, the compilation 
 will go through without complaint.  This seems quite clear.  
 What am I missing?

I agree that being forced to think about a trusted interface to 
that routine can contribute to safety. Not getting this is the 
price if DIP 1028 as is. Doing it the other way bears a different 
price.

Still, it does not change the amount of code that must be vetted 
during an audit, either way.

-- Bastiaan.

Bruce Carneal <bcarneal gmail.com> writes:

On Tuesday, 26 May 2020 at 20:38:17 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 16:31:57 UTC, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during 
 compilation. This is great as the C routine was not machine 
 checked, and generally can not be checked.  Post 1028, IIUC, 
 the compilation will go through without complaint.  This seems 
 quite clear.  What am I missing?

 I agree that being forced to think about a trusted interface to 
 that routine can contribute to safety. Not getting this is the 
 price if DIP 1028 as is. Doing it the other way bears a 
 different price.

 Still, it does not change the amount of code that must be 
 vetted during an audit, either way.

If you pick up 100% of the code that the post 1028 compiler lied 
about, then I agree. Unsafe is unsafe even if your compiler stays 
mum.

If you pick up less than 100%, I disagree.  In that situation the 
compiler will have "helped" you to reduce the amount of code to 
be audited.

And finally, were the 1028 extern!(D) default changed to  system, 
I disagree.  The amount of code in non-greenwashed  trusted 
wrappers should be smaller.  (note that I discount the one time 
audit of stable  system libraries)

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/26/20 12:31 PM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated extern 
 C routine will error out during compilation. This is great as the C 
 routine was not machine checked, and generally can not be checked.  Post 
 1028, IIUC, the compilation will go through without complaint.  This 
 seems quite clear.  What am I missing?

If that's the case, it's the death of DIP 1028.

Bruce Carneal <bcarneal gmail.com> writes:

On Wednesday, 27 May 2020 at 02:58:16 UTC, Andrei Alexandrescu 
wrote:
 On 5/26/20 12:31 PM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during 
 compilation. This is great as the C routine was not machine 
 checked, and generally can not be checked.  Post 1028, IIUC, 
 the compilation will go through without complaint.  This seems 
 quite clear.  What am I missing?

 If that's the case, it's the death of DIP 1028.

Walter has confirmed that this is indeed the case.  As you can 
read a few posts up his response to my "What am I missing?" query 
was "Nothing at all."

Yes, it's really that bad.

Your help in undoing this mistake would be greatly appreciated by 
me and by the authors of, literally, hundreds of posts in this 
thread opposing the DIP.

=?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:

On 2020-05-27 06:59:28 +0000, Bruce Carneal said:

 Walter has confirmed that this is indeed the case.  As you can read a 
 few posts up his response to my "What am I missing?" query was "Nothing 
 at all."
 
 Yes, it's really that bad.

Will it be possible to see a report of these "greenwashed" call-chains, 
so that at least there is a chance to detect them?

-- 
Robert M. M�nch
http://www.saphirion.com
smarter | better | faster

Bruce Carneal <bcarneal gmail.com> writes:

On Friday, 29 May 2020 at 06:55:07 UTC, Robert M. Münch wrote:
 On 2020-05-27 06:59:28 +0000, Bruce Carneal said:

 Walter has confirmed that this is indeed the case.  As you can 
 read a few posts up his response to my "What am I missing?" 
 query was "Nothing at all."
 
 Yes, it's really that bad.

 Will it be possible to see a report of these "greenwashed" 
 call-chains, so that at least there is a chance to detect them?

There was indeed activity on the "lets at least make this 
visible" front until Walter withdrew 1028 about 2 hours ago.  The 
thread title in the Announce forum is:

  DIP 1028 "Make  safe the default" is dead

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Tuesday, May 26, 2020 8:58:16 PM MDT Andrei Alexandrescu via Digitalmars-d-
announce wrote:
 On 5/26/20 12:31 PM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated extern
 C routine will error out during compilation. This is great as the C
 routine was not machine checked, and generally can not be checked.  Post
 1028, IIUC, the compilation will go through without complaint.  This
 seems quite clear.  What am I missing?

 If that's the case, it's the death of DIP 1028.

Walter has acknowledged the problem and seems to think that because it's the
programmer's responsibility to deal with extern(C) functions correctly
(since it's not possible for the compiler to do it), it's up to the
programmer to go and fix any existing code that should be marked  system and
isn't and that having the compiler incorrectly mark extern(C) declarations
as  safe isn't a big problem, because programmers need to be spending the
time to check them anyway. He's already created some PRs to try to fix some
issues with extern(C) declarations in druntime and explicitly markingthem as
 system but doesn't seem to think that it's ultimately a big deal.

- Jonathan M Davis

Paul Backus <snarwin gmail.com> writes:

On Thursday, 28 May 2020 at 02:47:01 UTC, Jonathan M Davis wrote:
 Walter has acknowledged the problem and seems to think that 
 because it's the programmer's responsibility to deal with 
 extern(C) functions correctly (since it's not possible for the 
 compiler to do it), it's up to the programmer to go and fix any 
 existing code that should be marked  system and isn't and that 
 having the compiler incorrectly mark extern(C) declarations as 
  safe isn't a big problem, because programmers need to be 
 spending the time to check them anyway. He's already created 
 some PRs to try to fix some issues with extern(C) declarations 
 in druntime and explicitly markingthem as  system but doesn't 
 seem to think that it's ultimately a big deal.

 - Jonathan M Davis

I've submitted a PR [1] to fix a whole bunch of these. It's 
currently blocked on what appear to be irrelevant CI failures 
(one of which is actually acknowledged as such in the discussion 
on Walter's PR). No one has given it any attention.

Is there something I'm doing wrong? Should I be splitting my PR 
by file, to make it easier to review? I have a similar PR for 
Phobos in the works, and it would be nice to have some feedback 
on this so that one doesn't get ignored too.

[1] https://github.com/dlang/druntime/pull/3117

Walter Bright <newshound2 digitalmars.com> writes:

On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated extern C 
 routine will error out during compilation. This is great as the C routine was 
 not machine checked, and generally can not be checked.  Post 1028, IIUC, the 
 compilation will go through without complaint.  This seems quite clear. 
What am 
 I missing?

Nothing at all.

But I doubt there is much legacy non-compiling code around.

Bruce Carneal <bcarneal gmail.com> writes:

On Wednesday, 27 May 2020 at 05:49:49 UTC, Walter Bright wrote:
 On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during 
 compilation. This is great as the C routine was not machine 
 checked, and generally can not be checked.  Post 1028, IIUC, 
 the compilation will go through without complaint.  This seems 
 quite clear.  What am I missing?

 Nothing at all.

 But I doubt there is much legacy non-compiling code around.

The intent of my above was to illustrate the difference between 
pre 1028 compiler behavior and post 1028 compiler behavior.  I'm 
not at all concerned with legacy non-compiling code of this 
nature.  I am concerned about the change in compiler behavior.

As you've confirmed, post 1028 the compiler will accept an  safe 
routine calling in to an un-annotated extern C function without 
complaint.

Andre, I hope this is clear enough for you.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of this nature.

Apparently you agree it is not an actual problem.

Bastiaan Veelo <Bastiaan Veelo.net> writes:

On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of 
 this nature.

 Apparently you agree it is not an actual problem.

Really? I don't know if you really missed the point being made, 
or you're being provocative. Both seem unlikely to me.

-- Bastiaan.

Timon Gehr <timon.gehr gmx.ch> writes:

On 27.05.20 11:34, Bastiaan Veelo wrote:
 On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of this nature.

 Apparently you agree it is not an actual problem.

 
 Really? I don't know if you really missed the point being made, or 
 you're being provocative. Both seem unlikely to me.
 
 -- Bastiaan.

It's just selective reading and confirmation bias.

Walter did not read past the quoted sentence as it successfully 
slaughters the straw man he set up in his previous post.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/27/2020 2:34 AM, Bastiaan Veelo wrote:
 On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of this nature.

 Apparently you agree it is not an actual problem.

 
 Really? I don't know if you really missed the point being made, or you're
being 
 provocative. Both seem unlikely to me.

His argument was:

"Currently a machine checked  safe function calling an unannotated extern C 
routine will error out during compilation. This is great as the C routine was 
not machine checked, and generally can not be checked.  Post 1028, IIUC, the 
compilation will go through without complaint.  This seems quite clear.  What
am 
I missing?"

I replied that it was unlikely that such legacy code existed.

He replied that he was not concerned about it.

I.e. working legacy code is not going break.

Patrick Schluter <Patrick.Schluter bbox.fr> writes:

On Wednesday, 27 May 2020 at 10:46:11 UTC, Walter Bright wrote:
 On 5/27/2020 2:34 AM, Bastiaan Veelo wrote:
 On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of 
 this nature.

 Apparently you agree it is not an actual problem.

 
 Really? I don't know if you really missed the point being 
 made, or you're being provocative. Both seem unlikely to me.

 His argument was:

 "Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during compilation. 
 This is great as the C routine was not machine checked, and 
 generally can not be checked.  Post 1028, IIUC, the compilation 
 will go through without complaint.  This seems quite clear.  
 What am I missing?"

 I replied that it was unlikely that such legacy code existed.

 He replied that he was not concerned about it.

 I.e. working legacy code is not going break.

The legacy code is not the issue, never was.
It always was about unsafe code that will become  safe with that 
DIP.
Safe code is safe and DIP doesn't change that.
It's all about UNSAFE code becoming magically labelled SAFE by 
the compiler but that is still UNSAFE in reality.

Bruce Carneal <bcarneal gmail.com> writes:

On Wednesday, 27 May 2020 at 10:46:11 UTC, Walter Bright wrote:
 On 5/27/2020 2:34 AM, Bastiaan Veelo wrote:
 On Wednesday, 27 May 2020 at 09:09:58 UTC, Walter Bright wrote:
 On 5/26/2020 11:20 PM, Bruce Carneal wrote:
 I'm not at all concerned with legacy non-compiling code of 
 this nature.

 Apparently you agree it is not an actual problem.

 
 Really? I don't know if you really missed the point being 
 made, or you're being provocative. Both seem unlikely to me.

 His argument was:

 "Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during compilation. 
 This is great as the C routine was not machine checked, and 
 generally can not be checked.  Post 1028, IIUC, the compilation 
 will go through without complaint.  This seems quite clear.  
 What am I missing?"

 I replied that it was unlikely that such legacy code existed.

 He replied that he was not concerned about it.

 I.e. working legacy code is not going break.

You continue to miss the point.

Additionally, there never was any "working legacy code".  As 
established, the pre 1080 compiler would have rejected the code.  
Does-not-compile != working-code.

Bruce Carneal <bcarneal gmail.com> writes:

On Wednesday, 27 May 2020 at 13:50:25 UTC, Bruce Carneal wrote:
 On Wednesday, 27 May 2020 at 10:46:11 UTC, Walter Bright wrote:
 [...]

 You continue to miss the point.

 Additionally, there never was any "working legacy code".  As 
 established, the pre 1080 compiler would have rejected the 
 code.  Does-not-compile != working-code.

pre 1028 compiler

John Colvin <john.loughran.colvin gmail.com> writes:

On Wednesday, 27 May 2020 at 05:49:49 UTC, Walter Bright wrote:
 On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an 
 unannotated extern C routine will error out during 
 compilation. This is great as the C routine was not machine 
 checked, and generally can not be checked.  Post 1028, IIUC, 
 the compilation will go through without complaint.  This seems 
 quite clear.  What am I missing?

 Nothing at all.

 But I doubt there is much legacy non-compiling code around.

The point isn't that incorrect legacy code that didn't compile 
now does (although that does matter a bit), it's that newly 
written code will compile when it shouldn't.

Existing code will be full of extern(C) declarations that are 
implicitly and correctly  system now and will become  safe with 
dip1028, which means that when I write new  safe code calling 
those (could be through a deep dependency chain of 
inferred- safety APIs, multiple dub packages...) I could easily 
find my new code compiling when it shouldn't.


Effectively, by silently  safe-ing things you can't infer, you 
will be changing APIs from  system to  trusted without any checks.


Just in case there's any confusion, here's a timeline:

1. library A is written containing a dangerous but useful 
extern(C) declaration assuming  system by default.
2. application B is written for and compiled with dip1028,  safe: 
at the top of every file.
3. B adds a dependency on A. It continues to compile as  safe, 
calling an unsafe C function.


This seems like one of those things where it's either wrong or a 
showstopper.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/27/20 1:49 AM, Walter Bright wrote:
 On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated 
 extern C routine will error out during compilation. This is great as 
 the C routine was not machine checked, and generally can not be 
 checked.  Post 1028, IIUC, the compilation will go through without 
 complaint.  This seems quite clear.  What am I missing?

 
 Nothing at all.

That means safe by default is effectively loosening D's notion of safety.

This DIP must go.

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Wednesday, May 27, 2020 3:30:32 AM MDT Andrei Alexandrescu via Digitalmars-
d-announce wrote:
 On 5/27/20 1:49 AM, Walter Bright wrote:
 On 5/26/2020 9:31 AM, Bruce Carneal wrote:
 Currently a machine checked  safe function calling an unannotated
 extern C routine will error out during compilation. This is great as
 the C routine was not machine checked, and generally can not be
 checked.  Post 1028, IIUC, the compilation will go through without
 complaint.  This seems quite clear.  What am I missing?

 Nothing at all.

 That means safe by default is effectively loosening D's notion of safety.

 This DIP must go.

Which is exactly what most of us have been arguing for weeks (months?). It
either needs to go or be ammended so that non-extern(D) declarations
continue to be treated as  system instead of automatically becoming  safe
with the DIP.

The result of all of that arguing is that Walter accepted the DIP and then
started this thread as his more detailed reply when there were a ton of
complaints about the DIP's acceptance - and of course, you've already read
and replied to his reasoning.

As far as I can tell, Walter understands the issues but fundamentally
disagrees with pretty much everyone else on the issue. He seems to think
that weakening  safe is worth doing, because it will ultimately mean that
more code will be treated as  safe and mechnically checked by the compiler,
whereas most everyone else thinks that weakening  safe is unacceptable. But
since Walter managed to convince Atila, the DIP has been accepted.

- Jonathan M Davis

Daniel Kozak <kozzi11 gmail.com> writes:

On Thu, May 28, 2020 at 4:56 AM Jonathan M Davis via
Digitalmars-d-announce <digitalmars-d-announce puremagic.com> wrote:
 As far as I can tell, Walter understands the issues but fundamentally
 disagrees with pretty much everyone else on the issue.

I do not think so, the issue is, that there could be more people who
agree with Walter (like me),
but because we agree we do not participate.

 He seems to think
 that weakening  safe is worth doing, because it will ultimately mean that
 more code will be treated as  safe and mechnically checked by the compiler,

And I believe he is right.

 whereas most everyone else thinks that weakening  safe is unacceptable.

You make bad assumptions, who is "most everyone else"? I have only see
few  members of dlang arguing about that. But there is much bigger
community around there.

 But since Walter managed to convince Atila, the DIP has been accepted.

 - Jonathan M Davis

So everything is OK right?

Timon Gehr <timon.gehr gmx.ch> writes:

On 28.05.20 10:50, Daniel Kozak wrote:
 He seems to think
 that weakening  safe is worth doing, because it will ultimately mean that
 more code will be treated as  safe and mechnically checked by the compiler,

 And I believe he is right.
 

No, it's a false dichotomy. Weakening  safe to allow more code to be 
 safe might have been sensible if there was no  trusted annotation. 
However, as things stand,  trusted is sufficient as a tool to introduce 
potentially wrong assumptions about memory safety, we don't need more, 
especially not implicit ones.

The reason why people are not using  safe is partly that it is not the 
default, but it is mostly that their library dependencies _including 
Phobos_ are not properly annotated. This needs actual work to fix.

If there is significant perceived value in performing  safety checks in 
 system code, we can add a new function attribute that causes 
non-transitive  safe checks but otherwise gets treated as  system.  safe 
does not have to take this role.

Johannes Pfau <nospam example.com> writes:

Am Thu, 28 May 2020 10:50:44 +0200 schrieb Daniel Kozak:

 On Thu, May 28, 2020 at 4:56 AM Jonathan M Davis via
 Digitalmars-d-announce <digitalmars-d-announce puremagic.com> wrote:
 As far as I can tell, Walter understands the issues but fundamentally
 disagrees with pretty much everyone else on the issue.

 
 I do not think so, the issue is, that there could be more people who
 agree with Walter (like me),
 but because we agree we do not participate.

You can not really assume any opinion for people who did not participate, 
unless you can really prove why there's a bias. I did not participate 
either and I do not agree with Walter. So now we can say the opinions of 
those who did not participate in the discussion are split 50:50 ;-)

We could assume there's a slight bias of those agreeing with Walter not 
responding because they don't have to actively convince anyone, as the 
DIP has been accepted. But given how much negative feedbcak there is, it's 
also likely people would voice their opinion to support the decision. 
Really the best thing we can assume is that the opionions of those not 
participating are split in the same way as the ones of those who are 
participating. The strawpoll posted recently suggests that as well:
https://www.strawpoll.me/20184671/r

-- 
Johannes

jmh530 <john.michael.hall gmail.com> writes:

On Wednesday, 27 May 2020 at 05:49:49 UTC, Walter Bright wrote:
 [snip]

 Nothing at all.

 But I doubt there is much legacy non-compiling code around.

I cannot speak for all the code out there, but I know of at least 
one binding out there [1] that will need to get updated in light 
of this DIP.

I also put together a wrapper [2] that sits on top of that 
binding (many years ago and no idea if the wrapper still works (I 
see no reason why the binding wouldn't), so this probably does 
count as legacy code) and calls the binding functions with 
templates and no safety annotations. So theoretically, after the 
DIP is accepted and without any other changes, nlopt.d becomes 
 safe and the binding that calls it also becomes  safe and any 
user code that calls it becomes  safe.

***************************

On a separate note, the current definition of  safe is that you 
cannot call  system functions. This prevents calling unannotated 
extern(C) prototypes from  safe code.

Ignoring DIP 1028 for a second, would you consider changing the 
rules so unannotated extern(C) prototypes are  safe instead of 
 system without any other change? Just this change by itself 
would mean that  safe functions can call unannotated extern(C) 
prototypes. Obviously this by itself could be rejected for 
special casing. However, I think the underlying question is 
should they be safe (ignoring  safe) or not. An argument made 
previously is that they should be safe and thus  safe. However, I 
think the overwhelming belief is that they are not safe and thus 
should not be  safe by default.

 From this perspective, you could think of the DIP as doing two 
things. 1) it makes unannotated extern(C) prototypes  safe 
instead of  system and 2) it makes all other unannotated 
functions  safe instead of  system. The big focus is clearly on 



So, do you think unannotated exten(C) prototypes safe? If you 
think they are safe, then there is no issue with making them 
 safe by default. However, if you do not think they are safe, 
then it raises serious concerns to make them  safe by default.

[1] 
https://github.com/DlangScience/nlopt/blob/master/source/nlopt.d
[2] https://github.com/jmh530/nloptd/blob/master/source/nloptd.d

Gregory <g.thompson.1892 gmall.com> writes:

I'm curious what the distribution looks like between whether 
people agree that extern(C) should be  safe, or  system. People 
that think it should be  trusted, vote  safe, it's pretty much 
the same thing.

  https://www.strawpoll.me/20184671

Bruce Carneal <bcarneal gmail.com> writes:

On Tuesday, 26 May 2020 at 16:20:23 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 [...]

 The compiler does not and cannot check inside  trusted. 
 Whether or not one requires extern(C[++]) to be behind or 
 within  trusted does not change what the compiler can or 
 cannot check.j

 Completely agree but my above says nothing about  trusted.

 [...]

 The amount of code that requires human auditing remains the 
 same. What matters is how to find that code, and how to 
 maintain the validity of the audits.

 Another distinction: pre 1028 your compilation will error out.
  Post 1028 it will not.

 Quite the opposite. Most code out there isn't marked as 
  safe/ trusted/ system. If I add a dub dependency and don't 
 bother with  safe, I can call anything, in any way. Post 
 1028... nope. Unmarked  system function definitions themselves 
 won't compile.

I think I see the difficulty.  We both support 1028's elimination 
of the "I can call anything, in any way" default.  I really like 
 safe by default for machine checkable code.  I don't support 
 safe by default for extern C.

Gregory <g.thompson.1892 gmall.com> writes:

On Tuesday, 26 May 2020 at 16:20:23 UTC, Atila Neves wrote:
 On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo 
 wrote:

  safe: the compiler checks

 The compiler does not and cannot check inside  trusted. 
 Whether or not one requires extern(C[++]) to be behind or 
 within  trusted does not change what the compiler can or 
 cannot check.j

 Completely agree but my above says nothing about  trusted.

  safe post 1028: the compiler checks, sometimes, just not in 
 the scary parts

 The amount of code that requires human auditing remains the 
 same. What matters is how to find that code, and how to 
 maintain the validity of the audits.

 Another distinction: pre 1028 your compilation will error out.
  Post 1028 it will not.

 Quite the opposite. Most code out there isn't marked as 
  safe/ trusted/ system. If I add a dub dependency and don't 
 bother with  safe, I can call anything, in any way. Post 
 1028... nope. Unmarked  system function definitions themselves 
 won't compile.

Which will just lead people to pure  trusted: at the top of their 
code to get it to compile again, with or without extern(C) being 
 safe by default. Then someone that uses it as dependency will 
mistaken think it is  safe. What's to stop this kind of 
"greenwashing" and why is greenwashing only important to prevent 
when talking about extern(C) but every other code that will break 
from this change?

Paul Backus <snarwin gmail.com> writes:

On Tuesday, 26 May 2020 at 17:50:58 UTC, Gregory wrote:
 Which will just lead people to pure  trusted: at the top of 
 their code to get it to compile again, with or without 
 extern(C) being  safe by default. Then someone that uses it as 
 dependency will mistaken think it is  safe. What's to stop this 
 kind of "greenwashing" and why is greenwashing only important 
 to prevent when talking about extern(C) but every other code 
 that will break from this change?

The reason extern function declarations are particularly 
problematic is that changing them from  system-by-default to 
 safe-by-default can cause *silent* breakage in existing, correct 
code. By "silent", I mean that there is no message from the 
compiler warning you that the memory safety of the code has been 
compromised. It will still compile, and you will only find out 
that something has gone wrong when you run into memory corruption 
at run-time.

This undermines the entire purpose of  safe, which is to 
automatically catch code with potential memory errors at compile 
time.

Gregory <g.thompson.1892 gmall.com> writes:

On Tuesday, 26 May 2020 at 20:32:13 UTC, Paul Backus wrote:
 On Tuesday, 26 May 2020 at 17:50:58 UTC, Gregory wrote:
 Which will just lead people to pure  trusted: at the top of 
 their code to get it to compile again, with or without 
 extern(C) being  safe by default. Then someone that uses it as 
 dependency will mistaken think it is  safe. What's to stop 
 this kind of "greenwashing" and why is greenwashing only 
 important to prevent when talking about extern(C) but every 
 other code that will break from this change?

 The reason extern function declarations are particularly 
 problematic is that changing them from  system-by-default to 
  safe-by-default can cause *silent* breakage in existing, 
 correct code. By "silent", I mean that there is no message from 
 the compiler warning you that the memory safety of the code has 
 been compromised. It will still compile, and you will only find 
 out that something has gone wrong when you run into memory 
 corruption at run-time.

 This undermines the entire purpose of  safe, which is to 
 automatically catch code with potential memory errors at 
 compile time.

Exactly, that's what I think as well. What I'm trying to convey 
is that there's a lot of greenwashing that is going to happen. 
Code doesn't work, slapping  trusted everywhere til it works is 
what people are going to do, it doesn't matter if extern(C) is 
 safe or not. Greenwashing occurs no matter what. It's only 
natural with such a big breaking changing with a feature that you 
have to "trust".

Consider this:

Someone is writing code, it's  safe cause it is now the default 
(yay!). They come across unannotated code while searching for an 
answer. They copy paste it into their source, ops compile error! 
Can't use X in  safe code. Ok, so they annotate it as  system. Oh 
but now their  safe code can't call  system code. So what do they 
do? Annotate it as  trusted without checking it.

If Walter believed greenwashing was actually a problem, then the 
best solution to prevent it would be to not make  safe by 
default. If it's not that serious of a problem that he will push 
through  safe by default, then greenwashing isn't actually a 
problem and extern(C) should remain  system.

Paul Backus <snarwin gmail.com> writes:

On Tuesday, 26 May 2020 at 22:47:03 UTC, Gregory wrote:
 If Walter believed greenwashing was actually a problem, then 
 the best solution to prevent it would be to not make  safe by 
 default. If it's not that serious of a problem that he will 
 push through  safe by default, then greenwashing isn't actually 
 a problem and extern(C) should remain  system.

I understand your point now, and I agree.

Walter Bright <newshound2 digitalmars.com> writes:

On 5/26/2020 1:32 PM, Paul Backus wrote:
 The reason extern function declarations are particularly problematic is that 
 changing them from  system-by-default to  safe-by-default can cause *silent* 
 breakage in existing, correct code.

Can you post an example of currently compiling and correctly working code that 
will break?

Setting aside use of __traits(compiles, ...).

Timon Gehr <timon.gehr gmx.ch> writes:

On 27.05.20 07:54, Walter Bright wrote:
 On 5/26/2020 1:32 PM, Paul Backus wrote:
 The reason extern function declarations are particularly problematic 
 is that changing them from  system-by-default to  safe-by-default can 
 cause *silent* breakage in existing, correct code.

 
 Can you post an example of currently compiling and correctly working 
 code that will break?
 
 Setting aside use of __traits(compiles, ...).

What exactly is your standard here? Are you saying we have to produce 
the following?

- Monolithic example, API breakage does not count.

- No __traits(compiles, ...)

- The code has to compile under both old and new rules.

- The code has to corrupt memory in  safe code under new rules.

This is clearly not possible, exactly because the old  safe rules are 
stronger. But why exactly should API breakage not count?

Walter Bright <newshound2 digitalmars.com> writes:

On 5/27/2020 3:01 AM, Timon Gehr wrote:
 This is clearly not possible, exactly because the old  safe rules are
stronger. 

Thank you. We can agree on something.

 But why exactly should API breakage not count?

I've addressed exactly this a dozen times or more, to you and others. Repeating 
myself has become pointless.

It's fine to disagree with me. Argue that point. But don't say I didn't address
it.

Timon Gehr <timon.gehr gmx.ch> writes:

On 27.05.20 12:51, Walter Bright wrote:
 On 5/27/2020 3:01 AM, Timon Gehr wrote:
 This is clearly not possible, exactly because the old  safe rules are 
 stronger. 

 
 Thank you. We can agree on something.
 ...

I am not sure if you noticed that I agree with most of your points, just 
not about their relevance to the topic at hand.

 But why exactly should API breakage not count?

 
 I've addressed exactly this a dozen times or more, to you

No. You did not. I went over all of your responses to my posts again to 
make sure. Why are you making this claim?

I haven't made API breakage a central point to any of my previous posts 
and you did not address any of my criticism in any depth. As far as I 
can tell, the only point you engaged with was that  trusted is not 
greenwashing.

 and others. 

I don't think you did, but I am not going to check.

 Repeating myself has become pointless.
 
 It's fine to disagree with me. Argue that point. But don't say I didn't 
 address it.

As far as I remember, you did not address this specific point, but if I 
had to extrapolate your response from previous points you made I would 
expect your opinion to be that implicitly broken APIs should be fixed by 
universal manual review of not explicitly annotated functions and that 
this is better than the compiler catching it for you because this way 
only people who are competent to judge which annotation should be there 
will notice that it is missing.

Claude <claudemr live.fr> writes:

On Wednesday, 27 May 2020 at 10:51:54 UTC, Walter Bright wrote:
 On 5/27/2020 3:01 AM, Timon Gehr wrote:
 I've addressed exactly this a dozen times or more, to you and 
 others. Repeating myself has become pointless.

 It's fine to disagree with me. Argue that point. But don't say 
 I didn't address it.

I'm trying to understand the logic of " safe by default for 
extern declaration".

So I setup a simplified real-life example that I had in my mind...

Let's say we have a project consisting of:
- an ASM file containing several function definitions (assembly 
language scares people away, so it's a good example).
- a .di file containing the extern(C) declarations of the former 
ASM functions.
- and a .d file using those functions with  safe code.

No function is annotated.

Before DIP-1028, it compiles and link without annotations.

If extern(C) decl are  system by default, it will not compile 
anymore, and a careless programmer will slap a " trusted:" at the 
top of the ".di" file (greenwashing), and it'll stay there.

On the other hand, a careful programmer will need to annotate 
 trusted ONLY on functions that he actually trusts AND make the 
project compile anyway (because he cannot afford to leave his 
project broken). So what does he do to make it compile ? Should 
he slap a " trusted:" at the beginning of the file just like the 
careless programmer and maybe explicitly annotate individually 
declarations that are actually trusted ? It's weird...

However, if extern(C) decl are  safe by default, he does not need 
to slap a " trusted" at the top of the ".di" file, he just needs 
to annotate individual declarations with  trusted to advertise 
(to the QA, or his colleague or future self) that he reviewed 
successfully the function implementation, AND it will still 
compile.


So that example would go in favor of Walter's point. So is it a 
good example? Does pinpoint the point?

Because again, as the others said, it's still controversial, it's 
hard to sell, it's convoluted... I dunno...

Johannes T <isrvoid gmail.com> writes:

On Wednesday, 27 May 2020 at 10:51:54 UTC, Walter Bright wrote:
 [..]

I was reading Bastiaan's inspiring ideas about provable 
verification of  trusted and something occurred to me. If we are 
serious about safety, we should consider  trusted annotation on 
extern C harmful. It provides little value and might be 
misleading like a bad comment:
// only call once, second call leaks
void waitForClose(size_t msTimeout);
The implementation was rewritten long ago, no one bothered to 
remove the remark.
The trustworthiness of extern code can be assessed by audit. It 
can only be set in stone by additional tools. git can nail down 
the version of a verified function in the CMake list. The 
compiler can't know what we are linking against or what's loaded 
at runtime.
 trusted extern C is meaningless. Pre DIP1028 we had to put it in 
for  safe code to compile.
I'm not sure how  system fits into this. It gets outdated the 
same way  trusted does.

Paul Backus <snarwin gmail.com> writes:

On Wednesday, 27 May 2020 at 05:54:32 UTC, Walter Bright wrote:
 On 5/26/2020 1:32 PM, Paul Backus wrote:
 The reason extern function declarations are particularly 
 problematic is that changing them from  system-by-default to 
  safe-by-default can cause *silent* breakage in existing, 
 correct code.

 Can you post an example of currently compiling and correctly 
 working code that will break?

 Setting aside use of __traits(compiles, ...).

I am paraphrasing an earlier post of mine in this thread:

https://forum.dlang.org/post/yvcjpcxfyyobqubjrrxv forum.dlang.org

The breakage is that functions which are currently correctly 
considered  system will instead be considered  safe. As a result, 
future changes to  safe code are at risk of introducing memory 
corruption.

Strictly speaking, the breakage does not occur until those future 
changes are made, so it is more accurate to say that changing 
un-annotated extern declarations to  safe-by-default will put 
existing code *at risk* of silent breakage. Given the prevalence 
of such declarations in existing code (e.g. [1]), I think it is 
inevitable that this risk will in fact lead to breakage in 
practice, so I do not consider the distinction particularly 
important to draw in this context.

[1] https://github.com/dlang/druntime/pull/3117

Bastiaan Veelo <Bastiaan Veelo.net> writes:

On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo wrote:

  safe: the compiler checks

 The compiler does not and cannot check inside  trusted. 
 Whether or not one requires extern(C[++]) to be behind or 
 within  trusted does not change what the compiler can or 
 cannot check.

 Completely agree but my above says nothing about  trusted.

But it matters. Even if your code is  safe, that doesn't mean it 
is completely safe. Your  safe code can call into somebody else's 
 safe code that can call  trusted code that calls  system code. 
If you want to guarantee that your code is safe, you'll have to 
find and audit the  trusted bits (as well as everything that is 
called from there).

If extern(C) is implicitly  system, you'll have to make calls 
 trusted and they are subject to auditing. If extern(C) is 
implicitly  safe, calls are still subject to auditing. Whatever 
your preference, I think the compiler can be of greater help in 
finding the bits that require auditing than grep can, and then 
the difference isn't all that important anymore. Safe wrappers 
around C libraries are still written the same way, audited the 
same way, and unsafe calls into C caught the same way.

I agree that making extern(!D)  system looks sound, like the way 
it should be. But it breaks compilation of existing code, and it 
is too easy to evade by slapping on an explicit  trusted or (God 
forbid)  safe. Or we could make extern(!D) to be anything other 
than  system an error, which would be really annoying. I think 
that what we all strive for is to reduce the strain of coding as 
well as to reduce the strain of auditing.

-- Bastiaan.

Bruce Carneal <bcarneal gmail.com> writes:

On Tuesday, 26 May 2020 at 18:06:00 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 16:10:24 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:54:31 UTC, Bastiaan Veelo wrote:
 On Tuesday, 26 May 2020 at 15:39:11 UTC, Bruce Carneal wrote:
 On Tuesday, 26 May 2020 at 15:01:06 UTC, Bastiaan Veelo 
 wrote:

  safe: the compiler checks

 The compiler does not and cannot check inside  trusted. 
 Whether or not one requires extern(C[++]) to be behind or 
 within  trusted does not change what the compiler can or 
 cannot check.

 Completely agree but my above says nothing about  trusted.

 But it matters. Even if your code is  safe, that doesn't mean 
 it is completely safe. Your  safe code can call into somebody 
 else's  safe code that can call  trusted code that calls 
  system code. If you want to guarantee that your code is safe, 
 you'll have to find and audit the  trusted bits (as well as 
 everything that is called from there).

Agree that the transitive closure is difficult, especially for 
humans.

 If extern(C) is implicitly  system, you'll have to make calls 
  trusted and they are subject to auditing. If extern(C) is 
 implicitly  safe, calls are still subject to auditing. Whatever 
 your preference, I think the compiler can be of greater help in 
 finding the bits that require auditing than grep can, and then 
 the difference isn't all that important anymore. Safe wrappers 
 around C libraries are still written the same way, audited the 
 same way, and unsafe calls into C caught the same way.

Agree.   trusted doesn't mean that the code so labelled should be 
blindly trusted.  The aim should be reduction of  trusted where 
practical.  Silently letting all extern C functions inside the 
tent is not a reduction.

 I agree that making extern(!D)  system looks sound, like the 
 way it should be. But it breaks compilation of existing code, 
 and it is too easy to evade by slapping on an explicit  trusted 
 or (God forbid)  safe. Or we could make extern(!D) to be 
 anything other than  system an error, which would be really 
 annoying. I think that what we all strive for is to reduce the 
 strain of coding as well as to reduce the strain of auditing.

I'm all for reducing the strain of writing correct code.

As others have implied, we're dealing with, at least, two issues 
here: 1) any transition from opt-in to opt-out safety will break 
code.  Some programmers will resort to greenwashing in an effort 
to "fix" it.  Will we be better off post transition? And 2) 
should uncheckable code be considered  safe, ever.

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Thursday, May 28, 2020 2:50:44 AM MDT Daniel Kozak via Digitalmars-d-
announce wrote:
 On Thu, May 28, 2020 at 4:56 AM Jonathan M Davis via

 Digitalmars-d-announce <digitalmars-d-announce puremagic.com> wrote:
 As far as I can tell, Walter understands the issues but fundamentally
 disagrees with pretty much everyone else on the issue.

 I do not think so, the issue is, that there could be more people who
 agree with Walter (like me),
 but because we agree we do not participate.

There may be some silent people who agree, but in all the discussions on
this DIP, almost no one has agreed with Walter on this. It has not been a
back and forth discussion with some on Walter's side and some against. It's
been pretty much everyone against Walter. He did unfortunately manage to
convince Atila, so the DIP has been accepted, but based on the discussions,
I think that you may be the only person I've seen say anything positive
about the DIP treating extern(C) functions as  safe. The fact that  safe
becomes the default has garned some debate with some in favor and some
against (with most seeming to be in favor), but the idea of making extern(C)
declarations  safe by default has been almost universally considered a bad
idea by anyone who has responded on the topic. Most DIPs do not get anywhere
close to this level of negative feedback.

 But since Walter managed to convince Atila, the DIP has been accepted.

 So everything is OK right?

Everything is okay? Because a bad DIP got accepted? No, most definitely not.
Quite the opposite. With the DIP in its current state,  safe becomes a lie.
The compiler no longer guarantees that  safe code is memory safe so long as
it doesn't call any  trusted code where the programmer incorrectly marked it
as  trusted. Instead, the compiler blindly treats non-extern(D) declarations
as  safe and invisibly introduces memory safety bugs into  safe code.
Nothing about that is "OK." From the way things currently look, we're going
to have to deal with that hole in  safe in D code in the future, because the
DIP has been accepted, but it adds yet another dark corner to the language
of the sort that folks here tend to make fun of C++ for. Going forward,
we're going to have to be explaining to people why  safe code doesn't
actually guarantee that code is memory safe (in spite of the fact that
that's what it claims to do) and why any and all non-extern(D) declarations
have to be treated with extreme caution to avoid invisibly introducing
memory safety bugs into your code.

Walter is very intelligent and has made many great decisions with D, but
IMHO, this is definitely not one of them.

- Jonathan M Davis

Sebastiaan Koppe <mail skoppe.eu> writes:

On Thursday, 28 May 2020 at 09:21:09 UTC, Jonathan M Davis wrote:
 He did unfortunately manage to convince Atila, so the DIP has 
 been accepted, but based on the discussions, I think that you 
 may be the only person I've seen say anything positive about 
 the DIP treating extern(C) functions as  safe.

 - Jonathan M Davis

I think Walter had to make a tough call with many tradeoffs. The 
defining feature of engineering I would say.

Is he wrong? Maybe, I don't know. The obvious path is far from 
always being a winner.

If it does come back to haunt him, he can always add a DIP to 
make extern(!D)  system by default. It won't invalidate any work.

Johannes Pfau <nospam example.com> writes:

Am Thu, 28 May 2020 12:28:16 +0000 schrieb Sebastiaan Koppe:

 On Thursday, 28 May 2020 at 09:21:09 UTC, Jonathan M Davis wrote:
 He did unfortunately manage to convince Atila, so the DIP has been
 accepted, but based on the discussions, I think that you may be the
 only person I've seen say anything positive about the DIP treating
 extern(C) functions as  safe.

 - Jonathan M Davis

 
 I think Walter had to make a tough call with many tradeoffs. The
 defining feature of engineering I would say.
 
 Is he wrong? Maybe, I don't know. The obvious path is far from always
 being a winner.
 
 If it does come back to haunt him, he can always add a DIP to make
 extern(!D)  system by default. It won't invalidate any work.

This would be another round of massively breaking user code. And this is 
going to be exactly the argument that will be used to dismiss any DIP 
trying to change the defaults later on.


-- 
Johannes

Sebastiaan Koppe <mail skoppe.eu> writes:

On Thursday, 28 May 2020 at 16:01:35 UTC, Johannes Pfau wrote:
 Am Thu, 28 May 2020 12:28:16 +0000 schrieb Sebastiaan Koppe:
 If it does come back to haunt him, he can always add a DIP to 
 make extern(!D)  system by default. It won't invalidate any 
 work.

 This would be another round of massively breaking user code.

The breakage will be split in two rounds, but the amount of code 
needed to be modified would be the same as with the 
safe-by-default-except-for-extern-non-c. With the possibility of 
not needing the second round.

I know, arguing for a lost cause...

jmh530 <john.michael.hall gmail.com> writes:

On Friday, 29 May 2020 at 11:33:01 UTC, Sebastiaan Koppe wrote:
 On Thursday, 28 May 2020 at 16:01:35 UTC, Johannes Pfau wrote:
 [snip]

 This would be another round of massively breaking user code.

 The breakage will be split in two rounds, but the amount of 
 code needed to be modified would be the same as with the 
 safe-by-default-except-for-extern-non-c. With the possibility 
 of not needing the second round.

 I know, arguing for a lost cause...

At the end of the day, I think people would accept "massively 
breaking user code" if there is a good justification and doesn't 
drive a hole through  safe.

Arine <arine1283798123 gmail.com> writes:

On Thursday, 28 May 2020 at 12:28:16 UTC, Sebastiaan Koppe wrote:
 On Thursday, 28 May 2020 at 09:21:09 UTC, Jonathan M Davis 
 wrote:
 He did unfortunately manage to convince Atila, so the DIP has 
 been accepted, but based on the discussions, I think that you 
 may be the only person I've seen say anything positive about 
 the DIP treating extern(C) functions as  safe.

 - Jonathan M Davis

 I think Walter had to make a tough call with many tradeoffs. 
 The defining feature of engineering I would say.

 Is he wrong? Maybe, I don't know. The obvious path is far from 
 always being a winner.

A winner for what. The reasoning given doesn't make sense. Lots 
of people have pointed that out, and Walter has yet to expand on 
that.

 If it does come back to haunt him, he can always add a DIP to 
 make extern(!D)  system by default. It won't invalidate any 
 work.

That's worse. It'll cause huge breakage again if you need to 
convert extern(C) back to  system after it was converted to 
 safe. If  system by default proves to be a mistake, you can just 
flip it to  safe and it won't break any code. Why take a gamble 
on something that can't be rationalized?

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Thu, May 28, 2020 at 03:21:09AM -0600, Jonathan M Davis via
Digitalmars-d-announce wrote:
[...]
 With the DIP in its current state,  safe becomes a lie.  The compiler
 no longer guarantees that  safe code is memory safe so long as it
 doesn't call any  trusted code where the programmer incorrectly marked
 it as  trusted. Instead, the compiler blindly treats non-extern(D)
 declarations as  safe and invisibly introduces memory safety bugs into
  safe code.  Nothing about that is "OK."

[...]

I see it already.  The next time someone wants to make a codebase  safe
but the compiler complains about some violation, just add `extern(C)` to
the function and move on.


T

-- 
Claiming that your operating system is the best in the world because more
people use it is like saying McDonalds makes the best food in the world. --
Carl B. Constantine