www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - trust is an encapsulation method, not an escape

reply Walter Bright <newshound2 digitalmars.com> writes:
Consider the following code excerpted from std.array.join:

   static U trustedCast(U, V)(V v)  trusted { return cast(U) v; }
   return trustedCast!RetType(result);

This is because the compiler would complain that the following line would not
be 
 safe:

   return cast(RetType)(result);

The rationale is "I know it's safe, so I'll create an  trusted wrapper to 
eliminate the error." What comes next is "that's cumbersome. How about a better 
syntax:"

   trusted {
      return cast(RetType)(result);
   }

? It's the rationale behind "unsafe" blocks that appear in other languages. It 
seems like a perfectly reasonable request.

The trouble with it is, what if the cast is converting from an integer to a 
pointer? That could lead to memory corruption. The code allows a potentially 
memory corrupting operation to be inserted into code that is otherwise  safe.

The only way to deal with it is to then manually review everything about 
'RetType' and 'result' to prove to oneself that it is not converting random bit 
patterns into pointers. In other words, one is manually reviewing  safe code
for 
memory corruption errors.

This is an abject failure of  safe,  trusted, and  system.

The solution is to regard  trusted as a means of encapsulating unsafe 
operations, not escaping them. Encapsulating them means that the interface from 
the  trusted code is such that it is usable from safe code without having to 
manually review the safe code for memory safety. For example (also from
std.array):

   static void trustedMemcopy(T[] dest, T[] src)  trusted
   {
     assert(src.length == dest.length);
     memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
   }

I don't have to review callers of trustedMemory() because it encapsulates an 
unsafe operation (memcpy) with a safe interface.

The reason  trusted applies only to functions, and not to blocks of code, is 
that functions are the construct in D that provides an interface. Arbitrary 
blocks of code do not have a structured interface. Adding  trusted { code } 
support will encourage incorrect uses like the opening example. The existence
of 
 trusted blocks will require review of every line of code in the function that 
encloses it, and transitively every function that calls it!

Adding  trusted as a function attribute, on the other hand, only requires
review 
of the function's interface to determine if it is acceptable to use in safe 
code. Safety review of its callers is unnecessary.
Feb 05 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
The fact that  trusted is contained in small block doesn't mean 
rest of  safe function doesn't need to be reviewed. Only 
difference is "review all manually" vs "review all manually with 
some help of compiler".
Feb 05 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/15 3:43 PM, Dicebot wrote:

 The fact that  trusted is contained in small block doesn't mean rest of
  safe function doesn't need to be reviewed. Only difference is "review
 all manually" vs "review all manually with some help of compiler".


As Steve and others have shown, unsafe code under the disguise of 
 trusted affects everything. You are fostering an illusion. -- Andrei
Feb 05 2015
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 5 February 2015 at 23:49:05 UTC, Andrei Alexandrescu 
wrote:

 On 2/5/15 3:43 PM, Dicebot wrote:

 The fact that  trusted is contained in small block doesn't 
 mean rest of
  safe function doesn't need to be reviewed. Only difference is 
 "review
 all manually" vs "review all manually with some help of 
 compiler".


 As Steve and others have shown, unsafe code under the disguise 
 of  trusted affects everything. You are fostering an illusion. 
 -- Andrei


Steve has shown exactly the opposite, proving quite some of my 
points ;) But it is not uncommon to see what one wants to see in 
identical piece of information.
Feb 05 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 5 February 2015 at 23:43:19 UTC, Dicebot wrote:

 The fact that  trusted is contained in small block doesn't mean 
 rest of  safe function doesn't need to be reviewed. Only 
 difference is "review all manually" vs "review all manually 
 with some help of compiler".


I believe that was also the reasoning behind H.S.Teoh proposal 
for refining  trusted semantics - to keep help from compiler but 
mark functions as  trusted to make it clear that it still needs 
to be reviewed in total.
Feb 05 2015
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Feb 05, 2015 at 11:50:05PM +0000, Dicebot via Digitalmars-d wrote:

 On Thursday, 5 February 2015 at 23:43:19 UTC, Dicebot wrote:

The fact that  trusted is contained in small block doesn't mean rest
of  safe function doesn't need to be reviewed. Only difference is
"review all manually" vs "review all manually with some help of
compiler".


 
 I believe that was also the reasoning behind H.S.Teoh proposal for
 refining  trusted semantics - to keep help from compiler but mark
 functions as  trusted to make it clear that it still needs to be
 reviewed in total.


This whole discussion has been very tiring and frustrating, because we
keep talking past each other. The reason we keep talking past each other
is because we keep conflating several distinct, though related, issues.

1) I think we all basically agree that std.file is a mess and some
solution is needed to fix this mess. I don't think anyone is actually
advocating *for* the code that's currently in std.file right now. So can
we pretty please agree that this is a given, and stop using it to beat
each other over the head?

2) I think we also all basically agree that the *intent* of  trusted is
to be an encapsulation mechanism, or to present a safe API, or however
you want to describe it. I.e., if a function is marked  safe, then it
must be impossible to cause it to do something unsafe by passing it the
wrong arguments. Whatever it does under the hood should not have any
observable unsafe effect on the outside world. I'm pretty sure everyone
also agrees with this; I don't think anyone is advocating that we should
changee this intent. So can we please also take this as a given, and
stop repeating it at each other as if we don't all already know it?

This leaves the core of the contention, which is that  safe/ trusted, as
currently implemented, presents maintenance difficulties. Walter &
Andrei don't seem to be convinced that this is a problem, but the
evidence is right before us. We Phobos committers did not deliberately
set out to sabotage  trusted by introducing blatant abuses of it just
for fun (see (1) and (2) above). The current ugly (and arguably totally
wrong) hacks are a symptom of the underlying maintenance difficulty that
the current system presents. It is a desperate attempt to contain a
maintenance problem that's growing out of hand.

You probably don't believe me, but let's assume that we all agree on (1)
and (2), and let's suppose that everything is implemented the way Walter
& Andrei envision it. That is, no ugly hacks like what's currently in
std.file, and  trusted is only ever used on functions that present a
 safe API. What maintenance issues might there be? There are at least
the following (and probably more):

- First, let's get one thing straight: the body of a function is rarely
  only written once and never touched again later. In the real world,
  function bodies tend to be written piecemeal -- you write an initial
  implementation, then refine it, then commit it, then somebody comes
  along later and fixes a bug, somebody else implements an enhancement,
  etc.. I.e., code is in a constant flux of changes. Especially in a
  public project like Phobos, these changes are pouring in everyday from
  all manner of contributors of varying levels of competence. No single
  person can possibly keep up with every code change that gets merged.

Now, with this fact as background, let's consider:

- The body of a  trusted function is completely unrestricted, and can
  contain any manner of  system operation. The compiler doesn't give you
  any help in ensuring there aren't careless mistakes that slipped in.
  Any unsafe operation you can imagine can be freely thrown into such a
  function, and the compiler will happily accept it and allow  safe code
  to freely call it.

- The fact that the compiler does zero verification of the body of a
   trusted function, should already be a sign of deficiency, since the
  responsibility of checking correctness of the *entire* function body
  now falls upon fallible human beings. How sure are we that even the
  initial review was foolproof? At the very least, one would hope that
  the compiler could give us some help where it can, and leave the parts
  it cannot to a few isolated places where human inspection can be more
  focused on.

- Now, couple this with the volume of changes pouring into Phobos
  constantly, which means that inadvertent mistakes will creep into the
  code, either in the function body directly, or indirectly into any of
  the functions that the  trusted function calls. Some of these mistakes
  may break the  trusted promise of the function, but currently there is
  NO WARNING for this. How confident are we that changes to the function
  after the initial review did not break the  trusted promise?

- The currently recommended approach is to thoroughly review all changes
  to  trusted functions before merging. Given that just today, somebody
  merged a PR that has no unittests for verifying the correctness of a
  bugfix and preventing regressions, how likely is it that changes to
   trusted function will be thoroughly reviewed for all possible
  implications before merging?

- Not to mention, this is a direct change to a function, that is
  immediately visible from the function itself.  We haven't considered
  yet the implications of changes to helper functions called by  trusted
  functions, which may have safety implications on previously-verified
   trusted functions. If a direct change passes review without a proper
  accompanying unittest, how much less will changes to functions
  *indirectly* called by  trusted functions cause reviewers to inspect
  the  trusted function that calls them?

  Sure, we *could* institute a draconian policy that we must keep track
  of every change to every function that might be called by a  trusted
  function, and review all those  trusted callers everytime any one of
  these functions change. That would ensure the promise of  trusted is
  not broken. That would also ensure that nobody will maintain Phobos,
  because the burden of such a review requirement is far too onerous to
  be practical.

So basically, once a function is marked  trusted, we basically have to
accept its safety based on faith and hope. Since reviewing the entire
call tree of the function is impractical, we just need to have faith
that whoever reviewed the initial version of the function 5 months ago
did his job correctly, and we hope that nothing has changed since that
may have broken the safety of the function. External  safe callers of
the function simply take its safety on faith, because it presents a
 safe API, and there's no reason whatsoever to doubt whether or not it
is truly safe. In any case, since the compiler happily accepts  safe
code calling  trusted code, and we can't live our lives in fear that
perhaps a recent PR has introduced a change that broke the safety
promise of said  trusted code, we just have to take it on faith that the
 trusted code is, in fact,  safe.

Meanwhile, the changes continue to pour in, and the compiler doesn't
even check the most basic  safe-ty concerns in the body of the  trusted
function. When a change touches something that causes a previously  safe
operation in said function body to become  system, the compiler happily
accepts it without any warning or indication whatsoever that perhaps
there is some kind of mistake, a previously  safe primitive has now
become un- safe, and we'd better re-review the  trusted function to
ensure that no new safety holes have been introduced.  But surely, this
isn't a problem at all, since we reviewed this  trusted function 5
months ago, and nothing could possibly have gone wrong in the interim,
-- we believe.

(After all, our reviewers are doing their jobs properly, right? That is,
the onerous obligation to review not only changes to  trusted functions,
but every change to every function that occurs in the downstream call
tree of every  trusted function. In a *volunteer* project, where people
only work on what interests them. Yes, this just fills me with
confidence that there's absolutely nothing wrong with  trusted. What do
you mean, the emperor has no clothes?)


T

-- 
Государство делает вид, что платит нам
зарплату, а мы делаем вид, что работаем.
Feb 05 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/15 5:23 PM, H. S. Teoh via Digitalmars-d wrote:

 It is a desperate attempt to contain a
 maintenance problem that's growing out of hand.


Before we get into this - do you have evidence of the maintenance 
problem that's growing out of hand? E.g. bugs in std.file that were 
fixed etc. Thanks. -- Andrei
Feb 05 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/15 5:23 PM, H. S. Teoh via Digitalmars-d wrote:

 This whole discussion has been very tiring and frustrating, because we
 keep talking past each other. The reason we keep talking past each other
 is because we keep conflating several distinct, though related, issues.


Bummer about that. Let's see.


 1) I think we all basically agree that std.file is a mess and some
 solution is needed to fix this mess. I don't think anyone is actually
 advocating *for* the code that's currently in std.file right now. So can
 we pretty please agree that this is a given, and stop using it to beat
 each other over the head?


There remains the issue - is the messy  trusted code a net negative or 
not? If so, why did it pass review? That would be an interesting case 
study for improving our review process. I find it difficult to reckon 
that that code got into a Phobos release.


 2) I think we also all basically agree that the *intent* of  trusted is
 to be an encapsulation mechanism, or to present a safe API, or however
 you want to describe it. I.e., if a function is marked  safe, then it
 must be impossible to cause it to do something unsafe by passing it the
 wrong arguments. Whatever it does under the hood should not have any
 observable unsafe effect on the outside world. I'm pretty sure everyone
 also agrees with this; I don't think anyone is advocating that we should
 changee this intent. So can we please also take this as a given, and
 stop repeating it at each other as if we don't all already know it?


Yah.


 This leaves the core of the contention, which is that  safe/ trusted, as
 currently implemented, presents maintenance difficulties.


I think these difficulties have not been demonstrated.


 Walter &
 Andrei don't seem to be convinced that this is a problem, but the
 evidence is right before us.


Pray tell.


 We Phobos committers did not deliberately
 set out to sabotage  trusted by introducing blatant abuses of it just
 for fun (see (1) and (2) above). The current ugly (and arguably totally
 wrong) hacks are a symptom of the underlying maintenance difficulty that
 the current system presents. It is a desperate attempt to contain a
 maintenance problem that's growing out of hand.


Could you please show the numerous bugs and subsequent commits fixing 
them that you are alluding to?

I'll stop quoting here. My understanding of your core argument is as 
follows:

"Currently there is no safety verification for  trusted functions, and 
it would be nice if there were more."

I'm totally on board with this. It would definitely be nice to have more 
verification. That said:

(1) I am not convinced this is a problem as large as you want it to be. 
I find most of your arguments to appeal to unsubstantiated hyperbole.

(2) I don't think the proposal you sketched at http://goo.gl/JWqafx is good.

Now, there must be a way to convey this to you without personally 
offending you: I think your proposal is not good. I don't buy it. It's 
poorly motivated, poorly argued, it's very incomplete, it complicates 
the language without visible benefit, it's just not what I'd think we 
should be doing.

If you have a better proposal, please let's have it. To be honest I 
don't think a reasonable amount of work on http://goo.gl/JWqafx will 
make it better.

How can I say "after carefully learning about your points and proposal I 
think we should do as Walter and I say and not as you say" without 
insulting you?


Andrei
Feb 05 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 6:26 PM, Andrei Alexandrescu wrote:

 (2) I don't think the proposal you sketched at http://goo.gl/JWqafx is good.


The goo link is disabled for violating goo's terms of service, whatever that
means.

What's the real url?
Feb 05 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/15 6:52 PM, Walter Bright wrote:

 On 2/5/2015 6:26 PM, Andrei Alexandrescu wrote:

 (2) I don't think the proposal you sketched at http://goo.gl/JWqafx is
 good.


 The goo link is disabled for violating goo's terms of service, whatever
 that means.

 What's the real url?


http://forum.dlang.org/thread/mb070a$1ok1$1 digitalmars.com#post-mailman.6039.1423163991.9932.digitalmars-d:40puremagic.com

Andrei
Feb 05 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 7:00 PM, Andrei Alexandrescu wrote:

 On 2/5/15 6:52 PM, Walter Bright wrote:

 On 2/5/2015 6:26 PM, Andrei Alexandrescu wrote:

 (2) I don't think the proposal you sketched at http://goo.gl/JWqafx is
 good.


 The goo link is disabled for violating goo's terms of service, whatever
 that means.

 What's the real url?


 http://forum.dlang.org/thread/mb070a$1ok1$1 digitalmars.com#post-mailman.6039.1423163991.9932.digitalmars-d:40puremagic.com


Thank you. Essentially, the proposal attaches safe/trusted/system to variables. 
I don't believe this works. Safety/unsafety can often be the emergent 
combination of state of many otherwise 'safe' variables.

In any case, it lacks any demonstration of how it would address the problems in 
std.array. I.e. how could it insure that the code that initializes the array in 
std.array.join() initializes every element? (That loop is in the 'safe'
section!)

I don't see how any proposal can work unless it specifies a safe interface to
an 
unsafe section of code. (I read a Rust tutorial that rather bluntly pointed
this 
out as well.)
Feb 05 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 03:14:59 UTC, Walter Bright wrote:

 I don't see how any proposal can work unless it specifies a 
 safe interface to an unsafe section of code. (I read a Rust 
 tutorial that rather bluntly pointed this out as well.)


Link?
Feb 05 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 7:39 PM, Zach the Mystic wrote:

 On Friday, 6 February 2015 at 03:14:59 UTC, Walter Bright wrote:

 I don't see how any proposal can work unless it specifies a safe interface to
 an unsafe section of code. (I read a Rust tutorial that rather bluntly pointed
 this out as well.)


 Link?


"A non-unsafe function using unsafe internally should be implemented to be safe 
to call; that is, there is no circumstance or set of arguments that can make
the 
function violate any invariants. If there are such circumstances, it should be 
marked unsafe."

"However, this is not the case, unsafe is just an implementation detail; if a 
safe function uses unsafe internally, it just means the author has been forced 
to step around the type system, but still exposes a safe interface."

http://huonw.github.io/blog/2014/07/what-does-rusts-unsafe-mean/
Feb 05 2015
prev sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
I figured that someone would have already objected to part of this, but 
the definition is stronger than I believe is intended for D:

On 2/5/2015 5:23 PM, H. S. Teoh via Digitalmars-d wrote:

 2) I think we also all basically agree that the *intent* of  trusted is
 to be an encapsulation mechanism, or to present a safe API, or however
 you want to describe it. I.e., if a function is marked  safe, then it
 must be impossible to cause it to do something unsafe by passing it the
 wrong arguments. Whatever it does under the hood should not have any
 observable unsafe effect on the outside world. I'm pretty sure everyone
 also agrees with this; I don't think anyone is advocating that we should
 changee this intent. So can we please also take this as a given, and
 stop repeating it at each other as if we don't all already know it?


Specifically, the 'impossible to cause it to do something unsafe by 
passing it the wrong arguments' part.

For example, this is valid  safe code:

 safe void setToZero(ubyte[] foo)
{
     foo[] = 0;
}

How about now:

void bar()
{
     ubyte[] a;
     a.ptr = 0; // arbitrary value as long as NOT from an allocator
     a.len = 1;

     setToZero(a);
}

 safe code is memory safe when passed good inputs.   safe code does not 
have to detect or prevent garbage input from causing harm.   The 
promises of  trusted aren't stronger than that either.
Feb 05 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 8:30 PM, Brad Roberts via Digitalmars-d wrote:

  safe code is memory safe when passed good inputs.   safe code does not have to
 detect or prevent garbage input from causing harm.   The promises of  trusted
 aren't stronger than that either.


This is correct. Thanks for catching it.
Feb 05 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 3:43 PM, Dicebot wrote:

 The fact that  trusted is contained in small block doesn't mean rest of  safe
 function doesn't need to be reviewed. Only difference is "review all manually"
 vs "review all manually with some help of compiler".


I did a review of all uses of  trusted in std.array:

   https://issues.dlang.org/show_bug.cgi?id=14127

About 90% of them resulted in the injection of unsafe code into safe functions, 
requiring a safety review of those allegedly mechanically checkable functions.

This is an abject failure of the technique of using  trusted as an escape than 
as encapsulation.

By definition, if an  trusted function presents itself with a safe interface, 
the calling code does not have to be reviewed. And reviewing the interface is a 
heluva lot easier than the whole rest of the code.
Feb 05 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 6 February 2015 at 00:04:26 UTC, Walter Bright wrote:

 On 2/5/2015 3:43 PM, Dicebot wrote:

 The fact that  trusted is contained in small block doesn't 
 mean rest of  safe
 function doesn't need to be reviewed. Only difference is 
 "review all manually"
 vs "review all manually with some help of compiler".


 I did a review of all uses of  trusted in std.array:

   https://issues.dlang.org/show_bug.cgi?id=14127

 About 90% of them resulted in the injection of unsafe code into 
 safe functions, requiring a safety review of those allegedly 
 mechanically checkable functions.


Yes, that was intended and not accidental. Again, we were dealing 
with limited set of faulty tools. Things got inevitably hacky.


 By definition, if an  trusted function presents itself with a 
 safe interface, the calling code does not have to be reviewed. 
 And reviewing the interface is a heluva lot easier than the 
 whole rest of the code.


I know this definition. It have tried it in practice and 
concluded as being absolutely useless. There is no way I am going 
to return back to this broken concept - better to ignore  safe 
completely as misfeature if you insist on doing things that way.
Feb 05 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 4:13 PM, Dicebot wrote:

 I know this definition. It have tried it in practice and concluded as being
 absolutely useless. There is no way I am going to return back to this broken
 concept - better to ignore  safe completely as misfeature if you insist on
doing
 things that way.


I'm sorry I haven't been able to convince you. I don't have any more arguments 
other than just repeating myself.

Moving forward, I must insist that use of  trusted in such a way as to make its 
surrounding context not mechanically checkable is no longer acceptable in
Phobos.

If need be, I will rewrite std.array myself to address this.

But D is a systems programming language, not a B&D language, and anyone will be 
free to ignore  safe and continue to use the other benefits of D, or use  safe 
in a way that conforms to their own formulation of best practices.
Feb 05 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/15 5:12 PM, Walter Bright wrote:

 On 2/5/2015 4:13 PM, Dicebot wrote:

 I know this definition. It have tried it in practice and concluded as
 being
 absolutely useless. There is no way I am going to return back to this
 broken
 concept - better to ignore  safe completely as misfeature if you
 insist on doing
 things that way.


 I'm sorry I haven't been able to convince you. I don't have any more
 arguments other than just repeating myself.

 Moving forward, I must insist that use of  trusted in such a way as to
 make its surrounding context not mechanically checkable is no longer
 acceptable in Phobos.

 If need be, I will rewrite std.array myself to address this.

 But D is a systems programming language, not a B&D language, and anyone
 will be free to ignore  safe and continue to use the other benefits of
 D, or use  safe in a way that conforms to their own formulation of best
 practices.


I second this. -- Andrei
Feb 05 2015
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Feb 05, 2015 at 05:12:16PM -0800, Walter Bright via Digitalmars-d wrote:

 On 2/5/2015 4:13 PM, Dicebot wrote:

I know this definition. It have tried it in practice and concluded as
being absolutely useless. There is no way I am going to return back
to this broken concept - better to ignore  safe completely as
misfeature if you insist on doing things that way.


 
 I'm sorry I haven't been able to convince you. I don't have any more
 arguments other than just repeating myself.
 
 Moving forward, I must insist that use of  trusted in such a way as to
 make its surrounding context not mechanically checkable is no longer
 acceptable in Phobos.


You are still not getting the point we've been trying to make since the
beginning of this tiresome interminable thread. WE AGREE WITH YOU THAT
 trusted IS MEANT TO BE USED IN THE WAY YOU DESCRIBED!! WE AGREE THAT
std.file NEEDS TO BE FIXED! I hope that gets the point across.

We are not arguing *for* the ugly hacks currently in std.file,
std.array, and wherever else.  What we're trying to tell you, is that
even if we were to rewrite everything the way you prescribe, it will
*still* present the same maintenance issues we have brought up, because
of the way  trusted is implemented. The way it gives free rein to use
whatever  system operations you fancy, with no warnings whatsoever
issued when you did something that's blatantly wrong.  The way it puts a
stamp of  safe-ty approval on what's essentially a  system function
after the initial review, and is never revoked thereafter. The way
changes to  trusted functions are never mechanically checked for basic
 safe sanity. The way a call to a previously  safe function from inside
a  trusted function does not trigger any warning when the callee now
becomes  system.

We are NOT asking you to justify the current state of std.file and
std.array, as you seem to believe we're asking for. Please stop fixating
on that, it's not helping this discussion at all. What we're asking for
is for the compiler to give us the tools to make Phobos maintenance more
tenable.  Like performing  safe checks inside  trusted functions where
possible.  Requiring explicit markup for operations that the compiler
thinks are unsafe, but we know otherwise, so that when somebody comes
along later and introduces more of these operations, the compiler will
issue warnings that prompt reviewers to carefully inspect what has
changed.  Some kind of mechanism for controlling what kind of changes
can be accepted in a  trusted function.

The current state of std.file/std.array/whatever is just a symptom. The
real disease is that we need some way of coping with the maintenance
issues introduced by  trusted. If those issues continue to be neglected,
then other symptoms will just show up elsewhere, even if you rewrite
std.file and std.array now. Until the real issue is addressed, you will
just be playing whack-a-mole on every apparent "abuse" of  trusted that
crops up. It will never end until the real issues are addressed.



 If need be, I will rewrite std.array myself to address this.


Please do. Maybe you will understand more where we're coming from if you
see it for yourself.  Obviously our words mean little to you.  And while
you're at it, try reviewing a few Phobos PR's related to std.array as
well, and observe for yourself the maintenance issues that arise.


T

-- 
"The whole problem with the world is that fools and fanatics are always so
certain of themselves, but wiser people so full of doubts." -- Bertrand
Russell. "How come he didn't put 'I think' at the end of it?" -- Anonymous
Feb 05 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/15 5:53 PM, H. S. Teoh via Digitalmars-d wrote:

 And while
 you're at it, try reviewing a few Phobos PR's related to std.array as
 well, and observe for yourself the maintenance issues that arise.


Again, could you please collect past evidence of such? According to you 
there must be many cases in which there was insufficient safety checking 
that was hiding bugs etc.

Evidence would do wonders for your argument.


Andrei
Feb 05 2015
parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 02:06:29 UTC, Andrei Alexandrescu 
wrote:

 On 2/5/15 5:53 PM, H. S. Teoh via Digitalmars-d wrote:

 And while
 you're at it, try reviewing a few Phobos PR's related to 
 std.array as
 well, and observe for yourself the maintenance issues that 
 arise.


 Again, could you please collect past evidence of such? 
 According to you there must be many cases in which there was 
 insufficient safety checking that was hiding bugs etc.

 Evidence would do wonders for your argument.


 Andrei


Evidence does wonders for any argument!
Feb 05 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 6 February 2015 at 01:12:18 UTC, Walter Bright wrote:

 On 2/5/2015 4:13 PM, Dicebot wrote:

 I know this definition. It have tried it in practice and 
 concluded as being
 absolutely useless. There is no way I am going to return back 
 to this broken
 concept - better to ignore  safe completely as misfeature if 
 you insist on doing
 things that way.


 I'm sorry I haven't been able to convince you. I don't have any 
 more arguments other than just repeating myself.


This is not about convincing but about showing the example 
yourself. My disagreement is not theoretical - I trust you to be 
much better in actual language design. But trusting alone won't 
make a difference if I have no idea how use it in practice (after 
trying and failing).

I am not even sure how you can show the example though, to be 
honest - implied issues are about maintaining code and not just 
writing it.


 But D is a systems programming language, not a B&D language, 
 and anyone will be free to ignore  safe and continue to use the 
 other benefits of D, or use  safe in a way that conforms to 
 their own formulation of best practices.


It always feel awkward to ignore what was supposed to be a major 
selling point for a language :(
Feb 05 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 8:20 PM, Dicebot wrote:

 I am not even sure how you can show the example though, to be honest - implied
 issues are about maintaining code and not just writing it.


Let's start with std.array.uninitializedArray():

    auto uninitializedArray(T, I...)(I sizes) nothrow  trusted

Note how it says it is 'trusted'. Yet it can be used to create an uninitialized 
array of pointers! There is no freakin' way this function should pass muster. 
Worse, multiple modules use it:

array.d:            return uninitializedArray!(Unqual!E[])(n);
array.d:auto uninitializedArray(T, I...)(I sizes) nothrow  trusted
array.d:    double[] arr = uninitializedArray!(double[])(100);
array.d:    double[][] matrix = uninitializedArray!(double[][])(42, 31);
array.d:    auto s1 = uninitializedArray!(int[])();
array.d:            return uninitializedArray!(T[])(n);
array.d:        auto result = uninitializedArray!(RetTypeElement[])(length);
array.d:            auto result = uninitializedArray!(RetTypeElement[])(length);
array.d:        auto result = uninitializedArray!(RetTypeElement[])(length);
file.d:    import std.array : uninitializedArray;
file.d:        auto buf = uninitializedArray!(ubyte[])(size);
file.d:        void[] result = uninitializedArray!(ubyte[])(initialAlloc);
numeric.d:    import std.array : uninitializedArray;
numeric.d:        auto memSpace = uninitializedArray!(lookup_t[])(2 * size);
numeric.d:        ret = uninitializedArray!(Complex!F[])(range.length);
numeric.d:        ret = uninitializedArray!(Complex!F[])(range.length);
parallelism.d:                auto buf = uninitializedArray!(MapType!(Args[0], 
functions)[])(len);
parallelism.d:                    temp = uninitializedArray!Temp(workUnitSize);
parallelism.d:                    temp = uninitializedArray!Temp(workUnitSize);
stdio.d:    import std.array : appender, uninitializedArray;
stdio.d:                buf = uninitializedArray!(char[])(i);
stdio.d:                buf = uninitializedArray!(char[])(i);
stdio.d:    import std.array : appender, uninitializedArray;
utf.d:        import std.array : uninitializedArray;
utf.d:        auto copy = uninitializedArray!(Unqual!OutChar[])(str.length + 1);

So I already know there are serious problems. Looking closer, I see there's 
another function:

auto minimallyInitializedArray(T, I...)(I sizes) nothrow  trusted

which initializes everything to 0, pointers and all. I agree this one can be 
trusted. The difference between their implementations is a parameter they pass 
to arrayAllocImpl(), which says whether to 0 it or not.

But really, it's only pointer types (and types composed of pointers) that make 
it system code. Since uninitializedArray() is a template, we can have two 
implementations for it that are statically selected based on the element type.
One is  system, when E is a type that contains a pointer. The other is
 trusted, 
when E is not a pointer.

That solves that problem without great effort.

Upstream of uninitializedArray(), the only problem then is when it is called to 
generate an uninitialized array of pointers. In order to make the call safe,
the 
caller must then ensure that the array gets initialized with safe values. And 
that code should be properly marked as  trusted, and not before.

I.e. start at the bottom, and work up until you find the right place to make it 
trusted. Rinse, repeat. I am not seeing where the problem with doing this is.
Feb 05 2015
prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:

   static void trustedMemcopy(T[] dest, T[] src)  trusted
   {
     assert(src.length == dest.length);
     memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
   }

 I don't have to review callers of trustedMemory() because it 
 encapsulates an unsafe operation (memcpy) with a safe interface.


It might have done so if it ensured that T was a proper value 
type, but unfortunately D's type system is not strong enough.

What happens if T is a unique_ptr style reference? Ouch, two 
unique references to the same object. Ouch, memory unsafe.

 safe is a leaky cauldron and will continue to be so until you 
provide a proof of language constructs and how they interact. The 
only sane way to do that is to do the proof over a simplified 
virtual machine and map all language constructs to it.
Feb 05 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 9:00 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:

   static void trustedMemcopy(T[] dest, T[] src)  trusted
   {
     assert(src.length == dest.length);
     memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
   }

 I don't have to review callers of trustedMemory() because it encapsulates an
 unsafe operation (memcpy) with a safe interface.


 It might have done so if it ensured that T was a proper value type, but
 unfortunately D's type system is not strong enough.

 What happens if T is a unique_ptr style reference? Ouch, two unique references
 to the same object. Ouch, memory unsafe.


Good point. Then a constraint can be added to the function signature that T is 
copyable.

D's type system is strong enough for that.
Feb 05 2015
next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Friday, 6 February 2015 at 05:32:48 UTC, Walter Bright wrote:

 On 2/5/2015 9:00 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright 
 wrote:

  static void trustedMemcopy(T[] dest, T[] src)  trusted
  {
    assert(src.length == dest.length);
    memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
  }

 I don't have to review callers of trustedMemory() because it 
 encapsulates an
 unsafe operation (memcpy) with a safe interface.


 It might have done so if it ensured that T was a proper value 
 type, but
 unfortunately D's type system is not strong enough.

 What happens if T is a unique_ptr style reference? Ouch, two 
 unique references
 to the same object. Ouch, memory unsafe.


 Good point. Then a constraint can be added to the function 
 signature that T is copyable.

 D's type system is strong enough for that.


What's the "D way" of checking if a parameter is a reftype, 
valuetype, etc?
Feb 05 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/5/2015 9:55 PM, weaselcat wrote:

 What's the "D way" of checking if a parameter is a reftype, valuetype, etc?


http://dlang.org/phobos/std_traits.html
Feb 05 2015
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 05:32:48 UTC, Walter Bright wrote:

 Good point. Then a constraint can be added to the function 
 signature that T is copyable.

 D's type system is strong enough for that.


Yes, if you remember to add it.  The ideal would be to have the 
default constrained to proper value-types. D inherits some of C's 
weaker typing properties by trying to stay Cish to the programmer.

The fact that so many messages has been posted to this thread 
before someone caught it is testament to the burden put upon 
reviewers with " trusted". After all, in debate-like threads like 
this people will look with higher scrutiny at posted code than in 
an average review...
Feb 06 2015
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:

   static void trustedMemcopy(T[] dest, T[] src)  trusted
   {
     assert(src.length == dest.length);
     memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
   }


Should be enforce: assert doesn't guard against malicious usage.
Feb 06 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 12:31 AM, Kagamin wrote:

 On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:

   static void trustedMemcopy(T[] dest, T[] src)  trusted
   {
     assert(src.length == dest.length);
     memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
   }


 Should be enforce: assert doesn't guard against malicious usage.


Cue my endless attempts to explain the difference between input errors and
logic 
errors :-(
Feb 06 2015
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:

 On 2/6/2015 12:31 AM, Kagamin wrote:

 On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright 
 wrote:

  static void trustedMemcopy(T[] dest, T[] src)  trusted
  {
    assert(src.length == dest.length);
    memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
  }


 Should be enforce: assert doesn't guard against malicious 
 usage.


 Cue my endless attempts to explain the difference between input 
 errors and logic errors :-(


So which one is it?

On one hand, it is clearly a logic error - passing arrays of 
different length is clearly a program bug.

On the other hand, this is a library function, and as you said, 
we can't know how it's going to be used - so the check has to be 
unconditional.
Feb 06 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 1:01 AM, Vladimir Panteleev wrote:

 On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:

 Cue my endless attempts to explain the difference between input errors and
 logic errors :-(


 So which one is it?


http://www.digitalmars.com/d/archives/digitalmars/D/Program_logic_bugs_vs_input_environmental_errors_244143.html
Feb 06 2015
next sibling parent "Kagamin" <spam here.lot> writes:
On Friday, 6 February 2015 at 10:28:38 UTC, Walter Bright wrote:

 On 2/6/2015 1:01 AM, Vladimir Panteleev wrote:

 On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright 
 wrote:

 Cue my endless attempts to explain the difference between 
 input errors and
 logic errors :-(


 So which one is it?


 http://www.digitalmars.com/d/archives/digitalmars/D/Program_logic_bugs_vs_input_environmental_errors_244143.html


Do you want to say, that safe code can still corrupt memory if 
it's crafted by a malevolent programmer or if it doesn't perform 
additional domain-specific logic checks of otherwise memory-safe 
data?
Feb 06 2015
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 6 February 2015 at 10:28:38 UTC, Walter Bright wrote:

 On 2/6/2015 1:01 AM, Vladimir Panteleev wrote:

 On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright 
 wrote:

 Cue my endless attempts to explain the difference between 
 input errors and
 logic errors :-(


 So which one is it?


 http://www.digitalmars.com/d/archives/digitalmars/D/Program_logic_bugs_vs_input_environmental_errors_244143.html


That doesn't answer my question.

A few years ago, I recall, you were arguing that for functions 
which are or may be exported to a DLL, thus all Phobos functions, 
it is impossible to predict how the functions will be used. Thus, 
you argued, the functions' input has to be validated, even if 
invalid parameters can only be passed to the function as a result 
of a program bug, and never user input.

So, to repeat my question: which one is it? Have you changed your 
mind, or are there exceptions to the rules in the post you quoted?
Feb 06 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 5:13 AM, Vladimir Panteleev wrote:

 On Friday, 6 February 2015 at 10:28:38 UTC, Walter Bright wrote:

 On 2/6/2015 1:01 AM, Vladimir Panteleev wrote:

 On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:

 Cue my endless attempts to explain the difference between input
 errors and
 logic errors :-(


 So which one is it?


 http://www.digitalmars.com/d/archives/digitalmars/D/Program_logic_bugs_vs_input_environmental_errors_244143.html


 That doesn't answer my question.

 A few years ago, I recall, you were arguing that for functions which are
 or may be exported to a DLL, thus all Phobos functions, it is impossible
 to predict how the functions will be used. Thus, you argued, the
 functions' input has to be validated, even if invalid parameters can
 only be passed to the function as a result of a program bug, and never
 user input.

 So, to repeat my question: which one is it? Have you changed your mind,
 or are there exceptions to the rules in the post you quoted?


Could you all please grant me this wish - let's not get into that vortex 
again? It renders everyone involved unproductive for days on end. 
Thanks. -- Andrei
Feb 06 2015
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 6 February 2015 at 16:19:00 UTC, Andrei Alexandrescu 
wrote:

 On 2/6/15 5:13 AM, Vladimir Panteleev wrote:

 That doesn't answer my question.

 A few years ago, I recall, you were arguing that for functions 
 which are
 or may be exported to a DLL, thus all Phobos functions, it is 
 impossible
 to predict how the functions will be used. Thus, you argued, 
 the
 functions' input has to be validated, even if invalid 
 parameters can
 only be passed to the function as a result of a program bug, 
 and never
 user input.

 So, to repeat my question: which one is it? Have you changed 
 your mind,
 or are there exceptions to the rules in the post you quoted?


 Could you all please grant me this wish - let's not get into 
 that vortex again? It renders everyone involved unproductive 
 for days on end. Thanks. -- Andrei


What is the problem? Sorry if my post sounded confrontational, 
but I wasn't going to argue - I just want to understand the 
language designers' current position on this topic.
Feb 06 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 9:41 AM, Vladimir Panteleev wrote:

 On Friday, 6 February 2015 at 16:19:00 UTC, Andrei Alexandrescu wrote:

 On 2/6/15 5:13 AM, Vladimir Panteleev wrote:

 That doesn't answer my question.

 A few years ago, I recall, you were arguing that for functions which are
 or may be exported to a DLL, thus all Phobos functions, it is impossible
 to predict how the functions will be used. Thus, you argued, the
 functions' input has to be validated, even if invalid parameters can
 only be passed to the function as a result of a program bug, and never
 user input.

 So, to repeat my question: which one is it? Have you changed your mind,
 or are there exceptions to the rules in the post you quoted?


 Could you all please grant me this wish - let's not get into that
 vortex again? It renders everyone involved unproductive for days on
 end. Thanks. -- Andrei


 What is the problem? Sorry if my post sounded confrontational, but I
 wasn't going to argue - I just want to understand the language
 designers' current position on this topic.


I was joking - whenever Walter gets into the assert vs. enforce 
distinction, the world economy is decreasing by 1%. -- Andrei
Feb 06 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 5:13 AM, Vladimir Panteleev wrote:

 So, to repeat my question: which one is it? Have you changed your mind, or are
 there exceptions to the rules in the post you quoted?


It means that you, the programmer, have to decide whether it is environmental 
input validation or a logic error in your code.

Follow the rules:

1. exceptions are not for debugging the logic of your program
2. do not use exceptions to recover from logic bugs in your program

I have pontificated on this at GREAT length in multiple threads in this n.g. If 
it is still a mystery to you or anyone else, I give up. Keep in mind the levels 
of expertise:

newbie: follow the rules because you're told they're the right thing to do
master: follow the rules because you understand they're the right thing to do
guru: transcend the rules because you know when they don't apply
Feb 06 2015
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 6 February 2015 at 21:08:21 UTC, Walter Bright wrote:

 1. exceptions are not for debugging the logic of your program
 2. do not use exceptions to recover from logic bugs in your 
 program


OK, this is nothing new, but still doesn't answer my question.

Would you say that the correct thing to do, then, would be an 
unconditional check that throws an Error? Such as, `if (!...) 
throw new Error(...)`, or `enforce!Error(...)`?

I recall there is a bug report on Bugzilla that Phobos contracts 
are removed in release (default) builds of the library, when 
ideally contracts of public functions should stay but inner 
asserts should be removed in optimized code. That would allow 
using them for validating parameters without resorting to 
explicitly-unconditional checks.
Feb 06 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 9:49 PM, Vladimir Panteleev wrote:

 On Friday, 6 February 2015 at 21:08:21 UTC, Walter Bright wrote:

 1. exceptions are not for debugging the logic of your program
 2. do not use exceptions to recover from logic bugs in your program


 OK, this is nothing new, but still doesn't answer my question.


You wrote "is clearly a program bug". Therefore use assert(). It's as simple as 
that.
Feb 06 2015
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Saturday, 7 February 2015 at 06:20:16 UTC, Walter Bright wrote:

 On 2/6/2015 9:49 PM, Vladimir Panteleev wrote:

 On Friday, 6 February 2015 at 21:08:21 UTC, Walter Bright 
 wrote:

 1. exceptions are not for debugging the logic of your program
 2. do not use exceptions to recover from logic bugs in your 
 program


 OK, this is nothing new, but still doesn't answer my question.


 You wrote "is clearly a program bug". Therefore use assert(). 
 It's as simple as that.


OK, thank you. A few years ago you were recommending something 
else for this situation. We were in disagreement then, and I 
agree with your current opinion.
Feb 07 2015
prev sibling next sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:

 On 2/6/2015 12:31 AM, Kagamin wrote:

 On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright 
 wrote:

  static void trustedMemcopy(T[] dest, T[] src)  trusted
  {
    assert(src.length == dest.length);
    memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
  }


 Should be enforce: assert doesn't guard against malicious 
 usage.


 Cue my endless attempts to explain the difference between input 
 errors and logic errors :-(


I agree with you, that this should be an assert, but then it 
cannot be  trusted. A  trusted function should provide it's 
guarantee for all input that can be crafted in  safe code. This 
is why this


void bar()
{
     ubyte[] a;
     a.ptr = 0; // arbitrary value as long as NOT from an allocator
     a.len = 1;

     setToZero(a);
}

is not a valid counter-example and that memcpy cannot be 
 trusted. It has no safe interface.


Back to topic. Since  trusted must provide a safe interface, I 
don't think it makes much sense to attach  trusted to smaller 
code blocks than functions. That might not be true for the 
opposite:  safe blocks where the compiler turns the safety check 
back on.

This prevents that a  trusted/ safe function becoming  system 
breaks the  trusted guarantee of called functions.

void bar(void* argument)  trusted { ... }

void forward()  trusted
{
     // REVIEW: forwards arguments to bar, which is  trusted, so 
we can
     // trust forward as well
     void* p = malloc(...);
     bar(arguments);
     free(p);
}

Changing bar to  system is a breaking change and currently a 
silent one. This is a maintenance problem I see as well. Solution:

void forward()  trusted
{
     void* p = malloc(...);
     // safe is bar is safe, so turn compiler checks on again.
      safe  { bar(p); }
     free(p);
}


Introducing an error in an  trusted function is _not_ an 
maintenance horror. While your  safe-ty guarantee is out of the 
window, it can be catched my reviewing one function: the one 
where the error is introduced in. Making a  trusted function 
 system forces us to review all  trusted functions calling the 
now  system function.
Feb 06 2015
parent "Tobias Pankrath" <tobias pankrath.net> writes:
 This prevents that a  trusted/ safe function becoming  system 
 breaks the  trusted guarantee of called functions.


Calling function, forward in the example.
Feb 06 2015
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:

 On 2/6/2015 12:31 AM, Kagamin wrote:

 On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright 
 wrote:

  static void trustedMemcopy(T[] dest, T[] src)  trusted
  {
    assert(src.length == dest.length);
    memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
  }


 Should be enforce: assert doesn't guard against malicious 
 usage.


 Cue my endless attempts to explain the difference between input 
 errors and logic errors :-(


A little offtop: if this function is compiled in release mode and 
compiler assumes assert holds, it's free to use dest.length 
instead of src.length and if at runtime dest is longer than src, 
this will create heartbleed-like bug in safe code.
Feb 06 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 4:17 AM, Kagamin wrote:

 On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:

 On 2/6/2015 12:31 AM, Kagamin wrote:

 On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:

  static void trustedMemcopy(T[] dest, T[] src)  trusted
  {
    assert(src.length == dest.length);
    memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
  }


 Should be enforce: assert doesn't guard against malicious usage.


 Cue my endless attempts to explain the difference between input errors and
 logic errors :-(


 A little offtop: if this function is compiled in release mode and compiler
 assumes assert holds, it's free to use dest.length instead of src.length and if
 at runtime dest is longer than src, this will create heartbleed-like bug in
safe
 code.


Sigh. Please visit your nearest Catholic school and ask one of the nuns to 
thwack your knuckles with a ruler!
Feb 06 2015
prev sibling parent "Kagamin" <spam here.lot> writes:
On Friday, 6 February 2015 at 08:58:05 UTC, Walter Bright wrote:

 On 2/6/2015 12:31 AM, Kagamin wrote:

 On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright 
 wrote:

  static void trustedMemcopy(T[] dest, T[] src)  trusted
  {
    assert(src.length == dest.length);
    memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
  }


 Should be enforce: assert doesn't guard against malicious 
 usage.


 Cue my endless attempts to explain the difference between input 
 errors and logic errors :-(


Well, ok, not enforce:

static void trustedMemcopy(T[] dest, T[] src)  trusted
{
   if(src.length != dest.length)assert(false);
   memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}
Feb 06 2015
prev sibling next sibling parent reply "Martin Krejcirik" <mk-junk i-line.cz> writes:
If I understand it correctly, Walter is against adding trusted 
blocks (trusted {...}) into  safe functions. But what about 
having safe blocks in  trusted functions ?

int somefunc()  trusted
{
    int a = systemfunc();
    int b;

     safe {
       b = safefunc(); // calling systemfunc() not allowed;
    }

    return a + b;
}
Feb 06 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 3:57 AM, Martin Krejcirik wrote:

 If I understand it correctly, Walter is against adding trusted blocks
 (trusted {...}) into  safe functions. But what about having safe blocks
 in  trusted functions ?


That would be sensible - perhaps the best step forward following this 
long discussion. -- Andrei
Feb 06 2015
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 6 February 2015 at 16:11:31 UTC, Andrei Alexandrescu 
wrote:

 On 2/6/15 3:57 AM, Martin Krejcirik wrote:

 If I understand it correctly, Walter is against adding trusted 
 blocks
 (trusted {...}) into  safe functions. But what about having 
 safe blocks
 in  trusted functions ?


 That would be sensible - perhaps the best step forward 
 following this long discussion. -- Andrei


It feels inelegant, but it might be the best way out of a bad 
situation.

I can instantly see this happening:

void foo()  trusted
{
      safe
     {
         //loads of code
     }

     //a few lines of system code, only safe due to context in the 
 safe blocks

      safe
     {
         \\loads of code
     }
}

Is that what we want? I can't see why not, but it feels off 
somehow... Effectively you've got  trusted blocks in an  trusted 
function, just inverted.
Feb 06 2015
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 16:19:26 UTC, John Colvin wrote:

 I can instantly see this happening:

 void foo()  trusted
 {


        auto p = malloc(…)


      safe
     {


             global_datastructure.push(p)

         //loads of code
     }


        free(p)


     //a few lines of system code, only safe due to context in 
 the  safe blocks

      safe
     {
         \\loads of code
     }
 }

 Is that what we want?


Nope.
Feb 06 2015
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 8:19 AM, John Colvin wrote:

 Is that what we want? I can't see why not, but it feels off somehow...
 Effectively you've got  trusted blocks in an  trusted function, just
 inverted.


That's the natural direction - the default assumption is weak, and you 
have the option to temporarily strengthen it. That said I'm not jazzed 
about the whole thing. What we have is the right balance. -- Andrei
Feb 06 2015
prev sibling next sibling parent "Tobias Pankrath" <tobias pankrath.net> writes:
On Friday, 6 February 2015 at 16:19:26 UTC, John Colvin wrote:

 It feels inelegant, but it might be the best way out of a bad 
 situation.

 I can instantly see this happening:

 void foo()  trusted
 {
      safe
     {
         //loads of code
     }

     //a few lines of system code, only safe due to context in 
 the  safe blocks

      safe
     {
         \\loads of code
     }
 }

 Is that what we want? I can't see why not, but it feels off 
 somehow... Effectively you've got  trusted blocks in an 
  trusted function, just inverted.


Some observations.

1. You cannot corrupt memory in the first  safe block. That's a 
plus.
2. It solves the 
"my- safe-function-turned- system-behind-my-back"-problem
3. It solves the problems the current  trusted-abuse in std.file 
tried to solve

And most importantly:
If that function can be made  trusted, there probably is a 
function bar, so that

void bar() { .. }  trusted

void foo()  safe
{

     safe {}
    bar();
     safe {}
}
Feb 06 2015
prev sibling next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 16:19:26 UTC, John Colvin wrote:

 On Friday, 6 February 2015 at 16:11:31 UTC, Andrei Alexandrescu 
 wrote:

 On 2/6/15 3:57 AM, Martin Krejcirik wrote:

 If I understand it correctly, Walter is against adding 
 trusted blocks
 (trusted {...}) into  safe functions. But what about having 
 safe blocks
 in  trusted functions ?


 That would be sensible - perhaps the best step forward 
 following this long discussion. -- Andrei


 It feels inelegant, but it might be the best way out of a bad 
 situation.

 I can instantly see this happening:

 void foo()  trusted
 {
      safe
     {
         //loads of code
     }

     //a few lines of system code, only safe due to context in 
 the  safe blocks

      safe
     {
         \\loads of code
     }
 }

 Is that what we want? I can't see why not, but it feels off 
 somehow... Effectively you've got  trusted blocks in an 
  trusted function, just inverted.


It's been suggested that ' system' be used to mark the blocks in 
question. The point would be to emphasize the dangerousness of 
the operation. The function is still  trusted, but inside, the 
 system code is marked as such.
Feb 06 2015
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 February 2015 at 17:13:09 UTC, Zach the Mystic wrote:

 It's been suggested that ' system' be used to mark the blocks 
 in question. The point would be to emphasize the dangerousness 
 of the operation. The function is still  trusted, but inside, 
 the  system code is marked as such.


This is precisely equivalent to just making the function  safe 
and using  trusted blocks inside (which is incidentally my 
preferred approach, and similar to what parts of Phobos try to 
emulate).

As I argued almost three years ago in my " trusted considered 
harmful" post,  trusted doesn't differ in meaning from  safe for 
API clients. Both mean that you can call the function from  safe 
code, nothing more, nothing less. I hope we agree on that.

You might argue that having  trusted in the function declaration 
serves as an indicator that the implementation requires extra 
careful review. However, there will *still* be a  trusted keyword 
sitting in the source code and staring at your face even with 
 trusted blocks. And if you don't have the source code for a 
function you call to begin with, all bets are off anyway (i.e. 
you need to trust its author) as  safe functions may always 
internally call  trusted ones.

If the leadership feels that your proposal is to be preferred 
because it keeps  trusted in the function signature (as he 
correctly remarked, the safety of  trusted blocks might depend on 
surrounding  safe code), then so be it. However, it makes an 
implementation detail part of the API (see " trusted considered 
harmful"), and furthermore is not backwards-compatible.

David
Feb 06 2015
next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 18:58:27 UTC, David Nadlinger wrote:

 On Friday, 6 February 2015 at 17:13:09 UTC, Zach the Mystic 
 wrote:

 It's been suggested that ' system' be used to mark the blocks 
 in question. The point would be to emphasize the dangerousness 
 of the operation. The function is still  trusted, but inside, 
 the  system code is marked as such.


 This is precisely equivalent to just making the function  safe 
 and using  trusted blocks inside (which is incidentally my 
 preferred approach, and similar to what parts of Phobos try to 
 emulate).


No it's not. Forcing  trusted functions, with  system blocks 
inside builds a redundancy into the system. Now someone from 
outside the can read the signature and know there might be a 
problem, and someone inside can find everywhere that problem 
could possibly have arisen from. The difference is you now have 
two signals instead of one - one in the function header, and the 
other(s) in the code itself.


 As I argued almost three years ago in my " trusted considered 
 harmful" post,  trusted doesn't differ in meaning from  safe 
 for API clients. Both mean that you can call the function from 
  safe code, nothing more, nothing less. I hope we agree on that.


I know. This was recently pointed out again by Steven 
Schveighoffer, and it's a great insight.


 You might argue that having  trusted in the function 
 declaration serves as an indicator that the implementation 
 requires extra careful review. However, there will *still* be a 
  trusted keyword sitting in the source code and staring at your 
 face even with  trusted blocks. And if you don't have the 
 source code for a function you call to begin with, all bets are 
 off anyway (i.e. you need to trust its author) as  safe 
 functions may always internally call  trusted ones.


It is a redundancy, yes, but one people might actually 
*appreciate*. People around here take memory safety very 
seriously, and probably with good reason!


 If the leadership feels that your proposal is to be preferred 
 because it keeps  trusted in the function signature (as he 
 correctly remarked, the safety of  trusted blocks might depend 
 on surrounding  safe code), then so be it. However, it makes an 
 implementation detail part of the API (see " trusted considered 
 harmful"), and furthermore is not backwards-compatible.


It's not backward-compatible, but it's not hard to fix at all. 
You get an error, and you enclose your unsafe code in a  system 
block. And the quality of your code improves at the same time.

The only reason the implementation detail,  trusted, was made 
part of the API was precisely because memory safety is taken 
extremely seriously by the language designers. There's no other 
reason I can think of, but for some reason, I can appreciate the 
thought. Also, I want to find a real solution and I doubt anyone 
is suggesting getting rid of  trusted at this point.
Feb 06 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 10:58 AM, David Nadlinger wrote:

  trusted doesn't differ in meaning from  safe for API clients. Both mean that
 you can call the function from  safe code, nothing more, nothing less. I hope
we
 agree on that.


That is correct.  trusted is a statement about the implementation of a
function, 
not its interface.

This suggests that  trusted should apply to blocks of code, not function 
declarations. Pedantically, I think that would be correct. But as we've seen in 
usage, this seductively makes for easy incorrect usage of  trusted. 
Pragmatically, the language should make it harder to write incorrect code.

Hence, I view it as the best compromise to make  trusted also apply only at the 
function level.
Feb 06 2015
parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 21:33:01 UTC, Walter Bright wrote:

 On 2/6/2015 10:58 AM, David Nadlinger wrote:

  trusted doesn't differ in meaning from  safe for API clients. 
 Both mean that
 you can call the function from  safe code, nothing more, 
 nothing less. I hope we
 agree on that.


 That is correct.  trusted is a statement about the 
 implementation of a function, not its interface.

 This suggests that  trusted should apply to blocks of code, not 
 function declarations. Pedantically, I think that would be 
 correct. But as we've seen in usage, this seductively makes for 
 easy incorrect usage of  trusted. Pragmatically, the language 
 should make it harder to write incorrect code.

 Hence, I view it as the best compromise to make  trusted also 
 apply only at the function level.


Please see this post:

http://forum.dlang.org/post/riphxcqazksykafumzcg forum.dlang.org
Feb 06 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 8:19 AM, John Colvin wrote:

 Is that what we want? I can't see why not, but it feels off somehow...
 Effectively you've got  trusted blocks in an  trusted function, just inverted.


Right. It just inverted the interface problem, the problem remains. So, no for 
this idea.

Another way to say it:

 safe =>  trusted must go through a safe interface

 trusted =>  safe must go through a safe interface
Feb 06 2015
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 February 2015 at 16:11:31 UTC, Andrei Alexandrescu 
wrote:

 On 2/6/15 3:57 AM, Martin Krejcirik wrote:

 If I understand it correctly, Walter is against adding trusted 
 blocks
 (trusted {...}) into  safe functions. But what about having 
 safe blocks
 in  trusted functions ?


 That would be sensible - perhaps the best step forward 
 following this long discussion. -- Andrei


This still does not solve the template inference problem though, 
unless you make it a "non- trusted" block instead of requiring 
 safe-ty. And then you'd end up with the—at least to my 
eyes—rather absurd situation that a template function that is 
marked  trusted might actually end up being  system.

David
Feb 06 2015
next sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
On Friday, 6 February 2015 at 16:40:10 UTC, David Nadlinger wrote:

 This still does not solve the template inference problem 
 though, unless you make it a "non- trusted" block instead of 
 requiring  safe-ty.


Don't understand. If that matters, the code shouldn't be marked 
 trusted
in the first place.


 And then you'd end up with the—at least to my eyes—rather 
 absurd situation that a template function that is marked 
  trusted might actually end up being  system.

 David


How?
Feb 06 2015
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 February 2015 at 16:50:51 UTC, Tobias Pankrath wrote:

 On Friday, 6 February 2015 at 16:40:10 UTC, David Nadlinger 
 wrote:

 This still does not solve the template inference problem 
 though, unless you make it a "non- trusted" block instead of 
 requiring  safe-ty.


 Don't understand. If that matters, the code shouldn't be marked 
  trusted
 in the first place.


Let's say you have a template function that accepts a range. For 
performance, you want to do some of the processing in a way that 
is  system, but can be verified to be correct for all inputs in 
this specific case. In other words, that piece of code can be 
rightfully  trusted. However, marking the whole function as 
 trusted would be a mistake, as the primitives of the range that 
is your input data might be  system.

Using  trusted blocks (which is what is currently emulated by the 
nested functions/lambdas), you can just mark your unsafe code as 
 trusted and let the compiler figure out the safety of the whole 
function.  safe blocks wouldn't work for this, as you'd 
inadvertently require the user-supplied input range to have 
 safe/ trusted primitives.


 And then you'd end up with the—at least to my eyes—rather 
 absurd situation that a template function that is marked 
  trusted might actually end up being  system.

 David


 How?


I was referring to a hypothetical "untrusted" block that might be 
used something like this:

---
void foo(Range)(Range r)  trusted {
   // ...

   untrusted {
     r.front;
   }

   // Your manually checked code.

   untrusted {
     r.popFront;
   }

   // …
}
---

This seems incredibly backwards. Not only is it confusing to 
read, it also encourages bugs where operations mistakenly end up 
being  trusted by forgetting to mark, say, an added call to 
r.empty as untrusted. On the other hand, it is much more unlikely 
that you accidentally add a new  trusted block. It seems obvious 
that explicitly whitelisting a small number of potentially 
dangerous but safe operations is much less error-prone approach 
than disabling compiler checks for everything and then having to 
remember to blacklist all unverified external dependencies.

David
Feb 06 2015
next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 17:12:40 UTC, David Nadlinger wrote:

 Let's say you have a template function that accepts a range. 
 For performance, you want to do some of the processing in a way 
 that is  system, but can be verified to be correct for all 
 inputs in this specific case. In other words, that piece of 
 code can be rightfully  trusted. However, marking the whole 
 function as  trusted would be a mistake, as the primitives of 
 the range that is your input data might be  system.

 Using  trusted blocks (which is what is currently emulated by 
 the nested functions/lambdas), you can just mark your unsafe 
 code as  trusted and let the compiler figure out the safety of 
 the whole function.  safe blocks wouldn't work for this, as 
 you'd inadvertently require the user-supplied input range to 
 have  safe/ trusted primitives.


I'm trying to promote suggesting ' system' blocks instead of 
' trusted'. ' trusted' functions, but ' system' blocks - which 
can only go in  trusted functions ( system block in  system 
functions are redundant). It's the same semantics, but it might 
win the day because the intent is to isolate the  system code, 
while still presenting a  trusted interface, as seems so 
important to the leadership.
Feb 06 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
 I'm trying to promote suggesting ' system' blocks instead of 
 ' trusted'. ' trusted' functions, but ' system' blocks - which 
 can only go in  trusted functions ( system block in  system 
 functions are redundant). It's the same semantics, but it might 
 win the day because the intent is to isolate the  system code, 
 while still presenting a  trusted interface, as seems so 
 important to the leadership.


That might be better than using  safe inside  trusted:

 trusted void myfunc() {
//implicitly safe
...
 system { //wouldn't compile otherwise.
    auto ptr = cast(ubyte*)4;
}

//implicitly safe again
}
Feb 06 2015
next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 17:36:27 UTC, Atila Neves wrote:

 I'm trying to promote suggesting ' system' blocks instead of 
 ' trusted'. ' trusted' functions, but ' system' blocks - which 
 can only go in  trusted functions ( system block in  system 
 functions are redundant). It's the same semantics, but it 
 might win the day because the intent is to isolate the  system 
 code, while still presenting a  trusted interface, as seems so 
 important to the leadership.


 That might be better than using  safe inside  trusted:

  trusted void myfunc() {
 //implicitly safe
 ...
  system { //wouldn't compile otherwise.
    auto ptr = cast(ubyte*)4;
 }

 //implicitly safe again
 }


Exactly. I think this addresses the concerns. If I read Walter's 
OP correctly, it's precisely the use of the word ' trusted' that 
he opposes, unless it's built into an interface like a function 
signature. Also, a  system block could be one statement long, if 
I'm not mistaken, in which case the above could look like:

 trusted void myfunc() {
//implicitly safe
...
 system auto ptr = cast(ubyte*)4;

//implicitly safe again
}
Feb 06 2015
parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 18:30:03 UTC, Zach the Mystic wrote:

 That might be better than using  safe inside  trusted:

  trusted void myfunc() {
 //implicitly safe
 ...
  system { //wouldn't compile otherwise.
   auto ptr = cast(ubyte*)4;
 }

 //implicitly safe again
 }


 Exactly. I think this addresses the concerns. If I read 
 Walter's OP correctly, it's precisely the use of the word 
 ' trusted' that he opposes, unless it's built into an interface 
 like a function signature. Also, a  system block could be one 
 statement long, if I'm not mistaken, in which case the above 
 could look like:

  trusted void myfunc() {
 //implicitly safe
 ...
  system auto ptr = cast(ubyte*)4;

 //implicitly safe again
 }


If this is a real solution, it's kind of exciting! :-)
Feb 06 2015
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 12:36 PM, Atila Neves wrote:

 I'm trying to promote suggesting ' system' blocks instead of
 ' trusted'. ' trusted' functions, but ' system' blocks - which can
 only go in  trusted functions ( system block in  system functions are
 redundant). It's the same semantics, but it might win the day because
 the intent is to isolate the  system code, while still presenting a
  trusted interface, as seems so important to the leadership.


 That might be better than using  safe inside  trusted:

  trusted void myfunc() {
 //implicitly safe
 ....
  system { //wouldn't compile otherwise.
     auto ptr = cast(ubyte*)4;
 }

 //implicitly safe again
 }


BTW, this is H.S. Teoh's suggestion too, and I like it better than mine 
after much thought.

-Steve
Feb 06 2015
prev sibling next sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
 I was referring to a hypothetical "untrusted" block that might 
 be used something like this:

 ---
 void foo(Range)(Range r)  trusted {
   // ...

   untrusted {
     r.front;
   }

   // Your manually checked code.

   untrusted {
     r.popFront;
   }

   // …
 }
 ---


Using current semantics we must not mark foo  trusted, if r.front 
and
r.popFront aren't. Using the proposed  safe-blocks (are those 
untrusted blocks the same?) we could guarantee that, by wrapping 
the use of r.front and r.popFront in  safe-blocks.

This is limiting because now we cannot provide an foo marked 
 system for all  system ranges without boiler plate or 
duplicating the function.

Correct?
Feb 06 2015
parent "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 February 2015 at 17:50:05 UTC, Tobias Pankrath wrote:

 I was referring to a hypothetical "untrusted" block that might 
 be used something like this:

 ---
 void foo(Range)(Range r)  trusted {
  // ...

  untrusted {
    r.front;
  }

  // Your manually checked code.

  untrusted {
    r.popFront;
  }

  // …
 }
 ---


 Using current semantics we must not mark foo  trusted, if 
 r.front and
 r.popFront aren't. Using the proposed  safe-blocks (are those 
 untrusted blocks the same?) we could guarantee that, by 
 wrapping the use of r.front and r.popFront in  safe-blocks.

 This is limiting because now we cannot provide an foo marked 
  system for all  system ranges without boiler plate or 
 duplicating the function.

 Correct?


Yes.

The "untrusted" blocks were an ad-hoc invention to show how  safe 
blocks could be modified to actually work in this case (for 
template functions, don't require  safe, but mark the function 
 system if the contents are unsafe), but that this modification 
results in a much less desirable design than just straight up 
having  trusted blocks.

David
Feb 06 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 17:12:40 UTC, David Nadlinger wrote:

 It seems obvious that explicitly whitelisting a small number of 
 potentially dangerous but safe operations is much less 
 error-prone approach than disabling compiler checks for 
 everything and then having to remember to blacklist all 
 unverified external dependencies.

 David


That seems obvious to me too. Isn't the whole purpose of having 
' trusted' in the first place to direct a programmer who's having 
memory safety problems to the potential sources those problems? 
But why have this and then stop at the function level? Why not 
force the programmer to tag precisely those portions of his code 
which cause him to tag his function  trusted to begin with? Why 
help him get to the function, and then leave him hanging out to 
dry once inside the function?
Feb 06 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 8:40 AM, David Nadlinger wrote:

 On Friday, 6 February 2015 at 16:11:31 UTC, Andrei Alexandrescu wrote:

 On 2/6/15 3:57 AM, Martin Krejcirik wrote:

 If I understand it correctly, Walter is against adding trusted blocks
 (trusted {...}) into  safe functions. But what about having safe blocks
 in  trusted functions ?


 That would be sensible - perhaps the best step forward following this
 long discussion. -- Andrei


 This still does not solve the template inference problem


What is that?


 though, unless
 you make it a "non- trusted" block instead of requiring  safe-ty. And
 then you'd end up with the—at least to my eyes—rather absurd situation
 that a template function that is marked  trusted might actually end up
 being  system.


 trusted functions are  system.


Andrei
Feb 06 2015
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 February 2015 at 17:16:19 UTC, Andrei Alexandrescu 
wrote:

 On 2/6/15 8:40 AM, David Nadlinger wrote:

 This still does not solve the template inference problem


 What is that?


See my reply to Tobias [1]. This seems to be the crux of our 
disagreement and/or misunderstanding, and is precisely the reason 
why I recommended you to try your hand at rewriting some of the 
std.array range algorithms yourself in the Bugzilla discussion. 
Let me know if the explanation in said post is not clear enough.


 though, unless
 you make it a "non- trusted" block instead of requiring 
  safe-ty. And
 then you'd end up with the—at least to my eyes—rather absurd 
 situation
 that a template function that is marked  trusted might 
 actually end up
 being  system.


  trusted functions are  system.


I meant " system" as in the part of the function type/API 
contract, not concerning the implementation. That is, a template 
function that is marked  trusted might not be able to be called 
from  safe code.

David


[1] 
http://forum.dlang.org/post/fwiivepmjfvqbxgagcrm forum.dlang.org
Feb 06 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 9:28 AM, David Nadlinger wrote:

 On Friday, 6 February 2015 at 17:16:19 UTC, Andrei Alexandrescu wrote:

 On 2/6/15 8:40 AM, David Nadlinger wrote:

 This still does not solve the template inference problem


 What is that?


 See my reply to Tobias [1]. This seems to be the crux of our
 disagreement and/or misunderstanding, and is precisely the reason why I
 recommended you to try your hand at rewriting some of the std.array
 range algorithms yourself in the Bugzilla discussion. Let me know if the
 explanation in said post is not clear enough.


It's clear. I just don't think it's a good point. -- Andrei
Feb 06 2015
parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu 
wrote:

 It's clear. I just don't think it's a good point. -- Andrei


I'm not making a point; I'm posing a problem. What is your 
solution?

David
Feb 06 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 10:42 AM, David Nadlinger wrote:

 On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu wrote:

 It's clear. I just don't think it's a good point. -- Andrei


 I'm not making a point; I'm posing a problem. What is your solution?


I think the problem is overstated. -- Andrei
Feb 06 2015
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Feb 06, 2015 at 10:52:45AM -0800, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 2/6/15 10:42 AM, David Nadlinger wrote:

On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu wrote:

It's clear. I just don't think it's a good point. -- Andrei


I'm not making a point; I'm posing a problem. What is your solution?


 
 I think the problem is overstated. -- Andrei


This is precisely why I have lost all interest in  safe. It's clear that
the present problematic situation will continue to hold, and the
decision makers are not interested to address it. I am not going to
waste any more time and energy on this topic.


T

-- 
Question authority. Don't ask why, just do it.
Feb 06 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 11:11 AM, H. S. Teoh via Digitalmars-d wrote:

 On Fri, Feb 06, 2015 at 10:52:45AM -0800, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 2/6/15 10:42 AM, David Nadlinger wrote:

 On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu wrote:

 It's clear. I just don't think it's a good point. -- Andrei


 I'm not making a point; I'm posing a problem. What is your solution?


 I think the problem is overstated. -- Andrei


 This is precisely why I have lost all interest in  safe. It's clear that
 the present problematic situation will continue to hold, and the
 decision makers are not interested to address it. I am not going to
 waste any more time and energy on this topic.


I've asked repeatedly for evidence of the "problematic situation", and 
all I got was doomsday predictions "maintenance nightmare!". If you have 
such, please show it. If not, thanks for a good course of action. -- Andrei
Feb 06 2015
next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 19:16:13 UTC, Andrei Alexandrescu 
wrote:

 This is precisely why I have lost all interest in  safe. It's 
 clear that
 the present problematic situation will continue to hold, and 
 the
 decision makers are not interested to address it. I am not 
 going to
 waste any more time and energy on this topic.


 I've asked repeatedly for evidence of the "problematic 
 situation", and all I got was doomsday predictions "maintenance 
 nightmare!". If you have such, please show it. If not, thanks 
 for a good course of action. -- Andrei


The best evidence I have it that everyone who actually worked on 
the phobos code you disapprove of says it's a problem. Argumentum 
ad populem, I know, but it calls for close scrutiny, to say the 
least.

My attitude is not based on evidence. It's based on just thinking 
about the problem:

http://forum.dlang.org/post/eeglnychgudcffpjcdvy forum.dlang.org

I really can't answer this question.
Feb 06 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 11:29 AM, Zach the Mystic wrote:

 The best evidence I have it that everyone who actually worked on the
 phobos code you disapprove of says it's a problem.


I see more like two. -- Andrei
Feb 06 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 11:29 AM, Zach the Mystic wrote:

 My attitude is not based on evidence. It's based on just thinking about the
 problem:

 http://forum.dlang.org/post/eeglnychgudcffpjcdvy forum.dlang.org

 I really can't answer this question.


You asked:

"Why not force the programmer to tag precisely those portions of his code
which cause him to tag his function  trusted to begin with?"

That question has been asked and answered repeatedly. The answer is that 
 trusted is not only to TAG unsafe code, but must provide a SAFE INTERFACE to
it.

     trusted {
       ... unsafe code ...
    }

provides no indication of what the interface to the unsafe code is. Addressing 
only the TAG aspect is insufficient.
Feb 06 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 21:56:40 UTC, Walter Bright wrote:

 On 2/6/2015 11:29 AM, Zach the Mystic wrote:

 My attitude is not based on evidence. It's based on just 
 thinking about the
 problem:

 http://forum.dlang.org/post/eeglnychgudcffpjcdvy forum.dlang.org

 I really can't answer this question.


 You asked:

 "Why not force the programmer to tag precisely those portions 
 of his code
 which cause him to tag his function  trusted to begin with?"

 That question has been asked and answered repeatedly. The 
 answer is that  trusted is not only to TAG unsafe code, but 
 must provide a SAFE INTERFACE to it.

     trusted {
       ... unsafe code ...
    }

 provides no indication of what the interface to the unsafe code 
 is. Addressing only the TAG aspect is insufficient.


No, at least three of us, Steven, H.S. Teoh and myself have 
confirmed that we've moved beyond requesting  trusted blocks. We 
are no longer requesting them. We are requesting * system* 
blocks, which can only appear in  trusted and  system functions. 
Any unsafe code appearing in a  trusted function which is not 
inside a  system block is an error. We've changed the name, but I 
think it will make a world of difference regarding how you will 
look at it. Marking ' system' code inside a  trusted function is 
exactly what is requested. Your message about ' trusted' being 
only acceptable as an interface has been heard. There will be no 
 trusted blocks, only  system blocks, which are the exact same 
thing, except they can only appear in  trusted and  system 
functions.

This solution appeals to me greatly. It pinpoints precisely where 
unsafe code can generate; it catches unintended safety violations 
in all  trusted code outside  system blocks, as requested by many 
people so far; it makes systems programming highly visible, with 
redundancy at the function signature and at the unsafe code 
itself. I really think it's spot on!
Feb 06 2015
next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote:

 No, at least three of us, Steven, H.S. Teoh and myself have 
 confirmed that we've moved beyond requesting  trusted blocks. 
 We are no longer requesting them. We are requesting * system* 
 blocks, which can only appear in  trusted and  system 
 functions. Any unsafe code appearing in a  trusted function 
 which is not inside a  system block is an error. We've changed 
 the name, but I think it will make a world of difference 
 regarding how you will look at it. Marking ' system' code 
 inside a  trusted function is exactly what is requested. Your 
 message about ' trusted' being only acceptable as an interface 
 has been heard. There will be no  trusted blocks, only  system 
 blocks, which are the exact same thing, except they can only 
 appear in  trusted and  system functions.


Also, to be clear, any  system block inside a  safe function is 
an automatic error.
Feb 06 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote:


 No, at least three of us, Steven, H.S. Teoh and myself have 
 confirmed that we've moved beyond requesting  trusted blocks. 
 We are no longer requesting them. We are requesting * system* 
 blocks, which can only appear in  trusted and  system 
 functions. Any unsafe code appearing in a  trusted function 
 which is not inside a  system block is an error. We've changed 
 the name, but I think it will make a world of difference 
 regarding how you will look at it. Marking ' system' code 
 inside a  trusted function is exactly what is requested. Your 
 message about ' trusted' being only acceptable as an interface 
 has been heard. There will be no  trusted blocks, only  system 
 blocks, which are the exact same thing, except they can only 
 appear in  trusted and  system functions.

 This solution appeals to me greatly. It pinpoints precisely 
 where unsafe code can generate; it catches unintended safety 
 violations in all  trusted code outside  system blocks, as 
 requested by many people so far; it makes systems programming 
 highly visible, with redundancy at the function signature and 
 at the unsafe code itself. I really think it's spot on!


this sounds interesting, is anyone going to make a DIP for it?
Feb 06 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 3:21 PM, weaselcat wrote:

 On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote:


 No, at least three of us, Steven, H.S. Teoh and myself have confirmed
 that we've moved beyond requesting  trusted blocks. We are no longer
 requesting them. We are requesting * system* blocks, which can only
 appear in  trusted and  system functions. Any unsafe code appearing in
 a  trusted function which is not inside a  system block is an error.
 We've changed the name, but I think it will make a world of difference
 regarding how you will look at it. Marking ' system' code inside a
  trusted function is exactly what is requested. Your message about
 ' trusted' being only acceptable as an interface has been heard. There
 will be no  trusted blocks, only  system blocks, which are the exact
 same thing, except they can only appear in  trusted and  system
 functions.

 This solution appeals to me greatly. It pinpoints precisely where
 unsafe code can generate; it catches unintended safety violations in
 all  trusted code outside  system blocks, as requested by many people
 so far; it makes systems programming highly visible, with redundancy
 at the function signature and at the unsafe code itself. I really
 think it's spot on!


 this sounds interesting, is anyone going to make a DIP for it?


Consider the previous code:

https://github.com/D-Programming-Language/phobos/blob/accb351b96bb04a6890bb7df018749337e55eccc/std/file.d#L194

that got replaced with:

https://github.com/D-Programming-Language/phobos/blob/master/std/file.d#L194

With the system proposal we're looking at something like:

version (Posix) void[] read(in char[] name, size_t upTo = size_t.max) 
 trusted
{
     import core.memory;
     // A few internal configuration parameters {
     enum size_t
         minInitialAlloc = 1024 * 4,
         maxInitialAlloc = size_t.max / 2,
         sizeIncrement = 1024 * 16,
         maxSlackMemoryAllowed = 1024;
     // }

      system
     {
         immutable fd = core.sys.posix.fcntl.open(name.tempCString(),
             core.sys.posix.fcntl.O_RDONLY);
     }
     cenforce(fd != -1, name);
     scope(exit) core.sys.posix.unistd.close(fd);

     stat_t statbuf = void;
      system
     {
         cenforce(trustedFstat(fd, trustedRef(statbuf)) == 0, name);
     }

     immutable initialAlloc = to!size_t(statbuf.st_size
         ? min(statbuf.st_size + 1, maxInitialAlloc)
         : minInitialAlloc);
     void[] result = uninitializedArray!(ubyte[])(initialAlloc);
     scope(failure) delete result;
     size_t size = 0;

     for (;;)
     {
          system
         {
             immutable actual = core.sys.posix.unistd.read(fd, 
result.ptr + size),
                 min(result.length, upTo) - size);
         }
         cenforce(actual != -1, name);
         if (actual == 0) break;
         size += actual;
         if (size < result.length) continue;
         immutable newAlloc = size + sizeIncrement;
          system
         {
             result = GC.realloc(result.ptr, newAlloc, 
GC.BlkAttr.NO_SCAN)[0 .. newAlloc];
         }

      system
     {
         return result.length - size >= maxSlackMemoryAllowed
             ? GC.realloc(result.ptr, size, GC.BlkAttr.NO_SCAN)[0 .. size]
             : result[0 .. size];
     }
}

We want to move D forward, folks. This is not it.


Andrei
Feb 06 2015
next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Saturday, 7 February 2015 at 01:43:01 UTC, Andrei Alexandrescu 
wrote:

 With the system proposal we're looking at something like:

 version (Posix) void[] read(in char[] name, size_t upTo = 
 size_t.max)  trusted
 {
     import core.memory;
     // A few internal configuration parameters {
     enum size_t
         minInitialAlloc = 1024 * 4,
         maxInitialAlloc = size_t.max / 2,
         sizeIncrement = 1024 * 16,
         maxSlackMemoryAllowed = 1024;
     // }

      system
     {
         immutable fd = 
 core.sys.posix.fcntl.open(name.tempCString(),
             core.sys.posix.fcntl.O_RDONLY);
     }
     cenforce(fd != -1, name);
     scope(exit) core.sys.posix.unistd.close(fd);

     stat_t statbuf = void;
      system
     {
         cenforce(trustedFstat(fd, trustedRef(statbuf)) == 0, 
 name);
     }

     immutable initialAlloc = to!size_t(statbuf.st_size
         ? min(statbuf.st_size + 1, maxInitialAlloc)
         : minInitialAlloc);
     void[] result = uninitializedArray!(ubyte[])(initialAlloc);
     scope(failure) delete result;
     size_t size = 0;

     for (;;)
     {
          system
         {
             immutable actual = core.sys.posix.unistd.read(fd, 
 result.ptr + size),
                 min(result.length, upTo) - size);
         }
         cenforce(actual != -1, name);
         if (actual == 0) break;
         size += actual;
         if (size < result.length) continue;
         immutable newAlloc = size + sizeIncrement;
          system
         {
             result = GC.realloc(result.ptr, newAlloc, 
 GC.BlkAttr.NO_SCAN)[0 .. newAlloc];
         }

      system
     {
         return result.length - size >= maxSlackMemoryAllowed
             ? GC.realloc(result.ptr, size, 
 GC.BlkAttr.NO_SCAN)[0 .. size]
             : result[0 .. size];
     }
 }

 We want to move D forward, folks. This is not it.


 Andrei


Oh I see. There are three posts, in the latter two of which the 
little  trusted functions are already removed. I had thought they 
were all identical, but you obviously realized the little 
functions should be removed.
Feb 06 2015
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 8:43 PM, Andrei Alexandrescu wrote:

 On 2/6/15 3:21 PM, weaselcat wrote:

 On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote:


 No, at least three of us, Steven, H.S. Teoh and myself have confirmed
 that we've moved beyond requesting  trusted blocks. We are no longer
 requesting them. We are requesting * system* blocks, which can only
 appear in  trusted and  system functions. Any unsafe code appearing in
 a  trusted function which is not inside a  system block is an error.
 We've changed the name, but I think it will make a world of difference
 regarding how you will look at it. Marking ' system' code inside a
  trusted function is exactly what is requested. Your message about
 ' trusted' being only acceptable as an interface has been heard. There
 will be no  trusted blocks, only  system blocks, which are the exact
 same thing, except they can only appear in  trusted and  system
 functions.

 This solution appeals to me greatly. It pinpoints precisely where
 unsafe code can generate; it catches unintended safety violations in
 all  trusted code outside  system blocks, as requested by many people
 so far; it makes systems programming highly visible, with redundancy
 at the function signature and at the unsafe code itself. I really
 think it's spot on!


 this sounds interesting, is anyone going to make a DIP for it?


 Consider the previous code:

 https://github.com/D-Programming-Language/phobos/blob/accb351b96bb04a6890bb7df018749337e55eccc/std/file.d#L194


 that got replaced with:

 https://github.com/D-Programming-Language/phobos/blob/master/std/file.d#L194


 With the system proposal we're looking at something like:


Please understand, Nobody is saying "let's replace incorrect code with 
the same incorrect code with different tags!"

The idea is to properly identify which code needs more scrutiny, and 
keep the mechanical checking of  safe where we can.

-Steve
Feb 07 2015
prev sibling next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Saturday, 7 February 2015 at 01:41:19 UTC, Andrei Alexandrescu 
wrote:

 Consider the previous code:
 [...]
     static trustedRead(int fildes, void* buf, size_t nbyte) 
  trusted
     {
         return core.sys.posix.unistd.read(fildes, buf, nbyte);
     }
     static trustedRealloc(void* p, size_t sz, uint ba = 0, 
 const TypeInfo ti = null)  trusted
     {
         return GC.realloc(p, sz, ba, ti);
     }
     static trustedPtrAdd(void[] buf, size_t s)  trusted
     {
         return buf.ptr+s;
     }
     static trustedPtrSlicing(void* ptr, size_t lb, size_t ub) 
  trusted
     {
         return ptr[lb..ub];
     }


First of all, these little  trusted functions are made obsolete 
by the new system. They should certainly be omitted.


      system
     {
         immutable fd = 
 core.sys.posix.fcntl.open(name.tempCString(),
             core.sys.posix.fcntl.O_RDONLY);
     }


Next, you have to realize that  system blocks are like 'try' 
blocks: You don't need brackets if there's only one statement in 
them.

Here's how I would rewrite what you have written using the new 
method:

version (Posix) void[] read(in char[] name, size_t upTo = 
size_t.max)
 trusted
{
     import core.memory;
     // A few internal configuration parameters {
     enum size_t
         minInitialAlloc = 1024 * 4,
         maxInitialAlloc = size_t.max / 2,
         sizeIncrement = 1024 * 16,
         maxSlackMemoryAllowed = 1024;
     // }

      system immutable fd = 
core.sys.posix.fcntl.open(name.tempCString(),
             core.sys.posix.fcntl.O_RDONLY);
     cenforce(fd != -1, name);
     scope(exit) core.sys.posix.unistd.close(fd);

     stat_t statbuf = void;
      system cenforce(trustedFstat(fd, trustedRef(statbuf)) == 0, 
name);

     immutable initialAlloc = to!size_t(statbuf.st_size
         ? min(statbuf.st_size + 1, maxInitialAlloc)
         : minInitialAlloc);
     void[] result = uninitializedArray!(ubyte[])(initialAlloc);
     scope(failure) delete result;
     size_t size = 0;

     for (;;)
     {
          system immutable actual = core.sys.posix.unistd.read(fd,
                 result.ptr + size, min(result.length, upTo) - 
size);
         cenforce(actual != -1, name);
         if (actual == 0) break;
         size += actual;
         if (size < result.length) continue;
         immutable newAlloc = size + sizeIncrement;
          system result = GC.realloc(
                 result.ptr, newAlloc, GC.BlkAttr.NO_SCAN)[0 .. 
newAlloc];
     }

      system return result.length - size >= maxSlackMemoryAllowed
         ? GC.realloc(result.ptr, size, 
GC.BlkAttr.NO_SCAN)[0..size]
             : result[0 .. size];
}

Note that I just mechanically your  system blocks with the better 
form. I didn't arrange for them to be elegant. There's nothing 
wrong with encapsulating a  trusted sequence in a  system block 
with brackets, to aid future reviewers in identifying subsequent 
code thought to be affected by the  system statements. Also, few 
functions will have their  system blocks more or less evenly 
distributed throughout like the above function does.

The new proposal will never let you add an unsafe operation 
without your knowing it. It's definitely the way forward.
Feb 06 2015
next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Saturday, 7 February 2015 at 05:35:51 UTC, Zach the Mystic 
wrote:

 Note that I just mechanically your  system blocks with the 
 better form.


...mechanically replaced, I mean, of course.
Feb 06 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Saturday, 7 February 2015 at 05:35:51 UTC, Zach the Mystic 
wrote:

 Here's how I would rewrite what you have written using the new 
 method:
 ...
     stat_t statbuf = void;
      system cenforce(trustedFstat(fd, trustedRef(statbuf)) == 
 0, name);


I didn't rewrite this because I didn't see the trustedXXX 
functions they referred to, but the basic rewrite would be:

 system cenforce(fstat(fd, reff(statbuf) == 0), name);

In other words, just copy the  system code directly without going 
through  trusted.
Feb 06 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 23:21:50 UTC, weaselcat wrote:

 On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic 
 wrote:


 No, at least three of us, Steven, H.S. Teoh and myself have 
 confirmed that we've moved beyond requesting  trusted blocks. 
 We are no longer requesting them. We are requesting * system* 
 blocks, which can only appear in  trusted and  system 
 functions. Any unsafe code appearing in a  trusted function 
 which is not inside a  system block is an error. We've changed 
 the name, but I think it will make a world of difference 
 regarding how you will look at it. Marking ' system' code 
 inside a  trusted function is exactly what is requested. Your 
 message about ' trusted' being only acceptable as an interface 
 has been heard. There will be no  trusted blocks, only  system 
 blocks, which are the exact same thing, except they can only 
 appear in  trusted and  system functions.

 This solution appeals to me greatly. It pinpoints precisely 
 where unsafe code can generate; it catches unintended safety 
 violations in all  trusted code outside  system blocks, as 
 requested by many people so far; it makes systems programming 
 highly visible, with redundancy at the function signature and 
 at the unsafe code itself. I really think it's spot on!


 this sounds interesting, is anyone going to make a DIP for it?


It was Teoh's idea. Maybe he should have the honors?
Feb 07 2015
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 3:02 PM, Zach the Mystic wrote:

 No, at least three of us, Steven, H.S. Teoh and myself have confirmed that
we've
 moved beyond requesting  trusted blocks. We are no longer requesting them. We
 are requesting * system* blocks, which can only appear in  trusted and  system
 functions. Any unsafe code appearing in a  trusted function which is not inside
 a  system block is an error. We've changed the name, but I think it will make a
 world of difference regarding how you will look at it. Marking ' system' code
 inside a  trusted function is exactly what is requested. Your message about
 ' trusted' being only acceptable as an interface has been heard. There will be
 no  trusted blocks, only  system blocks, which are the exact same thing, except
 they can only appear in  trusted and  system functions.

 This solution appeals to me greatly. It pinpoints precisely where unsafe code
 can generate; it catches unintended safety violations in all  trusted code
 outside  system blocks, as requested by many people so far; it makes systems
 programming highly visible, with redundancy at the function signature and at
the
 unsafe code itself. I really think it's spot on!


I suspect that such a feature would simply lull people into a false sense of 
security in that merely tagging an unsafe cast with  system and the compiler 
accepting it is good enough.

My evidence for this is how  trusted was used in Phobos.
Feb 06 2015
next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 February 2015 at 23:25:02 UTC, Walter Bright wrote:

 I suspect that such a feature would simply lull people into a 
 false sense of security in that merely tagging an unsafe cast 
 with  system and the compiler accepting it is good enough.

 My evidence for this is how  trusted was used in Phobos.


How is adding  system to some operations *in addition to* adding 
 trusted to the function declaration more likely to lull people 
into a false sense of security than just adding  trusted right 
now?

Let me also point out that the cases where the  system block 
equivalent is used right now (like in std.file, or the trustedXyz 
functions in std.array) are NOT the ones that actually have 
safety bugs in them (such as std.array.uninitializedArray or 
std.uuid.randomUUID). The two latter examples were actually 
written in your preferred style.

David
Feb 06 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 3:34 PM, David Nadlinger wrote:

 On Friday, 6 February 2015 at 23:25:02 UTC, Walter Bright wrote:

 I suspect that such a feature would simply lull people into a false sense of
 security in that merely tagging an unsafe cast with  system and the compiler
 accepting it is good enough.

 My evidence for this is how  trusted was used in Phobos.


 How is adding  system to some operations *in addition to* adding  trusted to
the
 function declaration more likely to lull people into a false sense of security
 than just adding  trusted right now?

 Let me also point out that the cases where the  system block equivalent is used
 right now (like in std.file, or the trustedXyz functions in std.array) are NOT
 the ones that actually have safety bugs in them (such as
 std.array.uninitializedArray or std.uuid.randomUUID). The two latter examples
 were actually written in your preferred style.


I've said that the usage of those functions was not actually buggy, what was 
wrong about them was that they required review of the surrounding supposedly 
safe context. I.e. they produced a false sense of safety. I fear the  system 
blocks, even if only allowed in  trusted functions, would produce a similar 
illusion of safety.

I agree with Andrei in that I do not believe that reviewing a  trusted
function, 
line by line, for safety is necessarily some sort of maintenance nightmare. If 
it is, then a refactoring should be considered to encapsulate the unsafe code
in 
a smaller, simpler manner.

I.e. let's make an effort to use  trusted correctly, and then see where we
stand.

Scott Meyer's excellent article (a classic):

   http://www.drdobbs.com/cpp/how-non-member-functions-improve-encapsu/184401197

describes this most eloquently. (Just substitute "private members" with
"trusted 
code".)
Feb 06 2015
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Feb 06, 2015 at 04:04:48PM -0800, Walter Bright via Digitalmars-d wrote:
[...]

 I agree with Andrei in that I do not believe that reviewing a  trusted
 function, line by line, for safety is necessarily some sort of
 maintenance nightmare. If it is, then a refactoring should be
 considered to encapsulate the unsafe code in a smaller, simpler
 manner.


[...]

This does not take into the account the fact that a  trusted function
may call other, non- trusted, functions. When one of those other
functions changes, the  trusted function necessarily needs to be
reviewed again.

However, under the current implementation, this is not done because when
the other, non- trusted, function is modified, nobody thinks to
re-review the  trusted function. They may not even be in the same
module. There is no mechanism in place to raise a warning flag when a
 trusted function's dependencies are modified. Thus, the proof of safety
of the  trusted function has been invalidated, but trust continues to be
conferred upon it.


T

-- 
Let's call it an accidental feature. -- Larry Wall
Feb 06 2015
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 4:29 PM, H. S. Teoh via Digitalmars-d wrote:

 This does not take into the account the fact that a  trusted function
 may call other, non- trusted, functions. When one of those other
 functions changes, the  trusted function necessarily needs to be
 reviewed again.


That's correct.


 However, under the current implementation, this is not done because when
 the other, non- trusted, function is modified, nobody thinks to
 re-review the  trusted function. They may not even be in the same
 module. There is no mechanism in place to raise a warning flag when a
  trusted function's dependencies are modified. Thus, the proof of safety
 of the  trusted function has been invalidated, but trust continues to be
 conferred upon it.


When the interface to an  system function is changed, all uses of that function 
have to be reviewed, whether one thinks of it or not. This is part of the
review 
process. Having  system blocks does not alter that.
Feb 06 2015
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 7 February 2015 at 00:31:41 UTC, H. S. Teoh wrote:

 This does not take into the account the fact that a  trusted 
 function
 may call other, non- trusted, functions. When one of those other
 functions changes, the  trusted function necessarily needs to be
 reviewed again.

 However, under the current implementation, this is not done 
 because when
 the other, non- trusted, function is modified, nobody thinks to
 re-review the  trusted function. They may not even be in the 
 same
 module. There is no mechanism in place to raise a warning flag 
 when a
  trusted function's dependencies are modified. Thus, the proof 
 of safety
 of the  trusted function has been invalidated, but trust 
 continues to be
 conferred upon it.


So what you should ask for is a way to "sign"  trusted with a 
timestamp that indicates when it was last proven to be safe, 
e.g.:  trusted("2015-01-01T12:30:12z")

Then ask for a fine grained dependency tracking tool that can 
extract changes from git. Such a dependency tracking tool could 
be a nice stepping stone for faster compilation (sub-file-level 
recompilation). So there is synergies in this.

The proposal to confuse  trusted/ safe with requiring  system 
within  trusted it not a language issue. It is a process issue 
and can be done with lint-like tooling.

Keep  trusted/ safe/ system simple. Enough convoluted semantics 
in D already.
Feb 07 2015
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 7:29 PM, H. S. Teoh via Digitalmars-d wrote:

 On Fri, Feb 06, 2015 at 04:04:48PM -0800, Walter Bright via Digitalmars-d
wrote:
 [...]

 I agree with Andrei in that I do not believe that reviewing a  trusted
 function, line by line, for safety is necessarily some sort of
 maintenance nightmare. If it is, then a refactoring should be
 considered to encapsulate the unsafe code in a smaller, simpler
 manner.


 [...]

 This does not take into the account the fact that a  trusted function
 may call other, non- trusted, functions. When one of those other
 functions changes, the  trusted function necessarily needs to be
 reviewed again.


This problem isn't solved by the proposal, however. If you are calling a 
 system function inside a  trusted function, and you've marked it as 
 system, then changing the  system function does not affect the call.

However, if you changed a  safe function into a  system function, then a 
call inside a  trusted function would have to now be annotated.

It's important to note that our proposal will not fix cases where 
something subtle happens inside a  system block. What it DOES do is 
limit this effect to the  system block instead of the whole function.

-Steve
Feb 09 2015
prev sibling next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 23:25:02 UTC, Walter Bright wrote:

 This solution appeals to me greatly. It pinpoints precisely 
 where unsafe code
 can generate; it catches unintended safety violations in all 
  trusted code
 outside  system blocks, as requested by many people so far; it 
 makes systems
 programming highly visible, with redundancy at the function 
 signature and at the
 unsafe code itself. I really think it's spot on!


 I suspect that such a feature would simply lull people into a 
 false sense of security in that merely tagging an unsafe cast 
 with  system and the compiler accepting it is good enough.

 My evidence for this is how  trusted was used in Phobos.


You do realize that our proposal *tightens* security, with no 
loosening at all? No code which currently fails to compile will 
start compiling with this proposal. This is literally a breaking 
change which does nothing but cause errors in existing code - for 
the explicit purpose of making all code safer, which it will do, 
possibly dramatically.
Feb 06 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 23:25:02 UTC, Walter Bright wrote:

 On 2/6/2015 3:02 PM, Zach the Mystic wrote:

 This solution appeals to me greatly. It pinpoints precisely 
 where unsafe code
 can generate; it catches unintended safety violations in all 
  trusted code
 outside  system blocks, as requested by many people so far; it 
 makes systems
 programming highly visible, with redundancy at the function 
 signature and at the
 unsafe code itself. I really think it's spot on!


 I suspect that such a feature would simply lull people into a 
 false sense of security in that merely tagging an unsafe cast 
 with  system and the compiler accepting it is good enough.

 My evidence for this is how  trusted was used in Phobos.


I'm also not saying phobos was written perfectly to begin with. I 
think that this whole  system blocks suggestion came up in a 
slightly different context than did your original discontent with 
std.file and std.array. I'm not sure you're ever going to stop 
careless programmers from getting their bad code to compile. But 
I think that's a different issue from giving good, careful 
programmers the tools they need. Right now,  trusted functions 
are the only tool they have to help people figure out where the 
unsafe code is, but it's not enough. Nested  trusted functions, 
and  trusted lambas are clearly being added as a workaround for 
not being able to "un trust" large portions of code intended to 
be  safe.
Feb 07 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 6 February 2015 at 23:02:54 UTC, Zach the Mystic wrote:

 This solution appeals to me greatly. It pinpoints precisely 
 where unsafe code can generate; it catches unintended safety 
 violations in all  trusted code outside  system blocks, as 
 requested by many people so far; it makes systems programming 
 highly visible, with redundancy at the function signature and 
 at the unsafe code itself. I really think it's spot on!


Even a call to a  system function inside a  trusted function must 
occur inside a  system block. It's that strict.
Feb 06 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 2:16 PM, Andrei Alexandrescu wrote:

 On 2/6/15 11:11 AM, H. S. Teoh via Digitalmars-d wrote:

 On Fri, Feb 06, 2015 at 10:52:45AM -0800, Andrei Alexandrescu via
 Digitalmars-d wrote:

 On 2/6/15 10:42 AM, David Nadlinger wrote:

 On Friday, 6 February 2015 at 18:39:28 UTC, Andrei Alexandrescu wrote:

 It's clear. I just don't think it's a good point. -- Andrei


 I'm not making a point; I'm posing a problem. What is your solution?


 I think the problem is overstated. -- Andrei


 This is precisely why I have lost all interest in  safe. It's clear that
 the present problematic situation will continue to hold, and the
 decision makers are not interested to address it. I am not going to
 waste any more time and energy on this topic.


 I've asked repeatedly for evidence of the "problematic situation", and
 all I got was doomsday predictions "maintenance nightmare!". If you have
 such, please show it. If not, thanks for a good course of action. -- Andrei


I think your strawman is overstated. The "doomsday" is the current 
situation to which you and Walter have objected. If you think having 
"better discipline" in reviews is going to fix it, I guess we will have 
to wait and see what the evidence eventually does show. There isn't 
evidence that either solution has worked, because neither has been 
employed yet.

Logically, it makes sense to me that we should adjust how  trusted 
operates to prevent preventable problems that you have identified. But 
we can just keep the status quo and rely on manual process improvements 
instead.

It's not terribly important to fix it right now, we can try your way 
first, I don't see how adjusting the meaning of  trusted in the future 
would be any more disruptive than it would be now.

If this is how it is to be, can we get some guidelines as to what should 
and should not pass review for  trusted?

-Steve
Feb 06 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 11:53 AM, Steven Schveighoffer wrote:

 I think your strawman is overstated. The "doomsday" is the current
 situation to which you and Walter have objected.


I see that just as: code in poor style made its way in Phobos. It 
doesn't improve anything (e.g. didn't find bugs in std.file.read) and is 
just a net negative resulting from the corrupted use of a feature.

There's no completely automated protection against poor style.


 If you think having
 "better discipline" in reviews is going to fix it, I guess we will have
 to wait and see what the evidence eventually does show. There isn't
 evidence that either solution has worked, because neither has been
 employed yet.

 Logically, it makes sense to me that we should adjust how  trusted
 operates to prevent preventable problems that you have identified. But
 we can just keep the status quo and rely on manual process improvements
 instead.

 It's not terribly important to fix it right now, we can try your way
 first, I don't see how adjusting the meaning of  trusted in the future
 would be any more disruptive than it would be now.

 If this is how it is to be, can we get some guidelines as to what should
 and should not pass review for  trusted?


Wise words. Walter has a PR on docs, you may want to review it: 
https://github.com/D-Programming-Language/dlang.org/pull/890


Andrei
Feb 06 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 11:11 AM, H. S. Teoh via Digitalmars-d wrote:

 This is precisely why I have lost all interest in  safe. It's clear that
 the present problematic situation will continue to hold, and the
 decision makers are not interested to address it. I am not going to
 waste any more time and energy on this topic.


In this thread at 8:20PM last night, Dicebot asked me:

"I am not even sure how you can show the example though, to be honest - implied
issues are about maintaining code and not just writing it."

And I replied with a specific example of how to fix one aspect of std.array. 
There have been no replies. What else can I do to address it?
Feb 06 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 4:48 PM, Walter Bright wrote:

 On 2/6/2015 11:11 AM, H. S. Teoh via Digitalmars-d wrote:

 This is precisely why I have lost all interest in  safe. It's clear that
 the present problematic situation will continue to hold, and the
 decision makers are not interested to address it. I am not going to
 waste any more time and energy on this topic.


 In this thread at 8:20PM last night, Dicebot asked me:

 "I am not even sure how you can show the example though, to be honest -
 implied
 issues are about maintaining code and not just writing it."

 And I replied with a specific example of how to fix one aspect of
 std.array. There have been no replies. What else can I do to address it?


In the case you bring up, maintenance is easy -- the code is incorrect, 
it needs to be fixed/rewritten. No solution ever implemented or proposed 
can stop this from happening.

The case being discussed by Dicebot, and most of us, involves a case 
where an entire  trusted function is PROPERLY implemented, yet someone 
adds incorrect code to it. The compiler does not complain.

Note that if the requested solution was implemented, each of these calls 
should be individual cases to inspect. I don't think anyone disagrees 
that uninitializedArray shouldn't be a public  trusted function. But 
individual cases of it CAN be properly safe.

-Steve
Feb 09 2015
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 6 February 2015 at 18:52:45 UTC, Andrei Alexandrescu 
wrote:

 I think the problem is overstated. -- Andrei


I think there could hardly be a more persuasive argument that 
this belief is wrong than Walter himself just having made the 
mistake several times, and not even immediately realizing what is 
wrong: https://github.com/D-Programming-Language/phobos/pull/2966 
[1]

Sorry for singling out this one example here. While it is 
particularly demonstrative, I am certainly not intending to put 
Walter in a bad light. It's simply hard to get that stuff right, 
as templates can make it hard to accurately determine the 
complete public interface of a function. As pointed out in the 
PR, here are some more examples for this class of bugs from 
Phobos code, also written and reviewed by top D coders:

https://issues.dlang.org/show_bug.cgi?id=14135
https://issues.dlang.org/show_bug.cgi?id=14136
https://issues.dlang.org/show_bug.cgi?id=14137
https://issues.dlang.org/show_bug.cgi?id=14138

Neither of those issues would have been prevented by your new 
guidelines; the code in question is already written in that way. 
Quite to the contrary, consequent application of minimal  trusted 
blocks or even the workaround you reject so strongly would have 
prevented all of the bugs except for 14138.

David


[1] The diff in question, for when the PR is fixed: 
https://github.com/klickverbot/phobos/commit/db647f62cb5279ae42ad98665cd60cdcdb9b3dd5
Feb 07 2015
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 7 February 2015 at 12:40:26 UTC, David Nadlinger 
wrote:

 Neither of those issues would have been prevented by your new 
 guidelines; the code in question is already written in that 
 way. Quite to the contrary, consequent application of minimal 
  trusted blocks or even the workaround you reject so strongly 
 would have prevented all of the bugs except for 14138.


This is an incredibly poor argument. The fact that there is no 
documentation for why the functions are  trusted and why they 
have to be  trusted is testament to a flawed process.

If you insist on relying on half-assed flawed verification you 
only catch those instances where it should not have been  trusted 
in the first place, and which would have been caught at an early 
stage with a decent process, but you will keep missing out on the 
hard to detect cases.

You will run into much more difficult problems if you don't do 
something about the safety review process.

Fix the weak typing rather than making the language more 
convoluted, the latter compounds the problem in the long run.
Feb 07 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
And I'll add this:

Please do not turn the compiler into a inadequate verification 
tool. The compiler should do what it can do well, but what it 
cannot do it should not do at all and leave to an external 
verification tool.

 trusted basically tells the compiler "this is beyond your 
capability so we left it to someone else".

Third parties should be able to provide incrementally improved 
verification tools, for various purposes, without mandating 
language or compiler changes.

So what you want from the language is simple clean semantics and 
reasonable best practice annotations for verification that can be 
extended based on the needs in a particular environment.
Feb 07 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/7/15 4:40 AM, David Nadlinger wrote:

 On Friday, 6 February 2015 at 18:52:45 UTC, Andrei Alexandrescu wrote:

 I think the problem is overstated. -- Andrei


 I think there could hardly be a more persuasive argument that this
 belief is wrong than Walter himself just having made the mistake several
 times, and not even immediately realizing what is wrong:
 https://github.com/D-Programming-Language/phobos/pull/2966 [1]

 Sorry for singling out this one example here. While it is particularly
 demonstrative, I am certainly not intending to put Walter in a bad
 light. It's simply hard to get that stuff right, as templates can make
 it hard to accurately determine the complete public interface of a
 function. As pointed out in the PR, here are some more examples for this
 class of bugs from Phobos code, also written and reviewed by top D coders:

 https://issues.dlang.org/show_bug.cgi?id=14135
 https://issues.dlang.org/show_bug.cgi?id=14136
 https://issues.dlang.org/show_bug.cgi?id=14137
 https://issues.dlang.org/show_bug.cgi?id=14138

 Neither of those issues would have been prevented by your new
 guidelines; the code in question is already written in that way. Quite
 to the contrary, consequent application of minimal  trusted blocks or
 even the workaround you reject so strongly would have prevented all of
 the bugs except for 14138.

 David


 [1] The diff in question, for when the PR is fixed:
 https://github.com/klickverbot/phobos/commit/db647f62cb5279ae42ad98665cd60cdcdb9b3dd5


Nice, thanks for this work. One good guideline here is to almost always 
let generic code rely on deduction instead of ascribing safety 
attributes to it.

Andrei
Feb 07 2015
parent "Tobias Pankrath" <tobias pankrath.net> writes:
 https://github.com/klickverbot/phobos/commit/db647f62cb5279ae42ad98665cd60cdcdb9b3dd5


 Nice, thanks for this work. One good guideline here is to 
 almost always let generic code rely on deduction instead of 
 ascribing safety attributes to it.

 Andrei


And making this work in functions that already marked  trusted 
and need to be, but should have some parts inferred is the hole 
point of this thread and the  trusted-misuse in phobos.
Feb 07 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 3:57 AM, Martin Krejcirik wrote:

 If I understand it correctly, Walter is against adding trusted blocks (trusted
 {...}) into  safe functions. But what about having safe blocks in  trusted
 functions ?

 int somefunc()  trusted
 {
     int a = systemfunc();
     int b;

      safe {
        b = safefunc(); // calling systemfunc() not allowed;
     }

     return a + b;
 }


The problem is, again:

      trusted code must have a safe interface

and with code blocks, there is no interface specified to them other than 
examining EVERY line of code in it.

The way interfaces are specified in D is to use functions.
Feb 06 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
First, I want to say that I didn't want to cause a huge rift between D 
developers with this, I didn't think this was such a drastic issue, just 
a possible idea. But here we are. I hope we can mend this, and move 
forward. But on to the discussion.

On 2/5/15 6:39 PM, Walter Bright wrote:

 Consider the following code excerpted from std.array.join:

    static U trustedCast(U, V)(V v)  trusted { return cast(U) v; }
    return trustedCast!RetType(result);

 This is because the compiler would complain that the following line
 would not be  safe:

    return cast(RetType)(result);

 The rationale is "I know it's safe, so I'll create an  trusted wrapper
 to eliminate the error." What comes next is "that's cumbersome. How
 about a better syntax:"

    trusted {
       return cast(RetType)(result);
    }


No. This was NEVER THE ARGUMENT.

Now, let me explain why the latter is BETTER.

It's better because I know where it is used. It's used in one place, and 
I can squash it right there saying "No, you can't do this in this one 
place." Instead of reviewing an API in ALL POSSBILE CONTEXTS (which if 
trustedCast is a public API, would be a lot), I have to review one call 
in ONE CONTEXT.

The former is WORSE because it can be used in 100 places. Now I have to 
go through and fix ALL THOSE FUNCTIONS that use it, because its 
interface was exposed to the whole of phobos.

The problem, as we have said many times, is maintenance. Sure, in both 
cases they never should have shown up in the first place. But there they 
are, and we now have to fix them. All of them.

And let's also talk about long term maintenance. Any time a  trusted 
function is amended or tweaked, we have to reconsider all the contexts. 
If you mark a single line as  trusted, then additional lines to the same 
function would not need as much scrutiny, especially if you warn when 
 trusted tainted data is touched.


 The trouble with it is, what if the cast is converting from an integer
 to a pointer? That could lead to memory corruption. The code allows a
 potentially memory corrupting operation to be inserted into code that is
 otherwise  safe.


And so, you reject that code in both cases, except in the case where you 
don't define a callable API, you don't have to worry about any other 
code or contexts. This has to be the worst example to explain your point.


 The only way to deal with it is to then manually review everything about
 'RetType' and 'result' to prove to oneself that it is not converting
 random bit patterns into pointers. In other words, one is manually
 reviewing  safe code for memory corruption errors.


 trusted code is not  safe code. You are reviewing that one line in its 
current context, not the whole function to see where it is called in 
various contexts.


 The solution is to regard  trusted as a means of encapsulating unsafe
 operations, not escaping them. Encapsulating them means that the
 interface from the  trusted code is such that it is usable from safe
 code without having to manually review the safe code for memory safety.
 For example (also from std.array):

    static void trustedMemcopy(T[] dest, T[] src)  trusted
    {
      assert(src.length == dest.length);
      memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
    }

 I don't have to review callers of trustedMemory() because it
 encapsulates an unsafe operation (memcpy) with a safe interface.


Sure. But let's consider it's called in one place:

trustedMemcopy(array[pos..pos+to_insert], tmp);

What if that became:

assert(tmp.length == to_insert); // same as src.length == dest.length
 trusted memcpy(array.ptr + pos, tmp.ptr, tmp.length);

What is missing here is, make sure the type of array and tmp is the 
same. All one has to do is review the function to see that. You could 
put in a static assert if you wish:

static assert(is(typeof(array) == typeof(tmp)));

If there are multiple places that it's called from, sure we can 
encapsulate that in a function:

static void trustedMemcopy(T[] dest, T[] src)
{
    assert(src.length == dest.length);
     trusted memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
}

There is very little difference here. Except if I add code to my version 
of trustedMemcopy that is  system, I have to properly mark that too, and 
that should imply greater scrutiny. I fail to see why making extra 
unintended  system calls inside a "trusted" function shouldn't require 
extra marking. Consider if this is a github review, and the context for 
the new lines is missing (i.e. you don't see that the new line is inside 
a  trusted function because github doesn't give you that line).


 The reason  trusted applies only to functions, and not to blocks of
 code, is that functions are the construct in D that provides an
 interface. Arbitrary blocks of code do not have a structured interface.
 Adding  trusted { code } support will encourage incorrect uses like the
 opening example. The existence of  trusted blocks will require review of
 every line of code in the function that encloses it, and transitively
 every function that calls it!


This is like opposite land! You don't have to review every function that 
calls a  safe function with  trusted escapes any more than you have to 
review every function that calls a  trusted function! That is the point 
of having the attribute on the function call.

However, if  trusted is improperly placed on a function, then I have to 
break every function that calls it if it now becomes  system. While the 
same would be for a  safe function that has incorrectly escaped  trusted 
calls, the temptation to make those small calls into a nice neat public 
API without context is not there.

Having  trusted as a function attribute *encourages* bad encapsulation 
of unsafe calls WITHOUT CONTEXT. This is so easy to do with templates.


 Adding  trusted as a function attribute, on the other hand, only
 requires review of the function's interface to determine if it is
 acceptable to use in safe code. Safety review of its callers is
 unnecessary.


I see zero difference between a  trusted function and a  safe function 
with  trusted escapes. You have to review both with the same scrutiny. 
Except less so for the one with escapes because you only have to 
scrutinize the  trusted calls.

You know, I'm surprised you have rejected H.S. Teoh's proposal because 
it seems like you are stuck on the concept that nobody should have to 
review  safe code for memory problems. Perhaps marking functions 
 trusted and then having escapes with  system is a better way of looking 
at it, because you do have to review that code for memory problems.

The bottom line of my reasoning is that code changes over time, by 
different people. Context is forgotten. It's much better to have the 
compiler verify you know what you are doing when working with  trusted 
than it is to just allow anyone to inject code anywhere.

-Steve
Feb 06 2015
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 13:28:59 UTC, Steven Schveighoffer 
wrote:

 The bottom line of my reasoning is that code changes over time, 
 by different people. Context is forgotten. It's much better to 
 have the compiler verify you know what you are doing when 
 working with  trusted than it is to just allow anyone to inject 
 code anywhere.


Actually, I think this argument goes against what you are arguing 
for. Anything within a  trusted section has a big warning sign 
attached to it that says "cannot modify this without detailed 
review". But the compiler cannot verify that a  safe function 
with local  trusted blocks are actually safe, so it only buys you 
a false sense of security.

It is also much easier to bring a large  trusted block to safety 
than a small one, e.g. to have one big  trusted chunk that does:

1. obtain resource
2. process resource
3. copy resource
4. free resource
5. return copy

The problem is the process around  trusted given that there will 
be no sound verification system in D.

Maybe it should have been called " manually_proven_safe" instead, 
to discourage use...
Feb 06 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 8:42 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Friday, 6 February 2015 at 13:28:59 UTC, Steven Schveighoffer wrote:

 The bottom line of my reasoning is that code changes over time, by
 different people. Context is forgotten. It's much better to have the
 compiler verify you know what you are doing when working with  trusted
 than it is to just allow anyone to inject code anywhere.


 Actually, I think this argument goes against what you are arguing for.
 Anything within a  trusted section has a big warning sign attached to it
 that says "cannot modify this without detailed review". But the compiler
 cannot verify that a  safe function with local  trusted blocks are
 actually safe, so it only buys you a false sense of security.


That is kind of the point behind H.S. Teoh's idea that  trusted code 
should be default  safe with  system escapes instead of today where it's 
default  system. If nothing else, it has discouraged the use of  trusted 
to mark code for extra scrutiny.

I'm coming to agree with him. Having a function marked  safe doesn't 
tell you whether there are  trusted blocks, and any  trusted blocks 
(despite the idea of having the compiler tell you when data was 
generated/touched by  trusted code) can bring into suspect the whole 
function. So marking a function  safe, and having it mean "this function 
has NO TRUSTED OR SYSTEM CODE in it whatsoever, is probably the right 
move, regardless of any other changes.

In fact, I would propose that it should be an error for  safe functions 
to have any non-static functions inside them (i.e. any place where the 
aliasing of  safe data is implicit). This is a departure from my 
previous position, but I think it still fits if we allow  trusted to 
mean what H.S. Teoh's proposal means.


 It is also much easier to bring a large  trusted block to safety than a
 small one, e.g. to have one big  trusted chunk that does:

 1. obtain resource
 2. process resource
 3. copy resource
 4. free resource
 5. return copy

 The problem is the process around  trusted given that there will be no
 sound verification system in D.


What we really want is:

1. A way to say "this function needs extra scrutiny"
2. Mechanical verification as MUCH AS POSSIBLE, and especially for 
changes to said function.

Yes, we can do 2 manually if necessary. But having a compiler that never 
misses on pointing out certain bad things is so much better than not 
having it.


 Maybe it should have been called " manually_proven_safe" instead, to
 discourage use...


 assume_safe would probably be the right moniker since that's what we 
use elsewhere. But it's water under the bridge now...

-Steve
Feb 06 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 15:10:18 UTC, Steven Schveighoffer 
wrote:

 into suspect the whole function. So marking a function  safe, 
 and having it mean "this function has NO TRUSTED OR SYSTEM CODE 
 in it whatsoever, is probably the right move, regardless of any 
 other changes.


But that would break if you want to call a  safe function with a 
 trusted function reference as a parameter? Or did I 
misunderstand what you meant here?

And... what happens if you bring in a new architecture that 
requires  trusted implementation of a library function that is 
 safe on other architectures?


 1. A way to say "this function needs extra scrutiny"
 2. Mechanical verification as MUCH AS POSSIBLE, and especially 
 for changes to said function.

 Yes, we can do 2 manually if necessary. But having a compiler 
 that never misses on pointing out certain bad things is so much 
 better than not having it.


I am not sure if it is worth the trouble. If you are gonna 
conduct a semi formal proof, then you should not have a 
mechanical sleeping pillow that makes you sloppy. ;-)

Also if you do safety reviews they should be separate from the 
functional review and only focus on safety.

Maybe it would be interesting to have an annotation for 
 notprovenyet, so that you could have regular reviews during 
development and then scan the source code for  trusted functions 
that need a safety review before you a release is permitted? That 
way you don't have to do the safety review for every single 
mutation of the  trusted function.


 Maybe it should have been called " manually_proven_safe" 
 instead, to
 discourage use...


  assume_safe would probably be the right moniker since that's 
 what we use elsewhere. But it's water under the bridge now...


Yeah, it was merely the psychological effect that one might 
hestitate to actually type in "I have proven this" without 
thinking twice about it. "I trust this code" is an easy claim... 
;-)
Feb 06 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 10:36 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Friday, 6 February 2015 at 15:10:18 UTC, Steven Schveighoffer wrote:

 into suspect the whole function. So marking a function  safe, and
 having it mean "this function has NO TRUSTED OR SYSTEM CODE in it
 whatsoever, is probably the right move, regardless of any other changes.


 But that would break if you want to call a  safe function with a
  trusted function reference as a parameter? Or did I misunderstand what
 you meant here?


The whole point of marking a function trusted instead of a block is that 
you have to follow the rules of function calling to get into that block, 
and your separate function only has access to variables you give it.

My point was that if you have  trusted escapes inside a function, 
whether it's marked  safe or not, you still have to review the whole 
function. If the compiler disallowed this outright, then you don't have 
that issue.

Separating the trusted code from the safe code via an API barrier has 
merits when it comes to code review.

Now,  trusted static nested functions that stand on their own are fine, 
they are no different than public ones, just not public.

 trusted static nested functions that are ONLY OK when called in certain 
ways, that is where we run into issues. At that point, you have to make 
a choice -- add (somewhat unnecessary) machinery to make sure the 
function is always called in a  safe way, or expand the scope of the 
 trusted portion, possibly even to the whole  safe function.

I see the point now that making sure  safe functions don't have escapes 
has the advantage of not requiring *as much* review as a  system or 
 trusted function. I am leaning so much towards H.S. Teoh's solution of 
making  trusted safe by default, and allowing escapes into  system code. 
That seems like the right abstraction.


 And... what happens if you bring in a new architecture that requires
  trusted implementation of a library function that is  safe on other
 architectures?


Then you create a  trusted wrapper around that API, ensuring when called 
from  safe code it can't corrupt memory.


 1. A way to say "this function needs extra scrutiny"
 2. Mechanical verification as MUCH AS POSSIBLE, and especially for
 changes to said function.

 Yes, we can do 2 manually if necessary. But having a compiler that
 never misses on pointing out certain bad things is so much better than
 not having it.


 I am not sure if it is worth the trouble. If you are gonna conduct a
 semi formal proof, then you should not have a mechanical sleeping pillow
 that makes you sloppy. ;-)


I see what you mean, but there are also really dumb things that people 
miss that a compiler won't. Having a mechanical set of eyes in addition 
to human eyes is still more eyes ;)


 Also if you do safety reviews they should be separate from the
 functional review and only focus on safety.

 Maybe it would be interesting to have an annotation for  notprovenyet,
 so that you could have regular reviews during development and then scan
 the source code for  trusted functions that need a safety review before
 you a release is permitted? That way you don't have to do the safety
 review for every single mutation of the  trusted function.


The way reviews are done isn't anything the language can require. 
Certainly we can provide guidelines, and we can require such review 
processes for phobos and druntime.

-Steve
Feb 06 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 18:51:34 UTC, Steven Schveighoffer 
wrote:

 My point was that if you have  trusted escapes inside a 
 function, whether it's marked  safe or not, you still have to 
 review the whole function. If the compiler disallowed this 
 outright, then you don't have that issue.


Ok, I also prefer that  trusted only apply to whole functions. I 
don't agree with the argument, but let's leave it at that ;-).


 Separating the trusted code from the safe code via an API 
 barrier has merits when it comes to code review.


Yes. And I actually don't think  trusted functions in Phobos 
should contain version() or other forms for conditional 
compilation since that makes it much harder to reason about.


 I see the point now that making sure  safe functions don't have 
 escapes has the advantage of not requiring *as much* review as 
 a  system or  trusted function. I am leaning so much towards 
 H.S. Teoh's solution of making  trusted safe by default, and 
 allowing escapes into  system code. That seems like the right 
 abstraction.


Just to make sure that I got this right:

I don't really understand why you need to escape to  system from 
 trusted. Isn't  trusted the same as  system but with a seal that 
says that it has been manually verified to be memory safe?  
 system simply allows the same internal semantics as  trusted but 
with no such declared guarantee to the caller?

Except that in  system you could potentially switch the stacks 
and do other unsafe operations that are not brought back to 
normal before leaving the system context... In  trusted you are 
required to restore the context to normal before returning.

So in the type system you only have  safe and  system,  trusted 
is just  safe with flexible manual verification rather than the 
limited automated verification DMD is capable of. Thus you only 
need to export  safe vs  system for separate compilation...?

Isn't that how it is supposed to work already?


 I see what you mean, but there are also really dumb things that 
 people miss that a compiler won't. Having a mechanical set of 
 eyes in addition to human eyes is still more eyes ;)


Well, if you can come up with something sound. The way I see it, 
 trusted functions are allowed to create a new context on entry 
as long as it restores the previous context before exiting.

This essentially means that what is  safe before entering 
 trusted no longer can be guaranteed to be safe. The floating 
point unit might work differently and result in memory unsafe 
operations etc etc.

So calling  safe from  trusted means that you are calling  safe 
as if it was  system. And therefore  safe code called from 
 trusted has to be proven correct just like  system code called 
from  trusted.


 The way reviews are done isn't anything the language can 
 require. Certainly we can provide guidelines, and we can 
 require such review processes for phobos and druntime.


Yes, in Phobos you need to impose additional stricter guarantees 
in order to support the formal review process and ensure that the 
formal review cannot be broken without a new review taking place. 
And yes, that is a process requirement, not a language 
requirement. A different process might impose other 
requirements...

I think you will have to define such a process though, because I 
don't think there is a solution for fully automated verification 
for D without going for a much more complex type system and 
mechanics (which I actually think is out of reach unless 
everything is designed around it).

Fortunately with 3 reviewers you can do quite well if the proof 
is available and they work independently without sharing results 
before everyone is done. If each reviewer has a 10% chance of 
failure then you end up with only 0.1% chance of all of them 
failing... So it is possible to get decent results with a formal 
process in place.

Complicated  trusted code (hard to prove safe) should be rejected 
and unnecessary  trusted code should be fixed in the compiler 
optimizer or by adding language features (like SIMD).
Feb 06 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 3:02 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Friday, 6 February 2015 at 18:51:34 UTC, Steven Schveighoffer wrote:

 I see the point now that making sure  safe functions don't have
 escapes has the advantage of not requiring *as much* review as a
  system or  trusted function. I am leaning so much towards H.S. Teoh's
 solution of making  trusted safe by default, and allowing escapes into
  system code. That seems like the right abstraction.


 Just to make sure that I got this right:

 I don't really understand why you need to escape to  system from
  trusted. Isn't  trusted the same as  system but with a seal that says
 that it has been manually verified to be memory safe?  system simply
 allows the same internal semantics as  trusted but with no such declared
 guarantee to the caller?


In the proposal,  trusted code is actually considered the same as  safe, 
but allows  system escapes.

I don't have any time to read your further points, but I will catch up 
with them later, sorry!

-Steve
Feb 06 2015
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 20:13:18 UTC, Steven Schveighoffer 
wrote:

 In the proposal,  trusted code is actually considered the same 
 as  safe, but allows  system escapes.


But that can't work:

 trusted_is_safe {

   auto tmp = get_hardware_config();

    system{
     mess_up_hardware_config();
   }

   // now this unsafe call is called in a  safe context, but is 
unsafe...
   // DMD does not catch this, so " trusted_is_safe" is broken

   call_safe_code_that_now_is_messed_up();

    system{
      restore_hardware_config(tmp);
   }
}
Feb 06 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 4:36 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 On Friday, 6 February 2015 at 20:13:18 UTC, Steven Schveighoffer wrote:

 In the proposal,  trusted code is actually considered the same as
  safe, but allows  system escapes.


 But that can't work:

  trusted_is_safe {

    auto tmp = get_hardware_config();

     system{
      mess_up_hardware_config();
    }

    // now this unsafe call is called in a  safe context, but is unsafe...
    // DMD does not catch this, so " trusted_is_safe" is broken

    call_safe_code_that_now_is_messed_up();

     system{
       restore_hardware_config(tmp);
    }
 }


The idea is that  trusted code still has to be reviewed for memory 
issues, but is mechanically checked for most of the function for obvious 
 safe violations. It limits to a degree the scrutiny one must apply to 
the  trusted function.

Remember, the whole point of a  trusted function is that it's manually 
verified.

-Steve
Feb 07 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 7 February 2015 at 11:32:41 UTC, Steven 
Schveighoffer wrote:

 The idea is that  trusted code still has to be reviewed for 
 memory issues, but is mechanically checked for most of the 
 function for obvious  safe violations. It limits to a degree 
 the scrutiny one must apply to the  trusted function.

 Remember, the whole point of a  trusted function is that it's 
 manually verified.


This is the wrong way to do it and this is a tooling issue, not a 
language issue.


The right way to do it is this:

1. annotate the  trusted region manually where it is needed

2. mechanically verify the whole  trusted region

Of course, then you also need a theorem prover...



You are trying to do this:

1. mechanically verify the whole  trusted region

2. manually verify the whole  trusted region, but be sloppy about 
it here an there

3. Ooops, we were sloppy in the wrong spot...


Not good.
Feb 07 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/7/15 7:11 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:


 You are trying to do this:

 1. mechanically verify the whole  trusted region

 2. manually verify the whole  trusted region, but be sloppy about it
 here an there

 3. Ooops, we were sloppy in the wrong spot...


No.

A  trusted function is manually verified, period.

But we also must tag potential points of leakage with  system. In fact, 
it probably could be a warning/error if you have a  trusted function 
without any  system escapes (it could just be marked  safe).

Think of it this way: the  system tags are the only places where issues 
can creep into the function. But then you have to apply the leaks to the 
whole function. It makes the problem of finding potential safety issues 
more tractable, because the compiler forces us to identify the root causes.

-Steve
Feb 09 2015
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 9 February 2015 at 14:40:36 UTC, Steven Schveighoffer 
wrote:

 On 2/7/15 7:11 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:


 You are trying to do this:

 1. mechanically verify the whole  trusted region

 2. manually verify the whole  trusted region, but be sloppy 
 about it
 here an there

 3. Ooops, we were sloppy in the wrong spot...


 No.

 A  trusted function is manually verified, period.

 But we also must tag potential points of leakage with  system. 
 In fact, it probably could be a warning/error if you have a 
  trusted function without any  system escapes (it could just be 
 marked  safe).


That's a nice migration path, btw. First, warn about  trusted 
functions without  system blocks and don't enforce  safe-ty 
inside them, later disallow them and do enforce  safe-ty in the 
others.
Feb 09 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/9/15 10:13 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 On Monday, 9 February 2015 at 14:40:36 UTC, Steven Schveighoffer wrote:

 On 2/7/15 7:11 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=
 <ola.fosheim.grostad+dlang gmail.com>" wrote:


 You are trying to do this:

 1. mechanically verify the whole  trusted region

 2. manually verify the whole  trusted region, but be sloppy about it
 here an there

 3. Ooops, we were sloppy in the wrong spot...


 No.

 A  trusted function is manually verified, period.

 But we also must tag potential points of leakage with  system. In
 fact, it probably could be a warning/error if you have a  trusted
 function without any  system escapes (it could just be marked  safe).


 That's a nice migration path, btw. First, warn about  trusted functions
 without  system blocks and don't enforce  safe-ty inside them, later
 disallow them and do enforce  safe-ty in the others.


Yes, that solves the problem of breaking code with this... Nice idea.

-Steve
Feb 09 2015
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 9 February 2015 at 14:40:36 UTC, Steven Schveighoffer 
wrote:

 But we also must tag potential points of leakage with  system. 
 In fact, it probably could be a warning/error if you have a 
  trusted function without any  system escapes (it could just be 
 marked  safe).

 Think of it this way: the  system tags are the only places 
 where issues can creep into the function. But then you have to 
 apply the leaks to the whole function. It makes the problem of 
 finding potential safety issues more tractable, because the 
 compiler forces us to identify the root causes.


The compiler should only verify what is needed for the type 
system to work. There is no need to differentiate between 
 trusted and  system for that.

If you require using " system" for annotation, then you will have 
to change the language every time you improve the verifier. 
Because this ad hoc annotation will be unsuitable for a more 
powerful verifier.

And it isn't obvious that treating  trusted as  safe will not 
lead to false positives.
Feb 09 2015
prev sibling parent reply "Meta" <jared771 gmail.com> writes:
On Friday, 6 February 2015 at 20:13:18 UTC, Steven Schveighoffer 
wrote:

 In the proposal,  trusted code is actually considered the same 
 as  safe, but allows  system escapes.


That seems like a good idea and in the spirit of what the goal 
is. However, won't it be a breaking change?
Feb 06 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/6/15 5:19 PM, Meta wrote:

 On Friday, 6 February 2015 at 20:13:18 UTC, Steven Schveighoffer wrote:

 In the proposal,  trusted code is actually considered the same as
  safe, but allows  system escapes.


 That seems like a good idea and in the spirit of what the goal is.
 However, won't it be a breaking change?


Yes. The big question is, is it worth it? I would say yes, since 
 trusted is already incorrectly used in most cases.

-Steve
Feb 07 2015
prev sibling next sibling parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Friday, 6 February 2015 at 13:42:40 UTC, Ola Fosheim Grøstad 
wrote:

 "cannot modify this without detailed review".


This quote from Ola, here?  That basically describes my job 
maintaining big piles of legacy C: the compiler verifies nothing, 
so every change to the anything in the API of "safe" functions or 
anything in their entire call chain must be painstakingly 
reviewed.

A single change generally takes me several days of research, 
verification, and testing.  I fixed about 150 potential memory 
issues (and several really dumb logic errors) with Clang's static 
analysis when I first inherited the code; it took weeks.  (And 
now writing new stuff using this "safe" API is turning up memory 
safety issues anyway!)

So from my perspective, calling this situation "completely 
impractical" reveals a stunning gift for understatement.  Is this 
really the best we can do after however many years?  Because it 
blows.

The current  trusted semantics (and accompanying politics) make 
it exceedingly clear that  safe is meaningless for anything 
beyond trivial, one-off tools that will never receive maintenance.

-Wyatt
Feb 06 2015
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 15:14:14 UTC, Wyatt wrote:

 So from my perspective, calling this situation "completely 
 impractical" reveals a stunning gift for understatement.  Is 
 this really the best we can do after however many years?  
 Because it blows.

 The current  trusted semantics (and accompanying politics) make 
 it exceedingly clear that  safe is meaningless for anything 
 beyond trivial, one-off tools that will never receive 
 maintenance.


I don't get this. If:

1.  safe actually works.

2.  trusted sections are written without dependencies

3.  trusted are formally proven safe

4.  trusted functions rarely change

5.  trusted is 0-2% of the codebase

Then how is this more work than implementing something like a 
linear type system that is both tedious for the programmer and 
has taken Rust 8 years to get into working shape...?

Jonathan recently said he would like to volunteer some, and he 
has mentioned a background with math/proofs. If he is willing to 
do safety review, then so I am, then so will someone else who 
feel like refreshing their training in program verification... 
Use the resources you have. Those resources, we do have, I think. 
Unused.

The resources we obviously don't have is experts on type systems 
and automated proof and verification engines.
Feb 06 2015
next sibling parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Friday, 6 February 2015 at 15:48:45 UTC, Ola Fosheim Grøstad 
wrote:

 2.  trusted sections are written without dependencies


This really won't happen unless statically enforced because 
humans are involved.


 3.  trusted are formally proven safe


...by humans?


 4.  trusted functions rarely change


Is this so?  Data, please.


 5.  trusted is 0-2% of the codebase


In Phobos, you mean?  You've checked?

2% in my world is already thousands of lines of code, and I'm far 
from having the largest of maintenance burdens.  Get it to a 
small fraction of a percent and then maybe we can talk.


 linear type system


Time and place, man.  I'm not even sure why you're bringing this 
up here.


 Jonathan recently said he would like to volunteer some, and he 
 has mentioned a background with math/proofs. If he is willing 
 to do safety review, then so I am, then so will someone else 
 who feel like refreshing their training in program 
 verification... Use the resources you have. Those resources, we 
 do have, I think. Unused.


That's great; thanks for that.  Seriously.  But to my mind, that 
adds a whole new stack of concerns.  How many people do you think 
you'll need to absorb the patch flow to Phobos?  In perpetuity?  
How do you separate the qualified from the overconfident?  How 
many people need to check something independently before you're 
reasonably certain there are no mistakes?  etc.  Any time you 
bind yourself to human process, you've created a bottleneck of 
uncertainty.

And that's just Phobos! You don't scale horizontally and it's 
kind of problematic to approach this with the assumption that 
everyone wanting to write something that even reasonably 
approximates safe code is a mathematician.  Rather, that doesn't 
bear out in practice at all.

Bottom Line: If it can't be even partially automated, it's not 
useful.

-Wyatt
Feb 06 2015
next sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
On Friday, 6 February 2015 at 17:02:44 UTC, Wyatt wrote:

 On Friday, 6 February 2015 at 15:48:45 UTC, Ola Fosheim Grøstad 
 wrote:

 2.  trusted sections are written without dependencies


 This really won't happen unless statically enforced because 
 humans are involved.


 3.  trusted are formally proven safe


 ...by humans?


 4.  trusted functions rarely change


 Is this so?  Data, please.


 5.  trusted is 0-2% of the codebase


 In Phobos, you mean?  You've checked?


At least I expect the amount of  trusted in phobos to be much 
higher than in normal user code.
Feb 06 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 9:07 AM, Tobias Pankrath wrote:

 At least I expect the amount of  trusted in phobos to be much higher
 than in normal user code.


Exactly. There's a bunch of low-level interfaces in Phobos and also some 
code that aims at maximum efficiency.

Andrei
Feb 06 2015
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 17:02:44 UTC, Wyatt wrote:

 3.  trusted are formally proven safe


 ...by humans?


It isn't that hard for typical library code that is required to 
be non-safe.

You don't have to do better than the compiler code in terms of 
probability of slipping things through... :-P


 4.  trusted functions rarely change


 Is this so?  Data, please.


That would be a requirement. That means you should have high 
requirements for design before allowing implementation of 
 trusted.


 5.  trusted is 0-2% of the codebase


 In Phobos, you mean?  You've checked?


That would be a requirement. If you have lots of  trusted, then 
there is something wrong with the abstraction level  trusted is 
used at (or the compiler features).


 linear type system


 Time and place, man.  I'm not even sure why you're bringing 
 this up here.


What is your alternative? You need to point at the alternative. 
The only alternative I see is to drop  safe or keep  trusted or 
change the type system.


 perpetuity?  How do you separate the qualified from the 
 overconfident?  How many people need to check something 
 independently before you're reasonably certain there are no 
 mistakes?


One semi-formal proof written down. 3 qualified (education) 
independent reviews of the proof.

What makes this more difficult for the standard library than for 
the compiler internals?


 etc.  Any time you bind yourself to human process, you've 
 created a bottleneck of uncertainty.


And the alternative is?


 And that's just Phobos! You don't scale horizontally and it's 
 kind of problematic to approach this with the assumption that 
 everyone wanting to write something that even reasonably 
 approximates safe code is a mathematician.  Rather, that 
 doesn't bear out in practice at all.

 Bottom Line: If it can't be even partially automated, it's not 
 useful.


Then drop  safe...
Feb 06 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 7:48 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 Then how is this more work than implementing something like a linear type
system
 that is both tedious for the programmer and has taken Rust 8 years to get into
 working shape...?


Rust has "unsafe" blocks with specific instructions that it cannot be verified 
mechanically and it is up to the programmer to ensure a safe interface to it.

So no, Rust didn't get that working, either, and it is far beyond current 
compiler technology to do it.
Feb 06 2015
parent reply "Meta" <jared771 gmail.com> writes:
On Friday, 6 February 2015 at 22:24:48 UTC, Walter Bright wrote:

 Rust has "unsafe" blocks with specific instructions that it 
 cannot be verified mechanically and it is up to the programmer 
 to ensure a safe interface to it.

 So no, Rust didn't get that working, either, and it is far 
 beyond current compiler technology to do it.


Rust guarantees, though, that all code outside of unsafe 
blocks/functions is completely safe, which D doesn't do because 
of trusted. I think that `unsafe` in Rust is more like  trust in 
D, but I'm not completely sure about that.
Feb 06 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 2:39 PM, Meta wrote:

 On Friday, 6 February 2015 at 22:24:48 UTC, Walter Bright wrote:

 Rust has "unsafe" blocks with specific instructions that it cannot be verified
 mechanically and it is up to the programmer to ensure a safe interface to it.

 So no, Rust didn't get that working, either, and it is far beyond current
 compiler technology to do it.


 Rust guarantees, though, that all code outside of unsafe blocks/functions is
 completely safe, which D doesn't do because of trusted. I think that `unsafe`
in
 Rust is more like  trust in D, but I'm not completely sure about that.


Rust guarantees no such thing, because it is explicitly up to the Rust 
programmer to verify a safe interface to unsafe code blocks.

It is no different from the D  trusted programmer verifying a safe interface to 
 trusted code.
Feb 06 2015
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 6 February 2015 at 15:14:14 UTC, Wyatt wrote:

 On Friday, 6 February 2015 at 13:42:40 UTC, Ola Fosheim Grøstad 
 wrote:

 "cannot modify this without detailed review".


 This quote from Ola, here?  That basically describes my job 
 maintaining big piles of legacy C: the compiler verifies 
 nothing, so every change to the anything in the API of "safe" 
 functions or anything in their entire call chain must be 
 painstakingly reviewed.

 A single change generally takes me several days of research, 
 verification, and testing.  I fixed about 150 potential memory 
 issues (and several really dumb logic errors) with Clang's 
 static analysis when I first inherited the code; it took weeks.
  (And now writing new stuff using this "safe" API is turning up 
 memory safety issues anyway!)

 So from my perspective, calling this situation "completely 
 impractical" reveals a stunning gift for understatement.  Is 
 this really the best we can do after however many years?  
 Because it blows.

 The current  trusted semantics (and accompanying politics) make 
 it exceedingly clear that  safe is meaningless for anything 
 beyond trivial, one-off tools that will never receive 
 maintenance.

 -Wyatt


Exactly the reason why I never liked C and C++ only makes it 
better if everyone on team stays away from Cisms and uses the 
safer alternatives.

Now the C and C++ world finally accepted the use of static 
analyzers, but I had the displeasure to try to find pointer 
related issues without one, back in the .com days. As you say, it 
takes weeks.

Regarding D, maybe  trusted should go away and only keep  system 
(== unsafe in other languages), as it can hardly give any extra 
security guarantee.

However, I don't see how to solve the human review process, as 
that is an halting problem.

--
Paulo
Feb 06 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 7:14 AM, Wyatt wrote:

 The current  trusted semantics (and accompanying politics) make it exceedingly
 clear that  safe is meaningless for anything beyond trivial, one-off tools that
 will never receive maintenance.


You are correct in how  trusted is currently (mis)used in Phobos. We aim to fix 
this.

 trusted must provide a safe interface. No exceptions. Yes, that requires
review 
of  trusted code by someone who thoroughly understands this, but there's no 
other way.
Feb 06 2015
prev sibling parent =?UTF-8?Q?Tobias=20M=C3=BCller?= <troplin bluewin.ch> writes:
"Ola Fosheim Grøstad" <ola.fosheim.grostad+dlang gmail.com> wrote:

 On Friday, 6 February 2015 at 13:28:59 UTC, Steven Schveighoffer wrote:

 The bottom line of my reasoning is that code changes over time, > by
 different people. Context is forgotten. It's much better to > have the
 compiler verify you know what you are doing when > working with  trusted
 than it is to just allow anyone to inject > code anywhere.


 
 Actually, I think this argument goes against what you are arguing for.
 Anything within a  trusted section has a big warning sign attached to it
 that says "cannot modify this without detailed review". But the compiler
 cannot verify that a  safe function with local  trusted blocks are
 actually safe, so it only buys you a false sense of security.


I'd go even further:
The compiler could even make optimizations in  safe code based on the
assumption that all  trusted function calls expose a safe interface. I
suspect this will lead to undefined behavior and very subtle bugs.
Feb 06 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2015 5:28 AM, Steven Schveighoffer wrote:

 It's better because I know where it is used. It's used in one place, and I can
 squash it right there saying "No, you can't do this in this one place." Instead
 of reviewing an API in ALL POSSBILE CONTEXTS (which if trustedCast is a public
 API, would be a lot), I have to review one call in ONE CONTEXT.

 The former is WORSE because it can be used in 100 places. Now I have to go
 through and fix ALL THOSE FUNCTIONS that use it, because its interface was
 exposed to the whole of phobos.


This is the crux of the problem - failing to define a safe interface to the 
trusted code block. Without defining an interface, you're right, you must
review 
all the context(s) that call it. With a safe interface you DO NOT. You only
have 
to review the interface.

A simple rule:

"If you need to do a safety review on the context in which  trusted code is 
called YOU ARE DOING IT WRONG because you've failed to provide a safe interface 
to the  trusted code."


It's like solving a physics problem and winding up with negative energy. If
that 
happens, you made a mistake. It is not a matter of judgement or opinion, it is 
an objective fact.

Going forward, all  trusted code that leaks unsafety into its context will be 
rejected for inclusion in Phobos. The code reviewer only has to review the 
 trusted block to determine this - he does not have to review the context.
Feb 06 2015
prev sibling next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
FWIW, and now that I think I understand how  trusted is supposed 
to be used, the arguments are valid. But I also:

1. Agree with H S Teoh on the maintainability aspect. Depending 
on humans reviewing the code is never going to work out. And even 
if it did, subsequent edits might break everything. It's a lot 
harder to review a 2-line diff to an existing function for 
trustworthiness than the original one when it was first written. 
I've lost count of how many code reviews I've approved at work 
that introduced subtle crashy bugs. And let's face it, reviewing 
a  trusted D function is no different from reviewing C code.

2. Agree with Dicebot that reviewing a 50-line function and 
making sure it isn't naughty is quite hard. Not impossible, but 
hard to guarantee its safety.

Unfortunately, I have no suggestions on how to make it better. 
The " safe block/local function in a  trusted function" idea 
sounds promising, but I don't know how well it'd work out in 
practice.

Atila


On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:

 Consider the following code excerpted from std.array.join:

   static U trustedCast(U, V)(V v)  trusted { return cast(U) v; }
   return trustedCast!RetType(result);

 This is because the compiler would complain that the following 
 line would not be  safe:

   return cast(RetType)(result);

 The rationale is "I know it's safe, so I'll create an  trusted 
 wrapper to eliminate the error." What comes next is "that's 
 cumbersome. How about a better syntax:"

   trusted {
      return cast(RetType)(result);
   }

 ? It's the rationale behind "unsafe" blocks that appear in 
 other languages. It seems like a perfectly reasonable request.

 The trouble with it is, what if the cast is converting from an 
 integer to a pointer? That could lead to memory corruption. The 
 code allows a potentially memory corrupting operation to be 
 inserted into code that is otherwise  safe.

 The only way to deal with it is to then manually review 
 everything about 'RetType' and 'result' to prove to oneself 
 that it is not converting random bit patterns into pointers. In 
 other words, one is manually reviewing  safe code for memory 
 corruption errors.

 This is an abject failure of  safe,  trusted, and  system.

 The solution is to regard  trusted as a means of encapsulating 
 unsafe operations, not escaping them. Encapsulating them means 
 that the interface from the  trusted code is such that it is 
 usable from safe code without having to manually review the 
 safe code for memory safety. For example (also from std.array):

   static void trustedMemcopy(T[] dest, T[] src)  trusted
   {
     assert(src.length == dest.length);
     memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
   }

 I don't have to review callers of trustedMemory() because it 
 encapsulates an unsafe operation (memcpy) with a safe interface.

 The reason  trusted applies only to functions, and not to 
 blocks of code, is that functions are the construct in D that 
 provides an interface. Arbitrary blocks of code do not have a 
 structured interface. Adding  trusted { code } support will 
 encourage incorrect uses like the opening example. The 
 existence of  trusted blocks will require review of every line 
 of code in the function that encloses it, and transitively 
 every function that calls it!

 Adding  trusted as a function attribute, on the other hand, 
 only requires review of the function's interface to determine 
 if it is acceptable to use in safe code. Safety review of its 
 callers is unnecessary.
Feb 06 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 6 February 2015 at 14:00:24 UTC, Atila Neves wrote:

 1. Agree with H S Teoh on the maintainability aspect. Depending 
 on humans reviewing the code is never going to work out.


Of course it can work out. You need a formalized process and a 
group of people with training in math or comp.sci to do the 
review.

Proving memory safety for a single function in a semi-formal 
manner is not like proving full correctness.

I suggest the following:

1.  trusted should only be used on high priority code (not to 
gain minor speed ups)

2.  trusted code should always carry the safety-proof as embedded 
comments

3.  trusted templates should carry the most stringent type 
constraints (and proof required to weaken them)

4. At least 3 reviewers with knowledge of compsci/math MUST 
verify the soundness of the proof.

5. Have a list of volunteers that are called in by email to 
review  trusted code.

If only 2% is  trusted then this should work out fine. The 
alternative is to give up  safe completely or completely change 
the typesystem.
Feb 06 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
6.  trusted code should be self contained so that changes in 
called functions don't break the proof...
Feb 06 2015
prev sibling parent reply "ponce" <contact gam3sfrommars.fr> writes:
On Thursday, 5 February 2015 at 23:39:39 UTC, Walter Bright wrote:

 The solution is to regard  trusted as a means of encapsulating 
 unsafe operations, not escaping them. Encapsulating them means 
 that the interface from the  trusted code is such that it is 
 usable from safe code without having to manually review the 
 safe code for memory safety. For example (also from std.array):

   static void trustedMemcopy(T[] dest, T[] src)  trusted
   {
     assert(src.length == dest.length);
     memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
   }

 I don't have to review callers of trustedMemory() because it 
 encapsulates an unsafe operation (memcpy) with a safe interface.


If I understand correctly, your rule o be a trusted function is:
"Unable to create a memory corrutpion whatever the arguments".

But here:
- dest or src could be the null slice
- the assert would go away in release

So I though this example _could_ corrupt memory?




 The reason  trusted applies only to functions, and not to 
 blocks of code, is that functions are the construct in D that 
 provides an interface. Arbitrary blocks of code do not have a 
 structured interface. Adding  trusted { code } support will 
 encourage incorrect uses like the opening example. The 
 existence of  trusted blocks will require review of every line 
 of code in the function that encloses it, and transitively 
 every function that calls it!

 Adding  trusted as a function attribute, on the other hand, 
 only requires review of the function's interface to determine 
 if it is acceptable to use in safe code. Safety review of its 
 callers is unnecessary.
Feb 07 2015
parent "ponce" <contact gam3sfrommars.fr> writes:
On Saturday, 7 February 2015 at 10:02:23 UTC, ponce wrote:

 If I understand correctly, your rule o be a trusted function is:
 "Unable to create a memory corrutpion whatever the arguments".

 But here:
 - dest or src could be the null slice
 - the assert would go away in release

 So I though this example _could_ corrupt memory?


I see know it was already addressed:
http://forum.dlang.org/post/mb1vmt$etn$1 digitalmars.com
Feb 07 2015