www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - trusted considered harmful

reply "David Nadlinger" <see klickverbot.at> writes:
 trusted in its current form needs to go. Its design is badly 
broken, as it leaks implementation details and encourages writing 
unsafe code.

The Problem
———————————

First, there is no point in having  trusted in the function 
signature. Why? From the perspective of the caller of the 
function in question,  safe and  trusted mean exactly the same 
thing. If you are not convinced about that, just consider that 
you can wrap any  trusted function into a  safe function to make 
it  safe, and vice versa.

So the current situation is similar to having two keywords `pure` 
and `pure2` in the language, which are completely equivalent for 
the consumer of an API. This is in itself a problem, since it is 
a pitfall for writing generic code. To stick with the example, 
it's easy to check only for `pure` in a template constraint when 
you really meant to accept both `pure` and `pure2` – and you 
_always_ want to accept both.

But is this alone enough to warrant a change to the language at 
this point? Probably not. But besides that, the current design 
also leads to problems for the implementation side:

One issue is that the distinction unnecessarily restricts the 
implementation in terms of interface stability. Yes,  safe and 
 trusted are equivalent from the caller's perspective, but they 
are mangled differently. This means that changing a function 
signature from one to the other is a breaking change to the ABI, 
and as the mangled name is available in the program (which is 
e.g. what std.traits.FunctionAttributes), also to the API.

Thus, you can't just change  trusted to  safe or vice versa on 
the implementation side if you make changes code which require 
 trusted, resp. cause it to be no longer needed. Sure, you can 
always move the implementation into a new, properly marked 
function, and make the original function just a wrapper around 
it. But this is kludgy at best, and might be unacceptable in the 
case of  safe ->  trusted for performance optimizations, if the 
inliner doesn't kick in. So, the only reasonable choice if you 
want to provide a stable interface in this case is to mark all 
functions which could ever possibly need to access unsafe code as 
 trusted, forgoing all the benefits of automatic safety checking.

But the much bigger problem is that  trusted doesn't play well 
with template attribute inference and makes it much too easy to 
accidentally mark a function as safe to call if it really isn't. 
Both things are a consequence of the fact that it can be applied 
at the function level only; there is no way to apply it 
selectively to only a part of the function.

As an example how this is problematic, consider that you are 
writing a function which takes some generic input data, and needs 
to do (unsafe) low-level buffer handling internally to 
efficiently do its job. You come up with a first implementation, 
maybe only accepting arrays for the sake of getting it working 
quickly, and add  trusted as your dirty buffer magic isn't 
visible from the outside, but does break attribute inference. 
Later, you decide that there is no reason not to take other range 
types as input. Fortunately, the actual implementation doesn't 
require any changes, so you just modify the template constraint 
as needed, and you are good. Well, no – you've just completely 
broken all safety guarantees for every program which calls your 
function, because empty/front/popFront of the passed range might 
be  system.

Now, you might argue that this is a contrived scenario. Yes, the 
mistake could have easily be avoided,  trusted on a template 
declaration should always raise a red flag. But cases like this 
_do_ occur in real-world code, and are easy to miss: The recently 
added std.uuid originally had a similar bug, which went unnoticed 
until during the vote [1] – at that point, a number of people, 
mostly experienced contributors, had reviewed the code. A safety 
system which is easy to break by accident is somewhat of a futile 
exercise.

Can you correctly implement such a template function with today's 
 trusted? Yes, there are workarounds, but it's not quite easy. 
One way is to explicitly detect the  safe-ty of the code accessed 
via template arguments and switch function prototypes using 
static ifs and string mixins to avoid code duplication. For an 
example of this, see Jonathan's new std.range.RefRange [2]. It 
works, but it isn't pretty. The average programmer will, just as 
done in the revised version of std.uuid [3], likely give up and 
accept the fact that the function isn't callable from safe code. 
Which is a pity, as we should really utilize the unique asset we 
got in SafeD to the fullest, but this only works if everything 
that could be  safe is marked as such. The situation won't get 
better as we continue to advocate the use of ranges, either.

To summarize, there are, at least as far as I can see, no 
advantages in distinguishing between  safe and  trusted in 
function signatures, and the function-level granularity of 
 trusted yields to avoidable bugs in real-world code. 
Fortunately, we should be able to resolve both of these issues 
fairly easily, as described below.


A Solution
——————————

Let me make something clear first: I am _not_ intending to remove 
 trusted from the language. As a bridge between the  safe and 
 system worlds, it is an integral part of SafeD. What I'm 
proposing is:

  1) Remove the distinction between  safe and  trusted at the 
interface (ABI, API) level. This implies changing the name 
mangling of  trusted to Nf, and consequently removing the 
distinction in DMD altogether (at least in user-facing parts like 
.stringof and error messages). In theory, this is a breaking 
change, but as any code that doesn't treat them the same is buggy 
anyway, it shouldn't be in practice. As for 
std.traits.FunctionAttribute, we could either make trusted an 
alias for safe, or just remove documentation for the former and 
keep it around for some time (there is no way to deprecate an 
enum member).

  2) The first step is necessary, but mainly of cosmetic nature 
(think `pure`, `pure2`). We still need to address for the 
granularity and attribute inference problem. The obvious solution 
is to add a " trusted" declaration/block, which would allow 
unsafe code in a certain region. Putting  trusted in the function 
header would still be allowed for backwards compatibility (but 
discouraged), and would have the same effect as marking the 
function  safe and wrapping its whole body in a  trusted block. 
It could e.g. look something like this (the   prefix definitely 
looks weird, but I didn't want to introduce a new keyword):

---
  void foo(T)(T t) {
    t.doSomething();
     trusted {
      // Do something dirty.
    }
    t.doSomethingElse();
     trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
  }
---

This is similar to other constructs we have today, for example 
debug {}, which allows impure code. It can be debated whether a 
block »argument« should introduce a new scope or not (like 
static if). The latter currently seems much more attractive to 
me, but I suppose it could be confusing for some.

In any case, while there is probably quite a bit of bikeshedding 
to be done for 2), I don't think there is much controversy about 
1). So, let's try to get this done shortly after the 2.060 
release – as discussed above, it is very unlikely that the 
change will break something, but the odds still increase over 
time. Also, there is currently a Phobos pull request [4] which 
will be influenced by the outcome of this discussion.

David



[1] 
http://forum.dlang.org/thread/jrpni1$rt$1 digitalmars.com?page=2#post-pcaaoymspzelodvmnbvc:40forum.dlang.org
[2] 
https://github.com/D-Programming-Language/phobos/blob/c005f334ec34d3b0295d5dbf48212972e17823d0/std/range.d#L7327
[3] 
https://github.com/D-Programming-Language/phobos/blob/c005f334ec34d3b0295d5dbf48212972e17823d0/std/uuid.d#L1193
[4] https://github.com/D-Programming-Language/phobos/pull/675
Jul 27 2012
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 02:08:28 David Nadlinger wrote:

 This is similar to other constructs we have today, for example
 debug {}, which allows impure code. It can be debated whether a
 block =C2=BBargument=C2=AB should introduce a new scope or not (like
 static if). The latter currently seems much more attractive to
 me, but I suppose it could be confusing for some.


I'd definitely vote for it _not_ introducing a new scope. It will be mu=
ch more=20
useful that way.


 In any case, while there is probably quite a bit of bikeshedding
 to be done for 2), I don't think there is much controversy about
 1). So, let's try to get this done shortly after the 2.060
 release =E2=80=93 as discussed above, it is very unlikely that the
 change will break something, but the odds still increase over
 time. Also, there is currently a Phobos pull request [4] which
 will be influenced by the outcome of this discussion.


I'm all for this. Templates and  trusted just don't get along very well=
, which=20
has a huge impact on ranges. The only real saving grace there is that=20=

attribute inferrence solves a lot of the problem if you just never use =
 safe=20
or  trusted on a templated function, but when you have to do  system st=
uff=20
within such a template, it's a real problem (as RefRange shows).

As for the syntax, I think that  trusted{ /+ system code+/ } makes perf=
ect=20
sense.

- Jonathan M Davis
Jul 27 2012
prev sibling next sibling parent reply =?ISO-8859-1?Q?Jos=E9_Armando_Garc=EDa_Sancio?= <jsancio gmail.com> writes:
On Fri, Jul 27, 2012 at 5:08 PM, David Nadlinger <see klickverbot.at> wrote:

  2) The first step is necessary, but mainly of cosmetic nature (think
 `pure`, `pure2`). We still need to address for the granularity and attribute
 inference problem. The obvious solution is to add a " trusted"
 declaration/block, which would allow unsafe code in a certain region.
 Putting  trusted in the function header would still be allowed for backwards
 compatibility (but discouraged), and would have the same effect as marking
 the function  safe and wrapping its whole body in a  trusted block. It could
 e.g. look something like this (the   prefix definitely looks weird, but I
 didn't want to introduce a new keyword):


Agreed. This is very similar to how Rust works. In Rust all the
functions are assumed to be  safe. Unsafe code can only be performed
in clearly marked blocks. Note: I am not suggesting D should implement
Rust's solution as David already pointed out.

-Jose
Jul 27 2012
parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Saturday, 28 July 2012 at 00:53:24 UTC, José Armando García 
Sancio wrote:

 On Fri, Jul 27, 2012 at 5:08 PM, David Nadlinger 
 <see klickverbot.at> wrote:

  2) The first step is necessary, but mainly of cosmetic nature 
 (think
 `pure`, `pure2`). We still need to address for the granularity 
 and attribute
 inference problem. The obvious solution is to add a " trusted"
 declaration/block, which would allow unsafe code in a certain 
 region.
 Putting  trusted in the function header would still be allowed 
 for backwards
 compatibility (but discouraged), and would have the same 
 effect as marking
 the function  safe and wrapping its whole body in a  trusted 
 block. It could
 e.g. look something like this (the   prefix definitely looks 
 weird, but I
 didn't want to introduce a new keyword):


 Agreed. This is very similar to how Rust works. In Rust all the
 functions are assumed to be  safe. Unsafe code can only be 
 performed
 in clearly marked blocks. Note: I am not suggesting D should 
 implement
 Rust's solution as David already pointed out.

 -Jose
Jul 27 2012
prev sibling next sibling parent reply "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Saturday, 28 July 2012 at 00:08:30 UTC, David Nadlinger wrote:


  2) [...] The obvious solution is to add a " trusted" 
 declaration/block, which would allow unsafe code in a certain 
 region. Putting  trusted in the function header would still be 
 allowed for backwards compatibility (but discouraged), and 
 would have the same effect as marking the function  safe and 
 wrapping its whole body in a  trusted block. It could e.g. look 
 something like this (the   prefix definitely looks weird, but I 
 didn't want to introduce a new keyword):

 ---
  void foo(T)(T t) {
    t.doSomething();
     trusted {
      // Do something dirty.
    }
    t.doSomethingElse();
     trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
  }
 ---


I don't see flaw with 1.

However 2 doesn't sound right.

      trusted {
       // Do something dirty.
     }

You aren't supposed to do dirty things in  trusted code. You're 
supposed to  safely wrap a system function to be usable by a safe 
function. The system function is supposed to be short and getting 
its hands dirty. Remember this is about memory safety and not 
lack of bugs safety.

The template issue needs fixed, but maybe it is the inference 
which needs expanded? Maybe a template is only inferred as safe 
or trusted and require explicitly system?

I think I was going to say more, but I'm not versed in the 
problems for this area, which I'm sure there are many, so this is 
probably good enough self butchering.
Jul 27 2012
next sibling parent "David Piepgrass" <qwertie256 gmail.com> writes:
 I don't see flaw with 1.

 However 2 doesn't sound right.

      trusted {
       // Do something dirty.
     }

 You aren't supposed to do dirty things in  trusted code. You're 
 supposed to  safely wrap a system function to be usable by a 
 safe function. The system function is supposed to be short and 
 getting its hands dirty.


True, but since the proposal is that all functions should be 
either  safe or  system, a  trusted block is necessary in a  safe 
function in order to call  system functions. Perhaps you would 
suggest that a  trusted block should be able to _call_  system 
code but not actually do anything unsafe directly? That sounds 
interesting, but it's not how  trusted currently works.
Jul 27 2012
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 04:05:14 Jesse Phillips wrote:

 On Saturday, 28 July 2012 at 00:08:30 UTC, David Nadlinger wrote:

  2) [...] The obvious solution is to add a " trusted"
 
 declaration/block, which would allow unsafe code in a certain
 region. Putting  trusted in the function header would still be
 allowed for backwards compatibility (but discouraged), and
 would have the same effect as marking the function  safe and
 wrapping its whole body in a  trusted block. It could e.g. look
 something like this (the   prefix definitely looks weird, but I
 didn't want to introduce a new keyword):
 
 ---
 
  void foo(T)(T t) {
  
    t.doSomething();
     trusted {
    
      // Do something dirty.
    
    }
    t.doSomethingElse();
     trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
  
  }
 
 ---


 
 I don't see flaw with 1.
 
 However 2 doesn't sound right.
 
       trusted {
        // Do something dirty.
      }
 
 You aren't supposed to do dirty things in  trusted code. You're
 supposed to  safely wrap a system function to be usable by a safe
 function. The system function is supposed to be short and getting
 its hands dirty. Remember this is about memory safety and not
 lack of bugs safety.


The whole point of  trusted is to mark  system code as being  safe. The 
programmer is certifying that the code is  safe, because they know that what 
it's doing is actually  safe in spite of the fact that the compiler can't 
verify that. It's perfectly acceptable to put "dirty code" in an  trusted 
function. The difference between  trusted and  system is that with  trusted, 
the programmer is guaranteeing that it's actually  safe regardless of what 
arguments it's given, whereas  system is not only doing unsafe things, but 
whether they're ultimately  safe or not depends on arguments and/or other 
code, so the programmer _can't_ guarantee that it's  safe.


 The template issue needs fixed, but maybe it is the inference
 which needs expanded? Maybe a template is only inferred as safe
 or trusted and require explicitly system?
 
 I think I was going to say more, but I'm not versed in the
 problems for this area, which I'm sure there are many, so this is
 probably good enough self butchering.


The main problem is when you have code that is  system in a templated function 
and while you can guarantee that that code is actually  safe and therefore 
mark it as  trusted, you can't guarantee that for the rest of the function.

With the current situation, you can't mark that function as  trusted, because 
that would be certifying that _everything_ in the function was  safe, which 
isn't necessarily true. Ideally, you would be able to mark the operations that 
are  system but you know are really  safe with  trusted and let whether the 
function as a whole is  safe or  system be inferred by the compiler. But right 
now, you either have to break up that  trusted code into a separate function 
(which isn't always reasonable) or play games with static if and having two 
versions of the same function where one is  safe and the other is  system. For 
instance, in the new std.range.RefRange, save is defined something like this:

private static void _testSave(R)(R* range)
{
    (*range).save;
}

static if(isSafe!(_testSave!R))
{
     property auto save()  trusted
    {
        mixin(_genSave());
    }
}
else
{
     property auto save()
    {
        mixin(_genSave());
    }
}

private static string _genSave()  safe pure nothrow
{
    return `import std.conv;` ~
           `alias typeof((*_range).save) S;` ~
           `static assert(isForwardRange!S, S.stringof ~ " is not a forward 
range.");` ~
           `auto mem = new void[S.sizeof];` ~
           `emplace!S(mem, cast(S)(*_range).save);` ~
           `return RefRange!S(cast(S*)mem.ptr);`;
}

The problem is that the emplace stuff is  system, and I know that it's really 
 safe and want to mark it with  trusted, but I _don't_ know that _range's save 
function is  safe, so I can't mark RefRange's save as  trusted. I'm stuck 
either doing nonsense like the code above or giving up on making it possible 
for save to be  safe. With David's suggestion, all of that can be reduced to 
this:

 property auto save()
{
    import std.conv;
    alias typeof((*_range).save) S;
    static assert(isForwardRange!S, S.stringof ~ " is not a forward range.");

     trusted
    {
        auto mem = new void[S.sizeof];`
        emplace!S(mem, cast(S)(*_range).save);`
        return RefRange!S(cast(S*)mem.ptr);
    }
}

That is _way_ cleaner.

Now, the actual code of in RefRange is actually even _more_ complicated 
because of the need to handle the possibility of making save const, but if 


8407 based on some comments by Andrei, but I have no idea what Walter will 
think of David's proposal. With both, functions like RefRange's save become 
reasonably small. Without them, you either give up on const and/or  safe, make 
your templated code require that the functions it uses be const and/or  safe 
(which can be way too restrictive), or you do what I did with RefRange, which 
works but is downright ugly.

- Jonathan M Davis
Jul 27 2012
next sibling parent reply "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis wrote:

 On Saturday, July 28, 2012 04:05:14 Jesse Phillips wrote:

 You aren't supposed to do dirty things in  trusted code. You're
 supposed to  safely wrap a system function to be usable by a 
 safe
 function. The system function is supposed to be short and 
 getting
 its hands dirty. Remember this is about memory safety and not
 lack of bugs safety.


 It's perfectly acceptable to put "dirty code" in an  trusted
 function. The difference between  trusted and  system is that 
 with  trusted,
 the programmer is guaranteeing that it's actually  safe 
 regardless of what
 arguments it's given, whereas  system is not only doing unsafe 
 things, but
 whether they're ultimately  safe or not depends on arguments 
 and/or other
 code, so the programmer _can't_ guarantee that it's  safe.


I realize you can, David Piepgrass was able to read into an idea 
which I didn't want to specifically state as it isn't as 
concerning. But why not limit  trusted to only do  safe and call 
 trusted. But I could see that as an extra layer of annoying, but 
it would keep the unsafe arithmetic and parameter passing in the 
 system functions. But again, not much thought on it and probably 
just be a layer of annoying.

For your example of the proposal, when does the compiler mark 
save() as  system over  trusted?

 property auto save()
{
    import std.conv;
    alias typeof((*_range).save) S;
    static assert(isForwardRange!S, S.stringof ~ " is not a 
forward range.");

     trusted
    {
        auto mem = new void[S.sizeof];`
        emplace!S(mem, cast(S)(*_range).save);`
        return RefRange!S(cast(S*)mem.ptr);
    }
}

You make a call to _range.save in the  trusted block. So why 
isn't the function marked  trusted?

Having taken some more thought on the guarantee provided by  safe 
even when calling  trusted code, I do see the issue of templates 
always being trusted. As it could then be the logic in of the 
 safe which is doing something unsafe. This could be true with 
 trusted but at least the author had the opportunity to make it 
safe.

But at this point I'm not seeing how this proposal is helping. 
You do want save() to be  system when the underlining save() is 
 system, and  trusted otherwise? So I'm not seeing that here.
Jul 28 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 21:53:25 Jesse Phillips wrote:

 For your example of the proposal, when does the compiler mark
 save() as  system over  trusted?
 
  property auto save()
 {
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a
 forward range.");
 
      trusted
     {
         auto mem = new void[S.sizeof];`
         emplace!S(mem, cast(S)(*_range).save);`
         return RefRange!S(cast(S*)mem.ptr);
     }
 }
 
 You make a call to _range.save in the  trusted block. So why
 isn't the function marked  trusted?


I screwed up my changes there. I made them in a bit of a hurry. And actually, 
in this particular case, using an  trusted block is still a bit entertaining, 
since the save call is in middle of an expression which is doing  system stuff. 
But the principle is the same regardless. If you need to do  system stuff in a 
function where you know that the  system stuff that you're doing is  safe but 
the function is also calling templated stuff which may or may not be  safe, you 
can't mark the whole function as  trusted, or you'll be marking code which is 
potentially truly unsafe as  trusted. With an  trusted block, you can mark 
sections of the code  trusted and let the rest be properly inferred based on 
what the the template was instantiated with.

In this particular case, it could still drastically reduce save, because only 
the line with  save would need to be duplicated. So, it could become something  
more like

 property auto save()
{
    import std.conv;
    alias typeof((*_range).save) S;
    static assert(isForwardRange!S, S.stringof ~ " is not a forward range.");

     trusted {auto mem = new void[S.sizeof];}

    static if(isSafelyCallable!((){(*_range).save;}))
    {
         trusted { emplace!S(mem, cast(S)(*_range).save); }
    }
    else
    {
        emplace!S(mem, cast(S)(*_range).save);
    }

     trusted {return RefRange!S(cast(S*)mem.ptr);}
}

It's still a bit ugly, but it's far better than what we can currently do. As 
it stands, you can't even put the emplace call in a separate function and mark 
it  trusted or  system depending on save, because the only way to mark the 
rest of the function as  trusted is to mark the whole thing as  trusted. You'd 
have to specifically put _everything else_ in its own function _and_ emplace in 
its own function (one  trusted, one  system) and have the outer function call 
both. It's a mess. Being able to mark statements as  trusted rather than just 
entire functons would be _huge_.

- Jonathan M Davis
Jul 28 2012
parent reply "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 20:22:27 UTC, Jonathan M Davis wrote:

  property auto save()
 {
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a 
 forward range.");

      trusted {auto mem = new void[S.sizeof];}

     static if(isSafelyCallable!((){(*_range).save;}))
     {
          trusted { emplace!S(mem, cast(S)(*_range).save); }
     }
     else
     {
         emplace!S(mem, cast(S)(*_range).save);
     }

      trusted {return RefRange!S(cast(S*)mem.ptr);}
 }

 It's still a bit ugly, but it's far better than what we can 
 currently do. As
 it stands, you can't even put the emplace call in a separate 
 function and mark
 it  trusted or  system depending on save, because the only way 
 to mark the
 rest of the function as  trusted is to mark the whole thing as 
  trusted. You'd
 have to specifically put _everything else_ in its own function 
 _and_ emplace in
 its own function (one  trusted, one  system) and have the outer 
 function call
 both. It's a mess. Being able to mark statements as  trusted 
 rather than just
 entire functons would be _huge_.


For the specific case of calling unsafe functions from otherwise 
safe code, which occurs quite frequently due to parts of Phobos 
not being properly annotated yet, I've been experimenting with an 
alternative solution in my code. It allows you to »apply 
trusted« at a function call level, and is easily implemented in 
the library with today's D – using it RefRange.save would look 
like this using it:

---
 property auto save()
{
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a 
forward range.");

     auto mem = new void[S.sizeof];
     TRUSTED!(emplace!S)(mem, cast(S)(*_range).save);
      trusted return RefRange!S(cast(S*)mem.ptr);
}
---

TRUSTED is just a  trusted template function which accepts a 
callable via an alias parameter and invokes it with the given 
parameters, adding  trusted to the function type via a cast (an 
application of std.traits.SetFunctionAttributes, by the way).

It seems to work fine, but there is an obvious catch: Just like 
 trusted, TRUSTED circumvents safety, and all of its uses must be 
carefully examined. But in contrast to the former, it is not part 
of the language, so anyone working on the code base, especially 
reviewers, must be aware of implications. The all-caps name is 
meant to help drawing attention to .

Maybe it would be a good idea to also allow 
` trusted(emplace!S)(mem, cast(S)(*_range).save)`, with semantics 
similar to TRUSTED? Or even applying  trusted to arbitrary 


David
Jul 28 2012
next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 20:52:33 UTC, David Nadlinger wrote:

 The all-caps name is meant to help drawing attention to .


Oh, I should proof-read my posts before submitting, but I guess 
it was clear what I meant – because TRUSTED can only rely on 
convention, I chose a name which I hoped would immediately catch 
one's eye when skimming through the code.

David
Jul 28 2012
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 22:52:32 David Nadlinger wrote:

 Maybe it would be a good idea to also allow
 ` trusted(emplace!S)(mem, cast(S)(*_range).save)`, with semantics
 similar to TRUSTED? Or even applying  trusted to arbitrary



Even the cast is unsafe. Basically, that entire line is  system and needs to 
be  trusted except for (*_range).save. Attribute inferrence needs to be used 
on that portion. So, aside from inventing yet more syntax like

 trusted { emplace!S(mem, cast(S) infer{(*_range).save}); }

I don't know how you'd avoid the code duplication here. But at least with 
 trusted blocks, it can be reduced to a single line rather than having to 
duplicate the whole function.

- Jonathan M Davis
Jul 28 2012
parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 21:03:20 UTC, Jonathan M Davis wrote:

 On Saturday, July 28, 2012 22:52:32 David Nadlinger wrote:

 Maybe it would be a good idea to also allow
 ` trusted(emplace!S)(mem, cast(S)(*_range).save)`, with 
 semantics
 similar to TRUSTED? Or even applying  trusted to arbitrary



 Even the cast is unsafe. Basically, that entire line is  system 
 and needs to
 be  trusted except for (*_range).save.


Aww, snap, missed that. Saving (*_range).save to a temporary 
would introduce an unnecessary copy, right?

In any case, I've found TRUSTED to be convenient when dealing 
with incorrectly marked Phobos/C library functions, but I'm still 
unsure it is worth the added maintenance liabilities incurred by 
it being non-standard. It certainly isn't a replacement for 
 trusted blocks.

Maybe allowing to apply  trusted at both expression and »block« 
level would really be an interesting direction: I've had a look 

integer overflow checking, and they work exactly like that.

I must admit that I never actually used them in the little amount 

well-designed language and similar enough to D that this makes me 
quite confident that implementing  trusted like that could be 
pulled off without feeling overly alien.

This still wouldn't solve your save() problem, though, as it acts 
»the wrong way« round, so I'm not sure if it would be worth the 
added complexity over allowing it just at the statement level…

David


[1] http://msdn.microsoft.com/en-us/library/74b4xzyw(v=vs.80).aspx
Jul 28 2012
prev sibling parent reply "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis wrote:

  property auto save()
 {
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a 
 forward range.");

      trusted
     {
         auto mem = new void[S.sizeof];`
         emplace!S(mem, cast(S)(*_range).save);`
         return RefRange!S(cast(S*)mem.ptr);
     }
 }

 That is _way_ cleaner.


But unfortunately wrong – you call S.save in the  trusted 
block… ;)

David
Jul 28 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 22:08:42 David Nadlinger wrote:

 On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis wrote:

  property auto save()
 {
=20
     import std.conv;
     alias typeof((*_range).save) S;
     static assert(isForwardRange!S, S.stringof ~ " is not a
=20
 forward range.");
=20
      trusted
     {
    =20
         auto mem =3D new void[S.sizeof];`
         emplace!S(mem, cast(S)(*_range).save);`
         return RefRange!S(cast(S*)mem.ptr);
    =20
     }
=20
 }
=20
 That is _way_ cleaner.


=20
 But unfortunately wrong =E2=80=93 you call S.save in the  trusted
 block=E2=80=A6 ;)


Yeah. I screwed that up. I was obviously in too much of a hurry when I =
wrote=20
it. And actually, in this particular case, since the part that can't be=
=20
 trusted is in the middle of an expression doing  system stuff, simply =
using an=20
 trusted block wouldn't do the trick. The basic principle is still good=
=20
though. Being able to mark specific statements or sections of code  tru=
sted=20
would greatly simplify templated functions which are making function ca=
lls=20
which _can't_ necessarily be  trusted.

- Jonathan M Davis
Jul 28 2012
parent reply "David Piepgrass" <qwertie256 gmail.com> writes:
 On Saturday, July 28, 2012 22:08:42 David Nadlinger wrote:

 On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis 
 But unfortunately wrong – you call S.save in the  trusted 
 block… ;)


 Yeah. I screwed that up. I was obviously in too much of a hurry 
 when I wrote
 it. And actually, in this particular case, since the part that 
 can't be
  trusted is in the middle of an expression doing  system stuff, 
 simply using an
  trusted block wouldn't do the trick.


Have you guys thought about the possibility that the language 
could simply not trust any calls that were resolved using a 
template argument?

I'm a bit tired so I may be missing something, but it seems to me 
that (in a  trusted template) if the compiler uses an 
instantiated template parameter (e.g. actual type Foo standing in 
for template parameter T) to choose a function to call, the 
compiler should require that the function be  safe, based on the 
principle that a template cannot vouch for what it can't control. 
IOW, since a template can't predict what function actually gets 
called, the compiler should require whatever function gets called 
to be  safe.

If the programmer actually does want his template function to be 
able to call _unpredictable_  system functions, he should mark 
his template as  system instead of  trusted.
Jul 28 2012
parent Artur Skawina <art.08.09 gmail.com> writes:
On 07/29/12 07:14, David Piepgrass wrote:

 On Saturday, July 28, 2012 22:08:42 David Nadlinger wrote:

 On Saturday, 28 July 2012 at 02:33:54 UTC, Jonathan M Davis But unfortunately
wrong – you call S.save in the  trusted block… ;)


 Yeah. I screwed that up. I was obviously in too much of a hurry when I wrote
 it. And actually, in this particular case, since the part that can't be
  trusted is in the middle of an expression doing  system stuff, simply using an
  trusted block wouldn't do the trick.


 
 Have you guys thought about the possibility that the language could simply not
trust any calls that were resolved using a template argument?
 
 I'm a bit tired so I may be missing something, but it seems to me that (in a
 trusted template) if the compiler uses an instantiated template parameter
(e.g. actual type Foo standing in for template parameter T) to choose a
function to call, the compiler should require that the function be  safe, based
on the principle that a template cannot vouch for what it can't control. IOW,
since a template can't predict what function actually gets called, the compiler
should require whatever function gets called to be  safe.


This wouldn't be enough, and would just paper over the real issue.
While the problem is probably most obvious with templates, it is also
present when no generics are involved.

If you write a  trusted non-templated function like

   class C { int field; /*...*/ }

   auto f(C c)  trusted {
do_something_potentially_unsafe_but_verified_with(c.field); }

it will be fine, as it is all obviously safe, right?

Well, until somebody else later changes 'C' to

   class C {  property field() { return unsafe_buggy_code(); } /*...*/ }

without realizing that 'C.field' is accessed in a  safe context.

The compiler will silently accept it and run the  system code just as
if it was marked as  safe or  trusted. The definition of 'C' and the
function 'f' could be in different modules or even libraries. Both will
still build, without even giving a hint that something may be wrong.

"You can't audit code that isn't available." -- that code may not have
been written yet. With the current ' trust' model even an "innocent"
access to non-local data can result in all  safe checks being bypassed.
Keeping all external accesses out of ' trusted' scopes is not really a
practical solution. Most accesses will be (perceived as) safe, so
programmers will choose to ignore the potential hazards in order to
keep the code readable.

artur
Jul 29 2012
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2012-07-28 00:08:28 +0000, "David Nadlinger" <see klickverbot.at> said:


  trusted in its current form needs to go. Its design is badly broken, 
 as it leaks implementation details and encourages writing unsafe code.


 trusted is a dangerous thing. Since the first time I tried to use it, 
I always found it troublesome. I agree it needs to be a scope. And for 
backward compatibility, a  trusted function should be exactly the same 
as a  safe function where the whole body is wrapped in a  trusted scope.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca/
Jul 27 2012
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2012-07-28 03:24:58 +0000, Michel Fortin <michel.fortin michelf.ca> said:


 On 2012-07-28 00:08:28 +0000, "David Nadlinger" <see klickverbot.at> said:
 

  trusted in its current form needs to go. Its design is badly broken, 
 as it leaks implementation details and encourages writing unsafe code.


 
  trusted is a dangerous thing. Since the first time I tried to use it, 
 I always found it troublesome. I agree it needs to be a scope. And for 
 backward compatibility, a  trusted function should be exactly the same 
 as a  safe function where the whole body is wrapped in a  trusted scope.


And when I say it "needs to be a scope", what I meat is a block. I 
don't think it should be a new scope unless someone finds a good reason 
for it (unsafe struct destructors?)


-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca/
Jul 27 2012
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, July 27, 2012 23:28:17 Michel Fortin wrote:

 On 2012-07-28 03:24:58 +0000, Michel Fortin <michel.fortin michelf.ca> said:

 On 2012-07-28 00:08:28 +0000, "David Nadlinger" <see klickverbot.at> said:

  trusted in its current form needs to go. Its design is badly broken,
 as it leaks implementation details and encourages writing unsafe code.


 
  trusted is a dangerous thing. Since the first time I tried to use it,
 I always found it troublesome. I agree it needs to be a scope. And for
 backward compatibility, a  trusted function should be exactly the same
 as a  safe function where the whole body is wrapped in a  trusted scope.


 
 And when I say it "needs to be a scope", what I meat is a block. I
 don't think it should be a new scope unless someone finds a good reason
 for it (unsafe struct destructors?)


If you need another scope, you can always just use a second set of braces, but 
if an  trusted block introduces another scope, there's no way to make it _not_ 
introduce one (not without adding more syntax to make it not introduce one 
anyway).

- Jonathan M Davis
Jul 27 2012
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 28-Jul-12 04:08, David Nadlinger wrote:

 Let me make something clear first: I am _not_ intending to remove
  trusted from the language. As a bridge between the  safe and  system
 worlds, it is an integral part of SafeD. What I'm proposing is:

   1) Remove the distinction between  safe and  trusted at the interface
 (ABI, API) level. This implies changing the name mangling of  trusted to
 Nf, and consequently removing the distinction in DMD altogether (at
 least in user-facing parts like .stringof and error messages). In
 theory, this is a breaking change, but as any code that doesn't treat
 them the same is buggy anyway, it shouldn't be in practice. As for
 std.traits.FunctionAttribute, we could either make trusted an alias for
 safe, or just remove documentation for the former and keep it around for
 some time (there is no way to deprecate an enum member).


No question here,  trusted should be usable in place of  safe 
_transparently_.


   2) The first step is necessary, but mainly of cosmetic nature (think
 `pure`, `pure2`). We still need to address for the granularity and
 attribute inference problem. The obvious solution is to add a " trusted"
 declaration/block, which would allow unsafe code in a certain region.
 Putting  trusted in the function header would still be allowed for
 backwards compatibility (but discouraged), and would have the same
 effect as marking the function  safe and wrapping its whole body in a
  trusted block. It could e.g. look something like this (the   prefix
 definitely looks weird, but I didn't want to introduce a new keyword):

 ---
   void foo(T)(T t) {
     t.doSomething();
      trusted {
       // Do something dirty.
     }
     t.doSomethingElse();
      trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
   }
 ---

 This is similar to other constructs we have today, for example debug {},
 which allows impure code. It can be debated whether a block »argument«
 should introduce a new scope or not (like static if). The latter
 currently seems much more attractive to me, but I suppose it could be
 confusing for some.


Finally proper granularity for trusted!
I'd say that SafeD is still unusable mostly because  trusted is too 
blunt (your example from std.uuid).
The day writeln works for all safe types (with safe/trusted toString or 
whatever) I'd call SafeD barely usable.

I believe it need not introduce another scope.



-- 
Dmitry Olshansky
Jul 28 2012
prev sibling next sibling parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 07/28/12 02:08, David Nadlinger wrote:

  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.


The problem with  trusted is that it is transitive.

 trusted should allow unsafe operations in the covered scope (right now -
the function), but disallow calling unsafe ( system) code. IOW a  safe
function should only be able to call a  safe or  trusted one, and the same
restriction should apply to  trusted functions. This way you can actually
audit the code - something which isn't typically possible right now, when
you have no control over what the "trusted" code might call.

There are two issues w/ such a change
1. Backward compatibility
2. Being able to override the restrictions (eg for debugging)

Both can be addressed by having a trust-me-harder mode, which can be
implemented as " trusted  safe", ie functions marked with both attributes
behave just as  trusted code does right now.

Yes,  safe and  trusted do not need different name mangling.

The ' attribute {}' syntax is something that is needed, but should probably
wait for /proper/ attribute handling, which should also allow for a saner
'synchronized' etc.

'{}' not introducing a scope is not the most intuitive approach (yes, this
includes the currently existing cases too).

artur
Jul 28 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/28/12 7:05 AM, Artur Skawina wrote:

 On 07/28/12 02:08, David Nadlinger wrote:

  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.


 The problem with  trusted is that it is transitive.

  trusted should allow unsafe operations in the covered scope (right now -
 the function), but disallow calling unsafe ( system) code.


No. Trusted means "hand-checked, good to go". It can do anything.

Andrei
Jul 28 2012
parent reply Artur Skawina <art.08.09 gmail.com> writes:
On 07/28/12 15:47, Andrei Alexandrescu wrote:

 On 7/28/12 7:05 AM, Artur Skawina wrote:

 On 07/28/12 02:08, David Nadlinger wrote:

  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.


 The problem with  trusted is that it is transitive.

  trusted should allow unsafe operations in the covered scope (right now -
 the function), but disallow calling unsafe ( system) code.


 
 No. Trusted means "hand-checked, good to go". It can do anything.


Exactly, but the only way for it to mean anything is if it really /can/
be hand-checked. A "trusted" function that calls arbitrary, potentially
unsafe code cannot be trusted. You can't audit code that isn't available.
So the result is bugs (like the ones mentioned in this thread), where  safe
is bypassed, because the  trusted functions aren't expecting to be used
with "unsafe" ones.  trusted bypasses *all* safety checks, not just those
in the hand-checked code. This is something that you will want sometimes,
but in most cases is neither necessary nor desirable. When dealing with
safety one has to be conservative. The proposals to limit the scope of
 trusted only address the symptoms, not the cause. If the design is fixed,
many of the reasons for introducing the finer-grained  trusted disappear,
and it is the truly unsafe ( trusted that calls  system) code that needs
the extra annotations -- which is a good thing. Papering over design bugs
never is.

artur
Jul 28 2012
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/28/12 10:34 AM, Artur Skawina wrote:

 On 07/28/12 15:47, Andrei Alexandrescu wrote:

 On 7/28/12 7:05 AM, Artur Skawina wrote:

 On 07/28/12 02:08, David Nadlinger wrote:

  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.


 The problem with  trusted is that it is transitive.

  trusted should allow unsafe operations in the covered scope (right now -
 the function), but disallow calling unsafe ( system) code.


 No. Trusted means "hand-checked, good to go". It can do anything.


 Exactly, but the only way for it to mean anything is if it really /can/
 be hand-checked.


It means someone stares at it until the goat dies.


 A "trusted" function that calls arbitrary, potentially
 unsafe code cannot be trusted.


I think you have it all wrong. Trusted means it's verified by a human, 
not by a formal method. The compiler allows it to do anything.


 You can't audit code that isn't available.


Correct. If you make e.g. syscalls into a closed-source OS you trust 
that function to not have bugs. It's a decision made by the human who 
annotates  trusted.


 So the result is bugs (like the ones mentioned in this thread), where  safe
 is bypassed, because the  trusted functions aren't expecting to be used
 with "unsafe" ones.  trusted bypasses *all* safety checks, not just those
 in the hand-checked code. This is something that you will want sometimes,
 but in most cases is neither necessary nor desirable. When dealing with
 safety one has to be conservative. The proposals to limit the scope of
  trusted only address the symptoms, not the cause. If the design is fixed,
 many of the reasons for introducing the finer-grained  trusted disappear,
 and it is the truly unsafe ( trusted that calls  system) code that needs
 the extra annotations -- which is a good thing. Papering over design bugs
 never is.


I don't understand what you suggest here. Is it a sort of a refinement 
of  trusted?


Andrei
Jul 28 2012
parent Artur Skawina <art.08.09 gmail.com> writes:
On 07/28/12 18:16, Andrei Alexandrescu wrote:

 On 7/28/12 10:34 AM, Artur Skawina wrote:

 So the result is bugs (like the ones mentioned in this thread), where  safe
 is bypassed, because the  trusted functions aren't expecting to be used
 with "unsafe" ones.  trusted bypasses *all* safety checks, not just those
 in the hand-checked code. This is something that you will want sometimes,
 but in most cases is neither necessary nor desirable. When dealing with
 safety one has to be conservative. The proposals to limit the scope of
  trusted only address the symptoms, not the cause. If the design is fixed,
 many of the reasons for introducing the finer-grained  trusted disappear,
 and it is the truly unsafe ( trusted that calls  system) code that needs
 the extra annotations -- which is a good thing. Papering over design bugs
 never is.


 
 I don't understand what you suggest here. Is it a sort of a refinement of
 trusted?


Maybe my other response to David makes things more clear. 

Yes, there would be a cost to this change (some lib and user code would need
additional annotations), but I think it'd be worth it, as right now it is much
too easy to 'break' the  safe guarantees, often w/o even realizing it.

artur
Jul 28 2012
prev sibling parent reply "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 11:05:34 UTC, Artur Skawina wrote:

 On 07/28/12 02:08, David Nadlinger wrote:

  trusted in its current form needs to go. Its design is badly 
 broken, as it leaks implementation details and encourages 
 writing unsafe code.


 The problem with  trusted is that it is transitive.

  trusted should allow unsafe operations in the covered scope 
 (right now -
 the function), but disallow calling unsafe ( system) code. IOW 
 a  safe
 function should only be able to call a  safe or  trusted one, 
 and the same
 restriction should apply to  trusted functions. This way you 
 can actually
 audit the code - something which isn't typically possible right 
 now, when
 you have no control over what the "trusted" code might call.


Sorry, while I think I know which problem you are referring to, I 
don't see what your suggestion does to address it. There is no 
such thing as allowing unsafe code »only in the current scope«. 
As soon as you are allowed to do unsafe things, you can e.g. just 
cast a  system function to  safe and call it. Or jump to it using 
inline assembly, etc.

My suggestion might not be perfect, but it addresses the problem 
by allowing you to trust only the specific piece of your code 
which actually need to perform unsafe things. Reviewers can 
concentrate on the smaller sections to make sure that what they 
do is indeed  safe (e.g. that they don't call unsafe external 
code), and if something unsafe is accidentally done outside, the 
compiler will catch it.

I also think implementing your suggestion, i.e. distinguishing 
between »local« and »external« code, would add much value for 
another reason: In my experience, the most frequent use case for 
 trusted is precisely to interface with external code that is 
 system, either for legacy reasons of because it just can't be 
safe in the general case (e.g. grep the Phobos sources for 
 trusted). It seems like the actual low-level pointer/inline asm 
magic is often encapsulated into separate ( system) functions 
anyway.


 The ' attribute {}' syntax is something that is needed, but 
 should probabl
 wait for /proper/ attribute handling, which should also allow 
 for a saner
 'synchronized' etc.


What is »proper attribute handling« supposed to be? How would 
it influence the implementation of  trusted{}? How is 
synchronized related to  trusted?

David
Jul 28 2012
parent Artur Skawina <art.08.09 gmail.com> writes:
On 07/28/12 17:05, David Nadlinger wrote:

 On Saturday, 28 July 2012 at 11:05:34 UTC, Artur Skawina wrote:

 On 07/28/12 02:08, David Nadlinger wrote:

  trusted in its current form needs to go. Its design is badly broken, as it
leaks implementation details and encourages writing unsafe code.


 The problem with  trusted is that it is transitive.

  trusted should allow unsafe operations in the covered scope (right now -
 the function), but disallow calling unsafe ( system) code. IOW a  safe
 function should only be able to call a  safe or  trusted one, and the same
 restriction should apply to  trusted functions. This way you can actually
 audit the code - something which isn't typically possible right now, when
 you have no control over what the "trusted" code might call.


 
 Sorry, while I think I know which problem you are referring to, I don't see
what your suggestion does to address it. There is no such thing as allowing
unsafe code »only in the current scope«. As soon as you are allowed to do
unsafe things, you can e.g. just cast a  system function to  safe and call it.
Or jump to it using inline assembly, etc.


Of course inside a  trusted scope you will always be able to bypass every
restriction.
But that does not mean that all restrictions should be disabled by default, so
that
you are not able to take advantage of them and have to reimplement them by
hand, like
checking if every called function is 'safe'.


 My suggestion might not be perfect, but it addresses the problem by allowing
you to trust only the specific piece of your code which actually need to
perform unsafe things. Reviewers can concentrate on the smaller sections to
make sure that what they do is indeed  safe (e.g. that they don't call unsafe
external code), and if something unsafe is accidentally done outside, the
compiler will catch it.


Well, what you're proposing is already possible (other than the
scope-lessness), your
suggestion would result in a nicer syntax. Which would be an improvement, I
agree. but
it's not really enough.  Say somebody writes a function like this one:

   struct S { size_t m()  safe { return 42; } }

   T* t(T)(T* p, size_t idx) {
      auto s = p+idx;
      auto i = s.m();
      return p+i;
   }

then later realizes it can't be called from  safe code. The easiest solution
will
be to mark the 't' function is trusted, which will work.

   T* t(T)(T* p, size_t idx)  trusted {
      auto s = p+idx;
      auto i = s.m();
      return p+i;
   }

Until 't' (much) later gets called with a different struct

   struct U { size_t m()  { return 666; } }

Which will succeed and  system code will silently run in a  safe context;
w/o even a warning.

What should have happened is that the function, instead of being marked as
 trusted, should have been rewritten as:

   T* t(T)(T* p, size_t idx)  safe {
       trusted gets() { return p+idx; }
      auto s = gets();
      auto i = s.m();
       trusted getn(size_t i) { return p+i; }
      return  getn(i);
   }

But, realistically, that is not what's going to happen most of the time...

Your suggestion is an improvement, in that it makes the above look like:

   T* t(T)(T* p, size_t idx)  safe {
       trusted { auto s = p+idx; }
      auto i = s.m();
       trusted { return p+i; }
   }

which is much better. But I'm afraid marking the whole function as trusted
will still be the 'easier' choice - so the problem won't go away, just become
less frequent.
Accidentally 'breaking'  safeness is a serious problem, at least if  safe is to
be taken seriously.


 I also think implementing your suggestion, i.e. distinguishing between
»local« and »external« code, would add much value for another reason: In my
experience, the most frequent use case for  trusted is precisely to interface
with external code that is  system, either for legacy reasons of because it
just can't be safe in the general case (e.g. grep the Phobos sources for
 trusted). It seems like the actual low-level pointer/inline asm magic is often
encapsulated into separate ( system) functions anyway.
 

 The ' attribute {}' syntax is something that is needed, but should probabl
 wait for /proper/ attribute handling, which should also allow for a saner
 'synchronized' etc.


 
 What is »proper attribute handling« supposed to be? How would it influence
the implementation of  trusted{}? How is synchronized related to  trusted?


There have been some discussions about attributes in the past on these lists. I
have some thoughts on »proper attribute handling«, but that's a different
topic.
What I mean here is that the /syntax/ could also be used for marking
scopes/blocks
with other attributes besides  trusted; synchronized is an example that would
also
fit in the ' attr {...}' scheme and is interesting because that also needs an
optional declaration block. 

artur
Jul 28 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/27/12 8:08 PM, David Nadlinger wrote:

 First, there is no point in having  trusted in the function signature.
 Why? From the perspective of the caller of the function in question,
  safe and  trusted mean exactly the same thing. If you are not convinced
 about that, just consider that you can wrap any  trusted function into a
  safe function to make it  safe, and vice versa.


If  trusted is not part of the signature, we can't enable e.g. analyzers 
that verify an entire program or package to be safe. This is not 
something that's currently used, but I'd hate to look back and say, 
"heck, I hate that we conflated  trusted with  safe!"


 One issue is that the distinction unnecessarily restricts the
 implementation in terms of interface stability. Yes,  safe and  trusted
 are equivalent from the caller's perspective, but they are mangled
 differently. This means that changing a function signature from one to
 the other is a breaking change to the ABI, and as the mangled name is
 available in the program (which is e.g. what
 std.traits.FunctionAttributes), also to the API.


I salute this quest for stability, but I don't find it the argument all 
that compelling. If one makes changes to a library, well, some changes 
will require clients to relink. Some don't. I don't see why we'd make a 
thing of the particular change  safe <->  trusted. Is this often, 
fundamental, big,...?


 Thus, you can't just change  trusted to  safe or vice versa on the
 implementation side if you make changes code which require  trusted,
 resp. cause it to be no longer needed.


Can't parse this sentence following "if you make changes code..."


 But the much bigger problem is that  trusted doesn't play well with
 template attribute inference and makes it much too easy to accidentally
 mark a function as safe to call if it really isn't. Both things are a
 consequence of the fact that it can be applied at the function level
 only; there is no way to apply it selectively to only a part of the
 function.


This could be a more serious problem. Could you please write a brief 
example that shows attribute deduction messing things up? I don't 
understand how marking a template as  trusted is bad.


 The obvious solution is to add a " trusted"
 declaration/block, which would allow unsafe code in a certain region.


This is sensible, but I fail to figure how it adds value over marking 
functions as  trusted. Sure, it's finer-grained, but it's also less 
structured.


Andrei
Jul 28 2012
next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 14:02:44 UTC, Andrei Alexandrescu 
wrote:

 If  trusted is not part of the signature, we can't enable e.g. 
 analyzers that verify an entire program or package to be safe. 
 This is not something that's currently used, but I'd hate to 
 look back and say, "heck, I hate that we conflated  trusted 
 with  safe!"


Could you elaborate on that? A  safe function is _identical_, 
from a client point of view, to a  trusted one. It can always 
call a  trusted function under the hood without the caller 
noticing, there is no way around that.

Thus, to be able to check that a program consists only of  safe 
code [1], you would need its complete source, i.e. including all 
the functions it can possibly invoke, to be able to check if 
 trusted code is called in any place. But with all the source 
available, you can just check the implementation for  trusted 
blocks [2], there is no advantage over having it in the signature.

Destroyed? :P

David


[1] Which is highly unlikely, by the way, as many parts of 
druntime just can't be safe.
[2] Or  trusted attributes in the function header – as 
described in the original post, they won't go away for backwards 
compatibility.
Jul 28 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Saturday, 28 July 2012 at 14:02:44 UTC, Andrei Alexandrescu 
wrote:

 But the much bigger problem is that  trusted doesn't play well 
 with
 template attribute inference and makes it much too easy to 
 accidentally
 mark a function as safe to call if it really isn't. Both 
 things are a
 consequence of the fact that it can be applied at the function 
 level
 only; there is no way to apply it selectively to only a part 
 of the
 function.


 This could be a more serious problem. Could you please write a 
 brief example that shows attribute deduction messing things up? 
 I don't understand how marking a template as  trusted is bad.


See the std.uuid discussion I linked in the original post for a 
real-world example of this bug.

The gist is: You can't ever mark a function which can end up 
execute code coming from a template parameter, for example a 
function accepting a range, as  trusted, because then you would 
vouch for all the passed in code as well, which might be  system. 
[1]

Templates parameters which just supply data are obviously not a 
problem.

David


[1] Unless you explicitly check whether the passed code is  safe, 
that is. If you go down this route, though, you need to duplicate 
the function declaration, which isn't pretty. See 
std.range.RefRange.save for an example of this.
Jul 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 28, 2012 10:02:43 Andrei Alexandrescu wrote:

 On 7/27/12 8:08 PM, David Nadlinger wrote:

 First, there is no point in having  trusted in the function signature.
 Why? From the perspective of the caller of the function in question,
  safe and  trusted mean exactly the same thing. If you are not convinced
 about that, just consider that you can wrap any  trusted function into a
  safe function to make it  safe, and vice versa.


 
 If  trusted is not part of the signature, we can't enable e.g. analyzers
 that verify an entire program or package to be safe. This is not
 something that's currently used, but I'd hate to look back and say,
 "heck, I hate that we conflated  trusted with  safe!"


But from the caller's perspective,  safe and  trusted are identical. And if 
you want a function to be  safe rather than  trusted, all you have to do is 
create an  trusted helper function which has the implementation and have the 
actual function call it, and the the function can be  safe. I don't see how it 
makes any difference at all whether a function is marked  safe or  trusted 
except for the fact that with  trusted, the implementation can _directly_ use 
 system stuff, whereas with  safe, there has to be another layer in between. 
There's really no difference from the outside and no difference in terms of the 
actual level of safety.

As far as I can tell, having  trusted being treated differently in the function 
signature buys us nothing. All of the difference is in whether the compiler 
should error out on  system stuff being use inside the functions. So, if we had 
 trusted blocks, and  trusted on a function meant that it was  safe with the 
entire body being inside of an  trusted block, as far as the caller and 
signature go, the behavior would be identical to now except that the mangling 
would be different.


 But the much bigger problem is that  trusted doesn't play well with
 template attribute inference and makes it much too easy to accidentally
 mark a function as safe to call if it really isn't. Both things are a
 consequence of the fact that it can be applied at the function level
 only; there is no way to apply it selectively to only a part of the
 function.


 
 This could be a more serious problem. Could you please write a brief
 example that shows attribute deduction messing things up? I don't
 understand how marking a template as  trusted is bad.


Marking pretty much _any_ templated function - especially range-based 
functions -  as  trusted is bad. If I have

auto func(R)(R range) if(isForwardRange!R)
{
    while(!r.empty && !condition)
    {
        //do  system stuff
    }

    return range.front;
}

and func does  system stuff which I _know_ is valid based on how it's used 
(emplace, scoped, pointer arithmetic, etc.), then ideally I'd be able to mark 
those operations as  trusted, but what about front, popFront, empty, save, 
etc.? I don't know what they do internally. I don't know whether they're  safe 
or  system. So, I can't mark func as  trusted, or I could be marking  system 
code as  trusted when it really isn't safe.

Take the new std.range.RefRange for example. The body of Its save function is 
supposed to look like this:

import std.conv;
alias typeof((*_range).save) S;
static assert(isForwardRange!S, S.stringof ~ " is not a forward range.");
auto mem = new void[S.sizeof];
emplace!S(mem, cast(S)(*_range).save);
return RefRange!S(cast(S*)mem.ptr);

All of the stuff related to emplace can be marked as  trusted, but save can't 
be marked as  trusted because of that call to _range's save, which may or may 
not be  safe. So, RefRange uses static ifs and mixins to create multiple 
versions of the function which are marked as  system or  trusted based on 
whether _range's save is  system or  safe. It's made even worse by the fact 

remains even if const or inout starts being inferred.

In some cases, you can use helper functions which are marked as  trusted to 
segregate the  system code that you know is safe, but in others, that just 
doesn't work (or is really messy if it does), and you need to use static ifs 
to provide multiple versions of the function. It's exactly the kind of 
situation for which we introduced attribute inferrence in the first place, 
except in this case, you _can't_ use inferrence, because you obviously can't 
infer  trusted.

By being able to mark blocks of code as  trusted instead of whole functions, 
the required code duplication is significantly reduced, and if the functions of 
unknown safety being called are separate enough from the code marked as 
 trusted (i.e. not intertwined like they are with the line using save in 
RefRange's save), then _no_ code duplication is required. But even if the 
potentially unsafe code is intertwined with the  trusted code, at least the 
code duplication can be restricted to just a few lines. So, with const/inout 
inferred and  trusted blocks, RefRange's save should be able to become 
something like this;

 property auto save()
{
    import std.conv;
    alias typeof((*_range).save) S;
    static assert(isForwardRange!S, S.stringof ~ " is not a forward range.");

    auto mem = new void[S.sizeof];

    static if(isSafelyCallable!((){(*_range).save;}))
         trusted { emplace!S(mem, cast(S)(*_range).save); }
    else
        emplace!S(mem, cast(S)(*_range).save);

     trusted { return RefRange!S(cast(S*)mem.ptr); }
}

which is _far_ shorter than what it is now (around 50 lines of code), since 
you only need _one_ function declaration instead of four.

- Jonathan M Davis
Jul 28 2012
prev sibling parent deadalnix <deadalnix gmail.com> writes:
Le 28/07/2012 16:02, Andrei Alexandrescu a écrit :

 This is sensible, but I fail to figure how it adds value over marking
 functions as  trusted. Sure, it's finer-grained, but it's also less
 structured.


Let me explain you the a problem I faced this week which illustrate 
pefectly the problem.

I have a function which is  system. The function caller have to ensure 
the parameters are correct. If it does, then the function is  trusted as 
of current semantic.

Let's call that function foo .

I have a second function, bar, which is a template function. This bar 
function uses reflection on an object and also use foo. The usage of foo 
is checked, so bar is supposed to be  trusted if all functions called by 
reflection are  safe or  trusted.

In pseudo code :

void foo(arguments)  system;

void bar(T)(T t) {
     // Reflect T and perform operations, sometime calling reflected 
methods.

     // call foo with checked arguments.
}

Now, as multiple reflected method can be called, it is really hard to 
check if bar is  trusted or  safe .

At the end, I ended up not marking the code  trusted which make it 
 system, even if it is safe in most cases. Simply because it is too 
difficult to know in which case it is  trusted.

With the proposal, code becomes :

void foo(arguments)  system;

void bar(T)(T t) {
     // Reflect T and perform operations, sometime calling reflected 
methods.

      trusted {
         // call foo with checked arguments.
     }
}

The problem is very real and the solution elegant. Any code analyzer 
will be able to look for  trusted block anyway, as it is certainly a 
concern to do more code review on  trusted blocks.
Jul 29 2012
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
Le 28/07/2012 02:08, David Nadlinger a écrit :

 As an example how this is problematic, consider that you are writing a
 function which takes some generic input data, and needs to do (unsafe)
 low-level buffer handling internally to efficiently do its job. You come
 up with a first implementation, maybe only accepting arrays for the sake
 of getting it working quickly, and add  trusted as your dirty buffer
 magic isn't visible from the outside, but does break attribute
 inference. Later, you decide that there is no reason not to take other
 range types as input. Fortunately, the actual implementation doesn't
 require any changes, so you just modify the template constraint as
 needed, and you are good. Well, no – you've just completely broken all
 safety guarantees for every program which calls your function, because
 empty/front/popFront of the passed range might be  system.

 Now, you might argue that this is a contrived scenario. Yes, the mistake
 could have easily be avoided,  trusted on a template declaration should
 always raise a red flag.


Run into that exact same problem this week. +1
Jul 29 2012
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 07/28/2012 02:08 AM, David Nadlinger wrote:

   1) Remove the distinction between  safe and  trusted at the interface
 (ABI, API) level. This implies changing the name mangling of  trusted to
 Nf, and consequently removing the distinction in DMD altogether (at
 least in user-facing parts like .stringof and error messages). In
 theory, this is a breaking change, but as any code that doesn't treat
 them the same is buggy anyway, it shouldn't be in practice. As for
 std.traits.FunctionAttribute, we could either make trusted an alias for
 safe, or just remove documentation for the former and keep it around for
 some time (there is no way to deprecate an enum member).


I haven't read through the whole thread, so pardon if that argument 
already came up.
I think you're missing an important point here.
SafeD is a subset of the language  trusted is not.
It's very reasonable to implement a compiler that supports only SafeD.
It's probably not feasible to translate  system to javascript for 
example [1].
Having the distinction at the API level makes perfectly sense for this.
It's also another use case for attribute overloading ;-) [2].

[1]: 
http://forum.dlang.org/thread/yfmgvgprfpiquakiyjlk forum.dlang.org#post-yfmgvgprfpiquakiyjlk:40forum.dlang.org

[2]: http://d.puremagic.com/issues/show_bug.cgi?id=9511
Feb 14 2013
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 02/15/2013 03:49 AM, David Nadlinger wrote:

 On Fri, Feb 15, 2013 at 3:31 AM, Martin Nowak <code dawg.eu> wrote:

 Having the distinction at the API level makes perfectly sense for this.


 Sorry, but where is the argument here?

 David


If you switch the perspective  trusted allows you to put a constrain on 
code. Especially you can chose to NOT trust.
The SafeD compiler was not a good example because you have the source 
code at hand and could inspect it for  trusted blocks.
But if you think of Chromes Native Client it becomes extremely valuable 
that one could distinguish a function that requires trusting vs. a 
function that is safe at the ABI level. One could for example intercept 
and disallow function calls to  trusted functions if a certain plugin is 
not trusted by the user.
Feb 14 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 15 February 2013 at 04:09:36 UTC, Martin Nowak wrote:

 On 02/15/2013 03:49 AM, David Nadlinger wrote:

 On Fri, Feb 15, 2013 at 3:31 AM, Martin Nowak <code dawg.eu>


 wrote:

 Having the distinction at the API level makes perfectly




 sense for this.

 Sorry, but where is the argument here?

 David


 If you switch the perspective  trusted allows you to put a 
 constrain on code. Especially you can chose to NOT trust.
 The SafeD compiler was not a good example because you have the 
 source code at hand and could inspect it for  trusted blocks.
 But if you think of Chromes Native Client it becomes extremely 
 valuable that one could distinguish a function that requires 
 trusting vs. a function that is safe at the ABI level. One 
 could for example intercept and disallow function calls to 
  trusted functions if a certain plugin is not trusted by the 
 user.


This is totally irrelevant to the current subject, as  trusted 
code can be called from  safe code.

You are asking for another functionality here.
Feb 14 2013
prev sibling parent reply "Timon Gehr" <timon.gehr hotmail.com> writes:
On Friday, 15 February 2013 at 02:31:08 UTC, Martin Nowak wrote:

 ...
 It's probably not feasible to translate  system to javascript 
 for example [1].
 ...


Yes it's feasible.
Feb 14 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 15 February 2013 at 06:27:55 UTC, Timon Gehr wrote:

 On Friday, 15 February 2013 at 02:31:08 UTC, Martin Nowak wrote:

 ...
 It's probably not feasible to translate  system to javascript 
 for example [1].
 ...


 Yes it's feasible.


http://asmjs.org/spec/latest/
Feb 15 2013
prev sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Saturday, 28 July 2012 at 00:08:30 UTC, David Nadlinger wrote:

  trusted in its current form needs to go. Its design is badly 
 broken, as it leaks implementation details and encourages 
 writing unsafe code.

 The Problem
 ———————————

 [...]

 A Solution
 ——————————

 Let me make something clear first: I am _not_ intending to 
 remove  trusted from the language. As a bridge between the 
  safe and  system worlds, it is an integral part of SafeD. What 
 I'm proposing is:

  1) Remove the distinction between  safe and  trusted at the 
 interface (ABI, API) level. This implies changing the name 
 mangling of  trusted to Nf, and consequently removing the 
 distinction in DMD altogether (at least in user-facing parts 
 like .stringof and error messages). In theory, this is a 
 breaking change, but as any code that doesn't treat them the 
 same is buggy anyway, it shouldn't be in practice. As for 
 std.traits.FunctionAttribute, we could either make trusted an 
 alias for safe, or just remove documentation for the former and 
 keep it around for some time (there is no way to deprecate an 
 enum member).

  2) The first step is necessary, but mainly of cosmetic nature 
 (think `pure`, `pure2`). We still need to address for the 
 granularity and attribute inference problem. The obvious 
 solution is to add a " trusted" declaration/block, which would 
 allow unsafe code in a certain region. Putting  trusted in the 
 function header would still be allowed for backwards 
 compatibility (but discouraged), and would have the same effect 
 as marking the function  safe and wrapping its whole body in a 
  trusted block. It could e.g. look something like this (the   
 prefix definitely looks weird, but I didn't want to introduce a 
 new keyword):

 ---
  void foo(T)(T t) {
    t.doSomething();
     trusted {
      // Do something dirty.
    }
    t.doSomethingElse();
     trusted phobosFunctionWhichHasNotBeenMarkedSafeYet();
  }
 ---

 This is similar to other constructs we have today, for example 
 debug {}, which allows impure code. It can be debated whether a 
 block »argument« should introduce a new scope or not (like 
 static if). The latter currently seems much more attractive to 
 me, but I suppose it could be confusing for some.

 In any case, while there is probably quite a bit of 
 bikeshedding to be done for 2), I don't think there is much 
 controversy about 1). So, let's try to get this done shortly 
 after the 2.060 release – as discussed above, it is very 
 unlikely that the change will break something, but the odds 
 still increase over time. Also, there is currently a Phobos 
 pull request [4] which will be influenced by the outcome of 
 this discussion.



I completely agree with the above.  trusted should be applied to 
as small blocks of code as possible -- preferably to individual 
statements -- to make it easier for the programmer to verify and 
maintain the safety of the code.

Lars
Feb 16 2013