www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Trusted Manifesto

reply Walter Bright <newshound2 digitalmars.com> writes:
Andrei and I have learned a lot from the  trusted discussions. It's clear the 
way we were approaching the problem was inadequate. So we came up with a 
proposal based on the ideas and criticisms of the participants in the 
references. It involves no language changes, but offers usage guidelines that
we 
believe are workable. Tell us what you think!
----------------------------------------------------

Trusted Manifesto
-----------------

Memory safety in D has the usual definition: a memory-safe program never reads
uninitialized memory and never reads or writes memory with a type incompatible
with the type it was written. The aim of D's
 safe,  trusted, and  system attributes is to provide as much
mechanically verified to be safe code as possible.


Functions
---------

Function signatures inform what must be true about a function. If
a function is marked as  safe, it must contain only  safe code. This
safety must be mechanically checkable.

Sometimes, an unsafe operation is needed even in a function
that is overall safe. For example,
here's a function that returns an upper case version of its
input string:

   string toUpper(string s)  safe
   {
      char[] r = new char[s.length];
      foreach (i, c; s)
	r[i] = toUpper(c);
      return cast(string)r; // <== unsafe operation
   }

The compiler rejects this function because it's declared as safe but contains an
unsafe operation.
Review of the whole function shows that the operation, in this instance, is
safe.
One way to deal with that is to do what is called an "escape", meaning
telling the compiler "I know what I'm doing, so allow this operation":

   string toUpper(string s)  safe
   {
      char[] r = new char[s.length];
      foreach (i, c; s)
	r[i] = toUpper(c);

      static string trustedCast(char[] r)  trusted
      {
         return cast(string)r;
      }

      return trustedCast(r);
   }

This will pass typechecking. However, the only reason it is safe is because the 
context
in which trustedCast() is called makes it safe. In other words, a manual review 
of the surrounding
context is necessary, which violates the  safe promise that its contents are 
mechanically
checkable.

I.e. the trustedCast() function is an "escape" that injects unsafety into its 
surrounding
contents.

This leads to:

RULE 1:  trusted code accessible from  safe code must expose a safe interface
to 
unsafe operations.

 trusted must not be used to inject unsafety into surrounding context that is 
marked  safe.
 safe code must be mechanically verifiable to be safe, and subverting that is 
not acceptable.

COROLLARY 1:  trusted functions should be as small as possible to encapsulate 
the unsafe operation
without injecting unsafety into  safe code.

In the case of toUpper(), it is necessary to review the entire function to 
verify that the cast is safe,
so it is properly written:

   string toUpper(string s)  trusted
   {
      char[] r = new char[s.length];
      foreach (i, c; s)
	r[i] = toUpper(c);
      return cast(string)r;
   }

Use of local  trusted functions with safe interfaces is encouraged to minimize 
the amount of safety code review required.


Generic Functions
-----------------

Generic functions are templates that accept compile time parameters in the form 
of types, values or aliases to other functions.
Whether the function is  safe or  system is not checkable until the template 
function is instantiated with explicit
arguments. If the template function is marked as  safe, then it can only be 
instantiated with arguments that expose
safe operations.

If the template function is marked  safe, then RULE 1 applies.

But that reduces the genericity of the function. The compiler is able to deduce 
whether a template function is  safe or  system
when it is instantiated. For maximum utility, we need a way to specify that:

     This template function is  safe if the generic and non-generic operations 
it uses are  safe as well,
     otherwise it is  system.

Consider a function to make an immutable array copy of a range:

   immutable(ElementType!Range)[] toArray(Range)(Range r)
   {
      alias ElementType!Range E;
      alias Unqual!E U;
      U[] a = new U[r.length];
      foreach (i, e; r)
	a[i] = e;
      return cast(immutable)a; // <== unsafe operation
   }

Being a template function without specified attributes, the compiler will infer 
the attributes.
But with the unsafe cast, toArray() will always be inferred to be  system. But 
the rest
of the code is safe. If toArray is marked as  trusted,

   immutable(ElementType!Range)[] toArray(Range)(Range r)  trusted
   {
      alias ElementType!Range E;
      alias Unqual!E U;
      U[] a = new U[r.length];
      foreach (i, e; r)
	a[i] = e;
      return cast(immutable)a; // <== unsafe operation
   }

then if the range primitives (front, empty, popFront) exposed by the argument
to r
happen to be  system, then those are invalidly assumed to be trustable. Every
usage
of toArray() would need to be reviewed for safety, which is impractical.

What is needed is a way to isolate the unsafe operation, and enable the
compiler to
infer the rest. In other words, a local exemption from overall safety deduction 
is needed.

Introducing the 'trusted' template to be put in std.conv:

    trusted auto trusted(alias fun)() { return fun(); }

and used:

   immutable(ElementType!Range)[] toArray(Range)(Range r)
   {
      alias ElementType!Range E;
      alias Unqual!E U;
      U[] a = new U[r.length];
      foreach (i, e; r)
	a[i] = e;

      import std.conv : trusted;
      auto result = trusted!(() => cast(immutable)a);
      return result;
   }

Use of the trusted escape requires the programmer to review the context to 
determine if
it really is safe. The compiler will infer safety from the rest of the
operations.

RULE 2: Usage of escapes are only allowable in functions for which safety is 
inferred,
and never when calling into as-of-yet not defined generic functions.

But how can it be verified that toArray() is safe otherwise?

RULE 3: An  safe unittest must be used to verify safety when escapes are used.

    safe unittest
   {
      ... TODO: test toArray() ...
   }

A unittest may also be constructed that verifies via static assert that if a
type
with  system operations is passed to the function, that the function is inferred
as  system.

The programmer must still verify that the usage of escapes that leak
unsafety into the surrounding context is safe.

RULE 4: Escape unsafety must not inject unsafety beyond the template function
it 
is used in.

Alternatives
------------

1. if(0) block

Provide an  trusted local function that fully encapsulates the unsafe code and 
its context,
providing a safe interface. For the operations on template parameters that may 
or may not be
safe inside the local function, represent them in an if(0) block of code:

   if (0) // safety inference
   {
     Unqual!T tmp = cast(Unqual!T)item;
     emplaceRef!(Unqual!T)(tmp, cast(Unqual!T)item);
   }

    trusted void emplace()
   {
     auto bigData = _data.arr.ptr[0 .. len + 1];

     emplaceRef!(Unqual!T)(bigData[len], cast(Unqual!T)item);

     //We do this at the end, in case of exceptions
     _data.arr = bigData;
   }
   emplace();

Although this works, it requires duplication of code in a rather careful, 
tedious, and essentially
unmaintainable manner. It also simply looks wrong, although it could be made 
more palatable by
enclosing it in a template.

2. isSafe!T template

Such a template would test that all operations on type T are  safe. The
template 
function could
then be marked  trusted. The troubles with this are (a) it is all or nothing 
with T, i.e. if a
template function only used an  safe subset of T, it still would not be
accepted 
and (b) it does
not do proper inference of the safety of a template function.

3.  system escape

 system would be used for escaping unsafe code in an  trusted function, or in
an 
un-attributed function
it would instruct compiler to not use the escaped code when deducing 
trustworthiness. Unsafe code in an  trusted function
not so marked would generate an error. While this works, it would essentially 
break every  trusted function
already in existence. It is a somewhat nicer syntax than the std.conv.trusted 
template, but the backwards compatibility
issue makes it unworkable. It offers a technical advantage over
std.conv.trusted 
in that  system will not be
allowed in  safe functions, while not allowing std.conv.trusted escapes in
 safe 
function would be by convention.

References
----------

https://github.com/D-Programming-Language/phobos/pull/2966

http://forum.dlang.org/post/mb0uvr$2fdb$1 digitalmars.com

Acknowledgements
----------------

Everyone who participated in the references!
Feb 09 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
   string toUpper(string s)  safe
   {
      char[] r = new char[s.length];
      foreach (i, c; s)
 	r[i] = toUpper(c);
      return cast(string)r; // <== unsafe operation
   }
Shouldn't that be `return assumeUnique(r)` instead? What about requiring to put in-code comment that mentions condition verified safety relies on? (here - actual uniqueness of r)
 Introducing the 'trusted' template to be put in std.conv:
 
  trusted auto trusted(alias fun)() { return fun(); }
Is this guaranteed to be inlined in frontend? Shouldn't it better be called `system` to denote operation is not actually trusted? ----------------------- In general, this is surprisingly good manifesto. The way it started I have abandoned all hope for any pragmatical compromise but it does address many of issues mentioned in discussion.
Feb 09 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2015 1:36 AM, Dicebot wrote:
   string toUpper(string s)  safe
   {
      char[] r = new char[s.length];
      foreach (i, c; s)
     r[i] = toUpper(c);
      return cast(string)r; // <== unsafe operation
   }
Shouldn't that be `return assumeUnique(r)` instead?
assumeUnique does a little more than assume the argument is unique - it also casts it, which is not a necessary consequence of holding a unique reference. For the purpose of this article, I'd rather have the unsafe cast be explicit rather than a side effect.
 What about requiring to put in-code comment that mentions condition verified
 safety relies on? (here - actual uniqueness of r)
Good idea.
 Introducing the 'trusted' template to be put in std.conv:

  trusted auto trusted(alias fun)() { return fun(); }
Is this guaranteed to be inlined in frontend?
pragma(inline, true) is not available yet!
 Shouldn't it better be called `system` to denote operation is not actually
trusted?
Andrei had the idea that one could simply grep the code for 'trusted' and thereby flag the code (trusted and trusted) that merits special attention. I agreed it was a good idea.
Feb 09 2015
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 9 February 2015 at 10:16:48 UTC, Walter Bright wrote:
 On 2/9/2015 1:36 AM, Dicebot wrote:
  string toUpper(string s)  safe
  {
     char[] r = new char[s.length];
     foreach (i, c; s)
    r[i] = toUpper(c);
     return cast(string)r; // <== unsafe operation
  }
Shouldn't that be `return assumeUnique(r)` instead?
assumeUnique does a little more than assume the argument is unique - it also casts it, which is not a necessary consequence of holding a unique reference. For the purpose of this article, I'd rather have the unsafe cast be explicit rather than a side effect.
What I have meant is that here cast is only safe under assumption that casted slice is unique (has not other referenced). `assumeUnique` is supposed to exactly communicate that to the reader in most idiomatic manner. It would be still inferred as system here so rules won't be compromised.
 Introducing the 'trusted' template to be put in std.conv:

  trusted auto trusted(alias fun)() { return fun(); }
Is this guaranteed to be inlined in frontend?
pragma(inline, true) is not available yet!
And this is exactly why I am asking ;) Wide usage of this idiom before inlining is guaranteed may result in performance regressions.
 Shouldn't it better be called `system` to denote operation is 
 not actually trusted?
Andrei had the idea that one could simply grep the code for 'trusted' and thereby flag the code (trusted and trusted) that merits special attention. I agreed it was a good idea.
Fine by me.
Feb 09 2015
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
On Monday, 9 February 2015 at 08:19:24 UTC, Walter Bright wrote:
 while not allowing std.conv.trusted escapes in  safe function 
 would be by convention.
Can SafeD be enforced given that?
Feb 09 2015
next sibling parent reply "Kagamin" <spam here.lot> writes:
Also why safe code is allowed to create unsafe lambdas?
Feb 09 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/9/15 2:09 AM, Kagamin wrote:
 Also why safe code is allowed to create unsafe lambdas?
Might be interesting to restrict that. -- Andrei
Feb 09 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/9/15 2:05 AM, Kagamin wrote:
 On Monday, 9 February 2015 at 08:19:24 UTC, Walter Bright wrote:
 while not allowing std.conv.trusted escapes in  safe function would be
 by convention.
Can SafeD be enforced given that?
At some point you have to define a trusted computing base. That includes the compiler and e.g. the safe/ trusted functions in the standard library. With those in tow it's easy to enforce safety in client code by ruling auto trusted constructs automatically. -- Andrei
Feb 09 2015
prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 9 February 2015 at 08:19:24 UTC, Walter Bright wrote:
 Andrei and I have learned a lot from the  trusted discussions. 
 It's clear the way we were approaching the problem was 
 inadequate. So we came up with a proposal based on the ideas 
 and criticisms of the participants in the references. It 
 involves no language changes, but offers usage guidelines that 
 we believe are workable. Tell us what you think!
 ----------------------------------------------------

 Trusted Manifesto
 -----------------

 Memory safety in D has the usual definition: a memory-safe 
 program never reads
 uninitialized memory and never reads or writes memory with a 
 type incompatible
 with the type it was written. The aim of D's
  safe,  trusted, and  system attributes is to provide as much
 mechanically verified to be safe code as possible.


 Functions
 ---------

 Function signatures inform what must be true about a function. 
 If
 a function is marked as  safe, it must contain only  safe code. 
 This
 safety must be mechanically checkable.

 Sometimes, an unsafe operation is needed even in a function
 that is overall safe. For example,
 here's a function that returns an upper case version of its
 input string:

   string toUpper(string s)  safe
   {
      char[] r = new char[s.length];
      foreach (i, c; s)
 	r[i] = toUpper(c);
      return cast(string)r; // <== unsafe operation
   }

 The compiler rejects this function because it's declared as 
 safe but contains an
 unsafe operation.
 Review of the whole function shows that the operation, in this 
 instance, is safe.
 One way to deal with that is to do what is called an "escape", 
 meaning
 telling the compiler "I know what I'm doing, so allow this 
 operation":

   string toUpper(string s)  safe
   {
      char[] r = new char[s.length];
      foreach (i, c; s)
 	r[i] = toUpper(c);

      static string trustedCast(char[] r)  trusted
      {
         return cast(string)r;
      }

      return trustedCast(r);
   }

 This will pass typechecking. However, the only reason it is 
 safe is because the context
 in which trustedCast() is called makes it safe. In other words, 
 a manual review of the surrounding
 context is necessary, which violates the  safe promise that its 
 contents are mechanically
 checkable.

 I.e. the trustedCast() function is an "escape" that injects 
 unsafety into its surrounding
 contents.

 This leads to:

 RULE 1:  trusted code accessible from  safe code must expose a 
 safe interface to unsafe operations.

  trusted must not be used to inject unsafety into surrounding 
 context that is marked  safe.
  safe code must be mechanically verifiable to be safe, and 
 subverting that is not acceptable.

 COROLLARY 1:  trusted functions should be as small as possible 
 to encapsulate the unsafe operation
 without injecting unsafety into  safe code.

 In the case of toUpper(), it is necessary to review the entire 
 function to verify that the cast is safe,
 so it is properly written:

   string toUpper(string s)  trusted
   {
      char[] r = new char[s.length];
      foreach (i, c; s)
 	r[i] = toUpper(c);
      return cast(string)r;
   }

 Use of local  trusted functions with safe interfaces is 
 encouraged to minimize the amount of safety code review 
 required.


 Generic Functions
 -----------------

 Generic functions are templates that accept compile time 
 parameters in the form of types, values or aliases to other 
 functions.
 Whether the function is  safe or  system is not checkable until 
 the template function is instantiated with explicit
 arguments. If the template function is marked as  safe, then it 
 can only be instantiated with arguments that expose
 safe operations.

 If the template function is marked  safe, then RULE 1 applies.

 But that reduces the genericity of the function. The compiler 
 is able to deduce whether a template function is  safe or 
  system
 when it is instantiated. For maximum utility, we need a way to 
 specify that:

     This template function is  safe if the generic and 
 non-generic operations it uses are  safe as well,
     otherwise it is  system.

 Consider a function to make an immutable array copy of a range:

   immutable(ElementType!Range)[] toArray(Range)(Range r)
   {
      alias ElementType!Range E;
      alias Unqual!E U;
      U[] a = new U[r.length];
      foreach (i, e; r)
 	a[i] = e;
      return cast(immutable)a; // <== unsafe operation
   }

 Being a template function without specified attributes, the 
 compiler will infer the attributes.
 But with the unsafe cast, toArray() will always be inferred to 
 be  system. But the rest
 of the code is safe. If toArray is marked as  trusted,

   immutable(ElementType!Range)[] toArray(Range)(Range r) 
  trusted
   {
      alias ElementType!Range E;
      alias Unqual!E U;
      U[] a = new U[r.length];
      foreach (i, e; r)
 	a[i] = e;
      return cast(immutable)a; // <== unsafe operation
   }

 then if the range primitives (front, empty, popFront) exposed 
 by the argument to r
 happen to be  system, then those are invalidly assumed to be 
 trustable. Every usage
 of toArray() would need to be reviewed for safety, which is 
 impractical.

 What is needed is a way to isolate the unsafe operation, and 
 enable the compiler to
 infer the rest. In other words, a local exemption from overall 
 safety deduction is needed.

 Introducing the 'trusted' template to be put in std.conv:

    trusted auto trusted(alias fun)() { return fun(); }

 and used:

   immutable(ElementType!Range)[] toArray(Range)(Range r)
   {
      alias ElementType!Range E;
      alias Unqual!E U;
      U[] a = new U[r.length];
      foreach (i, e; r)
 	a[i] = e;

      import std.conv : trusted;
      auto result = trusted!(() => cast(immutable)a);
      return result;
   }

 Use of the trusted escape requires the programmer to review the 
 context to determine if
 it really is safe. The compiler will infer safety from the rest 
 of the operations.

 RULE 2: Usage of escapes are only allowable in functions for 
 which safety is inferred,
 and never when calling into as-of-yet not defined generic 
 functions.

 But how can it be verified that toArray() is safe otherwise?

 RULE 3: An  safe unittest must be used to verify safety when 
 escapes are used.

    safe unittest
   {
      ... TODO: test toArray() ...
   }

 A unittest may also be constructed that verifies via static 
 assert that if a type
 with  system operations is passed to the function, that the 
 function is inferred
 as  system.

 The programmer must still verify that the usage of escapes that 
 leak
 unsafety into the surrounding context is safe.

 RULE 4: Escape unsafety must not inject unsafety beyond the 
 template function it is used in.

 Alternatives
 ------------

 1. if(0) block

 Provide an  trusted local function that fully encapsulates the 
 unsafe code and its context,
 providing a safe interface. For the operations on template 
 parameters that may or may not be
 safe inside the local function, represent them in an if(0) 
 block of code:

   if (0) // safety inference
   {
     Unqual!T tmp = cast(Unqual!T)item;
     emplaceRef!(Unqual!T)(tmp, cast(Unqual!T)item);
   }

    trusted void emplace()
   {
     auto bigData = _data.arr.ptr[0 .. len + 1];

     emplaceRef!(Unqual!T)(bigData[len], cast(Unqual!T)item);

     //We do this at the end, in case of exceptions
     _data.arr = bigData;
   }
   emplace();

 Although this works, it requires duplication of code in a 
 rather careful, tedious, and essentially
 unmaintainable manner. It also simply looks wrong, although it 
 could be made more palatable by
 enclosing it in a template.

 2. isSafe!T template

 Such a template would test that all operations on type T are 
  safe. The template function could
 then be marked  trusted. The troubles with this are (a) it is 
 all or nothing with T, i.e. if a
 template function only used an  safe subset of T, it still 
 would not be accepted and (b) it does
 not do proper inference of the safety of a template function.

 3.  system escape

  system would be used for escaping unsafe code in an  trusted 
 function, or in an un-attributed function
 it would instruct compiler to not use the escaped code when 
 deducing trustworthiness. Unsafe code in an  trusted function
 not so marked would generate an error. While this works, it 
 would essentially break every  trusted function
 already in existence. It is a somewhat nicer syntax than the 
 std.conv.trusted template, but the backwards compatibility
 issue makes it unworkable. It offers a technical advantage over 
 std.conv.trusted in that  system will not be
 allowed in  safe functions, while not allowing std.conv.trusted 
 escapes in  safe function would be by convention.

 References
 ----------

 https://github.com/D-Programming-Language/phobos/pull/2966

 http://forum.dlang.org/post/mb0uvr$2fdb$1 digitalmars.com

 Acknowledgements
 ----------------

 Everyone who participated in the references!
Looks great. It seems to me that rules 2 and 3 could be helped along by tooling (compiler or external).
Feb 09 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2015 2:54 AM, John Colvin wrote:
 It seems to me that rules 2 and 3 could be helped along by tooling (compiler or
 external).
Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
Feb 09 2015
next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Monday, 9 February 2015 at 11:43:00 UTC, Walter Bright wrote:
 On 2/9/2015 2:54 AM, John Colvin wrote:
 It seems to me that rules 2 and 3 could be helped along by 
 tooling (compiler or
 external).
Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
On the topic of safety, I seem to remember that bounds checking is disabled in trusted code is this true? If so, can we change that? I think it should only be disabled in system code, if at all.
Feb 09 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 9 February 2015 at 11:47:03 UTC, Meta wrote:
 On Monday, 9 February 2015 at 11:43:00 UTC, Walter Bright wrote:
 On 2/9/2015 2:54 AM, John Colvin wrote:
 It seems to me that rules 2 and 3 could be helped along by 
 tooling (compiler or
 external).
Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
On the topic of safety, I seem to remember that bounds checking is disabled in trusted code is this true? If so, can we change that? I think it should only be disabled in system code, if at all.
bounds checking *can be* disabled in trusted and system code, by choice of compiler flags. It can even be disabled in safe as well, with -boundscheck=off. It might be nice to have a -boundscheck=trusted option. pragma(boundscheck, true/false) would also be nice for functions.
Feb 09 2015
parent reply "Meta" <jared771 gmail.com> writes:
On Monday, 9 February 2015 at 12:02:12 UTC, John Colvin wrote:
 On Monday, 9 February 2015 at 11:47:03 UTC, Meta wrote:
 On Monday, 9 February 2015 at 11:43:00 UTC, Walter Bright 
 wrote:
 On 2/9/2015 2:54 AM, John Colvin wrote:
 It seems to me that rules 2 and 3 could be helped along by 
 tooling (compiler or
 external).
Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
On the topic of safety, I seem to remember that bounds checking is disabled in trusted code is this true? If so, can we change that? I think it should only be disabled in system code, if at all.
bounds checking *can be* disabled in trusted and system code, by choice of compiler flags. It can even be disabled in safe as well, with -boundscheck=off. It might be nice to have a -boundscheck=trusted option. pragma(boundscheck, true/false) would also be nice for functions.
Yes, but is it disabled by default in trusted code? That's what would be nice to change.
Feb 09 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2015 4:14 AM, Meta wrote:
 Yes, but is it disabled by default in  trusted code?
It is never disabled by default.
Feb 09 2015
parent reply "Meta" <jared771 gmail.com> writes:
On Monday, 9 February 2015 at 20:24:02 UTC, Walter Bright wrote:
 On 2/9/2015 4:14 AM, Meta wrote:
 Yes, but is it disabled by default in  trusted code?
It is never disabled by default.
Okay, I thought it was similar to system in that respect.
Feb 09 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2015 12:53 PM, Meta wrote:
 On Monday, 9 February 2015 at 20:24:02 UTC, Walter Bright wrote:
 On 2/9/2015 4:14 AM, Meta wrote:
 Yes, but is it disabled by default in  trusted code?
It is never disabled by default.
Okay, I thought it was similar to system in that respect.
system doesn't disable it, either.
Feb 09 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 9 February 2015 at 11:43:00 UTC, Walter Bright wrote:
 On 2/9/2015 2:54 AM, John Colvin wrote:
 It seems to me that rules 2 and 3 could be helped along by 
 tooling (compiler or
 external).
Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
Until you said this, I was feeling bad that language changes were just completely off the table. I'm not a long-term member of the D community, but a lot of the excitement I get from following D is its "relentless pursuit of perfection", to borrow someone else's slogan. I see a surprising willingness from the early adopters to endure breaking changes for the sake of the long run, and I feel like the pursuit of perfection should utilize this willingness as much as possible. (This might be one of the language changes which is "dfixable", granting it significantly lower switching costs.) That said, I really like your approach. It's like you're building the scaffolding first, and willing to build the statue itself (i.e. language changes) later, should the scaffolding prove too shaky. I think the risk here is that a number of users will commit to the scaffolding, only to find their code saturated with it when it is found to be inadequate. To accommodate this risk: 1. Adopt a tone of experimentation, rather than a tone of authority, when imposing these rules. This will help people understand the risks and encourage them to give feedback. 2. Bear in mind an escape hatch. Try to imagine how a codebase which enforces these rules might transition out of them, should it become necessary. As to the rules themselves, they seem like an admirable attempt to do what's possible within the existing language, to which I have no opposition, assuming they are found in unbiased assessment to work.
Feb 09 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
This is headed in the right direction, I think. Comments below.

On 2/9/15 3:19 AM, Walter Bright wrote:
 Andrei and I have learned a lot from the  trusted discussions. It's
 clear the way we were approaching the problem was inadequate. So we came
 up with a proposal based on the ideas and criticisms of the participants
 in the references. It involves no language changes, but offers usage
 guidelines that we believe are workable. Tell us what you think!
 ----------------------------------------------------

 Trusted Manifesto
 -----------------

 Memory safety in D has the usual definition: a memory-safe program never
 reads
 uninitialized memory and never reads or writes memory with a type
 incompatible
 with the type it was written. The aim of D's
  safe,  trusted, and  system attributes is to provide as much
 mechanically verified to be safe code as possible.


 Functions
 ---------

 Function signatures inform what must be true about a function. If
 a function is marked as  safe, it must contain only  safe code. This
 safety must be mechanically checkable.

 Sometimes, an unsafe operation is needed even in a function
 that is overall safe. For example,
 here's a function that returns an upper case version of its
 input string:

    string toUpper(string s)  safe
    {
       char[] r = new char[s.length];
       foreach (i, c; s)
      r[i] = toUpper(c);
       return cast(string)r; // <== unsafe operation
    }

 The compiler rejects this function because it's declared as safe but
 contains an
 unsafe operation.
 Review of the whole function shows that the operation, in this instance,
 is safe.
 One way to deal with that is to do what is called an "escape", meaning
 telling the compiler "I know what I'm doing, so allow this operation":

    string toUpper(string s)  safe
    {
       char[] r = new char[s.length];
       foreach (i, c; s)
      r[i] = toUpper(c);

       static string trustedCast(char[] r)  trusted
       {
          return cast(string)r;
       }

       return trustedCast(r);
    }

 This will pass typechecking. However, the only reason it is safe is
 because the context
 in which trustedCast() is called makes it safe. In other words, a manual
 review of the surrounding
 context is necessary, which violates the  safe promise that its contents
 are mechanically
 checkable.

 I.e. the trustedCast() function is an "escape" that injects unsafety
 into its surrounding
 contents.

 This leads to:

 RULE 1:  trusted code accessible from  safe code must expose a safe
 interface to unsafe operations.

  trusted must not be used to inject unsafety into surrounding context
 that is marked  safe.
  safe code must be mechanically verifiable to be safe, and subverting
 that is not acceptable.

 COROLLARY 1:  trusted functions should be as small as possible to
 encapsulate the unsafe operation
 without injecting unsafety into  safe code.

 In the case of toUpper(), it is necessary to review the entire function
 to verify that the cast is safe,
 so it is properly written:

    string toUpper(string s)  trusted
    {
       char[] r = new char[s.length];
       foreach (i, c; s)
      r[i] = toUpper(c);
       return cast(string)r;
    }
This I don't disagree with. The issue is mostly that the entire function marked as trusted allows for future additions that are unintentionally un- safe. But we can work with this for now and see how it goes.
 Use of local  trusted functions with safe interfaces is encouraged to
 minimize the amount of safety code review required.
An example here would be good. Perhaps the memcpy example.
 Generic Functions
 -----------------

 Generic functions are templates that accept compile time parameters in
 the form of types, values or aliases to other functions.
 Whether the function is  safe or  system is not checkable until the
 template function is instantiated with explicit
 arguments. If the template function is marked as  safe, then it can only
 be instantiated with arguments that expose
 safe operations.

 If the template function is marked  safe, then RULE 1 applies.

 But that reduces the genericity of the function. The compiler is able to
genericness
 deduce whether a template function is  safe or  system
 when it is instantiated. For maximum utility, we need a way to specify
 that:

      This template function is  safe if the generic and non-generic
 operations it uses are  safe as well,
      otherwise it is  system.

 Consider a function to make an immutable array copy of a range:

    immutable(ElementType!Range)[] toArray(Range)(Range r)
    {
       alias ElementType!Range E;
       alias Unqual!E U;
       U[] a = new U[r.length];
       foreach (i, e; r)
      a[i] = e;
       return cast(immutable)a; // <== unsafe operation
    }

 Being a template function without specified attributes, the compiler
 will infer the attributes.
 But with the unsafe cast, toArray() will always be inferred to be
  system. But the rest
 of the code is safe. If toArray is marked as  trusted,

    immutable(ElementType!Range)[] toArray(Range)(Range r)  trusted
    {
       alias ElementType!Range E;
       alias Unqual!E U;
       U[] a = new U[r.length];
       foreach (i, e; r)
      a[i] = e;
       return cast(immutable)a; // <== unsafe operation
    }

 then if the range primitives (front, empty, popFront) exposed by the
 argument to r
 happen to be  system, then those are invalidly assumed to be trustable.
 Every usage
 of toArray() would need to be reviewed for safety, which is impractical.

 What is needed is a way to isolate the unsafe operation, and enable the
 compiler to
 infer the rest. In other words, a local exemption from overall safety
 deduction is needed.

 Introducing the 'trusted' template to be put in std.conv:

     trusted auto trusted(alias fun)() { return fun(); }

 and used:

    immutable(ElementType!Range)[] toArray(Range)(Range r)
    {
       alias ElementType!Range E;
       alias Unqual!E U;
       U[] a = new U[r.length];
       foreach (i, e; r)
      a[i] = e;

       import std.conv : trusted;
       auto result = trusted!(() => cast(immutable)a);
       return result;
    }

 Use of the trusted escape requires the programmer to review the context
 to determine if
 it really is safe. The compiler will infer safety from the rest of the
 operations.
I actually like this. It is one step closer to the mechanically verified trusted function we have been asking for. In fact, it's exactly like that, as it will break existing if you happened to add system calls inside this function. However, there is a large hole in your example: Object[] o = ...; auto o2 = toArray(o); Now, we have o and o2. o2 is all immutable references to objects that are also in o as mutable. The above function is not safe. This is a demonstration that even with mechanically verified safe code with seemingly correct escapes, you cannot assume anything, especially with templates. I don't really like std.conv.trusted. It doesn't add enough value. What is wrong with using a lambda directly? auto result = (() trusted => cast(immutable)a)();
 RULE 2: Usage of escapes are only allowable in functions for which
 safety is inferred,
 and never when calling into as-of-yet not defined generic functions.

 But how can it be verified that toArray() is safe otherwise?

 RULE 3: An  safe unittest must be used to verify safety when escapes are
 used.

     safe unittest
    {
       ... TODO: test toArray() ...
    }
This rule is misworded. The unit test does not verify that it's safe, it only verifies that safe can be applied to the function, even if the function isn't actually safe. I'd say: RULE 3: A safe unittest must be used to verify the function can be called as safe when escapes are used. However, this does NOT verify safety, that still must be manually checked.
 A unittest may also be constructed that verifies via static assert that
 if a type
 with  system operations is passed to the function, that the function is
 inferred
 as  system.

 The programmer must still verify that the usage of escapes that leak
 unsafety into the surrounding context is safe.

 RULE 4: Escape unsafety must not inject unsafety beyond the template
 function it is used in.

 Alternatives
 ------------

 1. if(0) block

 Provide an  trusted local function that fully encapsulates the unsafe
 code and its context,
 providing a safe interface. For the operations on template parameters
 that may or may not be
 safe inside the local function, represent them in an if(0) block of code:

    if (0) // safety inference
    {
      Unqual!T tmp = cast(Unqual!T)item;
      emplaceRef!(Unqual!T)(tmp, cast(Unqual!T)item);
    }

     trusted void emplace()
    {
      auto bigData = _data.arr.ptr[0 .. len + 1];

      emplaceRef!(Unqual!T)(bigData[len], cast(Unqual!T)item);

      //We do this at the end, in case of exceptions
      _data.arr = bigData;
    }
    emplace();

 Although this works, it requires duplication of code in a rather
 careful, tedious, and essentially
 unmaintainable manner. It also simply looks wrong, although it could be
 made more palatable by
 enclosing it in a template.

 2. isSafe!T template

 Such a template would test that all operations on type T are  safe. The
 template function could
 then be marked  trusted. The troubles with this are (a) it is all or
 nothing with T, i.e. if a
 template function only used an  safe subset of T, it still would not be
 accepted and (b) it does
 not do proper inference of the safety of a template function.

 3.  system escape

  system would be used for escaping unsafe code in an  trusted function,
 or in an un-attributed function
 it would instruct compiler to not use the escaped code when deducing
 trustworthiness. Unsafe code in an  trusted function
 not so marked would generate an error. While this works, it would
 essentially break every  trusted function
 already in existence. It is a somewhat nicer syntax than the
 std.conv.trusted template, but the backwards compatibility
 issue makes it unworkable. It offers a technical advantage over
 std.conv.trusted in that  system will not be
 allowed in  safe functions, while not allowing std.conv.trusted escapes
 in  safe function would be by convention.
I understand the idea of rejecting this to avoid breaking code. However, let's consider 3 things: 1. you may break trusted code, but when you do, it points to code that may not have been scrutinized as well as it should have been. 2. It allows making safe functions with trusted escapes trusted functions with system escapes. This is a better marking system than what we have today. 3. Having the user tag properly what is non-safe and what is safe, allows for other mechanical verification later on. For example tagging data as system so normal trusted code cannot touch it without marking as well. -Steve
Feb 09 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/9/15 7:48 AM, Steven Schveighoffer wrote:
 auto result = (()  trusted => cast(immutable)a)();
I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Feb 09 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:
 On 2/9/15 7:48 AM, Steven Schveighoffer wrote:
 auto result = (()  trusted => cast(immutable)a)();
I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/generic -Steve
Feb 09 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/9/15 10:12 AM, Steven Schveighoffer wrote:
 On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:
 On 2/9/15 7:48 AM, Steven Schveighoffer wrote:
 auto result = (()  trusted => cast(immutable)a)();
I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/generic
Interesting, thanks. Also, M-W does not list "genericity". -- Andrei
Feb 09 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/9/15 1:18 PM, Andrei Alexandrescu wrote:
 On 2/9/15 10:12 AM, Steven Schveighoffer wrote:
 On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:
 On 2/9/15 7:48 AM, Steven Schveighoffer wrote:
 auto result = (()  trusted => cast(immutable)a)();
I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/generic
Interesting, thanks. Also, M-W does not list "genericity". -- Andrei
I know. It looked wrong to me (genericity), but I didn't know what would be the correct term, so I looked it up. FWIW, genericness doesn't look right to me either ;) -Steve
Feb 09 2015
parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 9 February 2015 at 18:36:49 UTC, Steven Schveighoffer 
wrote:
 On 2/9/15 1:18 PM, Andrei Alexandrescu wrote:
 On 2/9/15 10:12 AM, Steven Schveighoffer wrote:
 On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:
 On 2/9/15 7:48 AM, Steven Schveighoffer wrote:
 auto result = (()  trusted => cast(immutable)a)();
I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/generic
Interesting, thanks. Also, M-W does not list "genericity". -- Andrei
I know. It looked wrong to me (genericity), but I didn't know what would be the correct term, so I looked it up. FWIW, genericness doesn't look right to me either ;) -Steve
I prefer to program with "generitude"!
Feb 09 2015
prev sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Monday, 9 February 2015 at 18:18:23 UTC, Andrei Alexandrescu 
wrote:
 On 2/9/15 10:12 AM, Steven Schveighoffer wrote:
 Merriam Webster says otherwise ;)

 http://www.merriam-webster.com/dictionary/generic
Interesting, thanks. Also, M-W does not list "genericity". --
Webster needs to get over itself; according to usage, both are words. Though it looks like "genericity" is primarily used in the sciences, which is going to end up doing funny things to the connotation it carries. :) </linguist> -Wyatt
Feb 09 2015
prev sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2015-02-09 at 13:12 -0500, Steven Schveighoffer via Digitalmars-d w=
rote:
 On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:
 On 2/9/15 7:48 AM, Steven Schveighoffer wrote:
 auto result =3D (()  trusted =3D> cast(immutable)a)();
=20 I'm okay with this as with most of Steve's points. But=20 "genericness" is not a word :o). -- Andrei
=20 Merriam Webster says otherwise ;) =20 http://www.merriam-webster.com/dictionary/generic =20
But that dictionary doesn't matter, the OED is the one true keeper of=20 the English language. =46rom what I can see genericity is not officially an English word, even=20 though OUP have published a book with this "word" as title:=20 http://ukcatalogue.oup.com/product/9780199691807.do Genericness is definitely listed, and so is a valid word. On the other hand, terms of art (aka jargon) are allowed, and computer=20 science has determined that genericity is a valid word. As long as it=20 is used in a computer science, software context. :-) --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Feb 10 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/10/15 9:06 AM, Russel Winder via Digitalmars-d wrote:
 On Mon, 2015-02-09 at 13:12 -0500, Steven Schveighoffer via Digitalmars-d
wrote:
 On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:
 On 2/9/15 7:48 AM, Steven Schveighoffer wrote:
 auto result = (()  trusted => cast(immutable)a)();
I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/generic
But that dictionary doesn't matter, the OED is the one true keeper of the English language.
Sure it matters. If it's defined in M-W it should be defined in OED. What DOESN'T matter is the wiki-style dictionaries that have popped up everywhere.
  From what I can see genericity is not officially an English word
Well, there I found genericity too (and notes that is used especially for technical references, which means it's probably more appropriate), so apparently that is OK, I rescind my edit :P http://www.oed.com/view/Entry/272627 -Steve
Feb 10 2015
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Mon, Feb 09, 2015 at 12:19:10AM -0800, Walter Bright via Digitalmars-d wrote:
[...]
 Trusted Manifesto
 -----------------
[...]
 RULE 1:  trusted code accessible from  safe code must expose a safe
 interface to unsafe operations.
 
  trusted must not be used to inject unsafety into surrounding context
 that is marked  safe.
  safe code must be mechanically verifiable to be safe, and subverting
 that is not acceptable.
 
 COROLLARY 1:  trusted functions should be as small as possible to
 encapsulate the unsafe operation without injecting unsafety into  safe
 code.
[...]
 Generic Functions
 -----------------
[...]
 What is needed is a way to isolate the unsafe operation, and enable
 the compiler to infer the rest. In other words, a local exemption from
 overall safety deduction is needed.
 
 Introducing the 'trusted' template to be put in std.conv:
 
    trusted auto trusted(alias fun)() { return fun(); }
This seems to be a step in the right direction, but doesn't this contradict rule 1? Certainly, trusted() doesn't provide a safe API, yet it is marked trusted. What stops the following abuse of trusted via trusted()? int* myFunc(void* p) safe // <-- I'm claiming to be safe { // But actually I'm not! Though I can convince the // compiler that I am... return trusted!(() => cast(int*)p); } char c; auto p = myFunc(&c); // oops *p = 999; // kaboom Are we just relying on convention that trusted() will not be abused in this way? And how would we prevent users from defining similar templates for escaping safety without being readily detectable, e.g., by grepping for 'trusted'? auto sneaky(alias fun)() trusted { return fun(); } Or is this a case of "if you insist on shooting your own foot the compiler can't/won't help you"? [...]
 RULE 2: Usage of escapes are only allowable in functions for which
 safety is inferred, and never when calling into as-of-yet not defined
 generic functions.
I wonder if there's a way, in the current language, to make it illegal to call trusted() from a function explicitly marked safe...? [...]
 RULE 3: An  safe unittest must be used to verify safety when escapes
 are used.
 
    safe unittest
   {
      ... TODO: test toArray() ...
   }
 
 A unittest may also be constructed that verifies via static assert
 that if a type with  system operations is passed to the function, that
 the function is inferred as  system.
 
 The programmer must still verify that the usage of escapes that leak
 unsafety into the surrounding context is safe.
Makes sense, and does reflect current usage in Phobos. Thanks!!
 RULE 4: Escape unsafety must not inject unsafety beyond the template
 function it is used in.
IOW, the following is illegal? int* myTemplate(void* p) trusted // <-- illegal leakage of unsafety { return trusted!(() => cast(int*) p); }
 Alternatives
 ------------
 
 1. if(0) block
 
 Provide an  trusted local function that fully encapsulates the unsafe
 code and its context, providing a safe interface. For the operations
 on template parameters that may or may not be safe inside the local
 function, represent them in an if(0) block of code:
[...]
 Although this works, it requires duplication of code in a rather
 careful, tedious, and essentially unmaintainable manner. It also
 simply looks wrong, although it could be made more palatable by
 enclosing it in a template.
Yeah, this doesn't look viable.
 2. isSafe!T template
 
 Such a template would test that all operations on type T are  safe.
 The template function could then be marked  trusted. The troubles with
 this are (a) it is all or nothing with T, i.e. if a template function
 only used an  safe subset of T, it still would not be accepted and (b)
 it does not do proper inference of the safety of a template function.
What about isSafe!(T, method1, method2, ...)? I.e., test the safety of an explicit list of operations that the template function will be using? Of course, this isn't a fully-functional solution, since it doesn't handle the case where you *want* your template function to accept types with unsafe methods (just that the template function becomes system instead of safe). You'd have to copy-n-paste the function body and mark one safe (with isSafe!(...)) and the other system (with !isSafe!(...)). Unless there's a way of injecting function attributes based on the result of isSafe, like: // Hypothetical syntax auto myTemplate(R)(R range) { safe if (isSafe!(R, empty, front, popFront)) } { ... } But since we're trying not to require additional language support, this isn't really an option either.
 3.  system escape
 
  system would be used for escaping unsafe code in an  trusted
 function, or in an un-attributed function it would instruct compiler
 to not use the escaped code when deducing trustworthiness. Unsafe code
 in an  trusted function not so marked would generate an error. While
 this works, it would essentially break every  trusted function already
 in existence. It is a somewhat nicer syntax than the std.conv.trusted
 template, but the backwards compatibility issue makes it unworkable.
 It offers a technical advantage over std.conv.trusted in that  system
 will not be allowed in  safe functions, while not allowing
 std.conv.trusted escapes in  safe function would be by convention.
[...] If there was a way to force a compile error if someone tried to instantiate trusted() from inside an explicitly-marked safe function, we could have our cake and eat it too. One (very hackish) way might be a pragma or __trait that tests for explicit safe marking: auto trusted(alias fun)() trusted { // callerAttribute is a hypothetical trait. static if (__traits(callerAttribute, safe)) { static assert(0, "Abuse of trusted()"); } else { return fun(); } } void abusive(void* p) safe { int* ip = trusted!(() => cast(int*)p); // compile error: "Abuse of trusted()" } It's probably a bad idea to extend __traits in this way, but the point is that if we can *somehow* determine the safe marking of the caller, then we could prevent blatant abuses of trusted(). I don't know if the current language is able to achieve that, though. Can it? T -- Computers are like a jungle: they have monitor lizards, rams, mice, c-moss, binary trees... and bugs.
Feb 09 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2015 6:21 PM, H. S. Teoh via Digitalmars-d wrote:
 What stops the following abuse of  trusted via
 trusted()?

 	int* myFunc(void* p)  safe // <-- I'm claiming to be  safe
 	{
 		// But actually I'm not! Though I can convince the
 		// compiler that I am...
 		return trusted!(() => cast(int*)p);
 	}

 	char c;
 	auto p = myFunc(&c); // oops
 	*p = 999; // kaboom

 Are we just relying on convention that trusted() will not be abused in
 this way?
That's right. trusted will always rely on convention.
 And how would we prevent users from defining similar templates for
 escaping safety without being readily detectable, e.g., by grepping for
 'trusted'?

 	auto sneaky(alias fun)()  trusted { return fun(); }

 Or is this a case of "if you insist on shooting your own foot the
 compiler can't/won't help you"?
The very idea of trusted is you're trusting the programmer with something that cannot be mechanically checked.
 RULE 4: Escape unsafety must not inject unsafety beyond the template
 function it is used in.
IOW, the following is illegal? int* myTemplate(void* p) trusted // <-- illegal leakage of unsafety { return trusted!(() => cast(int*) p); }
Depends on the usage context, i.e. "beyond the template function it is used in".
 2. isSafe!T template

 Such a template would test that all operations on type T are  safe.
 The template function could then be marked  trusted. The troubles with
 this are (a) it is all or nothing with T, i.e. if a template function
 only used an  safe subset of T, it still would not be accepted and (b)
 it does not do proper inference of the safety of a template function.
What about isSafe!(T, method1, method2, ...)? I.e., test the safety of an explicit list of operations that the template function will be using?
Listing the operations to be used is the same as the if(0) solution.
Feb 09 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 03:36:14 UTC, Walter Bright wrote:
 On 2/9/2015 6:21 PM, H. S. Teoh via Digitalmars-d wrote:
 What stops the following abuse of  trusted via
 trusted()?

 	int* myFunc(void* p)  safe // <-- I'm claiming to be  safe
 	{
 		// But actually I'm not! Though I can convince the
 		// compiler that I am...
 		return trusted!(() => cast(int*)p);
 	}

 	char c;
 	auto p = myFunc(&c); // oops
 	*p = 999; // kaboom

 Are we just relying on convention that trusted() will not be 
 abused in
 this way?
That's right. trusted will always rely on convention.
You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.
Feb 09 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to save people
 the awkward burden of importing it from std.conv all the time. But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu 
wrote:
 On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to 
 save people
 the awkward burden of importing it from std.conv all the time. 
 But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
People were worried about reliable inlining, but maybe the compiler can just guarantee that somehow. Aren't we back where we were before? The only addition we were recommending is the equivalent of only allowing trusted lambdas like the one above in trusted and system functions, but not in safe ones.
Feb 09 2015
next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 04:18:16 UTC, Zach the Mystic 
wrote:
 On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei 
 Alexandrescu wrote:
 On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to 
 save people
 the awkward burden of importing it from std.conv all the 
 time. But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
People were worried about reliable inlining, but maybe the compiler can just guarantee that somehow. Aren't we back where we were before? The only addition we were recommending is the equivalent of only allowing trusted lambdas like the one above in trusted and system functions, but not in safe ones.
Hey, why use ' system' lambdas instead? They are already banned in safe code.
Feb 09 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/9/15 11:18 PM, Zach the Mystic wrote:
 On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:
 On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to save people
 the awkward burden of importing it from std.conv all the time. But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
People were worried about reliable inlining, but maybe the compiler can just guarantee that somehow. Aren't we back where we were before? The only addition we were recommending is the equivalent of only allowing trusted lambdas like the one above in trusted and system functions, but not in safe ones.
IIRC, gdc and ldc already do this. This is not an issue that should dictate convention, non-inlined code does not impact correctness, only performance. And once it is fixed in the compiler, the argument would be gone, and we would be left with a convention with no good reason. -Steve
Feb 09 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2015 9:04 PM, Steven Schveighoffer wrote:
 This is not an issue that should dictate convention, non-inlined code does not
 impact correctness, only performance. And once it is fixed in the compiler, the
 argument would be gone, and we would be left with a convention with no good
reason.
Exactly. The inlining issue is orthogonal to this and irrelevant, although important in its own right.
Feb 10 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu 
wrote:
 On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to 
 save people
 the awkward burden of importing it from std.conv all the time. 
 But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Erm. This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.
Feb 09 2015
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/10/15 12:06 AM, Dicebot wrote:
 On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:
 On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to save people
 the awkward burden of importing it from std.conv all the time. But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Erm. This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.
Perhaps an explosion of added delegate calls will motivate people to fix it ;) FWIW, I hate this kind of optimization based on a limitation. -Steve
Feb 09 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/9/15 9:06 PM, Dicebot wrote:
 On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:
 On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to save people
 the awkward burden of importing it from std.conv all the time. But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Erm. This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.
Correct. The new rules, however, prohibit using trusted code inside safe functions. -- Andrei
Feb 09 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 05:18:26 UTC, Andrei Alexandrescu 
wrote:
 On 2/9/15 9:06 PM, Dicebot wrote:
 On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei 
 Alexandrescu wrote:
 On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to 
 save people
 the awkward burden of importing it from std.conv all the 
 time. But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Erm. This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.
Correct. The new rules, however, prohibit using trusted code inside safe functions. -- Andrei
There's something weird going on. H.S. Teoh's idea for system blocks was actually an attempt to reconcile Walter's stated desire to have trusted interfaces at the named function level with the practical need to specify precisely which code was system and allow safety checking for all the rest. The system blocks were originally a way to still have a use for named trusted functions, even in the presence of specific system blocks. But named trusted functions become a redundancy when all system code is precisely marked. David Nadlinger originally pointed out almost three years ago that from the caller's perspective, there is *no* difference between safe and trusted. They are both effectively safe. The only ostensible explanation for the existence of trusted was to point a programmer who had already identified problems with memory safety to the possible locations of the unsafe code. This is arguably not a good enough reason to invent a whole new built-in function attribute, but there it is. When you mark a function safe, you are expected to make sure it's safe. It's not really the concern of a function attribute how you made sure it was safe. trusted lambdas are an effective way of marking unsafe code. But they are completely pointless, at this time, unless they are placed within the context a function marked (or inferred) safe. For them to become the idiomatic way of marking system code, there is no further purpose to any *named* trusted function, ever. To preserve the (admittedly marginal) utility of named trusted functions -- that is, the opportunity for programmers using a library which contains only the interface function signatures and a binary object file to more clearly identify which of the used functions might be causing their memory safety problems -- while also allowing safety checking of all trusted code outside specifically marked system blocks, would require a breaking change, as indicated in the system blocks proposal. Avoiding the breaking change requires using trusted lambdas in place of system blocks. This nullifies the original purpose of named trusted functions, as any code which effectively uses trusted lambdas for their intended purpose must (implicitly or explicitly) be marked safe. I never used the named trusted function for its original purpose anyway. I dont' see it as a big sacrifice, but anyway, you have three choices: 1. Keep *named* trusted functions for their original purpose, continue to rely on the programmer for manual verification of all trusted code, and break no code. (status quo) 2. Give up on *named* trusted functions as a lost cause, begin using trusted lambdas to isolate all system code, and break no code. 3. Keep named trusted functions, introduce system blocks to isolate system code, and break a lot of code. The way I see it, number 2 is the way to go, or at least the way Andrei is headed. 3 is elegant, and if I were in charge, I'd probably investigate how to make it work. 1, the status quo, is the worst, because it preserves the thing of low value (named trustedness) at the expense of the thing of greatest value (isolating unsafe code). I would say "destroy", but this isn't even an idea, just an analysis of the tradeoffs. So, ENJOY!
Feb 09 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 10 February 2015 at 06:35:08 UTC, Zach the Mystic 
wrote:
 I never used the named  trusted function for its original 
 purpose anyway. I dont' see it as a big sacrifice, but anyway, 
 you have three choices:

 1. Keep *named*  trusted functions for their original purpose, 
 continue to rely on the programmer for manual verification of 
 all  trusted code, and break no code. (status quo)

 2. Give up on *named*  trusted functions as a lost cause, begin 
 using  trusted lambdas to isolate all  system code, and break 
 no code.

 3. Keep named  trusted functions, introduce  system blocks to 
 isolate  system code, and break a lot of code.

 The way I see it, number 2 is the way to go, or at least the 
 way Andrei is headed. 3 is elegant, and if I were in charge, 
 I'd probably investigate how to make it work. 1, the status 
 quo, is the worst, because it preserves the thing of low value 
 (named  trustedness) at the expense of the thing of greatest 
 value (isolating unsafe code).

 I would say "destroy", but this isn't even an idea, just an 
 analysis of the tradeoffs. So, ENJOY!
As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics. This could either be kept indefinitely, or trusted functions without system blocks could slowly be deprecated and phased out, leading to (3).
Feb 10 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 12:27:29 UTC, Marc Schütz wrote:
 On Tuesday, 10 February 2015 at 06:35:08 UTC, Zach the Mystic 
 wrote:
 I never used the named  trusted function for its original 
 purpose anyway. I dont' see it as a big sacrifice, but anyway, 
 you have three choices:

 1. Keep *named*  trusted functions for their original purpose, 
 continue to rely on the programmer for manual verification of 
 all  trusted code, and break no code. (status quo)

 2. Give up on *named*  trusted functions as a lost cause, 
 begin using  trusted lambdas to isolate all  system code, and 
 break no code.

 3. Keep named  trusted functions, introduce  system blocks to 
 isolate  system code, and break a lot of code.

 The way I see it, number 2 is the way to go, or at least the 
 way Andrei is headed. 3 is elegant, and if I were in charge, 
 I'd probably investigate how to make it work. 1, the status 
 quo, is the worst, because it preserves the thing of low value 
 (named  trustedness) at the expense of the thing of greatest 
 value (isolating unsafe code).

 I would say "destroy", but this isn't even an idea, just an 
 analysis of the tradeoffs. So, ENJOY!
As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics.
But they *have* no semantics without disallowing system code in the rest of the trusted function.
 This could either be kept indefinitely, or  trusted functions 
 without  system blocks could slowly be deprecated and phased 
 out, leading to (3).
This is a transition to the inevitable breaking change of 3. It's the equivalent of making a DIP and then having a compiler switch "-DIPxxx" to turn it on or off.
Feb 10 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 15:47:14 UTC, Zach the Mystic 
wrote:
 On Tuesday, 10 February 2015 at 12:27:29 UTC, Marc Schütz wrote:
 On Tuesday, 10 February 2015 at 06:35:08 UTC, Zach the Mystic 
 wrote:
 I never used the named  trusted function for its original 
 purpose anyway. I dont' see it as a big sacrifice, but 
 anyway, you have three choices:

 1. Keep *named*  trusted functions for their original 
 purpose, continue to rely on the programmer for manual 
 verification of all  trusted code, and break no code. (status 
 quo)

 2. Give up on *named*  trusted functions as a lost cause, 
 begin using  trusted lambdas to isolate all  system code, and 
 break no code.

 3. Keep named  trusted functions, introduce  system blocks to 
 isolate  system code, and break a lot of code.

 The way I see it, number 2 is the way to go, or at least the 
 way Andrei is headed. 3 is elegant, and if I were in charge, 
 I'd probably investigate how to make it work. 1, the status 
 quo, is the worst, because it preserves the thing of low 
 value (named  trustedness) at the expense of the thing of 
 greatest value (isolating unsafe code).

 I would say "destroy", but this isn't even an idea, just an 
 analysis of the tradeoffs. So, ENJOY!
As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics.
But they *have* no semantics without disallowing system code in the rest of the trusted function.
Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).
Feb 10 2015
next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 15:49:24 UTC, Zach the Mystic
wrote:
 Wait a second... you're totally right. That is a cool solution. 
 The only hiccup is that it might be hard to implement in the 
 compiler because of flow tracking (i.e. the error comes before 
 the  system block, forcing a recheck of all preceding 
 functions.).
I mean all preceding statements.
Feb 10 2015
prev sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 15:49:24 UTC, Zach the Mystic 
wrote:
 As already pointed out in the other thread, there is a 
 non-breaking variant of (3):

 3a. Keep named  trusted functions, allow  system blocks 
 inside them, but only treat those with  system blocks with 
 the new semantics.
But they *have* no semantics without disallowing system code in the rest of the trusted function.
Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).
I'm sorry I misread you at first -- this is actually really cool (notwithstanding the hiccup)!
Feb 10 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 10 February 2015 at 15:57:28 UTC, Zach the Mystic 
wrote:
 On Tuesday, 10 February 2015 at 15:49:24 UTC, Zach the Mystic 
 wrote:
 As already pointed out in the other thread, there is a 
 non-breaking variant of (3):

 3a. Keep named  trusted functions, allow  system blocks 
 inside them, but only treat those with  system blocks with 
 the new semantics.
But they *have* no semantics without disallowing system code in the rest of the trusted function.
Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).
I'm sorry I misread you at first -- this is actually really cool (notwithstanding the hiccup)!
No problem! At first I thought it was only a nice deprecation path, but I realised that the intermediate stage could even be kept indefinitely. It probably wouldn't be too complicated to implement, because semantic analysis already happens in several stages. I think safe checks happen relatively late, which means that there has already been one complete traversal of the functions AST which can take a note whenever it sees an system block. If not, it would have to do a simple scan first, which doesn't seem to complicated either.
Feb 10 2015
parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 10 February 2015 at 16:04:05 UTC, Marc Schütz wrote:
 On Tuesday, 10 February 2015 at 15:57:28 UTC, Zach the Mystic 
 wrote:
 On Tuesday, 10 February 2015 at 15:49:24 UTC, Zach the Mystic 
 wrote:
 As already pointed out in the other thread, there is a 
 non-breaking variant of (3):

 3a. Keep named  trusted functions, allow  system blocks 
 inside them, but only treat those with  system blocks with 
 the new semantics.
But they *have* no semantics without disallowing system code in the rest of the trusted function.
Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).
I'm sorry I misread you at first -- this is actually really cool (notwithstanding the hiccup)!
No problem! At first I thought it was only a nice deprecation path, but I realised that the intermediate stage could even be kept indefinitely.
Eventually the error should be the default, I say, but even then, a compiler switch can be kept around indefinitely which turns the error off.
 It probably wouldn't be too complicated to implement, because 
 semantic analysis already happens in several stages. I think 
  safe checks happen relatively late, which means that there has 
 already been one complete traversal of the functions AST which 
 can take a note whenever it sees an  system block.
Well that's just jolly!
Feb 10 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2015 9:06 PM, Dicebot wrote:
 This was exactly how all those trustedFoo() nested functions have appeared
 - we wanted to localized unsafe operations with `()  trusted {}` but it wasn't
 properly inlined by DMD.
DMD will inline trivial functions like that. In what case does it not?
Feb 09 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 10 February 2015 at 07:02:23 UTC, Walter Bright wrote:
 On 2/9/2015 9:06 PM, Dicebot wrote:
 This was exactly how all those trustedFoo() nested functions 
 have appeared
 - we wanted to localized unsafe operations with `()  trusted 
 {}` but it wasn't
 properly inlined by DMD.
DMD will inline trivial functions like that. In what case does it not?
import std.stdio; void foo() system { writeln("foo"); } void main() safe { () trusted { foo(); } (); } // dmd -inline -O -g test.d Breakpoint 1, D main () at test.d:10 10 () trusted { foo(); } (); (gdb) disassemble Dump of assembler code for function _Dmain: 0x000000000046d3f0 <+0>: push %rbp 0x000000000046d3f1 <+1>: mov %rsp,%rbp => 0x000000000046d3f4 <+4>: callq 0x46d400 <_D4test4mainFNfZ9__lambda1FNeZv> 0x000000000046d3f9 <+9>: xor %eax,%eax 0x000000000046d3fb <+11>: pop %rbp 0x000000000046d3fc <+12>: retq End of assembler dump.\ As you may see in generated call it actually calls lambda function (_D4test4mainFNfZ9__lambda1FNeZv) instead of calling _D4test3fooFZv directly
Feb 09 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/9/2015 11:27 PM, Dicebot wrote:
 As you may see in generated call it actually calls lambda function
 (_D4test4mainFNfZ9__lambda1FNeZv) instead of calling _D4test3fooFZv directly
https://issues.dlang.org/show_bug.cgi?id=14164
Feb 10 2015
prev sibling parent "Kagamin" <spam here.lot> writes:
On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu 
wrote:
 On 2/9/15 8:03 PM, Zach the Mystic wrote:
 You could put the 'trusted' template right in object.d, to 
 save people
 the awkward burden of importing it from std.conv all the time. 
 But that
 would be a language change, of sorts.
We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Maybe, allow trusted safe functions, which will 1) like safe, disallow direct use of unsafe operations 2) unlike safe, allow trusted blocks 3) mostly obey safe codegen requirements and be hopefully a little safer than plain trusted functions 4) be reviewed as trusted
Feb 10 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/9/15 6:21 PM, H. S. Teoh via Digitalmars-d wrote:
 I wonder if there's a way, in the current language, to make it illegal
 to call trusted() from a function explicitly marked  safe...?
No, but that might be worthy future consideration. I think we now should move forward with this and see how it fares. -- Andrei
Feb 09 2015