digitalmars.D - Trusted Manifesto
- Walter Bright (210/210) Feb 09 2015 Andrei and I have learned a lot from the @trusted discussions. It's clea...
- Dicebot (11/21) Feb 09 2015 Shouldn't that be `return assumeUnique(r)` instead?
- Walter Bright (10/25) Feb 09 2015 assumeUnique does a little more than assume the argument is unique - it ...
- Dicebot (10/36) Feb 09 2015 What I have meant is that here cast is only safe under assumption
- Kagamin (2/4) Feb 09 2015 Can SafeD be enforced given that?
- Kagamin (1/1) Feb 09 2015 Also why safe code is allowed to create unsafe lambdas?
- Andrei Alexandrescu (2/3) Feb 09 2015 Might be interesting to restrict that. -- Andrei
- Andrei Alexandrescu (5/9) Feb 09 2015 At some point you have to define a trusted computing base. That includes...
- John Colvin (4/239) Feb 09 2015 Looks great.
- Walter Bright (4/6) Feb 09 2015 Sounds good, but I'd like to see how this works in practice before going...
- Meta (5/13) Feb 09 2015 On the topic of safety, I seem to remember that bounds checking
- John Colvin (6/20) Feb 09 2015 bounds checking *can be* disabled in @trusted and @system code,
- Meta (3/26) Feb 09 2015 Yes, but is it disabled by default in @trusted code? That's what
- Walter Bright (2/3) Feb 09 2015 It is never disabled by default.
- Meta (2/5) Feb 09 2015 Okay, I thought it was similar to @system in that respect.
- Walter Bright (2/8) Feb 09 2015 @system doesn't disable it, either.
- Zach the Mystic (28/36) Feb 09 2015 Until you said this, I was feeling bad that language changes were
- Steven Schveighoffer (41/256) Feb 09 2015 This I don't disagree with. The issue is mostly that the entire function...
- Andrei Alexandrescu (3/4) Feb 09 2015 I'm okay with this as with most of Steve's points. But "genericness" is
- Steven Schveighoffer (4/8) Feb 09 2015 Merriam Webster says otherwise ;)
- Andrei Alexandrescu (2/10) Feb 09 2015 Interesting, thanks. Also, M-W does not list "genericity". -- Andrei
- Steven Schveighoffer (5/17) Feb 09 2015 I know. It looked wrong to me (genericity), but I didn't know what would...
- Zach the Mystic (3/23) Feb 09 2015 I prefer to program with "generitude"!
- Wyatt (7/13) Feb 09 2015 Webster needs to get over itself; according to usage, both are
- Russel Winder via Digitalmars-d (22/33) Feb 10 2015 But that dictionary doesn't matter, the OED is the one true keeper of=20
- Steven Schveighoffer (9/24) Feb 10 2015 Sure it matters. If it's defined in M-W it should be defined in OED.
- H. S. Teoh via Digitalmars-d (77/150) Feb 09 2015 [...]
- Walter Bright (6/41) Feb 09 2015 The very idea of @trusted is you're trusting the programmer with somethi...
- Zach the Mystic (4/23) Feb 09 2015 You could put the 'trusted' template right in object.d, to save
- Andrei Alexandrescu (4/7) Feb 09 2015 We won't define it. Instead we'll go with Steve's idea: () @trusted =>
- Zach the Mystic (7/16) Feb 09 2015 People were worried about reliable inlining, but maybe the
- Zach the Mystic (4/21) Feb 09 2015 Hey, why use '@system' lambdas instead? They are already banned
- Steven Schveighoffer (7/21) Feb 09 2015 IIRC, gdc and ldc already do this.
- Walter Bright (3/6) Feb 10 2015 Exactly. The inlining issue is orthogonal to this and irrelevant, althou...
- Dicebot (5/14) Feb 09 2015 Erm. This was exactly how all those trustedFoo() nested functions
- Steven Schveighoffer (5/17) Feb 09 2015 Perhaps an explosion of added delegate calls will motivate people to fix...
- Andrei Alexandrescu (3/15) Feb 09 2015 Correct. The new rules, however, prohibit using trusted code inside
- Zach the Mystic (60/83) Feb 09 2015 There's something weird going on.
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (10/29) Feb 10 2015 As already pointed out in the other thread, there is a
- Zach the Mystic (6/40) Feb 10 2015 But they *have* no semantics without disallowing @system code in
- Zach the Mystic (6/43) Feb 10 2015 Wait a second... you're totally right. That is a cool solution.
- Zach the Mystic (3/8) Feb 10 2015 I mean all preceding statements.
- Zach the Mystic (4/18) Feb 10 2015 I'm sorry I misread you at first -- this is actually really cool
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (13/32) Feb 10 2015 No problem!
- Zach the Mystic (5/36) Feb 10 2015 Eventually the error should be the default, I say, but even then,
- Walter Bright (2/5) Feb 09 2015 DMD will inline trivial functions like that. In what case does it not?
- Dicebot (26/34) Feb 09 2015 import std.stdio;
- Walter Bright (2/4) Feb 10 2015 https://issues.dlang.org/show_bug.cgi?id=14164
- Kagamin (8/17) Feb 10 2015 Maybe, allow @trusted @safe functions, which will
- Andrei Alexandrescu (3/5) Feb 09 2015 No, but that might be worthy future consideration. I think we now should...
Andrei and I have learned a lot from the trusted discussions. It's clear the way we were approaching the problem was inadequate. So we came up with a proposal based on the ideas and criticisms of the participants in the references. It involves no language changes, but offers usage guidelines that we believe are workable. Tell us what you think! ---------------------------------------------------- Trusted Manifesto ----------------- Memory safety in D has the usual definition: a memory-safe program never reads uninitialized memory and never reads or writes memory with a type incompatible with the type it was written. The aim of D's safe, trusted, and system attributes is to provide as much mechanically verified to be safe code as possible. Functions --------- Function signatures inform what must be true about a function. If a function is marked as safe, it must contain only safe code. This safety must be mechanically checkable. Sometimes, an unsafe operation is needed even in a function that is overall safe. For example, here's a function that returns an upper case version of its input string: string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; // <== unsafe operation } The compiler rejects this function because it's declared as safe but contains an unsafe operation. Review of the whole function shows that the operation, in this instance, is safe. One way to deal with that is to do what is called an "escape", meaning telling the compiler "I know what I'm doing, so allow this operation": string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); static string trustedCast(char[] r) trusted { return cast(string)r; } return trustedCast(r); } This will pass typechecking. However, the only reason it is safe is because the context in which trustedCast() is called makes it safe. In other words, a manual review of the surrounding context is necessary, which violates the safe promise that its contents are mechanically checkable. I.e. the trustedCast() function is an "escape" that injects unsafety into its surrounding contents. This leads to: RULE 1: trusted code accessible from safe code must expose a safe interface to unsafe operations. trusted must not be used to inject unsafety into surrounding context that is marked safe. safe code must be mechanically verifiable to be safe, and subverting that is not acceptable. COROLLARY 1: trusted functions should be as small as possible to encapsulate the unsafe operation without injecting unsafety into safe code. In the case of toUpper(), it is necessary to review the entire function to verify that the cast is safe, so it is properly written: string toUpper(string s) trusted { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; } Use of local trusted functions with safe interfaces is encouraged to minimize the amount of safety code review required. Generic Functions ----------------- Generic functions are templates that accept compile time parameters in the form of types, values or aliases to other functions. Whether the function is safe or system is not checkable until the template function is instantiated with explicit arguments. If the template function is marked as safe, then it can only be instantiated with arguments that expose safe operations. If the template function is marked safe, then RULE 1 applies. But that reduces the genericity of the function. The compiler is able to deduce whether a template function is safe or system when it is instantiated. For maximum utility, we need a way to specify that: This template function is safe if the generic and non-generic operations it uses are safe as well, otherwise it is system. Consider a function to make an immutable array copy of a range: immutable(ElementType!Range)[] toArray(Range)(Range r) { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; return cast(immutable)a; // <== unsafe operation } Being a template function without specified attributes, the compiler will infer the attributes. But with the unsafe cast, toArray() will always be inferred to be system. But the rest of the code is safe. If toArray is marked as trusted, immutable(ElementType!Range)[] toArray(Range)(Range r) trusted { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; return cast(immutable)a; // <== unsafe operation } then if the range primitives (front, empty, popFront) exposed by the argument to r happen to be system, then those are invalidly assumed to be trustable. Every usage of toArray() would need to be reviewed for safety, which is impractical. What is needed is a way to isolate the unsafe operation, and enable the compiler to infer the rest. In other words, a local exemption from overall safety deduction is needed. Introducing the 'trusted' template to be put in std.conv: trusted auto trusted(alias fun)() { return fun(); } and used: immutable(ElementType!Range)[] toArray(Range)(Range r) { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; import std.conv : trusted; auto result = trusted!(() => cast(immutable)a); return result; } Use of the trusted escape requires the programmer to review the context to determine if it really is safe. The compiler will infer safety from the rest of the operations. RULE 2: Usage of escapes are only allowable in functions for which safety is inferred, and never when calling into as-of-yet not defined generic functions. But how can it be verified that toArray() is safe otherwise? RULE 3: An safe unittest must be used to verify safety when escapes are used. safe unittest { ... TODO: test toArray() ... } A unittest may also be constructed that verifies via static assert that if a type with system operations is passed to the function, that the function is inferred as system. The programmer must still verify that the usage of escapes that leak unsafety into the surrounding context is safe. RULE 4: Escape unsafety must not inject unsafety beyond the template function it is used in. Alternatives ------------ 1. if(0) block Provide an trusted local function that fully encapsulates the unsafe code and its context, providing a safe interface. For the operations on template parameters that may or may not be safe inside the local function, represent them in an if(0) block of code: if (0) // safety inference { Unqual!T tmp = cast(Unqual!T)item; emplaceRef!(Unqual!T)(tmp, cast(Unqual!T)item); } trusted void emplace() { auto bigData = _data.arr.ptr[0 .. len + 1]; emplaceRef!(Unqual!T)(bigData[len], cast(Unqual!T)item); //We do this at the end, in case of exceptions _data.arr = bigData; } emplace(); Although this works, it requires duplication of code in a rather careful, tedious, and essentially unmaintainable manner. It also simply looks wrong, although it could be made more palatable by enclosing it in a template. 2. isSafe!T template Such a template would test that all operations on type T are safe. The template function could then be marked trusted. The troubles with this are (a) it is all or nothing with T, i.e. if a template function only used an safe subset of T, it still would not be accepted and (b) it does not do proper inference of the safety of a template function. 3. system escape system would be used for escaping unsafe code in an trusted function, or in an un-attributed function it would instruct compiler to not use the escaped code when deducing trustworthiness. Unsafe code in an trusted function not so marked would generate an error. While this works, it would essentially break every trusted function already in existence. It is a somewhat nicer syntax than the std.conv.trusted template, but the backwards compatibility issue makes it unworkable. It offers a technical advantage over std.conv.trusted in that system will not be allowed in safe functions, while not allowing std.conv.trusted escapes in safe function would be by convention. References ---------- https://github.com/D-Programming-Language/phobos/pull/2966 http://forum.dlang.org/post/mb0uvr$2fdb$1 digitalmars.com Acknowledgements ---------------- Everyone who participated in the references!
Feb 09 2015
string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; // <== unsafe operation }Shouldn't that be `return assumeUnique(r)` instead? What about requiring to put in-code comment that mentions condition verified safety relies on? (here - actual uniqueness of r)Introducing the 'trusted' template to be put in std.conv: trusted auto trusted(alias fun)() { return fun(); }Is this guaranteed to be inlined in frontend? Shouldn't it better be called `system` to denote operation is not actually trusted? ----------------------- In general, this is surprisingly good manifesto. The way it started I have abandoned all hope for any pragmatical compromise but it does address many of issues mentioned in discussion.
Feb 09 2015
On 2/9/2015 1:36 AM, Dicebot wrote:assumeUnique does a little more than assume the argument is unique - it also casts it, which is not a necessary consequence of holding a unique reference. For the purpose of this article, I'd rather have the unsafe cast be explicit rather than a side effect.string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; // <== unsafe operation }Shouldn't that be `return assumeUnique(r)` instead?What about requiring to put in-code comment that mentions condition verified safety relies on? (here - actual uniqueness of r)Good idea.pragma(inline, true) is not available yet!Introducing the 'trusted' template to be put in std.conv: trusted auto trusted(alias fun)() { return fun(); }Is this guaranteed to be inlined in frontend?Shouldn't it better be called `system` to denote operation is not actually trusted?Andrei had the idea that one could simply grep the code for 'trusted' and thereby flag the code (trusted and trusted) that merits special attention. I agreed it was a good idea.
Feb 09 2015
On Monday, 9 February 2015 at 10:16:48 UTC, Walter Bright wrote:On 2/9/2015 1:36 AM, Dicebot wrote:What I have meant is that here cast is only safe under assumption that casted slice is unique (has not other referenced). `assumeUnique` is supposed to exactly communicate that to the reader in most idiomatic manner. It would be still inferred as system here so rules won't be compromised.assumeUnique does a little more than assume the argument is unique - it also casts it, which is not a necessary consequence of holding a unique reference. For the purpose of this article, I'd rather have the unsafe cast be explicit rather than a side effect.string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; // <== unsafe operation }Shouldn't that be `return assumeUnique(r)` instead?And this is exactly why I am asking ;) Wide usage of this idiom before inlining is guaranteed may result in performance regressions.pragma(inline, true) is not available yet!Introducing the 'trusted' template to be put in std.conv: trusted auto trusted(alias fun)() { return fun(); }Is this guaranteed to be inlined in frontend?Fine by me.Shouldn't it better be called `system` to denote operation is not actually trusted?Andrei had the idea that one could simply grep the code for 'trusted' and thereby flag the code (trusted and trusted) that merits special attention. I agreed it was a good idea.
Feb 09 2015
On Monday, 9 February 2015 at 08:19:24 UTC, Walter Bright wrote:while not allowing std.conv.trusted escapes in safe function would be by convention.Can SafeD be enforced given that?
Feb 09 2015
Also why safe code is allowed to create unsafe lambdas?
Feb 09 2015
On 2/9/15 2:09 AM, Kagamin wrote:Also why safe code is allowed to create unsafe lambdas?Might be interesting to restrict that. -- Andrei
Feb 09 2015
On 2/9/15 2:05 AM, Kagamin wrote:On Monday, 9 February 2015 at 08:19:24 UTC, Walter Bright wrote:At some point you have to define a trusted computing base. That includes the compiler and e.g. the safe/ trusted functions in the standard library. With those in tow it's easy to enforce safety in client code by ruling auto trusted constructs automatically. -- Andreiwhile not allowing std.conv.trusted escapes in safe function would be by convention.Can SafeD be enforced given that?
Feb 09 2015
On Monday, 9 February 2015 at 08:19:24 UTC, Walter Bright wrote:Andrei and I have learned a lot from the trusted discussions. It's clear the way we were approaching the problem was inadequate. So we came up with a proposal based on the ideas and criticisms of the participants in the references. It involves no language changes, but offers usage guidelines that we believe are workable. Tell us what you think! ---------------------------------------------------- Trusted Manifesto ----------------- Memory safety in D has the usual definition: a memory-safe program never reads uninitialized memory and never reads or writes memory with a type incompatible with the type it was written. The aim of D's safe, trusted, and system attributes is to provide as much mechanically verified to be safe code as possible. Functions --------- Function signatures inform what must be true about a function. If a function is marked as safe, it must contain only safe code. This safety must be mechanically checkable. Sometimes, an unsafe operation is needed even in a function that is overall safe. For example, here's a function that returns an upper case version of its input string: string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; // <== unsafe operation } The compiler rejects this function because it's declared as safe but contains an unsafe operation. Review of the whole function shows that the operation, in this instance, is safe. One way to deal with that is to do what is called an "escape", meaning telling the compiler "I know what I'm doing, so allow this operation": string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); static string trustedCast(char[] r) trusted { return cast(string)r; } return trustedCast(r); } This will pass typechecking. However, the only reason it is safe is because the context in which trustedCast() is called makes it safe. In other words, a manual review of the surrounding context is necessary, which violates the safe promise that its contents are mechanically checkable. I.e. the trustedCast() function is an "escape" that injects unsafety into its surrounding contents. This leads to: RULE 1: trusted code accessible from safe code must expose a safe interface to unsafe operations. trusted must not be used to inject unsafety into surrounding context that is marked safe. safe code must be mechanically verifiable to be safe, and subverting that is not acceptable. COROLLARY 1: trusted functions should be as small as possible to encapsulate the unsafe operation without injecting unsafety into safe code. In the case of toUpper(), it is necessary to review the entire function to verify that the cast is safe, so it is properly written: string toUpper(string s) trusted { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; } Use of local trusted functions with safe interfaces is encouraged to minimize the amount of safety code review required. Generic Functions ----------------- Generic functions are templates that accept compile time parameters in the form of types, values or aliases to other functions. Whether the function is safe or system is not checkable until the template function is instantiated with explicit arguments. If the template function is marked as safe, then it can only be instantiated with arguments that expose safe operations. If the template function is marked safe, then RULE 1 applies. But that reduces the genericity of the function. The compiler is able to deduce whether a template function is safe or system when it is instantiated. For maximum utility, we need a way to specify that: This template function is safe if the generic and non-generic operations it uses are safe as well, otherwise it is system. Consider a function to make an immutable array copy of a range: immutable(ElementType!Range)[] toArray(Range)(Range r) { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; return cast(immutable)a; // <== unsafe operation } Being a template function without specified attributes, the compiler will infer the attributes. But with the unsafe cast, toArray() will always be inferred to be system. But the rest of the code is safe. If toArray is marked as trusted, immutable(ElementType!Range)[] toArray(Range)(Range r) trusted { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; return cast(immutable)a; // <== unsafe operation } then if the range primitives (front, empty, popFront) exposed by the argument to r happen to be system, then those are invalidly assumed to be trustable. Every usage of toArray() would need to be reviewed for safety, which is impractical. What is needed is a way to isolate the unsafe operation, and enable the compiler to infer the rest. In other words, a local exemption from overall safety deduction is needed. Introducing the 'trusted' template to be put in std.conv: trusted auto trusted(alias fun)() { return fun(); } and used: immutable(ElementType!Range)[] toArray(Range)(Range r) { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; import std.conv : trusted; auto result = trusted!(() => cast(immutable)a); return result; } Use of the trusted escape requires the programmer to review the context to determine if it really is safe. The compiler will infer safety from the rest of the operations. RULE 2: Usage of escapes are only allowable in functions for which safety is inferred, and never when calling into as-of-yet not defined generic functions. But how can it be verified that toArray() is safe otherwise? RULE 3: An safe unittest must be used to verify safety when escapes are used. safe unittest { ... TODO: test toArray() ... } A unittest may also be constructed that verifies via static assert that if a type with system operations is passed to the function, that the function is inferred as system. The programmer must still verify that the usage of escapes that leak unsafety into the surrounding context is safe. RULE 4: Escape unsafety must not inject unsafety beyond the template function it is used in. Alternatives ------------ 1. if(0) block Provide an trusted local function that fully encapsulates the unsafe code and its context, providing a safe interface. For the operations on template parameters that may or may not be safe inside the local function, represent them in an if(0) block of code: if (0) // safety inference { Unqual!T tmp = cast(Unqual!T)item; emplaceRef!(Unqual!T)(tmp, cast(Unqual!T)item); } trusted void emplace() { auto bigData = _data.arr.ptr[0 .. len + 1]; emplaceRef!(Unqual!T)(bigData[len], cast(Unqual!T)item); //We do this at the end, in case of exceptions _data.arr = bigData; } emplace(); Although this works, it requires duplication of code in a rather careful, tedious, and essentially unmaintainable manner. It also simply looks wrong, although it could be made more palatable by enclosing it in a template. 2. isSafe!T template Such a template would test that all operations on type T are safe. The template function could then be marked trusted. The troubles with this are (a) it is all or nothing with T, i.e. if a template function only used an safe subset of T, it still would not be accepted and (b) it does not do proper inference of the safety of a template function. 3. system escape system would be used for escaping unsafe code in an trusted function, or in an un-attributed function it would instruct compiler to not use the escaped code when deducing trustworthiness. Unsafe code in an trusted function not so marked would generate an error. While this works, it would essentially break every trusted function already in existence. It is a somewhat nicer syntax than the std.conv.trusted template, but the backwards compatibility issue makes it unworkable. It offers a technical advantage over std.conv.trusted in that system will not be allowed in safe functions, while not allowing std.conv.trusted escapes in safe function would be by convention. References ---------- https://github.com/D-Programming-Language/phobos/pull/2966 http://forum.dlang.org/post/mb0uvr$2fdb$1 digitalmars.com Acknowledgements ---------------- Everyone who participated in the references!Looks great. It seems to me that rules 2 and 3 could be helped along by tooling (compiler or external).
Feb 09 2015
On 2/9/2015 2:54 AM, John Colvin wrote:It seems to me that rules 2 and 3 could be helped along by tooling (compiler or external).Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
Feb 09 2015
On Monday, 9 February 2015 at 11:43:00 UTC, Walter Bright wrote:On 2/9/2015 2:54 AM, John Colvin wrote:On the topic of safety, I seem to remember that bounds checking is disabled in trusted code is this true? If so, can we change that? I think it should only be disabled in system code, if at all.It seems to me that rules 2 and 3 could be helped along by tooling (compiler or external).Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
Feb 09 2015
On Monday, 9 February 2015 at 11:47:03 UTC, Meta wrote:On Monday, 9 February 2015 at 11:43:00 UTC, Walter Bright wrote:bounds checking *can be* disabled in trusted and system code, by choice of compiler flags. It can even be disabled in safe as well, with -boundscheck=off. It might be nice to have a -boundscheck=trusted option. pragma(boundscheck, true/false) would also be nice for functions.On 2/9/2015 2:54 AM, John Colvin wrote:On the topic of safety, I seem to remember that bounds checking is disabled in trusted code is this true? If so, can we change that? I think it should only be disabled in system code, if at all.It seems to me that rules 2 and 3 could be helped along by tooling (compiler or external).Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
Feb 09 2015
On Monday, 9 February 2015 at 12:02:12 UTC, John Colvin wrote:On Monday, 9 February 2015 at 11:47:03 UTC, Meta wrote:Yes, but is it disabled by default in trusted code? That's what would be nice to change.On Monday, 9 February 2015 at 11:43:00 UTC, Walter Bright wrote:bounds checking *can be* disabled in trusted and system code, by choice of compiler flags. It can even be disabled in safe as well, with -boundscheck=off. It might be nice to have a -boundscheck=trusted option. pragma(boundscheck, true/false) would also be nice for functions.On 2/9/2015 2:54 AM, John Colvin wrote:On the topic of safety, I seem to remember that bounds checking is disabled in trusted code is this true? If so, can we change that? I think it should only be disabled in system code, if at all.It seems to me that rules 2 and 3 could be helped along by tooling (compiler or external).Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
Feb 09 2015
On 2/9/2015 4:14 AM, Meta wrote:Yes, but is it disabled by default in trusted code?It is never disabled by default.
Feb 09 2015
On Monday, 9 February 2015 at 20:24:02 UTC, Walter Bright wrote:On 2/9/2015 4:14 AM, Meta wrote:Okay, I thought it was similar to system in that respect.Yes, but is it disabled by default in trusted code?It is never disabled by default.
Feb 09 2015
On 2/9/2015 12:53 PM, Meta wrote:On Monday, 9 February 2015 at 20:24:02 UTC, Walter Bright wrote:system doesn't disable it, either.On 2/9/2015 4:14 AM, Meta wrote:Okay, I thought it was similar to system in that respect.Yes, but is it disabled by default in trusted code?It is never disabled by default.
Feb 09 2015
On Monday, 9 February 2015 at 11:43:00 UTC, Walter Bright wrote:On 2/9/2015 2:54 AM, John Colvin wrote:Until you said this, I was feeling bad that language changes were just completely off the table. I'm not a long-term member of the D community, but a lot of the excitement I get from following D is its "relentless pursuit of perfection", to borrow someone else's slogan. I see a surprising willingness from the early adopters to endure breaking changes for the sake of the long run, and I feel like the pursuit of perfection should utilize this willingness as much as possible. (This might be one of the language changes which is "dfixable", granting it significantly lower switching costs.) That said, I really like your approach. It's like you're building the scaffolding first, and willing to build the statue itself (i.e. language changes) later, should the scaffolding prove too shaky. I think the risk here is that a number of users will commit to the scaffolding, only to find their code saturated with it when it is found to be inadequate. To accommodate this risk: 1. Adopt a tone of experimentation, rather than a tone of authority, when imposing these rules. This will help people understand the risks and encourage them to give feedback. 2. Bear in mind an escape hatch. Try to imagine how a codebase which enforces these rules might transition out of them, should it become necessary. As to the rules themselves, they seem like an admirable attempt to do what's possible within the existing language, to which I have no opposition, assuming they are found in unbiased assessment to work.It seems to me that rules 2 and 3 could be helped along by tooling (compiler or external).Sounds good, but I'd like to see how this works in practice before going further with it. The nice thing about this proposal is it involves no language changes. It'll allow us to gain experience before committing to language changes.
Feb 09 2015
This is headed in the right direction, I think. Comments below. On 2/9/15 3:19 AM, Walter Bright wrote:Andrei and I have learned a lot from the trusted discussions. It's clear the way we were approaching the problem was inadequate. So we came up with a proposal based on the ideas and criticisms of the participants in the references. It involves no language changes, but offers usage guidelines that we believe are workable. Tell us what you think! ---------------------------------------------------- Trusted Manifesto ----------------- Memory safety in D has the usual definition: a memory-safe program never reads uninitialized memory and never reads or writes memory with a type incompatible with the type it was written. The aim of D's safe, trusted, and system attributes is to provide as much mechanically verified to be safe code as possible. Functions --------- Function signatures inform what must be true about a function. If a function is marked as safe, it must contain only safe code. This safety must be mechanically checkable. Sometimes, an unsafe operation is needed even in a function that is overall safe. For example, here's a function that returns an upper case version of its input string: string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; // <== unsafe operation } The compiler rejects this function because it's declared as safe but contains an unsafe operation. Review of the whole function shows that the operation, in this instance, is safe. One way to deal with that is to do what is called an "escape", meaning telling the compiler "I know what I'm doing, so allow this operation": string toUpper(string s) safe { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); static string trustedCast(char[] r) trusted { return cast(string)r; } return trustedCast(r); } This will pass typechecking. However, the only reason it is safe is because the context in which trustedCast() is called makes it safe. In other words, a manual review of the surrounding context is necessary, which violates the safe promise that its contents are mechanically checkable. I.e. the trustedCast() function is an "escape" that injects unsafety into its surrounding contents. This leads to: RULE 1: trusted code accessible from safe code must expose a safe interface to unsafe operations. trusted must not be used to inject unsafety into surrounding context that is marked safe. safe code must be mechanically verifiable to be safe, and subverting that is not acceptable. COROLLARY 1: trusted functions should be as small as possible to encapsulate the unsafe operation without injecting unsafety into safe code. In the case of toUpper(), it is necessary to review the entire function to verify that the cast is safe, so it is properly written: string toUpper(string s) trusted { char[] r = new char[s.length]; foreach (i, c; s) r[i] = toUpper(c); return cast(string)r; }This I don't disagree with. The issue is mostly that the entire function marked as trusted allows for future additions that are unintentionally un- safe. But we can work with this for now and see how it goes.Use of local trusted functions with safe interfaces is encouraged to minimize the amount of safety code review required.An example here would be good. Perhaps the memcpy example.Generic Functions ----------------- Generic functions are templates that accept compile time parameters in the form of types, values or aliases to other functions. Whether the function is safe or system is not checkable until the template function is instantiated with explicit arguments. If the template function is marked as safe, then it can only be instantiated with arguments that expose safe operations. If the template function is marked safe, then RULE 1 applies. But that reduces the genericity of the function. The compiler is able togenericnessdeduce whether a template function is safe or system when it is instantiated. For maximum utility, we need a way to specify that: This template function is safe if the generic and non-generic operations it uses are safe as well, otherwise it is system. Consider a function to make an immutable array copy of a range: immutable(ElementType!Range)[] toArray(Range)(Range r) { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; return cast(immutable)a; // <== unsafe operation } Being a template function without specified attributes, the compiler will infer the attributes. But with the unsafe cast, toArray() will always be inferred to be system. But the rest of the code is safe. If toArray is marked as trusted, immutable(ElementType!Range)[] toArray(Range)(Range r) trusted { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; return cast(immutable)a; // <== unsafe operation } then if the range primitives (front, empty, popFront) exposed by the argument to r happen to be system, then those are invalidly assumed to be trustable. Every usage of toArray() would need to be reviewed for safety, which is impractical. What is needed is a way to isolate the unsafe operation, and enable the compiler to infer the rest. In other words, a local exemption from overall safety deduction is needed. Introducing the 'trusted' template to be put in std.conv: trusted auto trusted(alias fun)() { return fun(); } and used: immutable(ElementType!Range)[] toArray(Range)(Range r) { alias ElementType!Range E; alias Unqual!E U; U[] a = new U[r.length]; foreach (i, e; r) a[i] = e; import std.conv : trusted; auto result = trusted!(() => cast(immutable)a); return result; } Use of the trusted escape requires the programmer to review the context to determine if it really is safe. The compiler will infer safety from the rest of the operations.I actually like this. It is one step closer to the mechanically verified trusted function we have been asking for. In fact, it's exactly like that, as it will break existing if you happened to add system calls inside this function. However, there is a large hole in your example: Object[] o = ...; auto o2 = toArray(o); Now, we have o and o2. o2 is all immutable references to objects that are also in o as mutable. The above function is not safe. This is a demonstration that even with mechanically verified safe code with seemingly correct escapes, you cannot assume anything, especially with templates. I don't really like std.conv.trusted. It doesn't add enough value. What is wrong with using a lambda directly? auto result = (() trusted => cast(immutable)a)();RULE 2: Usage of escapes are only allowable in functions for which safety is inferred, and never when calling into as-of-yet not defined generic functions. But how can it be verified that toArray() is safe otherwise? RULE 3: An safe unittest must be used to verify safety when escapes are used. safe unittest { ... TODO: test toArray() ... }This rule is misworded. The unit test does not verify that it's safe, it only verifies that safe can be applied to the function, even if the function isn't actually safe. I'd say: RULE 3: A safe unittest must be used to verify the function can be called as safe when escapes are used. However, this does NOT verify safety, that still must be manually checked.A unittest may also be constructed that verifies via static assert that if a type with system operations is passed to the function, that the function is inferred as system. The programmer must still verify that the usage of escapes that leak unsafety into the surrounding context is safe. RULE 4: Escape unsafety must not inject unsafety beyond the template function it is used in. Alternatives ------------ 1. if(0) block Provide an trusted local function that fully encapsulates the unsafe code and its context, providing a safe interface. For the operations on template parameters that may or may not be safe inside the local function, represent them in an if(0) block of code: if (0) // safety inference { Unqual!T tmp = cast(Unqual!T)item; emplaceRef!(Unqual!T)(tmp, cast(Unqual!T)item); } trusted void emplace() { auto bigData = _data.arr.ptr[0 .. len + 1]; emplaceRef!(Unqual!T)(bigData[len], cast(Unqual!T)item); //We do this at the end, in case of exceptions _data.arr = bigData; } emplace(); Although this works, it requires duplication of code in a rather careful, tedious, and essentially unmaintainable manner. It also simply looks wrong, although it could be made more palatable by enclosing it in a template. 2. isSafe!T template Such a template would test that all operations on type T are safe. The template function could then be marked trusted. The troubles with this are (a) it is all or nothing with T, i.e. if a template function only used an safe subset of T, it still would not be accepted and (b) it does not do proper inference of the safety of a template function. 3. system escape system would be used for escaping unsafe code in an trusted function, or in an un-attributed function it would instruct compiler to not use the escaped code when deducing trustworthiness. Unsafe code in an trusted function not so marked would generate an error. While this works, it would essentially break every trusted function already in existence. It is a somewhat nicer syntax than the std.conv.trusted template, but the backwards compatibility issue makes it unworkable. It offers a technical advantage over std.conv.trusted in that system will not be allowed in safe functions, while not allowing std.conv.trusted escapes in safe function would be by convention.I understand the idea of rejecting this to avoid breaking code. However, let's consider 3 things: 1. you may break trusted code, but when you do, it points to code that may not have been scrutinized as well as it should have been. 2. It allows making safe functions with trusted escapes trusted functions with system escapes. This is a better marking system than what we have today. 3. Having the user tag properly what is non-safe and what is safe, allows for other mechanical verification later on. For example tagging data as system so normal trusted code cannot touch it without marking as well. -Steve
Feb 09 2015
On 2/9/15 7:48 AM, Steven Schveighoffer wrote:auto result = (() trusted => cast(immutable)a)();I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Feb 09 2015
On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:On 2/9/15 7:48 AM, Steven Schveighoffer wrote:Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/generic -Steveauto result = (() trusted => cast(immutable)a)();I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Feb 09 2015
On 2/9/15 10:12 AM, Steven Schveighoffer wrote:On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:Interesting, thanks. Also, M-W does not list "genericity". -- AndreiOn 2/9/15 7:48 AM, Steven Schveighoffer wrote:Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/genericauto result = (() trusted => cast(immutable)a)();I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Feb 09 2015
On 2/9/15 1:18 PM, Andrei Alexandrescu wrote:On 2/9/15 10:12 AM, Steven Schveighoffer wrote:I know. It looked wrong to me (genericity), but I didn't know what would be the correct term, so I looked it up. FWIW, genericness doesn't look right to me either ;) -SteveOn 2/9/15 12:57 PM, Andrei Alexandrescu wrote:Interesting, thanks. Also, M-W does not list "genericity". -- AndreiOn 2/9/15 7:48 AM, Steven Schveighoffer wrote:Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/genericauto result = (() trusted => cast(immutable)a)();I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Feb 09 2015
On Monday, 9 February 2015 at 18:36:49 UTC, Steven Schveighoffer wrote:On 2/9/15 1:18 PM, Andrei Alexandrescu wrote:I prefer to program with "generitude"!On 2/9/15 10:12 AM, Steven Schveighoffer wrote:I know. It looked wrong to me (genericity), but I didn't know what would be the correct term, so I looked it up. FWIW, genericness doesn't look right to me either ;) -SteveOn 2/9/15 12:57 PM, Andrei Alexandrescu wrote:Interesting, thanks. Also, M-W does not list "genericity". -- AndreiOn 2/9/15 7:48 AM, Steven Schveighoffer wrote:Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/genericauto result = (() trusted => cast(immutable)a)();I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- Andrei
Feb 09 2015
On Monday, 9 February 2015 at 18:18:23 UTC, Andrei Alexandrescu wrote:On 2/9/15 10:12 AM, Steven Schveighoffer wrote:Webster needs to get over itself; according to usage, both are words. Though it looks like "genericity" is primarily used in the sciences, which is going to end up doing funny things to the connotation it carries. :) </linguist> -WyattMerriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/genericInteresting, thanks. Also, M-W does not list "genericity". --
Feb 09 2015
On Mon, 2015-02-09 at 13:12 -0500, Steven Schveighoffer via Digitalmars-d w= rote:On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:But that dictionary doesn't matter, the OED is the one true keeper of=20 the English language. =46rom what I can see genericity is not officially an English word, even=20 though OUP have published a book with this "word" as title:=20 http://ukcatalogue.oup.com/product/9780199691807.do Genericness is definitely listed, and so is a valid word. On the other hand, terms of art (aka jargon) are allowed, and computer=20 science has determined that genericity is a valid word. As long as it=20 is used in a computer science, software context. :-) --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winderOn 2/9/15 7:48 AM, Steven Schveighoffer wrote:=20 Merriam Webster says otherwise ;) =20 http://www.merriam-webster.com/dictionary/generic =20auto result =3D (() trusted =3D> cast(immutable)a)();=20 I'm okay with this as with most of Steve's points. But=20 "genericness" is not a word :o). -- Andrei
Feb 10 2015
On 2/10/15 9:06 AM, Russel Winder via Digitalmars-d wrote:On Mon, 2015-02-09 at 13:12 -0500, Steven Schveighoffer via Digitalmars-d wrote:Sure it matters. If it's defined in M-W it should be defined in OED. What DOESN'T matter is the wiki-style dictionaries that have popped up everywhere.On 2/9/15 12:57 PM, Andrei Alexandrescu wrote:But that dictionary doesn't matter, the OED is the one true keeper of the English language.On 2/9/15 7:48 AM, Steven Schveighoffer wrote:Merriam Webster says otherwise ;) http://www.merriam-webster.com/dictionary/genericauto result = (() trusted => cast(immutable)a)();I'm okay with this as with most of Steve's points. But "genericness" is not a word :o). -- AndreiFrom what I can see genericity is not officially an English wordWell, there I found genericity too (and notes that is used especially for technical references, which means it's probably more appropriate), so apparently that is OK, I rescind my edit :P http://www.oed.com/view/Entry/272627 -Steve
Feb 10 2015
On Mon, Feb 09, 2015 at 12:19:10AM -0800, Walter Bright via Digitalmars-d wrote: [...]Trusted Manifesto -----------------[...]RULE 1: trusted code accessible from safe code must expose a safe interface to unsafe operations. trusted must not be used to inject unsafety into surrounding context that is marked safe. safe code must be mechanically verifiable to be safe, and subverting that is not acceptable. COROLLARY 1: trusted functions should be as small as possible to encapsulate the unsafe operation without injecting unsafety into safe code.[...]Generic Functions -----------------[...]What is needed is a way to isolate the unsafe operation, and enable the compiler to infer the rest. In other words, a local exemption from overall safety deduction is needed. Introducing the 'trusted' template to be put in std.conv: trusted auto trusted(alias fun)() { return fun(); }This seems to be a step in the right direction, but doesn't this contradict rule 1? Certainly, trusted() doesn't provide a safe API, yet it is marked trusted. What stops the following abuse of trusted via trusted()? int* myFunc(void* p) safe // <-- I'm claiming to be safe { // But actually I'm not! Though I can convince the // compiler that I am... return trusted!(() => cast(int*)p); } char c; auto p = myFunc(&c); // oops *p = 999; // kaboom Are we just relying on convention that trusted() will not be abused in this way? And how would we prevent users from defining similar templates for escaping safety without being readily detectable, e.g., by grepping for 'trusted'? auto sneaky(alias fun)() trusted { return fun(); } Or is this a case of "if you insist on shooting your own foot the compiler can't/won't help you"? [...]RULE 2: Usage of escapes are only allowable in functions for which safety is inferred, and never when calling into as-of-yet not defined generic functions.I wonder if there's a way, in the current language, to make it illegal to call trusted() from a function explicitly marked safe...? [...]RULE 3: An safe unittest must be used to verify safety when escapes are used. safe unittest { ... TODO: test toArray() ... } A unittest may also be constructed that verifies via static assert that if a type with system operations is passed to the function, that the function is inferred as system. The programmer must still verify that the usage of escapes that leak unsafety into the surrounding context is safe.Makes sense, and does reflect current usage in Phobos. Thanks!!RULE 4: Escape unsafety must not inject unsafety beyond the template function it is used in.IOW, the following is illegal? int* myTemplate(void* p) trusted // <-- illegal leakage of unsafety { return trusted!(() => cast(int*) p); }Alternatives ------------ 1. if(0) block Provide an trusted local function that fully encapsulates the unsafe code and its context, providing a safe interface. For the operations on template parameters that may or may not be safe inside the local function, represent them in an if(0) block of code:[...]Although this works, it requires duplication of code in a rather careful, tedious, and essentially unmaintainable manner. It also simply looks wrong, although it could be made more palatable by enclosing it in a template.Yeah, this doesn't look viable.2. isSafe!T template Such a template would test that all operations on type T are safe. The template function could then be marked trusted. The troubles with this are (a) it is all or nothing with T, i.e. if a template function only used an safe subset of T, it still would not be accepted and (b) it does not do proper inference of the safety of a template function.What about isSafe!(T, method1, method2, ...)? I.e., test the safety of an explicit list of operations that the template function will be using? Of course, this isn't a fully-functional solution, since it doesn't handle the case where you *want* your template function to accept types with unsafe methods (just that the template function becomes system instead of safe). You'd have to copy-n-paste the function body and mark one safe (with isSafe!(...)) and the other system (with !isSafe!(...)). Unless there's a way of injecting function attributes based on the result of isSafe, like: // Hypothetical syntax auto myTemplate(R)(R range) { safe if (isSafe!(R, empty, front, popFront)) } { ... } But since we're trying not to require additional language support, this isn't really an option either.3. system escape system would be used for escaping unsafe code in an trusted function, or in an un-attributed function it would instruct compiler to not use the escaped code when deducing trustworthiness. Unsafe code in an trusted function not so marked would generate an error. While this works, it would essentially break every trusted function already in existence. It is a somewhat nicer syntax than the std.conv.trusted template, but the backwards compatibility issue makes it unworkable. It offers a technical advantage over std.conv.trusted in that system will not be allowed in safe functions, while not allowing std.conv.trusted escapes in safe function would be by convention.[...] If there was a way to force a compile error if someone tried to instantiate trusted() from inside an explicitly-marked safe function, we could have our cake and eat it too. One (very hackish) way might be a pragma or __trait that tests for explicit safe marking: auto trusted(alias fun)() trusted { // callerAttribute is a hypothetical trait. static if (__traits(callerAttribute, safe)) { static assert(0, "Abuse of trusted()"); } else { return fun(); } } void abusive(void* p) safe { int* ip = trusted!(() => cast(int*)p); // compile error: "Abuse of trusted()" } It's probably a bad idea to extend __traits in this way, but the point is that if we can *somehow* determine the safe marking of the caller, then we could prevent blatant abuses of trusted(). I don't know if the current language is able to achieve that, though. Can it? T -- Computers are like a jungle: they have monitor lizards, rams, mice, c-moss, binary trees... and bugs.
Feb 09 2015
On 2/9/2015 6:21 PM, H. S. Teoh via Digitalmars-d wrote:What stops the following abuse of trusted via trusted()? int* myFunc(void* p) safe // <-- I'm claiming to be safe { // But actually I'm not! Though I can convince the // compiler that I am... return trusted!(() => cast(int*)p); } char c; auto p = myFunc(&c); // oops *p = 999; // kaboom Are we just relying on convention that trusted() will not be abused in this way?That's right. trusted will always rely on convention.And how would we prevent users from defining similar templates for escaping safety without being readily detectable, e.g., by grepping for 'trusted'? auto sneaky(alias fun)() trusted { return fun(); } Or is this a case of "if you insist on shooting your own foot the compiler can't/won't help you"?The very idea of trusted is you're trusting the programmer with something that cannot be mechanically checked.Depends on the usage context, i.e. "beyond the template function it is used in".RULE 4: Escape unsafety must not inject unsafety beyond the template function it is used in.IOW, the following is illegal? int* myTemplate(void* p) trusted // <-- illegal leakage of unsafety { return trusted!(() => cast(int*) p); }Listing the operations to be used is the same as the if(0) solution.2. isSafe!T template Such a template would test that all operations on type T are safe. The template function could then be marked trusted. The troubles with this are (a) it is all or nothing with T, i.e. if a template function only used an safe subset of T, it still would not be accepted and (b) it does not do proper inference of the safety of a template function.What about isSafe!(T, method1, method2, ...)? I.e., test the safety of an explicit list of operations that the template function will be using?
Feb 09 2015
On Tuesday, 10 February 2015 at 03:36:14 UTC, Walter Bright wrote:On 2/9/2015 6:21 PM, H. S. Teoh via Digitalmars-d wrote:You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.What stops the following abuse of trusted via trusted()? int* myFunc(void* p) safe // <-- I'm claiming to be safe { // But actually I'm not! Though I can convince the // compiler that I am... return trusted!(() => cast(int*)p); } char c; auto p = myFunc(&c); // oops *p = 999; // kaboom Are we just relying on convention that trusted() will not be abused in this way?That's right. trusted will always rely on convention.
Feb 09 2015
On 2/9/15 8:03 PM, Zach the Mystic wrote:You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:On 2/9/15 8:03 PM, Zach the Mystic wrote:People were worried about reliable inlining, but maybe the compiler can just guarantee that somehow. Aren't we back where we were before? The only addition we were recommending is the equivalent of only allowing trusted lambdas like the one above in trusted and system functions, but not in safe ones.You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
On Tuesday, 10 February 2015 at 04:18:16 UTC, Zach the Mystic wrote:On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:Hey, why use ' system' lambdas instead? They are already banned in safe code.On 2/9/15 8:03 PM, Zach the Mystic wrote:People were worried about reliable inlining, but maybe the compiler can just guarantee that somehow. Aren't we back where we were before? The only addition we were recommending is the equivalent of only allowing trusted lambdas like the one above in trusted and system functions, but not in safe ones.You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
On 2/9/15 11:18 PM, Zach the Mystic wrote:On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:IIRC, gdc and ldc already do this. This is not an issue that should dictate convention, non-inlined code does not impact correctness, only performance. And once it is fixed in the compiler, the argument would be gone, and we would be left with a convention with no good reason. -SteveOn 2/9/15 8:03 PM, Zach the Mystic wrote:People were worried about reliable inlining, but maybe the compiler can just guarantee that somehow. Aren't we back where we were before? The only addition we were recommending is the equivalent of only allowing trusted lambdas like the one above in trusted and system functions, but not in safe ones.You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
On 2/9/2015 9:04 PM, Steven Schveighoffer wrote:This is not an issue that should dictate convention, non-inlined code does not impact correctness, only performance. And once it is fixed in the compiler, the argument would be gone, and we would be left with a convention with no good reason.Exactly. The inlining issue is orthogonal to this and irrelevant, although important in its own right.
Feb 10 2015
On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:On 2/9/15 8:03 PM, Zach the Mystic wrote:Erm. This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
On 2/10/15 12:06 AM, Dicebot wrote:On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:Perhaps an explosion of added delegate calls will motivate people to fix it ;) FWIW, I hate this kind of optimization based on a limitation. -SteveOn 2/9/15 8:03 PM, Zach the Mystic wrote:Erm. This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
On 2/9/15 9:06 PM, Dicebot wrote:On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:Correct. The new rules, however, prohibit using trusted code inside safe functions. -- AndreiOn 2/9/15 8:03 PM, Zach the Mystic wrote:Erm. This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
On Tuesday, 10 February 2015 at 05:18:26 UTC, Andrei Alexandrescu wrote:On 2/9/15 9:06 PM, Dicebot wrote:There's something weird going on. H.S. Teoh's idea for system blocks was actually an attempt to reconcile Walter's stated desire to have trusted interfaces at the named function level with the practical need to specify precisely which code was system and allow safety checking for all the rest. The system blocks were originally a way to still have a use for named trusted functions, even in the presence of specific system blocks. But named trusted functions become a redundancy when all system code is precisely marked. David Nadlinger originally pointed out almost three years ago that from the caller's perspective, there is *no* difference between safe and trusted. They are both effectively safe. The only ostensible explanation for the existence of trusted was to point a programmer who had already identified problems with memory safety to the possible locations of the unsafe code. This is arguably not a good enough reason to invent a whole new built-in function attribute, but there it is. When you mark a function safe, you are expected to make sure it's safe. It's not really the concern of a function attribute how you made sure it was safe. trusted lambdas are an effective way of marking unsafe code. But they are completely pointless, at this time, unless they are placed within the context a function marked (or inferred) safe. For them to become the idiomatic way of marking system code, there is no further purpose to any *named* trusted function, ever. To preserve the (admittedly marginal) utility of named trusted functions -- that is, the opportunity for programmers using a library which contains only the interface function signatures and a binary object file to more clearly identify which of the used functions might be causing their memory safety problems -- while also allowing safety checking of all trusted code outside specifically marked system blocks, would require a breaking change, as indicated in the system blocks proposal. Avoiding the breaking change requires using trusted lambdas in place of system blocks. This nullifies the original purpose of named trusted functions, as any code which effectively uses trusted lambdas for their intended purpose must (implicitly or explicitly) be marked safe. I never used the named trusted function for its original purpose anyway. I dont' see it as a big sacrifice, but anyway, you have three choices: 1. Keep *named* trusted functions for their original purpose, continue to rely on the programmer for manual verification of all trusted code, and break no code. (status quo) 2. Give up on *named* trusted functions as a lost cause, begin using trusted lambdas to isolate all system code, and break no code. 3. Keep named trusted functions, introduce system blocks to isolate system code, and break a lot of code. The way I see it, number 2 is the way to go, or at least the way Andrei is headed. 3 is elegant, and if I were in charge, I'd probably investigate how to make it work. 1, the status quo, is the worst, because it preserves the thing of low value (named trustedness) at the expense of the thing of greatest value (isolating unsafe code). I would say "destroy", but this isn't even an idea, just an analysis of the tradeoffs. So, ENJOY!On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:Correct. The new rules, however, prohibit using trusted code inside safe functions. -- AndreiOn 2/9/15 8:03 PM, Zach the Mystic wrote:Erm. This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.You could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 09 2015
On Tuesday, 10 February 2015 at 06:35:08 UTC, Zach the Mystic wrote:I never used the named trusted function for its original purpose anyway. I dont' see it as a big sacrifice, but anyway, you have three choices: 1. Keep *named* trusted functions for their original purpose, continue to rely on the programmer for manual verification of all trusted code, and break no code. (status quo) 2. Give up on *named* trusted functions as a lost cause, begin using trusted lambdas to isolate all system code, and break no code. 3. Keep named trusted functions, introduce system blocks to isolate system code, and break a lot of code. The way I see it, number 2 is the way to go, or at least the way Andrei is headed. 3 is elegant, and if I were in charge, I'd probably investigate how to make it work. 1, the status quo, is the worst, because it preserves the thing of low value (named trustedness) at the expense of the thing of greatest value (isolating unsafe code). I would say "destroy", but this isn't even an idea, just an analysis of the tradeoffs. So, ENJOY!As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics. This could either be kept indefinitely, or trusted functions without system blocks could slowly be deprecated and phased out, leading to (3).
Feb 10 2015
On Tuesday, 10 February 2015 at 12:27:29 UTC, Marc Schütz wrote:On Tuesday, 10 February 2015 at 06:35:08 UTC, Zach the Mystic wrote:But they *have* no semantics without disallowing system code in the rest of the trusted function.I never used the named trusted function for its original purpose anyway. I dont' see it as a big sacrifice, but anyway, you have three choices: 1. Keep *named* trusted functions for their original purpose, continue to rely on the programmer for manual verification of all trusted code, and break no code. (status quo) 2. Give up on *named* trusted functions as a lost cause, begin using trusted lambdas to isolate all system code, and break no code. 3. Keep named trusted functions, introduce system blocks to isolate system code, and break a lot of code. The way I see it, number 2 is the way to go, or at least the way Andrei is headed. 3 is elegant, and if I were in charge, I'd probably investigate how to make it work. 1, the status quo, is the worst, because it preserves the thing of low value (named trustedness) at the expense of the thing of greatest value (isolating unsafe code). I would say "destroy", but this isn't even an idea, just an analysis of the tradeoffs. So, ENJOY!As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics.This could either be kept indefinitely, or trusted functions without system blocks could slowly be deprecated and phased out, leading to (3).This is a transition to the inevitable breaking change of 3. It's the equivalent of making a DIP and then having a compiler switch "-DIPxxx" to turn it on or off.
Feb 10 2015
On Tuesday, 10 February 2015 at 15:47:14 UTC, Zach the Mystic wrote:On Tuesday, 10 February 2015 at 12:27:29 UTC, Marc Schütz wrote:Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).On Tuesday, 10 February 2015 at 06:35:08 UTC, Zach the Mystic wrote:But they *have* no semantics without disallowing system code in the rest of the trusted function.I never used the named trusted function for its original purpose anyway. I dont' see it as a big sacrifice, but anyway, you have three choices: 1. Keep *named* trusted functions for their original purpose, continue to rely on the programmer for manual verification of all trusted code, and break no code. (status quo) 2. Give up on *named* trusted functions as a lost cause, begin using trusted lambdas to isolate all system code, and break no code. 3. Keep named trusted functions, introduce system blocks to isolate system code, and break a lot of code. The way I see it, number 2 is the way to go, or at least the way Andrei is headed. 3 is elegant, and if I were in charge, I'd probably investigate how to make it work. 1, the status quo, is the worst, because it preserves the thing of low value (named trustedness) at the expense of the thing of greatest value (isolating unsafe code). I would say "destroy", but this isn't even an idea, just an analysis of the tradeoffs. So, ENJOY!As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics.
Feb 10 2015
On Tuesday, 10 February 2015 at 15:49:24 UTC, Zach the Mystic wrote:Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).I mean all preceding statements.
Feb 10 2015
On Tuesday, 10 February 2015 at 15:49:24 UTC, Zach the Mystic wrote:I'm sorry I misread you at first -- this is actually really cool (notwithstanding the hiccup)!Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics.But they *have* no semantics without disallowing system code in the rest of the trusted function.
Feb 10 2015
On Tuesday, 10 February 2015 at 15:57:28 UTC, Zach the Mystic wrote:On Tuesday, 10 February 2015 at 15:49:24 UTC, Zach the Mystic wrote:No problem! At first I thought it was only a nice deprecation path, but I realised that the intermediate stage could even be kept indefinitely. It probably wouldn't be too complicated to implement, because semantic analysis already happens in several stages. I think safe checks happen relatively late, which means that there has already been one complete traversal of the functions AST which can take a note whenever it sees an system block. If not, it would have to do a simple scan first, which doesn't seem to complicated either.I'm sorry I misread you at first -- this is actually really cool (notwithstanding the hiccup)!Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics.But they *have* no semantics without disallowing system code in the rest of the trusted function.
Feb 10 2015
On Tuesday, 10 February 2015 at 16:04:05 UTC, Marc Schütz wrote:On Tuesday, 10 February 2015 at 15:57:28 UTC, Zach the Mystic wrote:Eventually the error should be the default, I say, but even then, a compiler switch can be kept around indefinitely which turns the error off.On Tuesday, 10 February 2015 at 15:49:24 UTC, Zach the Mystic wrote:No problem! At first I thought it was only a nice deprecation path, but I realised that the intermediate stage could even be kept indefinitely.I'm sorry I misread you at first -- this is actually really cool (notwithstanding the hiccup)!Wait a second... you're totally right. That is a cool solution. The only hiccup is that it might be hard to implement in the compiler because of flow tracking (i.e. the error comes before the system block, forcing a recheck of all preceding functions.).As already pointed out in the other thread, there is a non-breaking variant of (3): 3a. Keep named trusted functions, allow system blocks inside them, but only treat those with system blocks with the new semantics.But they *have* no semantics without disallowing system code in the rest of the trusted function.It probably wouldn't be too complicated to implement, because semantic analysis already happens in several stages. I think safe checks happen relatively late, which means that there has already been one complete traversal of the functions AST which can take a note whenever it sees an system block.Well that's just jolly!
Feb 10 2015
On 2/9/2015 9:06 PM, Dicebot wrote:This was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.DMD will inline trivial functions like that. In what case does it not?
Feb 09 2015
On Tuesday, 10 February 2015 at 07:02:23 UTC, Walter Bright wrote:On 2/9/2015 9:06 PM, Dicebot wrote:import std.stdio; void foo() system { writeln("foo"); } void main() safe { () trusted { foo(); } (); } // dmd -inline -O -g test.d Breakpoint 1, D main () at test.d:10 10 () trusted { foo(); } (); (gdb) disassemble Dump of assembler code for function _Dmain: 0x000000000046d3f0 <+0>: push %rbp 0x000000000046d3f1 <+1>: mov %rsp,%rbp => 0x000000000046d3f4 <+4>: callq 0x46d400 <_D4test4mainFNfZ9__lambda1FNeZv> 0x000000000046d3f9 <+9>: xor %eax,%eax 0x000000000046d3fb <+11>: pop %rbp 0x000000000046d3fc <+12>: retq End of assembler dump.\ As you may see in generated call it actually calls lambda function (_D4test4mainFNfZ9__lambda1FNeZv) instead of calling _D4test3fooFZv directlyThis was exactly how all those trustedFoo() nested functions have appeared - we wanted to localized unsafe operations with `() trusted {}` but it wasn't properly inlined by DMD.DMD will inline trivial functions like that. In what case does it not?
Feb 09 2015
On 2/9/2015 11:27 PM, Dicebot wrote:As you may see in generated call it actually calls lambda function (_D4test4mainFNfZ9__lambda1FNeZv) instead of calling _D4test3fooFZv directlyhttps://issues.dlang.org/show_bug.cgi?id=14164
Feb 10 2015
On Tuesday, 10 February 2015 at 04:05:35 UTC, Andrei Alexandrescu wrote:On 2/9/15 8:03 PM, Zach the Mystic wrote:Maybe, allow trusted safe functions, which will 1) like safe, disallow direct use of unsafe operations 2) unlike safe, allow trusted blocks 3) mostly obey safe codegen requirements and be hopefully a little safer than plain trusted functions 4) be reviewed as trustedYou could put the 'trusted' template right in object.d, to save people the awkward burden of importing it from std.conv all the time. But that would be a language change, of sorts.We won't define it. Instead we'll go with Steve's idea: () trusted => expr. It's not much longer and it's a teensy bit more awkward - exactly what the doctor prescribed. -- Andrei
Feb 10 2015
On 2/9/15 6:21 PM, H. S. Teoh via Digitalmars-d wrote:I wonder if there's a way, in the current language, to make it illegal to call trusted() from a function explicitly marked safe...?No, but that might be worthy future consideration. I think we now should move forward with this and see how it fares. -- Andrei
Feb 09 2015