www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D Article: Memory Safety

reply Jakob Ovrum <jakobovrum gmail.com> writes:
The article aims to explain how to use  safe,  system and 
importantly,  trusted, including all the hairy details of 
templates.

https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

Any and all feedback appreciated.
Jan 20 2016
next sibling parent default0 <schneider.cedric gmx.de> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:

 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.


Nice article! Feeling like I have a much better grasp on the 
whole attribute system now that I read it, thanks! :-)
Jan 20 2016
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:

 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.


I like the description of  trusted and template inference. 
Template inference, in particular, was not something that was 
obvious to me when first reading about D. I'm not sure how clear 
you make it that you can still mark templates  safe and what have 
you (you seem to just say don't make templates  trusted).

I wasn't aware of the point that " trusted nested functions in 
templated functions do not have to have a memory safe interface 
as long as all calls to the function are memory safe". 
Interesting.
Jan 20 2016
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Wednesday, 20 January 2016 at 15:28:05 UTC, jmh530 wrote:

 I like the description of  trusted and template inference. 
 Template inference, in particular, was not something that was 
 obvious to me when first reading about D. I'm not sure how 
 clear you make it that you can still mark templates  safe and 
 what have you (you seem to just say don't make templates 
  trusted).


Templated functions can still be explicitly annotated with 
attributes, which disables inference for those attributes. This 
is often a good idea even for templated functions when template 
arguments do not inject code, so that every instantiation has the 
same, known set of attributes. Attribute inference can handle it, 
but explicit annotations provide documentation value. I might 
incorporate this into the article, but I'm wary of it losing 
focus.


 I wasn't aware of the point that " trusted nested functions in 
 templated functions do not have to have a memory safe interface 
 as long as all calls to the function are memory safe". 
 Interesting.


It is a necessary evil to propagate attributes correctly. Don't 
use it when you don't have to.
Jan 20 2016
parent "H. S. Teoh via Digitalmars-d-announce" writes:
On Thu, Jan 21, 2016 at 04:38:08AM +0000, Jakob Ovrum via
Digitalmars-d-announce wrote:

 On Wednesday, 20 January 2016 at 15:28:05 UTC, jmh530 wrote:

I like the description of  trusted and template inference. Template
inference, in particular, was not something that was obvious to me
when first reading about D. I'm not sure how clear you make it that
you can still mark templates  safe and what have you (you seem to
just say don't make templates  trusted).


 
 Templated functions can still be explicitly annotated with attributes,
 which disables inference for those attributes. This is often a good
 idea even for templated functions when template arguments do not
 inject code, so that every instantiation has the same, known set of
 attributes. Attribute inference can handle it, but explicit
 annotations provide documentation value. I might incorporate this into
 the article, but I'm wary of it losing focus.


A common idiom used in Phobos related to this is to use an attributed
(pure, nothrow,  nogc, etc.) unittest that instantiates the template in
question, to ensure that it does not accidentally become non-pure,
throwing, etc. (the unittest will stop compiling if so). This way, we
ensure that the template itself doesn't contain any impure / throwing /
 system code, even though you can instantiate it with template
parameters that may be impure, throwing,  system, etc..



I wasn't aware of the point that " trusted nested functions in
templated functions do not have to have a memory safe interface as
long as all calls to the function are memory safe". Interesting.


 
 It is a necessary evil to propagate attributes correctly. Don't use it
 when you don't have to.


Wouldn't it be better to refactor the code to separate out the part that
needs to be  trusted into a separate place, with a safe API? Or are
there cases for which this is impossible?


T

-- 
The early bird gets the worm. Moral: ewww...
Jan 20 2016
prev sibling next sibling parent reply Dicebot <public dicebot.lv> writes:
`auto p = ()  trusted { return &t; } ();`

Huh, I thought Andrei was opposed to this idiom? Is it now 
considered reserved for templates or something has changed?
Jan 20 2016
parent reply "H. S. Teoh via Digitalmars-d-announce" writes:
On Wed, Jan 20, 2016 at 07:25:43PM +0000, Dicebot via Digitalmars-d-announce
wrote:

 `auto p = ()  trusted { return &t; } ();`
 
 Huh, I thought Andrei was opposed to this idiom? Is it now considered
 reserved for templates or something has changed?


Yeah, I thought this was exactly the case where some of us Phobos
contributors got lambasted by Andrei and Walter for abusing  trusted.

The thing is, this is too easy to abuse, and too prone to careless
mistakes with nasty consequences. Suppose you have a template function:

	auto func(T)(T t)
	{
		auto p = ()  trusted { return &t; } ();
		...
		p.bar();
		...
	}

The problem is, the separation between p and p.bar() can be very large
in a complicated function, and during maintenance, somebody accidentally
introduces unsafe operations on p in the function body without realizing
that it came from a function marked  trusted. However, the compiler
won't catch this, because of the  trusted annotation.

Any exception to the strict usage of  trusted to me smells like a time
bomb waiting to explode. It may not be today or tomorrow, but sooner or
later somebody is going to slip up and the compiler won't help you. It's
bad enough that every single change to a  trusted function must be
vetted to ensure actual safety; now we have to also vet any modification
to any function that contains  trusted anonymous functions? In a large
template function, it's too easy to miss these  trusted sub-functions,
because if the code change is far away enough, the  trusted annotation
won't even show up in the diff. So reviewers may not even realize it's a
change that may have broken  trusted.

I'm pretty sure somebody brought up a case where such a hack was
actually necessary... but I'd still tread very, very carefully before
recommending such a thing to the general D coder.


T

-- 
The best compiler is between your ears. -- Michael Abrash
Jan 20 2016
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Wednesday, 20 January 2016 at 19:55:45 UTC, H. S. Teoh wrote:

 On Wed, Jan 20, 2016 at 07:25:43PM +0000, Dicebot via 
 Digitalmars-d-announce wrote:

 `auto p = ()  trusted { return &t; } ();`
 
 Huh, I thought Andrei was opposed to this idiom? Is it now 
 considered reserved for templates or something has changed?


 Yeah, I thought this was exactly the case where some of us 
 Phobos contributors got lambasted by Andrei and Walter for 
 abusing  trusted.


That was for non-templated functions where this approach makes no 
sense. Indeed it is counterproductive, because  trusted on the 
whole function is a better indication of what needs to be 
reviewed for memory safety (the whole function!).


 Any exception to the strict usage of  trusted to me smells like 
 a time bomb waiting to explode. It may not be today or 
 tomorrow, but sooner or later somebody is going to slip up and 
 the compiler won't help you. It's bad enough that every single 
 change to a  trusted function must be vetted to ensure actual 
 safety; now we have to also vet any modification to any 
 function that contains  trusted anonymous functions? In a large 
 template function, it's too easy to miss these  trusted 
 sub-functions, because if the code change is far away enough, 
 the  trusted annotation won't even show up in the diff. So 
 reviewers may not even realize it's a change that may have 
 broken  trusted.


It is the only way to solve this problem.
Jan 20 2016
parent reply Dicebot <public dicebot.lv> writes:
On Thursday, 21 January 2016 at 04:31:25 UTC, Jakob Ovrum wrote:

 That was for non-templated functions where this approach makes 
 no sense. Indeed it is counterproductive, because  trusted on 
 the whole function is a better indication of what needs to be 
 reviewed for memory safety (the whole function!).


Thanks! I got confused because your used example actually leaves 
 safe hole with this specific usage of  trusted :

void foo(T)(T t) {
     auto p = ()  trusted { return &t; } ();
     p.bar();
}

struct S { int x; }
S* global;

void bar (S* ptr)  safe
{
     global = ptr;
}

void main ()  safe
{
     foo(S.init);
     global.x = 42; // oops, writing to some random stack memory
}

I'd suggest at the very least to add a comment before "p.bar();" 
saying "Must not escape 'p' pointer or  safe-ty will be 
compromised".
Jan 21 2016
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 21 January 2016 at 13:39:48 UTC, Dicebot wrote:

 I'd suggest at the very least to add a comment before 
 "p.bar();" saying "Must not escape 'p' pointer or  safe-ty will 
 be compromised".


I thought about this case, but it relies on UFCS which is 
controlled by the callee. The caller can't inject that call if 
the callee is careful with its imports.

For member functions, the this reference is `ref` and its address 
cannot be taken in  safe code.
Jan 21 2016
parent reply Dicebot <public dicebot.lv> writes:
On Thursday, 21 January 2016 at 13:42:13 UTC, Jakob Ovrum wrote:

 On Thursday, 21 January 2016 at 13:39:48 UTC, Dicebot wrote:

 I'd suggest at the very least to add a comment before 
 "p.bar();" saying "Must not escape 'p' pointer or  safe-ty 
 will be compromised".


 I thought about this case, but it relies on UFCS which is 
 controlled by the callee. The caller can't inject that call if 
 the callee is careful with its imports.

 For member functions, the this reference is `ref` and its 
 address cannot be taken in  safe code.


Reasonable, but the UFCS call can result from some other function 
defined in same module (Phobos modules are not small at all). 
Even small unlikely violation can completely destroy benefits of 
 safe so in my opinion one can't be overly cautious when 
documenting stuff that requires verification.
Jan 21 2016
parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 21 January 2016 at 13:52:57 UTC, Dicebot wrote:

 Reasonable, but the UFCS call can result from some other 
 function defined in same module (Phobos modules are not small 
 at all). Even small unlikely violation can completely destroy 
 benefits of  safe so in my opinion one can't be overly cautious 
 when documenting stuff that requires verification.


I agree. Do you think it's worth mentioning UFCS functions in the 
article?
Jan 21 2016
prev sibling next sibling parent reply Jon D <jond noreply.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:

 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.


Nice article. I got a much better understanding reading it.

A small thing - I immediately tried adding  safe to one of my 
programs and hit an issue related to the first example in the 
article:

     void main()  safe {
         import std.stdio;
         writeln("hello, world");
     }

This is passes the  safe constraint, but 'stdout.writeln()' and 
'stderr.writeln()' do not. (My program uses stderr.) 
stderr/stdout/stdin are __gshared and can't be referenced by safe 
code. The module level version of writeln, etc., access a trusted 
version of stdout to avoid this.

I don't have a specific suggestion for addressing this in the 
article. It's nicely written and delving into this may take away 
from the key points. But, I wanted to point this out in case 
others hit it.

(Note: This related to a recent thread on the learn forum: 
http://forum.dlang.org/thread/vkihzrwomhiwdzqelrxa forum.dlang.org)
Jan 20 2016
parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Wednesday, 20 January 2016 at 20:28:03 UTC, Jon D wrote:

 This is passes the  safe constraint, but 'stdout.writeln()' and 
 'stderr.writeln()' do not. (My program uses stderr.) 
 stderr/stdout/stdin are __gshared and can't be referenced by 
 safe code. The module level version of writeln, etc., access a 
 trusted version of stdout to avoid this.


Yeah, the standard library still has a ways to go even with  safe.

I always imagined that the standard pipes should use shared as 
opposed to __gshared. I don't think the current implementation is 
thread-safe, but I don't know how this affects in memory safety 
in this case.
Jan 20 2016
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:

 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.


Good work. Someone has to re-edit-it if not yet reeddited.

Altgough one thing, attributes are not the easy part of D. I've 
recently encountered a case were in the library attributes were 
allright, test OK, and then suddently when I've started to use 
the library in a real life context I had to remove them from the 
library... safe was unsustainable.

Dealing with attributes is the hardest part of D IMO.
No one is forced to btw, there are plenty of other cool things in 
D but to follow the D safety is hard...

congrats nice article.
Jan 20 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Thursday, 21 January 2016 at 04:59:01 UTC, Basile B. wrote:

 On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum 
 wrote:

 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.


 Good work. Someone has to re-edit-it if not yet reeddited.

 Altgough one thing, attributes are not the easy part of D. I've 
 recently encountered a case were in the library attributes were 
 allright, test OK, and then suddently when I've started to use 
 the library in a real life context I had to remove them from 
 the library... safe was unsustainable.

 Dealing with attributes is the hardest part of D IMO.
 No one is forced to btw, there are plenty of other cool things 
 in D but to follow the D safety is hard...

 congrats nice article.


I mean ' safe' at too low level is a handicap. It's like 'const'. 
They are hard to use, mostly because of transitivness. These 
attributes are never a noop.
Jan 20 2016
parent "H. S. Teoh via Digitalmars-d-announce" writes:
On Thu, Jan 21, 2016 at 05:09:48AM +0000, Basile B. via Digitalmars-d-announce
wrote:
[...]

 I mean ' safe' at too low level is a handicap. It's like 'const'. They
 are hard to use, mostly because of transitivness. These attributes are
 never a noop.


Transitivity also makes const really painful to use in a widespread way.
I've tried writing const-correct code before too, but gave up because it
quickly became too unwieldy to work with. I started spending more time
hunting down missing const attributes than actually writing useful code,
so I decided it was time to give up.

Generally, though, const is still useful in lower-level code (i.e., near
the leaf nodes of your function call tree), to prevent silly mistakes.
Knowing how to use const is also helpful in utility functions that need
to accept both immutable and mutable, etc.. Just like with (the current
state of)  safe, though, pervasive use of const is still too onerous
currently. Transitivity really makes it painful, especially when
important chunks of Phobos still isn't fully const-correct (or at least
const-compatible) yet. A lot of progress has been made, but, const being
transitive, all it takes is for one small Phobos function to be
non-const when it should be const, and your entire call tree can no
longer be const. Encounter this a handful of times, and it's hard not to
just throw in the towel instead of spending all of your time working
around const issues rather than writing useful code.

(Recently I'm slowly moving towards writing *all* my code as template
functions, and letting the compiler do the tedious work of attributing
my code instead of typing them out myself. My secret wish is that one
day, the compiler's attribute inference will be good enough that I could
just slap one or two const's (or  safe, etc.) on top of my modules and
everything will Just Work.)


T

-- 
Only boring people get bored. -- JM
Jan 20 2016
prev sibling parent "H. S. Teoh via Digitalmars-d-announce" writes:
On Thu, Jan 21, 2016 at 04:59:01AM +0000, Basile B. via Digitalmars-d-announce
wrote:
[...]

 Altgough one thing, attributes are not the easy part of D. I've
 recently encountered a case were in the library attributes were
 allright, test OK, and then suddently when I've started to use the
 library in a real life context I had to remove them from the
 library... safe was unsustainable.


Phobos/druntime still has some ways to go before using it from  safe
code will be painless.  Some pretty fundamental functionality still
isn't  safe (mainly some stuff in object.di that basically interacts
with too many other things that marking one thing as  safe will
percolate throughout pretty much everything, breaking a whole bunch of
stuff at once).

I once tried writing a  safe program, and it didn't take very long
before I threw that idea out the window.  Once main() is  safe, you're
so straitjacketed that you basically can't write anything too much more
complex than Hello World.  (Well, you *could* just slap  trusted on
whatever it is that's holding you back, but then that breaks the promise
of  safe, which defeats the purpose of the entire exercise.)

There's also still a good number of  safe-related bugs on Bugzilla,
several of which involve built-in language constructs that break
 safe-ty outright. Things have improved a bit since I last checked, but
it seems to me that  safe is still not quite ready to live up to its
promise just yet. Maybe in a few more years' time...



 Dealing with attributes is the hardest part of D IMO.  No one is
 forced to btw, there are plenty of other cool things in D but to
 follow the D safety is hard...


[...]

I think Walter has mentioned before that attribute inference is the way
to go, and I agree. Once you start writing carefully-attributed code,
you'll quickly find that your declarations become painfully verbose,
which is never a good sign (it encourages people not to use attributes).
However, attribute inference on templates and auto functions (proposed
last year, don't know if it's implemented yet) alleviates a lot of the
verbosity. Hopefully the scope of attribute inference will increase
until it makes attribute use more widespread in your everyday D code.


T

-- 
MS Windows: 64-bit rehash of 32-bit extensions and a graphical shell for a
16-bit patch to an 8-bit operating system originally coded for a 4-bit
microprocessor, written by a 2-bit company that can't stand 1-bit of
competition.
Jan 20 2016
prev sibling next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:

 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.


my experience with  safe:

okay, I'll just use  safe here... and nothing else in third party 
libraries/half of phobos is  safe friendly so I guess I'll wrap 
it in  trusted oh fuck it
Jan 20 2016
parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 21 January 2016 at 06:20:01 UTC, rsw0x wrote:

 okay, I'll just use  safe here... and nothing else in third 
 party libraries/half of phobos is  safe friendly so I guess 
 I'll wrap it in  trusted oh fuck it


Yeah, using  trusted like that is counterproductive. Just use 
 system or improve the dependencies.
Jan 20 2016
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:

 The article aims to explain how to use  safe


Um, no, the article doesn't explain how to use  safe, it shows 
patterns that can be used to write safe code. The target audience 
must already understand safety.
Jan 21 2016
prev sibling next sibling parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:

 snip


Thanks for all the feedback. I've pushed a revision with further 
changes, most of it based on the feedback in this thread.

https://github.com/JakobOvrum/jakobovrum.github.io/commit/07c270567097f6cae5d9b95c88bd4d6c8124498c

(I'll try to remember not to force push over this commit and 
break the link, but if it is broken in the future, sorry, I 
probably slipped up and forgot.)
Jan 21 2016
prev sibling next sibling parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 20 January 2016 at 14:04:53 UTC, Jakob Ovrum wrote:

 The article aims to explain how to use  safe,  system and 
 importantly,  trusted, including all the hairy details of 
 templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.


Nicely written, enjoyed reading it.

And the visual presentation is excellent, I guess because it's 
based on github's code presentation libraries.  Really a nice 
example of how you can find blog posts online that are an order 
of magnitude better than any book you've ever read, particularly 
the visuals.

Is it on reddit?  The dlang.org redesign, Walter's talk, and a 
couple other D topics did well on there this week, I think this 
would also.
Jan 21 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/20/2016 09:04 AM, Jakob Ovrum wrote:

 The article aims to explain how to use  safe,  system and importantly,
  trusted, including all the hairy details of templates.

 https://jakobovrum.github.io/d/2016/01/20/memory-safety.html

 Any and all feedback appreciated.


Good work, thanks! Has this been reddited yet? -- Andrei
Jan 21 2016
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 21 January 2016 at 17:39:11 UTC, Andrei Alexandrescu 
wrote:

 Good work, thanks! Has this been reddited yet? -- Andrei


I don't think so. Personally I don't think I have a reddit 
account, but people are more than welcome to post it wherever 
they like :)
Jan 21 2016
parent reply Brad Anderson <eco gnuk.net> writes:
On Thursday, 21 January 2016 at 17:42:02 UTC, Jakob Ovrum wrote:

 On Thursday, 21 January 2016 at 17:39:11 UTC, Andrei 
 Alexandrescu wrote:

 Good work, thanks! Has this been reddited yet? -- Andrei


 I don't think so. Personally I don't think I have a reddit 
 account, but people are more than welcome to post it wherever 
 they like :)


Someone has submitted it (about a half hour ago): 
https://www.reddit.com/r/programming/comments/420yhi/memory_safety_in_d/
Jan 21 2016
parent Brad Anderson <eco gnuk.net> writes:
On Thursday, 21 January 2016 at 17:56:19 UTC, Brad Anderson wrote:

 On Thursday, 21 January 2016 at 17:42:02 UTC, Jakob Ovrum wrote:

 On Thursday, 21 January 2016 at 17:39:11 UTC, Andrei 
 Alexandrescu wrote:

 Good work, thanks! Has this been reddited yet? -- Andrei


 I don't think so. Personally I don't think I have a reddit 
 account, but people are more than welcome to post it wherever 
 they like :)


 Someone has submitted it (about a half hour ago): 
 https://www.reddit.com/r/programming/comments/420yhi/memory_safety_in_d/


I've submitted it to Hacker News.
Jan 21 2016