www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - An interesting consequence of safety requirements

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Something just dawned on me: in safe mode, struct static member 
functions will be preferred to struct non-static member functions.

Why?

Consider:

struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
}

This code can't make it in safe mode because it takes the address of 
this. In general, the compiler must assume that you might have created a 
List object on the stack, e.g.:

List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
}

Now even if there is no ostensible local address taking, the code is in 
error because it has escaped the address of a local.

So prepend() cannot be compiled. The way to make it compile in safe mode is:


struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
}

Now the code compiles and is actually safe because it is impossible to 
pass the address of a local into prepend.

So get ready to use static a lot more ;o).


Andrei
Nov 04 2009
next sibling parent reply Leandro Lucarella <llucax gmail.com> writes:
Andrei Alexandrescu, el  4 de noviembre a las 11:24 me escribiste:

 Something just dawned on me: in safe mode, struct static member
 functions will be preferred to struct non-static member functions.
 
 Why?
 
 Consider:
 
 struct List(T) {
     T payload;
     List * next;
 
     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }
 
 This code can't make it in safe mode because it takes the address of
 this. In general, the compiler must assume that you might have
 created a List object on the stack, e.g.:
 
 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }
 
 Now even if there is no ostensible local address taking, the code is
 in error because it has escaped the address of a local.
 
 So prepend() cannot be compiled. The way to make it compile in safe mode is:
 
 
 struct List(T) {
     T payload;
     List * next;
 
     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }
 
 Now the code compiles and is actually safe because it is impossible
 to pass the address of a local into prepend.
 
 So get ready to use static a lot more ;o).


Or maybe th compiler should rewrite local.prepend(lst); as:
auto this_ = &local;
f(*_this);

Then the compiler can detect when taking the address of local is not
legal as it will do if you write the function as static.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Sometimes I think the sure sign that life exists elsewhere in the universe
Is that that none of them tried to contact us
Nov 04 2009
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Leandro Lucarella wrote:

 Andrei Alexandrescu, el  4 de noviembre a las 11:24 me escribiste:

 Something just dawned on me: in safe mode, struct static member
 functions will be preferred to struct non-static member functions.

 Why?

 Consider:

 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

 This code can't make it in safe mode because it takes the address of
 this. In general, the compiler must assume that you might have
 created a List object on the stack, e.g.:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }

 Now even if there is no ostensible local address taking, the code is
 in error because it has escaped the address of a local.

 So prepend() cannot be compiled. The way to make it compile in safe mode is:


 struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }

 Now the code compiles and is actually safe because it is impossible
 to pass the address of a local into prepend.

 So get ready to use static a lot more ;o).


 
 Or maybe th compiler should rewrite local.prepend(lst); as:
 auto this_ = &local;
 f(*_this);
 
 Then the compiler can detect when taking the address of local is not
 legal as it will do if you write the function as static.
 


Yah, but how would the compiler decide (in a separate compilation 
approach) that some functions are actually fine?

// inside List(T)
T getPayload() {
     return payload;
}


Andrei
Nov 04 2009
prev sibling parent Leandro Lucarella <llucax gmail.com> writes:
Leandro Lucarella, el  4 de noviembre a las 16:06 me escribiste:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }
 So get ready to use static a lot more ;o).


 
 Or maybe th compiler should rewrite local.prepend(lst); as:
 auto this_ = &local;
 f(*_this);


Damn! I mean:
auto this_ = &local;
List.prepend(*_this, lst);

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
You should've seen her face. It was the exact same look my father gave me when
I told him I wanted to be a ventriloquist.
	-- George Constanza
Nov 04 2009
prev sibling next sibling parent reply grauzone <none example.net> writes:
Andrei Alexandrescu wrote:

 Something just dawned on me: in safe mode, struct static member 
 functions will be preferred to struct non-static member functions.
 
 Why?
 
 Consider:
 
 struct List(T) {
     T payload;
     List * next;
 
     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }
 
 This code can't make it in safe mode because it takes the address of 
 this. In general, the compiler must assume that you might have created a 
 List object on the stack, e.g.:
 
 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }
 
 Now even if there is no ostensible local address taking, the code is in 
 error because it has escaped the address of a local.
 
 So prepend() cannot be compiled. The way to make it compile in safe mode 
 is:
 
 
 struct List(T) {
     T payload;
     List * next;
 
     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }
 
 Now the code compiles and is actually safe because it is impossible to 
 pass the address of a local into prepend.


Maybe I don't get it, but how to you call prepend() from safe code?

Also, does anybody really care about SafeD, or would it be better if we 
had some sort of valgrind for D? Maybe this is one of those features 
which first sounded nice, but then it turned out it's better to drop them.


 So get ready to use static a lot more ;o).
 
 
 Andrei
Nov 04 2009
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
grauzone wrote:

 Andrei Alexandrescu wrote:

 Something just dawned on me: in safe mode, struct static member 
 functions will be preferred to struct non-static member functions.

 Why?

 Consider:

 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

 This code can't make it in safe mode because it takes the address of 
 this. In general, the compiler must assume that you might have created 
 a List object on the stack, e.g.:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }

 Now even if there is no ostensible local address taking, the code is 
 in error because it has escaped the address of a local.

 So prepend() cannot be compiled. The way to make it compile in safe 
 mode is:


 struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }

 Now the code compiles and is actually safe because it is impossible to 
 pass the address of a local into prepend.


 
 Maybe I don't get it, but how to you call prepend() from safe code?


module(safe) wyda;

void main() {
     auto lst1 = new List;
     auto lst2 = new List;
     List.prepend(lst1, lst2)
}


 Also, does anybody really care about SafeD, or would it be better if we 
 had some sort of valgrind for D? Maybe this is one of those features 
 which first sounded nice, but then it turned out it's better to drop them.


I'm not sure, but clearly soundness is an important concern in 
contemporary languages.


Andrei
Nov 04 2009
next sibling parent David Gileadi <foo bar.com> writes:
Andrei Alexandrescu wrote:

 grauzone wrote:

 Maybe I don't get it, but how to you call prepend() from safe code?


 
 module(safe) wyda;
 
 void main() {
     auto lst1 = new List;
     auto lst2 = new List;
     List.prepend(lst1, lst2)
 }


This seems to be another case where automatic function rewriting would 
be nice, a la D's "Functions as Array Properties":

void main() {
     auto lst1 = new List;
     auto lst2 = new List;
     // rewritten by compiler to List.prepend(lst1, lst2):
     lst1.prepend(lst2);
}
Nov 04 2009
prev sibling parent reply grauzone <none example.net> writes:
Andrei Alexandrescu wrote:

 grauzone wrote:

 Andrei Alexandrescu wrote:

 Something just dawned on me: in safe mode, struct static member 
 functions will be preferred to struct non-static member functions.

 Why?

 Consider:

 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

 This code can't make it in safe mode because it takes the address of 
 this. In general, the compiler must assume that you might have 
 created a List object on the stack, e.g.:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }

 Now even if there is no ostensible local address taking, the code is 
 in error because it has escaped the address of a local.

 So prepend() cannot be compiled. The way to make it compile in safe 
 mode is:


 struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }

 Now the code compiles and is actually safe because it is impossible 
 to pass the address of a local into prepend.


 Maybe I don't get it, but how to you call prepend() from safe code?


 
 module(safe) wyda;
 
 void main() {
     auto lst1 = new List;
     auto lst2 = new List;
     List.prepend(lst1, lst2)
 }


I see... in this case, I'd say the user should be forced to allocate 
List on the heap by making List a class. Of course, you could say that 
it should be useable in other, unsafe, contexts too (maybe you want to 
construct a list of the stack); but then you could use... InSitu!(T), or 
what was it called?


 Also, does anybody really care about SafeD, or would it be better if 
 we had some sort of valgrind for D? Maybe this is one of those 
 features which first sounded nice, but then it turned out it's better 
 to drop them.


 
 I'm not sure, but clearly soundness is an important concern in 
 contemporary languages.
 
 
 Andrei
Nov 04 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
grauzone wrote:

 Andrei Alexandrescu wrote:

 grauzone wrote:

 Andrei Alexandrescu wrote:

 Something just dawned on me: in safe mode, struct static member 
 functions will be preferred to struct non-static member functions.

 Why?

 Consider:

 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }

 This code can't make it in safe mode because it takes the address of 
 this. In general, the compiler must assume that you might have 
 created a List object on the stack, e.g.:

 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }

 Now even if there is no ostensible local address taking, the code is 
 in error because it has escaped the address of a local.

 So prepend() cannot be compiled. The way to make it compile in safe 
 mode is:


 struct List(T) {
     T payload;
     List * next;

     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }

 Now the code compiles and is actually safe because it is impossible 
 to pass the address of a local into prepend.


 Maybe I don't get it, but how to you call prepend() from safe code?


 module(safe) wyda;

 void main() {
     auto lst1 = new List;
     auto lst2 = new List;
     List.prepend(lst1, lst2)
 }


 
 I see... in this case, I'd say the user should be forced to allocate 
 List on the heap by making List a class.


Well we'd still like to benefit from structs and pointers in safe mode. 
The point is to tighten the screws just as much as needed to achieve 
interesting properties, but not (much) tighter.


Andrei
Nov 04 2009
prev sibling next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from grauzone (none example.net)'s article

 Andrei Alexandrescu wrote:
 Also, does anybody really care about SafeD, or would it be better if we
 had some sort of valgrind for D? Maybe this is one of those features
 which first sounded nice, but then it turned out it's better to drop them.


I'm starting to get that feeling.  It seems like D was just not meant to be a
fully safe language.  It was built from the ground up to be a better C++, i.e. a
close to the metal language that assumes the programmer knows what he/she is
doing, albeit one with enough high level features that it doesn't require
attention to the irrelevant in day-to-day programming.  Shoe-horning D
(originally
designed as close-to-the-metal/programmer-knows-best) into a safe Java-like
language is going to work about as well as shoe-horning C (originally a very
low-level portable assembler) into C++ (a language that tries to be high-level
but
is too hobbled by C compatibility to actually achieve it).

Example (Simple program that has no chance of working in SafeD):

import std.getopt;

void main(string[] args) {
    uint foo;
    getopt(args, "someParam", &foo);  // Not allowed in SafeD.
}

Personally, if an example like that won't even compile, I refuse to ever use
SafeD
for any project I do because it is simply too rigid.  On the other hand, if it
does compile, SafeD provides no guarantees and is next to useless.
Nov 04 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
dsimcha wrote:

 == Quote from grauzone (none example.net)'s article

 Andrei Alexandrescu wrote:
 Also, does anybody really care about SafeD, or would it be better if we
 had some sort of valgrind for D? Maybe this is one of those features
 which first sounded nice, but then it turned out it's better to drop them.


 
 I'm starting to get that feeling.  It seems like D was just not meant to be a
 fully safe language.  It was built from the ground up to be a better C++, i.e.
a
 close to the metal language that assumes the programmer knows what he/she is
 doing, albeit one with enough high level features that it doesn't require
 attention to the irrelevant in day-to-day programming.  Shoe-horning D
(originally
 designed as close-to-the-metal/programmer-knows-best) into a safe Java-like
 language is going to work about as well as shoe-horning C (originally a very
 low-level portable assembler) into C++ (a language that tries to be high-level
but
 is too hobbled by C compatibility to actually achieve it).
 
 Example (Simple program that has no chance of working in SafeD):
 
 import std.getopt;
 
 void main(string[] args) {
     uint foo;
     getopt(args, "someParam", &foo);  // Not allowed in SafeD.
 }
 
 Personally, if an example like that won't even compile, I refuse to ever use
SafeD
 for any project I do because it is simply too rigid.  On the other hand, if it
 does compile, SafeD provides no guarantees and is next to useless.


getopt takes pointers just because variadic references aren't available. 
It's not a matter of principles. I agree that a useful module such as 
getopt should work in safe mode.

Andrei
Nov 04 2009
prev sibling next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
grauzone wrote:

 Also, does anybody really care about SafeD, or would it be better if we 
 had some sort of valgrind for D? Maybe this is one of those features 
 which first sounded nice, but then it turned out it's better to drop them.


valgrind is a runtime thing, so it depends on having a reasonably 
complete test suite, and cannot say anything about code that is not 
executed by the test suite. Even if the test suite has 100% coverage of 
all code paths, that is still no guarantee of not having undefined behavior.

SafeD offers a compile time guarantee.
Nov 04 2009
prev sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2009-11-04 14:15:47 -0500, grauzone <none example.net> said:


 Also, does anybody really care about SafeD, or would it be better if we 
 had some sort of valgrind for D? Maybe this is one of those features 
 which first sounded nice, but then it turned out it's better to drop 
 them.


I'm interested in SafeD a lot since it guards against buffer overruns 
and memory corruption errors, which represents a big slice of the most 
dangerous security risks.

Sure it comes with small performance drawbacks (array bound checks, 
forced dynamic allocation in some cases). But that shouldn't matter as 
you can move performance-critical code to unsafe/trusted modules as an 
optimization (hopefully with more security checkups on these), or just 
disable SafeD altogether if that really makes a difference.

But most of my code isn't performance critical and thus most of my code 
should be in SafeD.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Nov 04 2009
prev sibling parent reply Chad J <chadjoan __spam.is.bad__gmail.com> writes:
Andrei Alexandrescu wrote:

 Something just dawned on me: in safe mode, struct static member
 functions will be preferred to struct non-static member functions.
 
 Why?
 
 Consider:
 
 struct List(T) {
     T payload;
     List * next;
 
     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }
 
 This code can't make it in safe mode because it takes the address of
 this. In general, the compiler must assume that you might have created a
 List object on the stack, e.g.:
 
 List * someFun() {
     List local;
     List * lst = new List;
     local.prepend(lst);
     return lst;
 }
 
 Now even if there is no ostensible local address taking, the code is in
 error because it has escaped the address of a local.
 
 So prepend() cannot be compiled. The way to make it compile in safe mode
 is:
 
 
 struct List(T) {
     T payload;
     List * next;
 
     static void prepend(List * zis, List * newNode) {
         newNode.next = zis;
     }
 }
 
 Now the code compiles and is actually safe because it is impossible to
 pass the address of a local into prepend.
 
 So get ready to use static a lot more ;o).
 
 
 Andrei


This looks to me like it implicates ref parameters in general.


 struct List(T) {
     T payload;
     List * next;

     void prepend(List * newNode) {
         newNode.next = &this;
     }
 }


could be written as

struct List(T) {
    T payload;
    List * next;

    static void prepend(ref List zis, List * newNode) {
        newNode.next = &zis;
    }
}

It's not the same as the other static version, since zis is a ref and
not a pointer.

List * someFun() {
    List local;
    List * lst = new List;
    prepend(local,lst);    // <-- dubious; local is trying to hide
    return lst;            // <-- he got away
}

I think in the general case there might need to be some way to track
functions that have ref parameters.

Perhaps any ref parameter that has its address taken is marked as being
unable to accept local variables as arguments.  Additionally, it should
also not accept a parent function's ref parameters as arguments, since
those could be local to someone else.  Then if you rewrite member
functions as static versions with ref parameters as the zeroth arg, they
 will benefit from the analysis as well.

Maybe this is too complicated.  If not, hope it helps.
Nov 04 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Chad J wrote:

 Andrei Alexandrescu wrote:


[snip]

 So get ready to use static a lot more ;o).


 Andrei


 
 This looks to me like it implicates ref parameters in general.


Yah, and more precisely: ref parameters that you plan to take the 
address of.

It all holds water: if you take a pointer parameter in SafeD, you know 
for sure it's dynamically-allocated because the caller could not have 
taken the address of a stack variable.

[snip]

 Perhaps any ref parameter that has its address taken is marked as being
 unable to accept local variables as arguments.  Additionally, it should
 also not accept a parent function's ref parameters as arguments, since
 those could be local to someone else.  Then if you rewrite member
 functions as static versions with ref parameters as the zeroth arg, they
  will benefit from the analysis as well.
 
 Maybe this is too complicated.  If not, hope it helps.


Thanks. It's a reenactment of a discussion that Bartosz, Walter and I 
have had a few times: should the compiler collect the so-called 
"function summaries" during compilation and augment the signatures with 
additional properties, or should we require the user to annotate the 
signatures themselves?

Collecting function summaries is a classic in many program analysis, but 
is difficult to scale and to combine with separate compilation (usually 
interesting summaries require collecting info about all functions and 
then doing a sort of fixed point iteration).

So far dmd never relies on collecting a function summary, but that may 
change in the future.


Andrei
Nov 05 2009
parent Brad Roberts <braddr bellevue.puremagic.com> writes:
On Thu, 5 Nov 2009, Andrei Alexandrescu wrote:


 Thanks. It's a reenactment of a discussion that Bartosz, Walter and I have had
 a few times: should the compiler collect the so-called "function summaries"
 during compilation and augment the signatures with additional properties, or
 should we require the user to annotate the signatures themselves?
 
 Collecting function summaries is a classic in many program analysis, but is
 difficult to scale and to combine with separate compilation (usually
 interesting summaries require collecting info about all functions and then
 doing a sort of fixed point iteration).
 
 So far dmd never relies on collecting a function summary, but that may change
 in the future.
 
 
 Andrei


The key difference for me has always been that user specified traits (be 
they annotations, keywords, whatever) is that it's defining the intended 
contract.  Anything inferred is useful only so far in that it might allow 
usage where not explicitly intended.

For example, imagine a function declared to take a function pointer for 
which the function must be pure.  That explicitly allows any annotated 
function to be used (that matches the rest of the signature, obviously), 
but might also allow the use of any detected pure function.

The risk being that the detected as pure function might change at some 
future date -- since it's signature didn't require enforcement of that 
purity -- and break the call site.

From what I can see, there's three alternatives:

  1) strict policy -- no inference done at all

  2) strict policy -- inference allowed to influence optimization, but not 
change semantics

  3) inference allowed at the semantic layer, potentially allowing future 
issues.



Later,
Brad
Nov 05 2009