www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Coverity tool

reply bearophile <bearophileHUGS lycos.com> writes:
Rarely I have found a so fun description of a software, how they can (barely?)
make it a commercial product, the problems they find. The program is a bug

http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext

I think it's an interesting read for people that are creating a new language :-)

Bye,
bearophile
Feb 08 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
bearophile wrote:

 Rarely I have found a so fun description of a software, how they can
 (barely?) make it a commercial product, the problems they find. The

 http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext
 
 
 I think it's an interesting read for people that are creating a new
 language :-)



Coverity's business model seems to be based on being very vague about 
what their software actually does, and being even more evasive about 
what they charge to use it.

 From what I can infer from their various statements, the primary thing 
it does is pair functions that must be paired, like malloc/free, 
lock/unlock, fopen/fclose, etc.

This is ably handled in D using the scope guard statement.

It does some other things like very limited array bounds checking, also 
already handled by D which does full array bounds checking.
Feb 09 2010
next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Walter Bright wrote:

 Coverity's business model seems to be based on being very vague about 
 what their software actually does,



Found some stuff on the Cert web site:

https://www.securecoding.cert.org/confluence/display/seccode/Coverity+Prevent
Feb 09 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:

  From what I can infer from their various statements, the primary thing 
 it does is pair functions that must be paired, like malloc/free, 
 lock/unlock, fopen/fclose, etc.


I have never used their program, and probably I will never buy it, but I have
seen it used two times on the whole source code of CPython, and it has found a
good amount of bugs (that devs have fixed in some weeks) despite such C sources
are sometimes quite good (see for example:
http://svn.python.org/view/python/trunk/Modules/_collections
odule.c?view=markup ) compared to the messy C sources of Perl or Ruby. So it's
an useful tool. Using D2 compiled with DMD is not an option for CPython devs,
they can't even use C99.

Bye,
bearophile
Feb 09 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
bearophile wrote:

 Walter Bright:

 From what I can infer from their various statements, the primary
 thing it does is pair functions that must be paired, like
 malloc/free, lock/unlock, fopen/fclose, etc.


 
 I have never used their program, and probably I will never buy it,
 but I have seen it used two times on the whole source code of
 CPython, and it has found a good amount of bugs (that devs have fixed
 in some weeks) despite such C sources are sometimes quite good (see
 for example:
 http://svn.python.org/view/python/trunk/Modules/_collectionsmodule.c?view=markup
 ) compared to the messy C sources of Perl or Ruby. So it's an useful
 tool. Using D2 compiled with DMD is not an option for CPython devs,
 they can't even use C99.



I'm not disagreeing with you, I just think that Coverity doesn't have 
much use for D code, because what it checks for is already covered by 
the language.
Feb 09 2010
next sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 02/09/2010 02:18 PM, Walter Bright wrote:

 bearophile wrote:

 Walter Bright:

 From what I can infer from their various statements, the primary
 thing it does is pair functions that must be paired, like
 malloc/free, lock/unlock, fopen/fclose, etc.


 I have never used their program, and probably I will never buy it,
 but I have seen it used two times on the whole source code of
 CPython, and it has found a good amount of bugs (that devs have fixed
 in some weeks) despite such C sources are sometimes quite good (see
 for example:
 http://svn.python.org/view/python/trunk/Modules/_collectionsmodule.c?view=markup

 ) compared to the messy C sources of Perl or Ruby. So it's an useful
 tool. Using D2 compiled with DMD is not an option for CPython devs,
 they can't even use C99.



 I'm not disagreeing with you, I just think that Coverity doesn't have
 much use for D code, because what it checks for is already covered by
 the language.


I think a large part of the article proves your point: all those checks 
belong in the compiler, and not in a third party tool.

Still, there are lots of trivial or nontrivial or fuzzy or advanced 
analyses that DMD doesn't perform. A dumb example would be

version(1){
     pragma(msg, "1");
}else version(2){
     pragma(msg, "2");
}
Feb 09 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:


I just think that Coverity doesn't have much use for D code, because what it
checks for is already covered by the language.<


I have taken a look at this report about bugs found by this tool in open source
code:
http://scan.coverity.com/report/Coverity_White_Paper-Scan_Open_Source_Report_2009.pdf

The most common bugs found there by Coverity (I think they are mostly in C
code):

27.8% NULL Pointer Deference
23.3% Resource Leak
 9.7% Unintentional Ignored Expressions
 8.1% Use Before Test (NULL)
 5.9% Use After Free
 5.8% Buffer Overflow (statically allocated)
 5.3% Unsafe Use of Returned NULL
 8.4% Uninitialized Values Read
 3.9% Unsafe Use of Returned Negative
 1.1% Type and Allocation Size Mismatch
 0.2% Buffer Overflow (dynamically allocated)
 0.2% Use Before Test (negative)

Key:
- Unintentional Ignored Expressions: indicates that part of the source code is
not reachable on any execution path. One example involves if statements and
return statements. Consider when a programmer makes a mistake in the if()
statement’s test value, and the code block for the statement returns from the
current function. If the test value is always true, then all the code after the
if() block can never be reached.
- Unsafe Use of Returned values: results of a function are not tested if they
are NULL or if they are a error value (usually negative).
- Use Before Test: similar to Unsafe Use of Returned values. When programmers
test the value of a variable, this implies that they believe it is possible for
the variable to have the undesired value at the point of the test. If a code
path exists where the variable is used in an unsafe way before this is tested,
then the test does not accomplish its objective.


Regarding D code:

- Use After Free: probably less common in D code, because dellocations are done
by the GC.
- Buffer Overflows: less common in D code because of array bound tests at
runtime, and because some string operations/functions are safer.
- Type and Allocation Size Mismatch: malloc/calloc are not common in D code,
and the usage of the new statement avoids many bugs of this class.
- Unsafe Use of Returned error values: hopefully in all the code (but where
high-performance code is necessary) D programmers use exceptions to denote
error situations, avoiding some of this class of bugs.
- Resource Leak: in theory the GC can help a lot here. In practice the D GC is
conservative, so it leaks by design.
- Uninitialized Values Read: D "solves" this problem initializing by default
all variables. In practice probably some default-initialized values can cause
troubles where a default initialization was not the right thing.
- NULL Pointer Deference: this probably happens in D code, by default class
references can be null.

You can probably find the following bugs in D code:
- Use Before Test: here the bugs are in the code paths of the try-except.
- Unsafe Use of Returned NULL: the D2 type system doesn't help here.
- Unintentional Ignored Expressions


D language surely introduces new classes of bugs absent in C code. D2 has
several complex and hairy features, that can lead to bugs.

------------

Ellery Newcomer:
 	

I think a large part of the article proves your point: all those checks belong
in the compiler, and not in a third party tool.<


In theory you are right, but in practice compiler writers have already their
hands full, so I have nothing against the idea of a lint tool that does more
tests compared to the compiler.

The new Clang/Clang++ compiler (front-end based on LLVM. Few days ago Clang++
has compiled all llvm itself) is able to perform a good amount of (simple but
useful) static tests on the code, and it gives precise error messages.
Being the LLVM project very modular, with some luck a future LDC-like compiler
for D2 will find a way to re-use some of the analysis modules used by Clang on
D2 code, so maybe it can list unused variables, spot some cases of null
reference deferences, etc.

Bye,
bearophile
Feb 09 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
bearophile wrote:

 Regarding D code:
 
 - Use After Free: probably less common in D code, because
 dellocations are done by the GC.
 - Buffer Overflows: less common in D
 code because of array bound tests at runtime, and because some string
 operations/functions are safer.


 safe guarantees memory safety, so these are non-issues in D.



 - Type and Allocation Size Mismatch:
 malloc/calloc are not common in D code, and the usage of the new
 statement avoids many bugs of this class.


Using  safe makes these non-issues.



 - Unsafe Use of Returned
 error values: hopefully in all the code (but where high-performance
 code is necessary) D programmers use exceptions to denote error
 situations, avoiding some of this class of bugs.


As you say, exceptions solve these problems.



 - Resource Leak: in
 theory the GC can help a lot here. In practice the D GC is
 conservative, so it leaks by design.


It is a mistake to rely on the GC to free resources other than memory. 
Properly, either RAII or scope guard statements should be used, as these 
do not leak.



 - Uninitialized Values Read: D
 "solves" this problem initializing by default all variables. In
 practice probably some default-initialized values can cause troubles
 where a default initialization was not the right thing.


No analysis tool can help you if you initialize things to the wrong 
value. That's very, very different from initializing things to GARBAGE 
and then using those garbage values.



 - NULL Pointer Deference: this probably happens in D code, by default class
 references can be null.


We've certainly had long threads about this one. I'll just state that 
turning on the optimizer will detect many cases of dereferencing null 
pointers.



 You can probably find the following bugs in D code:



 - Use Before Test: here the bugs are in the code paths of the try-except.


Not sure what this means.



 - Unsafe Use of Returned NULL: the D2 type system doesn't help here.


Not sure what this means.



 - Unintentional Ignored Expressions


dmd already flags dead code.
Feb 09 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:


 safe guarantees memory safety, so these are non-issues in D.<



level features, because if they don't need that extra little percentage of

many more tools, software, cheap developers, etc. So D programs probably will
contain some unsafe code. Of course in a D project a smart programmer tries to
minimize the amount of unsafe lines of code.
So a lint tool (or a better compiler) to spot those bugs in unsafe D code can
be useful.



As you say, exceptions solve these problems.<


But there are many C++ programmers that don't use exceptions, for example they
are not allowed in Google C++ code:
http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Exceptions
So surely not 100% of D code will use exceptions. That's why I have said that D
avoids some of those problems but not all them :-)




- Resource Leak: in theory the GC can help a lot here. In practice the D GC is
conservative, so it leaks by design.<<





It is a mistake to rely on the GC to free resources other than memory.
Properly, either RAII or scope guard statements should be used, as these do not
leak.<


I was saying that by design a conservative GC can leak memory, and I have seen
that it actually does it if the programs allocates lot of memory.




- Uninitialized Values Read: D "solves" this problem initializing by default
all variables. In practice probably some default-initialized values can cause
troubles where a default initialization was not the right thing.<<





No analysis tool can help you if you initialize things to the wrong value.
That's very, very different from initializing things to GARBAGE and then using
those garbage values.<


What I meant is that I really like when the Java compiler tells me that a
variable is not initialized :-)




I'll just state that turning on the optimizer will detect many cases of
dereferencing null pointers.<


Very good, and future D compilers can improve some more on this :-)




- Unsafe Use of Returned NULL: the D2 type system doesn't help here.<<


Not sure what this means.<


I meant that in D2 the type system doesn't support nonnullable references yet.
Not-nullable references/pointers can't be null, so there's no need for the
programmer to remember to test if they are null, so I think they can avoid some
bugs caused by untested nulls. In practice I have never programmed with a
language with this feature so I don't know what side effects it can have.




- Use Before Test: here the bugs are in the code paths of the try-except.<<


Not sure what this means.<


Thinking more about it, I think I was probably wrong.

Thank you for your answers, as usual I am often wrong and you are usually
right; I am not even remotely as good as you, but I try to help :-)

Bye,
bearophile
Feb 09 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
bearophile wrote:

 But there are many C++ programmers that don't use exceptions, for
 example they are not allowed in Google C++ code: 
 http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Exceptions
  So surely not 100% of D code will use exceptions. That's why I have
 said that D avoids some of those problems but not all them :-)


I don't know why Google proscribes exceptions, but exceptions still have 
an uneasy relationship with C++. Older compilers may not support them 
properly, much C++ code uses them incorrectly and inconsistently, there 
isn't good support for exception safety or transaction safety, being 
able to throw value types is a big mistake, etc.
Feb 09 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:

 I don't know why Google proscribes exceptions,


If you look at that link, and you click on the arrow, you will see a nice list
of pro-cons :-)

Bye,
bearophile
Feb 09 2010
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-09 19:26:05 -0500, bearophile <bearophileHUGS lycos.com> said:


 Walter Bright:

 I don't know why Google proscribes exceptions,


 
 If you look at that link, and you click on the arrow, you will see a 
 nice list of pro-cons :-)


And below the pro-cons it says they avoid exceptions because they have 
a lot of exception-unsafe code that would break if you were to throw 
exceptions, suggesting it might be otherwise if it wasn't for that 
legacy code.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 09 2010
parent "Nick Sabalausky" <a a.a> writes:
"Michel Fortin" <michel.fortin michelf.com> wrote in message 
news:hkt3gq$19pp$1 digitalmars.com...

 On 2010-02-09 19:26:05 -0500, bearophile <bearophileHUGS lycos.com> said:


 Walter Bright:

 I don't know why Google proscribes exceptions,


 If you look at that link, and you click on the arrow, you will see a nice 
 list of pro-cons :-)


 And below the pro-cons it says they avoid exceptions because they have a 
 lot of exception-unsafe code that would break if you were to throw 
 exceptions, suggesting it might be otherwise if it wasn't for that legacy 
 code.


It doesn't just suggest it, it flat-out states it:
"Things would probably be different if we had to do it all over again from 
scratch."
Feb 10 2010
prev sibling parent Justin Johansson <no spam.com> writes:
Walter Bright wrote:

 bearophile wrote:

 But there are many C++ programmers that don't use exceptions, for
 example they are not allowed in Google C++ code: 
 http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Exceptions
  So surely not 100% of D code will use exceptions. That's why I have
 said that D avoids some of those problems but not all them :-)


 
 I don't know why Google proscribes exceptions, but exceptions still have 
 an uneasy relationship with C++. Older compilers may not support them 
 properly, much C++ code uses them incorrectly and inconsistently, there 
 isn't good support for exception safety or transaction safety, being 
 able to throw value types is a big mistake, etc.


If C++ exceptions are thrown and caught within, and are completely self-
-contained within, a (C++) library then nobody should be the wiser.

The only possible issue can be about whether the library should
be allowed to propagate exceptions across the API boundary and
into client code.

The pro and con arguments should take the client code interaction /
API context into consideration.  Otherwise it's just a blatant,
ill-considered consideration (ill-pun considered as well).

Cheers

Justin Johansson
Feb 10 2010
prev sibling next sibling parent reply sybrandy <sybrandy gmail.com> writes:
 - Use Before Test: here the bugs are in the code paths of the try-except.


 Not sure what this means.


If I remember correctly from when I had an opportunity to look at these 
types of tools, this is a case where, in C or C++, you have a variable 
that may be uninitialized or set to NULL, but you don't test for it 
until you have tried to use it.  I believe the root cause of this bug 
has nothing to do with the language, but with people going back into 
code they don't fully understand and trying to either A) use the 
variable and not know that the check is in there or B) they put the 
check in to "fix" a bug, but didn't put it in the correct spot.

Needless to say, with D setting variables to reasonable defaults, I 
believe this is mitigated, but not entirely prevented.
Feb 09 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
sybrandy wrote:

 - Use Before Test: here the bugs are in the code paths of the 
 try-except.


 Not sure what this means.


 
 If I remember correctly from when I had an opportunity to look at these 
 types of tools, this is a case where, in C or C++, you have a variable 
 that may be uninitialized or set to NULL, but you don't test for it 
 until you have tried to use it.  I believe the root cause of this bug 
 has nothing to do with the language, but with people going back into 
 code they don't fully understand and trying to either A) use the 
 variable and not know that the check is in there or B) they put the 
 check in to "fix" a bug, but didn't put it in the correct spot.


That makes sense.



 Needless to say, with D setting variables to reasonable defaults, I 
 believe this is mitigated, but not entirely prevented.


You're right.
Feb 09 2010
prev sibling parent reply Brad Roberts <braddr bellevue.puremagic.com> writes:
On Tue, 9 Feb 2010, Walter Bright wrote:


  safe guarantees memory safety, so these are non-issues in D.
 
 Using  safe makes these non-issues.
 
 As you say, exceptions solve these problems.
 
 It is a mistake to rely on the GC to free resources other than memory.
 Properly, either RAII or scope guard statements should be used, as these do
 not leak.
 
 No analysis tool can help you if you initialize things to the wrong value.
 That's very, very different from initializing things to GARBAGE and then using
 those garbage values.
 
 We've certainly had long threads about this one. I'll just state that turning
 on the optimizer will detect many cases of dereferencing null pointers.
 
 dmd already flags dead code.


Nice black and white world you are acting like you live in there Walter.  
I know you know better, so why do you pretend otherwise?

You're assuming perfect use of the language and largely simple code.  
Granted, there's a lot of things that D does better, and the tools in the 
language make it easier to write better code... but developers are corner 
cutters at heart.  Additionally, not everything can be done in  safe (or 
it wouldn't be an optional mode), and not everything can use RAII for 
resource management.

Coverity, as has been discussed here in the past, is a very powerful tool 
that solves very real world problems... at an astronomical price.  A 
sort-of close cousin to Coverity is Fortify.  Fortify has a security slant 
to it, but is solving essentially the same class of problems.

Compilers do, in concept, a lot of the things that Coverity does, but at a 
very different scale.  Coverty is aimed at whole application long range 
effect analysis.  It's got good tools for over-time analysis of builds to 
help ensure incremental improvements (ie, prevent quality slipping 
backwards).  All in all, it's well engineered for large systems and large 
teams.

I wish I had access to it for everthing I develop.  Sadly, the cost is too 
high.

Later,
Brad
Feb 09 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Brad Roberts wrote:

 Nice black and white world you are acting like you live in there Walter.  
 I know you know better, so why do you pretend otherwise?


First off, yes, if you program in C style, despite using D, your code is 
vulnerable to many of the problems that Coverity detects.


 You're assuming perfect use of the language and largely simple code.  
 Granted, there's a lot of things that D does better, and the tools in the 
 language make it easier to write better code... but developers are corner 
 cutters at heart.  Additionally, not everything can be done in  safe (or 
 it wouldn't be an optional mode), and not everything can use RAII for 
 resource management.


Yes, I am relying on convention to a degree for D to be helpful, and 
I've often enough railed against relying on convention. So why the 
contradiction?

Let's take the lock()/unlock() pairing problem. In C, you often wind up 
with a rat's nest of control flow that gets really hard to verify has 
the unlock() along all paths. Coverity is a win here. In C++, you can 
wrap the lock()/unlock() in a struct and use RAII, but that's as ugly as 
sin and many will eschew it, thus Coverity wins again. But in D,

     lock();
     scope(exit) unlock();

and you're done. There's nothing left to analyze. Nothing makes you 
write D code that way, but I contend that it is easy to write it that 
way (which addresses the corner cutters), and it is easy for a code 
review to verify it. No tangle of logic must be traced. So the problem 
is reduced from a 10 to a 1. It doesn't go away, but the payback of 
Coverity for these cases is greatly reduced.

(And the cost of Coverity isn't just the purchase price, it has 
significant administration and false positive costs.)

About  safe: a lot of Coverity is focussed on memory safety, for example 
it spends a lot of effort analyzing use of C string functions. With 
 safe, the D compiler guarantees your code is memory safe. No false 
positives. The idea is to reduce the  system parts of your code to as 
small a fraction as possible. Then, you aren't running a million lines 
of code through Coverity to check for memory safety, you're running only 
a few percent of it, and those few percent will be a lot easier to review.



 Compilers do, in concept, a lot of the things that Coverity does, but at a 
 very different scale.  Coverty is aimed at whole application long range 
 effect analysis.  It's got good tools for over-time analysis of builds to 
 help ensure incremental improvements (ie, prevent quality slipping 
 backwards).  All in all, it's well engineered for large systems and large 
 teams.
 
 I wish I had access to it for everthing I develop.  Sadly, the cost is too 
 high.


D has moved a lot towards supplying by default a lot of what Coverity 
claims to do. By making such an expensive tool irrelevant for D, we can 
make D much more cost effective.
Feb 09 2010
parent reply retard <re tard.com.invalid> writes:
Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:


 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we can
 make D much more cost effective.


D doesn't provide non-nullable types and local refs can still escape the 
scope easily. There are several kinds of bugs that cause null pointer 
exceptions / segfaults. Now it took me 2 minutes to come up two examples:

class foo {
  Object a;
  this(Object a) { a = a; }

  void bar() { a.toString(); }

  Object[] baz() {
    scope Object a = new Object;
    return  [ a ];
  }
}

void main() {
  auto a = new Object;
//  auto b = new foo(a);   // segfault
//  b.baz()[0].toString(); // segfault
}
Feb 10 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
retard wrote:

 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:
 

 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we can
 make D much more cost effective.


 
 D doesn't provide non-nullable types


Yes, there have been a couple long threads about that. Dereferencing a 
null pointer is a bug, but not a security/safety issue.


 and local refs can still escape the scope easily.


Yes, that is a problem, and we've been trying to fix that.
Feb 10 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound1 digitalmars.com> wrote in message 
news:hkv1mg$s6e$1 digitalmars.com...

 retard wrote:

 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:


 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we can
 make D much more cost effective.


 D doesn't provide non-nullable types


 Yes, there have been a couple long threads about that. Dereferencing a 
 null pointer is a bug, but not a security/safety issue.


Right, it's just not worthwhile for a language to provide help avoiding 
bugs.
Feb 10 2010
next sibling parent reply retard <re tard.com.invalid> writes:
Wed, 10 Feb 2010 15:29:06 -0500, Nick Sabalausky wrote:


 "Walter Bright" <newshound1 digitalmars.com> wrote in message
 news:hkv1mg$s6e$1 digitalmars.com...

 retard wrote:

 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:


 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we
 can make D much more cost effective.


 D doesn't provide non-nullable types


 Yes, there have been a couple long threads about that. Dereferencing a
 null pointer is a bug, but not a security/safety issue.


 Right, it's just not worthwhile for a language to provide help avoiding
 bugs.


Having such a language would cause unemployment among expensive 
professional programmers. Thus not worth it. That's why non-nullable 
types aren't even worth a try in an experimental branch of dmd unlike 
1000 times more useful features like built-in compiler support for 
regular expressions, opFish or having an integrated email client.
Feb 10 2010
parent "Nick Sabalausky" <a a.a> writes:
"retard" <re tard.com.invalid> wrote in message 
news:hkv5op$s4n$1 digitalmars.com...

 Wed, 10 Feb 2010 15:29:06 -0500, Nick Sabalausky wrote:


 "Walter Bright" <newshound1 digitalmars.com> wrote in message
 news:hkv1mg$s6e$1 digitalmars.com...

 Yes, there have been a couple long threads about that. Dereferencing a
 null pointer is a bug, but not a security/safety issue.


 Right, it's just not worthwhile for a language to provide help avoiding
 bugs.


 Having such a language would cause unemployment among expensive
 professional programmers. Thus not worth it. That's why non-nullable
 types aren't even worth a try in an experimental branch of dmd unlike
 1000 times more useful features like built-in compiler support for
 regular expressions, opFish or having an integrated email client.


opFish!!! Yes, I want it! :)

Although that would beg the question of what would be the appropriate way to 
express gratitude for it if it were to get implemented:

Option 1: Thanks for op the fish!
Option 2: Thanks for all the opFish!
Option 30: Icehouse (<--- Ok, that joke's probably *really* obscure)
Feb 10 2010
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Nick Sabalausky wrote:

 "Walter Bright" <newshound1 digitalmars.com> wrote in message 
 news:hkv1mg$s6e$1 digitalmars.com...

 retard wrote:

 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:


 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we can
 make D much more cost effective.


 D doesn't provide non-nullable types


 Yes, there have been a couple long threads about that. Dereferencing a 
 null pointer is a bug, but not a security/safety issue.


 
 Right, it's just not worthwhile for a language to provide help avoiding 
 bugs. 



It's a benefit/cost issue. There are no bug-preventing features that are 
without cost, the idea is to maximize the ratio.
Feb 10 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:

[about non-nullable types]

 It's a benefit/cost issue. There are no bug-preventing features that are 
 without cost, the idea is to maximize the ratio.


It surely has a cost (you can see it in the Cyclone language: it's safe but
quite fussy, so the costs for the programmer can be high enough to discourage
its use), but I have not tried this feature in other languages, so it's not
easy for me to measure its costs :-)

Bye,
bearophile
Feb 10 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"bearophile" <bearophileHUGS lycos.com> wrote in message 
news:hkv7u7$1a31$1 digitalmars.com...

 Walter Bright:

 [about non-nullable types]

 It's a benefit/cost issue. There are no bug-preventing features that are
 without cost, the idea is to maximize the ratio.


 It surely has a cost (you can see it in the Cyclone language: it's safe 
 but quite fussy, so the costs for the programmer can be high enough to 
 discourage its use), but I have not tried this feature in other languages, 
 so it's not easy for me to measure its costs :-)


I was assuming he meant cost of implementing that feature, but maybe you're 
right...?
Feb 10 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Nick Sabalausky wrote:

 I was assuming he meant cost of implementing that feature, but maybe you're 
 right...? 


I meant the cost to the user.
Feb 10 2010
parent retard <re tard.com.invalid> writes:
Wed, 10 Feb 2010 23:11:54 -0800, Walter Bright wrote:


 Nick Sabalausky wrote:

 I was assuming he meant cost of implementing that feature, but maybe
 you're right...?


 
 I meant the cost to the user.


The fact is that some of the references in a correct program are non-
nullable. If the program behaves according to the spec, those references 
will never point to null. What a language designer should do is to 
analyze the use cases and make the more common case the default. Type 
inference could aid the user here. I'm not sure how much trouble wrong 
guesses would cause, though.
Feb 11 2010
prev sibling parent reply retard <re tard.com.invalid> writes:
Wed, 10 Feb 2010 12:47:41 -0800, Walter Bright wrote:


 Nick Sabalausky wrote:

 "Walter Bright" <newshound1 digitalmars.com> wrote in message
 news:hkv1mg$s6e$1 digitalmars.com...

 retard wrote:

 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:


 D has moved a lot towards supplying by default a lot of what
 Coverity claims to do. By making such an expensive tool irrelevant
 for D, we can make D much more cost effective.


 D doesn't provide non-nullable types


 Yes, there have been a couple long threads about that. Dereferencing a
 null pointer is a bug, but not a security/safety issue.


 Right, it's just not worthwhile for a language to provide help avoiding
 bugs.


 
 
 It's a benefit/cost issue. There are no bug-preventing features that are
 without cost, the idea is to maximize the ratio.


What exactly is the cost here? The non-nullability invariant can be 
checked on compile time. It incurs no runtime overhead. Also the notation 
can be quite terse, as we have seen in Other Languages(tm).
Feb 10 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
retard wrote:

 What exactly is the cost here? The non-nullability invariant can be 
 checked on compile time. It incurs no runtime overhead. Also the notation 
 can be quite terse, as we have seen in Other Languages(tm).


If you want, you can create a template NonNullable that wraps an 
existing type, and try that out.
Feb 10 2010
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-10 17:29:09 -0500, Walter Bright <newshound1 digitalmars.com> said:


 retard wrote:

 What exactly is the cost here? The non-nullability invariant can be 
 checked on compile time. It incurs no runtime overhead. Also the 
 notation can be quite terse, as we have seen in Other Languages(tm).


 
 If you want, you can create a template NonNullable that wraps an 
 existing type, and try that out.


Just did that yesterday. It's probably not what most people expect 
though: it doesn't do static checking. But at least it catches null 
assignments early, before the null value breaks something elsewhere in 
the program. For your enjoyment:

/**
 Struct to make object references and pointers unable to accept a null value.
 The initial value is still null, but after the first assignment the value can
 never become null again. Attempting to assign null will throw an
 IllegalNullAssignment error.
 */
struct NonNullable(T) {
	private T _nonNullableValue;
	alias _getNonNullableValue this;
	
	this(T value) {
		opAssign(value);
	}
	this(NonNullable!T value) {
		// FIXME: Not polymorphic
		opAssign(value);
	}
	
	void opAssign(V, string file = __FILE__, int line = __LINE__)(V newValue) {
		if (isNull(newValue))
			IllegalNullAssignment.bailOut(file, line, "Attempt to assign null to "
										  "non-nullable " ~ T.stringof ~ ".");
		_nonNullableValue = newValue;
	}
	
	T _getNonNullableValue() {
		assert(_nonNullableValue, "Attempt to access uninitialized 
non-nullable " ~ T.stringof ~ ".");
		return _nonNullableValue;
	}
	
	private {
		bool isNull(T)(T value) if (__traits(compiles, value._nonNullableValue)) {
			return isNull(value._nonNullableValue);
		}
		bool isNull(T)(T value) if (__traits(compiles, value is null)) {
			return value is null;
		}
	}
}


/**
 Error thrown when attempting to assign a null value to a non-nullable.
 */
class IllegalNullAssignment : Error {
	this(string message) {
		super(message);
	}
	
	private static void bailOut(string file, int line, in char[] msg) {
		throw new IllegalNullAssignment(text(file, '(', line, "): ", msg));
	}
}


unittest {
	NonNullable!Object o;
	NonNullable!ClassInfo o2 = o.classinfo;
	NonNullable!TypeInfo o3;
	//NonNullable!Object o4 = o2; // FIXME: polymorphic NonNullable
	o = new Object;
	o = o2;
	try {
		o = o3; // should throw
		o.toString();
		assert(0, "prior assignment should have thrown");
	} catch (IllegalNullAssignment e) {
		// ok, error thrown.
	}
}



-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 10 2010
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Michel Fortin wrote:

 unittest {
     NonNullable!Object o;


Here o is null.


     NonNullable!ClassInfo o2 = o.classinfo;
     NonNullable!TypeInfo o3;
     //NonNullable!Object o4 = o2; // FIXME: polymorphic NonNullable
     o = new Object;
     o = o2;
     try {
         o = o3; // should throw
         o.toString();
         assert(0, "prior assignment should have thrown");
     } catch (IllegalNullAssignment e) {
         // ok, error thrown.
     }
 }
Feb 10 2010
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Ary Borenszweig wrote:

 Michel Fortin wrote:

 unittest {
     NonNullable!Object o;


 
 Here o is null.


Sorry, just read the class comments. But if this can't be implemented 
without compiler support... then it's not of much use. :-(
Feb 10 2010
next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-10 22:54:40 -0500, Ary Borenszweig <ary esperanto.org.ar> said:


 Ary Borenszweig wrote:

 Michel Fortin wrote:

 unittest {
     NonNullable!Object o;


 
 Here o is null.


 
 Sorry, just read the class comments. But if this can't be implemented 
 without compiler support... then it's not of much use. :-(


To recapitulate the documentation comment at the top of NonNullable: it 
is indeed null until the first assignment. It's non nullable in the 
sense that you can't make it null, but it still can be null if you 
assign nothing to it. It means that if you encounter a null value in a 
non-nullable you know it was never initialized. So the obvious place to 
look is where it should have been initialized. I really wonder how bad 
this would be in practice (I haven't used it much yet).

Unfortunately, there isn't a better way to implement NonNullable at the 
moment. An option could be to throw immediately at construction, but 
constructors with no arguments are not available for structs, and I'm 
not sure it'd be so great anyway.

There is also the issue of lack of polymorphism: a NonNullable!MyObject 
variable won't implicitly convert to NonNullable!Object, although I've 
been able to make it work for non-construction assignments using a 
template opAssign.

So I agree with you that it's of limited utility. I think it can still 
be useful at times. What would be great is non-nullable by default, but 
that's another thing entirely.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 10 2010
parent Ary Borenszweig <ary esperanto.org.ar> writes:
Michel Fortin wrote:

 On 2010-02-10 22:54:40 -0500, Ary Borenszweig <ary esperanto.org.ar> said:
 

 Ary Borenszweig wrote:

 Michel Fortin wrote:

 unittest {
     NonNullable!Object o;


 Here o is null.


 Sorry, just read the class comments. But if this can't be implemented 
 without compiler support... then it's not of much use. :-(


 
 To recapitulate the documentation comment at the top of NonNullable: it 
 is indeed null until the first assignment. It's non nullable in the 
 sense that you can't make it null, but it still can be null if you 
 assign nothing to it. It means that if you encounter a null value in a 
 non-nullable you know it was never initialized. So the obvious place to 
 look is where it should have been initialized. I really wonder how bad 
 this would be in practice (I haven't used it much yet).


That makes sense. :)


 Unfortunately, there isn't a better way to implement NonNullable at the 
 moment. An option could be to throw immediately at construction, but 
 constructors with no arguments are not available for structs, and I'm 
 not sure it'd be so great anyway.


Maybe in D2 you will be able to put  disable to the default constructor 
of the struct, so you'll have to explicity assign something to it.


 There is also the issue of lack of polymorphism: a NonNullable!MyObject 
 variable won't implicitly convert to NonNullable!Object, although I've 
 been able to make it work for non-construction assignments using a 
 template opAssign.
 
 So I agree with you that it's of limited utility. I think it can still 
 be useful at times. What would be great is non-nullable by default, but 
 that's another thing entirely.


Yes, I think non-nullable by default is better.
Feb 11 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Ary Borenszweig:


But if this can't be implemented without compiler support... then it's not of
much use. :-(<


There's another solution: to make D2/D3 itself flexible enough that you can
implement your own simple "extensions" to the type system, so you can actually
implement not-nullable references by default. This isn't impossible (there are
many papers about this, it can be done in Scheme, etc).

Once such flexibility is present, you can quickly create such plug-ins to the
type system. Like in Firefox, plug-ins are a very good way to experiment new
features, that can be later added to the main software if they are good.
(Recently GCC has gained the ability to be use plug-ins, but here I am talking
about something in the language, not in the compiler).

Related: I think the downsides of something like the AST macros (that is
community fragmentation, etc) can be quite mitigated if the main purpose of
them is to allow to create experimental features that later can be refused or
accepted into the official language :-)

Bye,
bearophile
Feb 11 2010
parent reply dennis luehring <dl.soluz gmx.net> writes:
 There's another solution: to make D2/D3 itself flexible enough that you can
implement your own simple "extensions" to the type system, so you
can actually implement not-nullable references by default. This isn't 


impossible (there are many papers about this,

it can be done in Scheme, etc).


"everything is possible with code" - its not the question of how to 
start an plugin/dll based function from descent compiler steps... thats 
too easy to even talk about

but how should the plugin-system work?
what compiler-functions should be changeable through plugins?
how do the different engine like parser, semantic analyser etc work 
together with the plugin, etc.? that is the nearly impossible to answer 
question - what do the LLVM-guys think about plugin like that

and firefox-plugins are far too easy in its possible feature-set 
compared to what is needed to add an type to an compiler
Feb 11 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
dennis luehring:

 but how should the plugin-system work?


I am ignorant still about type systems. But see something about pluggable type
systems:
http://lambda-the-ultimate.org/node/1311



what do the LLVM-guys think about plugin like that<


Chris Lattner, the boss of the LLVM project, has told me that he is not
interested in a plug-in system for LLVM similar to the new GCC one, because
LLVM has adopted kind of the opposite design strategy: it's made of loosely
connected components that you can recombine and reuse to do what you want.

Bye,
bearophile
Feb 11 2010
next sibling parent retard <re tard.com.invalid> writes:
Thu, 11 Feb 2010 16:20:55 -0500, bearophile wrote:


 dennis luehring:

 but how should the plugin-system work?


 
 I am ignorant still about type systems. But see something about
 pluggable type systems: http://lambda-the-ultimate.org/node/1311


In some educational Compiler Done Right (tm) projects the various parts 
of the compiler might be loosely coupled clean modules, but experience 
shows that in real life when you take e.g. efficiency into consideration, 
compilers tend to get nasty. Both parser and type system are two of the 
subsystems that almost everything else on a compiler depends on.

 
 

what do the LLVM-guys think about plugin like that<


 
 Chris Lattner, the boss of the LLVM project, has told me that he is not
 interested in a plug-in system for LLVM similar to the new GCC one,
 because LLVM has adopted kind of the opposite design strategy: it's made
 of loosely connected components that you can recombine and reuse to do
 what you want.
 
 Bye,
 bearophile
Feb 11 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
I've just found a nice article about this topic, "LCA: Static analysis with GCC
plugins", by Jonathan Corbet:
http://lwn.net/Articles/370717/

Beside allowing plug-ins, they have done something else to GCC:

The ability to attach attributes to objects in the compiled code makes it easy
to pass hints through to later processing steps. The new pass manager brings a
relatively modern structure to a compiler which did not originally have one.
And the GIMPLE intermediate representation provides much of the rest of what's
needed for code which needs to inspect other code.<


So now they are essentially extending the C++ type system using small
JavaScript programs:

Various other tools have been written. The final.js script (a dozen lines of
code which can be seen on this page) looks for C++ methods tagged with the
"final" attribute; any attempt to override those methods will result in a
compilation error. It is, in other words, a port of the Java final keyword to
C++. A checker which might be interesting in other environments - including the
kernel - is flow.js, which can add a constraint that all exits from a function
must flow through a specific label.<


The final.js code:
https://developer.mozilla.org/En/Dehydra/Using_Dehydra

/**
 * Helper function: returns true if a class is marked with the "final"
attribute.
 */
function isFinal(c) {
  if (!c.attributes)
    return false;
  for each (let a in c.attributes)
    if (a.name == 'user' && a.value == 'final')
      return true;
  return false;
}
function process_type(t) {
  if (t.bases)
    for each (let base in t.bases)
      if (isFinal(base.type))
        error("class " + t.name + " extends final class " + base.type.name,
t.loc);
}


Then in C++ the syntax of final class is not nice:
class __attribute__((user("final"))) MyClass {

In D you can probably write the same thing as (D already has final, so this is
just to compare):
 final class MyClass {

I think you can implement nonnullable references too (but you can't use the
nice "?" syntax for nullables).

Now please give me five minutes to dream about doing something similar using
Python for D :-) It can be even possible to use D to write those scripts,
especially once types become more first class values in D.

Bye,
bearophile
Feb 12 2010
next sibling parent reply retard <re tard.com.invalid> writes:
Fri, 12 Feb 2010 12:07:16 -0500, bearophile wrote:


 /**
  * Helper function: returns true if a class is marked with the "final"
  attribute. */
 function isFinal(c) {
   if (!c.attributes)
     return false;
   for each (let a in c.attributes)
     if (a.name == 'user' && a.value == 'final')
       return true;
   return false;
 }
 function process_type(t) {
   if (t.bases)
     for each (let base in t.bases)
       if (isFinal(base.type))
         error("class " + t.name + " extends final class " +
         base.type.name, t.loc);
 }


That kind of code is rather clumsy. Having written couple of toy 
languages, I really enjoy expressing the rules as immutable declarative 
axioms, e.g.:

final_class_rule: node.metatype == class && node.hasBases ==> !
node.bases.exists(n => n.attributes.contains("final"))

By using declarative notion, the ambiguity of rules can be tested and 
rules can be verified more easily.
Feb 12 2010
parent bearophile <bearophileHUGS lycos.com> writes:
retard Wrote:

 That kind of code is rather clumsy.


That code is probably not the best possible way to express those constraints.
But they have a good enough JavaScript JIT compiler, they know JavaScript well,
so using JS for that purpose can be good enough [TM] and it can be better than
not being able to do that.

Bye,
bearophile
Feb 12 2010
prev sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
bearophile wrote:

 Then in C++ the syntax of final class is not nice:
 class __attribute__((user("final"))) MyClass {


To be pedantic, C++ does not have final classes. That's a compiler 
extension.

Ali
Feb 12 2010
prev sibling parent reply Leandro Lucarella <llucax gmail.com> writes:
Walter Bright, el 10 de febrero a las 11:33 me escribiste:

 retard wrote:

Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:


D has moved a lot towards supplying by default a lot of what Coverity
claims to do. By making such an expensive tool irrelevant for D, we can
make D much more cost effective.


D doesn't provide non-nullable types


 
 Yes, there have been a couple long threads about that. Dereferencing
 a null pointer is a bug, but not a security/safety issue.


What about 3677?

Quote:
--------------------------------
This supposedly "safe" program under Mac OS X 10.6 doesn't give any error
neither at compile time nor at runtime, yet it isn't memory-safe at all as
it corrupts some part of the memory space.

struct S {
    uint[100000] a;
    uint b;
}

S* s = null;

 safe void main() {
    s.b = 1;
}

This happen because the offset for member "b" is very far, allowing it to
falls on another memory page which happen to exists.
--------------------------------

http://d.puremagic.com/issues/show_bug.cgi?id=3677

It's a strange case, I know...

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Feb 10 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Leandro Lucarella wrote:

 This supposedly "safe" program under Mac OS X 10.6 doesn't give any error
 neither at compile time nor at runtime, yet it isn't memory-safe at all as
 it corrupts some part of the memory space.



 safe is in its infancy, and I expect there to be some gaps. We'll get 
them plugged over time. In the meantime, it's great that you filed a bug 
report on it.
Feb 10 2010
parent Leandro Lucarella <llucax gmail.com> writes:
Walter Bright, el 10 de febrero a las 14:28 me escribiste:

 Leandro Lucarella wrote:

This supposedly "safe" program under Mac OS X 10.6 doesn't give any error
neither at compile time nor at runtime, yet it isn't memory-safe at all as
it corrupts some part of the memory space.


 
  safe is in its infancy, and I expect there to be some gaps. We'll
 get them plugged over time. In the meantime, it's great that you
 filed a bug report on it.


It wasn't me, but kudos for the reporter indeed =)

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Feb 11 2010