www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - About making Phobos safe

reply Seb <seb wilzba.ch> writes:
Making all high-level functions of Phobos  safe
-----------------------------------------------

There's are still some functions in Phobos which could be  safe, 
but aren't.
One of the high-level goals of 2018 [1] is:


 to enable large-scale uses of D with safety guarantees.


So what's missing regarding Phobos?

GH project: https://github.com/dlang/phobos/projects/5
Bugzilla issue: https://issues.dlang.org/show_bug.cgi?id=18110

or old-school with grep " system unittest"

DIP1000 & Phobos
----------------

Thanks to a lot of work of carblue, there's now a special 
dip1000.mak file, s.t. the last modules can be gradually tackled 
without regressions:

https://github.com/dlang/phobos/blob/master/dip1000.mak

GH project: https://github.com/dlang/phobos/projects/6


Meh, this is taking too long
----------------------------

So if it's still cold in your country, why not take 15 minutes 
and help in this crowd-sourced campaign to make Phobos  safer?
If you don't like  safety, there are a few other currently 
maintained GitHub projects [2].

[1] https://wiki.dlang.org/Vision/2018H1
[2] https://github.com/dlang/phobos/projects
Mar 22 2018
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/22/2018 12:14 PM, Seb wrote:

 Making all high-level functions of Phobos  safe
 -----------------------------------------------
 
 There's are still some functions in Phobos which could be  safe, but aren't.
 One of the high-level goals of 2018 [1] is:
 

 to enable large-scale uses of D with safety guarantees.


 
 So what's missing regarding Phobos?



To help focus on these things most important to realizing D's vision, I added 
the [Vision] Github tag to Phobos and Dmd, and added the [Vision] keyword to 
Bugzilla.
Mar 22 2018
prev sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Thursday, 22 March 2018 at 19:14:06 UTC, Seb wrote:

 Making all high-level functions of Phobos  safe
 -----------------------------------------------

 There's are still some functions in Phobos which could be 
  safe, but aren't.


I was going to ask this in Slack but since this thread is already 
here, why not.

What are we going to do about C library calls in std.zlib and 
std.zip? I'm really uncomfortable about adding  trusted to the 
zlib calls, as it's different than calling C functions from the 
std library. There's no issue in reality with marking a 
malloc/free pair as trusted when it's verified to not escape. But 
there's really no garuntee about the safety of third party 
libraries. What if there's a Heartbleed level bug in zlib and we 
marked it as  trusted?

Should we just resign ourselves to the fact that some functions 
are going to be  system no matter what?
Mar 23 2018
parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Friday, 23 March 2018 at 14:37:12 UTC, Jack Stouffer wrote:

 What are we going to do about C library calls in std.zlib and 
 std.zip? I'm really uncomfortable about adding  trusted to the 
 zlib calls, as it's different than calling C functions from the 
 std library. There's no issue in reality with marking a 
 malloc/free pair as trusted when it's verified to not escape. 
 But there's really no garuntee about the safety of third party 
 libraries. What if there's a Heartbleed level bug in zlib and 
 we marked it as  trusted?


 safe/ trusted doesn't mean "calling this function will not 
result in heartbleed."

What you're getting is that when you call this function there are 
no parameter values which you could pass such that you could 
cause memory corruption. If you are concerned about heartbleed 
then you don't need to review  safe code,  trusted code should be 
reviewed so that  safe code can't cause issue with any of the 
valid parameters (the way it calls  system functions should be 
safe). And  system code should reviewed for heartbleed. If you 
use zlib and it has a heartbleed bug then it is your fault for 
not reviewing the  system code, not the fault of D marking its 
usage as  system.

You mention mallac being ok to use with  trusted, but it has the 
same issue, if you didn't review the code behind your mallac call 
then it could introduce heartbleed, you don't know because you 
didn't review it.
Mar 23 2018
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Friday, 23 March 2018 at 17:31:09 UTC, Jesse Phillips wrote:

  safe/ trusted doesn't mean "calling this function will not 
 result in heartbleed."


If  safe doesn't protect against buffer overflows then chuck the 
whole thing out the window and start over.


 What you're getting is that when you call this function there 
 are no parameter values which you could pass such that you 
 could cause memory corruption. If you are concerned about 
 heartbleed then you don't need to review  safe code,  trusted 
 code should be reviewed so that  safe code can't cause issue 
 with any of the valid parameters (the way it calls  system 
 functions should be safe). And  system code should reviewed for 
 heartbleed. If you use zlib and it has a heartbleed bug then it 
 is your fault for not reviewing the  system code, not the fault 
 of D marking its usage as  system.


We can't review the zlib code which is the thrust of the issue. 
It's entirely possible that a bug  in zlib could write past the 
bounds of an array. We could in theory verify zlib, but

1. It would require including the C code in the distribution that 
was reviewed rather than linking to the system version

2. The review process would have to be repeated every time we 
update to a new zlib release.

 From a Phobos dev's perspective, zlib.compress2 is as much as a 
black box as malloc.


 You mention mallac being ok to use with  trusted, but it has 
 the same issue, if you didn't review the code behind your 
 mallac call then it could introduce heartbleed, you don't know 
 because you didn't review it.


Functions in the C standard library are a category difference 
because it's completely reasonable to assume that malloc isn't 
going to corrupt memory. The same assumption cannot be made with, 
to use the example again, libssl.
Mar 23 2018
next sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Friday, 23 March 2018 at 20:33:40 UTC, Jack Stouffer wrote:

 On Friday, 23 March 2018 at 17:31:09 UTC, Jesse Phillips wrote:

 [...]


 If  safe doesn't protect against buffer overflows then chuck 
 the whole thing out the window and start over.

 [...]


zlib sources are included in phobos. It doesn't link to the sys 
lib afaict.

https://github.com/dlang/phobos/tree/master/etc/c/zlib


 [...]
Mar 24 2018
prev sibling parent Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Friday, 23 March 2018 at 20:33:40 UTC, Jack Stouffer wrote:

 If  safe doesn't protect against buffer overflows then chuck 
 the whole thing out the window and start over.


Then chuck the whole thing out the window and start your own 
review over and include the  safe code this time.

You say it is reasonable to assume that mallac isn't a problem 
but I disagree depending on you needs to be secure.

In that space you can't rely on other programmers to have 
correctly verified.

The compiler checks  safe code, not  system or  trusted. These 
are there to indicate you need to review the code, not to 
indicate review has determined it to be bullet proof.

Yes zlib may be too much to review, so don't use it. Rewrite your 
C libraries in  safe so the compiler does the validation for you.
Mar 24 2018