www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Scenario: OpenSSL in D language, pros/cons

reply "Daniele M." <gdm85 iragan.com> writes:
I have read this excellent article by David A. Wheeler:

http://www.dwheeler.com/essays/heartbleed.html

And since D language was not there, I mentioned it to him as a 
possible good candidate due to its static typing and related 
features.

However, now I am asking the community here: would a D 
implementation (with GC disabled) of OpenSSL have been free from 
Heartbleed-type vulnerabilities? Specifically 
http://cwe.mitre.org/data/definitions/126.html and 
http://cwe.mitre.org/data/definitions/20.html as David mentions.

I find this perspective very interesting, please advise :)
May 04 2014
next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sun, 04 May 2014 08:34:19 +0000
"Daniele M. via Digitalmars-d" <digitalmars-d puremagic.com> wrote:


 I have read this excellent article by David A. Wheeler:

 http://www.dwheeler.com/essays/heartbleed.html

 And since D language was not there, I mentioned it to him as a
 possible good candidate due to its static typing and related
 features.

 However, now I am asking the community here: would a D
 implementation (with GC disabled) of OpenSSL have been free from
 Heartbleed-type vulnerabilities? Specifically
 http://cwe.mitre.org/data/definitions/126.html and
 http://cwe.mitre.org/data/definitions/20.html as David mentions.

 I find this perspective very interesting, please advise :)


 safe code protects against indexing an array out-of-bounds. So, if OpenSSL
had been implemented in D, and its heartbeat feature used  safe code, then
heartbleed would not have been possible. As soon as an attempt was made to
index passed the end of the array, it would have thrown a RangeError and
killed the program.

Now, even if OpenSSL had been implemented in D, if it had used  system or
 trusted code for its heartbeat feature, then it could have had the bug just
as easily in D as it did in C. And given all of the horrible practices in
OpenSSL, I very much doubt that having it written in D would have prevented
much, because anyone making the choices that the OpenSSL folks have been
making would likely have ended up with horrible D code which was mostly
 system and probably doing all kinds of things that are inherently risky,
forgoeing many of the benefits that D provides.

I think that it's safe to say that D makes it easier to catch problems like
this, but it doesn't entirely prevent them, and bad programming practices
can pretty much always get around protections that the language provides
unless the language provides no ways around those protections (which isn't
the case in D, because it's a systems language and needs to provide low-level
access and features for those programs that need them - it just doesn't use
those by default).

If I had more time, I'd actually be tempted to write an SSL implementation in
D, but even if I were to do an excellent job of it, it would still need to be
vetted by security experts to make sure that it didn't have horrible security
bugs in it (much as it would be likely that there would be fewer thanks to
the fact that it would be writen in D), and I suspect that it's the kind of
thing that many people aren't likely to trust because of how critical it is.
So, I don't know how good an idea it is at this point for someone to write
an implementation of SSL or TLS in D. Certainly, it's the type of thing where
we've generally tried to wrap existing C libraries in order to avoid having
to spend the time, effort, and expertise on in order to fully implement it
ourselves. The Go guys did it, but if I understand correctly, the fellow that
did it was one of the OpenSSL developers, so presumably he's already very
familiar with all of the ins and outs of SSL, and I don't know if any of us
here are (I'm certainly not - if I were doing it, I'd pretty much just have
to go off of its associated RFCs, for better or worse).

At this point though, if I were looking at using an existing implementation
of SSL, I'd probably be looking at GnuTLS rather than OpenSSL given how
horrible OpenSSL's codebase is. I don't know that GnuTLS is any better, but
it wouldn't be hard for it to be. OpenSSL is horrible both with regards to
its implementation and its API, and we'd all be better off if something
better replaced it (be it GnuTLS or something else). Unfortunately, even if
something better _were_ written in D, it's probably only the D folks who
would benefit, since it's not terribly likely at this point that very many
folks are going to wrap a D library in order to use it another language.

However, regardless of whether we ever end up with an SSL implementation in D,
I think that in the long run, D will show itself to be much, much better than
C or C++ at writing code that has a low number of security bugs.

- Jonathan M Davis
May 04 2014
parent reply "Daniele M." <gdm85 iragan.com> writes:
On Sunday, 4 May 2014 at 10:23:38 UTC, Jonathan M Davis via 
Digitalmars-d wrote:

 On Sun, 04 May 2014 08:34:19 +0000
 "Daniele M. via Digitalmars-d" <digitalmars-d puremagic.com> 
 wrote:


 I have read this excellent article by David A. Wheeler:

 http://www.dwheeler.com/essays/heartbleed.html

 And since D language was not there, I mentioned it to him as a
 possible good candidate due to its static typing and related
 features.

 However, now I am asking the community here: would a D
 implementation (with GC disabled) of OpenSSL have been free 
 from
 Heartbleed-type vulnerabilities? Specifically
 http://cwe.mitre.org/data/definitions/126.html and
 http://cwe.mitre.org/data/definitions/20.html as David 
 mentions.

 I find this perspective very interesting, please advise :)


  safe code protects against indexing an array out-of-bounds. 
 So, if OpenSSL
 had been implemented in D, and its heartbeat feature used  safe 
 code, then
 heartbleed would not have been possible. As soon as an attempt 
 was made to
 index passed the end of the array, it would have thrown a 
 RangeError and
 killed the program.


This is what I thought too.


 Now, even if OpenSSL had been implemented in D, if it had used 
  system or
  trusted code for its heartbeat feature, then it could have had 
 the bug just
 as easily in D as it did in C. And given all of the horrible 
 practices in
 OpenSSL, I very much doubt that having it written in D would 
 have prevented
 much, because anyone making the choices that the OpenSSL folks 
 have been
 making would likely have ended up with horrible D code which 
 was mostly
  system and probably doing all kinds of things that are 
 inherently risky,
 forgoeing many of the benefits that D provides.

 I think that it's safe to say that D makes it easier to catch 
 problems like
 this, but it doesn't entirely prevent them, and bad programming 
 practices
 can pretty much always get around protections that the language 
 provides
 unless the language provides no ways around those protections 
 (which isn't
 the case in D, because it's a systems language and needs to 
 provide low-level
 access and features for those programs that need them - it just 
 doesn't use
 those by default).


I should have said: a D implementation of OpenSSL *without* the 
malloc sorcerery, sorry for leaving it out of OP. Reason I'd like 
to ignore that for this discussion is that I consider it a bug. 
And we are all in the post-heartbleed scenario now so it looks 
huge, but Theo De Raadt already said it all, nothing to add on it.

And then comes my next question: except for that malloc-hack, 
would it have been possible to write it in  safe D? I guess that 
if not, module(s) could have been made un- safe. Not saying that 
a similar separation of concerns was not possible in OpenSSL 
itself, but that D could have made it less development-expensive 
in my opinion.


 If I had more time, I'd actually be tempted to write an SSL 
 implementation in
 D, but even if I were to do an excellent job of it, it would 
 still need to be
 vetted by security experts to make sure that it didn't have 
 horrible security
 bugs in it (much as it would be likely that there would be 
 fewer thanks to
 the fact that it would be writen in D), and I suspect that it's 
 the kind of
 thing that many people aren't likely to trust because of how 
 critical it is.


Nobody would expect/trust a single person to do this job :P
Working in an open source project would be best.


 So, I don't know how good an idea it is at this point for 
 someone to write
 an implementation of SSL or TLS in D. Certainly, it's the type 
 of thing where
 we've generally tried to wrap existing C libraries in order to 
 avoid having
 to spend the time, effort, and expertise on in order to fully 
 implement it
 ourselves. The Go guys did it, but if I understand correctly, 
 the fellow that
 did it was one of the OpenSSL developers, so presumably he's 
 already very
 familiar with all of the ins and outs of SSL, and I don't know 
 if any of us
 here are (I'm certainly not - if I were doing it, I'd pretty 
 much just have
 to go off of its associated RFCs, for better or worse).


I wasn't aware of this effort, interesting experiment.


 At this point though, if I were looking at using an existing 
 implementation
 of SSL, I'd probably be looking at GnuTLS rather than OpenSSL 
 given how
 horrible OpenSSL's codebase is. I don't know that GnuTLS is any 
 better, but
 it wouldn't be hard for it to be. OpenSSL is horrible both with 
 regards to
 its implementation and its API, and we'd all be better off if 
 something
 better replaced it (be it GnuTLS or something else).


I would pick LibreSSL:
http://www.libressl.org/

And merge patches as their progress goes on.


 Unfortunately, even if
 something better _were_ written in D, it's probably only the D 
 folks who
 would benefit, since it's not terribly likely at this point 
 that very many
 folks are going to wrap a D library in order to use it another 
 language.


Here I don't completely agree: if we can have a binary-compatible 
implementation done in D, then we would be able to modify 
software to eventually use it as a dependency. I don't see the 
necessary D dependencies as prohibitive here.


 However, regardless of whether we ever end up with an SSL 
 implementation in D,


I started this discussion just for the thought exercise, because 
I think it can give a few positive leads, but nice to see that 
other people too thought about a possible implementation :)


 I think that in the long run, D will show itself to be much, 
 much better than
 C or C++ at writing code that has a low number of security bugs.

 - Jonathan M Davis


Thanks for your post Jonathan, however I believe that attempts 
like Go's are what makes a language more real and less 
theoretical. The field test of D on system-level software would 
be in my opinion useful and sometimes I think the sooner the 
better, to eventually modify the evolution of this beautiful 
language (if there's still a viable window for this).
May 04 2014
next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sun, 04 May 2014 21:18:22 +0000
"Daniele M. via Digitalmars-d" <digitalmars-d puremagic.com> wrote:


 On Sunday, 4 May 2014 at 10:23:38 UTC, Jonathan M Davis via
 Digitalmars-d wrote:



 And then comes my next question: except for that malloc-hack,
 would it have been possible to write it in  safe D? I guess that
 if not, module(s) could have been made un- safe. Not saying that
 a similar separation of concerns was not possible in OpenSSL
 itself, but that D could have made it less development-expensive
 in my opinion.


I don't know what all OpenSSL is/was doing, and I haven't looked into it in
great detail. I'm familiar with what caused heartbleed, and I'm somewhat
familiar with OpenSSL's API from having dealt with it at work, but most of
what I know about OpenSSL, I know from co-workers who have had to deal with it
and other stuff that I've read about it, and in general, from what I
understand, it's just plain badly designed and badly written, and it's a
miracle that it works as well as it does. Most of the problems seem to stem
from how the project is managed (including having horrible coding style and
generally not liking to merge patches), but it's also certain that a number of
the choices that they've made make it easier for security problems to creep in
(e.g. using their own malloc in an attempt to gain some speed on some OSes).


From what I know of SSL itself (and I've read some of the spec, but not all of


it), very little of it (and probably none of it save for the actual operations
on the sockets) actually requires anything that's  system. The problem is when
you go to great lengths to optimize the code, which the OpenSSL guys seem to
have done. When you do that, you do things like turn off array bounds
checking and generally try and avoid many of the safety features that a
language like D provides, since many of them do incur at least some overhead.

Actually implementing SSL itself wouldn't take all that long from what I
understand. The main problem is in maintenance - probably in particular with
regards to the fact that you'd have to keep adding support for more encryption
methods as they come out (which technically aren't part of SSL itself), but
I'm not familiar enough with the details to know all of the nooks and crannies
that would cause maintenance nightmares. The base spec is less than 70 pages
long though. The fellow who answered the question here seems to think that
implementing SSL itself is actually fairly easy and that he's done it several
times for companies already:

http://security.stackexchange.com/questions/55465

I fully expect that if someone were to implement it in D, it would be safer
out of the box than a C implementation would be. But if you had to start
playing tricks to get it faster, that would increase the security risk, and in
order for folks to trust it, you'd have to get the code audited, which is a
whole other order of pain (and one that potentially costs money, depending on
who does the auditing).


 If I had more time, I'd actually be tempted to write an SSL
 implementation in
 D, but even if I were to do an excellent job of it, it would
 still need to be
 vetted by security experts to make sure that it didn't have
 horrible security
 bugs in it (much as it would be likely that there would be
 fewer thanks to
 the fact that it would be writen in D), and I suspect that it's
 the kind of
 thing that many people aren't likely to trust because of how
 critical it is.


 Nobody would expect/trust a single person to do this job :P
 Working in an open source project would be best.


If someone around here implemented SSL in D, I fully expect that it would be
open source, and I fully expect that one person could do it. It's just a
question of how long it would take them - though obviously sharing the work
among multiple people could make it faster. Where it's definitely required
that more people get involved is when you want the code audited to ensure that
it's actually safe and secure. And that's where implementing an SSL library is
fundamentally different from implementing most libraries - it's so integral to
security that it doesn't really cut it to just throw an implementation
together and toss it out there for folks to use it if they're interested.


 Unfortunately, even if
 something better _were_ written in D, it's probably only the D
 folks who
 would benefit, since it's not terribly likely at this point
 that very many
 folks are going to wrap a D library in order to use it another
 language.


 Here I don't completely agree: if we can have a binary-compatible
 implementation done in D, then we would be able to modify
 software to eventually use it as a dependency. I don't see the
 necessary D dependencies as prohibitive here.


If D were to be part of a typical Linux distro, a D compiler would have to be
part of a typical Linux distro. We're getting there with gdc as it's getting
merged into gcc, but I don't think that it's ended up on many distros yet, and
you then end up with the problem of compatabitily, because while D is far, far
more stable than it used to be, it's still pretty common that small things
change or break over time (sometimes by necessity, sometimes due to
regressions), so writing a D library that could be used on a Linx system
with the same compiler version for several years is probably still a bit
unrealistic (though we're getting closer). And without that, even with a solid
C wrapper around it, nothing that's normally part of a Linux distro is going
to use it. At most, you'd get the occasional, smaller project that used it.

And as for Windows, that's not terribly likely to work at this point, because
we only use the MS linker for 64-bit programs, and it's still frequently the
case that Windows programs are build for 32-bit, because the developrs don't
want to have to require 64-bit due to all of the 32-bit copies of Windows that
are still sold.

So, while it's theoretically feasible to have a D library wrapped in C and
then used in C and C++ programs by folks outside the D community, I don't
think that it's actually realistic at this point.


 Thanks for your post Jonathan, however I believe that attempts
 like Go's are what makes a language more real and less
 theoretical. The field test of D on system-level software would
 be in my opinion useful and sometimes I think the sooner the
 better, to eventually modify the evolution of this beautiful
 language (if there's still a viable window for this).


It's mostly an issue of time and man power. I have quite a few projects that
I'd like to work on for myself and for the D community at large, and I only
have the time to work on a fraction of them, if that. And the same goes for
many of the rest of the folks around here. We tend to try and direct our time
and energy to projects that are of greater benefit (which often means not
reimplementing stuff that exists in C and can easily be wrapped - especially
if reimplementing it in D would be time consuming), but regardless of where we
spend our time and energy, we just plain need more people to pitch in if we
want more stuff implemented in D.

Something like SSL is only going to be implemented in D if a member of the
community decides that it's worth it to them to spend the time doing it, and
they have the knowledge and expertise to pull it off. And given all of the
other things that need to be done, I'm not sure that it even makes sense to
spend time on implementing SSL in D as great as it would be to have such an
implementation. We can't do everything, much as we might like to.

- Jonathan M Davis
May 04 2014
prev sibling parent reply "JR" <zorael gmail.com> writes:
On Sunday, 4 May 2014 at 21:18:24 UTC, Daniele M. wrote:

 And then comes my next question: except for that malloc-hack, 
 would it have been possible to write it in  safe D? I guess 
 that if not, module(s) could have been made un- safe. Not 
 saying that a similar separation of concerns was not possible 
 in OpenSSL itself, but that D could have made it less 
 development-expensive in my opinion.


TDPL SafeD visions notwithstanding,  safe is very very limiting.

I/O is forbidden so simple Hello Worlds are right out, let alone 
advanced socket libraries.
May 05 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 5 May 2014 at 09:32:40 UTC, JR wrote:

 On Sunday, 4 May 2014 at 21:18:24 UTC, Daniele M. wrote:

 And then comes my next question: except for that malloc-hack, 
 would it have been possible to write it in  safe D? I guess 
 that if not, module(s) could have been made un- safe. Not 
 saying that a similar separation of concerns was not possible 
 in OpenSSL itself, but that D could have made it less 
 development-expensive in my opinion.


 TDPL SafeD visions notwithstanding,  safe is very very limiting.

 I/O is forbidden so simple Hello Worlds are right out, let 
 alone advanced socket libraries.


I/O is not forbidden, it's just that writeln and friends 
currently can't be made safe, but that is being worked on AFAIK. 
While I/O usually goes through the OS, the system calls can be 
manually verified and made  trusted.
May 05 2014
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 05 May 2014 10:24:27 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On Monday, 5 May 2014 at 09:32:40 UTC, JR wrote:

 On Sunday, 4 May 2014 at 21:18:24 UTC, Daniele M. wrote:

 And then comes my next question: except for that malloc-hack,
 would it have been possible to write it in  safe D? I guess
 that if not, module(s) could have been made un- safe. Not
 saying that a similar separation of concerns was not possible
 in OpenSSL itself, but that D could have made it less
 development-expensive in my opinion.


 TDPL SafeD visions notwithstanding,  safe is very very limiting.

 I/O is forbidden so simple Hello Worlds are right out, let
 alone advanced socket libraries.


 I/O is not forbidden, it's just that writeln and friends
 currently can't be made safe, but that is being worked on AFAIK.
 While I/O usually goes through the OS, the system calls can be
 manually verified and made  trusted.


As the underlying OS calls are all C functions, there will always be  system
code involved in I/O, but in most cases, we should be able to wrap those
functions in D functions which are  trusted.

Regarldess, I would think that SSL could be implemented without sockets - that
is, all of its operations should be able to operate on arbitrary data
regardless of whether that data is sent over a socket or not. And if that's
the case, then even if the socket operations themselves had to be  system,
then everything else should still be able to be  safe.

Most of the problems with  safe stem either from library functions that don't
use it like they should, or because the compiler does not yet do a good enough
job with attribute inference on templated functions. Both problems are being
addressed, so the situation will improve over time. Regardless, there's
nothing fundamentally limited about  safe except for operations which are
actually unsafe with regards to memory, and any case where something isn't
 safe when it's actually memory safe should be and will be fixed (as well as
any situation which isn't memory safe but is considered  safe anyway - we do
unfortunately still have a few of those).

- Jonathan M Davis
May 05 2014
next sibling parent "Daniele M." <gdm85 iragan.com> writes:
On Monday, 5 May 2014 at 10:41:41 UTC, Jonathan M Davis via 
Digitalmars-d wrote:

 On Mon, 05 May 2014 10:24:27 +0000
 via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On Monday, 5 May 2014 at 09:32:40 UTC, JR wrote:

 On Sunday, 4 May 2014 at 21:18:24 UTC, Daniele M. wrote:

 And then comes my next question: except for that 
 malloc-hack,
 would it have been possible to write it in  safe D? I guess
 that if not, module(s) could have been made un- safe. Not
 saying that a similar separation of concerns was not 
 possible
 in OpenSSL itself, but that D could have made it less
 development-expensive in my opinion.


 TDPL SafeD visions notwithstanding,  safe is very very 
 limiting.

 I/O is forbidden so simple Hello Worlds are right out, let
 alone advanced socket libraries.


 I/O is not forbidden, it's just that writeln and friends
 currently can't be made safe, but that is being worked on 
 AFAIK.
 While I/O usually goes through the OS, the system calls can be
 manually verified and made  trusted.


 As the underlying OS calls are all C functions, there will 
 always be  system
 code involved in I/O, but in most cases, we should be able to 
 wrap those
 functions in D functions which are  trusted.

 Regarldess, I would think that SSL could be implemented without 
 sockets - that
 is, all of its operations should be able to operate on 
 arbitrary data
 regardless of whether that data is sent over a socket or not. 
 And if that's
 the case, then even if the socket operations themselves had to 
 be  system,
 then everything else should still be able to be  safe.

 Most of the problems with  safe stem either from library 
 functions that don't
 use it like they should, or because the compiler does not yet 
 do a good enough
 job with attribute inference on templated functions. Both 
 problems are being
 addressed, so the situation will improve over time. Regardless, 
 there's
 nothing fundamentally limited about  safe except for operations 
 which are
 actually unsafe with regards to memory, and any case where 
 something isn't
  safe when it's actually memory safe should be and will be 
 fixed (as well as
 any situation which isn't memory safe but is considered  safe 
 anyway - we do
 unfortunately still have a few of those).

 - Jonathan M Davis


You nailed it. If we wanted to translate the theoretical exercise 
into something real, it would be nice to have an implementation 
of PolarSSL that works on ring buffers only, then leave network 
layer integration to clients. Much cleaner separation of concerns.
May 05 2014
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/05/2014 12:41 PM, Jonathan M Davis via Digitalmars-d wrote:

 Regardless, there's
 nothing fundamentally limited about  safe except for operations which are
 actually unsafe with regards to memory


What does 'actually unsafe' mean?  safe will happily ban statements that 
will never 'actually' result in memory corruption.
May 06 2014
parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 06 May 2014 09:56:11 +0200
Timon Gehr via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On 05/05/2014 12:41 PM, Jonathan M Davis via Digitalmars-d wrote:

 Regardless, there's
 nothing fundamentally limited about  safe except for operations
 which are actually unsafe with regards to memory


 What does 'actually unsafe' mean?  safe will happily ban statements
 that will never 'actually' result in memory corruption.


 safe is supposed to ban any code where it can't prove that it can't access
memory that it shouldn't - be it to read from it to write to it - as well as
code that it can't prove will not result in other code accessing memory that
it shouldn't have access to. It's quite possible that some code which
shouldn't be being banned is currently being banned, and it may be the case
that we'll be stuck with some code being banned simply because we can't make
the compiler smart enough to know that it doesn't need to ban it (though
hopefully that's kept to a minimum). But  trusted is there precisely for the
situations where an operation is unsafe in the general case but is perfectly
safe in a specific case based on information that the compiler does not have
access to but the programmer does (such as knowing that certain values will
never be passed to the given function).

However, I don't think that there's much question that at this point  safe
isn't quite correct. There are definitely cases right now that are considered
 safe when they shouldn't be (e.g.
https://issues.dlang.org/show_bug.cgi?id=8838 ), and I expect that there are
also cases where code is considered  system when the compiler should consider
it  safe (though I don't know of any of those off the top of my head).

- Jonathan M Davis
May 06 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 2:32 AM, JR wrote:

 On Sunday, 4 May 2014 at 21:18:24 UTC, Daniele M. wrote:

 And then comes my next question: except for that malloc-hack, would it
 have been possible to write it in  safe D? I guess that if not,
 module(s) could have been made un- safe. Not saying that a similar
 separation of concerns was not possible in OpenSSL itself, but that D
 could have made it less development-expensive in my opinion.


 TDPL SafeD visions notwithstanding,  safe is very very limiting.

 I/O is forbidden so simple Hello Worlds are right out, let alone
 advanced socket libraries.


Sounds like a library bug. Has it been submitted? -- Andrei
May 05 2014
parent reply "JR" <zorael gmail.com> writes:
On Monday, 5 May 2014 at 15:01:05 UTC, Andrei Alexandrescu wrote:

 On 5/5/14, 2:32 AM, JR wrote:

 On Sunday, 4 May 2014 at 21:18:24 UTC, Daniele M. wrote:

 And then comes my next question: except for that malloc-hack, 
 would it
 have been possible to write it in  safe D? I guess that if 
 not,
 module(s) could have been made un- safe. Not saying that a 
 similar
 separation of concerns was not possible in OpenSSL itself, 
 but that D
 could have made it less development-expensive in my opinion.


 TDPL SafeD visions notwithstanding,  safe is very very 
 limiting.

 I/O is forbidden so simple Hello Worlds are right out, let 
 alone
 advanced socket libraries.


 Sounds like a library bug. Has it been submitted? -- Andrei


When mentioned in #d it was met with replies of "well 
*obviously*", so I chalked it up to an irreconcilability of 
 safe's. Perhaps I'm expecting too much of the subset.

My code certainly does no pointer arithmetic, but adding  safe: 
in select locations quickly shows that string operations like 
indexOf are unsafe, as is everything in concurrency (including 
thisTid), getopt, std.conv.to (to avoid casts), and all socket 
operations. All of those can throw, but I don't see how they can 
corrupt memory.

Apologies for the negativity. It's not that much of a deal, but 
your code will have to be very unreliant upon phobos to be 
completely  safe. I appreciate these threads gauging community 
reactions and I hope the mood will be lighter post-dconf, but at 
present there's still a sour taste left in my mouth after the 
final-by-default pivot.

(Manu and Thaut, please don't leave~ :<   )
May 06 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/6/14, 2:37 AM, JR wrote:

 Apologies for the negativity. It's not that much of a deal, but your
 code will have to be very unreliant upon phobos to be completely  safe.


It's a huge deal. Most of phobos should be  safe or  trusted! Please 
submit everything you find as bugs. Thanks! -- Andrei
May 06 2014
prev sibling next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Sunday, 4 May 2014 at 08:34:20 UTC, Daniele M. wrote:

 I have read this excellent article by David A. Wheeler:

 http://www.dwheeler.com/essays/heartbleed.html

 And since D language was not there, I mentioned it to him as a 
 possible good candidate due to its static typing and related 
 features.

 However, now I am asking the community here: would a D 
 implementation (with GC disabled) of OpenSSL have been free 
 from Heartbleed-type vulnerabilities? Specifically 
 http://cwe.mitre.org/data/definitions/126.html and 
 http://cwe.mitre.org/data/definitions/20.html as David mentions.

 I find this perspective very interesting, please advise :)


While D is a somewhat safer language by *default*, it makes it 
fairly easy to escape from the safe part of the language and 
write unsafe code (array bounds checking can be turned off even 
for  safe code). Seeing as the OpenSSL devs went as far as to 
write an a buggy, custom implementation of malloc for a speed 
gain, turning off array bounds checking and ignoring  safe seems 
like the first thing they would do. The only language I would 
really trust is one in which it is impossible to write unsafe 
code, because you can then know that the developers can't use 
such unsafe hacks, even if they wanted to.
May 04 2014
next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 5/4/2014 9:29 AM, Meta wrote:

 While D is a somewhat safer language by *default*, it makes it fairly
 easy to escape from the safe part of the language and write unsafe code


Yea, I'm finding that in some ways, D accidentally encourages 
 system/ trusted code. For example, if you need some sensitive data 
zeroed out when done, and for whatever reason you aren't able to just 
have it live on the stack (to use RAII), then you need RefCounted which 
AFAICS throws any chance of being  safe out the window.

Maybe RefCounted could somehow take advantage of Unique to provide  safe-ty?
May 04 2014
prev sibling next sibling parent reply "Daniele M." <gdm85 iragan.com> writes:
On Sunday, 4 May 2014 at 13:29:34 UTC, Meta wrote:

 On Sunday, 4 May 2014 at 08:34:20 UTC, Daniele M. wrote:

 I have read this excellent article by David A. Wheeler:

 http://www.dwheeler.com/essays/heartbleed.html

 And since D language was not there, I mentioned it to him as a 
 possible good candidate due to its static typing and related 
 features.

 However, now I am asking the community here: would a D 
 implementation (with GC disabled) of OpenSSL have been free 
 from Heartbleed-type vulnerabilities? Specifically 
 http://cwe.mitre.org/data/definitions/126.html and 
 http://cwe.mitre.org/data/definitions/20.html as David 
 mentions.

 I find this perspective very interesting, please advise :)


 While D is a somewhat safer language by *default*, it makes it 
 fairly easy to escape from the safe part of the language and 
 write unsafe code (array bounds checking can be turned off even 
 for  safe code). Seeing as the OpenSSL devs went as far as to 
 write an a buggy, custom implementation of malloc for a speed 
 gain, turning off array bounds checking and ignoring  safe 
 seems like the first thing they would do. The only language I 
 would really trust is one in which it is impossible to write 
 unsafe code, because you can then know that the developers 
 can't use such unsafe hacks, even if they wanted to.


You are right, devs would eventually abuse everything possible, 
although it would make it for sure more visible: you cannot 
advertize an un- safe library as  safe, although I agree that a 
lot depends from devs/users culture.
May 04 2014
parent Jacob Carlborg <doob me.com> writes:
On 04/05/14 23:20, Daniele M. wrote:


 You are right, devs would eventually abuse everything possible, although
 it would make it for sure more visible: you cannot advertize an un- safe
 library as  safe, although I agree that a lot depends from devs/users
 culture.


In D, you can at least statically determine which part of the code is 
 safe and un- safe.

-- 
/Jacob Carlborg
May 04 2014
prev sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sun, 04 May 2014 13:29:33 +0000
Meta via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 The only language I would
 really trust is one in which it is impossible to write unsafe
 code, because you can then know that the developers can't use
 such unsafe hacks, even if they wanted to.


Realistically, I think that you ultimately have to rely on the developers
doing a good job. Good tools help a great deal (including a programming
language that's safe by default while still generally being efficient), but if
you try and restrict the programmer such that they can only do things that are
guaranteed to be safe, I think that you're bound to make it impossible to do a
number of things, which tends to not only be very frustrating to the
programmers, but it can also make it impossible to get the performance that
you need in some circumstances.

So, while you might be able to better trust a library written in a language
that's designed to make certain types of problems impossible, I don't think
that it's realistic for that language to get used much in anything performance
critical like an SSL implementation.

Ultimately, I think that the trick is to make things as safe as they can be
without actually making it so that the programmer can't do what they need to
be able to do. And while, I don't think that D hit the perfect balance on that
one (e.g. we should have made  safe the default if we wanted that), I think
that we've done a good job of it overall - certainly far better than C or C++.

- Jonathan M Davis
May 04 2014
parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 5 May 2014 at 06:35:07 UTC, Jonathan M Davis via 
Digitalmars-d wrote:

 On Sun, 04 May 2014 13:29:33 +0000
 Meta via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 The only language I would
 really trust is one in which it is impossible to write unsafe
 code, because you can then know that the developers can't use
 such unsafe hacks, even if they wanted to.


 Realistically, I think that you ultimately have to rely on the 
 developers
 doing a good job. Good tools help a great deal (including a 
 programming
 language that's safe by default while still generally being 
 efficient), but if
 you try and restrict the programmer such that they can only do 
 things that are
 guaranteed to be safe, I think that you're bound to make it 
 impossible to do a
 number of things, which tends to not only be very frustrating 
 to the
 programmers, but it can also make it impossible to get the 
 performance that
 you need in some circumstances.

 So, while you might be able to better trust a library written 
 in a language
 that's designed to make certain types of problems impossible, I 
 don't think
 that it's realistic for that language to get used much in 
 anything performance
 critical like an SSL implementation.

 Ultimately, I think that the trick is to make things as safe as 
 they can be
 without actually making it so that the programmer can't do what 
 they need to
 be able to do. And while, I don't think that D hit the perfect 
 balance on that
 one (e.g. we should have made  safe the default if we wanted 
 that), I think
 that we've done a good job of it overall - certainly far better 
 than C or C++.

 - Jonathan M Davis


Sometimes I wonder how much money have C design decisions cost 
the industry in terms of anti-virus, static and dynamic analyzers 
tools, operating systems security enforcements, security research 
and so on.

All avoidable with bound checking by default and no implicit 
conversions between arrays and pointers.

--
Paulo
May 05 2014
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 05 May 2014 07:39:13 +0000
Paulo Pinto via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Sometimes I wonder how much money have C design decisions cost
 the industry in terms of anti-virus, static and dynamic analyzers
 tools, operating systems security enforcements, security research
 and so on.

 All avoidable with bound checking by default and no implicit
 conversions between arrays and pointers.


Well, a number of years ago, the folks who started the codebase of the larger
products at the company I work at insisted on using COM everywhere, because we
_might_ have to interact with 3rd parties, and they _might_ not want to use
C++. So, foolishly, they mandated that _nowhere_ in the codebase should any
C++ objects be passed around except by pointer. They then had manual reference
counting on top of that to deal with memory management. That decision has cost
us man _years_ in time working on reference counting-related bugs. Simply
using smart pointers instead would probably have saved the company millions.
COM may have its place, but forcing a whole C++ codebase to function that way
was just stupid, especially when pretty much none of it ever had to interact
directly with 3rd party code (and even if it had, it should have been done
through strictly defined wrapper libraries; it doesn't make sense that 3rd
parties would hook into the middle of your codebase).

Seemingly simple decisions can have _huge_ consequences - especially when that
decision affects millions of lines of code, and that's definitely the case
with some of the decisions made for C. Some of them may have be unavoidable
give the hardware situation and programming climate at the time that C was
created, but we've been paying for them ever since. And unfortunately, the way
things are going at this point, nothing will ever really overthrow C. We'll
have to deal with it on some level for a long, long time to come.

- Jonathan M Davis
May 05 2014
parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 5 May 2014 at 08:04:24 UTC, Jonathan M Davis via 
Digitalmars-d wrote:

 On Mon, 05 May 2014 07:39:13 +0000
 Paulo Pinto via Digitalmars-d <digitalmars-d puremagic.com> 
 wrote:

 Sometimes I wonder how much money have C design decisions cost
 the industry in terms of anti-virus, static and dynamic 
 analyzers
 tools, operating systems security enforcements, security 
 research
 and so on.

 All avoidable with bound checking by default and no implicit
 conversions between arrays and pointers.


 Well, a number of years ago, the folks who started the codebase 
 of the larger
 products at the company I work at insisted on using COM 
 everywhere, because we
 _might_ have to interact with 3rd parties, and they _might_ not 
 want to use
 C++. So, foolishly, they mandated that _nowhere_ in the 
 codebase should any
 C++ objects be passed around except by pointer. They then had 
 manual reference
 counting on top of that to deal with memory management. That 
 decision has cost
 us man _years_ in time working on reference counting-related 
 bugs. Simply
 using smart pointers instead would probably have saved the 
 company millions.
 COM may have its place, but forcing a whole C++ codebase to 
 function that way
 was just stupid, especially when pretty much none of it ever 
 had to interact
 directly with 3rd party code (and even if it had, it should 
 have been done
 through strictly defined wrapper libraries; it doesn't make 
 sense that 3rd
 parties would hook into the middle of your codebase).



So the decision was made and CComPtr<> and _com_ptr_t<> weren't 
used?!

I feel your pain. Well, now the codebase is WinRT ready. :)


 Seemingly simple decisions can have _huge_ consequences - 
 especially when that
 decision affects millions of lines of code, and that's 
 definitely the case
 with some of the decisions made for C. Some of them may have be 
 unavoidable
 give the hardware situation and programming climate at the time 
 that C was
 created, but we've been paying for them ever since. And 
 unfortunately, the way
 things are going at this point, nothing will ever really 
 overthrow C. We'll
 have to deal with it on some level for a long, long time to 
 come.

 - Jonathan M Davis



I doubt those decisions really made sense, given the other system 
programming languages at the time. Most of them did bounds 
checking, had no implicit conversions and operating systems were 
being written with them.

Algol 60 reference compiler did not allow disabling bounds 
checking at all, for example.

Quote from Tony Hoare's ACM award article[1]:
"A consequence of this principle is that every occurrence of
every subscript of every subscripted variable was on every 
occasion
checked at run time against both the upper and the lower declared
bounds of the array. Many years later we asked our customers 
whether
they wished us to provide an option to switch off these checks in 
the
interests of efficiency on production runs. Unanimously, they 
urged
us not to--they already knew how frequently subscript errors occur
on production runs where failure to detect them could be 
disastrous.
I note with fear and horror that even in 1980 language designers 
and
users have not learned this lesson. In any respectable branch of
engineering, failure to observe such elementary precautions would 
have
long been against the law."


[1]http://www.labouseur.com/projects/codeReckon/papers/The-Emperors-Old-Clothes.pdf
May 05 2014
prev sibling parent reply Etienne <etcimon gmail.com> writes:
On 2014-05-04 4:34 AM, Daniele M. wrote:

 I have read this excellent article by David A. Wheeler:

 http://www.dwheeler.com/essays/heartbleed.html

 And since D language was not there, I mentioned it to him as a possible
 good candidate due to its static typing and related features.

 However, now I am asking the community here: would a D implementation
 (with GC disabled) of OpenSSL have been free from Heartbleed-type
 vulnerabilities? Specifically
 http://cwe.mitre.org/data/definitions/126.html and
 http://cwe.mitre.org/data/definitions/20.html as David mentions.

 I find this perspective very interesting, please advise :)


I'm currently working on a TLS library using only D. I've shared the 
ASN.1 parser here: https://github.com/globecsys/asn1.d

The ASN.1 format allows me to compile the data structures to D from the 
tls.asn1 in the repo I linked to. It uses the equivalent of D template 
structures extensively with what's called an Information Object Class.

Obviously, when it's done I need a DER serializer/deserializer which I 
intend on editing MsgPackD, and then I can do a handshake (read a ASN.1 
certificate) and encrypt/decrypt AES/RSA using the certificate 
information and this cryptography library: 
https://github.com/apartridge/crypto .

I've never expected any help so I'm not sure what the licensing will be. 
I'm currently working on the generation step for the ASN.1 to D 
compiler, it's very fun to make a compiler in D.
May 05 2014
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
05-May-2014 18:59, Etienne пишет:

 On 2014-05-04 4:34 AM, Daniele M. wrote:

 I have read this excellent article by David A. Wheeler:

 http://www.dwheeler.com/essays/heartbleed.html

 And since D language was not there, I mentioned it to him as a possible
 good candidate due to its static typing and related features.

 However, now I am asking the community here: would a D implementation
 (with GC disabled) of OpenSSL have been free from Heartbleed-type
 vulnerabilities? Specifically
 http://cwe.mitre.org/data/definitions/126.html and
 http://cwe.mitre.org/data/definitions/20.html as David mentions.

 I find this perspective very interesting, please advise :)


 I'm currently working on a TLS library using only D. I've shared the
 ASN.1 parser here: https://github.com/globecsys/asn1.d


Cool, keep us posted.


 The ASN.1 format allows me to compile the data structures to D from the
 tls.asn1 in the repo I linked to. It uses the equivalent of D template
 structures extensively with what's called an Information Object Class.

 Obviously, when it's done I need a DER serializer/deserializer which I
 intend on editing MsgPackD, and then I can do a handshake (read a ASN.1
 certificate) and encrypt/decrypt AES/RSA using the certificate
 information and this cryptography library:
 https://github.com/apartridge/crypto .

 I've never expected any help so I'm not sure what the licensing will be.
 I'm currently working on the generation step for the ASN.1 to D
 compiler, it's very fun to make a compiler in D.


Aye, D seems to be a nice choice for writing compilers.

-- 
Dmitry Olshansky
May 05 2014
prev sibling next sibling parent reply "Daniele M." <gdm85 iragan.com> writes:
On Monday, 5 May 2014 at 14:59:13 UTC, Etienne wrote:

 On 2014-05-04 4:34 AM, Daniele M. wrote:

 I have read this excellent article by David A. Wheeler:

 http://www.dwheeler.com/essays/heartbleed.html

 And since D language was not there, I mentioned it to him as a 
 possible
 good candidate due to its static typing and related features.

 However, now I am asking the community here: would a D 
 implementation
 (with GC disabled) of OpenSSL have been free from 
 Heartbleed-type
 vulnerabilities? Specifically
 http://cwe.mitre.org/data/definitions/126.html and
 http://cwe.mitre.org/data/definitions/20.html as David 
 mentions.

 I find this perspective very interesting, please advise :)


 I'm currently working on a TLS library using only D. I've 
 shared the ASN.1 parser here: 
 https://github.com/globecsys/asn1.d

 The ASN.1 format allows me to compile the data structures to D 
 from the tls.asn1 in the repo I linked to. It uses the 
 equivalent of D template structures extensively with what's 
 called an Information Object Class.

 Obviously, when it's done I need a DER serializer/deserializer 
 which I intend on editing MsgPackD, and then I can do a 
 handshake (read a ASN.1 certificate) and encrypt/decrypt 
 AES/RSA using the certificate information and this cryptography 
 library: https://github.com/apartridge/crypto .

 I've never expected any help so I'm not sure what the licensing 
 will be. I'm currently working on the generation step for the 
 ASN.1 to D compiler, it's very fun to make a compiler in D.


This is a quite radical approach, I am very interested to see its 
development! Have you thought about creating an SSL/TLS 
implementations tester instead? With the compiled information I 
see this goal quite well in range.
May 05 2014
parent Etienne <etcimon gmail.com> writes:
On 2014-05-05 2:54 PM, Daniele M. wrote:

 Have you thought about creating an SSL/TLS implementations tester instead?


You mean testing existing TLS libraries using this information?

The advantages of using all-D is having zero-copy buffers that inline 
with the other layers of streams when built inside another D project. I 
can also add processor-specific assembler-code algorithms of AES and RSA 
from openSSL (optimizing the critical parts can put it on par or better 
in speed).

To answer the question about safety, the code is very modular and so 
when you decide to zero out memory of keys before/after 
serialization/deserialization or even for the buffers, it happens for 
everything regardless of the complexity of the application. It's 
definitely easier to make it safer!
May 05 2014
prev sibling parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Monday, 5 May 2014 at 14:59:13 UTC, Etienne wrote:

 On 2014-05-04 4:34 AM, Daniele M. wrote:

 I have read this excellent article by David A. Wheeler:

 http://www.dwheeler.com/essays/heartbleed.html

 And since D language was not there, I mentioned it to him as a 
 possible
 good candidate due to its static typing and related features.

 However, now I am asking the community here: would a D 
 implementation
 (with GC disabled) of OpenSSL have been free from 
 Heartbleed-type
 vulnerabilities? Specifically
 http://cwe.mitre.org/data/definitions/126.html and
 http://cwe.mitre.org/data/definitions/20.html as David 
 mentions.

 I find this perspective very interesting, please advise :)


 I'm currently working on a TLS library using only D. I've 
 shared the ASN.1 parser here: 
 https://github.com/globecsys/asn1.d

 The ASN.1 format allows me to compile the data structures to D 
 from the tls.asn1 in the repo I linked to. It uses the 
 equivalent of D template structures extensively with what's 
 called an Information Object Class.

 Obviously, when it's done I need a DER serializer/deserializer 
 which I intend on editing MsgPackD, and then I can do a 
 handshake (read a ASN.1 certificate) and encrypt/decrypt 
 AES/RSA using the certificate information and this cryptography 
 library: https://github.com/apartridge/crypto .

 I've never expected any help so I'm not sure what the licensing 
 will be. I'm currently working on the generation step for the 
 ASN.1 to D compiler, it's very fun to make a compiler in D.


I've been working on an ASN.1 parser and D code generator [0].
Didn't get the BER encoder working but..
I thought it may be a good idea mention it here.

[0] https://github.com/rikkimax/asn1
May 06 2014