www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Re: Go: A new system programing language

reply Mike Hearn <mike plan99.net> writes:
 Whenever I give a talk on D, I start out by asking the audience who has 
 heard of it. In the last few years, nearly everyone raises their hand.

For what it's worth there's a segment of the Google engineering community that would love to use D internally (I'm one of them). Go is still very new and isn't used much here. Actually, I don't know of anything that it's used for off the top of my head. Google is based on C++ and Java with Python being used for a lot of glue/admin type stuff. Personally, I'd rather use D2 than Go for my next project - especially given the c++ compatibility. With a few minor improvements (eg namespace support) that'd save a lot of time. But I don't know of anybody doing the necessary work to make it usable here, and besides, there's a lot of resistance to introducing new languages without a really good reason. D2 is close to being a Really Good Reason all on its own IMO, but the inertia is huge. How do you find a code reviewer for something written in D? What about compiler quality? Who will write the style guideline and do readability reviews? (you have to pass a "readability" review for a language before you're allowed to check in code written with it).
Nov 12 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Mike Hearn wrote:
 Whenever I give a talk on D, I start out by asking the audience who
 has heard of it. In the last few years, nearly everyone raises
 their hand.

For what it's worth there's a segment of the Google engineering community that would love to use D internally (I'm one of them). Go is still very new and isn't used much here. Actually, I don't know of anything that it's used for off the top of my head. Google is based on C++ and Java with Python being used for a lot of glue/admin type stuff. Personally, I'd rather use D2 than Go for my next project - especially given the c++ compatibility. With a few minor improvements (eg namespace support) that'd save a lot of time. But I don't know of anybody doing the necessary work to make it usable here, and besides, there's a lot of resistance to introducing new languages without a really good reason. D2 is close to being a Really Good Reason all on its own IMO, but the inertia is huge. How do you find a code reviewer for something written in D? What about compiler quality? Who will write the style guideline and do readability reviews? (you have to pass a "readability" review for a language before you're allowed to check in code written with it).

This is very interesting to hear. If there is anything I can do to help any adoption of D at Google, please let me know.
Nov 12 2009
next sibling parent reply Mike Hearn <mike plan99.net> writes:
Walter Bright Wrote:
 This is very interesting to hear.
 
 If there is anything I can do to help any adoption of D at Google, 
 please let me know.

Thanks, I will. Off hand the only thing I can think of is "finish D2 and commit to it being supported/stable for a few years" :-) I realize there are already interesting ideas kicking around for D3 but if they can be integrated in a backwards compatible way .... Anyway, the amount of work required to integrate a new general purpose language at Google is pretty big, it'd probably be several 20% projects worth of work over a period of about half a year. For illustration here are a few of the tasks that'd be either required or strongly wanted (some stuff is still confidential and I can't mention). I'm not suggesting this list is typical of large companies but it might prove interesting anyway. - Integration of a compiler with our in-house build system (proprietary). If it's GCC based that's better. - Some kind of standard unit testing framework that the testing infrastructure knows how to drive. If it works similar to the ones we use for C++ that's better: http://code.google.com/p/googletest/ http://code.google.com/p/googlemock/ - Protocol buffer support http://code.google.com/p/protobuf/ - google style command line flags implementation: http://code.google.com/p/google-gflags/ - Style guide: It's an open question whether this would purely be how code looks or like the C++ guide also restrict some features. D is a very feature rich language even compared to C++. I'm not sure if expecting developers and reviewers to be familiar with *every* feature is realistic or worth the cost. http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml - The aforementioned readability reviews, reviewers need to be found (they are volunteers) and trained, Perforce needs to be taught about the new filetypes: http://1-800-magic.blogspot.com/2008/01/after-8-months-no-longer-noogler.html - Training materials so engineers who haven't encountered D before can get up to speed quickly, at least to the level where they can do code reviews even if they aren't a master of all the features. - By default emacs/vim at google import customizations for our environment, integrating a d-mode with that would be nice. - Bindings to the core libraries for accessing things like GFS/BigTable and doing RPCs: optional but the utility of any language that can't use them is limited. C++ compatibility would certainly make this easier but SWIG integration would make it even easier still, as we already have SWIG set up for Python. Figuring out an easy way to integrate the garbage collected world with the manually managed world is also a trick. I presume the Python bindings already figured this out but it'd obviously be nice if the bindings could be as thin as possible. Of all those, the last would be the most work. The google "standard library" is a huge collection of robust and well written code - everything from well known stuff like BigTable down to custom threading and malloc libraries. The nice things D brings to the table can't compensate for the loss of that codebase. I haven't tried binding stuff into D, although given that it's got some C/C++ compatibility it's way ahead of Python and Java already.
Nov 17 2009
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Mike Hearn:

With a few minor improvements (eg namespace support) that'd save a lot of time.<

This change to D language is not planned. You can explain why you think namespace support is useful (and you can explain those other minor improvements too).
- Integration of a compiler with our in-house build system (proprietary). If
it's GCC based that's better.<

There's a D compiled based on GCC, but at the moment the best D1 compiler is LDC, based on LLVM, especially if you care for top performance of the binary.
If it works similar to the ones we use for C++ that's better:<

I hope D will do better here :-) But it will take time.
D is a very feature rich language even compared to C++.<

But usually D features are designed to be safer, higher level, less tricky and more handy, and sometimes slower. Go designers have removed almost everything, so when you use Go you don't need a restrictive style guide like Google C++ one that forbids people to use several language features :-)
 - By default emacs/vim at google import customizations for our environment,
integrating a d-mode with that would be nice.<

This is something that probably needs to be done regardless possible D usage at Google.
I haven't tried binding stuff into D, although given that it's got some C/C++
compatibility it's way ahead of Python and Java already.<

C compatibility of D is good. C++ compatibility is currently limited by design. If Google hires Walter he may use 50% of his free time developing D2 (as Guido has 50% for Python itself, and 50% developing Python code). Bye, bearophile
Nov 17 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Bill Baxter wrote:
 On Tue, Nov 17, 2009 at 7:07 AM, bearophile <bearophileHUGS lycos.com> wrote:
 Mike Hearn:

 With a few minor improvements (eg namespace support) that'd save a lot of
time.<


To be more specific, Walter thinks is good enough namespace support (in addition to the fact that every module is a namespace).

Essentially all the uses of C++ namespaces I've seen map nicely onto D's module system.
Nov 17 2009
parent Walter Bright <newshound1 digitalmars.com> writes:
Bill Baxter wrote:
 It's pretty common to see bits of a namespace get spread across many files.
 There's no way to do that in D.  But I wouldn't go so far as to say
 it's a necessity.

Yeah, Andrei and I talked about that and if it was a worthwhile capability. We decided it was a misfeature in that it would subvert encapsulation. For example, if any source file can add functions to a namespace, how can function overloading be reliable? C++ namespaces are often considered a failure, and this is one of the reasons.
Nov 17 2009
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
Mike Hearn Wrote:
 
 - Bindings to the core libraries for accessing things like GFS/BigTable and
doing RPCs: optional but the utility of any language that can't use them is
limited. C++ compatibility would certainly make this easier but SWIG
integration would make it even easier still, as we already have SWIG set up for
Python. Figuring out an easy way to integrate the garbage collected world with
the manually managed world is also a trick. I presume the Python bindings
already figured this out but it'd obviously be nice if the bindings could be as
thin as possible.

RPC should be a part of messaging support, though possibly not right away. Socket IO in Phobos kind of stinks right now so that would need an overhaul first.
 Of all those, the last would be the most work. The google "standard library"
is a huge collection of robust and well written code - everything from well
known stuff like BigTable down to custom threading and malloc libraries. The
nice things D brings to the table can't compensate for the loss of that
codebase. I haven't tried binding stuff into D, although given that it's got
some C/C++ compatibility it's way ahead of Python and Java already.

Sounds like Google may want to use a custom runtime. It's pretty trivial to replace core.thread from a project perspective, but dropping in a custom thread implementation could take some work--the GC integration is tricky.
Nov 17 2009
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Mike Hearn wrote:
 - Bindings to the core libraries for accessing things like
 GFS/BigTable and doing RPCs: optional but the utility of any language
 that can't use them is limited. C++ compatibility would certainly
 make this easier but SWIG integration would make it even easier
 still, as we already have SWIG set up for Python. Figuring out an
 easy way to integrate the garbage collected world with the manually
 managed world is also a trick. I presume the Python bindings already
 figured this out but it'd obviously be nice if the bindings could be
 as thin as possible.
 
 
 Of all those, the last would be the most work. The google "standard
 library" is a huge collection of robust and well written code -
 everything from well known stuff like BigTable down to custom
 threading and malloc libraries. The nice things D brings to the table
 can't compensate for the loss of that codebase. I haven't tried
 binding stuff into D, although given that it's got some C/C++
 compatibility it's way ahead of Python and Java already.

Of course, D has direct access to any and all C code, being binary compatible with the C ABI. D, in fact, relies on being linked with the standard C runtime library and startup code. D2 has limited access to the C++ ABI: . name mangling (i.e. global function overloading) . single inheritance classes . binary compatibility with C++ function calling conventions I don't know the ABI for the Google libraries, but this should help. Mixing D's gc world with manually managed memory isn't hard, as long as the following rules are followed: 1. don't allocate in one language and expect to free in another 2. keep a 'root' to all gc allocated data in the D side of the fence (otherwise it may get collected)
Nov 17 2009
parent reply Sean Kelly <sean invisibleduck.org> writes:
Walter Bright Wrote:
 
 Mixing D's gc world with manually managed memory isn't hard, as long as 
 the following rules are followed:
 
 1. don't allocate in one language and expect to free in another
 2. keep a 'root' to all gc allocated data in the D side of the fence 
 (otherwise it may get collected)

This may actually work in D 2.0. core.thread has thread_attachThis() to make the D GC aware of an external thread, and gc_malloc() is an extern (C) function. I haven't tested this extensively however, so if you're keen to try it, please let me know if there are any problems.
Nov 17 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Sean Kelly wrote:
 Walter Bright Wrote:
 Mixing D's gc world with manually managed memory isn't hard, as
 long as the following rules are followed:
 
 1. don't allocate in one language and expect to free in another 2.
 keep a 'root' to all gc allocated data in the D side of the fence 
 (otherwise it may get collected)

This may actually work in D 2.0. core.thread has thread_attachThis() to make the D GC aware of an external thread, and gc_malloc() is an extern (C) function. I haven't tested this extensively however, so if you're keen to try it, please let me know if there are any problems.

There's a more fundamental problem. There is simply no reliable way to find all the static data segments in a program. I talked with Hans Boehm about this, he uses some horrific kludges to try and do it in the Boehm gc. This problem is on every OS I've checked. Windows has the bizarrely useless ability to find the beginning of your TLS data segments, but not the end! Useless. The D runtime knows about D's allocated segments because I fixed the object file generation to provide that information. But if it's some other compiler, it's hosed. Therefore, one has to keep on the D side of the fence roots to all data structures passed to foreign languages. It's pretty easy to do, just don't forget it!
Nov 17 2009
parent Sean Kelly <sean invisibleduck.org> writes:
Walter Bright Wrote:

 Sean Kelly wrote:
 Walter Bright Wrote:
 Mixing D's gc world with manually managed memory isn't hard, as
 long as the following rules are followed:
 
 1. don't allocate in one language and expect to free in another 2.
 keep a 'root' to all gc allocated data in the D side of the fence 
 (otherwise it may get collected)

This may actually work in D 2.0. core.thread has thread_attachThis() to make the D GC aware of an external thread, and gc_malloc() is an extern (C) function. I haven't tested this extensively however, so if you're keen to try it, please let me know if there are any problems.

There's a more fundamental problem. There is simply no reliable way to find all the static data segments in a program. I talked with Hans Boehm about this, he uses some horrific kludges to try and do it in the Boehm gc. This problem is on every OS I've checked.

Yeah, that's a problem. I've looked at this part of the Boehm collector too and it's absolutely horrifying. You'd think with the popularity (and history) of GC that there would be some API-level way to do this.
Nov 18 2009
prev sibling next sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Tue, Nov 17, 2009 at 7:07 AM, bearophile <bearophileHUGS lycos.com> wrote:
 Mike Hearn:

With a few minor improvements (eg namespace support) that'd save a lot of time.<

This change to D language is not planned. You can explain why you think namespace support is useful (and you can explain those other minor improvements too).

To be more specific, Walter thinks is good enough namespace support (in addition to the fact that every module is a namespace). struct Namespace { static: // ... } If you need something more, then it would be great if you could explain it. --bb
Nov 17 2009
prev sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Tue, Nov 17, 2009 at 4:12 PM, Walter Bright
<newshound1 digitalmars.com> wrote:
 Bill Baxter wrote:
 On Tue, Nov 17, 2009 at 7:07 AM, bearophile <bearophileHUGS lycos.com>
 wrote:
 Mike Hearn:

 With a few minor improvements (eg namespace support) that'd save a lot
 of time.<

This change to D language is not planned. You can explain why you think namespace support is useful (and you can explain those other minor improvements too).

To be more specific, Walter thinks is good enough namespace support (in addition to the fact that every module is a namespace).

Essentially all the uses of C++ namespaces I've seen map nicely onto D's module system.

It's pretty common to see bits of a namespace get spread across many files. There's no way to do that in D. But I wouldn't go so far as to say it's a necessity. --bb
Nov 17 2009