So, You Want To Write Your Own Language?
January 21, 2014
written by Walter Bright
When Andrew Binstock suggested I write an article about this, I thought about all the sage advice I could give. Then I realized my advice would sound like any of those innumerable forgettable business self-help books. I doubt I'd be able to stand to read such tripe myself.
Then I thought maybe if I just wrote a brief history of how the D programming language came to be, people would find that interesting. Business case history articles are far more interesting than platitudes about “be excellent”. So here goes.
I've liked figuring out how things worked and designing them from a very early age, I loved the Tom Swift books. In an earlier age I could see myself designing locomotives or Schneider Trophy racers. My degree was in mechanical engineering. But mechanical engineering was frustrating because of the large expense involved with building anything, and my rather poor fabrication skills.
With programming, on the other hand, one could build the most intricate machines imaginable, at no expense and just needing access to a computer. Like many programmers, I started with writing games. The appeal of writing games is sort of a god complex - you get to create a whole world that slavishly follows your rules. With the game Empire not only did I create such a world, but spent endless hours figuring out how to make the computer operate the enemy armies. Even now, I still think of ways to improve it.
The trouble, though, was the computer strategy's thirst for computer power far exceeded the abilities of the machines available, and so I became interested in how the compiler optimized the code. Compilers seemed like utterly magical devices that transformed source into machine code - how did this sorcery actually work? This was one of the Great Mysteries Of The Universe until BYTE magazine published the source code to Tiny Pascal. I devoured every line of that program, and once mastered I felt I'd been given the password to the inner sanctum.
Some years went by and in the early 80's and I found myself as part of a programming team developing software for MS-DOS. We were using C because it was the only high level language we could find that actually worked on the PC. The other language implementations were unbelievably awful. Even the C compilers were execrable, but at least they were usable. But the code they generated was terrible, the optimization nonexistent. I had the idea I could write a better C compiler myself.
I confided this to a colleague, and he suggested going to lunch with him and meet the local C guru who'd give me some advice on how to proceed. We went and my friend explained my ambition to him, and his contemptuous response is still etched on my brain, “who the hell do you think you are thinking you can write a C compiler?” I have to retroactively thank him for that, because I found the desire to show him up to be powerfully motivating.
(I often read on the internet about how people are discouraged from doing this or that because of negative reactions from others. I must be weird because not much gets me going stronger than people telling me “you can't do that”. I suppose it runs in the family, my grandmother was a mechanic in the age when women weren't allowed to be, and her sister borrowed money from the bank and started her own successful business when women weren't allowed to do that, either. They both had reputations for doing whatever the hell they wanted to.)
I went on to implement the C compiler, known as Datalight C. True to my interest in optimization, it was the first on the PC to have a data flow optimizing compiler. Such was new enough that the compiler got into trouble in the computer magazine benchmarks because the optimizer figured out that the benchmarks did nothing and so deleted all that dead code - the journalist assumed my compiler was broken or cheating and Datalight C got a bad review. (This of course made me pretty mad, but there was no internet in those days where I could post a riposte.)
It was later reincarnated as Zortech C. By then there were many other C compilers on the PC (I think I counted 30 at one point) and I was looking around for a competitive edge, and found Bjarne Stroustrup's book on C++ in the bookstore. I thought “heh, add a couple of new keywords, and in a couple of months, and I'll have a C++ compiler!” Probably the understatement of the programming century. If I'd known what I was getting into, perhaps I would have believed the people who told me I couldn't do it.
Anyhow, the late 1980's was when lots of people were working on successors to C. You can even find references to one such called “D” on usenet of the time. But for various reasons, including the fact that Zortech had an inexpensive C++ compiler ready to go on the most popular platform of the day, C++ buried those other languages and dominated the 1990's of programming, and I was carried along with that boom.
Along the way (we're now in 199x), I also wrote a Java compiler that generated native code, and a Javascript compiler/interpreter. These products were unsuccessful. As an aside, I'll note that working on stuff that I needed has fared quite a bit better than working on stuff that I was told others need.
For example, I was out jogging one day with a programmer friend who said “You know, what the world is desperate for is a Java compiler that generates native code. You'll make a mint off of that! I use Java and this is really needed.” I told him that, coincidentally, I had written one and he could start using it right away. Of course he never did.
I invented the Empire game because it was the game I wanted to play. I developed a C compiler because I needed a better compiler to develop Empire (and other projects) with. The Java and Javascript compilers were other peoples' ideas.
But back to where D came from. Nobody could possibly work on compilers for 15+ years and not come up with ideas for language improvements. I tried out many language improvements in the C and C++ compilers, and they all fell flat. Nobody was interested in language extensions - they wanted standard compliant languages, and they were right. Unfortunately, getting ideas adopted by the standards committees is an arduous, multiyear bureaucratic process, a process I am not very patient with.
In 1999 I thought I was retired, and spent about 6 weeks watching TV until I thought I'd go mad. Time to get back to living. Whining about perceived problems with existing languages had gone on long enough; it was time to power up the machine shop. When tackling a problem like this, I am always reminded of Gimli the dwarf: “Certainty of death. Small chance of success. What are we waiting for?” Why not? At least I'll go down sword in hand fighting the glorious fight.
So D started out, stumbling along as a solo project. I've never been comfortable programming in a language I hadn't written a compiler for (I know, another oddity), but I'd had enough experience implementing various languages that I was pretty confident I could get D working. I also had the huge advantage of the ecosystem of the existing C++ compiler to work with.
A couple years later, D first appeared on Slashdot, and suddenly it started attracting users and collaborators. Turns out I am hardly the unique person in what I want from a language that everyone told me I was! Along with collaborators, D also grew dramatically in ambition. One great thing about the internet is one can have collaborators that are all over the world. It's so amazing that we can all be in constant realtime contact with each other, and it wasn't until a few months ago at DConf 2013 that we even knew what each other looked like. (This is one of the greatest aspects of the internet revolution - one can work successfully with others while knowing nothing about their sex, age, looks, race, religion, language, culture, disabilities, histories, etc. It's as pure a meritocracy as it gets. Only your ideas, contributions, and how you present yourself matter.)
D wouldn't exist without the internet. How else could the disparate enthusiasts have ever gotten together? The rise of collaborative tools like github and bugzilla have been absolutely critical to D's development (along with the open source model, of course!).
But I've had to learn things, too, often the hard way.
- My natural tendency is to work solo on things. My work performance reviews at the various jobs I've had usually included comments like “Walter needs to learn to work better with others.” My desire to control everything nearly wrecked the D community at one point, and I've had to change.
- I've had to learn how to manage a project where people are all volunteers and work for free on it. Since I don't pay anybody anything, I can't tell anybody to do anything. I have to find other ways. I rank pretty close to “nerd” on personality tests, this is not natural for me. This is actually a fascinating challenge.
- On the D forums, proposals for new language features come every day, and have since the forums started. I have to say no a lot, which is hard to do, and harder for the proposers to hear.
- Management is a hard job, and I'm not very suited for it. I didn't do a very good job of it at Zortech, and at various company jobs I was (wisely) never promoted into management. But with D this couldn't be avoided - I have to learn how to do it, and do it reasonably well. I'm pretty motivated to try and be a better manager, because I want D to succeed.
- As my friends know, I love to debate. I'm a participant in many internet flame wars. But I can't do that with D, because nobody ever changes their mind as a result of a flame war, no matter how wrong they are (!). But all this experience actually lends a curious advantage - I can tell when a conversation thread is starting to slip into unproductive argument, and can take steps to lower the voltage and defuse it. I still have to fight the urge to go “shields up! phasers on vaporize!”
Of course, as anyone on the D forums will agree, this is a work in progress for me.
D has been and continues to be an amazing experience - it's borrowed the best ideas from many other languages, and is the summation of enormous and selfless effort by many, many collaborators from all over the world, who can all be justly proud of what they've accomplished.
There has been one result in all this that I especially treasure - I love programming in D. It's the language I wish I'd always had available. I'm glad I never listened to the naysayers, the debbie downers, and of course that nameless C guru from long ago, who I owe thanks to.
So, you want to write your own language? All I can say is: certainty of death. Small chance of success. What are you waiting for?
Acknowledgements
Thanks to Andrei Alexandrescu for his helpful suggestions on this article.