digitalmars.D - Are we getting better at designing programming languages?
- Jonathan A Dunlap (2/2) Jul 23 2013 Check it out: D is listed rather favorably on the chart..
- H. S. Teoh (11/13) Jul 23 2013 I'm skeptical about using LOC/commit as a metric for measuring
- =?UTF-8?B?QWxpIMOHZWhyZWxp?= (66/71) Jul 23 2013 Agreed. How many lines of code is the following?
- Kagamin (2/6) Jul 25 2013 Huh, never seen such pattern in C. I just return error code.
- H. S. Teoh (27/35) Jul 25 2013 I think what he meant is code like this:
- monarch_dodra (4/19) Jul 26 2013 Ditto.
- Tofu Ninja (4/6) Jul 23 2013 From my understanding of the article, lower on the graph means
- Peter Williams (6/12) Jul 23 2013 You're right. And the consequence is that D didn't do very well at all
- Brad Anderson (4/6) Jul 23 2013 Conversation from the last time Redmonk's Expressiveness index
- JS (25/25) Jul 26 2013 I think the next step in languages it the mutli-level
- Kagamin (2/2) Jul 26 2013 Syntax matters because it shouldn't be an obstacle. And D has
- H. S. Teoh (251/273) Jul 26 2013 There is much value to be had for working with the minimum possible
- JS (140/530) Jul 27 2013 I used to program in assembly and loved it. The problem was that
Check it out: D is listed rather favorably on the chart.. http://redmonk.com/dberkholz/2013/07/23/are-we-getting-better-at-designing-programming-languages/
Jul 23 2013
On Tue, Jul 23, 2013 at 06:24:14PM +0200, Jonathan A Dunlap wrote:Check it out: D is listed rather favorably on the chart.. http://redmonk.com/dberkholz/2013/07/23/are-we-getting-better-at-designing-programming-languages/I'm skeptical about using LOC/commit as a metric for measuring expressiveness. First of all, does it take comments into account? 'cos if it does, then it's basically a useless metric. It's also too easily influenced by developer practices: smaller, more frequent commits vs. larger, less frequent commits. And it's biased by quality of code: just because a piece of code is buggy and has many 1-line fixes applied to it, says nothing about the expressiveness of the language itself. T -- There are two ways to write error-free programs; only the third one works.
Jul 23 2013
On 07/23/2013 09:47 AM, H. S. Teoh wrote:On Tue, Jul 23, 2013 at 06:24:14PM +0200, Jonathan A Dunlap wrote:http://redmonk.com/dberkholz/2013/07/23/are-we-getting-better-at-designing-programming-languages/Check it out: D is listed rather favorably on the chart..I'm skeptical about using LOC/commit as a metric for measuring expressiveness.Agreed. How many lines of code is the following? import std.algorithm; import std.range; import std.traits; import std.conv; auto evensJoined(R)(R r) if (isNumeric!(ElementType!R)) in { assert(r.length > 3); } out { // I don't know... } body { return r .filter!(a => !(a % 2)) .map!(a => a.to!string) .joiner(","); } unittest { assert(evensJoined([ 0, 1, 2, 3, 4 ]).array == "0,2,4"); } void main() {} My answer is 1 line of code. Also, comparing C, C++, and D, the three languages that I know well, I am shocked that C does not get at least twice as many lines as the others. My hunch is that that C code is buggy code. First of all, C does not have exceptions so every single thing must be done as two lines: err = do_something(); goto_finally_if_error(err); err = do_something_else(); goto_finally_if_error(err); Additionally, C does not have constructors and destructors so every object and variable must be properly and explicitly initialized and cleaned up. Further, since the return value must be reserved for the error code, the results must be returned as an out-parameter: // (Sorry, I haven't compiled this C code) int foo(int arg, int * ret_result) { int err = 0; int * p = NULL; MyType * m = NULL; // Function precondition checks if (arg < 42) { goto finally; } goto_finally_if_null(ret_result); *ret_result = 0; p = calloc(/* ... */); goto_finally_if_null(p); err = make_MyType(arg, p, &m); goto_finally_if_error(err); goto_finally_if_null(m); // use m... *ret_result = m.some_member; finally: free(p); destroy_MyType(&m); return err; } Yeah, I think the article is looking at suboptimal C code... Ali
Jul 23 2013
On Tuesday, 23 July 2013 at 18:14:10 UTC, Ali Çehreli wrote:
First of all, C does not have exceptions so every single thing
must be done as two lines:
err = do_something();
goto_finally_if_error(err);
Huh, never seen such pattern in C. I just return error code.
Jul 25 2013
On Fri, Jul 26, 2013 at 06:36:21AM +0200, Kagamin wrote:On Tuesday, 23 July 2013 at 18:14:10 UTC, Ali Çehreli wrote:I think what he meant is code like this: #define OK 0 #define ERR -1 void myfunc(int x, int y, int z) { void *buf = malloc(100); int err; if ((err = fun1(x,y,z)) != OK) goto EXIT; if ((err = fun2(x,y,z)) != OK) goto EXIT; if ((err = fun3(x,y,z)) != OK) goto EXIT; if ((err = fun4(x,y,z)) != OK) goto EXIT; /* Finally finished! */ err = OK; EXIT: /* Do cleanup */ free(buf); return err; } I have to write code like this every day at work, and it's a royal pain in the neck. T -- The best way to destroy a cause is to defend it poorly.First of all, C does not have exceptions so every single thing must be done as two lines: err = do_something(); goto_finally_if_error(err);Huh, never seen such pattern in C. I just return error code.
Jul 25 2013
"monarch_dodra" <monarchdodra gmail.com> On Friday, 26 July 2013 at 05:12:55 UTC, H. S. Teoh wrote:On Fri, Jul 26, 2013 at 06:36:21AM +0200, Kagamin wrote:Ditto. I think it is a pretty common pattern for legacy C programs. Thank god for RAII.On Tuesday, 23 July 2013 at 18:14:10 UTC, Ali Çehreli wrote:I think what he meant is code like this: [SNIP] I have to write code like this every day at work, and it's a royal pain in the neck. TFirst of all, C does not have exceptions so every single thing must be done as two lines: err = do_something(); goto_finally_if_error(err);Huh, never seen such pattern in C. I just return error code.
Jul 26 2013
On Tuesday, 23 July 2013 at 16:24:15 UTC, Jonathan A Dunlap wrote:Check it out: D is listed rather favorably on the chart.. http://redmonk.com/dberkholz/2013/07/23/are-we-getting-better-at-designing-programming-languages/From my understanding of the article, lower on the graph means more 'expressive' and higher means less 'expressive'? Correct me if i am wrong.
Jul 23 2013
On 24/07/13 04:02, Tofu Ninja wrote:On Tuesday, 23 July 2013 at 16:24:15 UTC, Jonathan A Dunlap wrote:You're right. And the consequence is that D didn't do very well at all on the chart. Whether the metric used to measure "expressiveness" is valid is another question? My personal view would be that it's a highly dubious metric. PeterCheck it out: D is listed rather favorably on the chart.. http://redmonk.com/dberkholz/2013/07/23/are-we-getting-better-at-designing-programming-languages/From my understanding of the article, lower on the graph means more 'expressive' and higher means less 'expressive'? Correct me if i am wrong.
Jul 23 2013
On Tuesday, 23 July 2013 at 16:24:15 UTC, Jonathan A Dunlap wrote:Check it out: D is listed rather favorably on the chart.. http://redmonk.com/dberkholz/2013/07/23/are-we-getting-better-at-designing-programming-languages/Conversation from the last time Redmonk's Expressiveness index came up: http://forum.dlang.org/post/spgnwpchvtbfqhalpjhn forum.dlang.org
Jul 23 2013
I think the next step in languages it the mutli-level abstraction. Right now we have the base level core programming and the preprocessing/template/generic level above that. There is no reason why language can't/shouldn't keep going. The ability to control and help the compiler do it's job better is the next frontier. Analogous to how C++ allowed for abstraction of data, template allow for abstraction of functionality, we then need to abstract "templates"(or rather meta programming). Unfortunately each level is less useful and more complex to implement but I think it will open new doors once done(similar to how oop changes the face of programing). For example, why are there built in types? There is no inherit reason this is so except this allows compilers to achieve certain performance results... but having a higher level of abstraction of meta programming should allow us to bridge the internals of the compiler more effectively. I don't see anything like this happening so depending on your scale, I don't think we are getting better, but just chasing our tails... how many more languages do we need that just change the syntax of C++? Why do people think syntax matters? Semantics is what is important but there seems to be little focus on it. Of course, we must express semantics through syntax so for practical purposes it mattes to some degree.... But not nearly as much as the number of programming languages suggest.
Jul 26 2013
Syntax matters because it shouldn't be an obstacle. And D has more features than C and C++ - more, different semantics.
Jul 26 2013
On Fri, Jul 26, 2013 at 03:02:32PM +0200, JS wrote:I think the next step in languages it the mutli-level abstraction. Right now we have the base level core programming and the preprocessing/template/generic level above that. There is no reason why language can't/shouldn't keep going. The ability to control and help the compiler do it's job better is the next frontier. Analogous to how C++ allowed for abstraction of data, template allow for abstraction of functionality, we then need to abstract "templates"(or rather meta programming).There is much value to be had for working with the minimum possible subset of features that can achieve what you want with a minimum of hassle. The problem with going too far with abstraction is that you start living in an imaginary idealistic dreamworld that has nothing to do with how the hardware actually implements the stuff. You start writing some idealistic code and then wonder why it doesn't work, or why performance is so poor. As Knuth once said: By understanding a machine-oriented language, the programmer will tend to use a much more efficient method; it is much closer to reality. -- D. Knuth People who are more than casually interested in computers should have at least some idea of what the underlying hardware is like. Otherwise the programs they write will be pretty weird. -- D. Knuth If I ever had the chance to teach programming, my first course would be assembly language programming, followed by C, then by other languages, starting with a functional language (Lisp, Haskell, or Concurrent Clean), then other imperative languages, like Java. (Then finally I'll teach them D and tell them to forget about the others. :-P) Will I expect my students to write large applications in assembly language or C? Nope. But will I require them to pass the final exam in assembly language? YOU BETCHA. I had the benefit of learning assembly while I was still a teenager, and I can't tell you how much that experience has shaped my programming skills. Even though I haven't written a single line of assembly for at least 10 years, understanding what the machine ultimately runs gave me deep insights into why certain things are done in a certain way, and how to take advantage of that. It helps you build *useful* abstractions that map well to the underlying machine code, which therefore gives you good performance and overall behaviour while providing ease of use. By contrast, my encounters with people who grew up with Java or Pascal consistently showed that most of them haven't the slightest clue how the machine even works, and as a result, just like Knuth said, they tend to have some pretty weird ideas about how to write their programs. They tend to build messy abstractions or idealistic abstractions that don't map well to the underlying hardware, and as a result, their programs are often bloated, needlessly complex, and run poorly. [...]For example, why are there built in types?You need to learn assembly language to understand the answer to that one. ;-)There is no inherit reason this is so except this allows compilers to achieve certain performance results...Nah... performance isn't the *only* reason. But like I said, you need to understand the foundations (i.e., assembly language) before you can understand why, to use a physical analogy, you can't just freely move load-bearing walls around.but having a higher level of abstraction of meta programming should allow us to bridge the internals of the compiler more effectively.Andrei mentions several times in TDPL that we programmers don't like artificial distinctions between built-in types and user-defined types, and I agree with that sentiment. Fortunately, things like alias this and opCast allow us to define user-defined types that, for all practical purposes, behave as though they were built-in types. This is a good thing, and we should push it to the logical conclusion: to allow user-defined types to be optimized in analogous ways to built-in types. That is something I've always found lacking in the languages I know, and something I'd love to explore, but given that we're trying to stabilize D2 right now, it isn't gonna happen in the near future. Maybe if we ever get to D3... Nevertheless, having said all that, if you truly want to make the machine dance, you gotta sing to its tune. In the old days, the saying was that premature optimization is the root of all evils. These days, I'd like to say, premature *generalization* is the root of all evils. I've seen software that suffered from premature generalization... It was a system that was essentially intended to be a nice interface to a database, with some periodic background monitoring functions. The person(s) who designed it decided to build this awesome generic framework with all sorts of fancy features. For example, users don't have to understand what SQL stands for, yet they can formulate complex queries by means of nicely-abstracted OO interfaces. Hey, OO is all the rage these days, so what can be better than to wrap SQL in OO in such a way that the user wouldn't even know it's SQL underneath? I mean, what if we wanted to switch to, oh, Berkeley DB one of these days?! But abstracting a database isn't good enough. There's also this incredible generic framework that handles timers and events, such that you don't have to understand what an event loop is and you can write event-driven code, just like that. Oh, and to run all of these complicated fancy features, we have to put it inside its own standalone daemon, so that if it crashes, we can use another super-powerful generic framework to handle crashes and automatically restart so that the user doesn't even have to know the database engine is crashing underneath him; the daemon will pick up the query and continue running it after it restarts! Isn't that cool? But of course, since it runs as a separate daemon, we have to use IPC to interface it with user code. It all makes total sense! But since it provides so much functionality in such generic terms, we need to auto-generate header files that wrap around its API functions, so that user code doesn't have to know IPC is happening underneath the hood! So we'll write this generic script that auto-generates 20,000-line C++ files (with the respective header files) containing 8000+ wrapper functions that basically bundles the function arguments up and sends it over the IPC link to the master daemon that Does Everything(tm). Whaddya mean, it's a problem that these monstrous files take 30 minutes *each* to compile? We're talking generic framework here! Go take a coffee break, while we work magic behind your backs so that you don't even have to know how to use the system, before you can start using its API! What's that you say? The daemon is always crashing? Not a problem! We'll just write this super-powerful, super-generic templated system that abstracts away the dirty details of OS signal-handling, and use that as a generic framework to auto-restart our daemon when it crashes! Oh, and since performance isn't as good as we'd hoped for, we're now gonna use threading to boost throughput! So we're gonna write another super powerful, super-generic framework that handles concurrency locks, mutexes, and everything -- it's called RAII, you see, you instantiate this guard object at the beginning of every function, and it automatically handles reference counting and releases the lock at the end. What's that you say? We have deadlocks in our system? Well, not a problem! We'll ... er... uhm... ahem... well, I guess to work around this deadlock, we'll have to make some exceptions to the generic API, so if you call this workaround function before your code, then it will return an RAII object that will work magic behind your back that somehow eliminates the deadlock by hacking past the 6 layers of abstraction we've built up over the last year's worth of coding! What's that? You said you have a database query that you don't know how to formulate using our super-generic OO-powered abstractions? Not a problem! We'll just add another object here and ... uh... uhm... ahem, well let's see. The guy who originally wrote the database abstraction class hierarchy's transferred to another project, and we don't really understand what he did. But have no fear! We'll just add this new API that lets you write SQL directly, so that you can do any query you want! Yeah you'll have to know SQL, and yeah we're breaking the 6 layers of supercool OO abstractions, but hey, you wanted to do stuff the generic framework couldn't do, right? What did you say? Oh, you need to add a new feature to the daemon? Not a problem, we'll just add another abstraction over this thing here, and ... uh... uhm... ahem... well you see, the guy who wrote this part of the code's left the company, and we've no idea what he did, 'cos it's 6 layers of abstraction deep and we don't really have the time to go digging into it. So tell ya what. We'll just add this flag over here, that you can use to temporarily turn off that part of the daemon's built-in functionality that's interfering with what you're trying to do, and so you can just write your own code here, do your stuff, then turn the flag off after you're done. See? No problem! It's all a cinch. ... After about 3 years worth of this, the system has become a giant behemoth, awesome (and awful) to behold, slumbering onwards in unyielding persistence, soaking up all RAM everywhere it can find any, and peaking at 99% CPU when you're not looking (gotta keep those savvy customers who know how to use 'top' happy, y'know?). The old OO abstraction layers for the database are mostly no longer used, nowadays we're just writing straight SQL anyway, but some core code still uses them, so we daren't delete them just yet. The resource acquisition code has mutated under the CPU's electromagnetic radiation, and has acquired 5 or 6 different ways of acquiring mutex locks, each written by different people who couldn't figure out how to use the previous person's code. None of these locks could be used simultaneously with each other, for they interact in mysterious, and often disastrous, ways. Adding more features to the daemon is a road through a minefield filled with the remains of less savvy C++ veterans. Then one day, I was called upon to implement something that required making an IPC call to this dying but stubbornly still-surviving daemon. bloatedness of the superdooper generic framework, is completely isolated wrapper library to it, because that would pull in 8000+ IPC wrapper functions from that horrific auto-generated header file, which in turn requires linking in all the C++-based framework libraries, which in turn pulls in yet more subsidiary supporting libraries, which if you add it all up, adds about 600MB to the C library size. Which is Not Acceptable(tm). So what to do? Well, first write a separate library to handle interfacing with the 1 or 2 IPC calls that I can't do without, to keep the nasty ugly hacks in one place. Next, in this library, since we can't talk to the C++ part directly, write out function arguments using fwrite() into a temporary file, then fork() and exec() a C++ wrapper executable that *can* link with the C++ IPC code. This wrapper executable then reads the temporary file and unpacks the function arguments, then hands them over to the IPC code that repacks them in the different format understood by the daemon, then sends it off. Inside the daemon, write more code to recognize this special request, unpack its arguments once again, then do some setup work (y'know, acquire those nasty mutexes, create some OO abstraction objects, the works), then actually call the real function that does the work. But we're not done; that function must return some results, so after carefully cleaning up after ourselves (making sure that the "RAII" objects are destructed in the right order to prevent nasty problems like deadlocks or double-free()'s), we repackage the function's return value and send it back over the IPC link. On the other end, the IPC library decodes that and returns it to the wrapper executable, which now must fwrite() it into another temporary file, and then exit with a specific exit code so that the C library that fork-and-exec'd it will know to look for the function results in the temporary file, so that it can read them back in, unpack them, then return to the original caller. This nasty piece of work was done EVERY SINGLE TIME AN IPC FUNCTION WAS CALLED. What's that you say? Performance is poor? Well, that's just because you need to upgrade to our new, latest-release, shiny hardware! We'll double the amount of RAM and the speed of the CPU -- we'll throw in an extra core or two, too -- and you'll be up and running in no time! Meanwhile, back in the R&D department (nicely insulated from customer support), I say to myself, gee I wonder why performance is so bad... After years of continual horrendous problems, nasty deadlock bugs, hair-pulling sessions, bugfixes that introduced yet more bugs because the whole thing has become a tower of cards, the PTBs finally was convinced that we needed to do something about it. Long story short, we trashed the ENTIRE C++ generic framework, and went back to using straight int's and char's and good ole single-threaded C code, with no IPCs or mutex RAII objects or 5-layer DB abstractions -- the result was a system at the most 20% of the size of the original, ran 5 times faster, and was more flexible in handling DB queries than the previous system ever could. These days, whenever I hear the phrase "generic framework", esp. if it has "OO" in it, I roll my eyes and go home and happily work on my D code that deals directly with int's and char's. :) That's not to say that abstractions are worthless. On the contrary, having the *right* abstractions can be extremely powerful -- things like D ranges, for example, literally revolutionized the way I write iterative code. The *wrong* abstractions, OTOH... let's just say it's on the path toward that proverbial minefield littered with the remains of less-than-savvy programmer-wannabes. What constitutes a *good* abstraction, though, while easy to define in general terms, is rather elusive in practice. It takes a lot of skill and experience to be able to come up with useful abstractions. Unfortunately, it's all too easy to come up with idealistic abstractions that actually detract, rather than add -- and people reinvent them all the time. The good thing is that usually other people will fail to see any value in them ('cos there is none) so they get quickly forgotten, like they should be. The bad thing is that they keep coming back through people who don't know better. Again I come back to Knuth's insightful quote -- to truly build a useful abstraction, you have to understand what it translates to. You have to understand how the machine works, and how all lower layers of abstraction works, before you can build something new *and* useful. That's why I said that in order to make the machine dance, you must sing its tune. You can't just pull an abstraction out of thin air and expect that it will all somehow work out in the end. Before we master the new abstractions introduced by D -- like ranges -- we're not really in the position to discover better abstractions that improve upon them.I don't see anything like this happening so depending on your scale, I don't think we are getting better, but just chasing our tails... how many more languages do we need that just change the syntax of C++? Why do people think syntax matters? Semantics is what is important but there seems to be little focus on it. Of course, we must express semantics through syntax so for practical purposes it mattes to some degree.... But not nearly as much as the number of programming languages suggest.Actually, I would say that in terms of semantics, it all ultimately maps to Turing machines anyway, so ultimately it's all the same. You can write anything in assembly language. (Or, to quote Larry Wall's tongue-in-cheek way of putting it: you can write assembly code in any language. :-P) That's already been known and done. What matters is, what kind of abstractions can we build on top of it, that allows us maximum expressivity and usability? The seeming simplicity of Turing machines (or assembly language) belies the astounding computational power hidden behind the apparent simplicity. The goal of programming language design is to discover ways of spanning this range of computational power in a way that's easy to understand, easy to use, and efficient to run in hardware. Syntax is part of the "easy to use" equation, a small part, but no less important (ever tried writing code in lambda calculus?). The harder part is the balancing act between expressiveness and implementability (i.e. efficiency, or more generally, how to make your computation run in a reasonable amount of time with a reasonable amount of space -- a program that can solve all your problems is useless if it will take 10 billion years to produce the answer; so is a program that requires more memory than you can afford to buy). That's where the abstractions come in -- what kind of abstractions will you have, and how well do they map to the underlying machine? It's all too easy to think in terms of the former, and neglect the latter -- you end up with something that works perfectly in theory but requires unreasonable amounts of time/memory in practice or is just a plain mess when mapped to actual hardware. T -- Food and laptops don't mix.
Jul 26 2013
On Friday, 26 July 2013 at 23:19:45 UTC, H. S. Teoh wrote:On Fri, Jul 26, 2013 at 03:02:32PM +0200, JS wrote:I used to program in assembly and loved it. The problem was that one could not write large programs. Not because it is impossible in assembly but because there was no(or little) ways to abstract. Large programs MUST be able to be broken into manageable pieces. OOP is what allows the large programs of our times... Irrelevant if it was done in assembly or not. E.g., it is not impossible to think of an assembly like language(low level) that has many high level concepts(classes, templates, etc...) and a compiler that has many safety features(type checking, code analysis, bounds checking, etc...)... But what you end up with then is probably something similar to D with every function's body written in asm.I think the next step in languages it the mutli-level abstraction. Right now we have the base level core programming and the preprocessing/template/generic level above that. There is no reason why language can't/shouldn't keep going. The ability to control and help the compiler do it's job better is the next frontier. Analogous to how C++ allowed for abstraction of data, template allow for abstraction of functionality, we then need to abstract "templates"(or rather meta programming).There is much value to be had for working with the minimum possible subset of features that can achieve what you want with a minimum of hassle. The problem with going too far with abstraction is that you start living in an imaginary idealistic dreamworld that has nothing to do with how the hardware actually implements the stuff. You start writing some idealistic code and then wonder why it doesn't work, or why performance is so poor. As Knuth once said: By understanding a machine-oriented language, the programmer will tend to use a much more efficient method; it is much closer to reality. -- D. Knuth People who are more than casually interested in computers should have at least some idea of what the underlying hardware is like. Otherwise the programs they write will be pretty weird. -- D. Knuth If I ever had the chance to teach programming, my first course would be assembly language programming, followed by C, then by other languages, starting with a functional language (Lisp, Haskell, or Concurrent Clean), then other imperative languages, like Java. (Then finally I'll teach them D and tell them to forget about the others. :-P) Will I expect my students to write large applications in assembly language or C? Nope. But will I require them to pass the final exam in assembly language? YOU BETCHA. I had the benefit of learning assembly while I was still a teenager, and I can't tell you how much that experience has shaped my programming skills. Even though I haven't written a single line of assembly for at least 10 years, understanding what the machine ultimately runs gave me deep insights into why certain things are done in a certain way, and how to take advantage of that. It helps you build *useful* abstractions that map well to the underlying machine code, which therefore gives you good performance and overall behaviour while providing ease of use.By contrast, my encounters with people who grew up with Java or Pascal consistently showed that most of them haven't the slightest clue how the machine even works, and as a result, just like Knuth said, they tend to have some pretty weird ideas about how to write their programs. They tend to build messy abstractions or idealistic abstractions that don't map well to the underlying hardware, and as a result, their programs are often bloated, needlessly complex, and run poorly. [...]I've spend several years in assembly when I was young... But you need to go a step further. Electronics deals with only 1's and 0's... not nibbles, bytes, words, dwords, qwords, etc.... These groups only help people, not computers. Surely we generally have optimal performance by using types that are multiples of the bus size, but that is irrelevant. Sure many cpu's have some idea of the basic types but this is only because it is in hardware and they can't predict the type you want to create. For the most part it is irrelevant because all complex types are built from fundamental types. BUT we are talking about compilers and not cpu's. Compilers are software and can be written to "know" future types(by modifying/informing the compiler). Everything that can be done in any HLL can be done with 1's and 0's in a hex editor... in fact, it must be so or you end up with a program that can't run. So this alone proves that a HLL is only for abstraction to make life easier(essentially to mimic human thinking the best it can). The problem that I've always run across is that all compilers are rather limited in some way that makes life harder, not easier. Sometimes this is bugs, other times it is lack of as simple feature that would make thinks much easier to deal with. D goes a long way in the feature set but seems to have a lot more bugs than normal and has a few down sides. D is what got me back language(I find it very cohesive and well thought out) but unfortunately do not want to be restricted to .NET(a great library and well put together too).For example, why are there built in types?You need to learn assembly language to understand the answer to that one. ;-)Again, at the lowest level cpu's work on bits, nothing else. Even most cpu's are abstracted for performance reasons, but it doesn't change that fact. By bits I do not mean a 1-bit computer but simply a computer that works on a bit stream with no fixed size "word". Think of a turing machine.There is no inherit reason this is so except this allows compilers to achieve certain performance results...Nah... performance isn't the *only* reason. But like I said, you need to understand the foundations (i.e., assembly language) before you can understand why, to use a physical analogy, you can't just freely move load-bearing walls around.I think those are cool features, and it's the kinda stuff that draws me to the language. Stuff that makes life easier rather than harder. But all these concepts are simply "configuring" the compiler to do things that traditionally they didn't do. Alias this is telling the compiler to treat a class as a specific type or to replace it's usage with a function. Where did this come from? C/C++ doesn't have it! Why? Because it either wasn't thought of or wasn't thought of as useful. Those kinds of things hold compilers back. If someone will use it then it is useful. I understand that in reality there are limitations but when someone makes the decision "No one will need this" then every ultimately suffers. It's a very egocentric decision that assumes that the person knows everything. Luckily Walter seems to have had the foresight to avoid making such decisions.but having a higher level of abstraction of meta programming should allow us to bridge the internals of the compiler more effectively.Andrei mentions several times in TDPL that we programmers don't like artificial distinctions between built-in types and user-defined types, and I agree with that sentiment. Fortunately, things like alias this and opCast allow us to define user-defined types that, for all practical purposes, behave as though they were built-in types. This is a good thing, and we should push it to the logical conclusion: to allow user-defined types to be optimized in analogous ways to built-in types. That is something I've always found lacking in the languages I know, and something I'd love to explore, but given that we're trying to stabilize D2 right now, it isn't gonna happen in the near future. Maybe if we ever get to D3...Nevertheless, having said all that, if you truly want to make the machine dance, you gotta sing to its tune. In the old days, the saying was that premature optimization is the root of all evils. These days, I'd like to say, premature *generalization* is the root of all evils.Sure. Ultimately someone designed the cpu in a certain way and for you to take advantage of all it's potential you have to work within the limitations/constraints they set up(which, sometimes, is not known fully). Also, it is useless, from a practical matter, to create a program in a language that can never be ran but not theoretically useless. It ultimately depends on the goals... I imagine when someone wants to create the best they can, then sometimes it's very easy to go overboard... sometimes the tools are simply not available to reach the goal. But such attitudes is what pushes the boundaries and generally pays off in the long run... without them we would at the most still be using punch cards.I've seen software that suffered from premature generalization... It was a system that was essentially intended to be a nice interface to a database, with some periodic background monitoring functions. The person(s) who designed it decided to build this awesome generic framework with all sorts of fancy features. For example, users don't have to understand what SQL stands for, yet they can formulate complex queries by means of nicely-abstracted OO interfaces. Hey, OO is all the rage these days, so what can be better than to wrap SQL in OO in such a way that the user wouldn't even know it's SQL underneath? I mean, what if we wanted to switch to, oh, Berkeley DB one of these days?! But abstracting a database isn't good enough. There's also this incredible generic framework that handles timers and events, such that you don't have to understand what an event loop is and you can write event-driven code, just like that. Oh, and to run all of these complicated fancy features, we have to put it inside its own standalone daemon, so that if it crashes, we can use another super-powerful generic framework to handle crashes and automatically restart so that the user doesn't even have to know the database engine is crashing underneath him; the daemon will pick up the query and continue running it after it restarts! Isn't that cool? But of course, since it runs as a separate daemon, we have to use IPC to interface it with user code. It all makes total sense! ... After about 3 years worth of this, the system has become a giant behemoth, awesome (and awful) to behold, slumbering onwards in unyielding persistence, soaking up all RAM everywhere it can find any, and peaking at 99% CPU when you're not looking (gotta keep those savvy customers who know how to use 'top' happy, y'know?). The old OO abstraction layers for the database are mostly no longer used, nowadays we're just writing straight SQL anyway, but some core code still uses them, so we daren't delete them just yet. The resource acquisition code has mutated under the CPU's electromagnetic radiation, and has acquired 5 or 6 different ways of acquiring mutex locks, each written by different people who couldn't figure out how to use the previous person's code. None of these locks could be used simultaneously with each other, for they interact in mysterious, and often disastrous, ways. Adding more features to the daemon is a road through a minefield filled with the remains of less savvy C++ veterans. Then one day, I was called upon to implement something that required making an IPC call to this dying but stubbornly still-surviving daemon. to the bloatedness of the superdooper generic framework, is completely isolated C++ IPC wrapper library to it, because that would pull in 8000+ IPC wrapper functions from that horrific auto-generated header file, which in turn requires linking in all the C++-based framework libraries, which in turn pulls in yet more subsidiary supporting libraries, which if you add it all up, adds about 600MB to the C library size. Which is Not Acceptable(tm). So what to do? Well, first write a separate library to handle interfacing with the 1 or 2 IPC calls that I can't do without, to keep the nasty ugly hacks in one place. Next, in this library, since we can't talk to the C++ part directly, write out function arguments using fwrite() into a temporary file, then fork() and exec() a C++ wrapper executable that *can* link with the C++ IPC code. This wrapper executable then reads the temporary file and unpacks the function arguments, then hands them over to the IPC code that repacks them in the different format understood by the daemon, then sends it off. Inside the daemon, write more code to recognize this special request, unpack its arguments once again, then do some setup work (y'know, acquire those nasty mutexes, create some OO abstraction objects, the works), then actually call the real function that does the work. But we're not done; that function must return some results, so after carefully cleaning up after ourselves (making sure that the "RAII" objects are destructed in the right order to prevent nasty problems like deadlocks or double-free()'s), we repackage the function's return value and send it back over the IPC link. On the other end, the IPC library decodes that and returns it to the wrapper executable, which now must fwrite() it into another temporary file, and then exit with a specific exit code so that the C library that fork-and-exec'd it will know to look for the function results in the temporary file, so that it can read them back in, unpack them, then return to the original caller. This nasty piece of work was done EVERY SINGLE TIME AN IPC FUNCTION WAS CALLED. What's that you say? Performance is poor? Well, that's just because you need to upgrade to our new, latest-release, shiny hardware! We'll double the amount of RAM and the speed of the CPU -- we'll throw in an extra core or two, too -- and you'll be up and running in no time! Meanwhile, back in the R&D department (nicely insulated from customer support), I say to myself, gee I wonder why performance is so bad... After years of continual horrendous problems, nasty deadlock bugs, hair-pulling sessions, bugfixes that introduced yet more bugs because the whole thing has become a tower of cards, the PTBs finally was convinced that we needed to do something about it. Long story short, we trashed the ENTIRE C++ generic framework, and went back to using straight int's and char's and good ole single-threaded C code, with no IPCs or mutex RAII objects or 5-layer DB abstractions -- the result was a system at the most 20% of the size of the original, ran 5 times faster, and was more flexible in handling DB queries than the previous system ever could. These days, whenever I hear the phrase "generic framework", esp. if it has "OO" in it, I roll my eyes and go home and happily work on my D code that deals directly with int's and char's. :) That's not to say that abstractions are worthless. On the contrary, having the *right* abstractions can be extremely powerful -- things like D ranges, for example, literally revolutionized the way I write iterative code. The *wrong* abstractions, OTOH... let's just say it's on the path toward that proverbial minefield littered with the remains of less-than-savvy programmer-wannabes. What constitutes a *good* abstraction, though, while easy to define in general terms, is rather elusive in practice. It takes a lot of skill and experience to be able to come up with useful abstractions. Unfortunately, it's all too easy to come up with idealistic abstractions that actually detract, rather than add -- and people reinvent them all the time. The good thing is that usually other people will fail to see any value in them ('cos there is none) so they get quickly forgotten, like they should be. The bad thing is that they keep coming back through people who don't know better.I agree, it is very hard, and I think that is why the compiler must make such things easier. I think the problems tend to be more that compilers get in the way or are difficult to implement the abstracts and end up causing problems down the road due to hacks and workarounds rather than the other way around. While it is true that it is difficult to abstract things and take into account unforeseen events, a properly abstracted system should be able to be general enough to deal with them... when it's not, then I'm sure it is much more difficult to rectify than a concrete implementation. Abstraction is difficult, requires a good memory, and the intelligence to deal with the complexity involved... but the rewards are well worth it. We, as a civilization, can't get better at it without workings at it. You can't expect everyone to get it right all the time. Most people are idiots, simple as that. You can't expect most people to comprehend complex systems, or even have the desire to do so if they are capable. Most people want to blindly apply a familiar pattern that has worked before they don't know any better. This is not necessarily bad except when the pattern isn't the right one. Even the real intelligent people that are capable of dealing with the complexity can only do so for so long. A human brain, no matter how good, can't deal with exponential factors... at some point it will become too much to handle. I'm one of those believers that at some point you have to scrap the broken way and start afresh, learned from your mistakes to make something better. This is what you guys did when implementing a simpler system... as what you learned was simple was better. One of the great things though, is that breaking complexity into simpler parts always arrives at a set of simple enough pieces that can be dealt with. The problem is that someone still has to understand all the complexity more or less.Again I come back to Knuth's insightful quote -- to truly build a useful abstraction, you have to understand what it translates to. You have to understand how the machine works, and how all lower layers of abstraction works, before you can build something new *and* useful. That's why I said that in order to make the machine dance, you must sing its tune. You can't just pull an abstraction out of thin air and expect that it will all somehow work out in the end. Before we master the new abstractions introduced by D -- like ranges -- we're not really in the position to discover better abstractions that improve upon them.I think we agree, basically what I was getting at above.Yes! So what is different is only what the language itself has to offer to make life easier to abstract... if we didn't want abstraction we would just write in 0's and 1's... or if we had the memory and intelligence, it would be the easiest way(instead of waiting for multiple alias this's ;))I don't see anything like this happening so depending on your scale, I don't think we are getting better, but just chasing our tails... how many more languages do we need that just change the syntax of C++? Why do people think syntax matters? Semantics is what is important but there seems to be little focus on it. Of course, we must express semantics through syntax so for practical purposes it mattes to some degree.... But not nearly as much as the number of programming languages suggest.Actually, I would say that in terms of semantics, it all ultimately maps to Turing machines anyway, so ultimately it's all the same. You can write anything in assembly language. (Or, to quote Larry Wall's tongue-in-cheek way of putting it: you can write assembly code in any language. :-P) That's already been known and done.What matters is, what kind of abstractions can we build on top of it, that allows us maximum expressivity and usability? The seeming simplicity of Turing machines (or assembly language) belies the astounding computational power hidden behind the apparent simplicity. The goal of programming language design is to discover ways of spanning this range of computational power in a way that's easy to understand, easy to use, and efficient to run in hardware. Syntax is part of the "easy to use" equation, a small part, but no less important (ever tried writing code in lambda calculus?). The harder part is the balancing act between expressiveness and implementability (i.e. efficiency, or more generally, how to make your computation run in a reasonable amount of time with a reasonable amount of space -- a program that can solve all your problems is useless if it will take 10 billion years to produce the answer; so is a program that requires more memory than you can afford to buy). That's where the abstractions come in -- what kind of abstractions will you have, and how well do they map to the underlying machine? It's all too easy to think in terms of the former, and neglect the latter -- you end up with something that works perfectly in theory but requires unreasonable amounts of time/memory in practice or is just a plain mess when mapped to actual hardware.One of the most useful aspects of programming, and what makes it so powerful, is the ability to leverage what others have done. Unfortunately what happens is people write the same old stuff other people have written. This problem has gotten better over the last few years with the internet making it easier but still is an issue. Just think of all the man hours wasted due to people writing the same code, debugging code because of bugs, or giving up because they didn't have the code they needed(but it existed). Think things were optimal from the get go.... how much further we'd be along. I don't mind people making mistakes in the theoretical vs practical tug of war... because I think the theoretical is what pushes boundaries and the practical is with strengthens them. Much of recent mathematics is purely theoretical, which in turn has fueled practical things... mathematics started out purely practical. Maybe this ebb and flow is a natural thing in life. But I believe just because sometime is practical doesn't mean it is best. For example, suppose you are able to design a programming language that somehow is provably better than all other languages combined. The problem is, it requires a new CPU design that is difficult and expensive to create. Should the language not be used because it is not practical? Of course not. Luckily all of humanity does not have to stop while such things are being built. We need people pushing the boundaries to keep us from spinning our wheels and we need people keeping the wheels turning.
Jul 27 2013