www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - What is the right level of abstractions for D?

reply Joakim <dlang joakim.fea.st> writes:
I noticed that Richard Gabriel, one of the designers of Common 
Lisp, is on the committee for the new <Programming> conference 
that Andrei mentioned here, so I went back and checked out his 
website again: http://dreamsongs.com.  I mentioned his famous 
"Worse is better" essay here a couple years ago:

http://forum.dlang.org/thread/wpyswlvwhmhoutqdwcmo forum.dlang.org

He has an out-of-print 1996 book of essays up on his site, called 
Patterns of Software.  I started reading it, pretty good so far.  
Here is a snippet worth thinking about from pages 19-20:

"Abstractions must be carefully and expertly designed, especially 
when reuse or compression is intended. However, because 
abstractions are designed in a partic-
ular context and for a particular purpose, it is hard to design 
them while antici- pating all purposes and forgetting all 
purposes, which is the hallmark of the well- designed 
abstractions.

This implies that abstractions are best designed by experts. 
Worse, average programmers are not well-equipped to design 
abstractions that have universal usage, even though the 
programming languages used by average programmers and the 
programming language texts and courses average programmers read 
and attend to learn their trade emphasize the importance of doing 
exactly that. Although the designers of the programming language 
and the authors of texts and course instructors can probably 
design abstractions well, the intended audience of the 
language—average programmers—cannot and are therefore left out. 
That is, languages that encourage abstraction lead to less 
habitable software, because its expected inhabitants—average 
programmers working on code years after the original designers 
have disappeared—are not easily able to grasp, modify, and grow 
the abstraction-laden code they must work on.

Not everyone is a poet, but most anybody can write usable 
documentation for small programs—we don’t expect poets to do this 
work. Yet we seem to expect that the equivalent of poets will use 
high-level programming languages, because only program-poets are 
able to use them. In light of this observation, is it any wonder 
that abstraction-poor languages like C are by far the most 
popular and that abstraction-rich ones like Lisp and Smalltalk 
are niche languages?"

Of course, this completely ignores the fact that the abstractions 
are not costless, ie Smalltalk will usually be much slower than 
C, but he does mention C++ and its attempt at much lower-cost 
abstractions in an earlier chapter.

This is something D people have to think heavily about, as the 
market for programming languages has split into at least three 
groups since he wrote that more than two decades ago:

1. low-level compiled languages like C++, D, Rust, and Swift, 
meant for performance and usually experts who want to squeeze it 
out


vast middle of day-to-day programmers to crank out libraries and 
apps that perform reasonably well

3. high-level "scripting" languages like Ruby and Python, meant 
for those who don't care too much for performance but just want 
to get working code

I think D is positioned somewhere between 1 and 2, though closer 
to 1. However, there is sometimes talk of using D for all three, 
though perhaps that is only meant as an added benefit for people 
already using it for 1 or 2, ie those who already know the 
language better.

I have the impression that the recent  nogc and C++ push means 
that a concerted effort is being made to go more after 1.  Some 
thought needs to be put into which of those groups D is going 
after and whether it's possible to go after more than one of 
those groups today.
Oct 26 2016
next sibling parent Joakim <dlang joakim.fea.st> writes:
On Thursday, 27 October 2016 at 06:22:03 UTC, Joakim wrote:
 I noticed that Richard Gabriel, one of the designers of Common 
 Lisp, is on the committee for the new <Programming> conference 
 that Andrei mentioned here, so I went back and checked out his 
 website again: http://dreamsongs.com.  I mentioned his famous 
 "Worse is better" essay here a couple years ago:

 [...]
Forgot to post the link for the book, a pdf link is available here: https://www.dreamsongs.com/Books.html
Oct 26 2016
prev sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 10/27/2016 02:22 AM, Joakim wrote:
 1. low-level compiled languages like C++, D, Rust, and Swift, meant for
 performance and usually experts who want to squeeze it out


 middle of day-to-day programmers to crank out libraries and apps that
 perform reasonably well

 3. high-level "scripting" languages like Ruby and Python, meant for
 those who don't care too much for performance but just want to get
 working code

 I think D is positioned somewhere between 1 and 2, though closer to 1.
 However, there is sometimes talk of using D for all three, though
 perhaps that is only meant as an added benefit for people already using
 it for 1 or 2, ie those who already know the language better.
You're falling into the common fallacy that those groups are mutually exclusive and that a single language can't be appropriate for more than one. D is all about proving that wrong, and is meant for, and good at, all three. I've noticed that, for many of the people who don't "get" D, the problem they're hitting is that they're minds are so twisted around by the "polyglot" culture, that they're looking for "the one" tiny little niche that D is for, not seeing that, and thus missing the whole entire point.
Oct 27 2016
parent reply Joakim <dlang joakim.fea.st> writes:
On Thursday, 27 October 2016 at 17:03:09 UTC, Nick Sabalausky 
wrote:
 On 10/27/2016 02:22 AM, Joakim wrote:
 1. low-level compiled languages like C++, D, Rust, and Swift, 
 meant for
 performance and usually experts who want to squeeze it out


 the vast
 middle of day-to-day programmers to crank out libraries and 
 apps that
 perform reasonably well

 3. high-level "scripting" languages like Ruby and Python, 
 meant for
 those who don't care too much for performance but just want to 
 get
 working code

 I think D is positioned somewhere between 1 and 2, though 
 closer to 1.
 However, there is sometimes talk of using D for all three, 
 though
 perhaps that is only meant as an added benefit for people 
 already using
 it for 1 or 2, ie those who already know the language better.
You're falling into the common fallacy that those groups are mutually exclusive and that a single language can't be appropriate for more than one. D is all about proving that wrong, and is meant for, and good at, all three.
There are good reasons for this split, and yes, it is probably impossible for one language to attract all three groups. You could use languages from 1 and 2 for all three, with a bit more work, but I don't see many scripts written in C++. :) The reason for the split is that there are different levels of software expertise and performance needs, and each of those groups is geared for a different level. Show templates to a scripting user and they probably run away screaming. D can probably do well with groups 1 and 2, but the level of power and expertise that is needed for those lower levels will scare away people from 3. Those already using it for 1 and 2 may also be comfortable with reusing D for scripting, but that's not attracting people from group 3, ie those who only want something easy to use and don't want to know the difference between a static and dynamic array.
 I've noticed that, for many of the people who don't "get" D, 
 the problem they're hitting is that they're minds are so 
 twisted around by the "polyglot" culture, that they're looking 
 for "the one" tiny little niche that D is for, not seeing that, 
 and thus missing the whole entire point.
Yes, this has definitely hurt D, being stuck between 1 and 2. People from 2 probably look at D and think it's too low-level. People from 1 are always looking to squeeze out _more_ performance, so the GC is a way for them to just write off D. You and I think they're making a mistake, but maybe they're not wrong for their own uses. As I said, the recent push for nogc and C++ compatibility suggests that a renewed effort is being made to focus on group 1, particularly when combined with the D benchmarks for regex and recently math. I'm happy in that space between 1 and 2, and the recent push to move languages from 2 to AoT compilation suggests that is a good place to be. So maybe group 2 will also come to us. :)
Oct 28 2016
next sibling parent reply Danni Coy via Digitalmars-d <digitalmars-d puremagic.com> writes:
I am going to talk as a person who mostly works with group 3 languages
but will use whatever they need to use to get the job done (tm).

 The reason for the split is that there are different levels of software
 expertise and performance needs, and each of those groups is geared for a
 different level.  Show templates to a scripting user and they probably run
 away screaming.
The bit that sends group 3 people off screaming is static typing, once you have accepted the need to explicitly type everything then Template functions are pretty straight forward in D.
 D can probably do well with groups 1 and 2, but the level
 of power and expertise that is needed for those lower levels will scare away
 people from 3.  Those already using it for 1 and 2 may also be comfortable
 with reusing D for scripting, but that's not attracting people from group 3,
 ie those who only want something easy to use and don't want to know the
 difference between a static and dynamic array.
Again this boils down to static typing, once your group 3 programmer has accepted this then is is a minor issue. At worst case I could see a grumble then avoiding static arrays.
 I've noticed that, for many of the people who don't "get" D, the problem
 they're hitting is that they're minds are so twisted around by the
 "polyglot" culture, that they're looking for "the one" tiny little niche
 that D is for, not seeing that, and thus missing the whole entire point.
Yes, this has definitely hurt D, being stuck between 1 and 2. People from 2 probably look at D and think it's too low-level. People from 1 are always looking to squeeze out _more_ performance, so the GC is a way for them to just write off D. You and I think they're making a mistake, but maybe they're not wrong for their own uses. As I said, the recent push for nogc and C++ compatibility suggests that a renewed effort is being made to focus on group 1, particularly when combined with the D benchmarks for regex and recently math. I'm happy in that space between 1 and 2, and the recent push to move languages from 2 to AoT compilation suggests that is a good place to be. So maybe group 2 will also come to us. :)
Apart from in application scripting, I have replaced python with D at work. (I was largely using python because my brain and shell scripting are largely incompatible). The reasons for this was mostly because with D I can compile and distribute the binary without having to worry if the right version of python was installed on the target machine. For this whip up a tool to automate some file processing quickly I found D as pleasant as anything else I have used. The only thing missing are bindings for a high quality crossplatform gui toolkit. Where I see the future potential of D in regards to the levels 1,2 and 3 is having a team where different programmers are working at different levels within the same language on the same project. Take a game engine, typically you have low level engine guys (level 1) some guys on the engine team who aren't quite as good but don't have to be (effectively level 2) and the guys who do the level / ai scripting (level 3). I would quite happily use D as a high level language in a team where I had a good lower level team designing most of the stuff I use with my level of expertise in mind as they designed it. To this end the really critical things for me are that my level 1 guys can create code that performs in low latency environments without missing deadlines and can present their APIs in such a way that is easy/enjoyable for us level 3 guys to use (see the batch processing thread a little while back).
Oct 29 2016
parent reply Joakim <dlang joakim.fea.st> writes:
On Saturday, 29 October 2016 at 22:38:07 UTC, Danni Coy wrote:
 The bit that sends group 3 people off screaming is static 
 typing, once you have accepted the need to explicitly type 
 everything then Template functions are pretty straight forward 
 in D.

 D can probably do well with groups 1 and 2, but the level
 of power and expertise that is needed for those lower levels 
 will scare away
 people from 3.  Those already using it for 1 and 2 may also be 
 comfortable
 with reusing D for scripting, but that's not attracting people 
 from group 3,
 ie those who only want something easy to use and don't want to 
 know the
 difference between a static and dynamic array.
Again this boils down to static typing, once your group 3 programmer has accepted this then is is a minor issue. At worst case I could see a grumble then avoiding static arrays.
As someone who uses all three, I think you underestimate the reluctance of many in group 3 to learn enough to drop down into the lower-level languages.
 Apart from in application scripting, I have replaced python 
 with D at work. (I was largely using python because my brain 
 and shell scripting are largely incompatible). The reasons for 
 this was mostly because with D I can compile and distribute the 
 binary without having to worry if the right version of python 
 was installed on the target machine.
I have done something similiar with a scripting language before, shipping the scripting interpreter bundled with the app, so that nothing else has to be installed.
 For this whip up a tool to automate some file processing 
 quickly I
 found D as pleasant as anything else I have used.
I would only use D for such a task too, but I think that is more that we are in group 1/2 and are reusing the tool we know. I'm talking about attracting those from group 3.
 The only thing missing are bindings for a high quality 
 crossplatform
 gui toolkit.
Have you tried DlangUI? https://github.com/buggins/dlangui I haven't tried it much, so I can't speak to quality, but it certainly is very cross-platform.
 Where I see the future potential of D in regards to the levels 
 1,2 and 3 is having a team where different programmers are 
 working at different levels within the same language on the 
 same project. Take a game engine, typically you have low level 
 engine guys (level 1) some guys on the engine team who aren't 
 quite as good but don't have to be (effectively level 2) and 
 the guys who do the level / ai scripting (level 3). I would 
 quite happily use D as a high level language in a team where I 
 had a good lower level team designing most of the stuff I use 
 with my level of expertise in mind as they designed it.

 To this end the really critical things for me are that my level 
 1 guys can create code that performs in low latency 
 environments without missing deadlines and can present their 
 APIs in such a way that is easy/enjoyable for us level 3 guys 
 to use (see the batch processing thread a little while back).
That's an interesting use, guess Ethan is heading down that road with his Binderoo effort. If D can be used like this for more than just games, it would create a nice niche, but I'm skeptical of attracting much of group 3. On Sunday, 30 October 2016 at 23:25:03 UTC, Laeeth Isharc wrote:
 Those categories - I am not sure how well they fit.

 When I learnt to program, C was considered a high level 
 language,
  and now Swift is considered low level.  The world has changed 
 a little, but that isn't my main point.
There were higher-level languages than C even back then, but computers were much slower, so C was considered the highest practical level. The difference is that computers are so fast now that many can actually use lisp or smalltalk.
 Implicitly in what you wrote is the idea that low level 
 programmers are the ones with real ability,  and people who 
 write in Python might be productive,  but are of a different 
 level of overall ability.

 Once that kind of high level / low level mapping to ability 
 might have made sense,  but if it's still useful,  I think it's 
 much less applicable now.   There are plenty of mediocre 
 embedded device programmers,  and plenty of people who used to 
 write in C and now write in Python.   (I am sure ESR still 
 writes in C, but he write about his love for python some time 
 back).
I never mentioned anything like "real ability," whatever that means, only that you cannot abstract away the computer as much when you need more performance. Those in group 3 don't need performance, so they are abstracted away from types, virtual methods, and memory allocation, low-level details that they don't want or need to deal with. Mediocre embedded guys still have to know more about the hardware than a Python programmer, that's the knowledge/skill I'm saying they have more of.
 And to call python a scripting language is misleading 
 terminology - conventionally so,  but still  true - for 
 example,  AHL,  the large quant money management firm,  wrote 
 their risk management systems entirely in Python. You may be an 
 enthusiast of Fisher Black's Noise paper and think that people 
 in money management are fooling themselves,  and I am 
 sympathetic to some of that,  but my impression is that 
 technically this firm is decent.
Scripting is just how they're named, obviously you can write competent software with them too, but I'd be surprised if AHL's software were that fast. Maybe they don't need it to be speedier, so python is best for them, but I never said anybody in group 3 is not technically "decent."
 And everything gets more mixed up when you can compile a Ruby 
 dialect and have it appear at the top of the performance tables.
Which dialect is that? I did mention that group 2 is going AoT-compiled and speeding up, so it is possible with more work to speed up parts of the higher-level languages, but those are subsets.
  It was a scripting language before,  and now it's not?  (it's 
 control and level of abstraction rather than performance that 
 distinguishes level of a language,  but in the past these 
 things went together).
You probably can't use rails with your ruby dialect, with its extensive use of eval(), so there go most of your users. Abstractions often cost performance, which is why those higher-level languages are slower. D tries to keep its abstractions as low-cost as possible, similar to C++, but that comes with complexity that group 3 doesn't want to deal with.
 It seems to me you are reifying your class structure of 
 languages when I am not sure it is a good representation of how 
 things are.
I think that hierarchy is pretty uncontroversial nowadays.
 The reason scripting applications don't use D if it's not 
 already used for other things is libraries and polish.   D has 
 great python interoperability and also a nice package manager.
  Well,  try compiling a program as a python library depending 
 on vibed using dub.   It won't go very well because the fPIC 
 isn't propagated and you might need to compile in a little C 
 code got the constructors.   And what needs to be done isn't 
 completely documented anywhere. So at this point,  it's a less 
 appealing proposition because for the people that currently use 
 D,  these things don't bother them as much and because it's 
 still a small community with a lot of work to do. John Colvin 
 is working on it,
  and maybe it will be fixed soon - because it did bother me.

 But this isn't a consequence of the Platonic essence of D as a 
 language.   It's merely contingent on the particular stage of 
 development and it couldn't have been otherwise because you 
 have to get the foundation right before putting sugar on top.
Libraries and polish are part of the problem, but not what I'm talking about. You cannot make every power tool as easy to use as a screwdriver. You want performance, you have to pay for it, by learning how to get it. D makes an effort to pay less and does a remarkable job, but I think group 3 is out of reach.
 The experience of social learning is that every person that 
 follows a course somewhat mysteriously makes it easier for 
 those that follow.   It's not only mysterious,  because the 
 signposts and guides get better too.   D is not an easy 
 language to learn,  but even today it's far from exceptionally 
 difficult,  and it will get easier with time.

 If you want a C and C++ ABI,  want to have control over memory 
 and to go down to a low level if you need it,  but are someone 
 in a position where you need to get stuff done,  and don't 
 think modern C++ is the answer,  what choices do you have?  Not 
 all that many.

 And I have thought for a while that people were recklessly 
 squandering performance and memory usage.   That's not good 
 craftsmanship - it might be necessary to make compromises in a 
 practical situation, but it's hardly something to be proud of,  
 as people almost seem to be.   What Knuth said in his paper and 
 speech is not what people take his words out of context to 
 mean,  and on the other hand look at the everyday experience of 
 using applications written by people who follow this philosophy 
 to see that there must be something wrong with it.
As I've said in this forum before, if you want performance, I wouldn't recommend anything other than D.
 A language really takes off not because of something it started 
 doing that it didn't before,  and all of a sudden everything is 
 a success.   It takes off when you have the seeds of something,
  and external conditions change to the point that suddenly it 
 starts to become a much more favourable environment for 
 language adoption.
I think the recent move towards performance, because of the more constrained mobile environ, is that change.
 Herb Sutter wrote a long time back about the end of the free 
 lunch from Moore's Law.   And its already visible today - 
 somehow one ends up being CPU and memory bound more than Guido 
 would have us believe (I am not sure quite often that Python is 
 fast enough).   It's why I adopted D, and another chap from 
 hedge fund land who has spoken at dconf.   He said to me this 
 year that things had a bit the feel of python scene in the 90s 
 before it really took off.

 Dataset sizes keep growing.  And I don't know,  but it seems to 
 be that the ACM paper I posted earlier about the consequences 
 of new fast non-volatile storage technology is right - storage 
 won't be the bottleneck for much longer, and increasingly CPU 
 will.   Intel described the technology as the most significant 
 development since the Internet.   I am not sure that is merely 
 hyperbole.

 When you see a change in relative price of the sort they are 
 talking,  and it's something so basic,  then everything changes.
  Not overnight,  but over the course of the next five to ten 
 years.
Yes, some big changes are happening, that favor group 1 again. It's why we see a renewed focus there, with new entrants like Rust, Swift, and Go, and C++ finally reforming.
 I can't imagine this shift will be bad for D,  and if that's 
 the case then the pattern of people drawn to explore D at the 
 margin won't necessarily fit your categories.   I should think 
 the determinants will be more things like how much work and 
 data do you have in relation to easily available processing 
 power (yes,  the cloud is cheap,  but there are institutional 
 and contractual barriers to moving data outside the firm,  and 
 these change slowly), how open are you to new technologies,  do 
 you have the authority to try it yourself without persuading a 
 committee,  can you and your people learn something new without 
 extensive training courses,  are you addicted to depending on 
 support,  can you already program in C?  And so on.
They still fit the categories, it's just the relative importance of each is changing, with the broader patterns you point out favoring group 1 more.
 So look at recent adopters - that probably gives you a little 
 idea about what's coming.

 It seems to me the recent low level improvements might be seen 
 as just a general process of maturation that is reflected now 
 in lower level stuff,  previously in documentation 
 improvements,  and tomorrow in something else.   It's not like 
 there is a central direction of any effectiveness on what 
 people work on.
There is a central direction: what the co-architects work on, which has recently been nogc, C++, and safety. All are geared towards group 1, as safety has become a recent concern for group 1.
 A push for C++ interoperability is a good thing,  but 
 strategically what has that to do with being low level?   It 
 makes it easier to use C++ libraries,  which is also good for 
 scripting...
It is far more likely for a group 1 user to bind to a C++ library than a group 3 user, though it potentially helps him too. I'm not complaining, D's spot between group 1 and 2 is great for me. I have also argued against those who think D should specialize even more, by narrowing in on some niche within those groups, like games or webdev. I think some thought has to be put into which of those three larger groups D is going after, and tweak some of the messaging because group 3 is out of reach.
Nov 05 2016
parent reply Danni Coy via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, Nov 5, 2016 at 5:47 PM, Joakim via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On Saturday, 29 October 2016 at 22:38:07 UTC, Danni Coy wrote:

 The bit that sends group 3 people off screaming is static typing, once
 you have accepted the need to explicitly type everything then Template
 functions are pretty straight forward in D.

 D can probably do well with groups 1 and 2, but the level
 of power and expertise that is needed for those lower levels will scare
 away
 people from 3.  Those already using it for 1 and 2 may also be
 comfortable
 with reusing D for scripting, but that's not attracting people from
 group 3,
 ie those who only want something easy to use and don't want to know the
 difference between a static and dynamic array.
Again this boils down to static typing, once your group 3 programmer has accepted this then is is a minor issue. At worst case I could see a grumble then avoiding static arrays.
As someone who uses all three, I think you underestimate the reluctance of many in group 3 to learn enough to drop down into the lower-level languages.
What I am saying that you loose a whole bunch of people as soon as you make the language statically typed. Of the people that you have left at worst you will have them grumble a bit and perhaps avoid using static arrays.
 For this whip up a tool to automate some file processing quickly I
 found D as pleasant as anything else I have used.
I would only use D for such a task too, but I think that is more that we are in group 1/2 and are reusing the tool we know. I'm talking about attracting those from group 3.
I honestly don't think the D code I wrote was any less readable or understandable than the python code I wrote, In both cases there are a couple of idioms you have to learn, but the code is really concise and to the point once you know the idioms (something that makes me a big fan of both languages). In fact the file processing required less steps than the Python version. I don't think I would have a harder time explaining what the code does to a novice coder.
 The only thing missing are bindings for a high quality crossplatform
 gui toolkit.
Have you tried DlangUI? https://github.com/buggins/dlangui I haven't tried it much, so I can't speak to quality, but it certainly is very cross-platform.
When I mean high quality I mean competitive with Qt (the current least bad cross platform toolkit), DLangUI gets compared to the Java UI offerings which leaves me somewhat cold. I have never met a java program with a UI I liked.
 Where I see the future potential of D in regards to the levels 1,2 and 3
 is having a team where different programmers are working at different
 levels within the same language on the same project. Take a game engine,
 typically you have low level engine guys (level 1) some guys on the engine
 team who aren't quite as good but don't have to be (effectively level 2)
 and the guys who do the level / ai scripting (level 3). I would quite
 happily use D as a high level language in a team where I had a good lower
 level team designing most of the stuff I use with my level of expertise in
 mind as they designed it.

 To this end the really critical things for me are that my level 1 guys
 can create code that performs in low latency environments without missing
 deadlines and can present their APIs in such a way that is easy/enjoyable
 for us level 3 guys to use (see the batch processing thread a little while
 back).
That's an interesting use, guess Ethan is heading down that road with his Binderoo effort. If D can be used like this for more than just games, it would create a nice niche, but I'm skeptical of attracting much of group 3.
Perhaps - Taking a look at my D as a scripting language code, I can't see anything that would really turn a scripter off. I basically use auto for every variable declaration. Probably the most complicated thing I do is to use std.algorithm.filter and map. In the former case it might be nice to have some syntactic sugar in the way that maxscript does. for a in group where a.count > 0 do ( is easier to approach than foreach (a; group.filter!(a.count > 0)()) { but that is a fairly minor qwibble. http://pastebin.com/VyAiRpyh
Nov 06 2016
parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 11/07/2016 02:00 AM, Danni Coy via Digitalmars-d wrote:
 When I mean high quality I mean competitive with Qt (the current least bad
 cross platform toolkit), DLangUI gets compared to the Java UI offerings
 which leaves me somewhat cold. I have never met a java program with a UI I
 liked.
Yea. DLangUI sounds nice and all to me, but the lack of native UI backend is a nonstarter. Have you tried QtE5? I admit I haven't had a chance to try it out yet myself, but from what I can tell, it looks like what I've been wanting in a D GUI lib: https://github.com/MGWL/QtE5
 In the former case it might be nice to have some syntactic sugar in the way
 that maxscript does.

 for a in group where a.count > 0 do (
I know I'm not a group 3 person, but I find that difficult to read, and its similarity to SQL's undisciplined syntax frightens me.
 is easier to approach than

 foreach (a; group.filter!(a.count > 0)()) {

 but that is a fairly minor qwibble.
I like to "stack" for/foreach/if's: foreach(group; setOfGroups) foreach(a; group) if(a.count > 0) { //... }
Nov 08 2016
prev sibling parent Laeeth Isharc <laeethnospam nospamlaeeth.com> writes:
Those categories - I am not sure how well they fit.

When I learnt to program, C was considered a high level language, 
  and now Swift is considered low level.  The world has changed a 
little, but that isn't my main point.

To grow in a healthy way, D doesn't need to think in terms of 
dominating the world in its totality in a single bound.  All that 
is necessary is to appeal to people around the world in different 
lines of work who are unhappy with what they have now and are 
searching for something better,  or who know about D but where 
there is some kind of barrier to adopting it (barriers are not 
impossible to overcome).

It's a big world,  and the work of Polanyi and Hayek should 
remind us that its very difficult to know where users will come 
from,  because that requires a knowledge of time and place that 
we don't have.   But at this size in relation to the total pool 
of programmers to grow,  and Walter's point about listening to 
your current customers who wish to become bigger ones is a good 
one.

Implicitly in what you wrote is the idea that low level 
programmers are the ones with real ability,  and people who write 
in Python might be productive,  but are of a different level of 
overall ability.

Once that kind of high level / low level mapping to ability might 
have made sense,  but if it's still useful,  I think it's much 
less applicable now.   There are plenty of mediocre embedded 
device programmers,  and plenty of people who used to write in C 
and now write in Python.   (I am sure ESR still writes in C, but 
he write about his love for python some time back).

And to call python a scripting language is misleading terminology 
- conventionally so,  but still  true - for example,  AHL,  the 
large quant money management firm,  wrote their risk management 
systems entirely in Python. You may be an enthusiast of Fisher 
Black's Noise paper and think that people in money management are 
fooling themselves,  and I am sympathetic to some of that,  but 
my impression is that technically this firm is decent.

And everything gets more mixed up when you can compile a Ruby 
dialect and have it appear at the top of the performance tables.  
  It was a scripting language before,  and now it's not?  (it's 
control and level of abstraction rather than performance that 
distinguishes level of a language,  but in the past these things 
went together).

It seems to me you are reifying your class structure of languages 
when I am not sure it is a good representation of how things are.

The reason scripting applications don't use D if it's not already 
used for other things is libraries and polish.   D has great 
python interoperability and also a nice package manager.   Well,  
try compiling a program as a python library depending on vibed 
using dub.   It won't go very well because the fPIC isn't 
propagated and you might need to compile in a little C code got 
the constructors.   And what needs to be done isn't completely 
documented anywhere. So at this point,  it's a less appealing 
proposition because for the people that currently use D,  these 
things don't bother them as much and because it's still a small 
community with a lot of work to do. John Colvin is working on it, 
  and maybe it will be fixed soon - because it did bother me.

But this isn't a consequence of the Platonic essence of D as a 
language.   It's merely contingent on the particular stage of 
development and it couldn't have been otherwise because you have 
to get the foundation right before putting sugar on top.

The experience of social learning is that every person that 
follows a course somewhat mysteriously makes it easier for those 
that follow.   It's not only mysterious,  because the signposts 
and guides get better too.   D is not an easy language to learn,  
but even today it's far from exceptionally difficult,  and it 
will get easier with time.

If you want a C and C++ ABI,  want to have control over memory 
and to go down to a low level if you need it,  but are someone in 
a position where you need to get stuff done,  and don't think 
modern C++ is the answer,  what choices do you have?  Not all 
that many.

And I have thought for a while that people were recklessly 
squandering performance and memory usage.   That's not good 
craftsmanship - it might be necessary to make compromises in a 
practical situation, but it's hardly something to be proud of,  
as people almost seem to be.   What Knuth said in his paper and 
speech is not what people take his words out of context to mean,  
and on the other hand look at the everyday experience of using 
applications written by people who follow this philosophy to see 
that there must be something wrong with it.

A language really takes off not because of something it started 
doing that it didn't before,  and all of a sudden everything is a 
success.   It takes off when you have the seeds of something,  
and external conditions change to the point that suddenly it 
starts to become a much more favourable environment for language 
adoption.

Herb Sutter wrote a long time back about the end of the free 
lunch from Moore's Law.   And its already visible today - somehow 
one ends up being CPU and memory bound more than Guido would have 
us believe (I am not sure quite often that Python is fast 
enough).   It's why I adopted D, and another chap from hedge fund 
land who has spoken at dconf.   He said to me this year that 
things had a bit the feel of python scene in the 90s before it 
really took off.

Dataset sizes keep growing.  And I don't know,  but it seems to 
be that the ACM paper I posted earlier about the consequences of 
new fast non-volatile storage technology is right - storage won't 
be the bottleneck for much longer, and increasingly CPU will.   
Intel described the technology as the most significant 
development since the Internet.   I am not sure that is merely 
hyperbole.

When you see a change in relative price of the sort they are 
talking,  and it's something so basic,  then everything changes.  
  Not overnight,  but over the course of the next five to ten 
years.

I can't imagine this shift will be bad for D,  and if that's the 
case then the pattern of people drawn to explore D at the margin 
won't necessarily fit your categories.   I should think the 
determinants will be more things like how much work and data do 
you have in relation to easily available processing power (yes,  
the cloud is cheap,  but there are institutional and contractual 
barriers to moving data outside the firm,  and these change 
slowly), how open are you to new technologies,  do you have the 
authority to try it yourself without persuading a committee,  can 
you and your people learn something new without extensive 
training courses,  are you addicted to depending on support,  can 
you already program in C?  And so on.

So look at recent adopters - that probably gives you a little 
idea about what's coming.

It seems to me the recent low level improvements might be seen as 
just a general process of maturation that is reflected now in 
lower level stuff,  previously in documentation improvements,  
and tomorrow in something else.   It's not like there is a 
central direction of any effectiveness on what people work on.

A push for C++ interoperability is a good thing,  but 
strategically what has that to do with being low level?   It 
makes it easier to use C++ libraries,  which is also good for 
scripting...


Friday, 28 October 2016 at 08:46:05 UTC, Joakim wrote:
 On Thursday, 27 October 2016 at 17:03:09 UTC, Nick Sabalausky 
 wrote:
 On 10/27/2016 02:22 AM, Joakim wrote:
 1. low-level compiled languages like C++, D, Rust, and Swift, 
 meant for
 performance and usually experts who want to squeeze it out


 the vast
 middle of day-to-day programmers to crank out libraries and 
 apps that
 perform reasonably well

 3. high-level "scripting" languages like Ruby and Python, 
 meant for
 those who don't care too much for performance but just want 
 to get
 working code

 I think D is positioned somewhere between 1 and 2, though 
 closer to 1.
 However, there is sometimes talk of using D for all three, 
 though
 perhaps that is only meant as an added benefit for people 
 already using
 it for 1 or 2, ie those who already know the language better.
You're falling into the common fallacy that those groups are mutually exclusive and that a single language can't be appropriate for more than one. D is all about proving that wrong, and is meant for, and good at, all three.
There are good reasons for this split, and yes, it is probably impossible for one language to attract all three groups. You could use languages from 1 and 2 for all three, with a bit more work, but I don't see many scripts written in C++. :) The reason for the split is that there are different levels of software expertise and performance needs, and each of those groups is geared for a different level. Show templates to a scripting user and they probably run away screaming. D can probably do well with groups 1 and 2, but the level of power and expertise that is needed for those lower levels will scare away people from 3. Those already using it for 1 and 2 may also be comfortable with reusing D for scripting, but that's not attracting people from group 3, ie those who only want something easy to use and don't want to know the difference between a static and dynamic array.
 I've noticed that, for many of the people who don't "get" D, 
 the problem they're hitting is that they're minds are so 
 twisted around by the "polyglot" culture, that they're looking 
 for "the one" tiny little niche that D is for, not seeing 
 that, and thus missing the whole entire point.
Yes, this has definitely hurt D, being stuck between 1 and 2. People from 2 probably look at D and think it's too low-level. People from 1 are always looking to squeeze out _more_ performance, so the GC is a way for them to just write off D. You and I think they're making a mistake, but maybe they're not wrong for their own uses. As I said, the recent push for nogc and C++ compatibility suggests that a renewed effort is being made to focus on group 1, particularly when combined with the D benchmarks for regex and recently math. I'm happy in that space between 1 and 2, and the recent push to move languages from 2 to AoT compilation suggests that is a good place to be. So maybe group 2 will also come to us. :)
Oct 30 2016