www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D compilation is too slow and I am forking the compiler

reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
Nov 21 2018
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:

 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/


You gave me a fright there with the title there for a moment. 
Awesome stuff though. Not sure how easy it will be to upstream 
considering this needs to not wreck Windows and needs to work 
with LDC/GDC (at least we have inlining in the backend).
Nov 21 2018
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 21 November 2018 at 08:32:39 UTC, Nicholas Wilson 
wrote:

 You gave me a fright there with the title there for a moment.


:)


 Awesome stuff though. Not sure how easy it will be to upstream 
 considering this needs to not wreck Windows and needs to work 
 with LDC/GDC (at least we have inlining in the backend).


All the DMD-side logic is all encapsulated in one function:

https://github.com/CyberShadow/dmd/blob/dmdforker/src/dmd/mars.d#L501-L673

Its body can be versioned out in incompatible 
platforms/implementations.
Nov 21 2018
prev sibling next sibling parent bauss <jj_1337 live.dk> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:

 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/


Not only an interesting read, but also interesting research!
Nov 21 2018
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:

 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/


You might want to have a brush up on which direction C++ modules 
are heading in.  Notable talks would be those given at the GNU 
Cauldron for both 2017 and 2018.  The general run-down as I 
understand it.

===
Problem to solve: Compiler asks an Oracle about module A.

Phrased this way, Compiler is a client, Oracle is a server.

Oracle could be a file, socket, remote server, anything that can 
be read from or written to.

Communication can be done via a standard format (such as json).

This means that the Oracle (the implementation of) that keeps 
track of compilation and dependencies of the build is now someone 
else's problem as far as the Compiler is concerned.
===

I think what you've already started would fit well into this.

Iain.
Nov 21 2018
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 12:07 AM, Vladimir Panteleev wrote:

 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-f
rking-the-compiler/ 


I implemented precompiled headers for Digital Mars C++. It took a loooong time 
to work all the bugs out of it. It's also a brittle system. It works by 
allocating memory from a memory-mapped file, which serves as the precompiled
header.
Nov 21 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 21 November 2018 at 09:46:44 UTC, Walter Bright 
wrote:

 It works by allocating memory from a memory-mapped file, which 
 serves as the precompiled header.


Hey, that's a great idea! Can we do this for DMD? :D

On a more serious note: do you think that with D's features (type 
system / metaprogramming), you could have avoided some of those 
bugs?

For example, one thing we can do in D which is still impossible 
in C++ is to automatically serialize/deserialize all fields of a 
struct/class (using tupleof / allMembers).
Nov 21 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 2:16 AM, Vladimir Panteleev wrote:

 On Wednesday, 21 November 2018 at 09:46:44 UTC, Walter Bright wrote:

 It works by allocating memory from a memory-mapped file, which serves as the 
 precompiled header.


 
 Hey, that's a great idea! Can we do this for DMD? :D
 
 On a more serious note: do you think that with D's features (type system / 
 metaprogramming), you could have avoided some of those bugs?
 
 For example, one thing we can do in D which is still impossible in C++ is to 
 automatically serialize/deserialize all fields of a struct/class (using
tupleof 
 / allMembers).
 


Memory mapped files really were the key to success, because if you could reload 
the mmf at the same address, the pointers did not have to be patched. In the 
DMC++ source code, "dehydrating" a pointer meant subtracting a value from it so 
it was correct for the base address of the mmf, and "hydrating" a pointer was 
the inverse.

The two bug prone problems were:

1. separating out the tangled data structures into what goes into the pch, and 
what does not. Obviously, nothing in the pch could point outside of it.

2. .h files are simply not compatible with this, so you've got to detect when
it 
won't work. For example, anything like a command line switch or a macro that 
might cause different code to be generated in the pch had to invalidate it.

Maybe I should have done your fork idea? :-)

My experience with this drove many design decisions for D modules, for example, 
D modules are unaffected by where they are imported, the order they are 
imported, or the number of times they are imported. (Yes, I know about 
https://digitalmars.com/d/archives/digitalmars/D/D_needs_to_be_honest_320976.html)

Anyhow, what I've thought about doing since the beginning was make DMD 
multithreaded. The language is designed to support multithreaded compilation. 
For example, lexing, parsing, semantic analysis, optimization, and code 
generation can all be done concurrently.

DMD 1.0 would read imports in a separate thread. This would speed things up if 
you were using a slow filesystem, like NAS or a USB stick, but it was
eventually 
disabled because there wasn't a perceptible speedup with current filesystems.

Wouldn't it be awesome to have the lexing/parsing of the imports all done in 
parallel? The main difficulty in getting that to work is dealing with the
shared 
string table.
Nov 21 2018
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright 
wrote:

 Wouldn't it be awesome to have the lexing/parsing of the 
 imports all done in parallel? The main difficulty in getting 
 that to work is dealing with the shared string table.


What about creating a new Fiber for each module needing 
lexing/parsing/semantic to be ran?  Compilation of one module 
would then get as far as it can until it needs to defer, then 
calls yield() to continue compilation of the next module.  This 
in hope that when the round trip returns, the AST will be 
sufficiently complete enough for compilation to continue.
Nov 21 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 3:19 AM, Iain Buclaw wrote:

 On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright wrote:

 Wouldn't it be awesome to have the lexing/parsing of the imports all done in 
 parallel? The main difficulty in getting that to work is dealing with the 
 shared string table.


 
 What about creating a new Fiber for each module needing
lexing/parsing/semantic 
 to be ran?  Compilation of one module would then get as far as it can until
it 
 needs to defer, then calls yield() to continue compilation of the next module.
 
 This in hope that when the round trip returns, the AST will be sufficiently 
 complete enough for compilation to continue.


Since the lexing/parsing does not do any blocking calls, fibers won't help. It 
has to be another hardware thread.

Trying to parallelize semantic over multiple modules will be very hard to do.
Nov 21 2018
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2018-11-21 11:56, Walter Bright wrote:


 Wouldn't it be awesome to have the lexing/parsing of the imports all 
 done in parallel? The main difficulty in getting that to work is dealing 
 with the shared string table.


Would it be possible to have one string table per thread and merge them 
to one single shared string table before continuing with the next phase?

-- 
/Jacob Carlborg
Nov 23 2018
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/23/2018 2:12 AM, Jacob Carlborg wrote:

 Would it be possible to have one string table per thread and merge them to one 
 single shared string table before continuing with the next phase?


It'd probably be even slower because one would have to rewrite all the pointers 
into the string table.
Nov 23 2018
prev sibling parent reply welkam <wwwelkam gmail.com> writes:
On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright 
wrote:

 Wouldn't it be awesome to have the lexing/parsing of the 
 imports all done in parallel?


 From my testing lexing/parsing takes small amount of build time 
so running it in parallel might be small gain. We should consider 
running in parallel more heavy hitting features like CTFE and 
templates.

Since we are in wish land here is my wishes. Currently D reads 
the all files that are passed in command line before starting 
lexing/parsing, but in principle we could start lexing/parsing 
after first file is read. In fact we could start after first 
file`s first line is read. Out of all operation before semantic 
pass, reading from hard disk should be the slowest so it might be 
possible to decode utf-8, lex and parse at the speed of reading 
from hard disk. If we run these steps in different thread on the 
same core with SMT we could better use core`s resources. Reading 
file with kernel, decoding UTF-8 with vector instructions and 
lexing/parsing with scalar operations while all communication is 
done trough L1 and L2 cache.

I thought about using memory mapped files to unblock file reading 
as a first step but lack of good documentation about mmf and lack 
of thorough understanding of front end made me postpone this 
modification. Its a change with little benefit.


 The main difficulty in getting that to work is dealing with the 
 shared string table.


At begging of parsing a thread could get a read-only shared slice 
of string table. All strings not in table are put in local string 
table. After parsing tables are merged and shared slice is 
updated so new thread could start with bigger table. this assumes 
that table is not sorted
Nov 23 2018
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 23 November 2018 at 13:23:22 UTC, welkam wrote:

 If we run these steps in different thread on the same core with 
 SMT we could better use core`s resources. Reading file with 
 kernel, decoding UTF-8 with vector instructions and 
 lexing/parsing with scalar operations while all communication 
 is done trough L1 and L2 cache.


You might save some pages from the data cache, but by doing more 
work at once, the code might stop fitting in the 
execution-related caches (code pages, microcode, branch 
prediction) instead.
Nov 23 2018
parent reply welkam <wwwelkam gmail.com> writes:
On Friday, 23 November 2018 at 14:32:39 UTC, Vladimir Panteleev 
wrote:

 On Friday, 23 November 2018 at 13:23:22 UTC, welkam wrote:

 If we run these steps in different thread on the same core 
 with SMT we could better use core`s resources. Reading file 
 with kernel, decoding UTF-8 with vector instructions and 
 lexing/parsing with scalar operations while all communication 
 is done trough L1 and L2 cache.


 You might save some pages from the data cache, but by doing 
 more work at once, the code might stop fitting in the 
 execution-related caches (code pages, microcode, branch 
 prediction) instead.


Its not about saving tlb pages or fitting better in cache. 
Compilers are considered streaming applications - they dont 
utilize cpu caches effectively. You cant read one character and 
emit machine code then read next character you have to go over 
all data multiple times while you modify it. I can find white 
papers if you interested where people test GCC with different 
cache architectures and it doesnt make much of a difference. GCC 
is popular application when testing caches.

Here are profiling data from DMD
  Performance counter stats for 'dmd -c main.d':


utilized



M/sec

GHz

frontend cycles idle

backend cycles idle

per cycle

stalled cycles per insn

M/sec

all branches

        0.747720395 seconds time elapsed

        0.497698000 seconds user
        0.104165000 seconds sys

Most important data in this conversation is 0.82  insn per cycle. 
My CPU could do ~2 IPC so there are plenty of CPU resources 
available. New Intel desktop processors are designed to do 4 
insn/cycle. What is limiting DMD performance is slow RAM, data 
fetching and not what you listed.
code pages - you mean TLB here?

microcode cache. Not all processors have it and those who have 
only benefit trivial loops. DMD have complex loops.

branch prediction. More entries in branch predictor wont help 
here because branches are missed because data is unpredictable 
not because there are too many branches. Also branch 
missprediction penalty is around 30 cycles while reading from RAM 
could be over 200 cycles.

L1 code cache. You didnt mention this but running those tasks in 
SMT mode might trash L1$ so execution might not be optimal.

Instead of parallel reading of imports DMD needs more data 
oriented data structures instead of old OOP inspired data 
structures. Ill give you example why its the case.

Consider
struct {
     bool isAlive;
     <other data at least 7 bytes of size>
}

If you want to read data from that bool CPU needs to fetch 8 
bytes of data(cache line of 64 bits). What this means is that for 
one bit of information CPU fetches 64 bits of data resulting in 
1/64 = 0.015625 or ~1.6 % signal to noise ratio. This is terrible!

AFAIK DMD doesnt make this kind of mistake but its full of large 
structs and classes that are not efficient to read.  To fix this 
we need to split those large data structures into smaller ones 
that only contain what is needed for particular algorithm. I 
predict 2x speed improvement if we transform all data structures 
in DMD. Thats improvement without improving algorithms only 
changing data structures. This getting too longs so i will stop 
right now
Nov 23 2018
parent welkam <wwwelkam gmail.com> writes:
On Friday, 23 November 2018 at 16:21:53 UTC, welkam wrote:

 If you want to read data from that bool CPU needs to fetch 8 
 bytes of data(cache line of 64 bits). What this means is that 
 for one bit of information CPU fetches 64 bits of data 
 resulting in 1/64 = 0.015625 or ~1.6 % signal to noise ratio. 
 This is terrible!


Cache line of 64 bits. 64 BITS. This forum is full of 
knowledgeable people and they should have spoted this mistake or 
they didnt read it. Cache lines on most processors are 64 bytes. 
Now I know why it felt weird when I wrote that post. So the real 
math for when you read one bit in cache line is 1/(64*8) = 
0.001953125 or ~ 0.2% signal to noise ratio
Dec 11 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/23/2018 5:23 AM, welkam wrote:

 Currently D reads the all files 
 that are passed in command line before starting lexing/parsing, but in
principle 
 we could start lexing/parsing after first file is read. In fact we could start 
 after first file`s first line is read.


DMD used to do that. But it was removed because:

1. nobody understood the logic

2. it didn't seem to make a difference

You can still see the vestiges by the:

     static if (ASYNCREAD)

blocks in the code.
Nov 23 2018
parent welkam <wwwelkam gmail.com> writes:
On Friday, 23 November 2018 at 19:21:03 UTC, Walter Bright wrote:

 On 11/23/2018 5:23 AM, welkam wrote:

 Currently D reads the all files that are passed in command 
 line before starting lexing/parsing, but in principle we could 
 start lexing/parsing after first file is read. In fact we 
 could start after first file`s first line is read.


 DMD used to do that. But it was removed because:

 1. nobody understood the logic

 2. it didn't seem to make a difference

 You can still see the vestiges by the:

     static if (ASYNCREAD)

 blocks in the code.


I didnt expect huge wins. This would be useful when you start 
your computer and files have to be read from old spinning rust 
and the project has many files. Otherwise files will be cached 
and memcopy is fast. I was surprised on how fast modern computers 
copy data from one place to another.

Speaking of memcpy here is a video you might like. It has memcpy, 
assembler and a bit of compiler. Its very easy watch for when you 
want to relax.
Level1 Diagnostic: Fixing our Memcpy Troubles (for Looking Glass)
https://www.youtube.com/watch?v=idauoNVwWYE
Nov 23 2018
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:

 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
Nov 21 2018
next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson 
wrote:

 On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
 Panteleev wrote:

 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/





I would be wary of such titles.

Whatever will remain in minds will be something like "D 
compilation is slow" which isn't accurate when compared to the 
competition.

The article is clever, the title is clever, but most people will 
only read the title.
Nov 21 2018
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 5:55 AM, Guillaume Piolat wrote:

 On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson wrote:

 On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote:

 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-f
rking-the-compiler/ 





 
 I would be wary of such titles.
 
 Whatever will remain in minds will be something like "D compilation is slow" 
 which isn't accurate when compared to the competition.
 
 The article is clever, the title is clever, but most people will only read the 
 title.


Unfortunately, you're right. The title will leave the impression "D is slow at 
compiling". You have to carefully read the article to see otherwise, and few 
will do that.
Nov 21 2018
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
I neglected point out, however, that the article itself is a home run! Thank 
you, Vladimir!
Nov 21 2018
prev sibling next sibling parent Daniel Kozak <kozzi11 gmail.com> writes:
I would say opposite :)

On Wed, Nov 21, 2018 at 9:55 PM Walter Bright via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On 11/21/2018 5:55 AM, Guillaume Piolat wrote:

 On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson wrote:

 On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev




 wrote:






 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/





 I would be wary of such titles.

 Whatever will remain in minds will be something like "D compilation is


 slow"

 which isn't accurate when compared to the competition.

 The article is clever, the title is clever, but most people will only


 read the

 title.


 Unfortunately, you're right. The title will leave the impression "D is
 slow at
 compiling". You have to carefully read the article to see otherwise, and
 few
 will do that.





 Nov 21 2018



prev sibling next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com>  writes:


On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
wrote:

 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully read 
 the article to see otherwise, and few will do that.


Sorry about that. I'll have to think of two titles next time, one 
for the D community and one for everyone else.

If it's of any consolation, the top comments in both discussion 
threads point out that the title is inaccurate on purpose.



 Nov 21 2018



 next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/21/2018 8:48 PM, Vladimir Panteleev wrote:

 On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright wrote:

 Unfortunately, you're right. The title will leave the impression "D is slow at 
 compiling". You have to carefully read the article to see otherwise, and few 
 will do that.


 
 Sorry about that. I'll have to think of two titles next time, one for the D 
 community and one for everyone else.
 
 If it's of any consolation, the top comments in both discussion threads point 
 out that the title is inaccurate on purpose.


It has indeed done well. In fact, the article is so good it is probably worth
it 
to have that attention-getting title. It is a risky strategy, though :-)



 Nov 21 2018



prev sibling next sibling parent reply welkam <wwwelkam gmail.com>  writes:


On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir Panteleev 
wrote:

 Sorry about that. I'll have to think of two titles next time, 
 one for the D community and one for everyone else.

 If it's of any consolation, the top comments in both discussion 
 threads point out that the title is inaccurate on purpose.


Your post on reddit received more comments than D front ends 
inclusion to GCC. If you titled your post differently you 
probably wouldn't had such success so from my perspective its a 
net positive. Sure there are few people that took the wrong 
message but there are more people who saw your post



 Nov 23 2018



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/23/2018 6:37 AM, welkam wrote:

 Your post on reddit received more comments than D front ends inclusion to GCC. 
 If you titled your post differently you probably wouldn't had such success so 
 from my perspective its a net positive. Sure there are few people that took
the 
 wrong message but there are more people who saw your post


It definitely shows the value of a provocative title!



 Nov 23 2018



prev sibling  parent reply Guillaume Piolat <first.last gmail.com>  writes:


On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir Panteleev 
wrote:

 On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
 wrote:

 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully 
 read the article to see otherwise, and few will do that.


 Sorry about that. I'll have to think of two titles next time, 
 one for the D community and one for everyone else.

 If it's of any consolation, the top comments in both discussion 
 threads point out that the title is inaccurate on purpose.


Please don't get me wrong, it's an excellent article, a 
provocative title, and fantastic work going on. I didn't meant to 
hurt!

In my opinion language adoption is a seduction/sales process very 
much like business-to-consumer is, the way I see it it's 
strikingly similar to marketing B2C apps, unless there will be no 
"impulse buy".

Actually no less than 3 programmer friends came to (I'm the 
weirdo-using-D and people are _always_ in disbelief and invent 
all sorts of reasons not to try) saying they saw an article on D 
on HN, with "D compilation is slow", and on further examination 
they didn't read or at best the first paragraph. But they did 
remember the title. They may rationally think their opinion of D 
hasn't changed: aren't we highly capable people?

I'm not making that up! So why is it a problem ?

HN may be the only time they hear about D. The words of the title 
may be their only contact with it. The first 3 words of the title 
may be the only thing associated with the "D language" chunk in 
their brain.

The associative mind doesn't know _negation_ so even a title like 
"D compilation wasn't fast so I forked the compiler" is better 
from a marketing point of view since it contains the word "fast" 
in it! That's why marketing people have the annoying habit of 
using positive words, you may think this stuff is unimportant but 
this is actually the important meat.

Reasonable people may think marketing and biases don't apply to 
them but they do, it works without your consent.



 Nov 26 2018



 next sibling parent reply Joakim <dlang joakim.fea.st>  writes:


On Monday, 26 November 2018 at 16:00:36 UTC, Guillaume Piolat 
wrote:

 On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir 
 Panteleev wrote:

 On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
 wrote:

 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully 
 read the article to see otherwise, and few will do that.


 Sorry about that. I'll have to think of two titles next time, 
 one for the D community and one for everyone else.

 If it's of any consolation, the top comments in both 
 discussion threads point out that the title is inaccurate on 
 purpose.


 Please don't get me wrong, it's an excellent article, a 
 provocative title, and fantastic work going on. I didn't meant 
 to hurt!

 In my opinion language adoption is a seduction/sales process 
 very much like business-to-consumer is, the way I see it it's 
 strikingly similar to marketing B2C apps, unless there will be 
 no "impulse buy".


I find that hard to believe: we are talking about a technical 
tool here.

Also, regardless of how languages are chosen as they get into the 
majority, D is very much still in the innovators/early-adopters 
stage:

https://en.m.wikipedia.org/wiki/Technology_adoption_life_cycle

That is a very different type of sales process, much more geared 
towards what the new tech can actually do.


 Actually no less than 3 programmer friends came to (I'm the 
 weirdo-using-D and people are _always_ in disbelief and invent 
 all sorts of reasons not to try) saying they saw an article on 
 D on HN, with "D compilation is slow", and on further 
 examination they didn't read or at best the first paragraph. 
 But they did remember the title. They may rationally think 
 their opinion of D hasn't changed: aren't we highly capable 
 people?


With people like that, it's almost impossible to get them in the 
early adopter stage. They will only jump on the bandwagon once 
it's full, ie as part of the late majority.


 I'm not making that up! So why is it a problem ?

 HN may be the only time they hear about D. The words of the 
 title may be their only contact with it. The first 3 words of 
 the title may be the only thing associated with the "D 
 language" chunk in their brain.

 The associative mind doesn't know _negation_ so even a title 
 like "D compilation wasn't fast so I forked the compiler" is 
 better from a marketing point of view since it contains the 
 word "fast" in it! That's why marketing people have the 
 annoying habit of using positive words, you may think this 
 stuff is unimportant but this is actually the important meat.

 Reasonable people may think marketing and biases don't apply to 
 them but they do, it works without your consent.


I agree that it was a risky title, as many who don't know D will 
simply see it and go, "Yet another slow compiler, eh, I'll pass" 
and not click on the link. Whereas others who have heard 
something of D will be intrigued, as they know it's already 
supposed to compile fast. And yet more others will click on it 
purely for the controversy, just to gawk at some technical 
bickering.

Given how well it did on HN/reddit/lobste.rs, I think Vlad's 
gamble probably paid off. We can't run the counterfactual of 
choosing a safer title to see if it would have done even better, 
let's just say it did well enough. ;)



 Nov 26 2018



 next sibling parent reply bachmeier <no spam.net>  writes:


On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote:


 I agree that it was a risky title, as many who don't know D 
 will simply see it and go, "Yet another slow compiler, eh, I'll 
 pass" and not click on the link. Whereas others who have heard 
 something of D will be intrigued, as they know it's already 
 supposed to compile fast. And yet more others will click on it 
 purely for the controversy, just to gawk at some technical 
 bickering.


I don't actually think it was risky. What are the odds that 
someone was going to start using D for a major project but then 
changed her mind upon seeing a title on HN or Reddit? Probably 
very small that even one person did that.

On the other hand, it says a lot of other things:

- There's an active community that cares about the language.
- It's not a dying language.
- Fast compilation is a realistic possibility.
- There are users with the technical ability to make the compiler 
faster.

And then there is always the fact that there was a story on 
HN/Reddit about D. It's hard for publicity for a language like D 
to be bad when so few people use it.



 Nov 26 2018



  parent reply Joakim <dlang joakim.fea.st>  writes:


On Monday, 26 November 2018 at 16:42:40 UTC, bachmeier wrote:

 On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote:


 I agree that it was a risky title, as many who don't know D 
 will simply see it and go, "Yet another slow compiler, eh, 
 I'll pass" and not click on the link. Whereas others who have 
 heard something of D will be intrigued, as they know it's 
 already supposed to compile fast. And yet more others will 
 click on it purely for the controversy, just to gawk at some 
 technical bickering.


 I don't actually think it was risky. What are the odds that 
 someone was going to start using D for a major project but then 
 changed her mind upon seeing a title on HN or Reddit? Probably 
 very small that even one person did that.


Yes, but you're ignoring the much larger group I mentioned- those 
who only vaguely heard of D, if at all- and the negative title 
gives them a reason not to look into it further.


 And then there is always the fact that there was a story on 
 HN/Reddit about D. It's hard for publicity for a language like 
 D to be bad when so few people use it.


The quote that "there's no such thing as bad publicity" comes 
from art and show business though, don't think it's true for tech 
and other markets. When your audience is looking for a tool and 
not entertainment, there's lots of ways for bad publicity to sink 
it.

Anyway, I noted in this case that the provocative title may 
actually have gotten more people to read a positive post, so the 
pros likely outweighed the cons. We can just never know how large 
the unclicked-on downside was: you can never measure how many 
people heard of but _didn't_ buy your book, because they didn't 
like the title or something else about its exterior.

On Monday, 26 November 2018 at 16:53:59 UTC, Guillaume Piolat 
wrote:

 On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote:

 In my opinion language adoption is a seduction/sales process 
 very much like business-to-consumer is, the way I see it it's 
 strikingly similar to marketing B2C apps, unless there will 
 be no "impulse buy".


 I find that hard to believe: we are talking about a technical 
 tool here.


 How many times have you been in this conversation:

 --------------------------

 - What language are you using?
 - D.
 - I know next to nothing about D.
 - Oh, it's very good, I even built a business on it! <laundry 
 list of arguments and features>.
 - Oh no thanks. I should try Rust, it's secure, fast, modern 
 blah blah; facts don't matter to me. But in reality I won't 
 even learn a new language, I'm happy with a language without 
 multi-threading.

 --------------------------

 It happens to me ALL THE TIME.
 This pattern is so predictable it's becoming boring so now I 
 just keep silent.


Never, I don't go around trying to convince people one-on-one to 
use D. I have given talks to groups introducing the language, 
that's how I go about it.


 What happens? Rust / Go have outmarketed us with words.

 The battle (of marketing) is on words not technical features, 
 Rust happen to own "programming language" + "safety", what do 
 we own? D is good in all kinds of directions and the marketing 
 message is less simple.

 The leaders choose to own the word "fast" (see our new motto 
 "fast code, fast" which is very accurate) and it's important to 
 get aligned.


I'll note that in your example they haven't actually learnt Rust 
either. I don't think marketing is that relevant for D at this 
stage, nor for Rust/Go either.

The way anything- tech, fashion, TV shows- becomes popular is 
that some early tastemaker decides that it's worth using or 
backing. Eventually, enough early adopters find value that it 
spreads out to the majority, who simply follow their lead.

Most people aren't early adopters of most things. They like to 
think they are, so they'll give you all kinds of 
rational-sounding reasons for why they don't like some new tech, 
but the real underlying thought process goes something like this, 
"I have no idea if this new tech will do well or not. I could 
take a risk on it but it's safer not to, so I will just wait and 
see if it gets popular, then follow the herd."

Very few will admit this though, hence the list of 
plausible-sounding reasons that don't actually make sense! ;)

As Laeeth always says, you're best off looking for people who're 
actually capable and empowered to make such risky decisions, 
rather than aiming for the majority too early, because they only 
jump on board once the bandwagon is stuffed and rolling downhill.


 Also, regardless of how languages are chosen as they get into 
 the majority, D is very much still in the 
 innovators/early-adopters stage:


 But the current state of D would very much accomodate the 
 middle-of-the-curve adopters. The language rarely breaks stuff. 
 People making money with it, making long-term bets.

 Hell, I could make a laundry list of things that are better in 
 D versus any alternatives! That doesn't bring users.


I'm not talking about the quality of the product. I'm talking 
about the current size of the userbase, which is still in the 
early adopter stage.


 With people like that, it's almost impossible to get them in 
 the early adopter stage. They will only jump on the bandwagon 
 once it's full, ie as part of the late majority.


 There is a gap where we are, but "People like that" are almost 
 everyone.


Yes, this is why you must ignore almost everyone. It is a waste 
of time, because they will not take a risk with new tech.


 Those people actually are middle-of-the-curve adopter, if you 
 see a true late adopter in the wild it takes 3 relatives 
 programming in D so that they start to be interested.


Heh, perhaps.


 Who doesn't want to be out of the early adopter stage, and get 
 into the "officially endorsed safe choice" cohort?


All of us, but you cannot skip steps, not how it works.


 D is remarkably ready as a safe choice for lots of software.



 Given how well it did on HN/reddit/lobste.rs, I think Vlad's 
 gamble probably paid off. We can't run the counterfactual of 
 choosing a safer title to see if it would have done even 
 better, let's just say it did well enough. ;)


 Alternative darker view: ever remarked how D articles often 
 goes downvoted on HN? The title who says something bad about D 
 is upvoted ; it's easy to see events as going our way. I, for 
 one, didn't really read the article. Who has time for that?


Maybe you should read it, some of the proggitors even created a 
new reddit sub based on this post, coining the term ClickGold, 
for when a clickbait article ends up being much _more_ 
interesting than its leading title: ;)

https://www.reddit.com/r/programming/comments/9z36xg/comment/ea6afce



 Nov 27 2018



  parent  Guillaume Piolat <first.last gmail.com>  writes:


On Tuesday, 27 November 2018 at 14:19:12 UTC, Joakim wrote:

 As Laeeth always says, you're best off looking for people 
 who're actually capable and empowered to make such risky 
 decisions, rather than aiming for the majority too early, 
 because they only jump on board once the bandwagon is stuffed 
 and rolling downhill.


I get the (I read "D compilation is slow" + "I'm not interested") 
vibe from people that actually _are_ early adopters, very 
interested in programming, who adopted Scala and Nim despite 
obvious risks.

I disagree with your opinion but just lacks the time to defend my 
own in a lengthy forum post. Let me make a giant argument by 
authority: I sell B2C software for a living. Convincing people to 
try software despite their will is what I do for a living. "Build 
it and they will come" would be a fantasy to believe if we hadn't 
competitors who had people come way before they built it.

Complicated triple negation arguments (D compile fast! no wait, 
it compiles slowly! no wait, it compiles fast!) don't work. In a 
few months whenever someone bring out D, forum replies will say 
"but D compilation is not that fast".



 Nov 27 2018



prev sibling  parent reply Guillaume Piolat <first.last gmail.com>  writes:


On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote:

 In my opinion language adoption is a seduction/sales process 
 very much like business-to-consumer is, the way I see it it's 
 strikingly similar to marketing B2C apps, unless there will be 
 no "impulse buy".


 I find that hard to believe: we are talking about a technical 
 tool here.


How many times have you been in this conversation:

--------------------------

- What language are you using?
- D.
- I know next to nothing about D.
- Oh, it's very good, I even built a business on it! <laundry 
list of arguments and features>.
- Oh no thanks. I should try Rust, it's secure, fast, modern blah 
blah; facts don't matter to me. But in reality I won't even learn 
a new language, I'm happy with a language without multi-threading.

--------------------------

It happens to me ALL THE TIME.
This pattern is so predictable it's becoming boring so now I just 
keep silent.

What happens? Rust / Go have outmarketed us with words.

The battle (of marketing) is on words not technical features, 
Rust happen to own "programming language" + "safety", what do we 
own? D is good in all kinds of directions and the marketing 
message is less simple.

The leaders choose to own the word "fast" (see our new motto 
"fast code, fast" which is very accurate) and it's important to 
get aligned.



 Also, regardless of how languages are chosen as they get into 
 the majority, D is very much still in the 
 innovators/early-adopters stage:


But the current state of D would very much accomodate the 
middle-of-the-curve adopters. The language rarely breaks stuff. 
People making money with it, making long-term bets.

Hell, I could make a laundry list of things that are better in D 
versus any alternatives! That doesn't bring users.



 With people like that, it's almost impossible to get them in 
 the early adopter stage. They will only jump on the bandwagon 
 once it's full, ie as part of the late majority.


There is a gap where we are, but "People like that" are almost 
everyone.

Those people actually are middle-of-the-curve adopter, if you see 
a true late adopter in the wild it takes 3 relatives programming 
in D so that they start to be interested.

Who doesn't want to be out of the early adopter stage, and get 
into the "officially endorsed safe choice" cohort?

D is remarkably ready as a safe choice for lots of software.



 Given how well it did on HN/reddit/lobste.rs, I think Vlad's 
 gamble probably paid off. We can't run the counterfactual of 
 choosing a safer title to see if it would have done even 
 better, let's just say it did well enough. ;)


Alternative darker view: ever remarked how D articles often goes 
downvoted on HN? The title who says something bad about D is 
upvoted ; it's easy to see events as going our way. I, for one, 
didn't really read the article. Who has time for that?



 Nov 26 2018



  parent  "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com>  writes:


On 11/26/18 11:53 AM, Guillaume Piolat wrote:

 
 How many times have you been in this conversation:
 
 --------------------------
 
 - What language are you using?
 - D.
 - I know next to nothing about D.
 - Oh, it's very good, I even built a business on it! <laundry list of 
 arguments and features>.
 - Oh no thanks. I should try Rust, it's secure, fast, modern blah blah; 
 facts don't matter to me. But in reality I won't even learn a new 
 language, I'm happy with a language without multi-threading.
 
 --------------------------
 
 It happens to me ALL THE TIME.
 This pattern is so predictable it's becoming boring so now I just keep 
 silent.
 


That's no surprise. The modern tech world is absolutely flooded with 
imbeciles and fashion-driven people.



 Nov 27 2018



prev sibling  parent reply Laeeth Isharc <laeeth laeeth.com>  writes:


On Monday, 26 November 2018 at 16:00:36 UTC, Guillaume Piolat 
wrote:

 On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir 
 Panteleev wrote:

 On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
 wrote:

 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully 
 read the article to see otherwise, and few will do that.


 Sorry about that. I'll have to think of two titles next time, 
 one for the D community and one for everyone else.

 If it's of any consolation, the top comments in both 
 discussion threads point out that the title is inaccurate on 
 purpose.


 Please don't get me wrong, it's an excellent article, a 
 provocative title, and fantastic work going on. I didn't meant 
 to hurt!

 In my opinion language adoption is a seduction/sales process 
 very much like business-to-consumer is, the way I see it it's 
 strikingly similar to marketing B2C apps, unless there will be 
 no "impulse buy".


I think that there are different strategies - decent appeal to a 
broad market and having a very high appeal to a small market (but 
there has better be something good about your potential customer 
base ie 'D, if you find VBA too difficult' is probably not a good 
strategy!).  And you probably don't get to pick which situation 
you are in, and then one had better realise it and play the game 
you're in.  The particular kind of market will shape what works - 
in my business you approach a retail client base differently from 
regular institutional investors and then the worlds' largest 
pools of money involved something else again.

D isn't really marketed and it's definitely not sold.  That's an 
implicit strategy in itself.

Nassim Taleb raises the question of how do you choose between two 
surgeons, both recommended.  One looks the part and hangs his 
many certificates on his office wall.  The other looks scruffy 
with the appearance of a tradesman.  Who do you pick?  Taleb says 
pick the guy who doesn't look the part because if he got there 
without signalling he must have something going for him.

But in general you can appeal on merits mostly to an audience 
that is highly discerning and very capable.  If you haven't got 
any money to appeal to an audience that judges based on 
heuristics and social factors well then you can try to avoid 
accidentally putting people off, you can be creative with 
guerilla marketing but the key thing is to make the most of what 
you got.  If everyone else does things a certain way then if for 
some reason that's closed off to you for now then if you look 
closely, with active perception,you may well see opportunities 
that are neglected to approach the problem another way.


 Actually no less than 3 programmer friends came to (I'm the 
 weirdo-using-D and people are _always_ in disbelief and invent 
 all sorts of reasons not to try) saying they saw an article on 
 D on HN, with "D compilation is slow", and on further 
 examination they didn't read or at best the first paragraph. 
 But they did remember the title. They may rationally think 
 their opinion of D hasn't changed: aren't we highly capable 
 people?


It doesn't matter what most people think.  It matters what people 
who are on the fence or using D already a bit think.  Or people 
who have a lot of problems to which D is in part a solution only 
they didn't know about or think of D yet.

The messenger matters too.  If someone you trust and rate highly 
tells you something based on their experience that counts for a 
lot more than all the blog posts in the world.  And working code 
and lived experience dominates the social talk about it.

I've talked about D with the CTO of Bloomberg, the outgoing COO 
of Barclays investment bank, the number two guy at a 30bn hedge 
fund, the COO of the largest hedge fund in the world (depending 
on how you count) and more.  That's not going to change anything 
tomorrow but in time those kinds of conversations matter much 
more than what people might say on Reddit. It's not either /or of 
course, but it's just not worth sweating your reviews.

Finally the reasons people buy things are not what you might 
reasonably think!  Ask Walter how he was able to compete 
successfully for so long as a one man band with Microsoft.  I 
don't think his edge was in the beginning something calculated.



 Reasonable people may think marketing and biases don't apply to 
 them but they do, it works without your consent.


The thing is that we had a bubble in synthetic manufactured 
marketing.  And now increasingly people are tired of that and 
seek what's authentic, real and that doesn't pretend to be 
perfect.

That doesn't mean a bit of thought is a bad idea,just that it 
might matter less than you think that the D community isn't 
particularly interested in marketing.  Sometimes one can see that 
hidden in what superficially seems to be a weakness is a strength.



 Nov 28 2018



 next sibling parent reply Guillaume Piolat <first.last gmail.com>  writes:


On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
wrote:

 D isn't really marketed and it's definitely not sold.  That's 
 an implicit strategy in itself.



What I see in my (absurdly competitive) market is that the people 
that truly do no-marketing tend to close shop, sometimes despite 
very competitive offerings.

It colors my perception of course, since it can be very tempting 
to appeal to a limited pool of discerning customers; but that 
would mean death.

Imho the ones that succeed doing "no marketing" are actively 
telling others they don't do marketing. It can be sometimes 
funny, when the "no marketing" products comes with walls of text 
and videos that very much sound like... storytelling. But

Like in the "no-makeup makeup", there is still makeup but not 
obvious. 2018's marketing trend is to subtly fake complete 
honesty.



 Nov 28 2018



  parent  Laeeth isharc <laeeth laeeth.com>  writes:


On Wednesday, 28 November 2018 at 13:05:34 UTC, Guillaume Piolat 
wrote:

 On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
 wrote:

 D isn't really marketed and it's definitely not sold.  That's 
 an implicit strategy in itself.



 What I see in my (absurdly competitive) market is that the 
 people that truly do no-marketing tend to close shop, sometimes 
 despite very competitive offerings.

 It colors my perception of course, since it can be very 
 tempting to appeal to a limited pool of discerning customers; 
 but that would mean death.


What is the ratio of expenditure of your best customer to an 
average customer?   Not much.  That's one main reason why your 
intuition developed by organising your emotions according to your 
business domain fits this domain less.

What is the ratio of expenditure of the biggest 'customer' of 
Python to the average 'customer'?  Measured by resources lent to 
the community directly or indirectly, or by the wage bill of 
programmers at that firm working in Python this ratio is enormous.



 Nov 29 2018



prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com>  writes:


On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
wrote:

 Nassim Taleb raises the question of how do you choose between 
 two surgeons, both recommended.  One looks the part and hangs 
 his many certificates on his office wall.  The other looks 
 scruffy with the appearance of a tradesman.  Who do you pick?  
 Taleb says pick the guy who doesn't look the part because if he 
 got there without signalling he must have something going for 
 him.



It's definately the kind of surgeon one should choose - 
programmers that are not necessarily well groomed etc.. - but is 
it the kind of surgeon people will actually recommend? I'm 
doubtful.

If X has the social signalling then people will recommend X even 
without trying, because it's socially safe.

If one doesn't have the signalling, I've found the hard way even 
supporters will hesitate a bit before making recommendations, 
because of the social standing _cost_ it may have.

But then, perhaps recommendations don't matter, because opinions 
don't matter much? I think they matter to be even heard on public 
places.

And I think early adopters need a nudge, the influent need to be 
bothered by less influents (influencers are not especially on the 
lookout for new options, as they are already influent). Above all 
I think the niche of early-adopters is smaller than the larger 
market for languages, and the early-adopters are going elsewhere.



 Nov 28 2018



  parent reply Laeeth isharc <laeeth laeeth.com>  writes:


On Wednesday, 28 November 2018 at 13:30:37 UTC, Guillaume Piolat 
wrote:

 On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
 wrote:

 Nassim Taleb raises the question of how do you choose between 
 two surgeons, both recommended.  One looks the part and hangs 
 his many certificates on his office wall.  The other looks 
 scruffy with the appearance of a tradesman.  Who do you pick?  
 Taleb says pick the guy who doesn't look the part because if 
 he got there without signalling he must have something going 
 for him.



 It's definately the kind of surgeon one should choose - 
 programmers that are not necessarily well groomed etc.. - but 
 is it the kind of surgeon people will actually recommend? I'm 
 doubtful.

 If X has the social signalling then people will recommend X 
 even without trying, because it's socially safe.

 If one doesn't have the signalling, I've found the hard way 
 even supporters will hesitate a bit before making 
 recommendations, because of the social standing _cost_ it may 
 have.

 But then, perhaps recommendations don't matter, because 
 opinions don't matter much? I think they matter to be even 
 heard on public places.

 And I think early adopters need a nudge, the influent need to 
 be bothered by less influents (influencers are not especially 
 on the lookout for new options, as they are already influent). 
 Above all I think the niche of early-adopters is smaller than 
 the larger market for languages, and the early-adopters are 
 going elsewhere.


The innovator's dilemma, which is really an insight that dates 
back to Toynbee, and before that Ibn Khaldun, is not so obvious.  
I am not sure that you have understood it.  I suggest reading the 
book if you are interested, but otherwise I unfortunately don't 
have so much time at the moment to try to persuade you of what 
this phenomenon is like and there's limited value to talking 
about talking rather than having a discussion based on a shared 
understanding of what this is about.



 Nov 29 2018



  parent  Guillaume Piolat <first.last gmail.com>  writes:


On Thursday, 29 November 2018 at 09:18:35 UTC, Laeeth isharc 
wrote:

 The innovator's dilemma, which is really an insight that dates 
 back to Toynbee, and before that Ibn Khaldun, is not so 
 obvious.  I am not sure that you have understood it.  I suggest 
 reading the book if you are interested, but otherwise I 
 unfortunately don't have so much time at the moment to try to 
 persuade you of what this phenomenon is like and there's 
 limited value to talking about talking rather than having a 
 discussion based on a shared understanding of what this is 
 about.


No, indeed I've not read about it (yet, it looks like a tale of 
our lives).

My market has indeed a distribution where the best customer 
brings 3:1 vs the average, it's not inf:1 like it would be in 
programming language "customers". "Impulse buy" is predominent 
too, which does not exist for technical decisions.

I understand that a few big players will bring a lot more than 
hundreds of smaller ones. Especially if the smaller ones keep 
writing on the D newsgroup :)

In my market too, I prefer if D is kind of unpopular! I don't 
tell details to competitors other than "works for me". And they 
aren't much interested, which is satisfying.

This secrecy has to be balanced by the fact we have to hire, and 
D being a critical piece of infrastructure I'd like it to simply 
receive more money, being a small player I contribute what I can 
to the Foundation - and direct others that came to Dplug to do so.



 Nov 29 2018



prev sibling  parent  Nicholas Wilson <iamthewilsonator hotmail.com>  writes:


On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
wrote:

 I think that there are different strategies - decent appeal to 
 a broad market and having a very high appeal to a small market 
 (but there has better be something good about your potential 
 customer base ie 'D, if you find VBA too difficult' is probably 
 not a good strategy!).  And you probably don't get to pick 
 which situation you are in, and then one had better realise it 
 and play the game you're in.  The particular kind of market 
 will shape what works - in my business you approach a retail 
 client base differently from regular institutional investors 
 and then the worlds' largest pools of money involved something 
 else again.

 D isn't really marketed and it's definitely not sold.  That's 
 an implicit strategy in itself.


But one doesn't decide to have no strategy (at least if they any 
common sense!), one simply has no strategy. Unfortunately I think 
D falls into the latter, certainly not more than "Build it and 
they will come", irrespective of it effectiveness.


 Actually no less than 3 programmer friends came to (I'm the 
 weirdo-using-D and people are _always_ in disbelief and invent 
 all sorts of reasons not to try) saying they saw an article on 
 D on HN, with "D compilation is slow", and on further 
 examination they didn't read or at best the first paragraph. 
 But they did remember the title. They may rationally think 
 their opinion of D hasn't changed: aren't we highly capable 
 people?




I hope so!


 It doesn't matter what most people think.  It matters what 
 people who are on the fence or using D already a bit think.  Or 
 people who have a lot of problems to which D is in part a 
 solution only they didn't know about or think of D yet.


Then we should try to subtly (for some value of subtlety) make 
ourselves noticed.



 Nov 28 2018



prev sibling  parent reply Andrej Mitrovic <andrej.mitrovich gmail.com>  writes:


On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
wrote:

 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully read 
 the article to see otherwise, and few will do that.


Well comparative to itself sometimes it is. When you initially 
write D code you get used to the blazing fast speeds, but when 
eventually the compilation speed slows down as a project grows 
then this has a real effect on productivity.

Maybe a better title would have been "D compilation sometimes 
slows down too much", but it wouldn't get as many hits. On the 
upside, people who read the article - or even just read the 
comments section, will quickly realize that D's compilation speed 
is still miles faster than the competition. They might actually 
try the language. :)



 Nov 22 2018



  parent reply Paolo Invernizzi <paolo.invernizzi gmail.com>  writes:


On Thursday, 22 November 2018 at 10:51:45 UTC, Andrej Mitrovic 
wrote:

<mega snip>

BTW, it's nice to see again the Secret Squirrel on the forum, in 
these days: welcome back Andrej!

/Paolo



 Nov 22 2018



  parent reply Andrej Mitrovic <andrej.mitrovich gmail.com>  writes:


On Thursday, 22 November 2018 at 11:16:26 UTC, Paolo Invernizzi 
wrote:

 BTW, it's nice to see again the Secret Squirrel on the forum, 
 in these days: welcome back Andrej!

 /Paolo


Oh hey there too! I'm sorry if I can't recall you, though. 
¯\_(ツ)_/¯

I mostly lurk around here these days. But I still use D heavily, 
at work.



 Nov 22 2018



  parent  Paolo Invernizzi <paolo.invernizzi gmail.com>  writes:


On Thursday, 22 November 2018 at 13:19:58 UTC, Andrej Mitrovic 
wrote:

 On Thursday, 22 November 2018 at 11:16:26 UTC, Paolo Invernizzi 
 wrote:

 BTW, it's nice to see again the Secret Squirrel on the forum, 
 in these days: welcome back Andrej!

 /Paolo


 Oh hey there too! I'm sorry if I can't recall you, though. 
 ¯\_(ツ)_/¯


Oh, no problem... eheh


 I mostly lurk around here these days.


Yep, the same...


 But I still use D heavily, at work.


Well, the same here; not so heavily right now, my CTO is not sure 
anymore about the "case for D", but well, we have just delivered 
a D (medical) codebase to one of our customer...

Let's see...

/Paolo



 Nov 22 2018



prev sibling  parent  Vladimir Panteleev <thecybershadow.lists gmail.com>  writes:


On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson 
wrote:




Also on reddit:

https://www.reddit.com/r/programming/comments/9z36xg/d_compilation_is_too_slow_and_i_am_forking_the/



 Nov 21 2018



prev sibling  parent reply Atila Neves <atila.neves gmail.com>  writes:


On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:

 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/


Very interesting.

I'm also currently working on a project to save my bloodstream 
from the cortisol drip that happens when anything a computer does 
takes over a second, which these days means waiting for dmd to 
compile my code so I can see the result of the tests. I'll share 
more details when I have time.

But: one of the things I want to do is a version of this / 
precompiled headers. I've complained before that compiling 
std.path with -unittest takes forever (0.5s or so, and most of it 
due to std.uni). That's a price I pay every time I make a one 
line change to any file, and the linker hasn't even been invoked 
yet. Here's the thing: Phobos only changes from one release to 
the next. Why am I waiting to recompile a read-only file that 
won't change unless I update the compiler, over and over again?

I'd love it if I could precompile Phobos and just use the 
digested information every time I'm iterating.



 Nov 21 2018



  parent reply Vladimir Panteleev <thecybershadow.lists gmail.com>  writes:


On Wednesday, 21 November 2018 at 11:35:02 UTC, Atila Neves wrote:

 I'm also currently working on a project to save my bloodstream 
 from the cortisol drip that happens when anything a computer 
 does takes over a second, which these days means waiting for 
 dmd to compile my code so I can see the result of the tests. 
 I'll share more details when I have time.


Looking forward to it!


 But: one of the things I want to do is a version of this / 
 precompiled headers. I've complained before that compiling 
 std.path with -unittest takes forever (0.5s or so, and most of 
 it due to std.uni). That's a price I pay every time I make a 
 one line change to any file, and the linker hasn't even been 
 invoked yet. Here's the thing: Phobos only changes from one 
 release to the next. Why am I waiting to recompile a read-only 
 file that won't change unless I update the compiler, over and 
 over again?


That particular problem is in large part due to that the 
-unittest switch is not namespaced. I ran into the same issue 
with -allinst - with std.path, it breaks only if -unittest is 
also specified. I don't want to compile std.path unit tests, just 
my own!

Have we tried disabling -unittest for modules that aren't on the 
compiler's command line yet (or, in case of -i, not excluded)?


 I'd love it if I could precompile Phobos and just use the 
 digested information every time I'm iterating.


I agree, it would be nice if we could ship some "precompiled 
module" files along with Phobos .lib / .so files, but it looks 
like implementing this feature correctly might be non-trivial.

Maybe this hack could be developed further into a more generic 
"compiler server" idea.



 Nov 21 2018



 next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com>  writes:


On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir 
Panteleev wrote:

 Have we tried disabling -unittest for modules that aren't on 
 the compiler's command line yet (or, in case of -i, not 
 excluded)?


Not that I know of, thats a great idea!


 Maybe this hack could be developed further into a more generic 
 "compiler server" idea.


Wasn't that Robert's Masters thesis (Dconf 2013(?) presentation)? 
;)



 Nov 21 2018



 next sibling parent  Stefan Koch <uplink.coder googlemail.com>  writes:


On Wednesday, 21 November 2018 at 13:05:27 UTC, Nicholas Wilson 
wrote:

 On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir 
 Panteleev wrote:

 Have we tried disabling -unittest for modules that aren't on 
 the compiler's command line yet (or, in case of -i, not 
 excluded)?


 Not that I know of, thats a great idea!


 Maybe this hack could be developed further into a more generic 
 "compiler server" idea.


 Wasn't that Robert's Masters thesis (Dconf 2013(?) 
 presentation)? ;)


The main problem with this, is the amount of context a compilers 
needs.
And the current design of DMD does not lend itself splitting out 
the context.
If you wanted you could consider the semantic pass of the 
compiler as a database, which answers queries such as:

  - which size does this type have.
  - which arguments does this function have
  - can the type A be casted to type B
  - which conversion function should be invoked for (B)A ?
  - is this function known to be pure?

The data-base containing this information needs to be maintained 
on the compile-nodes, and that possibly leads to many 
data-dependencies.
Which may degrade the performance of the "compiler server" to the 
point where it is quicker to do it locally.

I am currently working (albeit very slowly due to lack of time 
and focus) to enable programmers to circumvent slow parts in 
compiler. When completed this should make a compiler-server 
unnecessary for some time to come.



 Nov 21 2018



prev sibling  parent  Seb <seb wilzba.ch>  writes:


On Wednesday, 21 November 2018 at 13:05:27 UTC, Nicholas Wilson 
wrote:

 On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir 
 Panteleev wrote:

 Have we tried disabling -unittest for modules that aren't on 
 the compiler's command line yet (or, in case of -i, not 
 excluded)?


 Not that I know of, thats a great idea!



Yes it's sadly a well-known problem e.g. 
https://github.com/dlang/dmd/pull/8124



 Nov 21 2018



prev sibling  parent  Atila Neves <atila.neves gmail.com>  writes:


On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir 
Panteleev wrote:

 On Wednesday, 21 November 2018 at 11:35:02 UTC, Atila Neves 
 wrote:

 [...]


 Looking forward to it!


 [...]


 That particular problem is in large part due to that the 
 -unittest switch is not namespaced. I ran into the same issue 
 with -allinst - with std.path, it breaks only if -unittest is 
 also specified. I don't want to compile std.path unit tests, 
 just my own!

 Have we tried disabling -unittest for modules that aren't on 
 the compiler's command line yet (or, in case of -i, not 
 excluded)?


I'm not sure what the real issue is here or what the solution 
would be. There was a PR to fix it that was closed without 
merging.



 Nov 21 2018