www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Recompiling D code

reply Rikki Cattermole <alphaglosined gmail.com> writes:
Hello everyone.
As you all may know I've been working on recompiling D for web services 
last few weeks.
Its both good news and bad news.

Good:
Reloading definitely possible. With dependency handling using dub.

Bad:
Its slow. And not in my code sort of way.

You're welcome to atest my speeds that I have gotten so far (older well 
used hdd, Win7 x64), its around 2-3 seconds for a basic go download 
webpage and not much more.

I've done a bit of working with it and with 2.066-rc2 got it to the 
lower end of 2s.

Atleast from my experience with this, its dmd thats actually taking the 
time.

I know my code isn't the best and optimized, so I have compared mine 
against dmd itself. 100-200ms difference normally.

For those who don't know why I'm making a fuss over 2 seconds, its 
because you actually do notice this when doing web dev stuff. Without 
including start up time, this could become a lot larger.

[0] https://github.com/rikkimax/livereload
Aug 14 2014
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 14 August 2014 at 11:54:57 UTC, Rikki Cattermole 
wrote:

 Atleast from my experience with this, its dmd thats actually 
 taking the time.


I can't glean this from looking at the code, but are you 
recompiling the entire program? Web development is a perfect fit 
for incremental compilation, so when changing one web page's 
template or view code, you should need to compile only one .d 
file.

ld and gold seem to support incremental linking, that could help 
speed things up as well.
Aug 14 2014
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/08/2014 12:47 a.m., Vladimir Panteleev wrote:

 On Thursday, 14 August 2014 at 11:54:57 UTC, Rikki Cattermole wrote:

 Atleast from my experience with this, its dmd thats actually taking
 the time.


 I can't glean this from looking at the code, but are you recompiling the
 entire program? Web development is a perfect fit for incremental
 compilation, so when changing one web page's template or view code, you
 should need to compile only one .d file.

 ld and gold seem to support incremental linking, that could help speed
 things up as well.


Dub automatically handles caching of dependencies such as vibe-d. So 
they are not rebuilt.

The only things that get recompiled for example is a single code unit. 
This is defined as being the dependency between a route file and 
template files (simplified).

The necessary dependencies such as vibe-d then should be added as part 
of the build process.

Incremental builds of things such as routes and templates are not a good 
idea. Same goes for data models. It could change the code unit output a 
little too much. Possibly even cause a corrupt binary from missing 
symbols ext.

But in this case none of this matters much. Pretty much I'm only testing 
against a single route module. No dependencies outside phobos and vibe-d.
Aug 14 2014
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 14 August 2014 at 12:56:10 UTC, Rikki Cattermole 
wrote:

 On 15/08/2014 12:47 a.m., Vladimir Panteleev wrote:

 On Thursday, 14 August 2014 at 11:54:57 UTC, Rikki Cattermole 
 wrote:

 Atleast from my experience with this, its dmd thats actually 
 taking
 the time.


 I can't glean this from looking at the code, but are you 
 recompiling the
 entire program? Web development is a perfect fit for 
 incremental
 compilation, so when changing one web page's template or view 
 code, you
 should need to compile only one .d file.

 ld and gold seem to support incremental linking, that could 
 help speed
 things up as well.


 Dub automatically handles caching of dependencies such as 
 vibe-d. So they are not rebuilt.

 The only things that get recompiled for example is a single 
 code unit. This is defined as being the dependency between a 
 route file and template files (simplified).

 The necessary dependencies such as vibe-d then should be added 
 as part of the build process.

 Incremental builds of things such as routes and templates are 
 not a good idea. Same goes for data models. It could change the 
 code unit output a little too much. Possibly even cause a 
 corrupt binary from missing symbols ext.

 But in this case none of this matters much. Pretty much I'm 
 only testing against a single route module. No dependencies 
 outside phobos and vibe-d.


My experience with these sort of things suggests that it'll be 
the linker taking the time. Dynamic libraries are the solution.

Dub needs proper support for dynamic library dependencies.
Aug 14 2014
parent reply Justin Whear <justin economicmodeling.com> writes:
On Thu, 14 Aug 2014 14:58:20 +0000, John Colvin wrote:


 My experience with these sort of things suggests that it'll be the
 linker taking the time. Dynamic libraries are the solution.
 
 Dub needs proper support for dynamic library dependencies.


I'll second that suggestion.  Can you run the timings using -c?  
Obviously you won't get binaries out the end, but it'd help diagnose the 
problem.
Aug 14 2014
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/08/2014 3:09 a.m., Justin Whear wrote:

 On Thu, 14 Aug 2014 14:58:20 +0000, John Colvin wrote:


 My experience with these sort of things suggests that it'll be the
 linker taking the time. Dynamic libraries are the solution.

 Dub needs proper support for dynamic library dependencies.


 I'll second that suggestion.  Can you run the timings using -c?
 Obviously you won't get binaries out the end, but it'd help diagnose the
 problem.


Yes its definitely a lot faster. Around 600ms - 800ms. Even with other 
applications running.
Aug 14 2014
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 14 August 2014 at 12:56:10 UTC, Rikki Cattermole 
wrote:

 Dub automatically handles caching of dependencies such as 
 vibe-d. So they are not rebuilt.

 The only things that get recompiled for example is a single 
 code unit. This is defined as being the dependency between a 
 route file and template files (simplified).


How many D modules / object files is that?


 The necessary dependencies such as vibe-d then should be added 
 as part of the build process.

 Incremental builds of things such as routes and templates are 
 not a good idea. Same goes for data models. It could change the 
 code unit output a little too much. Possibly even cause a 
 corrupt binary from missing symbols ext.


That approach sounds much more high-level than what I meant, and 
I don't see why you'd need to abstract things to that level. I'm 
not familiar with how dub builds things, but inter-module 
dependencies are a solved problem, thanks to DMD's -v and -deps 
switches. Incremental compilation was implemented in older 
(D1-era) build tools, such as xfbuild, however has never picked 
up due to DMD bugs with incrementally compiling more than one 
object file at a time (which might very well be fixed by now).
Aug 14 2014
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/08/2014 2:58 a.m., Vladimir Panteleev wrote:

 On Thursday, 14 August 2014 at 12:56:10 UTC, Rikki Cattermole wrote:

 Dub automatically handles caching of dependencies such as vibe-d. So
 they are not rebuilt.

 The only things that get recompiled for example is a single code unit.
 This is defined as being the dependency between a route file and
 template files (simplified).


 How many D modules / object files is that?


I haven't gone into that, I don't really want to go around modifying dub 
if I can help it.


 The necessary dependencies such as vibe-d then should be added as part
 of the build process.

 Incremental builds of things such as routes and templates are not a
 good idea. Same goes for data models. It could change the code unit
 output a little too much. Possibly even cause a corrupt binary from
 missing symbols ext.


 That approach sounds much more high-level than what I meant, and I don't
 see why you'd need to abstract things to that level. I'm not familiar
 with how dub builds things, but inter-module dependencies are a solved
 problem, thanks to DMD's -v and -deps switches. Incremental compilation
 was implemented in older (D1-era) build tools, such as xfbuild, however
 has never picked up due to DMD bugs with incrementally compiling more
 than one object file at a time (which might very well be fixed by now).
Aug 14 2014
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 15 August 2014 at 06:54:28 UTC, Rikki Cattermole wrote:

 On 15/08/2014 2:58 a.m., Vladimir Panteleev wrote:

 How many D modules / object files is that?


 I haven't gone into that, I don't really want to go around 
 modifying dub if I can help it.


Well, to be blunt, I doubt you'll get satisfactory results unless 
you're ready to get down to the nitty-gritty.
Aug 15 2014
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 16/08/2014 11:29 a.m., Vladimir Panteleev wrote:

 On Friday, 15 August 2014 at 06:54:28 UTC, Rikki Cattermole wrote:

 On 15/08/2014 2:58 a.m., Vladimir Panteleev wrote:

 How many D modules / object files is that?


 I haven't gone into that, I don't really want to go around modifying
 dub if I can help it.


 Well, to be blunt, I doubt you'll get satisfactory results unless you're
 ready to get down to the nitty-gritty.


Unfortunately I agree with you. But from what I've heard this isn't bad 
results considering native language and all.

I may work towards using something like parrotVM or lua. Could be 
interesting for dev vs production. Don't see why I can't get that 
working rather well.
Aug 15 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-14 13:54, Rikki Cattermole wrote:

 Hello everyone.
 As you all may know I've been working on recompiling D for web services
 last few weeks.
 Its both good news and bad news.

 Good:
 Reloading definitely possible. With dependency handling using dub.

 Bad:
 Its slow. And not in my code sort of way.


Has anyone thought of trying to JIT using LDC.

-- 
/Jacob Carlborg
Aug 14 2014
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/08/2014 6:06 a.m., Jacob Carlborg wrote:

 On 2014-08-14 13:54, Rikki Cattermole wrote:

 Hello everyone.
 As you all may know I've been working on recompiling D for web services
 last few weeks.
 Its both good news and bad news.

 Good:
 Reloading definitely possible. With dependency handling using dub.

 Bad:
 Its slow. And not in my code sort of way.


 Has anyone thought of trying to JIT using LDC.


Supposedly the llvm is slower for generation then dmd's backend. Atleast 
what was said during the SDC DCONF talk.

But I'm all for this, could also use it to cache code that is generated 
ext. in memory. Less usage on the hdd can only be a good thing.

I'll leave this to somebody who does know this area of things.
Aug 14 2014