www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Speed up `dub`.

reply ciechowoj <keepitsimplesirius gmail.com> writes:
I'm using `dub` to build project. And every time I run `dub` it 
seems to check if dependencies are up to date, which takes some 
time. Is there a way to switch of that checking? Or any other way 
to speed up building process? It really slows down my 
modify-compile-check iteration time.
Mar 07 2016
next sibling parent reply Daniel Kozak via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
maybe: dub build --nodeps

Dne 7.3.2016 v 10:18 ciechowoj via Digitalmars-d-learn napsal(a):

 I'm using `dub` to build project. And every time I run `dub` it seems 
 to check if dependencies are up to date, which takes some time. Is 
 there a way to switch of that checking? Or any other way to speed up 
 building process? It really slows down my modify-compile-check 
 iteration time.
Mar 07 2016
parent ciechowoj <keepitsimplesirius gmail.com> writes:
On Monday, 7 March 2016 at 09:22:16 UTC, Daniel Kozak wrote:

 maybe: dub build --nodeps

 Dne 7.3.2016 v 10:18 ciechowoj via Digitalmars-d-learn 
 napsal(a):

 I'm using `dub` to build project. And every time I run `dub` 
 it seems to check if dependencies are up to date, which takes 
 some time. Is there a way to switch of that checking? Or any 
 other way to speed up building process? It really slows down 
 my modify-compile-check iteration time.




I've tried that one already - it makes no difference.
Currently when I run `dub test` it takes about 9s (project is 
already built). When running the executable directly takes only 
0.2s :/.
Mar 07 2016
prev sibling next sibling parent Luis <luis.panadero gmail.com> writes:
On Monday, 7 March 2016 at 09:18:37 UTC, ciechowoj wrote:

 I'm using `dub` to build project. And every time I run `dub` it 
 seems to check if dependencies are up to date, which takes some 
 time. Is there a way to switch of that checking? Or any other 
 way to speed up building process? It really slows down my 
 modify-compile-check iteration time.


On my case, I don't see taking too long. Could you check what 
version of dub are you using, and what OS & D compiler and 
version ?
And better, if you can past here your dub.sdl/dub.json .
Mar 07 2016
prev sibling next sibling parent reply Martin Tschierschke <mt smartdolphin.de> writes:
On Monday, 7 March 2016 at 09:18:37 UTC, ciechowoj wrote:

 I'm using `dub` to build project. And every time I run `dub` it 
 seems to check if dependencies are up to date, which takes some 
 time. Is there a way to switch of that checking? Or any other 
 way to speed up building process? It really slows down my 
 modify-compile-check iteration time.


A few hours ago dub.pm alias code.dlang.org was down or slow, so 
I used:

$> dub --skip-registry=all build

might be thats the reason?
Mar 07 2016
parent reply ciechowoj <keepitsimplesirius gmail.com> writes:
dub --version
DUB version 0.9.24+161-gb9ce700, built on Feb 23 2016

`dub.json` is `dub.json` of dstep

`dub test --skip-registry=all`
Do not helps.
Mar 07 2016
next sibling parent Daniel Kozak via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
I would say dub is broken and should be fixed. Even dub run is really 
slow and try to build everything. So please report a bug: 
https://github.com/D-Programming-Language/dub/issues

Dne 7.3.2016 v 19:58 ciechowoj via Digitalmars-d-learn napsal(a):

 dub --version
 DUB version 0.9.24+161-gb9ce700, built on Feb 23 2016

 `dub.json` is `dub.json` of dstep

 `dub test --skip-registry=all`
 Do not helps.
Mar 07 2016
prev sibling next sibling parent reply WebFreak001 <janju007 web.de> writes:
On Monday, 7 March 2016 at 18:58:55 UTC, ciechowoj wrote:

 dub --version
 DUB version 0.9.24+161-gb9ce700, built on Feb 23 2016

 `dub.json` is `dub.json` of dstep

 `dub test --skip-registry=all`
 Do not helps.


Its because of dub going through all those JSON files. For 
example just querying the import paths would probably take like 
7s in your case. Try "dub describe --import-paths" in your 
project. Its something you can't do anything against except 
reporting it to the dub issues
Mar 07 2016
parent reply xcvn <xcvn xcvn.vn> writes:
On Monday, 7 March 2016 at 19:58:19 UTC, WebFreak001 wrote:

 On Monday, 7 March 2016 at 18:58:55 UTC, ciechowoj wrote:

 dub --version
 DUB version 0.9.24+161-gb9ce700, built on Feb 23 2016

 `dub.json` is `dub.json` of dstep

 `dub test --skip-registry=all`
 Do not helps.


 Its because of dub going through all those JSON files. For 
 example just querying the import paths would probably take like 
 7s in your case. Try "dub describe --import-paths" in your 
 project. Its something you can't do anything against except 
 reporting it to the dub issues


Yes but if he wants to build then `--describe` will not help that 
much !
Mar 07 2016
parent WebFreak001 <janju007 web.de> writes:
On Monday, 7 March 2016 at 21:49:59 UTC, xcvn wrote:

 Yes but if he wants to build then `--describe` will not help 
 that much !


dub uses that internally for finding the files for compiling, so 
thats the cause of the slowdown
Mar 08 2016
prev sibling parent reply Luis <luis.panadero gmail.com> writes:
On Monday, 7 March 2016 at 18:58:55 UTC, ciechowoj wrote:

 dub --version
 DUB version 0.9.24+161-gb9ce700, built on Feb 23 2016

 `dub.json` is `dub.json` of dstep

 `dub test --skip-registry=all`
 Do not helps.


I try to grab dstep with dub fetch step (dub version 0.9.24, 
built on Aug 19 2015, on Ubuntu), and try to run dub build step. 
I can confirm that on my machine takes around 5 seconds to check 
if the dependencies are update.
Mar 07 2016
parent reply rcorre <ryan rcorre.net> writes:
On Monday, 7 March 2016 at 22:47:13 UTC, Luis wrote:


 I try to grab dstep with dub fetch step (dub version 0.9.24, 
 built on Aug 19 2015, on Ubuntu), and try to run dub build 
 step. I can confirm that on my machine takes around 5 seconds 
 to check if the dependencies are update.


I had this same experience. Thought dub would be a convenient way 
to install dstep, but dub run took so long I ended up just 
creating an alias to run the binary directly.
Mar 08 2016
parent Luis <luis.panadero gmail.com> writes:
On Tuesday, 8 March 2016 at 15:43:28 UTC, rcorre wrote:

 On Monday, 7 March 2016 at 22:47:13 UTC, Luis wrote:


 I try to grab dstep with dub fetch step (dub version 0.9.24, 
 built on Aug 19 2015, on Ubuntu), and try to run dub build 
 step. I can confirm that on my machine takes around 5 seconds 
 to check if the dependencies are update.


 I had this same experience. Thought dub would be a convenient 
 way to install dstep, but dub run took so long I ended up just 
 creating an alias to run the binary directly.


dub not should have something to install executables without the 
need of using dub run to execute it ? Like pip install does a 
true install of the package.
Mar 08 2016
prev sibling next sibling parent reply Seb <seb wilzba.ch> writes:
On Monday, 7 March 2016 at 09:18:37 UTC, ciechowoj wrote:

 I'm using `dub` to build project. And every time I run `dub` it 
 seems to check if dependencies are up to date, which takes some 
 time. Is there a way to switch of that checking? Or any other 
 way to speed up building process? It really slows down my 
 modify-compile-check iteration time.


Use ld.gold - it will speed up your linking quite dramatically!

https://code.dawg.eu/reducing-vibed-turnaround-time-part-1-faster-linking.html
Mar 07 2016
parent reply ciechowoj <keepitsimplesirius gmail.com> writes:
On Monday, 7 March 2016 at 21:56:11 UTC, Seb wrote:

 Use ld.gold - it will speed up your linking quite dramatically!

 https://code.dawg.eu/reducing-vibed-turnaround-time-part-1-faster-linking.html


Thanks, it improves things a little. However I've just had idea 
that it should be possible to implement 'speculative dependency 
checking' for dub. What I mean by that is to start build process 
and dependency checking in parallel and then if something changes 
(with dependencies), apply the changes and restart the build 
process, and if dependencies are OK continue.
May 14 2016
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Sat, 2016-05-14 at 16:36 +0000, ciechowoj via Digitalmars-d-learn
wrote:

 On Monday, 7 March 2016 at 21:56:11 UTC, Seb wrote:

=20
 Use ld.gold - it will speed up your linking quite dramatically!
=20
 https://code.dawg.eu/reducing-vibed-turnaround-time-part-1-faster-l
 inking.html


 Thanks, it improves things a little. However I've just had idea=C2=A0
 that it should be possible to implement 'speculative dependency=C2=A0
 checking' for dub. What I mean by that is to start build process=C2=A0
 and dependency checking in parallel and then if something changes=C2=A0
 (with dependencies), apply the changes and restart the build=C2=A0
 process, and if dependencies are OK continue.


Anecdotal evidence from SCons, Gant, Gradle, Rant, and indeed Make,
seems to indicate that full dependency resolution in the ADG of the
build is the right thing to do for consistent and repeatable builds.=C2=A0

Various debates have been had in the SCons and Gradle communities over
the years about slow builds, and always, in the end, dependency
resolution up front has always won out.

Interestingly though "dependency resolution" actually has slightly
different meanings in different contexts. In the SCons context it is
about ensuring the ADG of the source =E2=86=92 object =E2=86=92 executable|=
library and
the presence of any referenced libraries. In the Ant/Maven/Gant/Gradle
context is is about downloading references libraries and ensuring the
ADG of build tasks is right.

In all cases building the ADG and constructing the resolution list is
not the big problem except in very, very large builds. For example folk
at Intel created Parts as a layer over SCons so as to parallelize and
create partial caches for the build, a bit along the lines of Vesta but
far better.

I cannot be certain, but I am not sure there are any D codebases quite
the same size as the Intel C++ ones.
=C2=A0
With SCons and Gradle there were various algorithmic and implementation
speedups to ameliorate things and that has helped a lot. I wonder if
there are opportunities to do the same for Dub. =C2=A0There are many
functionality things that need to be added to Dub to get it into the
Cargo and Gradle end of functionality mayhap performance can be
improved along the way.

I am not sure about DStep build, but for the two projects I have (using
GtkD) build times using Dub were sufficiently long that I switched to
SCons and it is OK and allows for a reasonable development rhythm.
=C2=A0
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
May 14 2016
parent ciechowoj <keepitsimplesirius gmail.com> writes:
On Sunday, 15 May 2016 at 06:49:00 UTC, Russel Winder wrote:

 I am not sure about DStep build, but for the two projects I 
 have (using
 GtkD) build times using Dub were sufficiently long that I 
 switched to
 SCons and it is OK and allows for a reasonable development 
 rhythm.


That SCons looks very nice but don't have time (and dub isn't 
that bad) to research it more/can't migrate to it. If I have more 
time I'll certainly try it out.
May 19 2016
prev sibling next sibling parent reply cy <dlang verge.info.tm> writes:
On Monday, 7 March 2016 at 09:18:37 UTC, ciechowoj wrote:

 I'm using `dub` to build project. And every time I run `dub` it 
 seems to check if dependencies are up to date, which takes some 
 time. Is there a way to switch of that checking? Or any other 
 way to speed up building process? It really slows down my 
 modify-compile-check iteration time.



dub build --nodeps

It's amazing.

dub has a few uh, issues with dependencies. It pulls all 
dependencies, even for disabled configurations, even for 
unselected optional dependencies, then just... leaves them pulled 
I guess, and doesn't link with them?
https://github.com/dlang/dub/issues/844

I don't know exactly what's going wrong. But after you run `dub 
build` the first time, everything will be downloaded and 
installed now, so you can recompile by specifying --nodeps and it 
just jumps straight past all those buggy issues.

ld.gold also will vastly speed up modify-compile-check cycles. 
dmd is fast at compiling, but linking it all up is really slow 
with vanilla ld.
May 14 2016
parent reply ciechowoj <keepitsimplesirius gmail.com> writes:
On Sunday, 15 May 2016 at 01:48:41 UTC, cy wrote:

 dub build --nodeps

 It's amazing.


I tried it, doesn't seem to do anything, maybe something is 
broken.
May 19 2016
parent reply cy <dlang verge.info.tm> writes:
On Thursday, 19 May 2016 at 17:50:44 UTC, ciechowoj wrote:

 dub build --nodeps


 I tried it, doesn't seem to do anything, maybe something is 
 broken.


Perhaps you didn't complete "dub build" without --nodeps 
beforehand? You have to do that once, and it's still as 
annoyingly inefficient as anything, especially since it checks 
dependencies online even if you have the selected version 
downloaded already. But after that first compile, I add --nodeps 
to the dub build command, and it's lightning fast to recompile.

It doesn't fix the issues in DUB that slow down compiling 
programs, but it does speed up the compile-modify-check cycle.
May 23 2016
parent reply ciechowoj <keepitsimplesirius gmail.com> writes:
Hahahaa. Who could possibly think that `build.sh` builds dub in 
debug mode? With -release -O -inline -m64 it runs 5 times faster 
: P. It made my day...
Jun 01 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-06-01 23:33, ciechowoj wrote:

 Hahahaa. Who could possibly think that `build.sh` builds dub in debug
 mode? With -release -O -inline -m64 it runs 5 times faster : P. It made
 my day...


Haha really?

-- 
/Jacob Carlborg
Jun 02 2016
parent reply ciechowoj <keepitsimplesirius gmail.com> writes:
On Thursday, 2 June 2016 at 12:20:50 UTC, Jacob Carlborg wrote:

 On 2016-06-01 23:33, ciechowoj wrote:

 Hahahaa. Who could possibly think that `build.sh` builds dub 
 in debug
 mode? With -release -O -inline -m64 it runs 5 times faster : 
 P. It made
 my day...


 Haha really?


Yes, I should have done it at the beginning, but yesterday I 
decided to see what is really happening and compiled dub with 
`-profile` and found that an assert from `std/path.d:3168` 
(`globMatch`) contributes a major amount to the running time of 
dub.

```
     assert(balancedParens(pattern, '[', ']', 0));
     assert(balancedParens(pattern, '{', '}', 0));
```
Jun 02 2016
next sibling parent reply ciechowoj <keepitsimplesirius gmail.com> writes:
Maybe it would be worth to write something about the issue here 
https://github.com/dlang/dub#installation . I'm curious how the 
pre-compiled versions are built.
Jun 02 2016
next sibling parent Daniel Kozak via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
I am not sure but on Arch linux I always recompile dub myself because by 
default it is build without release flags too.


Dne 2.6.2016 v 15:07 ciechowoj via Digitalmars-d-learn napsal(a):

 Maybe it would be worth to write something about the issue here 
 https://github.com/dlang/dub#installation . I'm curious how the 
 pre-compiled versions are built.
Jun 02 2016
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-06-02 15:07, ciechowoj wrote:

 Maybe it would be worth to write something about the issue here
 https://github.com/dlang/dub#installation . I'm curious how the
 pre-compiled versions are built.


Yeah, I really hope they're built with optimizations enabled.

-- 
/Jacob Carlborg
Jun 02 2016
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Thursday, 2 June 2016 at 13:04:00 UTC, ciechowoj wrote:

 and found that an assert from `std/path.d:3168` (`globMatch`) 
 contributes a major amount to the running time of dub.

 ```
     assert(balancedParens(pattern, '[', ']', 0));
     assert(balancedParens(pattern, '{', '}', 0));
 ```


Hmm.. that sounds like the dub binary that's distributed is 
running its unittests at start. If that's true I don't think they 
should be part of the released binary..
Jun 05 2016
parent ciechowoj <keepitsimplesirius gmail.com> writes:
On Sunday, 5 June 2016 at 21:20:20 UTC, Andrej Mitrovic wrote:

 On Thursday, 2 June 2016 at 13:04:00 UTC, ciechowoj wrote:

 and found that an assert from `std/path.d:3168` (`globMatch`) 
 contributes a major amount to the running time of dub.

 ```
     assert(balancedParens(pattern, '[', ']', 0));
     assert(balancedParens(pattern, '{', '}', 0));
 ```


 Hmm.. that sounds like the dub binary that's distributed is 
 running its unittests at start. If that's true I don't think 
 they should be part of the released binary..


Actually, that assertions are invoked from precondition block of 
`globMatch`. On the other hand `globMatch` is invoked from 
`std.file.dirEntry`. So, I suppose it is fine if it goes about 
unit tests.
Jun 06 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-03-07 10:18, ciechowoj wrote:

 I'm using `dub` to build project. And every time I run `dub` it seems to
 check if dependencies are up to date, which takes some time. Is there a
 way to switch of that checking? Or any other way to speed up building
 process? It really slows down my modify-compile-check iteration time.


I think that by default Dub should not check dependencies. There should 
be an explicit command to install the dependencies.

-- 
/Jacob Carlborg
May 24 2016
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Tue, 2016-05-24 at 09:00 +0200, Jacob Carlborg via Digitalmars-d-
learn wrote:

=20


[=E2=80=A6]

 I think that by default Dub should not check dependencies. There
 should=C2=A0
 be an explicit command to install the dependencies.



Dub must check all dependencies before starting a build, the question
is whether is should download and build them or just fail.

Gradle currently checks and downloads as required. This can be very
annoying in that there is an assumption of "always connected" and
"infinite data allowance". This can be very, very annoying when not
connected or connected with a very limited connection. On the other
hand, the build never fails due to ailed dependencies unless the
dependency doesn't actually exist in the search path. For JVM language
builds most people put both Bintray and Maven Central in the path and
more or less all sensible dependencies live in one of those place =E2=80=93=
 for
FOSS stuff of course, for proprietary you are on your own :-)

Gradle does now though build C++ stuff so there is already a dependency
system. I am likely to try creating a D plugin for Gradle and then copy
the Gradle way of doing things for SCons.

I wonder of Dub should focus on using the same repositories for
dependencies as Gradle does, not to avoif the Dub repository so much as
to hijack native code build of C++ and ensure those people have D on
offer.=C2=A0

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Jun 04 2016
parent Jacob Carlborg <doob me.com> writes:
On 04/06/16 10:04, Russel Winder via Digitalmars-d-learn wrote:


 Dub must check all dependencies before starting a build, the question
 is whether is should download and build them or just fail.


Yeah, I meant check online.

-- 
/Jacob Carlborg
Jun 05 2016