www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D, SCons, Dub

reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:


I have submitted a pull request that adds ProgramAllAtOnce as a builder
in the dmd, ldc, and gdc tools of SCons. This does an 'all at once'
compilation in a single compiler instantiation, unlike the standard
module at a time compilation and then link the program. There are lots
of arguments about whether "all at once" is at all useful, I have added
it simply because it is the only way Unit-Threaded works.

If you want to use it prior to it being accepted and released, then
feel free to use the tools in the repository https://github.com/russel/
SCons_D_Experiment

Also in there you will spot a new tool dub. I am now using this to grab
unit-threaded from the Dub repository in a SCons build. See for example
 https://github.com/russel/ApproxGC

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 17 2017
next sibling parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:



 I have submitted a pull request that adds ProgramAllAtOnce as a 
 builder in the dmd, ldc, and gdc tools of SCons. This does an 
 'all at once' compilation in a single compiler instantiation, 
 unlike the standard module at a time compilation and then link 
 the program. There are lots of arguments about whether "all at 
 once" is at all useful, I have added it simply because it is 
 the only way Unit-Threaded works.


Nice. I'm using SCons in my everyday development. I'll give your 
feature a try.

Meson has recently added a similar feature. Meson calls it a 
"unity build".


 If you want to use it prior to it being accepted and released, 
 then feel free to use the tools in the repository 
 https://github.com/russel/ SCons_D_Experiment

 Also in there you will spot a new tool dub. I am now using this 
 to grab
 unit-threaded from the Dub repository in a SCons build. See for 
 example
  https://github.com/russel/ApproxGC


Cool.

Thanks.
Apr 17 2017
parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2017-04-17 at 18:53 +0000, Nordl=C3=B6w via Digitalmars-d wrote:

=20


[=E2=80=A6]

 Nice. I'm using SCons in my everyday development. I'll give your=C2=A0
 feature a try.


My pull request is currently being reviewed and some updates are
needed. These will appear immediately in the SCons_D_Experiment. I'll
post once the pull request gets merged into SCons mainline. Then it
just awaits a release, unless you use default/tip from the mainline
repository.


 Meson has recently added a similar feature. Meson calls it a=C2=A0
 "unity build".


Is there documentation? I need this immediately as unit-threaded only
works with whole source compilation.


 If you want to use it prior to it being accepted and released,=C2=A0
 then feel free to use the tools in the repository=C2=A0
 https://github.com/russel/ SCons_D_Experiment
=20
 Also in there you will spot a new tool dub. I am now using this=C2=A0
 to grab
 unit-threaded from the Dub repository in a SCons build. See for=C2=A0
 example
 =C2=A0https://github.com/russel/ApproxGC


=20
 Cool.
=20


:-)

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 17 2017
prev sibling next sibling parent reply Alex <alexibu.nospam me.com> writes:
On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:



 I have submitted a pull request that adds ProgramAllAtOnce


I have been trying to do the opposite with scons - incremental 
builds. In c++ the .h and .cpp files allowed the compilation from 
many edits to be limited to one translation unit.

One thing people are often confused about is whether everything 
needs to be in the header (or di file).

An influence is that most of the code they see written by skilled 
language users is libraries like the standard library. These 
typically benefit little from interfaces files because everything 
is templates and must be in the header to compile. Hence why 
boost is mostly headers in c++ land and the D ecosystem 
encourages all at once / all everytime compilation.

In a large code base this becomes cumbersome. Changing a unit 
test should not result in everything recompiling.

I think there is a difference between library style code and 
application code. My application code doesn't have templates and 
can make use of di files. It makes sense that the language 
developers spend most of their time on library type code, and 
have been focused on it.

I just need to work out how to get scons to recognise two targets 
from one builder properly : the object file and the di file.
I can add DFLAGs to make a Object builder make the di files too, 
and the d scanner finds di files where d files are not available, 
but the build tree dependency from d to di file is missing. Dies 
anyone know how to do this ?
Apr 17 2017
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 17 April 2017 at 20:56:55 UTC, Alex wrote:

 On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:



 I have submitted a pull request that adds ProgramAllAtOnce


 I have been trying to do the opposite with scons - incremental 
 builds. In c++ the .h and .cpp files allowed the compilation 
 from many edits to be limited to one translation unit.


In D, this is nearly always slower (I measured). Compiling per 
package, however, is a lot faster.

Atila
Apr 17 2017
parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2017-04-17 at 22:59 +0000, Atila Neves via Digitalmars-d wrote:

 [=E2=80=A6]
 In D, this is nearly always slower (I measured). Compiling per=C2=A0
 package, however, is a lot faster.


I have only ever needed file-by-file or whole source compilation, I
have never needed package-oriented compilation. It sounds like this
needs adding to SCons and Meson. What is needed is a really small
project showing the idea so that it can be used as a test case.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 17 2017
prev sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2017-04-17 at 20:56 +0000, Alex via Digitalmars-d wrote:

 On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:


=20
 I have submitted a pull request that adds ProgramAllAtOnce


=20
 I have been trying to do the opposite with scons - incremental=C2=A0
 builds. In c++ the .h and .cpp files allowed the compilation from=C2=A0
 many edits to be limited to one translation unit.


Per module compilation then linking is the default for the Program
builder in SCons, that is already there and has been for over a decade.


 One thing people are often confused about is whether everything=C2=A0
 needs to be in the header (or di file).


Uuurrr=E2=80=A6 there is no separation of header and code in D, just module=
s.
And packages. SCons currently has no "per package" builder. I guess I
need to add this.


 An influence is that most of the code they see written by skilled=C2=A0
 language users is libraries like the standard library. These=C2=A0
 typically benefit little from interfaces files because everything=C2=A0
 is templates and must be in the header to compile. Hence why=C2=A0
 boost is mostly headers in c++ land and the D ecosystem=C2=A0
 encourages all at once / all everytime compilation.
=20
 In a large code base this becomes cumbersome. Changing a unit=C2=A0
 test should not result in everything recompiling.
=20
 I think there is a difference between library style code and=C2=A0
 application code. My application code doesn't have templates and=C2=A0
 can make use of di files. It makes sense that the language=C2=A0
 developers spend most of their time on library type code, and=C2=A0
 have been focused on it.
=20
 I just need to work out how to get scons to recognise two targets=C2=A0
 from one builder properly : the object file and the di file.
 I can add DFLAGs to make a Object builder make the di files too,=C2=A0
 and the d scanner finds di files where d files are not available,=C2=A0
 but the build tree dependency from d to di file is missing. Dies=C2=A0
 anyone know how to do this ?


Currently there is no explicit handling of di files in SCons. If you
can come up with a small project that exhibits the problem you need
solved, I can add it as a test case in the SCons system thereby
creating a need for a fix.

(Tests in SCons for the tools are end-to-end tests, so need a project
exhibiting a problem.)
=20
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 17 2017
parent reply Alex <alexibu.nospam me.com> writes:
On Tuesday, 18 April 2017 at 06:51:51 UTC, Russel Winder wrote:

 I have been trying to do the opposite with scons - incremental 
 builds. In c++ the .h and .cpp files allowed the compilation 
 from many edits to be limited to one translation unit.


 Per module compilation then linking is the default for the 
 Program builder in SCons, that is already there and has been 
 for over a decade.


I know, I have been using your work in SCons to build in this 
mode since it was first available.


 One thing people are often confused about is whether 
 everything needs to be in the header (or di file).


 Uuurrr… there is no separation of header and code in D, just 
 modules. And packages. SCons currently has no "per package" 
 builder. I guess I need to add this.


I realise there are no headers, that is the problem. SCons can't 
tell whether a change to a module affects the interface, only 
affects the implementation or only affects a unit test. This 
means everything that imports a module is recompiled on every 
edit, even if only the unit test was modified, or a formatting 
change was made.

I am not suggesting the D way is bad compared to C++, I am 
suggesting we can have the best of both worlds by automatically 
generating 'headers' or .di files from modules. Then having 
dependent modules read only the .di file instead of the .d file.
This way SCons won't recompile dependencies unless necessary. 
This is the main purpose of using SCons (perfect incremental 
builds) which is largely defeated currently by D.



 Currently there is no explicit handling of di files in SCons.


I know, I have been working through the guts of SCons for (a 
little bit too long to mention)  trying to make it both output 
two products with proper dependencies (.o and .di) and to find 
the di files correctly. SCons seems to have an ability to find 
the original .d files in the project even if the compiler can't.


 If you can come up with a small project that exhibits the 
 problem you need solved, I can add it as a test case in the 
 SCons system thereby creating a need for a fix.


Thanks for that kind offer Russel, I will make a minimal example 
project.


 (Tests in SCons for the tools are end-to-end tests, so need a 
 project
 exhibiting a problem.)


OK.
Apr 18 2017
next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-18 at 07:34 +0000, Alex via Digitalmars-d wrote:

=20


[=E2=80=A6]

 I know, I have been using your work in SCons to build in this=C2=A0
 mode since it was first available.


Excellent. :-)

[=E2=80=A6]

=20




 I realise there are no headers, that is the problem. SCons can't=C2=A0
 tell whether a change to a module affects the interface, only=C2=A0
 affects the implementation or only affects a unit test. This=C2=A0
 means everything that imports a module is recompiled on every=C2=A0
 edit, even if only the unit test was modified, or a formatting=C2=A0
 change was made.


SCons should be immune to whitespace changes, but actual reformatting
is likely to be a change.

Hummm=E2=80=A6 it is not clear in my mind just now whether that dependency =
is
build into the SCons build DAG.=C2=A0Whilst the C and C++ builders go to
great lengths to deal with the #include dependency tree, I am not sure
the D tools parse the D source import dependency tree. It probably
should, but just now =E2=80=93 I believe =E2=80=93 it is just module by mod=
ule
compilation. If there are API changes then SCons will not follow that
through for D codes.=20
=20

 I am not suggesting the D way is bad compared to C++, I am=C2=A0
 suggesting we can have the best of both worlds by automatically=C2=A0
 generating 'headers' or .di files from modules. Then having=C2=A0
 dependent modules read only the .di file instead of the .d file.
 This way SCons won't recompile dependencies unless necessary.=C2=A0
 This is the main purpose of using SCons (perfect incremental=C2=A0
 builds) which is largely defeated currently by D.


I have never trodden the .di file path and so the SCons tools do not
generate them. A flag could easily be added so that an emitter of the
builder could generate the .di as well as the .o file. The tools are
not set up for that as yet, but it could be done.

Being sent small projects as test cases to drive adding features is
likely the best route. Feel free to send in issues on  https://github.c
om/russel/SCons_D_Experiment

[=E2=80=A6]

 I know, I have been working through the guts of SCons for (a=C2=A0
 little bit too long to mention)=C2=A0=C2=A0trying to make it both output=


=C2=A0

 two products with proper dependencies (.o and .di) and to find=C2=A0
 the di files correctly. SCons seems to have an ability to find=C2=A0
 the original .d files in the project even if the compiler can't.


It maybe that the missing link here is that the D builders currently do
not have emitters. With an emitter added doing the .di as well as .o
files should be very straightforward =E2=80=93 but maybe not easy.

[=E2=80=A6]
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 18 2017
prev sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Apr 18, 2017 at 07:34:34AM +0000, Alex via Digitalmars-d wrote:
[...]

 I realise there are no headers, that is the problem. SCons can't tell
 whether a change to a module affects the interface, only affects the
 implementation or only affects a unit test. This means everything that
 imports a module is recompiled on every edit, even if only the unit
 test was modified, or a formatting change was made.


This is not a problem if you use env.Decider('content'), in which case
the worst that will happen is that the module with the whitespace /
unittest change will be recompiled, but everything else that depends on
that module won't be touched because the object file will have the same
content as before.



 I am not suggesting the D way is bad compared to C++, I am suggesting
 we can have the best of both worlds by automatically generating
 'headers' or .di files from modules. Then having dependent modules
 read only the .di file instead of the .d file.


[...]

This is not workable for D code that uses a lot of templates. In my own
projects I never use .di files because it suffices to import the .d file
directly.  In the D world, generally the only time you'd use .di files
is (1) interfacing with proprietary libraries, or (2) interfacing with
C/C++ code (but even in this latter case, it's common practice to simply
declare C/C++ functions in extern(C) or extern(C++) blocks inside a .d
file, rather than have a .di file). It's pretty unnatural to use .di
files anywhere else in the typical D project.


T

-- 
"I'm running Windows '98." "Yes." "My computer isn't working now." "Yes, you
already said that." -- User-Friendly
Apr 18 2017
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:



 I have submitted a pull request that adds ProgramAllAtOnce as a 
 builder in the dmd, ldc, and gdc tools of SCons. This does an 
 'all at once' compilation in a single compiler instantiation, 
 unlike the standard module at a time compilation and then link 
 the program. There are lots of arguments about whether "all at 
 once" is at all useful, I have added it simply because it is 
 the only way Unit-Threaded works.


It's the only way __traits(getUnitTests) works. Unfortunately. 
I'm just going to fix that dmd bug myself.

Atila
Apr 17 2017
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/17/17 6:58 PM, Atila Neves wrote:

 On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:



 I have submitted a pull request that adds ProgramAllAtOnce as a 
 builder in the dmd, ldc, and gdc tools of SCons. This does an 'all at 
 once' compilation in a single compiler instantiation, unlike the 
 standard module at a time compilation and then link the program. There 
 are lots of arguments about whether "all at once" is at all useful, I 
 have added it simply because it is the only way Unit-Threaded works.


 
 It's the only way __traits(getUnitTests) works. Unfortunately. I'm just 
 going to fix that dmd bug myself.


Thanks!! -- Andrei
Apr 17 2017
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 18 April 2017 at 00:14:40 UTC, Andrei Alexandrescu 
wrote:

 On 4/17/17 6:58 PM, Atila Neves wrote:

 On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:



 I have submitted a pull request that adds ProgramAllAtOnce as 
 a builder in the dmd, ldc, and gdc tools of SCons. This does 
 an 'all at once' compilation in a single compiler 
 instantiation, unlike the standard module at a time 
 compilation and then link the program. There are lots of 
 arguments about whether "all at once" is at all useful, I 
 have added it simply because it is the only way Unit-Threaded 
 works.


 
 It's the only way __traits(getUnitTests) works. Unfortunately. 
 I'm just going to fix that dmd bug myself.


 Thanks!! -- Andrei


https://github.com/dlang/dmd/pull/6727

Getting it past the autotester was the hardest part of that 
odyssey!

Atila
Jun 06 2017
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 18 April 2017 at 00:14:40 UTC, Andrei Alexandrescu 
wrote:

 On 4/17/17 6:58 PM, Atila Neves wrote:

 On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:



 I have submitted a pull request that adds ProgramAllAtOnce as 
 a builder in the dmd, ldc, and gdc tools of SCons. This does 
 an 'all at once' compilation in a single compiler 
 instantiation, unlike the standard module at a time 
 compilation and then link the program. There are lots of 
 arguments about whether "all at once" is at all useful, I 
 have added it simply because it is the only way Unit-Threaded 
 works.


 
 It's the only way __traits(getUnitTests) works. Unfortunately. 
 I'm just going to fix that dmd bug myself.


 Thanks!! -- Andrei


So... I have, and it's languishing waiting for someone to merge 
it. Or just to give me a LGTM, I'd merge it myself then.

https://github.com/dlang/dmd/pull/6727

Atila
Aug 28 2017