Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

Phobos' posix.mak offers the ability to only run unittests for one module:

make std/range/primitives.test BUILD=debug -j8

... or package:

make std/range.test BUILD=debug -j8

It runs module tests in parallel and everything. This is definitely 
awesome. But say I misspell things by using a dot instead of the slash:

make std.range.test BUILD=debug -j8

Instead of an error, I get a no-op result that looks like success. How 
can that situation be converted to an error?


Thanks,

Andrei

ketmar <ketmar ketmar.no-ip.org> writes:

Andrei Alexandrescu wrote:

 Phobos' posix.mak offers the ability to only run unittests for one module:

 make std/range/primitives.test BUILD=debug -j8

 ... or package:

 make std/range.test BUILD=debug -j8

 It runs module tests in parallel and everything. This is definitely 
 awesome. But say I misspell things by using a dot instead of the slash:

 make std.range.test BUILD=debug -j8

 Instead of an error, I get a no-op result that looks like success. How 
 can that situation be converted to an error?

maybe by creating phony targets with such names?

ketmar <ketmar ketmar.no-ip.org> writes:

Andrei Alexandrescu wrote:

 Phobos' posix.mak offers the ability to only run unittests for one module:

 make std/range/primitives.test BUILD=debug -j8

 ... or package:

 make std/range.test BUILD=debug -j8

 It runs module tests in parallel and everything. This is definitely 
 awesome. But say I misspell things by using a dot instead of the slash:

 make std.range.test BUILD=debug -j8

 Instead of an error, I get a no-op result that looks like success. How 
 can that situation be converted to an error?

p.s.: or replacing make-based build system with D-based. as we need working 
D compiler to compile dmd anyway, i see no reason to not use it more.

=?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:

On 06/11/2017 12:27 PM, ketmar wrote:

 p.s.: or replacing make-based build system with D-based. as we need
 working D compiler to compile dmd anyway, i see no reason to not use it
 more.

I had the pleasure of working with Eyal Lotem, main author of buildsome. 
The buildsome team are aware of all pitfalls of all build systems and 
offer build*some* as an awe*some* ;) and correct build system:

   http://buildsome.github.io/buildsome/

Ali

ketmar <ketmar ketmar.no-ip.org> writes:

Ali Çehreli wrote:

 On 06/11/2017 12:27 PM, ketmar wrote:

 p.s.: or replacing make-based build system with D-based. as we need
 working D compiler to compile dmd anyway, i see no reason to not use it
 more.

 I had the pleasure of working with Eyal Lotem, main author of buildsome. 
 The buildsome team are aware of all pitfalls of all build systems and 
 offer build*some* as an awe*some* ;) and correct build system:

    http://buildsome.github.io/buildsome/

haskell? no, thanks. sorry.

Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:

On Sunday, June 11, 2017 16:47:30 Ali Çehreli via Digitalmars-d wrote:
 On 06/11/2017 12:27 PM, ketmar wrote:
  > p.s.: or replacing make-based build system with D-based. as we need
  > working D compiler to compile dmd anyway, i see no reason to not use it
  > more.

 I had the pleasure of working with Eyal Lotem, main author of buildsome.
 The buildsome team are aware of all pitfalls of all build systems and
 offer build*some* as an awe*some* ;) and correct build system:

    http://buildsome.github.io/buildsome/

Atila did some work to get dmd, druntime, phobos, etc. building with
reggae/dub, and I _think_ that he had it all working. As I understand it,
the main barrier to switching to it officially was political. A number of us
would _love_ to see the makefiles killed off, and there's really no
technical barrier to doing so. It's really a question of convincing folks
like Walter, Andrei, and Martin, and I get the impression that to an extent,
there's an attitude of not wanting to mess with what's working (though I
dispute that it works all that well from a maintenance perspective).

It's certainly a pain to edit the makefiles though, and I think that we'd be
far better off in the long run if we switched to something like reggae - and
since reggae is written in D and uses dub, we'd be dogfooding our own stuff
in the process.

- Jonathan M Davis

ketmar <ketmar ketmar.no-ip.org> writes:

Jonathan M Davis wrote:

 It's certainly a pain to edit the makefiles though

and don't forget those Great Copying Lists to copy modules. forgetting to 
include module in one of the lists was happened before, not once or twice...

Sebastien Alaiwan <ace17 free.fr> writes:

On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:

 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules. 
 forgetting to include module in one of the lists was happened 
 before, not once or twice...

I don't get it, could you please show an example?

ketmar <ketmar ketmar.no-ip.org> writes:

Sebastien Alaiwan wrote:

 On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:

 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules. forgetting 
 to include module in one of the lists was happened before, not once or 
 twice...

 I don't get it, could you please show an example?

ah, lol, those lists were killed some time ago, and i failed to notice it. 
silly me. sorry. but they *were* a PITA back then, and memory remains. ;-)

ketmar <ketmar ketmar.no-ip.org> writes:

Sebastien Alaiwan wrote:

 On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:

 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules. forgetting 
 to include module in one of the lists was happened before, not once or 
 twice...

 I don't get it, could you please show an example?

ah, nope, not killed. it's just me being blind. look at druntime/mak 
directory, for example. those lists are *required*, and they are maintained 
MANUALLY.

Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:

On Monday, June 12, 2017 06:34:31 Sebastien Alaiwan via Digitalmars-d wrote:
 On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:
 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules.
 forgetting to include module in one of the lists was happened
 before, not once or twice...

 I don't get it, could you please show an example?

posix.mak is a lot better than it used to be, but with win{32,64}.mak, you
have to list the modules all over the place. So, adding or removing a module
becomes a royal pain, and it's very easy to screw up. Ideally, we'd just
list the modules once in one file that was then used across all of the
platforms rather than having to edit several files every time we add or
remove anything. And the fact that we're using make for all of this makes
that difficult if not impossible (especially with the very limited make that
we're using on Windows). IMHO, switching to something like reggae would be
an enormous improvement.

It's true that we don't have to constantly edit the makefiles, so it's not a
constant pain point, but it does come up every time we add or remove any
modules, and the pain in dealing with the makefiles and the time wasted with
them adds up over time.

- Jonathan M Davis

Jacob Carlborg <doob me.com> writes:

On 2017-06-12 09:00, Jonathan M Davis via Digitalmars-d wrote:

 It's true that we don't have to constantly edit the makefiles, so it's not a
 constant pain point, but it does come up every time we add or remove any
 modules, and the pain in dealing with the makefiles and the time wasted with
 them adds up over time.

For some reason, in most of the PRs I've made to DMD I had to modify the 
makefiles, just recently [1]. And not just the makefiles, now there's a 
Visual Studio project that needs to kept up to date as well.

[1] https://github.com/dlang/dmd/pull/6837

-- 
/Jacob Carlborg

Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:

On Mon, 2017-06-12 at 21:27 +0200, Jacob Carlborg via Digitalmars-d
wrote:
=20

[=E2=80=A6]
 the=C2=A0
 makefiles, just recently [1]. And not just the makefiles, now there's
 a=C2=A0
 Visual Studio project that needs to kept up to date as well.
=20
 [1] https://github.com/dlang/dmd/pull/6837


There should only ever be one specification of the project and how to
build it. IDE project files should be generated from that
specification, never hand built.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

Sebastien Alaiwan <ace17 free.fr> writes:

On Monday, 12 June 2017 at 07:00:46 UTC, Jonathan M Davis wrote:
 On Monday, June 12, 2017 06:34:31 Sebastien Alaiwan via 
 Digitalmars-d wrote:
 On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:
 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules. 
 forgetting to include module in one of the lists was 
 happened before, not once or twice...

 I don't get it, could you please show an example?

 posix.mak is a lot better than it used to be, but with 
 win{32,64}.mak, you have to list the modules all over the 
 place. So, adding or removing a module becomes a royal pain, 
 and it's very easy to screw up. Ideally, we'd just list the 
 modules once in one file that was then used across all of the 
 platforms rather than having to edit several files every time 
 we add or remove anything. And the fact that we're using make 
 for all of this makes that difficult if not impossible 
 (especially with the very limited make that we're using on 
 Windows).

Are you implying that we are currently keeping compatibility with 
NMAKE (the 'make' from MS)?

GNU make inclusion mechanism makes it possible and easy to share 
a list of modules between makefiles.

Before switching to a fancy BS, we might benefit from learning to 
fully take advantage of the one we currently have!

Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:

On Tuesday, June 13, 2017 05:38:37 Sebastien Alaiwan via Digitalmars-d 
wrote:
 On Monday, 12 June 2017 at 07:00:46 UTC, Jonathan M Davis wrote:
 On Monday, June 12, 2017 06:34:31 Sebastien Alaiwan via

 Digitalmars-d wrote:
 On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:
 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules.
 forgetting to include module in one of the lists was
 happened before, not once or twice...

 I don't get it, could you please show an example?

 posix.mak is a lot better than it used to be, but with
 win{32,64}.mak, you have to list the modules all over the
 place. So, adding or removing a module becomes a royal pain,
 and it's very easy to screw up. Ideally, we'd just list the
 modules once in one file that was then used across all of the
 platforms rather than having to edit several files every time
 we add or remove anything. And the fact that we're using make
 for all of this makes that difficult if not impossible
 (especially with the very limited make that we're using on
 Windows).

 Are you implying that we are currently keeping compatibility with
 NMAKE (the 'make' from MS)?

 GNU make inclusion mechanism makes it possible and easy to share
 a list of modules between makefiles.

 Before switching to a fancy BS, we might benefit from learning to
 fully take advantage of the one we currently have!

As I understand it, the make that we use on Windows is one that comes with
dmc and is rather primitive in comparison to other versions of make, which
is why win32.mak and win64.mak suck so much in comparison to posix.mak.

Regardless, IMHO, make is a terrible build solution regardless of which
version of make you're using. It's better than using shell scripts, but in
this day and age, there are a plethora of solutions which are vastly
superior to make in terms of usability.

- Jonathan M Davis

Meta <jared771 gmail.com> writes:

On Monday, 12 June 2017 at 06:34:31 UTC, Sebastien Alaiwan wrote:
 On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:

 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules. 
 forgetting to include module in one of the lists was happened 
 before, not once or twice...

 I don't get it, could you please show an example?

https://github.com/dlang/phobos/pull/3843

Sebastien Alaiwan <ace17 free.fr> writes:

On Monday, 12 June 2017 at 15:57:13 UTC, Meta wrote:
 On Monday, 12 June 2017 at 06:34:31 UTC, Sebastien Alaiwan 
 wrote:
 On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:

 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules. 
 forgetting to include module in one of the lists was happened 
 before, not once or twice...

 I don't get it, could you please show an example?

 https://github.com/dlang/phobos/pull/3843

Thanks!

Please, tell me that the Digital Mars implementation of Make 
*does* support generic rules...

Meta <jared771 gmail.com> writes:

On Tuesday, 13 June 2017 at 06:05:16 UTC, Sebastien Alaiwan wrote:
 On Monday, 12 June 2017 at 15:57:13 UTC, Meta wrote:
 On Monday, 12 June 2017 at 06:34:31 UTC, Sebastien Alaiwan 
 wrote:
 On Monday, 12 June 2017 at 06:30:16 UTC, ketmar wrote:
 Jonathan M Davis wrote:

 It's certainly a pain to edit the makefiles though

 and don't forget those Great Copying Lists to copy modules. 
 forgetting to include module in one of the lists was 
 happened before, not once or twice...

 I don't get it, could you please show an example?

 https://github.com/dlang/phobos/pull/3843

 Thanks!

 Please, tell me that the Digital Mars implementation of Make 
 *does* support generic rules...

I have no idea. I'm not very well versed in Make to begin with, 
much less Digital Mars Make.

Atila Neves <atila.neves gmail.com> writes:

On Monday, 12 June 2017 at 00:19:18 UTC, Jonathan M Davis wrote:
 On Sunday, June 11, 2017 16:47:30 Ali Çehreli via Digitalmars-d 
 wrote:
    [...]

 Atila did some work to get dmd, druntime, phobos, etc. building 
 with reggae/dub, and I _think_ that he had it all working. As I 
 understand it, the main barrier to switching to it officially 
 was political. A number of us would _love_ to see the makefiles 
 killed off, and there's really no technical barrier to doing 
 so. It's really a question of convincing folks like Walter, 
 Andrei, and Martin, and I get the impression that to an extent, 
 there's an attitude of not wanting to mess with what's working 
 (though I dispute that it works all that well from a 
 maintenance perspective).

It was only phobos, but other than that, correct.

Atila

Sebastien Alaiwan <ace17 free.fr> writes:

On Sunday, 11 June 2017 at 23:47:30 UTC, Ali Çehreli wrote:
 I had the pleasure of working with Eyal Lotem, main author of 
 buildsome. The buildsome team are aware of all pitfalls of all 
 build systems and offer build*some* as an awe*some* ;) and 
 correct build system:

   http://buildsome.github.io/buildsome/

Very interesting!

The selling points, to me, are:
1) the automatic dependency detection through filesystem hooks
2) recipes also are dependencies
3) the genericity/low-level. I believe build systems should let 
me define my own abstractions, instead of trying to define for me 
what an "executable" or a "static library" should be.

- Make has 3)
- Ninja has 2), 3)
- tup and buildsome have 1), 2), 3)

However, buildsome also seems to have a (simplified) make-like 
syntax.
Why did they have to write it in Haskell, for god's sake!

ketmar <ketmar ketmar.no-ip.org> writes:

Sebastien Alaiwan wrote:

 The selling points, to me, are:
 1) the automatic dependency detection through filesystem hooks
 2) recipes also are dependencies
 3) the genericity/low-level. I believe build systems should let me define 
 my own abstractions, instead of trying to define for me what an 
 "executable" or a "static library" should be.

i'm not that all excited by "1", though. tbh, i prefer simplier things, 
like regexp scanning. while regexp scanners may find more dependencies than 
there really are, they doesn't require any dirty tricks to work.

i have my own fork of Perforce Jam, and such scanners works surprisingly 
well with it (for years). i even extended it to support D.

one of the great things in Jam is that it can simply call a "rule" (this is 
jam's name for something like function) when it need to get dependencies. 
and i can do regexp scan there, or run arbitrary system command and get 
it's result. also, while my jam comes with a set of predefined rules, there 
is nothing hardcoded, and everything can be redefined. it is also highly 
extensible via special "hook rules", which are empty by default.

the simpliest Jamrule can be as simple as this:

	Main myapp : [ Glob src/*.c ] ;

it also correctly processes subdirs (not via recursive invocations), can 
replace "configure" to some extend, and so on. i was using it exclusively 
for years before i completely switched to D (and even after the switch, to 
build projects with GDC).

Sebastien Alaiwan <ace17 free.fr> writes:

On Monday, 12 June 2017 at 06:38:34 UTC, ketmar wrote:
 Sebastien Alaiwan wrote:

 The selling points, to me, are:
 1) the automatic dependency detection through filesystem hooks
 2) recipes also are dependencies
 3) the genericity/low-level. I believe build systems should 
 let me define my own abstractions, instead of trying to define 
 for me what an "executable" or a "static library" should be.

 i'm not that all excited by "1", though. tbh, i prefer simplier 
 things, like regexp scanning. while regexp scanners may find 
 more dependencies than there really are, they doesn't require 
 any dirty tricks to work.

I understand your point ; I was explaining to my colleagues 
yesterday that "1" was a "good step in the wrong direction".

I think dependencies should come from above, they should be 
forced at the build system level. No more 'include' or 'imports' 
(Bazel took one step into this direction).
The consequence is that now you can't consider the build system 
of your project as a second class citizen. It's the only place 
where you're forced to express something  vaguely resembling to a 
high-level architecture.

Instead of letting module implementations happily create 
dependencies to any other module implementation (which is an 
architectural sin!) and then resorting to system-level hacks to 
try to re-create the DAG of this mess (and good luck with 
generated files).

However: "1" is still a "good" step. Compared to where we are 
now, it's in theory equivalent to perfectly doing regexp/gcc-MM 
scanning, in a langage agnostic way. It's a net win!

ketmar <ketmar ketmar.no-ip.org> writes:

Sebastien Alaiwan wrote:

 On Monday, 12 June 2017 at 06:38:34 UTC, ketmar wrote:
 Sebastien Alaiwan wrote:

 The selling points, to me, are:
 1) the automatic dependency detection through filesystem hooks
 2) recipes also are dependencies
 3) the genericity/low-level. I believe build systems should let me 
 define my own abstractions, instead of trying to define for me what an 
 "executable" or a "static library" should be.

 i'm not that all excited by "1", though. tbh, i prefer simplier things, 
 like regexp scanning. while regexp scanners may find more dependencies 
 than there really are, they doesn't require any dirty tricks to work.

 I understand your point ; I was explaining to my colleagues yesterday 
 that "1" was a "good step in the wrong direction".

..
 However: "1" is still a "good" step. Compared to where we are now, it's 
 in theory equivalent to perfectly doing regexp/gcc-MM scanning, in a 
 langage agnostic way. It's a net win!

it is still a kludge. and it has a huge chance to stay with us for a very 
long time -- this is what usually happens with kludges. ;-)

Ralph Amissah via Digitalmars-d <digitalmars-d puremagic.com> writes:

On Sun, Jun 11 2017, Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

And where do existing D build systems fit into the picture, I seem to
recall this general discussion around the announcement of "button", which
seems interesting, potentially awesome even:

Button announcement
https://forum.dlang.org/post/uhozcvatvyztfuhivrjq forum.dlang.org

Features
https://code.dlang.org/packages/button
- Implicit dependency detection.
- Correct incremental builds.
- Can display a graph of the build.
- Recursive. Can generate a build description as part of the build.
- Very general. Does not make any assumptions about the structure of your
project.
- Detects and displays cyclic dependencies.
- Detects race conditions.

example build description provided (the build description is done in
lua)
https://github.com/jasonwhite/dmd/blob/button/src/BUILD.lua

http://jasonwhite.github.io/button/

I have not used it, but if it works as described on the box most should
be pleased to get to know and to use and promote, no?... I assume it
will work well with dub if necessary? Other D based build systems have
been mentioned, who is able to review them?

 On Sunday, June 11, 2017 16:47:30 Ali Çehreli via Digitalmars-d wrote:
 On 06/11/2017 12:27 PM, ketmar wrote:
  > p.s.: or replacing make-based build system with D-based. as we need
  > working D compiler to compile dmd anyway, i see no reason to not use it
  > more.

 I had the pleasure of working with Eyal Lotem, main author of buildsome.
 The buildsome team are aware of all pitfalls of all build systems and
 offer build*some* as an awe*some* ;) and correct build system:

    http://buildsome.github.io/buildsome/

 Atila did some work to get dmd, druntime, phobos, etc. building with
 reggae/dub, and I _think_ that he had it all working. As I understand it,
 the main barrier to switching to it officially was political. A number of us
 would _love_ to see the makefiles killed off, and there's really no
 technical barrier to doing so. It's really a question of convincing folks
 like Walter, Andrei, and Martin, and I get the impression that to an extent,
 there's an attitude of not wanting to mess with what's working (though I
 dispute that it works all that well from a maintenance perspective).

 It's certainly a pain to edit the makefiles though, and I think that we'd be
 far better off in the long run if we switched to something like reggae - and
 since reggae is written in D and uses dub, we'd be dogfooding our own stuff
 in the process.

 - Jonathan M Davis

--
ralph.amissah gmail.com

ketmar <ketmar ketmar.no-ip.org> writes:

Ralph Amissah wrote:

 On Sun, Jun 11 2017, Jonathan M Davis via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 And where do existing D build systems fit into the picture, I seem to
 recall this general discussion around the announcement of "button", which
 seems interesting, potentially awesome even:

 Button announcement
 https://forum.dlang.org/post/uhozcvatvyztfuhivrjq forum.dlang.org

"tracing system calls". a no-no for any general solution. no sane person 
with even rudimentary care for privaly will give *build* *system* 
privileges for injecting .so libraries or trace system calls.

actually, requiring such privileges is my main objection agains any "modern 
automagic build system".

Guillaume Boucher <guillaume.boucher.d gmail.com> writes:

On Sunday, 11 June 2017 at 19:17:36 UTC, Andrei Alexandrescu 
wrote:
 Instead of an error, I get a no-op result that looks like 
 success. How can that situation be converted to an error?

That makefile target is poorly written.

It was probably intended to have a dependency on the directory 
itself, so by adding "%/" to the pattern it correctly produces an 
error (since std.algorithm.d doesn't exist):

%.test : $(LIB) %/
         ...

To allow for your case, just add this line:

std.%.test : std/%.test


While that should work (at least for the first level), it still 
is very poorly written.  It doesn't use the white-listed 
modules/packages despite the rest of the makefile seems to use 
that fairly consistently.

I would replace all the ".test" targets with the following code:


phobos library).


concurrent unittesting.
$(addsuffix .test,$(D_MODULES)): %.test : %.d $(LIB)
         T=`mktemp -d /tmp/.dmd-run-test.XXXXXX` &&                
                                               \
           (                                                       
                                               \
             $(DMD) -od$$T $(DFLAGS) -main -unittest $(LIB) 
-defaultlib= -debuglib= $(LINKDL) -cov -run $< ;     \
             RET=$$? ; rm -rf $$T ; exit $$RET                     
                                               \
           )


a package,

define PACKAGETEST_template
$(1).test: $$(patsubst %,$(1)/%.test,$$(PACKAGE_$(subst 
/,_,$(1))))
endef
$(foreach package,$(STD_PACKAGES),$(eval $(call 
PACKAGETEST_template,$(package))))


using dot as a separator.

define MODULESYNTAXTEST_template
$(subst /,.,$(1)).test : $(1).test
endef
$(foreach module,$(STD_PACKAGES) $(D_MODULES),$(eval $(call 
MODULESYNTAXTEST_template,$(module))))


(I'm not going to make a pull request though.)

Jacob Carlborg <doob me.com> writes:

On 2017-06-11 21:17, Andrei Alexandrescu wrote:
 Phobos' posix.mak offers the ability to only run unittests for one module:

 make std/range/primitives.test BUILD=debug -j8

 ... or package:

 make std/range.test BUILD=debug -j8

 It runs module tests in parallel and everything. This is definitely
 awesome. But say I misspell things by using a dot instead of the slash:

 make std.range.test BUILD=debug -j8

 Instead of an error, I get a no-op result that looks like success. How
 can that situation be converted to an error?

Wouldn't it be more natural if the following syntax was used:

make test std/range/primitives.d

That is, "make test" then the path to a file or directory.

But I rather see the make build system completely replaced with 
something else.

-- 
/Jacob Carlborg

Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:

On Mon, 2017-06-12 at 13:18 +0200, Jacob Carlborg via Digitalmars-d
wrote:
 [=E2=80=A6]
=20
 But I rather see the make build system completely replaced with=20
 something else.

Make was a revelation and revolution in 1977.

Make is the assembly language of build.

There are much better abstractions of build these days.

CMake/Make
CMake/Ninja
Meson/Ninja
SCons

to name some obvious high profile examples. And then there are:

Gradle
Bazel
Cargo
Go
Dub

to name some alternate models of build.

That Chapel and D chose Make as their build system is rather annoying,
so backward looking, for supposedly forward looking systems. Oh well.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

"H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:

On Mon, Jun 12, 2017 at 06:09:52PM +0100, Russel Winder via Digitalmars-d wrote:
 On Mon, 2017-06-12 at 13:18 +0200, Jacob Carlborg via Digitalmars-d
 wrote:
 […]
 
 But I rather see the make build system completely replaced with
 something else.


+1.


 Make was a revelation and revolution in 1977.
 
 Make is the assembly language of build.
 
 There are much better abstractions of build these days.
 
 CMake/Make
 CMake/Ninja
 Meson/Ninja
 SCons

Don't forget tup, and others inspired by it, which use modern OS
features to reduce the cost of determining what to build to an O(1)
database lookup rather than an O(n) whole-source tree scan. I.e., much
faster code-compile-test cycle.  Much as I have come to love SCons, I
think ultimately the build system of the future will be something
derived from the tup model.


[...]
 That Chapel and D chose Make as their build system is rather annoying,
 so backward looking, for supposedly forward looking systems. Oh well.

[...]

I don't agree with the notion that everything must be "forward looking"
in order to be "good".  If something existing works well and does what
we need it to do, there's no reason to keep reinventing the wheel.

However, I do agree that where there is room for improvement,
improvements ought to be made. On this front, make leaves a lot to be
desired.  It requires far too much babysitting, manual work, and
dependence on human infallibility (which we all know works all too
well). Not to mention that it fails the basic tenet of a reliable build
system: reproducible builds, i.e., given *any* arbitrary workspace
state, running the build command ought to produce products (executables,
etc.) that are *guaranteed* to reflect the current state of the source
code.

Just working with a dmd PR these past few days brought me right back to
the bad ole days of always running `make clean` before `make`, just to
be sure that the executables accurately reflect the source. I had shed
such redundant steps long ago when I adopted SCons as my go-to build
system, but it didn't take long working with the dmd sources to discover
that sometimes make doesn't produce the correct executable, but picks up
detritus from the environment (stale object files from previous builds,
temporary files that the bash script snippets expect to be absent or
don't expect to be still lingering, etc.).

(It's not as bad in druntime/Phobos where all the makefile does is
basically to pass all the source files to a single invocation of dmd.
More or less. But if you have to work with documentation or running the
test suite, the flaws of make surface all over again pretty soon.
Updating the interminable lists of files in posix.mak, for example, is
quite reminiscient of working with a C source tree that's heavily
dependent on leaky macros. Quite the nostalgia trip, I have to say.)

One can always argue that make is time-tested and everybody knows it,
therefore it can't be all that bad.  Well, programming in C is also
time-tested, a large number of veteran programmers know it, so it can't
be all that bad, can it?  I mean, all you have to do is to make sure
your arrays are never overrun, avoid poorly-designed standard library
features that lead to security holes, make sure you manage your own
memory well to avoid leaks and double-frees, make sure your pointers are
always valid, make sure you don't write leaky macros, memorize the
operator precedence table, learn the intricacies of 100 different
gotchas that can come back and bite you in the behind, and ... There's
really no need to program in another language, like D, right? :-P

Another common objection is, there's no clear "winner" among the make
alternatives to have an obvious choice to pick, so we better just stick
with make for now. Well, arguably, from a certain perspective, there's
no obvious "winner" among successors to C/C++ either... there's D, Go,
Rust, Java, etc.. So let's just stick with C, the lowest common
denominator, until a clear winner emerges.


T

-- 
Жил-был король когда-то, при нём блоха жила.

ketmar <ketmar ketmar.no-ip.org> writes:

H. S. Teoh wrote:

 Don't forget tup, and others inspired by it, which use modern OS
 features to reduce the cost of determining what to build to an O(1)
 database lookup rather than an O(n) whole-source tree scan.

added complexity for nothing. disks gradually replacing with ssd, amount of 
RAM allows to cache alot, and CPUs are faster and faster (and has more cores).

i still have HDD, 8GB of RAM, and 32-bit system. and it even worse some 
time ago. in the last ~8 years i was using my k8jam for various projects: 
several kb to multimegabytes of code and thousand files/alot of subdirs. 
source tree scanning and dependency resolving NEVER was any significant 
factor.

and i really mean it: it was less than a second even for a huge projects, 
where linking alone took long enough that i could get coffee and cigarette. ;-)

worse is better. i hate to admit it, but most tricks we're learned over the 
years becomes more and more useless. it doesn't matter if the program will 
do it's work in 0.001 msecs or in 0.0001 msecs: this is not even a 
difference between 10 minutes and 5 minutes anymore.

"H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:

On Mon, Jun 12, 2017 at 09:53:42PM +0300, ketmar via Digitalmars-d wrote:
 H. S. Teoh wrote:
 
 Don't forget tup, and others inspired by it, which use modern OS
 features to reduce the cost of determining what to build to an O(1)
 database lookup rather than an O(n) whole-source tree scan.

 
 added complexity for nothing. disks gradually replacing with ssd,
 amount of RAM allows to cache alot, and CPUs are faster and faster
 (and has more cores).

CPU speed has nothing to do with this, disk roundtrip is always going to
be the bottleneck, even with SSD.


 i still have HDD, 8GB of RAM, and 32-bit system. and it even worse
 some time ago. in the last ~8 years i was using my k8jam for various
 projects: several kb to multimegabytes of code and thousand files/alot
 of subdirs. source tree scanning and dependency resolving NEVER was
 any significant factor.

The problem is not building the entire source tree -- that's not the use
case we're targeting.  We're talking about incremental builds, which are
not relevant to make because in make they are inherently unreliable.

Several KB is kids' play.  I work with a source tree with at least 800MB
of source code (not counting auxiliary scripts and other resources that
must be compiled as part of the build process), producing a 130MB
archive of *compressed* executables, libraries, and other resources.
The only way I can even maintain my sanity is to `cd
deeply/nested/subdir; make` to compile the specific target I'm working
on. Scanning dependencies take a *long* time, and can mean the
difference between being able to test a 1 line change in 1 second vs. 5
*minutes* while the thing scans 800MB worth of source code just to
recompile 2 source files.

It's either that, or scan the timestamps instead of the file contents,
like make does, which will be fast, but you then have the lovely
side-effect of unreliable builds and non-existent bugs that appear in
the executable but has no representation in the source code.  *Then* you
have to ditch incremental buids altogether and do wholesale 'make clean;
make', so it's time to take a nap for 20 mins while it recompiles the
entire source tree just to account for a 1-line bug fix.


 and i really mean it: it was less than a second even for a huge
 projects, where linking alone took long enough that i could get coffee
 and cigarette.  ;-)

[...]

Your project is not huge enough. :-D


T

-- 
Do not reason with the unreasonable; you lose by definition.

ketmar <ketmar ketmar.no-ip.org> writes:

H. S. Teoh wrote:

 and i really mean it: it was less than a second even for a huge
 projects, where linking alone took long enough that i could get coffee
 and cigarette.  ;-)

 [...]

 Your project is not huge enough. :-D

~20MB, >2000 source files. for *this* it was something like 0.5-3 seconds 
(it obviously oscillates). and of course, i'm not talking about full 
rebuilds. this is *all* the time k8jam spent before invoking compiler/linker.

and for pathological 800MB use cases... don't do that. you obviously don't 
need to have thing thing as a one huge project (althru i'm sure that k8jam 
can do it).

k8jam can use timestamps and md5 sums to detect changes (althru i'm usually 
using only timestamps, and had zero problems with it ever), and it can 
optionally cache gathered info.

note that even for small "helloworld" C project, k8jam also checks *all* 
standard libc include files that which brought into the project even by 
simple `#inlcude <stdlib.h>`! and i never bothered to optimize this, 'cause 
it takes no time anyway.

"H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:

On Mon, Jun 12, 2017 at 10:41:13PM +0300, ketmar via Digitalmars-d wrote:
[...]
 ~20MB, >2000 source files.

[...]

Toy project. :-D ;-)  The project I work on has more than 78,000 files,
of which more than 50,000 are source code.


 and for pathological 800MB use cases... don't do that. you obviously
 don't need to have thing thing as a one huge project (althru i'm sure
 that k8jam can do it).

[...]

Unfortunately, I don't get to make this decision; I'm only an employee.

Besides, putting everything in an 800MB source tree is actually
necessary because this is software for an embedded system -- we have to
basically build the entire OS along with all its utilities and other
application software that will run on it.


T

-- 
ASCII stupid question, getty stupid ANSI.

Paolo Invernizzi <paolo.invernizzi gmail.com> writes:

On Monday, 12 June 2017 at 20:04:22 UTC, H. S. Teoh wrote:
 On Mon, Jun 12, 2017 at 10:41:13PM +0300, ketmar via we have to 
 basically build the entire OS along with all its utilities and 
 other application software that will run on it.


... and tup can do it [1]... ;-P

/P

[1] http://gittup.org/gittup/

ketmar <ketmar ketmar.no-ip.org> writes:

H. S. Teoh wrote:

i'm pretty sure that i *don't* want to know more. ;-)

ketmar <ketmar ketmar.no-ip.org> writes:

p.s.: btw, k8jam is not a "make wrapper", it is a self-contained build 
system, in one ~100 kb binary. ;-)

ketmar <ketmar ketmar.no-ip.org> writes:

p.s.: like, i've seen one Fallout 2 recreatoin project where they're doing 
translation of screen coords to hex coords not with two simple formulas, 
but by looping over 40000 (200x200) hex objects (objects!), and calling 
*virtual* method to ask if a hex contains such coords. and you know what? 
*nobody* noticed any slowdown. 'cause there is none.

it took me ~5 mins to visualise scewed coordinate axes and derive the 
formulas. it took 'em ~30 seconds to write a loop (i guess). and their work 
is actually *better* than mine: they had 4.30 mins free, while i was doing 
useless things.

Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:

On Mon, 2017-06-12 at 10:39 -0700, H. S. Teoh via Digitalmars-d wrote:
=20

[=E2=80=A6]
 Don't forget tup, and others inspired by it, which use modern OS
 features to reduce the cost of determining what to build to an O(1)
 database lookup rather than an O(n) whole-source tree scan. I.e.,
 much
 faster code-compile-test cycle.=C2=A0=C2=A0Much as I have come to love SC=

ons, I
 think ultimately the build system of the future will be something
 derived from the tup model.

Ninja.

However good Tup may be, I think Ninja has the mindshare as the build
engine underneath CMake, Meson, etc.

I am ambivalent about SCons. It is a huge improvement over Make, great
strides are being made in performance along with the Python 2 =E2=86=92 Pyt=
hon
3 thing (removing all the Python < 2.7 stuff and replacing with modern
Python code), but it's model is from the 2000s and as you say the 2010s
have new things which are not getting in to SCons, mostly because it is
a pure volunteer project with no companies putting any resources in. I
am doing work on it for D builds mostly because I can't really cope
with Dub as a build tool.

 [...]
=20
 I don't agree with the notion that everything must be "forward
 looking"
 in order to be "good".=C2=A0=C2=A0If something existing works well and do=

es
 what
 we need it to do, there's no reason to keep reinventing the wheel.

Forward looking is always good, this does not exclude leaving things as
is, or bringing things that got forgotten from the past, it is about
thinking about what is right next year, not what we have last year.

 However, I do agree that where there is room for improvement,
 improvements ought to be made. On this front, make leaves a lot to be
 desired.=C2=A0=C2=A0It requires far too much babysitting, manual work, an=

d
 dependence on human infallibility (which we all know works all too
 well). Not to mention that it fails the basic tenet of a reliable
 build
 system: reproducible builds, i.e., given *any* arbitrary workspace
 state, running the build command ought to produce products
 (executables,
 etc.) that are *guaranteed* to reflect the current state of the
 source
 code.
=20

[=E2=80=A6]

It all comes down to two things:

1. Is anyone willing to create the new build.
2. Will the new build be accepted and used.

Given some people's tone here, and the lack of contribution from Walter
and Andrei on replacing Make, the answer to 2 is NO. Thus the answer to
1 is "don't bother". Thus this thread is useless. Thus D will continue
to use an outdated and increasingly difficult to use build system. Not
a way of attracting new contributors. Its not the outdated so much as
the difficult to use.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

Jonathan Marler <johnnymarler gmail.com> writes:

On Sunday, 11 June 2017 at 19:17:36 UTC, Andrei Alexandrescu 
wrote:
 Phobos' posix.mak offers the ability to only run unittests for 
 one module:

 make std/range/primitives.test BUILD=debug -j8

 ... or package:

 make std/range.test BUILD=debug -j8

 It runs module tests in parallel and everything. This is 
 definitely awesome. But say I misspell things by using a dot 
 instead of the slash:

 make std.range.test BUILD=debug -j8

 Instead of an error, I get a no-op result that looks like 
 success. How can that situation be converted to an error?


 Thanks,

 Andrei

I've shared this same frustration.  I once took a stab at 
creating a "DMake" system, where you could use standard MAKE 
syntax, or D code side by side.  I never finished it but you can 
find the concept here.  If there is interest I'd be willing to 
finish it.

https://github.com/marler8997/dmake