www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Common makefile (gasp) stuff

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Hey folks,


Over the time a number of stuff has become quite duplicated across our 
makefiles for dmd, druntime, and phobos.

These include fetching OS and model but (newer) general-purpose macros 
for e.g. rebasing repos, see 
https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

I think it's time to reduce clutter and duplication by migrating such 
common stuff into one common.mak file. Question is, where should that 
file sit? One candidate is tools/ but it's not really a tool. Another 
candidate is dmd/src/ because that would be the first thing anyone 
depends on.

Ideas?


Andrei
Jan 13 2015
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 13 Jan 2015 15:20:12 -0800
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 Hey folks,
=20
=20
 Over the time a number of stuff has become quite duplicated across our=20
 makefiles for dmd, druntime, and phobos.
=20
 These include fetching OS and model but (newer) general-purpose macros=20
 for e.g. rebasing repos, see=20
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak=


#L37.

=20
 I think it's time to reduce clutter and duplication by migrating such=20
 common stuff into one common.mak file. Question is, where should that=20
 file sit? One candidate is tools/ but it's not really a tool. Another=20
 candidate is dmd/src/ because that would be the first thing anyone=20
 depends on.
=20
 Ideas?


i thing that dmd is the right place. there is little sense in building
brand new phobos without brand new dmd with all fixes and enhancements
anyway, so i believe that most people who wants to rebuild phobos and
druntime are already has dmd repo cloned.

besides, this is a basic thing, which is of almost no use for ldc and
gdc anyway.

i vote for dmd.
Jan 13 2015
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Jan 13, 2015 at 03:20:12PM -0800, Andrei Alexandrescu via Digitalmars-d
wrote:

 Hey folks,
 
 
 Over the time a number of stuff has become quite duplicated across our
 makefiles for dmd, druntime, and phobos.
 
 These include fetching OS and model but (newer) general-purpose macros for
 e.g. rebasing repos, see
https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
 
 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.
 
 Ideas?


[...]

Actually, my first reaction is to go the opposite direction. Currently,
we have way too many complicated interdependencies between different
repos. For example, the 'html' target of phobos/posix.mak depends on
files in the dlang.org repos, and the website build scripts of dlang.org
in turn depends on files generated by the Phobos ddoc targets. This may
be fine for the core devs and hard-core bleeding-edge people like myself
who don't mind running off git HEAD and dealing with breakages, but it's
bad for anybody else:

1) These interdependencies are currently expressed by hard-coded
filesystem paths, which presumes a specific directory layout for
checking out dmd, dlang.org, phobos, druntime. This means the build will
break if somebody doesn't know the correct directory structure to use
when checking out these repos.

2) This makes the build fragile, because any changes to makefiles that
might affect these dependencies have to be merged in sync between
multiple repos, e.g., a PR that changes the layout of dlang.org needs to
have a corresponding PR that updates the Phobos makefiles as well. This
is kinda what we already have to do with compiler-dependent changes to
druntime/phobos anyway, but in the case of documentation, any oversight
is not caught by the autotester and may wind up on the public-facing
dlang.org website before the problem is noticed.

3) Due to these interdependencies, you always have to checkout all 4
repos (in a specific directory layout), otherwise the build will fail
one way or another. Again, not a problem for core devs, but a gratuitous
obstacle for the casual user who wants to try things out. (Actually, to
this day I'm almost certain I *still* don't have the exact layout
assumed by the build scripts, because dlang.org appears to deposit
generated html files in dlang.org/web, whereas phobos/posix.mak's 'html'
target generates them in ../web/, which in my setup is one directory
level off. I hacked around the problem with a symlink from web to
dlang.org/web, but I'm pretty sure many users who wanted to experiment
with Phobos docs gave up because they didn't have the patience to figure
out how to get the thing to work properly.)

Because of this, my inclination right now is to *reduce*
interdependencies between these repos, rather than introduce yet more.

Either that, or we should just forget about trying to manually keep 3-4
disparate repos in sync, and just use a single repo for dmd, druntime,
phobos, dlang.org, etc.. Except that some people will then complain that
they just want the bare compiler and nothing else, so why should they
have to clone excess baggage like druntime/phobos. *shrug* Some days you
just can't win.

(Of course, the *ideal* solution is that the makefiles will tell you if
something is missing and where to get it and how to set it up correctly.
But I'm skeptical whether this will ever materialize.  I'm not holding
my breath.)


T

-- 
Why waste time reinventing the wheel, when you could be reinventing the engine?
-- Damian Conway
Jan 13 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/13/15 3:55 PM, H. S. Teoh via Digitalmars-d wrote:

 On Tue, Jan 13, 2015 at 03:20:12PM -0800, Andrei Alexandrescu via
Digitalmars-d wrote:

 Hey folks,


 Over the time a number of stuff has become quite duplicated across our
 makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros for
 e.g. rebasing repos, see
https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.

 Ideas?


 [...]

 Actually, my first reaction is to go the opposite direction. Currently,
 we have way too many complicated interdependencies between different
 repos. For example, the 'html' target of phobos/posix.mak depends on
 files in the dlang.org repos, and the website build scripts of dlang.org
 in turn depends on files generated by the Phobos ddoc targets.


Yah but when you look closer it's not that bad; just an attribution 
tradeoff. Building the phobos docs belongs with the phobos makefile 
because that's where the module informations resides; however, part of 
the build requires macro definitions that are at best consolidated in 
the website directory.

Building phobos's docs without dlang.org around is not sensible, just 
the same as building dlang.org without phobos. So I'm not too worried 
about that.

If someone builds phobos without caring about the docs, that's fine as 
long as they don't try to build html.

One simplification would be to not set STDDOC in 
https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L95, 
and refuse to build html if STDDOC is not set from the outside. But 
again I see this as just minor progress.


 This may
 be fine for the core devs and hard-core bleeding-edge people like myself
 who don't mind running off git HEAD and dealing with breakages, but it's
 bad for anybody else:

 1) These interdependencies are currently expressed by hard-coded
 filesystem paths, which presumes a specific directory layout for
 checking out dmd, dlang.org, phobos, druntime. This means the build will
 break if somebody doesn't know the correct directory structure to use
 when checking out these repos.


Yah, that should be in a wiki. I keep on thinking about producing a 
10-minutes video "Your First Contribution to D" in which I explain step 
by step how to get a pull request in starting from a scratch.

We also have tools/update.sh which should probably be more prominently 
advertised.

But in the grand scheme of things I don't see a foolproof solution to 
that - you need dmd for building druntime, druntime for building phobos, 
etc. If you got dmd, druntime, and phobos in the wrong places then 
You're Gonna Have A Bad Time.


 2) This makes the build fragile, because any changes to makefiles that
 might affect these dependencies have to be merged in sync between
 multiple repos, e.g., a PR that changes the layout of dlang.org needs to
 have a corresponding PR that updates the Phobos makefiles as well. This
 is kinda what we already have to do with compiler-dependent changes to
 druntime/phobos anyway, but in the case of documentation, any oversight
 is not caught by the autotester and may wind up on the public-facing
 dlang.org website before the problem is noticed.


I don't understand how this is a problem, how my suggestion makes it any 
worse, what a solution to it all would be.


 3) Due to these interdependencies, you always have to checkout all 4
 repos (in a specific directory layout), otherwise the build will fail
 one way or another.


This is expected, no? What is an alternative? How would one e.g. build 
druntime without dmd?


 Again, not a problem for core devs, but a gratuitous
 obstacle for the casual user who wants to try things out. (Actually, to
 this day I'm almost certain I *still* don't have the exact layout
 assumed by the build scripts, because dlang.org appears to deposit
 generated html files in dlang.org/web, whereas phobos/posix.mak's 'html'
 target generates them in ../web/, which in my setup is one directory
 level off. I hacked around the problem with a symlink from web to
 dlang.org/web, but I'm pretty sure many users who wanted to experiment
 with Phobos docs gave up because they didn't have the patience to figure
 out how to get the thing to work properly.)


That's an inconsistency we should fix. Right now phobos is best build 
starting from the dlang.org build. We should probably disable phobos' 
own docs build.


 Because of this, my inclination right now is to *reduce*
 interdependencies between these repos, rather than introduce yet more.

 Either that, or we should just forget about trying to manually keep 3-4
 disparate repos in sync, and just use a single repo for dmd, druntime,
 phobos, dlang.org, etc.. Except that some people will then complain that
 they just want the bare compiler and nothing else, so why should they
 have to clone excess baggage like druntime/phobos. *shrug* Some days you
 just can't win.


Yah, that's why I think a set of simple tactical tools a la 
tools/update.sh are the proportional response here.


 (Of course, the *ideal* solution is that the makefiles will tell you if
 something is missing and where to get it and how to set it up correctly.
 But I'm skeptical whether this will ever materialize.  I'm not holding
 my breath.)


dlang.org does something of that kind already - "make rebase".


Andrei
Jan 13 2015
parent reply Paul O'Neil <redballoon36 gmail.com> writes:
On 01/13/2015 09:35 PM, Andrei Alexandrescu wrote:
<snip>

 1) These interdependencies are currently expressed by hard-coded
 filesystem paths, which presumes a specific directory layout for
 checking out dmd, dlang.org, phobos, druntime. This means the build will
 break if somebody doesn't know the correct directory structure to use
 when checking out these repos.


 
 Yah, that should be in a wiki. I keep on thinking about producing a
 10-minutes video "Your First Contribution to D" in which I explain step
 by step how to get a pull request in starting from a scratch.
 
 We also have tools/update.sh which should probably be more prominently
 advertised.
 


<snip>

This shouldn't go (only) in the wiki.  It should be in the README of
every repository.  And if not in the README, then the CONTRIBUTING file.
 A minimal list of "clone these things, run make in these directories"
is fine here with a reference to the wiki for more details.

I put off some documentation updates a few months ago because it wasn't
worth the investment yet to figure out how to build the website.  The
wiki pages appeared to be about building the compiler, which I didn't
realize I should be interested in.  This barrier is too high.

-- 
Paul O'Neil
Github / IRC: todayman
Jan 13 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/13/15 8:29 PM, Paul O'Neil wrote:

 On 01/13/2015 09:35 PM, Andrei Alexandrescu wrote:
 <snip>

 1) These interdependencies are currently expressed by hard-coded
 filesystem paths, which presumes a specific directory layout for
 checking out dmd, dlang.org, phobos, druntime. This means the build will
 break if somebody doesn't know the correct directory structure to use
 when checking out these repos.


 Yah, that should be in a wiki. I keep on thinking about producing a
 10-minutes video "Your First Contribution to D" in which I explain step
 by step how to get a pull request in starting from a scratch.

 We also have tools/update.sh which should probably be more prominently
 advertised.


 <snip>

 This shouldn't go (only) in the wiki.  It should be in the README of
 every repository.  And if not in the README, then the CONTRIBUTING file.
   A minimal list of "clone these things, run make in these directories"
 is fine here with a reference to the wiki for more details.

 I put off some documentation updates a few months ago because it wasn't
 worth the investment yet to figure out how to build the website.  The
 wiki pages appeared to be about building the compiler, which I didn't
 realize I should be interested in.  This barrier is too high.


Great ideas! I created https://issues.dlang.org/show_bug.cgi?id=13980 to 
track them. Thanks, Andrei
Jan 13 2015
prev sibling parent reply "Mike" <none none.com> writes:
On Wednesday, 14 January 2015 at 04:29:57 UTC, Paul O'Neil wrote:

 I put off some documentation updates a few months ago because 
 it wasn't
 worth the investment yet to figure out how to build the 
 website.  The
 wiki pages appeared to be about building the compiler, which I 
 didn't
 realize I should be interested in.  This barrier is too high.


I know the feeling.  I finally figured out how to get a single
ddoc file to html so I could help update the documentation.  I
added a few notes to the dlang.org CONTRIBUTING.md [1] file a
while back to help other users with that.  It's probably not very
noticeable there, but I don't understand the build process well
enough to do a proper job.

[1]
https://github.com/D-Programming-Language/dlang.org/blob/master/CONTRIBUTING.md

Why does documentation need to be "built" anyway.  I understand
the autogenerated stuff from the source code, but not the
website.  Would it be better to wikify it, somehow, in GitHub
pages?

Mike
Jan 13 2015
parent "Mike" <none none.com> writes:
On Wednesday, 14 January 2015 at 07:55:31 UTC, Mike wrote:


 Would it be better to wikify it, somehow, in GitHub
 pages?


I think wiki is the wrong term here.  I mean, is there a way to 
leverage Markdown (or some other markup language) with GitHub's 
source code management features to make it easier for 
contributors to fork, edit in their browser, verify in their 
repository, and create pull requests without having to build with 
a compiler?

Mike
Jan 14 2015
prev sibling next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message 
news:m9497d$1pv2$1 digitalmars.com...


 Over the time a number of stuff has become quite duplicated across our 
 makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros for 
 e.g. rebasing repos, see 
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

 I think it's time to reduce clutter and duplication by migrating such 
 common stuff into one common.mak file. Question is, where should that file 
 sit? One candidate is tools/ but it's not really a tool. Another candidate 
 is dmd/src/ because that would be the first thing anyone depends on.

 Ideas?


If it's optional extra stuff like git shortcuts, then removing it from the 
makefiles and putting it in tools makes sense.  If it's mandatory stuff like 
OS detection, then it needs to stay where it is.

What's the actual problem you're trying to solve with this?  IMO duplication 
is less of a problem than overcomplicating the makefiles.
Jan 13 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/13/15 7:44 PM, Daniel Murphy wrote:

 "Andrei Alexandrescu"  wrote in message
 news:m9497d$1pv2$1 digitalmars.com...


 Over the time a number of stuff has become quite duplicated across our
 makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros
 for e.g. rebasing repos, see
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.


 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.

 Ideas?


 If it's optional extra stuff like git shortcuts, then removing it from
 the makefiles and putting it in tools makes sense.  If it's mandatory
 stuff like OS detection, then it needs to stay where it is.


Hmmm... I wonder why the distinction.


 What's the actual problem you're trying to solve with this?  IMO
 duplication is less of a problem than overcomplicating the makefiles.


The problem is basic code duplication with its known liabilities. I'm 
looking at stuff like this:

https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L58

https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L31

https://github.com/D-Programming-Language/druntime/blob/master/posix.mak#L8

Three's a charm. That code is relatively verbose (required newlines). I 
trust it doesn't need updates often, but it occupies real estate in a 
readily accessible position (beginning of file).

The real issue here is that acquiescing to such duplication discourages 
attempts at uniformization before they occur. I'm thinking of simpler, 
clearer naming conventions and uniform ways of doing the same things 
(such as unittesting or documentation).

Getting back to REBASE 
(https://github.com/D-Programming-Language/dlang.org/blob/ma
ter/posix.mak#L37), 
it greatly simplifies my building the entire website in a way I trust is 
nice:

   make rebase -j && make clean && make rsync -j

(Sadly make clean is necessary because of dub, which makes the process 
take a lot longer than it should; with luck, somebody will improve on 
that soon.)

Now "make rebase" is really useful for druntime and phobos as well. But 
between duplicating more code or just doing it by hand, one may as well 
give up and choose the latter.

I'd also like to define "make rebase-dirty" based on this:

REBASE_DIRTY = MYBRANCH=`git rev-parse --abbrev-ref HEAD` &&\
  git stash &&\
  git co master &&\
  git pull --ff-only upstream master &&\
  git co $$MYBRANCH &&\
  git rebase master &&\
  git stash pop

which is useful because it lets me rebase on top of work in progress. 
Again the prospect of duplicating this other macro across 3-4 projects 
is fairly unattractive.



Andrei
Jan 13 2015
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message 
news:m94vfk$2gdh$1 digitalmars.com...


 Hmmm... I wonder why the distinction.


OS detection is required to build successfully, a rebase shortcut is not.


 The problem is basic code duplication with its known liabilities. I'm 
 looking at stuff like this:

 Three's a charm. That code is relatively verbose (required newlines). I 
 trust it doesn't need updates often, but it occupies real estate in a 
 readily accessible position (beginning of file).

 The real issue here is that acquiescing to such duplication discourages 
 attempts at uniformization before they occur. I'm thinking of simpler, 
 clearer naming conventions and uniform ways of doing the same things (such 
 as unittesting or documentation).


The problem here is really that make sucks, right?  And makefiles suck. 
Maybe we should start seriously looking at replacing them with D scripts.


 Getting back to REBASE 
 (https://github.com/D-Programming-Language/dlang.org/blob/ma
ter/posix.mak#L37), 
 it greatly simplifies my building the entire website in a way I trust is 
 nice:

    make rebase -j && make clean && make rsync -j

 (Sadly make clean is necessary because of dub, which makes the process 
 take a lot longer than it should; with luck, somebody will improve on that 
 soon.)

 Now "make rebase" is really useful for druntime and phobos as well. But 
 between duplicating more code or just doing it by hand, one may as well 
 give up and choose the latter.


Honestly I don't think automating rebase is a good idea, because it 
discourages actually learning how to do it.

eg if all you're trying to do is get the current branch rebased on top of 
upstream's master, you can do this:

git pull upstream master --rebase


 I'd also like to define "make rebase-dirty" based on this:

 REBASE_DIRTY = MYBRANCH=`git rev-parse --abbrev-ref HEAD` &&\
   git stash &&\
   git co master &&\
   git pull --ff-only upstream master &&\
   git co $$MYBRANCH &&\
   git rebase master &&\
   git stash pop

 which is useful because it lets me rebase on top of work in progress. 
 Again the prospect of duplicating this other macro across 3-4 projects is 
 fairly unattractive.


I've managed to get by fine with using shell scripts for stuff like this.
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 12:29 AM, Daniel Murphy wrote:

 The problem here is really that make sucks, right?  And makefiles suck.
 Maybe we should start seriously looking at replacing them with D scripts.


I agree that make sucks etc. but duplication sucks more (and would suck 
regardless of what tool we use). Here we do have a simple method 
(include files) that is suitable for addressing that. I'd assume we'd 
use something similar with whichever tool we'd use.


 Honestly I don't think automating rebase is a good idea, because it
 discourages actually learning how to do it.

 eg if all you're trying to do is get the current branch rebased on top
 of upstream's master, you can do this:

 git pull upstream master --rebase


Great. As an aside I just ran that and got:

error: unknown option `rebase'
usage: ...

Whether it's a command or a couple, that doesn't change the fact that I 
need to run the same command four times, and in parallel if at all 
possible. It's a perfect candidate for automation, and I haven't seen an 
argument the makefile isn't a good place for that.


 I'd also like to define "make rebase-dirty" based on this:

 REBASE_DIRTY = MYBRANCH=`git rev-parse --abbrev-ref HEAD` &&\
   git stash &&\
   git co master &&\
   git pull --ff-only upstream master &&\
   git co $$MYBRANCH &&\
   git rebase master &&\
   git stash pop

 which is useful because it lets me rebase on top of work in progress.
 Again the prospect of duplicating this other macro across 3-4 projects
 is fairly unattractive.


 I've managed to get by fine with using shell scripts for stuff like this.


Do you need to upload the dlang.org website quickly and without error?

Overall I have difficulty understanding counterarguments.

Challenge: "We should eliminate some unpleasant duplication."

Response: "Make sucks. Make includes are unreliable for mythical 
unexplained reasons. You don't know git. You should do scripts instead."

The response just doesn't follow. Care to clarify what you think we 
should be doing here?


Andrei
Jan 14 2015
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message 
news:m95ds3$2vm4$1 digitalmars.com...


 I agree that make sucks etc. but duplication sucks more (and would suck 
 regardless of what tool we use). Here we do have a simple method (include 
 files) that is suitable for addressing that. I'd assume we'd use something 
 similar with whichever tool we'd use.


I don't mind using include files, I mind adding more dependencies between 
projects.


 Great. As an aside I just ran that and got:

 error: unknown option `rebase'
 usage: ...


Maybe it's 'git pull --rebase upstream master', I didn't try it and I don't 
type it manually very often because I have a script to do this.


 Whether it's a command or a couple, that doesn't change the fact that I 
 need to run the same command four times, and in parallel if at all 
 possible. It's a perfect candidate for automation, and I haven't seen an 
 argument the makefile isn't a good place for that.


I guess we have different usage patterns, which isn't really a surprise. 
Anyway putting rebase in the makefile is a secondary issue, that's probably 
not worth discussing now.


 I've managed to get by fine with using shell scripts for stuff like 
 this.


 Do you need to upload the dlang.org website quickly and without error?


Do enough people need to that it warrants being in the makefile instead of a 
local shell script?  Does everyone do it the same way?


 Overall I have difficulty understanding counterarguments.

 Challenge: "We should eliminate some unpleasant duplication."

 Response: "Make sucks. Make includes are unreliable for mythical 
 unexplained reasons. You don't know git. You should do scripts instead."

 The response just doesn't follow. Care to clarify what you think we should 
 be doing here?


Apologies for not being clear.

Challenge: "We should eliminate some unpleasant duplication."

Response:  "We should avoid adding more dependencies between projects at all 
costs."
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 2:09 AM, Daniel Murphy wrote:

 "Andrei Alexandrescu"  wrote in message
 news:m95ds3$2vm4$1 digitalmars.com...


 I agree that make sucks etc. but duplication sucks more (and would
 suck regardless of what tool we use). Here we do have a simple method
 (include files) that is suitable for addressing that. I'd assume we'd
 use something similar with whichever tool we'd use.


 I don't mind using include files, I mind adding more dependencies
 between projects.


Why? Druntime depends on dmd. Phobos depends on dmd and druntime. 
Dlang.org and tools depend on dmd, druntime, and phobos. It's a fact of 
life.


 Great. As an aside I just ran that and got:

 error: unknown option `rebase'
 usage: ...


 Maybe it's 'git pull --rebase upstream master', I didn't try it and I
 don't type it manually very often because I have a script to do this.


 Whether it's a command or a couple, that doesn't change the fact that
 I need to run the same command four times, and in parallel if at all
 possible. It's a perfect candidate for automation, and I haven't seen
 an argument the makefile isn't a good place for that.


 I guess we have different usage patterns, which isn't really a surprise.
 Anyway putting rebase in the makefile is a secondary issue, that's
 probably not worth discussing now.


 I've managed to get by fine with using shell scripts for stuff like
 this.


 Do you need to upload the dlang.org website quickly and without error?


 Do enough people need to that it warrants being in the makefile instead
 of a local shell script?  Does everyone do it the same way?


Yes.

Currently I'm the site build master by default. In the near future 
there'll be several. I'd rather have anyone run a simple command than 
telling them to write their scripts.

So if next month I transfer this responsibility to Martin, I can tell: 
"to update dlang.org, run 'make rebase -j && make rsync -j'" as opposed 
to "yeah, I have some scripts somewhere, do your own or let me email 
them to you".

I don't really understand your position here. It's like you reject 
robust automation over tribal knowledge and folklore.


 Overall I have difficulty understanding counterarguments.

 Challenge: "We should eliminate some unpleasant duplication."

 Response: "Make sucks. Make includes are unreliable for mythical
 unexplained reasons. You don't know git. You should do scripts instead."

 The response just doesn't follow. Care to clarify what you think we
 should be doing here?


 Apologies for not being clear.

 Challenge: "We should eliminate some unpleasant duplication."

 Response:  "We should avoid adding more dependencies between projects at
 all costs."


Why? As an aside, "at all costs" is suspicious in many contexts.


Andrei
Jan 14 2015
next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message news:m95i3q$1t6$1 digitalmars.com...


 I don't mind using include files, I mind adding more dependencies
 between projects.


 Why? Druntime depends on dmd. Phobos depends on dmd and druntime. 
 Dlang.org and tools depend on dmd, druntime, and phobos. It's a fact of 
 life.


Druntime, phobos and dlang.org don't (currently) depend on dmd's source.
Jan 14 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 3:29 AM, Daniel Murphy wrote:

 "Andrei Alexandrescu"  wrote in message
 news:m95i3q$1t6$1 digitalmars.com...


 I don't mind using include files, I mind adding more dependencies
 between projects.


 Why? Druntime depends on dmd. Phobos depends on dmd and druntime.
 Dlang.org and tools depend on dmd, druntime, and phobos. It's a fact
 of life.


 Druntime, phobos and dlang.org don't (currently) depend on dmd's source.


I see your point, though I'll note that DMD everywhere defaults to 
../dmd/src/dmd (and that's a huge improvement from the time is was 
defined as just dmd). But granted setting it in the command line would 
make those independent of the dmd location.

So the question is - is there a place that's common to all?


Andrei
Jan 14 2015
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Jan 14, 2015 at 02:58:02AM -0800, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 1/14/15 2:09 AM, Daniel Murphy wrote:


[...]

Do enough people need to that it warrants being in the makefile
instead of a local shell script?  Does everyone do it the same way?


 
 Yes.
 
 Currently I'm the site build master by default. In the near future
 there'll be several. I'd rather have anyone run a simple command than
 telling them to write their scripts.
 
 So if next month I transfer this responsibility to Martin, I can tell:
 "to update dlang.org, run 'make rebase -j && make rsync -j'" as
 opposed to "yeah, I have some scripts somewhere, do your own or let me
 email them to you".
 
 I don't really understand your position here. It's like you reject
 robust automation over tribal knowledge and folklore.


What stops us from adding said scripts to the dlang.org repo? Or tools,
for that matter, which is probably more appropriate.

Not everything needs to be shoehorned into the makefiles. 



Overall I have difficulty understanding counterarguments.

Challenge: "We should eliminate some unpleasant duplication."

Response: "Make sucks. Make includes are unreliable for mythical
unexplained reasons. You don't know git. You should do scripts
instead."

The response just doesn't follow. Care to clarify what you think we
should be doing here?


Apologies for not being clear.

Challenge: "We should eliminate some unpleasant duplication."

Response:  "We should avoid adding more dependencies between projects
at all costs."


 
 Why? As an aside, "at all costs" is suspicious in many contexts.


Because it imposes a specific external directory structure which the
casual user has no idea about. This stuff is documented nowhere
currently -- I had to read the makefiles to figure it out for myself;
not many people have that much patience.  It makes the whole thing more
fragile because as soon as one or more repos go out-of-sync (not on
github, but in the user's local clones, which can happen very easily)
things can break horribly without any warning.

I'm astonished this has to be spelled out in, of all places, a D forum.
Surely, of all people, we D people must be all too familiar with exactly
why things like #include <../../../mylib/mymod/myheader.h> are evil? I
mean, why else would we have D modules rather than continue this
glorious C heritage? Yet here we're proposing to add the equivalent of
such out-of-tree #include's to our makefiles *and* we're straining to
find reasons why we shouldn't do it? I'm at a loss of words to describe
the cognitive dissonance here.


T

-- 
Creativity is not an excuse for sloppiness.
Jan 14 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:

 What stops us from adding said scripts to the dlang.org repo? Or tools,
 for that matter, which is probably more appropriate.

 Not everything needs to be shoehorned into the makefiles.


As I explained in my previous message it's a sensible solution. -- Andrei
Jan 14 2015
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:

 Because it imposes a specific external directory structure which the
 casual user has no idea about.


I think that's a given already and has been for a long time. -- Andrei
Jan 14 2015
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:

 I'm astonished this has to be spelled out in, of all places, a D forum.
 Surely, of all people, we D people must be all too familiar with exactly
 why things like #include <../../../mylib/mymod/myheader.h> are evil?


I think that would be applying one canned lesson to a situation where it 
doesn't quite apply. -- Andrei
Jan 14 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:

 What stops us from adding said scripts to the dlang.org repo? Or tools,
 for that matter, which is probably more appropriate.


Oh, I just figured that would make everything dependent on dlang.org. 
That would be a new dependency :o). -- Andrei
Jan 14 2015
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/13/2015 3:20 PM, Andrei Alexandrescu wrote:

 Over the time a number of stuff has become quite duplicated across our
makefiles
 for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros for e.g.
 rebasing repos, see
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

 I think it's time to reduce clutter and duplication by migrating such common
 stuff into one common.mak file. Question is, where should that file sit? One
 candidate is tools/ but it's not really a tool. Another candidate is dmd/src/
 because that would be the first thing anyone depends on.

 Ideas?



It's a few lines of duplication, as opposed to adding another point of failure 
to the build process, i.e. common.mak isn't found or the wrong one is found,
etc.

I've always tended to avoid using make macro include's for this reason.
Jan 13 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/13/15 8:49 PM, Walter Bright wrote:

 It's a few lines of duplication, as opposed to adding another point of
 failure to the build process, i.e. common.mak isn't found or the wrong
 one is found, etc.


I'd have difficulty agreeing with this. If dmd is found then whatever is 
there is found. It's a deterministic process, not an arbitrary path 
lookup dependent on system vagaries.

Plus... duplication only grows.


 I've always tended to avoid using make macro include's for this reason.


I think I failed to grasp the reason.


Andrei
Jan 13 2015
prev sibling next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2015-01-13 at 15:20 -0800, Andrei Alexandrescu via Digitalmars-d
wrote:

 Hey folks,
=20
=20
 Over the time a number of stuff has become quite duplicated across our=


=20

 makefiles for dmd, druntime, and phobos.


Are these three separate things with separate builds, or are they three
components of the same system? What is the relationship between these
three, LDC and GDC?


 These include fetching OS and model but (newer) general-purpose macros=


=20

 for e.g. rebasing repos, see=20
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak=


#L37.

=20
 I think it's time to reduce clutter and duplication by migrating such=20
 common stuff into one common.mak file. Question is, where should that=20
 file sit? One candidate is tools/ but it's not really a tool. Another=20
 candidate is dmd/src/ because that would be the first thing anyone=20
 depends on.
=20
 Ideas?


Are all three things in the same Git repository? Can I clone the three
independently or do I have to have them all? Can any of them be built
without building the other or does a DMD build require a new druntime
and phobos build.

Basically what I am trying to say here is that I suspect the extraction
of commonality is going to create an inappropriate coupling. The
solution is to have the common material as a fourth thing that can be
brought in to any or all of the three. With SCons and Waf this is
achieved by having a tool, easy simple and keeps the coupling low.

I haven't used Make in 12 years, so I have no idea what it can do these
days, But unless you mandate GNU Make, I suspect it is going to be awful
trying to build across Windows, OSX, Debian, Arch, Fedora, FreeBSD,
OpenBSD, etc. without hacking (in the worst sense).

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Jan 14 2015
prev sibling next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
One thought because of somethings being said, dmd-fe is being converted 
to D. Which means to compile dmd you need a D compiler already.
We could make all the build scripts, D instead.

It would mean one less external to D ecosystem dependency.
But it could easily be a nightmare.
Jan 14 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-01-14 00:20, Andrei Alexandrescu wrote:

 Hey folks,


 Over the time a number of stuff has become quite duplicated across our
 makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros
 for e.g. rebasing repos, see
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.


 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.


Oh God, please no. I really, really hate that the makefiles are are 
looking in folders outside of the repositories.


 Ideas?


If you really want to do something like this I would recommend creating 
a new repository, acting like a super repository, including the other 
repositories as submodules. Then you can create a new makefile that 
builds everything. Although, I still don't want the makefiles in the 
existing repositories be dependent on the new repository.

-- 
/Jacob Carlborg
Jan 14 2015
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Jan 14, 2015 at 04:27:10PM +0100, Jacob Carlborg via Digitalmars-d
wrote:

 On 2015-01-14 00:20, Andrei Alexandrescu wrote:


[...]

I think it's time to reduce clutter and duplication by migrating such
common stuff into one common.mak file. Question is, where should that
file sit? One candidate is tools/ but it's not really a tool. Another
candidate is dmd/src/ because that would be the first thing anyone
depends on.


 
 Oh God, please no. I really, really hate that the makefiles are are
 looking in folders outside of the repositories.


+1. Makefiles that reference paths outside the repo are really really
evil. They only ever work for the specific setup the author has, and
breaks for everybody else.



Ideas?


 
 If you really want to do something like this I would recommend
 creating a new repository, acting like a super repository, including
 the other repositories as submodules. Then you can create a new
 makefile that builds everything. Although, I still don't want the
 makefiles in the existing repositories be dependent on the new
 repository.


[...]

+1, this is a much better approach than introducing yet more extra-repo
hard-coded paths into the makefiles. There are too many of those
already.


T

-- 
Truth, Sir, is a cow which will give [skeptics] no more milk, and so
they are gone to milk the bull. -- Sam. Johnson
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:01 AM, H. S. Teoh via Digitalmars-d wrote:

 On Wed, Jan 14, 2015 at 04:27:10PM +0100, Jacob Carlborg via Digitalmars-d
wrote:

 On 2015-01-14 00:20, Andrei Alexandrescu wrote:


 [...]

 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.


 Oh God, please no. I really, really hate that the makefiles are are
 looking in folders outside of the repositories.


 +1. Makefiles that reference paths outside the repo are really really
 evil. They only ever work for the specific setup the author has, and
 breaks for everybody else.


Currently there must be a place where druntime finds dmd for building 
purposes. Also phobos needs to find druntime. Also dlang.ord needs to 
find the source of both phobos and druntime.

There's no specific setup of one person as much as a boring:

/some/dir/
   dmd
   druntime
   phobos
   dlang.org
   tools

I have a hard time making it any simpler.


 Ideas?


 If you really want to do something like this I would recommend
 creating a new repository, acting like a super repository, including
 the other repositories as submodules. Then you can create a new
 makefile that builds everything. Although, I still don't want the
 makefiles in the existing repositories be dependent on the new
 repository.


 [...]

 +1, this is a much better approach than introducing yet more extra-repo
 hard-coded paths into the makefiles. There are too many of those
 already.


Where would that super-repo sit?


Andrei
Jan 14 2015
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Jan 14, 2015 at 11:50:39AM -0800, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 1/14/15 9:01 AM, H. S. Teoh via Digitalmars-d wrote:

On Wed, Jan 14, 2015 at 04:27:10PM +0100, Jacob Carlborg via Digitalmars-d
wrote:

On 2015-01-14 00:20, Andrei Alexandrescu wrote:


[...]

I think it's time to reduce clutter and duplication by migrating
such common stuff into one common.mak file. Question is, where
should that file sit? One candidate is tools/ but it's not really a
tool. Another candidate is dmd/src/ because that would be the first
thing anyone depends on.


Oh God, please no. I really, really hate that the makefiles are are
looking in folders outside of the repositories.


+1. Makefiles that reference paths outside the repo are really really
evil. They only ever work for the specific setup the author has, and
breaks for everybody else.


 
 Currently there must be a place where druntime finds dmd for building
 purposes. Also phobos needs to find druntime. Also dlang.ord needs to
 find the source of both phobos and druntime.
 
 There's no specific setup of one person as much as a boring:
 
 /some/dir/
   dmd
   druntime
   phobos
   dlang.org
   tools
 
 I have a hard time making it any simpler.


I think you're missing the point. The problem is not that a specific
setup is required. The problem is that the user has to (1) know exactly
what this setup is, and (2) set it up *manually*. The first point is a
dire lack of documentation, the second point begs the question "why
require the user to do this when we can do this for them?". I.e., via
submodules, see below.

Not to mention, your proposed setup does not match dlang.org's
makefile's idea of what the setup should be. (For example, it generates
html files in dlang.org/web whereas the Phobos makefile generates html
files in phobos/../web, which appears to suggest that
{dmd,druntime,phobos} ought to be nested inside dlang.org instead of
being siblings as you have shown.) If even you can't keep things
consistent with what's in dlang.org's makefiles, what hope do the rest
of us have, let alone a casual user who's only tentatively interested in
D?



Ideas?


If you really want to do something like this I would recommend
creating a new repository, acting like a super repository, including
the other repositories as submodules. Then you can create a new
makefile that builds everything. Although, I still don't want the
makefiles in the existing repositories be dependent on the new
repository.


[...]

+1, this is a much better approach than introducing yet more extra-repo
hard-coded paths into the makefiles. There are too many of those
already.


 
 Where would that super-repo sit?


[...]

Huh?? What do you mean, where it would sit? If you set up a
"super-repo", it would look something like this:







Then checking out this new dlang repo (with the requisite git flags)
will create the above structure for you, including clone all the
sub-repos. *This* is what will address point (2) that I raised earlier:
the user shouldn't need to manually setup some arbitrary directory
structure for you -- automate this with git submodules instead!

Having such a super-repo will also solve the problem of git bisecting
earlier toolchain releases -- because the super-repo references the
sub-repos by SHA hashes, which are history-tracked by git, so we could
set it up so that each commit of the super-repo corresponds with a valid
version set of dmd/druntime/phobos. Then you could checkout a specific
commit of the super-repo and update git submodules, and you'll get a
snapshot of the entire toolchain as it stood at the time of the commit.
If we do this right, we could even have the autotester update the
super-repo automatically upon successful completion of the test suite
for the entire toolchain.

And ideally, stuff common to multiple subrepos should be in this
"super-repo", such as the stuff you're trying to factor out right now.
Including the master makefile that will build everything WITHOUT needing
to reference out-of-tree paths. And ideally, any out-of-tree references
needed by the subrepos will be via configurable parameters (e.g. $(DMD),
$(DRUNTIME_PATH), etc.), which the master makefile can set to the
correct values, so the subrepos don't have to know or hardcode where the
exact out-of-tree location ought to be.

Such a super-repo will also make release management easier: instead of
manually managing release tags in each of dmd/druntime/phobos, we can
tag the super-repo, which encapsulates the exact set of
dmd/druntime/phobos commits for that release.


T

-- 
There are 10 kinds of people in the world: those who can count in binary, and
those who can't.
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 12:18 PM, H. S. Teoh via Digitalmars-d wrote:

 If you set up a
 "super-repo", it would look something like this:







 Then checking out this new dlang repo (with the requisite git flags)
 will create the above structure for you, including clone all the
 sub-repos. *This* is what will address point (2) that I raised earlier:
 the user shouldn't need to manually setup some arbitrary directory
 structure for you -- automate this with git submodules instead!


Agreed. Ah, this is getting interesting, thanks. So I've wanting to look 
into submodules for a while now but have been discouraged by the bad 
press they seem to be getting:

https://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/

http://ayende.com/blog/4746/the-problem-with-git-submodules

http://blogs.atlassian.com/2013/03/git-submodules-workflows-tips/

http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/

http://somethingsinistral.net/blog/git-submodules-are-probably-not-the-answer/ 
(though I have no idea whether what's relevant for puppet is relevant to us)


 Having such a super-repo will also solve the problem of git bisecting
 earlier toolchain releases -- because the super-repo references the
 sub-repos by SHA hashes, which are history-tracked by git, so we could
 set it up so that each commit of the super-repo corresponds with a valid
 version set of dmd/druntime/phobos. Then you could checkout a specific
 commit of the super-repo and update git submodules, and you'll get a
 snapshot of the entire toolchain as it stood at the time of the commit.
 If we do this right, we could even have the autotester update the
 super-repo automatically upon successful completion of the test suite
 for the entire toolchain.

 And ideally, stuff common to multiple subrepos should be in this
 "super-repo", such as the stuff you're trying to factor out right now.
 Including the master makefile that will build everything WITHOUT needing
 to reference out-of-tree paths. And ideally, any out-of-tree references
 needed by the subrepos will be via configurable parameters (e.g. $(DMD),
 $(DRUNTIME_PATH), etc.), which the master makefile can set to the
 correct values, so the subrepos don't have to know or hardcode where the
 exact out-of-tree location ought to be.

 Such a super-repo will also make release management easier: instead of
 manually managing release tags in each of dmd/druntime/phobos, we can
 tag the super-repo, which encapsulates the exact set of
 dmd/druntime/phobos commits for that release.


I hear you but The Internet says submodules are a kind of a bummer.


Andrei
Jan 14 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-01-14 21:54, Andrei Alexandrescu wrote:


 I hear you but The Internet says submodules are a kind of a bummer.


Most issues I have read about is because they don't know how submodules 
work. It's the same thing with most tools, if you don't know how to use 
it the experience is going to be crappy.

I'm hoping the submodules can be automatically update, i.e. on a push to 
some of the submodules. I'm expecting most work to be done in the 
submodules and the super repository will be kept in sync automatically.

-- 
/Jacob Carlborg
Jan 14 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 15 January 2015 at 07:33:38 UTC, Jacob Carlborg 
wrote:

 On 2015-01-14 21:54, Andrei Alexandrescu wrote:


 I hear you but The Internet says submodules are a kind of a 
 bummer.


 Most issues I have read about is because they don't know how 
 submodules work. It's the same thing with most tools, if you 
 don't know how to use it the experience is going to be crappy.

 I'm hoping the submodules can be automatically update, i.e. on 
 a push to some of the submodules. I'm expecting most work to be 
 done in the submodules and the super repository will be kept in 
 sync automatically.


We have been using git submodules for internal dependencies of 
Sociomantic projects for quite a while. It is not perfect but it 
works good if you defines terms of usage strict enough.

In this specific case I think it may work with help of few shell 
scripts that may reside in same top-level repo. Actually I do 
remember someone already trying that setup but can't find the 
links now. Anyone with better memory?
Jan 15 2015
parent reply "Dicebot" <public dicebot.lv> writes:
Important thing here is that starting with git 1.8.2 it is 
possible to set submodule to track head of remote branch instead 
of fixed commit hash - which fixes the maintenance issue and is 
exactly the thing needed for meta-dlang repo.
Jan 15 2015
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Jan 15, 2015 at 04:08:14PM +0000, Dicebot via Digitalmars-d wrote:

 Important thing here is that starting with git 1.8.2 it is possible to
 set submodule to track head of remote branch instead of fixed commit
 hash - which fixes the maintenance issue and is exactly the thing
 needed for meta-dlang repo.


Very nice!! I wasn't aware of that... I must be behind the times. :-P

 Andrei: I think this nullifies about 50% of the naysayers' arguments
against git submodules. Another issue is also that people used git
submodules for things it wasn't intended for, and got burned as a
result. Not really the fault of the feature. There *are* still some
things to be desired, though, so it's not quite perfect yet. But it
should be a lot more usable than before.


T

-- 
What are you when you run out of Monet? Baroque.
Jan 15 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/15/15 10:27 AM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, Jan 15, 2015 at 04:08:14PM +0000, Dicebot via Digitalmars-d wrote:

 Important thing here is that starting with git 1.8.2 it is possible to
 set submodule to track head of remote branch instead of fixed commit
 hash - which fixes the maintenance issue and is exactly the thing
 needed for meta-dlang repo.


 Very nice!! I wasn't aware of that... I must be behind the times. :-P

  Andrei: I think this nullifies about 50% of the naysayers' arguments
 against git submodules. Another issue is also that people used git
 submodules for things it wasn't intended for, and got burned as a
 result. Not really the fault of the feature. There *are* still some
 things to be desired, though, so it's not quite perfect yet. But it
 should be a lot more usable than before.


I'd be okay with reorganizing our repos as a module with the submodules 
dmd, druntime, phobos, and tools. For that we'd need a champion who is 
willing to try it, experiment with it (e.g. in a separate repo), 
demonstrate its advantages and discuss its disadvantages, and carry the 
actual porting work.

It seems to me we'd need to derive a few more advantages than solving 
the particular matter I opened this thread with.


Andrei
Jan 15 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-01-15 19:45, Andrei Alexandrescu wrote:


 I'd be okay with reorganizing our repos as a module with the submodules
 dmd, druntime, phobos, and tools. For that we'd need a champion who is
 willing to try it, experiment with it (e.g. in a separate repo),
 demonstrate its advantages and discuss its disadvantages, and carry the
 actual porting work.


I hope it will only require minimal changes in the individual 
submodules. Most things should be done in the super repository. There's 
nothing that needs to be done for a git repository to make it a 
submodule in another repository.

-- 
/Jacob Carlborg
Jan 15 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-01-15 17:08, Dicebot wrote:


 Important thing here is that starting with git 1.8.2 it is possible to
 set submodule to track head of remote branch instead of fixed commit
 hash - which fixes the maintenance issue and is exactly the thing needed
 for meta-dlang repo.


Yeah, but in practice I haven't noticed any difference. You still need 
to manually update the submodules with a command and make a commit in 
the super repository.

I did some experiments with this at work when we moved from SVN to Git.

-- 
/Jacob Carlborg
Jan 15 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-01-14 20:50, Andrei Alexandrescu wrote:


 Currently there must be a place where druntime finds dmd for building
 purposes. Also phobos needs to find druntime. Also dlang.ord needs to
 find the source of both phobos and druntime.

 There's no specific setup of one person as much as a boring:

 /some/dir/
    dmd
    druntime
    phobos
    dlang.org
    tools

 I have a hard time making it any simpler.


Last time I tried to build the docs in dlang.org, quite a long time ago. 
The d-programming-language.org repository just had been moved to 
dlang.org. So I update the repository URL, pull the new code and tried 
building the docs. Nothing worked because I had forget to rename the 
directory. A super repository with submodules would have solved this.


 Where would that super-repo sit?


https://github.com/D-Programming-Language/dlang for example

-- 
/Jacob Carlborg
Jan 14 2015
prev sibling parent reply "Mathias LANG" <geod24 gmail.com> writes:
On Tuesday, 13 January 2015 at 23:20:13 UTC, Andrei Alexandrescu 
wrote:

 Hey folks,


 Over the time a number of stuff has become quite duplicated 
 across our makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose 
 macros for e.g. rebasing repos, see 
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

 I think it's time to reduce clutter and duplication by 
 migrating such common stuff into one common.mak file. Question 
 is, where should that file sit? One candidate is tools/ but 
 it's not really a tool. Another candidate is dmd/src/ because 
 that would be the first thing anyone depends on.

 Ideas?


 Andrei


IMO helpers (such as git helpers) are not part of the build 
process, so they should not be part of the build files. Few 
people care about make rebase. I'm not saying we should delete 
them (they obviously are useful to you), but moving them to tools 
might be a good start.

For druntime / phobos build... That's really a shame we don't 
have a standard D tool to build D code, cross platform.
If we had, not only would we promote an even saner image of our 
tooling ('hey look, dogfooding'), we won't have to solve the same 
problems over and over again, so both druntime / phobos and the 
tool would benefit from this, but ofc it will require some 
initial work.
If someone wants to get this project started, I have a suggestion 
for the name: dub

Okay, sarcasms appart, would that be a possibility for us to 
switch to dub in the long run ? I'm not sure dub is ready for 
that ATM, but I'm confident it could be made ready without much 
work.
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 8:50 AM, Mathias LANG wrote:

 IMO helpers (such as git helpers) are not part of the build process, so
 they should not be part of the build files. Few people care about make
 rebase. I'm not saying we should delete them (they obviously are useful
 to you), but moving them to tools might be a good start.


This is about scaling up and passing responsibility around. I could 
write a script and then tell whoever maintains dlang.org "first run that 
script then run make" etc.

With make it turns out it's dead simple to get parallel rebase going:

REBASE = git pull upstream master --rebase

rebase: rebase-dlang rebase-dmd rebase-druntime rebase-phobos
rebase-dlang: ; $(REBASE)
rebase-dmd: ; cd $(DMD_DIR) && $(REBASE)
rebase-druntime: ; cd $(DRUNTIME_DIR) && $(REBASE)
rebase-phobos: ; cd $(PHOBOS_DIR) && $(REBASE)

So I'm done in 6 lines and it's parallel automatically, no code needed. 
With zsh I'd probably go (untested)...

===========================

set -e

function rebase() {
   ( cd $1 && git pull upstream master --rebase )
}

rebase . &&
rebase ../dmd &&
rebase ../druntime &&
rebase ../phobos &&

wait
===========================

But wait, there's less. The script:

* does not return nonzero if either subtask fails - and doing so needs 
e.g. temporary files and takes extra code. So I need to carefully look 
at the textual garbled output to make sure it all works.

* does not accept setting variables DMD_DIR etc. It can be made to, but 
that's yet more code.

* cannot run serially if I want to debug something, only parallel. It 
can be made do, but that's again more code!

So we're looking at an ideological response "don't use make, make sucks" 
to a pragmatic matter - I want to get work done, and we shouldn't use 
inferior/more intensive technical solutions on an ideological basis 
alone. I agree make sucks! But that doesn't suddenly eliminate my common 
sense.


 For druntime / phobos build... That's really a shame we don't have a
 standard D tool to build D code, cross platform.
 If we had, not only would we promote an even saner image of our tooling
 ('hey look, dogfooding'), we won't have to solve the same problems over
 and over again, so both druntime / phobos and the tool would benefit
 from this, but ofc it will require some initial work.
 If someone wants to get this project started, I have a suggestion for
 the name: dub

 Okay, sarcasms appart, would that be a possibility for us to switch to
 dub in the long run ? I'm not sure dub is ready for that ATM, but I'm
 confident it could be made ready without much work.


I'd be glad to switch to dub but I confess at this time I can't invest 
the time. I'm already spread too thin.


Andrei
Jan 14 2015
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Wednesday, 14 January 2015 at 20:07:12 UTC, Andrei 
Alexandrescu wrote:

 On 1/14/15 8:50 AM, Mathias LANG wrote:

 IMO helpers (such as git helpers) are not part of the build 
 process, so
 they should not be part of the build files. Few people care 
 about make
 rebase. I'm not saying we should delete them (they obviously 
 are useful
 to you), but moving them to tools might be a good start.


 This is about scaling up and passing responsibility around. I 
 could write a script and then tell whoever maintains dlang.org 
 "first run that script then run make" etc.

 With make it turns out it's dead simple to get parallel rebase 
 going:

 REBASE = git pull upstream master --rebase

 rebase: rebase-dlang rebase-dmd rebase-druntime rebase-phobos
 rebase-dlang: ; $(REBASE)
 rebase-dmd: ; cd $(DMD_DIR) && $(REBASE)
 rebase-druntime: ; cd $(DRUNTIME_DIR) && $(REBASE)
 rebase-phobos: ; cd $(PHOBOS_DIR) && $(REBASE)

 So I'm done in 6 lines and it's parallel automatically, no code 
 needed. With zsh I'd probably go (untested)...

 ===========================

 set -e

 function rebase() {
   ( cd $1 && git pull upstream master --rebase )
 }

 rebase . &&
 rebase ../dmd &&
 rebase ../druntime &&
 rebase ../phobos &&

 wait
 ===========================

 But wait, there's less. The script:

 * does not return nonzero if either subtask fails - and doing 
 so needs e.g. temporary files and takes extra code. So I need 
 to carefully look at the textual garbled output to make sure it 
 all works.

 * does not accept setting variables DMD_DIR etc. It can be made 
 to, but that's yet more code.

 * cannot run serially if I want to debug something, only 
 parallel. It can be made do, but that's again more code!

 So we're looking at an ideological response "don't use make, 
 make sucks" to a pragmatic matter - I want to get work done, 
 and we shouldn't use inferior/more intensive technical 
 solutions on an ideological basis alone. I agree make sucks! 
 But that doesn't suddenly eliminate my common sense.


 For druntime / phobos build... That's really a shame we don't 
 have a
 standard D tool to build D code, cross platform.
 If we had, not only would we promote an even saner image of 
 our tooling
 ('hey look, dogfooding'), we won't have to solve the same 
 problems over
 and over again, so both druntime / phobos and the tool would 
 benefit
 from this, but ofc it will require some initial work.
 If someone wants to get this project started, I have a 
 suggestion for
 the name: dub

 Okay, sarcasms appart, would that be a possibility for us to 
 switch to
 dub in the long run ? I'm not sure dub is ready for that ATM, 
 but I'm
 confident it could be made ready without much work.


 I'd be glad to switch to dub but I confess at this time I can't 
 invest the time. I'm already spread too thin.


 Andrei


So a dub file for phobos is needed? And maybe druntime too? I'll 
see what I can do.
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 3:14 PM, Rikki Cattermole wrote:

 So a dub file for phobos is needed? And maybe druntime too? I'll see
 what I can do.


That would be very interesting, thanks. Prolly druntime is easiest  - 
one less dependency. Please share all discoveries you make with the core 
team here. -- Andrei
Jan 14 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 12:24 p.m., Andrei Alexandrescu wrote:

 On 1/14/15 3:14 PM, Rikki Cattermole wrote:

 So a dub file for phobos is needed? And maybe druntime too? I'll see
 what I can do.


 That would be very interesting, thanks. Prolly druntime is easiest  -
 one less dependency. Please share all discoveries you make with the core
 team here. -- Andrei


Phobos is done (2.066 branch). I'll submit PR when soon.
Oh there will also be atleast two fixes.
Somebody forgot to add some version statements in linux/osx only code 
and since I'm lazy no way I'm specifying every single file to dub.
Jan 14 2015
parent reply "Mathias LANG" <geod24 gmail.com> writes:
On Wednesday, 14 January 2015 at 23:58:45 UTC, Rikki Cattermole 
wrote:

 On 15/01/2015 12:24 p.m., Andrei Alexandrescu wrote:

 On 1/14/15 3:14 PM, Rikki Cattermole wrote:

 So a dub file for phobos is needed? And maybe druntime too? 
 I'll see
 what I can do.


 That would be very interesting, thanks. Prolly druntime is 
 easiest  -
 one less dependency. Please share all discoveries you make 
 with the core
 team here. -- Andrei


 Phobos is done (2.066 branch). I'll submit PR when soon.
 Oh there will also be atleast two fixes.
 Somebody forgot to add some version statements in linux/osx 
 only code and since I'm lazy no way I'm specifying every single 
 file to dub.


excludedSourceFiles accept patterns and platform suffixes.
Jan 14 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 1:15 p.m., Mathias LANG wrote:

 On Wednesday, 14 January 2015 at 23:58:45 UTC, Rikki Cattermole wrote:

 On 15/01/2015 12:24 p.m., Andrei Alexandrescu wrote:

 On 1/14/15 3:14 PM, Rikki Cattermole wrote:

 So a dub file for phobos is needed? And maybe druntime too? I'll see
 what I can do.


 That would be very interesting, thanks. Prolly druntime is easiest  -
 one less dependency. Please share all discoveries you make with the core
 team here. -- Andrei


 Phobos is done (2.066 branch). I'll submit PR when soon.
 Oh there will also be atleast two fixes.
 Somebody forgot to add some version statements in linux/osx only code
 and since I'm lazy no way I'm specifying every single file to dub.


 excludedSourceFiles accept patterns and platform suffixes.


I'm using it for std.metastrings. Not really a loss since the 
distributed version doesn't include it (also doesn't compile).
In the case of the version statements? Its two line additions in total. 
They definitely should be there considering.
Jan 14 2015
prev sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 12:24 p.m., Andrei Alexandrescu wrote:

 On 1/14/15 3:14 PM, Rikki Cattermole wrote:

 So a dub file for phobos is needed? And maybe druntime too? I'll see
 what I can do.


 That would be very interesting, thanks. Prolly druntime is easiest  -
 one less dependency. Please share all discoveries you make with the core
 team here. -- Andrei


Druntime and phobos now can be built and even more importantly you can 
have druntime as a dependency to phobos. In fact it is.
Phobos supports being built as a dynamic library (there is a 
configuration for for it).
So if you want to switch to dynamic library at any time its as simple as 
adding:
"subConfigurations": {
	"phobos": "dynamicLibrary"
}
I'm pretty sure anyway.

Because of the current state of assembly files, we really need to sort 
it out. I understand that not all things are really possible in an 
inline assembler. But at the very least we should be able to compile the 
damn thing.

Dub can run external processes but its best to leave it up to the end 
dev to run the c/assembly generation. When they want it regenerated.
I think its best we commit most of the binary outputs.
Mostly c/assembly generation. Not the D stuff.

I've added information to the README on how to build based upon dub.
Docs so far have not been included. But that adds one more repo I need 
to get my hands dirty with.
Jan 14 2015
next sibling parent reply "Mathias LANG" <geod24 gmail.com> writes:
On Thursday, 15 January 2015 at 01:48:29 UTC, Rikki Cattermole 
wrote:

 Druntime and phobos now can be built and even more importantly 
 you can have druntime as a dependency to phobos. In fact it is.
 Phobos supports being built as a dynamic library (there is a 
 configuration for for it).
 So if you want to switch to dynamic library at any time its as 
 simple as adding:
 "subConfigurations": {
 	"phobos": "dynamicLibrary"
 }
 I'm pretty sure anyway.


Thanks !
Last time I checked, building dynamic library with dub was buggy. 
In addition, we can't have druntime has a dependency to phobos 
with dynamic library (yet). See 
https://github.com/D-Programming-Language/dub/blob/master/source/dub/generators/generator.d#L152

But having the static version in is a good start :)
Jan 14 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 2:58 p.m., Mathias LANG wrote:

 On Thursday, 15 January 2015 at 01:48:29 UTC, Rikki Cattermole wrote:

 Druntime and phobos now can be built and even more importantly you can
 have druntime as a dependency to phobos. In fact it is.
 Phobos supports being built as a dynamic library (there is a
 configuration for for it).
 So if you want to switch to dynamic library at any time its as simple
 as adding:
 "subConfigurations": {
     "phobos": "dynamicLibrary"
 }
 I'm pretty sure anyway.


 Thanks !
 Last time I checked, building dynamic library with dub was buggy. In
 addition, we can't have druntime has a dependency to phobos with dynamic
 library (yet). See
 https://github.com/D-Programming-Language/dub/blob/master/source/dub/generators/generator.d#L152


 But having the static version in is a good start :)


The configuration is there. Atleast I'm prepared!
Anyway static version is default, so as long as dynamic isn't advertised 
too much we're good.
Jan 14 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-01-15 02:48, Rikki Cattermole wrote:


 Dub can run external processes but its best to leave it up to the end
 dev to run the c/assembly generation. When they want it regenerated.
 I think its best we commit most of the binary outputs.
 Mostly c/assembly generation. Not the D stuff.


Why don't you want to build the C/assembly stuff? The user is already 
expected to have a C compiler installed.

-- 
/Jacob Carlborg
Jan 14 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 8:35 p.m., Jacob Carlborg wrote:

 On 2015-01-15 02:48, Rikki Cattermole wrote:


 Dub can run external processes but its best to leave it up to the end
 dev to run the c/assembly generation. When they want it regenerated.
 I think its best we commit most of the binary outputs.
 Mostly c/assembly generation. Not the D stuff.


 Why don't you want to build the C/assembly stuff? The user is already
 expected to have a C compiler installed.


Imagine your wanting to get some new flash feature (doesn't require 
compiler change) by just adding druntime/phobos as a dependency via dub.
You don't really want to recompile the assembly/c stuff that almost 
definitely hasn't changed.

Not many people may want to do that. But it should be reasonable to do so.
But this is mostly a Windows issue then posix. There its pretty well 
defined. Windows ergg.
Jan 15 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-01-15 09:30, Rikki Cattermole wrote:


 Imagine your wanting to get some new flash feature (doesn't require
 compiler change) by just adding druntime/phobos as a dependency via dub.
 You don't really want to recompile the assembly/c stuff that almost
 definitely hasn't changed.


Why not? It's more likely you'll run in to problems with forgetting to 
compile it when it really has changed.


 Not many people may want to do that. But it should be reasonable to do so.
 But this is mostly a Windows issue then posix. There its pretty well
 defined. Windows ergg.


If it's only a problem on Windows, don't compile it on Windows :). The 
other platforms should suffer because of this.

-- 
/Jacob Carlborg
Jan 15 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 16/01/2015 1:19 a.m., Jacob Carlborg wrote:

 On 2015-01-15 09:30, Rikki Cattermole wrote:


 Imagine your wanting to get some new flash feature (doesn't require
 compiler change) by just adding druntime/phobos as a dependency via dub.
 You don't really want to recompile the assembly/c stuff that almost
 definitely hasn't changed.


 Why not? It's more likely you'll run in to problems with forgetting to
 compile it when it really has changed.


In this case I think its pretty close between checking in and not.
Atleast for me, I generally keep my D environment separate from e.g. VC 
so things like this could get difficult unless it was configured per 
machine like it is for dmd.


 Not many people may want to do that. But it should be reasonable to do
 so.
 But this is mostly a Windows issue then posix. There its pretty well
 defined. Windows ergg.


 If it's only a problem on Windows, don't compile it on Windows :). The
 other platforms should suffer because of this.


Technically there is one assembly file required by druntime for Windows 
that can't even be compiled on any 32 or 64bit computer... and is 
checked into git.
Jan 15 2015