www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Phobos posix.mak -> D file using reggae: round 2

reply Atila Neves <atila.neves gmail.com> writes:
Here's[1] another attempt at converting the Makefile for POSIX 
systems to D using reggae[2]. I first tried my hand at writing a 
parser for Makefiles using Pegged and machine translating to D. I 
learned a lot about parsing and GNU make in the process and also 
that:

1. It was about two orders of magnitude more work than I thought 
to support the subset of GNU make used by posix.mak
2. The generated D code wasn't exactly pretty

I concluded that worse is better and hand translated all of 
posix.mak instead. It isn't as pretty as possible either, but 
that's on purpose - since I'll have to mentally apply any further 
diffs and hand translate, I tried to make it as close as possible 
to the existing Makefile. That way the edits should be obvious.

AFAICT by testing manually, it does everything that the original 
Makefile does, including all the different unit test targets and 
documentation. Then again, I only tested on Linux. I've been 
trying to bend the auto-tester to my will to make sure all 
posix.mak systems are still green but I ran into configuration 
issues.

I'll carry on maintaining the D description as updates to 
posix.mak happen for a year. If we haven't switched by then we 
probably never will.

To try it out yourself, get & build/run reggae from dub in this 
branch at the root dir of phobos selecting the backend of choice 
with `-b <backend>`. I suggest the binary one since one of the 
points of this is to remove dependencies from other build 
systems. Then run make/ninja/tup/./build with the same targets as 
usual. If that doesn't work... open an issue on github.

Destroy!

Atila


[1] https://github.com/atilaneves/phobos/blob/reggae/reggaefile.d
[2] https://github.com/atilaneves/reggae
Apr 18 2016
next sibling parent Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 18 April 2016 at 15:15:26 UTC, Atila Neves wrote:

 Here's[1] another attempt at converting the Makefile for POSIX 
 systems to D using reggae[2].


Great work! This transition cannot happen soon enough. 
Maintaining the make files is a huge pain and something always 
falls through the cracks.
Apr 18 2016
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 18 Apr 2016 15:15:26 +0000
schrieb Atila Neves <atila.neves gmail.com>:

Just remember that makefiles are well integrated with other
systems (i.e. package managers) and there are some
conventions, about what certain environment variables mean.
To name a few:

MAKEOPTS=-j4  (parallel build with N processes)
DC=/opt/ldc2-0.17/bin/ldc2  (Dlang compiler)
DC_FLAGS=-O5  (flags for that compiler)
LDFLAGS=-L=-O1  (flags for the linker)
PREFIX=/usr  (installation prefix)

Dlang projects already have some make files that handle linker
flag prefixes while others will pass them verbatim to gcc and
some make files accept a DMD (ldmd2, gdmd, dmd) and others a
DC (ldc2, gdc, dmd). The more things we can override, the
better.
As a practical example, on Gentoo, the package management sets
all kinds of environment variables, most importantly PREFIX,
LDFLAGS, CFLAGS and CXXFLAGS. It would be great if any
replacement system would also honor those (translated to D).
Even better if it could handle GDC and LDC2 directly (since
gdmd and ldmd2 don't do all the way).
LDFLAGS need translation from the default (GCC) syntax to DMD
and LDC2. I currently do this in a helper script and would not
depend on the build script doing that.

-- 
Marco
Apr 18 2016
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 18 April 2016 at 23:17:39 UTC, Marco Leise wrote:

 Am Mon, 18 Apr 2016 15:15:26 +0000
 schrieb Atila Neves <atila.neves gmail.com>:

 Just remember that makefiles are well integrated with other 
 systems (i.e. package managers) and there are some conventions, 
 about what certain environment variables mean. To name a few:

 MAKEOPTS=-j4  (parallel build with N processes)


This is make-dependent though. ninja and the reggae binary 
backend run with the number of cores by default.


 DC=/opt/ldc2-0.17/bin/ldc2  (Dlang compiler)


DC, CC, and CXX all work already.


 DC_FLAGS=-O5  (flags for that compiler)
 LDFLAGS=-L=-O1  (flags for the linker)
 PREFIX=/usr  (installation prefix)


These I need to add. Well, if there's enough of a use-case anyway.


 Dlang projects already have some make files that handle linker
 flag prefixes while others will pass them verbatim to gcc and
 some make files accept a DMD (ldmd2, gdmd, dmd) and others a
 DC (ldc2, gdc, dmd). The more things we can override, the
 better.


The phobos makefile only has DMD.


 As a practical example, on Gentoo, the package management sets
 all kinds of environment variables, most importantly PREFIX,
 LDFLAGS, CFLAGS and CXXFLAGS. It would be great if any
 replacement system would also honor those (translated to D).


I didn't even know CFLAGS was a default!


 Even better if it could handle GDC and LDC2 directly (since
 gdmd and ldmd2 don't do all the way).


Here it'd be better to improve gdmd and ldmd2, I think. In any 
case it's a leaky abstraction. I've never seen a CMake build that 
doesn't have ifs for what compiler is being used in order to 
choose the appropriate flags.


 LDFLAGS need translation from the default (GCC) syntax to DMD
 and LDC2. I currently do this in a helper script and would not
 depend on the build script doing that.


Why does it need translating? If you use the system linker 
directly, use LDFLAGS as is. If not, use whatever linker flags 
are appropriate for the compiler being used (-L-l... for 
instance).

Atila
Apr 19 2016
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 19 Apr 2016 17:02:43 +0000
schrieb Atila Neves <atila.neves gmail.com>:


 DC_FLAGS=-O5  (flags for that compiler)
 LDFLAGS=-L=-O1  (flags for the linker)
 PREFIX=/usr  (installation prefix)  


 
 These I need to add. Well, if there's enough of a use-case anyway.


It's just normal on Gentoo, to be able to set all the compiler
and linker flags and have atypical configurations for their
software. One guy with a security enhanced Linux and GCC
installation required -fPIC to be passed. This is easy when
the build system provides a facility for it as otherwise I
need to maintain patches.


 Dlang projects already have some make files that handle linker
 flag prefixes while others will pass them verbatim to gcc and
 some make files accept a DMD (ldmd2, gdmd, dmd) and others a
 DC (ldc2, gdc, dmd). The more things we can override, the
 better.  


 
 The phobos makefile only has DMD.


Well, it has the compiler that ships with the sources anyways.
Point taken.


 As a practical example, on Gentoo, the package management sets
 all kinds of environment variables, most importantly PREFIX,
 LDFLAGS, CFLAGS and CXXFLAGS. It would be great if any
 replacement system would also honor those (translated to D).  


 
 I didn't even know CFLAGS was a default!


It is for makefiles. These conventions that grew over time
make them convenient to the expert. But they are hardly
visible so easily missed when seeking to replace make files.
https://en.wikipedia.org/wiki/CFLAGS
On a typical Gentoo "ricer" system they'd include e.g.
  -O3 -flto -march=native


 Even better if it could handle GDC and LDC2 directly (since
 gdmd and ldmd2 don't do all the way).  


 
 Here it'd be better to improve gdmd and ldmd2, I think. In any 
 case it's a leaky abstraction. I've never seen a CMake build that 
 doesn't have ifs for what compiler is being used in order to 
 choose the appropriate flags.


Maybe you are right. I hope passing "native" arguments to gdc
and ldc2 through their wrappers works reliably and gdmd stops
reading dmd.conf.


 LDFLAGS need translation from the default (GCC) syntax to DMD
 and LDC2. I currently do this in a helper script and would not
 depend on the build script doing that.  


 
 Why does it need translating? If you use the system linker 
 directly, use LDFLAGS as is. If not, use whatever linker flags 
 are appropriate for the compiler being used (-L-l... for 
 instance).


I can only speak for Gentoo, where this env. variable is
assumed to be passed as flags to GCC in the linking step, so
the options are already prefixed with "-Wl,". For passing them
to the linker the prefixes needs to be stripped and for passing
them to ldc2 they need to be replaced by -L=. I have code in
place in the compiler selection logic for Dlang packages that
performs this translation. I don't know how exactly you plan
to handle this in your script. E.g. translate LDFLAGS form GCC
to DMD style, expect the user to set them in DMD style or
handle all three compilers by translating to each of them.

-- 
Marco
Apr 19 2016
prev sibling next sibling parent reply Piotrek <starpit tlen.pl> writes:
On Monday, 18 April 2016 at 15:15:26 UTC, Atila Neves wrote:

 Here's[1] another attempt at converting the Makefile for POSIX 
 systems to D using reggae[2].


...

 Destroy!

 Atila


I know you your intention was to keep it similar to makefile, but 
for me it looks unnecessarily complex.

What whould you say about different approach e.g.:

void configure()
{
     with (targets["linux shared library"])
     {
         inheritConfig("linux 32");
         sources = "lib/*.d";
         dependency ~= "objects";
         output = "somthing.so";
     }
}


Piotrek
Apr 22 2016
parent Atila Neves <atila.neves gmail.com> writes:
On Friday, 22 April 2016 at 17:00:21 UTC, Piotrek wrote:

 On Monday, 18 April 2016 at 15:15:26 UTC, Atila Neves wrote:

 Here's[1] another attempt at converting the Makefile for POSIX 
 systems to D using reggae[2].


 ...

 Destroy!

 Atila


 I know you your intention was to keep it similar to makefile, 
 but for me it looks unnecessarily complex.

 What whould you say about different approach e.g.:

 void configure()
 {
     with (targets["linux shared library"])
     {
         inheritConfig("linux 32");
         sources = "lib/*.d";
         dependency ~= "objects";
         output = "somthing.so";
     }
 }


 Piotrek


I think simpler would be better and that it could definitely be 
made simpler.

But none of that matters unless we can switch, and we won't be 
able to if I can't maintain it in parallel.

It's complex because posix.mak is complex.

Atila
Apr 23 2016
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Monday, 18 April 2016 at 15:15:26 UTC, Atila Neves wrote:

 Here's[1] another attempt at converting the Makefile for POSIX 
 systems to D using reggae[2]. I first tried my hand at writing 
 a parser for Makefiles using Pegged and machine translating to 
 D. I learned a lot about parsing and GNU make in the process 
 and also that:

 1. It was about two orders of magnitude more work than I 
 thought to support the subset of GNU make used by posix.mak
 2. The generated D code wasn't exactly pretty

 I concluded that worse is better and hand translated all of 
 posix.mak instead. It isn't as pretty as possible either, but 
 that's on purpose - since I'll have to mentally apply any 
 further diffs and hand translate, I tried to make it as close 
 as possible to the existing Makefile. That way the edits should 
 be obvious.

 AFAICT by testing manually, it does everything that the 
 original Makefile does, including all the different unit test 
 targets and documentation. Then again, I only tested on Linux. 
 I've been trying to bend the auto-tester to my will to make 
 sure all posix.mak systems are still green but I ran into 
 configuration issues.

 I'll carry on maintaining the D description as updates to 
 posix.mak happen for a year. If we haven't switched by then we 
 probably never will.

 To try it out yourself, get & build/run reggae from dub in this 
 branch at the root dir of phobos selecting the backend of 
 choice with `-b <backend>`. I suggest the binary one since one 
 of the points of this is to remove dependencies from other 
 build systems. Then run make/ninja/tup/./build with the same 
 targets as usual. If that doesn't work... open an issue on 
 github.

 Destroy!

 Atila


 [1] 
 https://github.com/atilaneves/phobos/blob/reggae/reggaefile.d
 [2] https://github.com/atilaneves/reggae


Phobos posix.mak in D:

https://github.com/atilaneves/phobos/blob/new_reggae/reggaefile.d

I've given up on making it look like the makefile for maintenance 
reasons and tried to write it "properly". At the end of the day, 
that phobos posix.mak just does a lot of stuff so even though I 
tried getting rid of cruft it's still rather large.

Notice however how there are no explicit listings of packages or 
converting them or anything like that. This isn't the best I can 
do, but it was a lot of work already so I thought I'd share.

This new attempt does everything posix.mak at that point in time 
did except for running unit tests per package. I think that's the 
test runner's job anyway. I also changed the individual unit test 
targets from e.g. std/array.test to std.array.test because, well, 
it makes a lot more sense.

Atila
Jun 29 2016