www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DMD test suite not runnable on Debian/Linux

reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
I'm trying to track down an autotester failure in a DMD PR, and getting
very frustrated that the DMD test suite does not run on Debian/Linux
because of the whole PIE-by-default fiasco that necessitates compiling
everything with -fPIC.

(1) Currently, there is no way to customize how test/Makefile compiles
d_do_tests.d, so there is no way to insert -fPIC into the offending
compile line.

(2) d_do_tests itself runs compile commands with `-conf=`, so putting
-fPIC inside dmd.conf has no effect (where otherwise it does the right
thing when dmd is invoked "normally"). There's no obvious way to make
customize the compile command used for each test: I tried setting the
environment variable ARGS but that just messed up the test runner's
expected outputs. Tried to add -fPIC to the DMD environment variable but
that blew up in a different way.

So basically, I'm stuck up PIE creek without a paddle, and have no way
to locally test DMD changes without hogging more load on the autotester.

Why is something so simple so hard to do?!


T

-- 
Жил-был король когда-то, при нём блоха жила.
Jun 09 2017
next sibling parent reply Seb <seb wilzba.ch> writes:
On Friday, 9 June 2017 at 22:14:30 UTC, H. S. Teoh wrote:

 I'm trying to track down an autotester failure in a DMD PR, and 
 getting very frustrated that the DMD test suite does not run on 
 Debian/Linux because of the whole PIE-by-default fiasco that 
 necessitates compiling everything with -fPIC.

 (1) Currently, there is no way to customize how test/Makefile 
 compiles d_do_tests.d, so there is no way to insert -fPIC into 
 the offending compile line.


Once this PR is merged, it will at least use the default conf 
file:

https://github.com/dlang/dmd/pull/6870

(the config file is now in generated and this lead to nice errors 
on my machine)


 (2) d_do_tests itself runs compile commands with `-conf=`, so 
 putting -fPIC inside dmd.conf has no effect (where otherwise it 
 does the right thing when dmd is invoked "normally"). There's 
 no obvious way to make customize the compile command used for 
 each test: I tried setting the environment variable ARGS but 
 that just messed up the test runner's expected outputs. Tried 
 to add -fPIC to the DMD environment variable but that blew up 
 in a different way.


Do you mean DFLAGS with DMD environment variable?


 So basically, I'm stuck up PIE creek without a paddle, and have 
 no way to locally test DMD changes without hogging more load on 
 the autotester.


Don't worry about that. It's just a computer. As a temporary 
workaround, docker might be worth considering.


 Why is something so simple so hard to do?!


I think it's simply because not many people use the DMD test 
suite and those who do are used to the pain or have their 
custom-built tricks.
Also I guess that there are two groups of people:

A) wow that's complicated and frustrating -> move on to the next 
great thing in life
B) eh what the heck. I have seen worse...

(And not many in-between).
Jun 09 2017
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Jun 09, 2017 at 10:34:00PM +0000, Seb via Digitalmars-d wrote:

 On Friday, 9 June 2017 at 22:14:30 UTC, H. S. Teoh wrote:

 I'm trying to track down an autotester failure in a DMD PR, and
 getting very frustrated that the DMD test suite does not run on
 Debian/Linux because of the whole PIE-by-default fiasco that
 necessitates compiling everything with -fPIC.
 
 (1) Currently, there is no way to customize how test/Makefile
 compiles d_do_tests.d, so there is no way to insert -fPIC into the
 offending compile line.


 
 Once this PR is merged, it will at least use the default conf file:
 
 https://github.com/dlang/dmd/pull/6870
 
 (the config file is now in generated and this lead to nice errors on
 my machine)


But you see, that's the problem. What happens if the auto-generated
config file needs further adjustment?



 (2) d_do_tests itself runs compile commands with `-conf=`, so
 putting -fPIC inside dmd.conf has no effect (where otherwise it does
 the right thing when dmd is invoked "normally"). There's no obvious
 way to make customize the compile command used for each test: I
 tried setting the environment variable ARGS but that just messed up
 the test runner's expected outputs. Tried to add -fPIC to the DMD
 environment variable but that blew up in a different way.


 
 Do you mean DFLAGS with DMD environment variable?


There is no DFLAGS defined in d_do_tests.d.  The only configurable
parameter I can see is the DMD variable, so I tried to bluff d_do_tests
by setting DMD="dmd -fPIC", but that also didn't work, maybe because it
doesn't go through shell interpolation so it tries to find a
non-existent executable named "dmd -fPIC".

Anyway, I eventually went back to my trusty old anti-makefile
workaround: create a wrapper shell script that does this:


	/path/to/dmd -fPIC -I/path/to/druntime/import -I/path/to/phobos "$ "

(Yes, the -I's are necessary, I don't know why and don't really want to
at this point.)

Save this to /path/to/dmdwrapper, and then run make with
DMD=/path/to/dmdwrapper, and lo and behold, finally the miserable test
suite is runnable.

That was 2 hours' worth of frustration just to get the testsuite
running.  I dunno about you, but being unable to run the dmd testsuite
seems to me to be a big turnoff for potential dmd contributors. I
surmise not every potential dmd contributor is as persistent as I am.
:-D



 So basically, I'm stuck up PIE creek without a paddle, and have no
 way to locally test DMD changes without hogging more load on the
 autotester.


 
 Don't worry about that. It's just a computer. As a temporary
 workaround, docker might be worth considering.


Wow.  Way to bring out the shotgun to kill an ant!  I think installing
docker would be total overkill for something that really ought to work
out-of-the-box...!



 Why is something so simple so hard to do?!


 
 I think it's simply because not many people use the DMD test suite and
 those who do are used to the pain or have their custom-built tricks.
 Also I guess that there are two groups of people:
 
 A) wow that's complicated and frustrating -> move on to the next great
 thing in life B) eh what the heck. I have seen worse...
 
 (And not many in-between).


Ha, so now I've invented my own custom-built trick to make it work, and
so that's no help to the next would-be dmd contributor that comes along
and can't figure out how to make it work. :-/

//

On a tangential note, now that I finally got the miserable test suite
running, I've managed to figure out that the problem in my dmd PR was
caused by src/ddmd/globals.h not being auto-generated from
src/ddmd/globals.d, so when I added new fields to the Global struct, the
C++ code that use globals.h with the old definition crashed. I suppose
this is just a relic from the pre-self-hosting days, but it would be
nice if these last .h files were gotten rid of in the near future.

That, or we should (ab)use the .di generation facility or some sed
script to generate these .h files, so that they won't go out-of-sync
inadvertently.


T

-- 
LINUX = Lousy Interface for Nefarious Unix Xenophobes.
Jun 09 2017
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
H. S. Teoh wrote:


 On a tangential note, now that I finally got the miserable test suite
 running, I've managed to figure out that the problem in my dmd PR was
 caused by src/ddmd/globals.h not being auto-generated from
 src/ddmd/globals.d, so when I added new fields to the Global struct, the
 C++ code that use globals.h with the old definition crashed. I suppose
 this is just a relic from the pre-self-hosting days, but it would be
 nice if these last .h files were gotten rid of in the near future.

 That, or we should (ab)use the .di generation facility or some sed
 script to generate these .h files, so that they won't go out-of-sync
 inadvertently.


we can use UDAs to do that! just make the generator import the relevand dmd 
parts, and then use compile-time introspection to build tables, so 
generator can just write 'em to file in runtime. ;-)
Jun 09 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Jun 10, 2017 at 02:27:13AM +0300, ketmar via Digitalmars-d wrote:

 H. S. Teoh wrote:
 

 On a tangential note, now that I finally got the miserable test
 suite running, I've managed to figure out that the problem in my dmd
 PR was caused by src/ddmd/globals.h not being auto-generated from
 src/ddmd/globals.d, so when I added new fields to the Global struct,
 the C++ code that use globals.h with the old definition crashed. I
 suppose this is just a relic from the pre-self-hosting days, but it
 would be nice if these last .h files were gotten rid of in the near
 future.
 
 That, or we should (ab)use the .di generation facility or some sed
 script to generate these .h files, so that they won't go out-of-sync
 inadvertently.


 
 we can use UDAs to do that! just make the generator import the
 relevand dmd parts, and then use compile-time introspection to build
 tables, so generator can just write 'em to file in runtime. ;-)


Sure!  But ... uhm... what has UDAs gotta do with that? A straight
`foreach (memb; __traits(allMembers, T))` should do the trick.


T

-- 
A mathematician is a device for turning coffee into theorems. -- P. Erdos
Jun 09 2017
next sibling parent reply Seb <seb wilzba.ch> writes:
On Saturday, 10 June 2017 at 00:16:13 UTC, H. S. Teoh wrote:

 On Sat, Jun 10, 2017 at 02:27:13AM +0300, ketmar via 
 Digitalmars-d wrote:

 H. S. Teoh wrote:
 

 [...]


 
 we can use UDAs to do that! just make the generator import the 
 relevand dmd parts, and then use compile-time introspection to 
 build tables, so generator can just write 'em to file in 
 runtime. ;-)


 Sure!  But ... uhm... what has UDAs gotta do with that? A 
 straight
 `foreach (memb; __traits(allMembers, T))` should do the trick.


 T


Are you aware of this excellent PR?

https://github.com/dlang/dmd/pull/5082
Jun 09 2017
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Jun 10, 2017 at 12:33:39AM +0000, Seb via Digitalmars-d wrote:

 On Saturday, 10 June 2017 at 00:16:13 UTC, H. S. Teoh wrote:

 On Sat, Jun 10, 2017 at 02:27:13AM +0300, ketmar via Digitalmars-d
 wrote:

 H. S. Teoh wrote:
 

 [...]


 
 we can use UDAs to do that! just make the generator import the
 relevand dmd parts, and then use compile-time introspection to build
 tables, so generator can just write 'em to file in runtime. ;-)


 
 Sure!  But ... uhm... what has UDAs gotta do with that? A straight
 `foreach (memb; __traits(allMembers, T))` should do the trick.
 
 
 T


 
 Are you aware of this excellent PR?
 
 https://github.com/dlang/dmd/pull/5082


Whoa. This is exactly what I've been looking for!


T

-- 
The best way to destroy a cause is to defend it poorly.
Jun 09 2017
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
H. S. Teoh wrote:


 On Sat, Jun 10, 2017 at 02:27:13AM +0300, ketmar via Digitalmars-d wrote:

 H. S. Teoh wrote:
 

 On a tangential note, now that I finally got the miserable test
 suite running, I've managed to figure out that the problem in my dmd
 PR was caused by src/ddmd/globals.h not being auto-generated from
 src/ddmd/globals.d, so when I added new fields to the Global struct,
 the C++ code that use globals.h with the old definition crashed. I
 suppose this is just a relic from the pre-self-hosting days, but it
 would be nice if these last .h files were gotten rid of in the near
 future.
 That, or we should (ab)use the .di generation facility or some sed
 script to generate these .h files, so that they won't go out-of-sync
 inadvertently.


 we can use UDAs to do that! just make the generator import the
 relevand dmd parts, and then use compile-time introspection to build
 tables, so generator can just write 'em to file in runtime. ;-)


 Sure!  But ... uhm... what has UDAs gotta do with that? A straight
 `foreach (memb; __traits(allMembers, T))` should do the trick.


i thought about situations where you may need to exclude something from 
generated header... but this surely has no sense, as one cannot exclude 
fields or virtual methods from generated classes, so `extern(C++)` is 
surely enough. well, i like to overengineer! ;-)
Jun 09 2017
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-06-10 00:34, Seb wrote:


 Don't worry about that. It's just a computer. As a temporary workaround,
 docker might be worth considering.


Wow. Just modify the Makefile.

-- 
/Jacob Carlborg
Jun 10 2017
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
it's even worser than that: unittests ultimatively fails with binutils 2.28 
(at least on x86 without PIC), so i was forced to downgrade binutils to 2.26.
Jun 09 2017