www.digitalmars.com         C & C++   DMDScript  

D.gnu - GDMD D port ready for alpha-testing

reply Johannes Pfau <nospam example.com> writes:
I've finally finished the D port of GDMD and pushed everything
including a detailed README to
https://github.com/D-Programming-GDC/GDMD/tree/dport

All parameters are tested for simple cases but in complex
combinations there could still be some unknown bugs.


I think the D port should already be much more useful than the perl
script though. It supports gdmd -main, gdmd -run and gdmd -lib as well
as most other options in DMD 2.066*. And it properly creates all output
folders which dmd-script sometimes doesn't do...

So I'd like to ship this gdmd with the binaries starting with the next
release.  Iain if you've got no objections I'd also like to merge this
into the GDMD master branch ASAP. We have to keep dmd-script in master
though for travis-ci.


* There are some DMD options which can't work the same way in GDC. For
  example treating deprecation as an error but all other warnings as
  warnings.
Mar 26 2016
next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 26 Mar 2016 11:09:10 +0100
schrieb Johannes Pfau <nospam example.com>:


 I've finally finished the D port of GDMD and pushed everything
 including a detailed README to
 https://github.com/D-Programming-GDC/GDMD/tree/dport
 
 All parameters are tested for simple cases but in complex
 combinations there could still be some unknown bugs.
 
 
 I think the D port should already be much more useful than the perl
 script though. It supports gdmd -main, gdmd -run and gdmd -lib as well
 as most other options in DMD 2.066*. And it properly creates all
 output folders which dmd-script sometimes doesn't do...
 
 So I'd like to ship this gdmd with the binaries starting with the next
 release.  Iain if you've got no objections I'd also like to merge this
 into the GDMD master branch ASAP. We have to keep dmd-script in master
 though for travis-ci.
 
 
 * There are some DMD options which can't work the same way in GDC. For
   example treating deprecation as an error but all other warnings as
   warnings.


I forgot to mention that the initial code was written by quickfur aka
H. S. Teoh. Thanks for that!
Mar 26 2016
prev sibling next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:

 I've finally finished the D port of GDMD and pushed everything 
 including a detailed README to 
 https://github.com/D-Programming-GDC/GDMD/tree/dport


Thanks for this.

I see that parsing response files isn't implemented yet. Response 
files are practically required when used with rdmd on Windows, 
because of the command line length limit.

std.process.escapeWindowsArgument is available on all platforms 
for this purpose:



(The response files use the Windows syntax, regardless of the 
host platform.)
Mar 26 2016
next sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 26 March 2016 at 10:35:44 UTC, Vladimir Panteleev 
wrote:

 On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:

 I've finally finished the D port of GDMD and pushed everything 
 including a detailed README to 
 https://github.com/D-Programming-GDC/GDMD/tree/dport


 Thanks for this.

 I see that parsing response files isn't implemented yet. 
 Response files are practically required when used with rdmd on 
 Windows, because of the command line length limit.

 std.process.escapeWindowsArgument is available on all platforms 
 for this purpose:



 (The response files use the Windows syntax, regardless of the 
 host platform.)


Erm, sorry, of course gdmd needs to parse, not generate response 
files. Ignore the above.
Mar 26 2016
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 26 Mar 2016 10:35:44 +0000
schrieb Vladimir Panteleev <thecybershadow.lists gmail.com>:


 On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:

 I've finally finished the D port of GDMD and pushed everything 
 including a detailed README to 
 https://github.com/D-Programming-GDC/GDMD/tree/dport  


 
 Thanks for this.
 
 I see that parsing response files isn't implemented yet. Response 
 files are practically required when used with rdmd on Windows, 
 because of the command line length limit.
 
 std.process.escapeWindowsArgument is available on all platforms 
 for this purpose:
 

 
 (The response files use the Windows syntax, regardless of the 
 host platform.)
 


I didn't realize response file parsing is that important for windows,
thanks for letting me know ;-) I guess there's no proper documentation
for response files or any test files?


There are some open questions:

* Are escape sequences allowed outside of quoted strings? 
* Which characters end a comment?
* How does this parse: foo\"bar
* Or this: foo"bar


BTW:
The DMD implementation is using the backend license, so I
certainly couldn't copy it. It tried to follow clean-room design
principles when reimplementing it (writing a spec first, then using the
spec to implement the new parser) and choose a completely different
parsing approach (parsing/unescaping separated, range based) so I hope
it's different enough to avoid copyright or licensing issues....

https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
It'd be great if you could have a look at the unittests and tell me if
the parsing is correct ;-)

Now I still have to generate a response file for passing arguments to
GDC. First have to check whether GCC even uses the windows escaping
rules though. Do you know how many characters can be passed without
response files?
Mar 26 2016
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
Thanks for working on this :)

On Saturday, 26 March 2016 at 22:37:53 UTC, Johannes Pfau wrote:

 I didn't realize response file parsing is that important for 
 windows, thanks for letting me know ;-) I guess there's no 
 proper documentation for response files or any test files?


It uses the CommandLineToArgvW syntax.

That's pretty much all there is to it.


 There are some open questions:

 * Are escape sequences allowed outside of quoted strings?


Apparently, though rdmd will never generate such response files.


 * Which characters end a comment?


There are no comments in response files (as far as I know).


 * How does this parse: foo\"bar


foo"bar


 * Or this: foo"bar


foobar

I suggest writing a small program to experiment, e.g.

void main(string[] args) { args.each!writeln(); }


 BTW:
 The DMD implementation is using the backend license, so I
 certainly couldn't copy it. It tried to follow clean-room design
 principles when reimplementing it (writing a spec first, then 
 using the
 spec to implement the new parser) and choose a completely 
 different
 parsing approach (parsing/unescaping separated, range based) so 
 I hope
 it's different enough to avoid copyright or licensing issues....


I'm not sure this approach is the best - I think unescaping and 
splitting should be done in one pass. Essentially, I think 
arguments are separated by an unescaped space.

If you have a Windows machine, you can verify it yourself by 
comparing the results with CommandLineToArgvW.


 https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
 It'd be great if you could have a look at the unittests and 
 tell me if
 the parsing is correct ;-)


The unit tests *look* OK, but I can't say for sure without having 
a way to verify them.

I suggest you have a look at the escaping code and tests from 
std.process. There should be no licensing conflict with Boost, I 
think. I went to great lengths to ensure that code is correct, 
there is even a test which brute-forces special character 
combinations and another that tries random strings to find bugs 
in the escaping code.

IIRC the simple regex that was in the Perl version of gdmd was 
also sufficient for parsing the response files generated by rdmd.


 Now I still have to generate a response file for passing 
 arguments to GDC. First have to check whether GCC even uses the 
 windows escaping rules though. Do you know how many characters 
 can be passed without response files?


rdmd uses a limit of 1024, but I think that's lower than the OS 
limit.
Mar 27 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 08:35:00 +0000
schrieb Vladimir Panteleev <thecybershadow.lists gmail.com>:


 Thanks for working on this :)
 
 On Saturday, 26 March 2016 at 22:37:53 UTC, Johannes Pfau wrote:

 I didn't realize response file parsing is that important for 
 windows, thanks for letting me know ;-) I guess there's no 
 proper documentation for response files or any test files?  


 
 It uses the CommandLineToArgvW syntax.
 
 That's pretty much all there is to it.
 

 There are some open questions:

 * Are escape sequences allowed outside of quoted strings?  


 
 Apparently, though rdmd will never generate such response files.
 

 * Which characters end a comment?  


 
 There are no comments in response files (as far as I know).
 [...]


Well, that's the problem: There's no specification for response files.
The DMD implementation certainly does support comments in response
files. LDMD simply reuses the DMD code because of these issues...

The entries in the response file are formatted according to
CommandLineToArgvW, this part is simple.

But the entries are separated by 'whitespace' which according to dmd is
'\r' '\n' '\t' ' ' and '\0'! This leads to many questions such as does

comment and 0x1a always ends the file...

DMD also only does the '\' escaping for double quotes, not for single
quotes. I think the code I've pushed should be as DMD-compatible as
possible.


 I'm not sure this approach is the best - I think unescaping and 
 splitting should be done in one pass. Essentially, I think 
 arguments are separated by an unescaped space.


Sure, escaping needs to be considered when splitting the string (to
differentiate between whitespace in quoted strings / whitespace and to
differentiate escaped quotes from normal quotes). But the code is nicer
if the splitting pass simply returns a slice to the original content
and does not assemble the unescaped strings on the fly.


 If you have a Windows machine, you can verify it yourself by 
 comparing the results with CommandLineToArgvW.
 

 https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
 It'd be great if you could have a look at the unittests and 
 tell me if
 the parsing is correct ;-)  


 
 The unit tests *look* OK, but I can't say for sure without having 
 a way to verify them.
 
 I suggest you have a look at the escaping code and tests from 
 std.process. There should be no licensing conflict with Boost, I 
 think. I went to great lengths to ensure that code is correct, 
 there is even a test which brute-forces special character 
 combinations and another that tries random strings to find bugs 
 in the escaping code.
 
 IIRC the simple regex that was in the Perl version of gdmd was 
 also sufficient for parsing the response files generated by rdmd.
 


Yeah, probably supporting the sane subset of response files actually
used is good enough. I tried to match the dmd implementation, so any
weird special feature works, but maybe that's not required.


 Now I still have to generate a response file for passing 
 arguments to GDC. First have to check whether GCC even uses the 
 windows escaping rules though. Do you know how many characters 
 can be passed without response files?  


 
 rdmd uses a limit of 1024, but I think that's lower than the OS 
 limit.
 


OK, thanks!
Mar 27 2016
parent Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 13:31:34 +0200
schrieb Johannes Pfau <nospam example.com>:


 Am Sun, 27 Mar 2016 08:35:00 +0000
 schrieb Vladimir Panteleev <thecybershadow.lists gmail.com>:
 
 

 I'm not sure this approach is the best - I think unescaping and 
 splitting should be done in one pass. Essentially, I think 
 arguments are separated by an unescaped space.  


 
 Sure, escaping needs to be considered when splitting the string (to
 differentiate between whitespace in quoted strings / whitespace and to
 differentiate escaped quotes from normal quotes). But the code is
 nicer if the splitting pass simply returns a slice to the original
 content and does not assemble the unescaped strings on the fly.
 


Thinking about this some more, it probably can't work in all cases. So
I've changed it to do unescaping on the fly.

I've also integrated the phobos unittests:
* My parser doesn't report "" as an empty argument. I could fix this,
  but then we don't use empty arguments, so why bother?


However, MS is completely insane:

`"C:\abc\" def" foo` => `C:\abc" def`, `foo`
`"C:\abc\"def" foo` => `C:\abc"def`, `foo`
`"C:\abc\\" def" foo` => `C:\abc\`, `def foo`
`"C:\abc\\\" def" foo` => `C:\abc\" def`, `foo`

`"C:\abc\\"def" foo` => `C:\abc\def foo`
The last one is not supported in the parser cause I don't want to guess
any further what arbitrary rules MS has made up...
Mar 27 2016
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:

 I've finally finished the D port of GDMD and pushed everything 
 including a detailed README to 
 https://github.com/D-Programming-GDC/GDMD/tree/dport

 All parameters are tested for simple cases but in complex 
 combinations there could still be some unknown bugs.


Good news. I'll follow the project.

Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after 
a renaming it was OK with a simple CE project. (linux x86_64 and 
latest stable gdc)
Mar 26 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 04:25:27 +0000
schrieb Basile B. <b2.temp gmx.com>:


 On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:

 I've finally finished the D port of GDMD and pushed everything 
 including a detailed README to 
 https://github.com/D-Programming-GDC/GDMD/tree/dport

 All parameters are tested for simple cases but in complex 
 combinations there could still be some unknown bugs.  


 
 Good news. I'll follow the project.
 
 Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after 
 a renaming it was OK with a simple CE project. (linux x86_64 and 
 latest stable gdc)


This is actually the first time I heard about gcc-ar...
But it seems using gcc-ar is indeed better:
http://manpages.ubuntu.com/manpages/trusty/man1/aarch64-linux-gnu-gcc-ar-4.7.1.html

Thanks for reporting this!
Mar 27 2016
next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 13:36:19 +0200
schrieb Johannes Pfau <nospam example.com>:


 Am Sun, 27 Mar 2016 04:25:27 +0000
 schrieb Basile B. <b2.temp gmx.com>:
 

 On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:  

 I've finally finished the D port of GDMD and pushed everything 
 including a detailed README to 
 https://github.com/D-Programming-GDC/GDMD/tree/dport

 All parameters are tested for simple cases but in complex 
 combinations there could still be some unknown bugs.    


 
 Good news. I'll follow the project.
 
 Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after 
 a renaming it was OK with a simple CE project. (linux x86_64 and 
 latest stable gdc)  


 
 This is actually the first time I heard about gcc-ar...
 But it seems using gcc-ar is indeed better:
 http://manpages.ubuntu.com/manpages/trusty/man1/aarch64-linux-gnu-gcc-ar-4.7.1.html
 
 Thanks for reporting this!


https://github.com/D-Programming-GDC/GDMD/commit/c2e6bd7d83fdec28b249b4f36eadb64e4206dd52

Could you please check if it's working now?
Mar 27 2016
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 13:36:19 +0200
schrieb Johannes Pfau <nospam example.com>:


 Am Sun, 27 Mar 2016 04:25:27 +0000
 schrieb Basile B. <b2.temp gmx.com>:
 

 On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:  

 I've finally finished the D port of GDMD and pushed everything 
 including a detailed README to 
 https://github.com/D-Programming-GDC/GDMD/tree/dport

 All parameters are tested for simple cases but in complex 
 combinations there could still be some unknown bugs.    


 
 Good news. I'll follow the project.
 
 Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after 
 a renaming it was OK with a simple CE project. (linux x86_64 and 
 latest stable gdc)  


 
 This is actually the first time I heard about gcc-ar...
 But it seems using gcc-ar is indeed better:
 http://manpages.ubuntu.com/manpages/trusty/man1/aarch64-linux-gnu-gcc-ar-4.7.1.html
 
 Thanks for reporting this!


Turns out gcc-ar will not work if binutils was built without
plugin support. So I've reverted that change and GDMD will have to use
ar instead.

There was a unrelated problem on windows where the '.exe' extension was
not appended when searching ar. This should be fixed. There's still the
problem that ar is only available as 'ar' with the windows binary
toolchains. So if you use gdmd -gdc=gdc it will find ar, but if you
use gdmd -gdc=x86_64-w64-mingw32-gdc it will search for
x86_64-w64-mingw32-ar and won't find it.
Mar 27 2016
parent Basile B. <b2.temp gmx.com> writes:
On Sunday, 27 March 2016 at 16:55:18 UTC, Johannes Pfau wrote:

 There was a unrelated problem on windows where the '.exe' 
 extension was not appended when searching ar. This should be 
 fixed. There's still the problem that ar is only available as 
 'ar' with the windows binary toolchains. So if you use gdmd 
 -gdc=gdc it will find ar, but if you use gdmd 
 -gdc=x86_64-w64-mingw32-gdc it will search for 
 x86_64-w64-mingw32-ar and won't find it.


Right, sorry for the misleading report. After adding the 
additional '-gdc=gdc' parameter gcc-ar is detected.
Mar 27 2016