D.gnu - GDMD D port ready for alpha-testing

• Johannes Pfau (16/16) Mar 26 2016 I've finally finished the D port of GDMD and pushed everything
• Johannes Pfau (4/26) Mar 26 2016 I forgot to mention that the initial code was written by quickfur aka
• Vladimir Panteleev (10/13) Mar 26 2016 Thanks for this.
• Vladimir Panteleev (4/17) Mar 26 2016 Erm, sorry, of course gdmd needs to parse, not generate response
• Johannes Pfau (24/43) Mar 26 2016 I didn't realize response file parsing is that important for windows,
• Vladimir Panteleev (27/53) Mar 27 2016 It uses the CommandLineToArgvW syntax.
• Johannes Pfau (23/76) Mar 27 2016 Well, that's the problem: There's no specification for response files.
• Johannes Pfau (15/29) Mar 27 2016 Thinking about this some more, it probably can't work in all cases. So
• Basile B. (5/10) Mar 26 2016 Good news. I'll follow the project.
• Johannes Pfau (6/19) Mar 27 2016 This is actually the first time I heard about gcc-ar...
• Johannes Pfau (4/26) Mar 27 2016 https://github.com/D-Programming-GDC/GDMD/commit/c2e6bd7d83fdec28b249b4f...
• Johannes Pfau (11/33) Mar 27 2016 Turns out gcc-ar will not work if binutils was built without
• Basile B. (3/10) Mar 27 2016 Right, sorry for the misleading report. After adding the
Johannes Pfau <nospam example.com> writes:
I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

All parameters are tested for simple cases but in complex
combinations there could still be some unknown bugs.

I think the D port should already be much more useful than the perl
script though. It supports gdmd -main, gdmd -run and gdmd -lib as well
as most other options in DMD 2.066*. And it properly creates all output
folders which dmd-script sometimes doesn't do...

So I'd like to ship this gdmd with the binaries starting with the next
release.  Iain if you've got no objections I'd also like to merge this
into the GDMD master branch ASAP. We have to keep dmd-script in master
though for travis-ci.

* There are some DMD options which can't work the same way in GDC. For
example treating deprecation as an error but all other warnings as
warnings.

Mar 26 2016
Johannes Pfau <nospam example.com> writes:
Am Sat, 26 Mar 2016 11:09:10 +0100
schrieb Johannes Pfau <nospam example.com>:

I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

All parameters are tested for simple cases but in complex
combinations there could still be some unknown bugs.

I think the D port should already be much more useful than the perl
script though. It supports gdmd -main, gdmd -run and gdmd -lib as well
as most other options in DMD 2.066*. And it properly creates all
output folders which dmd-script sometimes doesn't do...

So I'd like to ship this gdmd with the binaries starting with the next
release.  Iain if you've got no objections I'd also like to merge this
into the GDMD master branch ASAP. We have to keep dmd-script in master
though for travis-ci.

* There are some DMD options which can't work the same way in GDC. For
example treating deprecation as an error but all other warnings as
warnings.

I forgot to mention that the initial code was written by quickfur aka
H. S. Teoh. Thanks for that!

Mar 26 2016
On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

Thanks for this.

I see that parsing response files isn't implemented yet. Response
files are practically required when used with rdmd on Windows,
because of the command line length limit.

std.process.escapeWindowsArgument is available on all platforms
for this purpose:

http://dlang.org/phobos/std_process.html#.escapeWindowsArgument

(The response files use the Windows syntax, regardless of the
host platform.)

Mar 26 2016
On Saturday, 26 March 2016 at 10:35:44 UTC, Vladimir Panteleev
wrote:
On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

Thanks for this.

I see that parsing response files isn't implemented yet.
Response files are practically required when used with rdmd on
Windows, because of the command line length limit.

std.process.escapeWindowsArgument is available on all platforms
for this purpose:

http://dlang.org/phobos/std_process.html#.escapeWindowsArgument

(The response files use the Windows syntax, regardless of the
host platform.)

Erm, sorry, of course gdmd needs to parse, not generate response
files. Ignore the above.

Mar 26 2016
Johannes Pfau <nospam example.com> writes:
Am Sat, 26 Mar 2016 10:35:44 +0000

On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

Thanks for this.

I see that parsing response files isn't implemented yet. Response
files are practically required when used with rdmd on Windows,
because of the command line length limit.

std.process.escapeWindowsArgument is available on all platforms
for this purpose:

http://dlang.org/phobos/std_process.html#.escapeWindowsArgument

(The response files use the Windows syntax, regardless of the
host platform.)

I didn't realize response file parsing is that important for windows,
thanks for letting me know ;-) I guess there's no proper documentation
for response files or any test files?

There are some open questions:

* Are escape sequences allowed outside of quoted strings?
* Which characters end a comment?
* How does this parse: foo\"bar
* Or this: foo"bar

BTW:
The DMD implementation is using the backend license, so I
certainly couldn't copy it. It tried to follow clean-room design
principles when reimplementing it (writing a spec first, then using the
spec to implement the new parser) and choose a completely different
parsing approach (parsing/unescaping separated, range based) so I hope
it's different enough to avoid copyright or licensing issues....

https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
It'd be great if you could have a look at the unittests and tell me if
the parsing is correct ;-)

Now I still have to generate a response file for passing arguments to
GDC. First have to check whether GCC even uses the windows escaping
rules though. Do you know how many characters can be passed without
response files?

Mar 26 2016
Thanks for working on this :)

On Saturday, 26 March 2016 at 22:37:53 UTC, Johannes Pfau wrote:
I didn't realize response file parsing is that important for
windows, thanks for letting me know ;-) I guess there's no
proper documentation for response files or any test files?

It uses the CommandLineToArgvW syntax.

That's pretty much all there is to it.

There are some open questions:

* Are escape sequences allowed outside of quoted strings?

Apparently, though rdmd will never generate such response files.

* Which characters end a comment?

There are no comments in response files (as far as I know).

* How does this parse: foo\"bar

foo"bar

* Or this: foo"bar

foobar

I suggest writing a small program to experiment, e.g.

void main(string[] args) { args.each!writeln(); }

BTW:
The DMD implementation is using the backend license, so I
certainly couldn't copy it. It tried to follow clean-room design
principles when reimplementing it (writing a spec first, then
using the
spec to implement the new parser) and choose a completely
different
parsing approach (parsing/unescaping separated, range based) so
I hope
it's different enough to avoid copyright or licensing issues....

I'm not sure this approach is the best - I think unescaping and
splitting should be done in one pass. Essentially, I think
arguments are separated by an unescaped space.

If you have a Windows machine, you can verify it yourself by
comparing the results with CommandLineToArgvW.

https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
It'd be great if you could have a look at the unittests and
tell me if
the parsing is correct ;-)

The unit tests *look* OK, but I can't say for sure without having
a way to verify them.

I suggest you have a look at the escaping code and tests from
std.process. There should be no licensing conflict with Boost, I
think. I went to great lengths to ensure that code is correct,
there is even a test which brute-forces special character
combinations and another that tries random strings to find bugs
in the escaping code.

IIRC the simple regex that was in the Perl version of gdmd was
also sufficient for parsing the response files generated by rdmd.

Now I still have to generate a response file for passing
arguments to GDC. First have to check whether GCC even uses the
windows escaping rules though. Do you know how many characters
can be passed without response files?

rdmd uses a limit of 1024, but I think that's lower than the OS
limit.

Mar 27 2016
Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 08:35:00 +0000

Thanks for working on this :)

On Saturday, 26 March 2016 at 22:37:53 UTC, Johannes Pfau wrote:
I didn't realize response file parsing is that important for
windows, thanks for letting me know ;-) I guess there's no
proper documentation for response files or any test files?

It uses the CommandLineToArgvW syntax.

That's pretty much all there is to it.

There are some open questions:

* Are escape sequences allowed outside of quoted strings?

Apparently, though rdmd will never generate such response files.

* Which characters end a comment?

There are no comments in response files (as far as I know).
[...]

Well, that's the problem: There's no specification for response files.
The DMD implementation certainly does support comments in response
files. LDMD simply reuses the DMD code because of these issues...

The entries in the response file are formatted according to
CommandLineToArgvW, this part is simple.

But the entries are separated by 'whitespace' which according to dmd is
'\r' '\n' '\t' ' ' and '\0'! This leads to many questions such as does
'\0' end a comment? Additionally dmd treats '#' as the start of a
comment and 0x1a always ends the file...

DMD also only does the '\' escaping for double quotes, not for single
quotes. I think the code I've pushed should be as DMD-compatible as
possible.

I'm not sure this approach is the best - I think unescaping and
splitting should be done in one pass. Essentially, I think
arguments are separated by an unescaped space.

Sure, escaping needs to be considered when splitting the string (to
differentiate between whitespace in quoted strings / whitespace and to
differentiate escaped quotes from normal quotes). But the code is nicer
if the splitting pass simply returns a slice to the original content
and does not assemble the unescaped strings on the fly.

If you have a Windows machine, you can verify it yourself by
comparing the results with CommandLineToArgvW.

https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
It'd be great if you could have a look at the unittests and
tell me if
the parsing is correct ;-)

The unit tests *look* OK, but I can't say for sure without having
a way to verify them.

I suggest you have a look at the escaping code and tests from
std.process. There should be no licensing conflict with Boost, I
think. I went to great lengths to ensure that code is correct,
there is even a test which brute-forces special character
combinations and another that tries random strings to find bugs
in the escaping code.

IIRC the simple regex that was in the Perl version of gdmd was
also sufficient for parsing the response files generated by rdmd.

Yeah, probably supporting the sane subset of response files actually
used is good enough. I tried to match the dmd implementation, so any
weird special feature works, but maybe that's not required.

Now I still have to generate a response file for passing
arguments to GDC. First have to check whether GCC even uses the
windows escaping rules though. Do you know how many characters
can be passed without response files?

rdmd uses a limit of 1024, but I think that's lower than the OS
limit.

OK, thanks!

Mar 27 2016
Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 13:31:34 +0200
schrieb Johannes Pfau <nospam example.com>:

Am Sun, 27 Mar 2016 08:35:00 +0000

I'm not sure this approach is the best - I think unescaping and
splitting should be done in one pass. Essentially, I think
arguments are separated by an unescaped space.

Sure, escaping needs to be considered when splitting the string (to
differentiate between whitespace in quoted strings / whitespace and to
differentiate escaped quotes from normal quotes). But the code is
nicer if the splitting pass simply returns a slice to the original
content and does not assemble the unescaped strings on the fly.

I've changed it to do unescaping on the fly.

I've also integrated the phobos unittests:
* My parser doesn't report "" as an empty argument. I could fix this,
but then we don't use empty arguments, so why bother?

However, MS is completely insane:

"C:\abc\" def" foo => C:\abc" def, foo
"C:\abc\"def" foo => C:\abc"def, foo
"C:\abc\\" def" foo => C:\abc\, def foo
"C:\abc\\\" def" foo => C:\abc\" def, foo

"C:\abc\\"def" foo => C:\abc\def foo
The last one is not supported in the parser cause I don't want to guess
any further what arbitrary rules MS has made up...

Mar 27 2016
Basile B. <b2.temp gmx.com> writes:
On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

All parameters are tested for simple cases but in complex
combinations there could still be some unknown bugs.

Good news. I'll follow the project.

Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after
a renaming it was OK with a simple CE project. (linux x86_64 and
latest stable gdc)

Mar 26 2016
Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 04:25:27 +0000
schrieb Basile B. <b2.temp gmx.com>:

On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

All parameters are tested for simple cases but in complex
combinations there could still be some unknown bugs.

Good news. I'll follow the project.

Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after
a renaming it was OK with a simple CE project. (linux x86_64 and
latest stable gdc)

This is actually the first time I heard about gcc-ar...
But it seems using gcc-ar is indeed better:
http://manpages.ubuntu.com/manpages/trusty/man1/aarch64-linux-gnu-gcc-ar-4.7.1.html

Thanks for reporting this!

Mar 27 2016
Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 13:36:19 +0200
schrieb Johannes Pfau <nospam example.com>:

Am Sun, 27 Mar 2016 04:25:27 +0000
schrieb Basile B. <b2.temp gmx.com>:

On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

All parameters are tested for simple cases but in complex
combinations there could still be some unknown bugs.

Good news. I'll follow the project.

Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after
a renaming it was OK with a simple CE project. (linux x86_64 and
latest stable gdc)

This is actually the first time I heard about gcc-ar...
But it seems using gcc-ar is indeed better:
http://manpages.ubuntu.com/manpages/trusty/man1/aarch64-linux-gnu-gcc-ar-4.7.1.html

Thanks for reporting this!

Could you please check if it's working now?

Mar 27 2016
Johannes Pfau <nospam example.com> writes:
Am Sun, 27 Mar 2016 13:36:19 +0200
schrieb Johannes Pfau <nospam example.com>:

Am Sun, 27 Mar 2016 04:25:27 +0000
schrieb Basile B. <b2.temp gmx.com>:

On Saturday, 26 March 2016 at 10:09:10 UTC, Johannes Pfau wrote:
I've finally finished the D port of GDMD and pushed everything
https://github.com/D-Programming-GDC/GDMD/tree/dport

All parameters are tested for simple cases but in complex
combinations there could still be some unknown bugs.

Good news. I'll follow the project.

Small bug: gdmd is looking for "ar" instead of "gcc-ar", so after
a renaming it was OK with a simple CE project. (linux x86_64 and
latest stable gdc)

This is actually the first time I heard about gcc-ar...
But it seems using gcc-ar is indeed better:
http://manpages.ubuntu.com/manpages/trusty/man1/aarch64-linux-gnu-gcc-ar-4.7.1.html

Thanks for reporting this!

Turns out gcc-ar will not work if binutils was built without
plugin support. So I've reverted that change and GDMD will have to use

There was a unrelated problem on windows where the '.exe' extension was
not appended when searching ar. This should be fixed. There's still the
problem that ar is only available as 'ar' with the windows binary
toolchains. So if you use gdmd -gdc=gdc it will find ar, but if you
use gdmd -gdc=x86_64-w64-mingw32-gdc it will search for
x86_64-w64-mingw32-ar and won't find it.

Mar 27 2016
Basile B. <b2.temp gmx.com> writes:
On Sunday, 27 March 2016 at 16:55:18 UTC, Johannes Pfau wrote:
There was a unrelated problem on windows where the '.exe'
extension was not appended when searching ar. This should be
fixed. There's still the problem that ar is only available as
'ar' with the windows binary toolchains. So if you use gdmd
-gdc=gdc it will find ar, but if you use gdmd
-gdc=x86_64-w64-mingw32-gdc it will search for
x86_64-w64-mingw32-ar and won't find it.