www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Subtle bug in ddox vs. ddoc macro handling

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
So, I've spent the better part of this morning tracing what seems to be 
a subtle bug in ddox' macro treatment. I defined these macros:

IGNORESECOND = [$1]
DOLLARZERO = dzbegin $0 dzend
TEST = before $(IGNORESECOND $(DOLLARZERO one, two)) after

I inserted $(TEST) in a Phobos module. When processing the module with 
ddoc, the macro expands to:

before [dzbegin one, two dzend] after

wherease with ddox it expands to:

before [dzbegin one] after

i.e. the comma "escapes" the confines of the macro DOLLARZERO.

The "right" behavior is ddoc's, for several reasons. Is there a distinct 
macro engine powering ddox generation? Who is maintaining that?


Thanks,

Andrei
Sep 16 2016
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 16 September 2016 at 18:40:03 UTC, Andrei Alexandrescu 
wrote:

 So, I've spent the better part of this morning tracing what 
 seems to be a subtle bug in ddox' macro treatment. I defined 
 these macros:

 IGNORESECOND = [$1]
 DOLLARZERO = dzbegin $0 dzend
 TEST = before $(IGNORESECOND $(DOLLARZERO one, two)) after

 I inserted $(TEST) in a Phobos module. When processing the 
 module with ddoc, the macro expands to:

 before [dzbegin one, two dzend] after

 wherease with ddox it expands to:

 before [dzbegin one] after

 i.e. the comma "escapes" the confines of the macro DOLLARZERO.

 The "right" behavior is ddoc's, for several reasons. Is there a 
 distinct macro engine powering ddox generation? Who is 
 maintaining that?


 Thanks,

 Andrei


I belive ddox is maintained by soenke ludwig
Sep 16 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/16/2016 03:11 PM, Stefan Koch wrote:

 I belive ddox is maintained by soenke ludwig


Thx, he's cc'd. Workaround for the time being: 
https://github.com/dlang/dlang.org/pull/1475, could someone please 
review. Thanks!

Andrei
Sep 16 2016
prev sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
There is indeed a separate macro engine [1] contained in the ddox repo. 
Neither DDMD, nor libddoc existed at the time and DMD's implementation 
was written in, from a D POV, very alien C style, with no suitable 
structure to extract high level information, so instead of porting it, I 
instead tried to implement a processor from the spec [1]. However, it 
turned out that the spec is not sufficient at all, so a lot of changes 
were necessary over time to work out those small differences. I'd like 
to improve the docs, but I'm still not confident that my 
implementation's behavior is always equivalent (as this issue shows).

I've opened a GitHub ticket now [2] and will have a look at it shortly.

[1]: https://github.com/rejectedsoftware/ddox/blob/master/source/ddox/ddoc.d
[2]: http://dlang.org/spec/ddoc.html#phases_of_processing
[3]: https://github.com/rejectedsoftware/ddox/issues/130
Sep 16 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/16/2016 11:53 PM, Sönke Ludwig wrote:

 There is indeed a separate macro engine [1] contained in the ddox repo. Neither
 DDMD, nor libddoc existed at the time and DMD's implementation was written in,
 from a D POV, very alien C style, with no suitable structure to extract high
 level information, so instead of porting it, I instead tried to implement a
 processor from the spec [1]. However, it turned out that the spec is not
 sufficient at all, so a lot of changes were necessary over time to work out
 those small differences. I'd like to improve the docs, but I'm still not
 confident that my implementation's behavior is always equivalent (as this issue
 shows).


I've factored out the Ddoc macro code into a separate component:

https://digitalmars.com/sargon/textmac.html
Sep 17 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/17/16 5:29 AM, Walter Bright wrote:

 On 9/16/2016 11:53 PM, Sönke Ludwig wrote:

 There is indeed a separate macro engine [1] contained in the ddox
 repo. Neither
 DDMD, nor libddoc existed at the time and DMD's implementation was
 written in,
 from a D POV, very alien C style, with no suitable structure to
 extract high
 level information, so instead of porting it, I instead tried to
 implement a
 processor from the spec [1]. However, it turned out that the spec is not
 sufficient at all, so a lot of changes were necessary over time to
 work out
 those small differences. I'd like to improve the docs, but I'm still not
 confident that my implementation's behavior is always equivalent (as
 this issue
 shows).


 I've factored out the Ddoc macro code into a separate component:

 https://digitalmars.com/sargon/textmac.html


Cool. The way the whole establishment should work is that all surface 
syntax rewrites itself into macros, and expand() is invoked as the last 
step. -- Andrei
Sep 17 2016
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/17/16 2:53 AM, Sönke Ludwig wrote:

 There is indeed a separate macro engine [1] contained in the ddox repo.
 Neither DDMD, nor libddoc existed at the time and DMD's implementation
 was written in, from a D POV, very alien C style, with no suitable
 structure to extract high level information, so instead of porting it, I
 instead tried to implement a processor from the spec [1]. However, it
 turned out that the spec is not sufficient at all, so a lot of changes
 were necessary over time to work out those small differences. I'd like
 to improve the docs, but I'm still not confident that my
 implementation's behavior is always equivalent (as this issue shows).

 I've opened a GitHub ticket now [2] and will have a look at it shortly.

 [1]:
 https://github.com/rejectedsoftware/ddox/blob/master/source/ddox/ddoc.d
 [2]: http://dlang.org/spec/ddoc.html#phases_of_processing
 [3]: https://github.com/rejectedsoftware/ddox/issues/130


Cool beans, thanks. Do you think we should unify the implementations? 
Perhaps by moving your code (assuming we get semantic convergence) into 
dmd so we get the benefit of hyphenation? Say e.g. we want to add 
markdown input, should we continue with two codebases or unify? Thanks! 
-- Andrei
Sep 17 2016