www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Cross referencing in Ddoc

reply Jacob Carlborg <doob me.com> writes:
If nothing has happened recently the current situation of cross 
referencing in Ddoc sucks. What's currently being used in the Phobos 
documentation is the XREF, CXREF and ECXREF ddoc macros. These macros 
take two arguments, append "std", "core" or "etc" and form a link of the 
arguments. The problem with this is that it doesn't work so good for 
referring to symbols in a deeper package hierarchy.

Example, adding a reference to std.path.buildPath works fine but symbols 
that are any deeper than that will not work very well, like 
std.digest.digest.toHexString.

$(XREF path, buildPath)

The above will generate a link to std_path.html#buildPath with the text 
"std.path.buildPath". If we want to create a link to 
std.digest.digest.toHexString we get in to some problems, XREF doesn't 
accept a variable amount of arguments, the following will not work:

$(XREF digest, digest, toHexString)

What will work is this:

$(XREF digest_digest, toHexString)

The above will properly create a link to 
std_digest_digest.html#toHexString but that relies on the fact that DMD 
replaces dots with underscores in module names when generating 
documentation. But the text of the link will be 
"std.digest_digest.toHexString" which doesn't look very pretty.

The XREF macro also hard codes the root package, making it only work for 
"std".

I think we need a better solution for creating cross referencing links 
in ddoc. Here are a couple of ideas:

A. Automatic cross reference

Automatically create links for all matching symbols in the current 
scope. That means to create a link to a symbol in the current scope it's 
enough to mention it's name in the documentation. To refer to a symbol 
in another module the fully qualified name is used. Examples:

"Read and validates (using $(XREF utf, validate)) a text file."

Would become:

"Read and validates std.utf.validate a text file."

And

"The default $(XREF _algorithm, move) performs a potentially"

Would become:

"The default move performs a potentially"

Advantages:

* Everything happens automatically, the developer doesn't have to do 
anything special to add cross referencing

Disadvantages:

* It might create links where it's not wanted
* Requires support from the compiler

B. REF macro

A special macro that the compiler knows about. Pass the symbol name 
(local or fully qualified) to the macro and the compiler will generate 
the link. Examples:

"Read and validates (using $(XREF utf, validate)) a text file."

Would become:

"Read and validates $(REF std.utf.validate) a text file."

And

"The default $(XREF _algorithm, move) performs a potentially"

Would become:

"The default $(REF move) performs a potentially"

Advantages:

* No links are created where it's not wanted

Disadvantages:

* The developer need to manually add macros, still better than today though
* Requires support from the compiler

C. A combination of A and B

By default ddoc behaves as option B, but in See_Also sections ddoc 
behaves as option A.

Advantages:

* No links are created where it's not wanted

Disadvantages:

* The developer need to manually add macros but hopefully not in as many 
places as with option B
* Requires support from the compiler

What do you think?

-- 
/Jacob Carlborg
Dec 29 2013
next sibling parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 29.12.2013 18:38, schrieb Jacob Carlborg:

 A. Automatic cross reference


This is done for the DDOX based docs that were supposed to end up on the
home page at some point:

http://forum.dlang.org/thread/l9poef$210u$1 digitalmars.com
http://vibed.org/temp/dlang.org/library/index.html


 * It might create links where it's not wanted


The DDOC specs say that preceding an identifier with an underscore
avoids it getting highlighted. DDOX currently uses the same indicator to
avoid it getting a link.

See the "Emphasis" section in http://dlang.org/ddoc.html
Dec 29 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-29 19:35, Sönke Ludwig wrote:


 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:

 http://forum.dlang.org/thread/l9poef$210u$1 digitalmars.com
 http://vibed.org/temp/dlang.org/library/index.html


Right, forgot about that. When _are_ DDOX becoming the official 
documentation format. I really don't want to manually add cross referencing.


 The DDOC specs say that preceding an identifier with an underscore
 avoids it getting highlighted. DDOX currently uses the same indicator to
 avoid it getting a link.

 See the "Emphasis" section in http://dlang.org/ddoc.html


I see.

-- 
/Jacob Carlborg
Dec 29 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-29 19:35, Sönke Ludwig wrote:


 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:


BTW, although DDOX seems to be the best solution currently. It still 
feels like a workaround for something that should be fixed in the compiler.

-- 
/Jacob Carlborg
Dec 30 2013
parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 30.12.2013 11:24, schrieb Jacob Carlborg:

 On 2013-12-29 19:35, Sönke Ludwig wrote:
 

 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:


 
 BTW, although DDOX seems to be the best solution currently. It still
 feels like a workaround for something that should be fixed in the compiler.
 


Although I think that it is more important to have a well defined
documentation format that can be processed by external tools, it would
indeed be much nicer if for example DMD could automatically emit $(XREF
...) or similar for recognized symbol names. Another thing that would
sometimes be nice is to group functions by category (even though most of
the time using different modules is arguably the better choice). Using
custom HTML macros is sub optimal for many reasons and some kind of
little DDOC syntax similar to sections could do wonders there.

All in all, it's good to have a built-in solution available (and
especially important to facilitate a standard documentation format), but
many things that a separate tool can do (global indexes, search, custom
page structure, per declaration pages, etc.) would IMHO be overkill to
have in the compiler. It's still a compiler after all.
Dec 30 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 12:39, Sönke Ludwig wrote:


 Although I think that it is more important to have a well defined
 documentation format that can be processed by external tools, it would
 indeed be much nicer if for example DMD could automatically emit $(XREF
 ...) or similar for recognized symbol names. Another thing that would
 sometimes be nice is to group functions by category (even though most of
 the time using different modules is arguably the better choice). Using
 custom HTML macros is sub optimal for many reasons and some kind of
 little DDOC syntax similar to sections could do wonders there.

 All in all, it's good to have a built-in solution available (and
 especially important to facilitate a standard documentation format), but
 many things that a separate tool can do (global indexes, search, custom
 page structure, per declaration pages, etc.) would IMHO be overkill to
 have in the compiler. It's still a compiler after all.


Sure, it always leads back to the same problem: have the compiler 
available as a library. I guess the JSON output is some kind of middle 
ground.

-- 
/Jacob Carlborg
Dec 30 2013
parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 30.12.2013 13:47, schrieb Jacob Carlborg:

 On 2013-12-30 12:39, Sönke Ludwig wrote:
 

 Although I think that it is more important to have a well defined
 documentation format that can be processed by external tools, it would
 indeed be much nicer if for example DMD could automatically emit $(XREF
 ...) or similar for recognized symbol names. Another thing that would
 sometimes be nice is to group functions by category (even though most of
 the time using different modules is arguably the better choice). Using
 custom HTML macros is sub optimal for many reasons and some kind of
 little DDOC syntax similar to sections could do wonders there.

 All in all, it's good to have a built-in solution available (and
 especially important to facilitate a standard documentation format), but
 many things that a separate tool can do (global indexes, search, custom
 page structure, per declaration pages, etc.) would IMHO be overkill to
 have in the compiler. It's still a compiler after all.


 
 Sure, it always leads back to the same problem: have the compiler
 available as a library. I guess the JSON output is some kind of middle
 ground.
 


Absolutely! I hope at least the lexer will find it's way into Phobos
soon. A lot of potential is effectively sleeping there.
Dec 30 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/30/13 4:55 AM, Sönke Ludwig wrote:

 Am 30.12.2013 13:47, schrieb Jacob Carlborg:

 On 2013-12-30 12:39, Sönke Ludwig wrote:


 Although I think that it is more important to have a well defined
 documentation format that can be processed by external tools, it would
 indeed be much nicer if for example DMD could automatically emit $(XREF
 ...) or similar for recognized symbol names. Another thing that would
 sometimes be nice is to group functions by category (even though most of
 the time using different modules is arguably the better choice). Using
 custom HTML macros is sub optimal for many reasons and some kind of
 little DDOC syntax similar to sections could do wonders there.

 All in all, it's good to have a built-in solution available (and
 especially important to facilitate a standard documentation format), but
 many things that a separate tool can do (global indexes, search, custom
 page structure, per declaration pages, etc.) would IMHO be overkill to
 have in the compiler. It's still a compiler after all.


 Sure, it always leads back to the same problem: have the compiler
 available as a library. I guess the JSON output is some kind of middle
 ground.


 Absolutely! I hope at least the lexer will find it's way into Phobos
 soon. A lot of potential is effectively sleeping there.


Yah, https://github.com/Hackerpilot/lexer-work/blob/master/lexer.d is 
great! More pre-review discussion?

 Brian Schott: would be awesome to encode a couple more lexer examples, 
such as HTML (recognition of all tags and character entities - there are 
comprehensive lists on the net) and C++?

Andrei
Dec 30 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/30/13 3:39 AM, Sönke Ludwig wrote:

 Am 30.12.2013 11:24, schrieb Jacob Carlborg:

 On 2013-12-29 19:35, Sönke Ludwig wrote:


 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:


 BTW, although DDOX seems to be the best solution currently. It still
 feels like a workaround for something that should be fixed in the compiler.


 Although I think that it is more important to have a well defined
 documentation format that can be processed by external tools, it would
 indeed be much nicer if for example DMD could automatically emit $(XREF
 ...) or similar for recognized symbol names.


I'm not sure automatic cross-references are a good thing. Too many 
litter the document with links, and catch casual uses of words that 
happen to be indexable (consider e.g. "in" would be cross-referenced 
automatically).


Andrei
Dec 30 2013
parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 30.12.2013 16:57, schrieb Andrei Alexandrescu:

 On 12/30/13 3:39 AM, Sönke Ludwig wrote:

 Am 30.12.2013 11:24, schrieb Jacob Carlborg:

 On 2013-12-29 19:35, Sönke Ludwig wrote:


 This is done for the DDOX based docs that were supposed to end up on
 the
 home page at some point:


 BTW, although DDOX seems to be the best solution currently. It still
 feels like a workaround for something that should be fixed in the
 compiler.


 Although I think that it is more important to have a well defined
 documentation format that can be processed by external tools, it would
 indeed be much nicer if for example DMD could automatically emit $(XREF
 ...) or similar for recognized symbol names.


 
 I'm not sure automatic cross-references are a good thing. Too many
 litter the document with links, and catch casual uses of words that
 happen to be indexable (consider e.g. "in" would be cross-referenced
 automatically).
 
 
 Andrei
 
 


In my experience it is far more annoying to have the source files
littered with $(MACROS) instead of putting a few _underscores to avoid
bogus links (not to mention that most of the time this results in no
links at all due to laziness). In some modules it may happen more
frequently, but in general this seems to be rare enough for not really
having to worry about it - at least in the examples that I've seen up to
now.

Also remember that you often have to escape such words anyway, because
they would otherwise still be highlighted as D code even without the
cross references.
Dec 30 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/30/13 8:10 AM, Sönke Ludwig wrote:

 In my experience it is far more annoying to have the source files
 littered with $(MACROS) instead of putting a few _underscores to avoid
 bogus links (not to mention that most of the time this results in no
 links at all due to laziness).


Oh the default is to link and then disable manually? That's still 
manual, just the default is different :o). I can work with that.

Andrei
Dec 30 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:

 Oh the default is to link and then disable manually? That's still manual, just
 the default is different :o). I can work with that.


As I mentioned elsewhere, I dislike the requirement to use the inverse feature. 
Consider maintenance - I add a symbol, and then have to grep the rest of the 
module for any inadvertent collisions? Features should be positive ones, not 
negative.
Dec 30 2013
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 30.12.2013 22:00, schrieb Walter Bright:

 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:

 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.


 
 As I mentioned elsewhere, I dislike the requirement to use the inverse
 feature. Consider maintenance - I add a symbol, and then have to grep
 the rest of the module for any inadvertent collisions? Features should
 be positive ones, not negative.
 


I agree the reverse would be better, but that's what is already in DDOC
now. Also, quoting http://dlang.org/ddoc.html:


 Identifiers in documentation comments that are function parameters or


are names that are in scope at the associated declaration are emphasized
in the output.

So the same problem is already reality - you already have to go through
the documentation to see if there are any mis-highlighted words.
Dec 30 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 1:25 PM, Sönke Ludwig wrote:

 Identifiers in documentation comments that are function parameters or
 are names that are in scope at the associated declaration are emphasized
 in the output.

 So the same problem is already reality - you already have to go through
 the documentation to see if there are any mis-highlighted words.


1. having a bad idea is not justification for doing it again :-)

2. the requirement that the name be "in scope" does a pretty good job of 
reducing the problem. But cross references, are by definition CROSS references, 
i.e. they cross scopes. The whole point of them is they cross scopes, which 
means this becomes a much bigger issue.
Dec 30 2013
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 30.12.2013 23:13, schrieb Walter Bright:

 On 12/30/2013 1:25 PM, Sönke Ludwig wrote:

 Identifiers in documentation comments that are function parameters or
 are names that are in scope at the associated declaration are emphasized
 in the output.

 So the same problem is already reality - you already have to go through
 the documentation to see if there are any mis-highlighted words.


 
 1. having a bad idea is not justification for doing it again :-)


Of course not, but my point was that the mistake has already been made,
so we may as well make the best out of it... not really doing it again
in that sense.

But if the decision is that in fact this was a mistake, it should get
removed and replaced by a proper solution sooner rather than later (it's
not that bad to lose some highlighting and Phobos uses $(D ...) anyway,
so just removing it wouldn't hurt much). And in that regard I'd like to
point to "D's goals for embedded documentation"
<http://dlang.org/ddoc.htm>. I very much agree with them, but the
current reality looks like the opposite - lots of macro-pseudo-HTML
markup, very difficult to read unprocessed in many places. So something

marked symbols would be a *really* nice to have.


 
 2. the requirement that the name be "in scope" does a pretty good job of
 reducing the problem. But cross references, are by definition CROSS
 references, i.e. they cross scopes. The whole point of them is they
 cross scopes, which means this becomes a much bigger issue.
 


This is how I see it: single identifiers are only searched in the
current scope and in parent scopes of the same module. Cross references
to foreign scopes need to use qualified names (either ClassName.member
or pack.mod.Symbol). This avoids any inter-module issues and also a lot
of intra-module issues. Symbols in code blocks on the other hand can be
cross referenced completely automatically according to the usual D
lookup rules (including imports).
Dec 30 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 3:13 PM, Sönke Ludwig wrote:

 Am 30.12.2013 23:13, schrieb Walter Bright:

 On 12/30/2013 1:25 PM, Sönke Ludwig wrote:

 Identifiers in documentation comments that are function parameters or
 are names that are in scope at the associated declaration are emphasized
 in the output.

 So the same problem is already reality - you already have to go through
 the documentation to see if there are any mis-highlighted words.


 1. having a bad idea is not justification for doing it again :-)


 Of course not, but my point was that the mistake has already been made,
 so we may as well make the best out of it... not really doing it again
 in that sense.

 But if the decision is that in fact this was a mistake, it should get
 removed and replaced by a proper solution sooner rather than later (it's
 not that bad to lose some highlighting and Phobos uses $(D ...) anyway,
 so just removing it wouldn't hurt much).


Ddoc is very heavily used. I don't believe we can make such silent, breaking 
changes.



 And in that regard I'd like to
 point to "D's goals for embedded documentation"
 <http://dlang.org/ddoc.htm>. I very much agree with them, but the
 current reality looks like the opposite - lots of macro-pseudo-HTML
 markup, very difficult to read unprocessed in many places. So something

 marked symbols would be a *really* nice to have.


I don't see any particular advantage to using (name) rather than $(REF name), 
and some serious disadvantages with false positives like func(param). Oops! A 
markup language should really strive to minimize special syntax. (I'm often 
tripped up by false positives in the wacky markup languages in Skype, Github, 
Reddit, etc., each of which feels compelled to reinvent markup.)



 2. the requirement that the name be "in scope" does a pretty good job of
 reducing the problem. But cross references, are by definition CROSS
 references, i.e. they cross scopes. The whole point of them is they
 cross scopes, which means this becomes a much bigger issue.


 This is how I see it: single identifiers are only searched in the
 current scope and in parent scopes of the same module. Cross references
 to foreign scopes need to use qualified names (either ClassName.member
 or pack.mod.Symbol). This avoids any inter-module issues and also a lot
 of intra-module issues. Symbols in code blocks on the other hand can be
 cross referenced completely automatically according to the usual D
 lookup rules (including imports).


My bias is towards being extremely conservative in adding new features to Ddoc, 
instead we should strive to get the max out of the existing system, and even 
rely on external post processors.
Dec 30 2013
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 31.12.2013 00:37, schrieb Walter Bright:

 On 12/30/2013 3:13 PM, Sönke Ludwig wrote:

 Am 30.12.2013 23:13, schrieb Walter Bright:

 On 12/30/2013 1:25 PM, Sönke Ludwig wrote:

 Identifiers in documentation comments that are function parameters or
 are names that are in scope at the associated declaration are
 emphasized
 in the output.

 So the same problem is already reality - you already have to go through
 the documentation to see if there are any mis-highlighted words.


 1. having a bad idea is not justification for doing it again :-)


 Of course not, but my point was that the mistake has already been made,
 so we may as well make the best out of it... not really doing it again
 in that sense.

 But if the decision is that in fact this was a mistake, it should get
 removed and replaced by a proper solution sooner rather than later (it's
 not that bad to lose some highlighting and Phobos uses $(D ...) anyway,
 so just removing it wouldn't hurt much).


 
 Ddoc is very heavily used. I don't believe we can make such silent,
 breaking changes.
 


It's just word highlighting! Breaking changes in the language are one
thing, but this kind of fix is a whole different category. Keeping it
will only make the problem worse over time. If there is a reasonable
deprecation path, that would be better, of course, but the current
situation is the worst of all possibilities.


 

 And in that regard I'd like to
 point to "D's goals for embedded documentation"
 <http://dlang.org/ddoc.htm>. I very much agree with them, but the
 current reality looks like the opposite - lots of macro-pseudo-HTML
 markup, very difficult to read unprocessed in many places. So something

 marked symbols would be a *really* nice to have.


 
 I don't see any particular advantage to using (name) rather than $(REF
 name), and some serious disadvantages with false positives like
 func(param). Oops! A markup language should really strive to minimize
 special syntax. (I'm often tripped up by false positives in the wacky
 markup languages in Skype, Github, Reddit, etc., each of which feels
 compelled to reinvent markup.)


It's not "(name)", but "name()" or "name(param)", which is recognized as
a function call. "#name" also forces recognition as an identifier, as
well as "::name" in C++'s case.

I agree that going overboard with special syntax is a bad idea, but
symbols in a code documentation language is such a fundamental thing
that it should surely be among the first things, if not *the* first
thing, that deserves a concise syntax.

Also, the official first four goals of DDOC are:


 1. It looks good as embedded documentation, not just after it is
    extracted and processed.
 2. It's easy and natural to write, i.e. minimal reliance on <tags> and
    other clumsy forms one would never see in a finished document.
 3. It does not repeat information that the compiler already knows from
    parsing the code.
 4. It doesn't rely on embedded HTML, as such will impede extraction
    and formatting for other purposes.


Using $(XREF), $(TABLE $(TR $(TD ...))), $(LINK a b) $(D ...) and so on
is nothing but markup. If we go that route, we could as well remove all
syntax, except for the macros:

$(SUMMARY ...)
$(SECTION See_Also
   ...
)
$(PARAMS
  $(PARAM x ...)
  $(PARAM y ...)
)
$(SECTION Example
  $(CODEBLOCK
    ...
  )
)

But you added the syntax for a reason back then (readability ...and
writability). And the same goes for the auto-detection of identifiers. I
think the only problem is that the default would have been better off
the other way around.
Dec 30 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 01:05, Sönke Ludwig wrote:


 Also, the official first four goals of DDOC are:


 1. It looks good as embedded documentation, not just after it is
     extracted and processed.




It certainly does not. Just have a look at the top of std.algorithm or 
std.uni. Hundreds of lines of Ddoc madness.


 2. It's easy and natural to write, i.e. minimal reliance on <tags> and
     other clumsy forms one would never see in a finished document.




Fails again. As soon as you need something more advanced than the 
standard sections you need to use macros. The macros are just as tags, 
just with a different syntax and name.


 3. It does not repeat information that the compiler already knows from
     parsing the code.




You sure need to do if you want good documentation. Take a look at 
std.algorithm, it has over 300 lines of Ddoc code to create a symbol 
table that Ddoc should be able to automatically generate.


 4. It doesn't rely on embedded HTML, as such will impede extraction
     and formatting for other purposes.




As far as I know this isn't very useful. For the other formats we use, 
like PDF, it uses the HTML output as a base.

To me it looks like it failed on all four of these goals.


 Using $(XREF), $(TABLE $(TR $(TD ...))), $(LINK a b) $(D ...) and so on
 is nothing but markup. If we go that route, we could as well remove all
 syntax, except for the macros:

 $(SUMMARY ...)
 $(SECTION See_Also
     ...
 )
 $(PARAMS
    $(PARAM x ...)
    $(PARAM y ...)
 )
 $(SECTION Example
    $(CODEBLOCK
      ...
    )
 )

 But you added the syntax for a reason back then (readability ...and
 writability). And the same goes for the auto-detection of identifiers. I
 think the only problem is that the default would have been better off
 the other way around.


I completely agree.

-- 
/Jacob Carlborg
Dec 31 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/31/13 4:26 AM, Jacob Carlborg wrote:

 On 2013-12-31 01:05, Sönke Ludwig wrote:


 Also, the official first four goals of DDOC are:


 1. It looks good as embedded documentation, not just after it is
     extracted and processed.




 It certainly does not. Just have a look at the top of std.algorithm or
 std.uni. Hundreds of lines of Ddoc madness.


That's an exaggeration. The top of std.algorithm produces complex output 
(two tables). The rest of the documentation is nice and legible.


 2. It's easy and natural to write, i.e. minimal reliance on <tags> and
     other clumsy forms one would never see in a finished document.




 Fails again. As soon as you need something more advanced than the
 standard sections you need to use macros. The macros are just as tags,
 just with a different syntax and name.


As one who's written both, I can testify this isn't true. Ddoc macros 
need to be mentioned once and then the closing paren completes them. 
Additionally, they can expand to arbitrarily complex output. In 
contrast, tags need to be mentioned twice (an often-mentioned issue with 
writing HTML by hand), are syntactically heavier, and therefore litter 
the text a lot more.


 3. It does not repeat information that the compiler already knows from
     parsing the code.




 You sure need to do if you want good documentation. Take a look at
 std.algorithm, it has over 300 lines of Ddoc code to create a symbol
 table that Ddoc should be able to automatically generate.


No. There are two tables there. One is a categorization of algorithms, 
which is human-made. There are several solutions to materializing that 
table but they all would require the intervention of a human being. I 
figured std.algorithm adds new algorithms rarely enough to allow myself 
a solution that requires manual intervention.

The second table is a "cheat sheet" with human-produced short 
descriptions for each function. Those short descriptions wouldn't fit 
well with the longer descriptions.


 4. It doesn't rely on embedded HTML, as such will impede extraction
     and formatting for other purposes.




 As far as I know this isn't very useful. For the other formats we use,
 like PDF, it uses the HTML output as a base.


Again no. You are misinformed. The PDF manual is generated via LaTeX 
from the same ddoc sources as the HTML pages.


Andrei
Dec 31 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 31, 2013 at 09:08:07AM -0800, Andrei Alexandrescu wrote:

 On 12/31/13 4:26 AM, Jacob Carlborg wrote:


[...]

4. It doesn't rely on embedded HTML, as such will impede extraction
and formatting for other purposes.




As far as I know this isn't very useful. For the other formats we
use, like PDF, it uses the HTML output as a base.


 
 Again no. You are misinformed. The PDF manual is generated via LaTeX
 from the same ddoc sources as the HTML pages.


[...]

This is true.

I will note, however, that limitations in Ddoc make the LaTeX output
less than ideal. Many fine points of LaTeX formatting are ignored or
simply not possible (e.g., proper use of em- and en-dashes, proper use
of '.\ ', non-breaking spaces, hyphenation hints, etc.). In some cases,
postprocessing is needed to take full advantage of the LaTeX format
(esp. for display of math formulas, which is one of LaTeX's biggest
selling points).

Even for HTML, the lack of semantic support for paragraphs means XHTML
compliance is out of the question, and any proper tag nesting where
paragraphs are involved is a fragile hack that is likely non-HTML
compliant. But nobody notices this because most browsers are too
permissive in what they accept.

So even though I think Ddoc isn't as bad as some people make it out to
be, I'm still on the fence about it.


T

-- 
By understanding a machine-oriented language, the programmer will tend
to use a much more efficient method; it is much closer to reality. -- D.
Knuth
Dec 31 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/31/13 11:51 AM, H. S. Teoh wrote:

 On Tue, Dec 31, 2013 at 09:08:07AM -0800, Andrei Alexandrescu wrote:

 On 12/31/13 4:26 AM, Jacob Carlborg wrote:


 [...]

 4. It doesn't rely on embedded HTML, as such will impede extraction
 and formatting for other purposes.




 As far as I know this isn't very useful. For the other formats we
 use, like PDF, it uses the HTML output as a base.


 Again no. You are misinformed. The PDF manual is generated via LaTeX
 from the same ddoc sources as the HTML pages.


 [...]

 This is true.

 I will note, however, that limitations in Ddoc make the LaTeX output
 less than ideal. Many fine points of LaTeX formatting are ignored or
 simply not possible (e.g., proper use of em- and en-dashes, proper use
 of '.\ ', non-breaking spaces, hyphenation hints, etc.). In some cases,
 postprocessing is needed to take full advantage of the LaTeX format
 (esp. for display of math formulas, which is one of LaTeX's biggest
 selling points).

 Even for HTML, the lack of semantic support for paragraphs means XHTML
 compliance is out of the question, and any proper tag nesting where
 paragraphs are involved is a fragile hack that is likely non-HTML
 compliant. But nobody notices this because most browsers are too
 permissive in what they accept.


These should be fixed in ddoc.

Andrei
Dec 31 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 31, 2013 at 06:12:02PM -0800, Andrei Alexandrescu wrote:

 On 12/31/13 11:51 AM, H. S. Teoh wrote:

On Tue, Dec 31, 2013 at 09:08:07AM -0800, Andrei Alexandrescu wrote:

On 12/31/13 4:26 AM, Jacob Carlborg wrote:


[...]

4. It doesn't rely on embedded HTML, as such will impede
extraction and formatting for other purposes.




As far as I know this isn't very useful. For the other formats we
use, like PDF, it uses the HTML output as a base.


Again no. You are misinformed. The PDF manual is generated via LaTeX
from the same ddoc sources as the HTML pages.


[...]

This is true.

I will note, however, that limitations in Ddoc make the LaTeX output
less than ideal. Many fine points of LaTeX formatting are ignored or
simply not possible (e.g., proper use of em- and en-dashes, proper
use of '.\ ', non-breaking spaces, hyphenation hints, etc.). In some
cases, postprocessing is needed to take full advantage of the LaTeX
format (esp. for display of math formulas, which is one of LaTeX's
biggest selling points).

Even for HTML, the lack of semantic support for paragraphs means
XHTML compliance is out of the question, and any proper tag nesting
where paragraphs are involved is a fragile hack that is likely
non-HTML compliant. But nobody notices this because most browsers are
too permissive in what they accept.


 
 These should be fixed in ddoc.


[...]

For HTML:

	https://d.puremagic.com/issues/show_bug.cgi?id=9731

For LaTeX, though, it would require some pretty radical changes to ddoc,
that I'm not sure will be accepted.


T

-- 
Blunt statements really don't have a point.
Jan 01 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 18:08, Andrei Alexandrescu wrote:


 That's an exaggeration. The top of std.algorithm produces complex output
 (two tables). The rest of the documentation is nice and legible.


std.algorithm contains 335 lines of ddoc comments. I would consider that 
"hundreds", or does "hundreds" mean something else. What I'm saying is 
that those two tables should be automatically generated, at least the 
first one.


 As one who's written both, I can testify this isn't true. Ddoc macros
 need to be mentioned once and then the closing paren completes them.
 Additionally, they can expand to arbitrarily complex output. In
 contrast, tags need to be mentioned twice (an often-mentioned issue with
 writing HTML by hand), are syntactically heavier, and therefore litter
 the text a lot more.


I'm considering it to be a sort of tag, just with a different syntax. 
Who said tags need to have an end tag. A less verbose a minimal reliance 
on tags would be Markdown or similar. Creating a list in Markdown is 
simple as:

* this
* is
* a
* list
* in Markdown


 No. There are two tables there. One is a categorization of algorithms,
 which is human-made. There are several solutions to materializing that
 table but they all would require the intervention of a human being. I
 figured std.algorithm adds new algorithms rarely enough to allow myself
 a solution that requires manual intervention.


If there's a specific macro that would indicate a category then it would 
be easy to generate a table like that.

/// $(CATEGORY searching)
void foo ()


 The second table is a "cheat sheet" with human-produced short
 descriptions for each function. Those short descriptions wouldn't fit
 well with the longer descriptions.


That can also be generated automatically. Ddoc can just insert the 
summary [1] for each symbol.

[1] The Ddoc documentation says:

"The first section is the Summary, and does not have a section name. It 
is first paragraph, up to a blank line or a section name. While the 
summary can be any length, try to keep it to one line."

http://dlang.org/ddoc.html


 Again no. You are misinformed. The PDF manual is generated via LaTeX
 from the same ddoc sources as the HTML pages.


Ok, fair enough.

-- 
/Jacob Carlborg
Jan 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/1/14 4:56 AM, Jacob Carlborg wrote:

 On 2013-12-31 18:08, Andrei Alexandrescu wrote:


 That's an exaggeration. The top of std.algorithm produces complex output
 (two tables). The rest of the documentation is nice and legible.


 std.algorithm contains 335 lines of ddoc comments. I would consider that
 "hundreds", or does "hundreds" mean something else.


I replied to "Hundreds of lines of Ddoc madness" with "that's an 
exaggeration". Do you really think I was refuting the factual 335 lines?


 What I'm saying is
 that those two tables should be automatically generated, at least the
 first one.


Yah, that would be nice.


 As one who's written both, I can testify this isn't true. Ddoc macros
 need to be mentioned once and then the closing paren completes them.
 Additionally, they can expand to arbitrarily complex output. In
 contrast, tags need to be mentioned twice (an often-mentioned issue with
 writing HTML by hand), are syntactically heavier, and therefore litter
 the text a lot more.


 I'm considering it to be a sort of tag, just with a different syntax.
 Who said tags need to have an end tag. A less verbose a minimal reliance
 on tags would be Markdown or similar. Creating a list in Markdown is
 simple as:

 * this
 * is
 * a
 * list
 * in Markdown


I prefer ddoc to markdown for its flexibility. Ddoc is an almost pure 
macro expansion system with minimal syntax. Markdown defines a bunch of 
syntax but has (last time I looked) is weak on macro capability.

Case in point: markdown lists will be styled one way. With DDoc macros 
one can define any style of lists needed. (We've defined BOOKTABLE in 
addition to a couple of other TABLEs, which works beautifully in HTML 
and LaTeX; not sure how can be done in Markdown.


 If there's a specific macro that would indicate a category then it would
 be easy to generate a table like that.

 /// $(CATEGORY searching)
 void foo ()


The issue is with writing text out of order, something DDoc is not very 
apt at. Would be nice to improve on that (we do some of it with javascript).


Andrei
Jan 01 2014
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 01, 2014 at 06:41:50PM -0800, Andrei Alexandrescu wrote:

 On 1/1/14 4:56 AM, Jacob Carlborg wrote:


[...]

I'm considering it to be a sort of tag, just with a different syntax.
Who said tags need to have an end tag. A less verbose a minimal
reliance on tags would be Markdown or similar. Creating a list in
Markdown is simple as:

* this
* is
* a
* list
* in Markdown


 
 I prefer ddoc to markdown for its flexibility. Ddoc is an almost
 pure macro expansion system with minimal syntax. Markdown defines a
 bunch of syntax but has (last time I looked) is weak on macro
 capability.


Well, ddoc was designed to be a macro system, so from that POV, it's
pretty good at what it was intended to do. Whether it's the best system
for *generating documentation*, though, is a matter of opinion. Markdown
has the advantage of having very close visual appearance in the input as
compared to the output -- a stated goal of ddoc, but obviously the
unified macro syntax was a higher priority in ddoc's case. For example,
using `backticks` to write code snippets is definitely more readable
than writing $(D backticks), among many other things. But that does
introduce more syntax, which makes parsing more involved and also
requires learning more syntax. So, it's a trade-off.



 Case in point: markdown lists will be styled one way. With DDoc
 macros one can define any style of lists needed. (We've defined
 BOOKTABLE in addition to a couple of other TABLEs, which works
 beautifully in HTML and LaTeX; not sure how can be done in Markdown.


Styling is an orthogonal issue. Markdown was designed to translate to
HTML, so you'd just write CSS stylesheets to customize the output. Ddoc,
OTOH, was designed to be a generic macro system, so obviously it's
better at doing macro-based stuff.



If there's a specific macro that would indicate a category then it
would be easy to generate a table like that.

/// $(CATEGORY searching)
void foo ()


 
 The issue is with writing text out of order, something DDoc is not
 very apt at. Would be nice to improve on that (we do some of it with
 javascript).


[...]

The limitation of macro systems is that, fundamentally speaking, no
semantics are assigned to macros. A macro is just an arbitrary name that
gets recursively substituted with some pre-specified pattern.  The macro
engine doesn't know what any of the macros "mean", and therefore has no
concept of things like cross-referencing, categorization, etc., which
are all higher-level, logical concepts, that are required in order to
move data around.

There are, of course, ways of simulating this with macros, such as with
a global top-level macro that takes an arbitrary number of arguments and
can permute them at will, but this is ultimately just a hack. Macro
systems simply don't provide enough expressive power to build
abstractions like "this block of text is a `declaration` with
`identifier` as key, so add `identifier` to the index structure in the
output". Postprocessing is the only way to reliably add this kind of
semantics to it.

But postprocessing also removes some of the advantage of ddoc: if you
translate everything down to HTML tags, the postprocessor will have to
reparse the HTML and resynthesize information on where sections
begin/end, what constitutes a declaration, etc., which may not be
reliable. So now you have to use an intermediate format (like docbook)
that still captures some of the original structure, so that the
postprocessor can reliably recover the original document structure from
it. But then, you might as well just write in the intermediate format to
begin with, instead of using ddoc.

So while I do agree that ddoc is pretty good at what it does --
expanding macros -- I'm still on the fence as to whether that's the best
approach to documentation generation.


T

-- 
Sometimes the best solution to morale problems is just to fire all of the
unhappy people. -- despair.com
Jan 02 2014
next sibling parent =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 02.01.2014 22:38, schrieb H. S. Teoh:

 For example,
 using `backticks` to write code snippets is definitely more readable
 than writing $(D backticks), among many other things. But that does
 introduce more syntax, which makes parsing more involved and also
 requires learning more syntax. So, it's a trade-off.


I don't want to argue for as much syntax as Markdown has - even if I'd 
actually find that great, Markdown has become a kind of de-facto 
standard in the programming area - but, overall, at least the aspect of 
learning new syntax is not really worse than learning new macros. There 
may be cases where a form of syntax is more difficult to remember, but 
there are also many cases where the macros are actually harder to learn 
and remember.

Having said that, Markdown, if not implemented using a series of 
regexes, is a crazy format to parse. There is an existing parser in 
vibe.d, though.



 Case in point: markdown lists will be styled one way. With DDoc
 macros one can define any style of lists needed. (We've defined
 BOOKTABLE in addition to a couple of other TABLEs, which works
 beautifully in HTML and LaTeX; not sure how can be done in Markdown.


 Styling is an orthogonal issue. Markdown was designed to translate to
 HTML, so you'd just write CSS stylesheets to customize the output. Ddoc,
 OTOH, was designed to be a generic macro system, so obviously it's
 better at doing macro-based stuff.


Also, an important point is that markdown defines *semantic* primitives. 
They can be translated to whatever is required by the output format. In 
DDOC's case that could simply be some standard macros ($(UL), $(LI)).

BTW to Markdown HTML basically is what macros are to DDOC (just that it 
also has defined semantics). Inline HTML explicitly supported to handle 
advanced cases and, if restricted to semantic tags, provides a pretty 
good basis for translating into other output formats.
Jan 02 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-02 22:38, H. S. Teoh wrote:


 The limitation of macro systems is that, fundamentally speaking, no
 semantics are assigned to macros. A macro is just an arbitrary name that
 gets recursively substituted with some pre-specified pattern.  The macro
 engine doesn't know what any of the macros "mean", and therefore has no
 concept of things like cross-referencing, categorization, etc., which
 are all higher-level, logical concepts, that are required in order to
 move data around.


We could start adding semantic meaning to some macros. They would become 
built-in macros ddoc knows about.

-- 
/Jacob Carlborg
Jan 03 2014
prev sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, January 02, 2014 13:38:22 H. S. Teoh wrote:

 But postprocessing also removes some of the advantage of ddoc: if you
 translate everything down to HTML tags, the postprocessor will have to
 reparse the HTML and resynthesize information on where sections
 begin/end, what constitutes a declaration, etc., which may not be
 reliable. So now you have to use an intermediate format (like docbook)
 that still captures some of the original structure, so that the
 postprocessor can reliably recover the original document structure from
 it. But then, you might as well just write in the intermediate format to
 begin with, instead of using ddoc.


That depends on how you go about post-processing. If you design your ddoc 
macros with the idea that there's going to be post-processing, then they can 
convert to other ddoc macros which are designed for the post-processor to be 
able to handle, and then the post-processor can convert those into another set 
of macros that you run ddoc on again to get the final result. That's 
essentially what I've done when using ddoc for stuff like a table of contents 
or index.

Now, if the post-processing is really the last step rather than just post one 
of multiple runs of ddoc, then it's stuck dealing with whatever the target 
format is, but if you do multiple runs of ddoc with a separate program 
operating on the intermediate results of the macro expansion and then 
generating a new set of ddoc macros for another run of ddoc, then your post-
processor doesn't have to understand the target format.

- Jonathan M Davis
Jan 02 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-01-03 01:56, Jonathan M Davis wrote:


 That depends on how you go about post-processing. If you design your ddoc
 macros with the idea that there's going to be post-processing, then they can
 convert to other ddoc macros which are designed for the post-processor to be
 able to handle, and then the post-processor can convert those into another set
 of macros that you run ddoc on again to get the final result. That's
 essentially what I've done when using ddoc for stuff like a table of contents
 or index.

 Now, if the post-processing is really the last step rather than just post one
 of multiple runs of ddoc, then it's stuck dealing with whatever the target
 format is, but if you do multiple runs of ddoc with a separate program
 operating on the intermediate results of the macro expansion and then
 generating a new set of ddoc macros for another run of ddoc, then your post-
 processor doesn't have to understand the target format.


Seems unnecessary complicated.

-- 
/Jacob Carlborg
Jan 03 2014
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, January 03, 2014 12:12:44 Jacob Carlborg wrote:

 On 2014-01-03 01:56, Jonathan M Davis wrote:

 That depends on how you go about post-processing. If you design your ddoc
 macros with the idea that there's going to be post-processing, then they
 can convert to other ddoc macros which are designed for the
 post-processor to be able to handle, and then the post-processor can
 convert those into another set of macros that you run ddoc on again to
 get the final result. That's essentially what I've done when using ddoc
 for stuff like a table of contents or index.
 
 Now, if the post-processing is really the last step rather than just post
 one of multiple runs of ddoc, then it's stuck dealing with whatever the
 target format is, but if you do multiple runs of ddoc with a separate
 program operating on the intermediate results of the macro expansion and
 then generating a new set of ddoc macros for another run of ddoc, then
 your post- processor doesn't have to understand the target format.


 
 Seems unnecessary complicated.


It depends on what you're trying to do. For generating something like a table 
of contents, it's pretty straightforward.

For instance,

$(CHAPTER Introduction,
    $(P Contents of chapter)
 )

could be processed by a short program to have that chapter's name put in the 
table of contents which it generates (which is then also ddoc). e.g.

$(TABLE_OF_CONTENTS
    $(CHAPTER_ENTRY Introduction, 1)
    $(CHAPTER_ENTRY Some Other Chapter, 2)
    $(CHAPTER_ENTRY Conclusion, 3)
)

and if any other operations need to be done on CHAPTER to make it suitable for 
your target formats, then it can be tweaked as necessary.

You then take the resulting .dd file and generate the documentation with the 
ddoc file that corresponds to the target format (e.g. html.ddoc or latex.ddoc 
or whatever). I've found that it works quite well.

But that's for generating a book, not documentation for code. You could do 
something similar to generate cross-references in code documentation, I'm 
sure, but I wasn't necessarily suggesting that we do. I was just pointing out 
that it's quite possible (and not necessarily difficult) to manipulate a .dd
file 
(or a .d file with ddoc in it) to generate another .dd file which you then 
actually use to generate the html or latex or whatever you need to generate 
and that you don't necessarily need to manipulate the target format to get 
what you want.

- Jonathan M Davis
Jan 03 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
XSLT is pretty useful for format conversion. It is also built 
into most browsers, so you can just emit XML attach a XSLT 
stylesheet and send it to a browser for transformation and 
rendering.

Would be an advantage for anyone writing a book or article on D 
(transform XML over to docbook, and from there to any format).
Jan 03 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 00:37, Walter Bright wrote:


 I don't see any particular advantage to using (name) rather than $(REF
 name), and some serious disadvantages with false positives like
 func(param). Oops! A markup language should really strive to minimize
 special syntax. (I'm often tripped up by false positives in the wacky
 markup languages in Skype, Github, Reddit, etc., each of which feels
 compelled to reinvent markup.)


You never addressed any of my other suggestions. Would suggestion B, 
that is having a special macro that compiler knows about, be an 
acceptable solution? Because the current behavior is not good enough.


 My bias is towards being extremely conservative in adding new features
 to Ddoc, instead we should strive to get the max out of the existing
 system, and even rely on external post processors.


If Ddoc requires post processing to be useful it's a complete failure. 
Why don't we instead strive to make it the best documentation generator 
for D?

-- 
/Jacob Carlborg
Dec 31 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/31/2013 4:14 AM, Jacob Carlborg wrote:

 If Ddoc requires post processing to be useful it's a complete failure.


Not at all. The same ddoc sources are used to generate an ebook and a Windows 
help file - which are based on html but require different html to be generated, 
a difference handled by ddoc. The ebook and chm both require post processing 
with tools supplied by Amazon and Microsoft, respectively.

It works, it is not a failure because the source files do not need to be 
modified to get this working. A failure would be having to modify the source 
files each time a different output needs to be generated.



 Why don't we instead strive to make it the best documentation generator for D?


We all of course want that - but there is certainly room for disagreement about 
what "best" is.

Reinventing Amazon's "kindlegen.exe" seems pointless to me.
Dec 31 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-31 22:08, Walter Bright wrote:


 Not at all. The same ddoc sources are used to generate an ebook and a
 Windows help file - which are based on html but require different html
 to be generated, a difference handled by ddoc. The ebook and chm both
 require post processing with tools supplied by Amazon and Microsoft,
 respectively.

 It works, it is not a failure because the source files do not need to be
 modified to get this working. A failure would be having to modify the
 source files each time a different output needs to be generated.


Ok, I would not consider it a failure if it needs to be converted to a 
different format. But adding cross referencing, generating an index or 
similar would still be a failure, if that requires post processing.


 We all of course want that - but there is certainly room for
 disagreement about what "best" is.


Yes, but who would not want cross referencing to work properly?


 Reinventing Amazon's "kindlegen.exe" seems pointless to me.


Yes, I agree.

-- 
/Jacob Carlborg
Jan 01 2014
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 31 Dec 2013 00:13:41 +0100
schrieb S=C3=B6nke Ludwig <sludwig outerproduct.org>:


 [...] I very much agree with them, but the
 current reality looks like the opposite - lots of macro-pseudo-HTML
 markup, very difficult to read unprocessed in many places. So something

 marked symbols would be a *really* nice to have.



for identifier references! It is also very verbose to create
bullet-point lists and you always have to double-check you
don't forget to replace all ) with $(RPAREN). If you misspell
anything it is lost in the output.

--=20
Marco
Dec 31 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 10:31, Marco Leise wrote:



 for identifier references! It is also very verbose to create
 bullet-point lists and you always have to double-check you
 don't forget to replace all ) with $(RPAREN). If you misspell
 anything it is lost in the output.


I completely agree. I suggested the $(REF) macro as an alternative to 
automatically add cross referencing because I thought that would have a 
greater chance of being accepted.

-- 
/Jacob Carlborg
Dec 31 2013
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 31 Dec 2013 13:27:17 +0100
schrieb Jacob Carlborg <doob me.com>:


 On 2013-12-31 10:31, Marco Leise wrote:
 


 for identifier references! It is also very verbose to create
 bullet-point lists and you always have to double-check you
 don't forget to replace all ) with $(RPAREN). If you misspell
 anything it is lost in the output.


 
 I completely agree. I suggested the $(REF) macro as an alternative to 
 automatically add cross referencing because I thought that would have a 
 greater chance of being accepted.


Yes that's more practical, unfortunately. :p

-- 
Marco
Dec 31 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/30/13 1:00 PM, Walter Bright wrote:

 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:

 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.


 As I mentioned elsewhere, I dislike the requirement to use the inverse
 feature. Consider maintenance - I add a symbol, and then have to grep
 the rest of the module for any inadvertent collisions? Features should
 be positive ones, not negative.


Your arguments are convincing.

Andrei
Dec 30 2013
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:

 On 12/30/13 1:00 PM, Walter Bright wrote:

 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:

 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.


 As I mentioned elsewhere, I dislike the requirement to use the inverse
 feature. Consider maintenance - I add a symbol, and then have to grep
 the rest of the module for any inadvertent collisions? Features should
 be positive ones, not negative.


 
 Your arguments are convincing.
 
 Andrei
 


The argument in isolation is sound, no doubt. But alone it doesn't lead
to a solution*. There are a lot of arguments regarding the current
situation and if the decision is not to use the existing _negative rule,
then all of those should be carefully taken into account.

 - Visual noise and effort when using $(XREF) style macros
 - $(XREF) isn't sufficient! A multitude of macros is actually needed
 - There is no standard set of macros, it's project specific
 -> The whole documentation needs to be revised when moving code between
    projects!
 - Macros need to be carefully crafted so that they don't encode
   knowledge about the final documentation/file structure
 - Macros alone can't cope with nested directory structures
 - The cognitive load for multiple such macros would very likely cause
   them to go unused in many cases

My favorite solution would be to introduce a simple #identifier.chain or
 identifier.chain syntax to let the compiler or documentation generator
insert the proper links or macro calls. Using a sepecial $(REF
identifier.chain) pseudo macro, like Jacob suggested, would work as
well, but I can't see how the additional syntax/visual/typing overhead
is justified for such a common feature - at least when trying to find an
optimal solution, compared to the current macro based system it would of
course still be a big improvement.

But still, considering that there is precedence for the _negative
approach (i.e. the problems of it are already there), I find using that
acceptable, too, even if the concept is not as pretty/clean.


* *Some* solution is needed. The different structure of the DDOX
documentation for example makes the $(XREF) macros worthless, so they
are currently completely disabled and automatic link generation is used
instead.
Dec 31 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tuesday, 31 December 2013 at 09:29:20 UTC, Sönke Ludwig wrote:

 My favorite solution would be to introduce a simple 
 #identifier.chain or
  identifier.chain syntax to let the compiler or documentation 
 generator
 insert the proper links or macro calls. Using a sepecial $(REF
 identifier.chain) pseudo macro, like Jacob suggested, would 
 work as
 well, but I can't see how the additional syntax/visual/typing 
 overhead
 is justified for such a common feature - at least when trying 
 to find an
 optimal solution, compared to the current macro based system it 
 would of
 course still be a big improvement.


I agree that a good compromise would be to lighten the syntactic 
overhead.

How about using the $(D) macro instead of $(REF)? This macro 
syntax-highlights its parameter as D code, and in Phobos it is 
already used for identifiers, so I think a good solution would be 
to find and create cross-reference links only for $(D contents).

Also, I'd like to mention a previous idea I posted here: this 
problem, and several other problems we have with DDoc, would be 
gone if we'd put what we already have with what we already have, 
and allow writing D code that gets interpreted to translate 
documentation comments into documentation output, using our 
existing CTFE capabilities. This way, the entire DDoc syntax can 
be reimplemented using "library" D code, interpreted by the 
compiler during documentation generation, and anyone could build 
upon it or completely replace it with another syntax.

P.S. How come your user agent (Thunderbird) is not emitting 
format=flowed messages? According to [1], it supports 
format=flowed, and can only be disabled via editing option 
strings.

[1]: 
http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format
Dec 31 2013
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 31.12.2013 11:05, schrieb Vladimir Panteleev:

 P.S. How come your user agent (Thunderbird) is not emitting
 format=flowed messages? According to [1], it supports format=flowed, and
 can only be disabled via editing option strings.

 [1]:
 http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format


Although I can't promise that I didn't tamper with these settings, I 
didn't find any non-default value in about:config in that area, but 
there is a default setting "mailnews.send_plaintext_flowed = false". 
This is now manually set to true, let's see.
Dec 31 2013
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tuesday, 31 December 2013 at 10:37:23 UTC, Sönke Ludwig wrote:

 Am 31.12.2013 11:05, schrieb Vladimir Panteleev:

 P.S. How come your user agent (Thunderbird) is not emitting
 format=flowed messages? According to [1], it supports 
 format=flowed, and
 can only be disabled via editing option strings.

 [1]:
 http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format


 Although I can't promise that I didn't tamper with these 
 settings, I didn't find any non-default value in about:config 
 in that area, but there is a default setting 
 "mailnews.send_plaintext_flowed = false". This is now manually 
 set to true, let's see.


Did you install the Enigmail extension? It seems to have a bug 
which turns off format=flowed generation even if you never use it.
Dec 31 2013
parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 01.01.2014 02:21, schrieb Vladimir Panteleev:

 On Tuesday, 31 December 2013 at 10:37:23 UTC, Sönke Ludwig wrote:

 Am 31.12.2013 11:05, schrieb Vladimir Panteleev:

 P.S. How come your user agent (Thunderbird) is not emitting
 format=flowed messages? According to [1], it supports format=flowed, and
 can only be disabled via editing option strings.

 [1]:
 http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format


 Although I can't promise that I didn't tamper with these settings, I
 didn't find any non-default value in about:config in that area, but
 there is a default setting "mailnews.send_plaintext_flowed = false".
 This is now manually set to true, let's see.


 Did you install the Enigmail extension? It seems to have a bug which
 turns off format=flowed generation even if you never use it.


Yep, that seems to be it. Good to know, I've never noticed that up to now.
Jan 01 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-31 11:05, Vladimir Panteleev wrote:


 I agree that a good compromise would be to lighten the syntactic overhead.

 How about using the $(D) macro instead of $(REF)? This macro
 syntax-highlights its parameter as D code, and in Phobos it is already
 used for identifiers, so I think a good solution would be to find and
 create cross-reference links only for $(D contents).

 Also, I'd like to mention a previous idea I posted here: this problem,
 and several other problems we have with DDoc, would be gone if we'd put
 what we already have with what we already have, and allow writing D code
 that gets interpreted to translate documentation comments into
 documentation output, using our existing CTFE capabilities. This way,
 the entire DDoc syntax can be reimplemented using "library" D code,
 interpreted by the compiler during documentation generation, and anyone
 could build upon it or completely replace it with another syntax.


Isn't this something Sönke has done with DDOX and with the help of the 
JSON output?

I've been thinking that it probably wouldn't be that hard to create a 
__trait that return the ddoc comments of a symbol. Don't know how much 
that would be different from the JSON output though.

-- 
/Jacob Carlborg
Dec 31 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 31 Dec 2013 10:28:46 +0100
schrieb S=C3=B6nke Ludwig <sludwig outerproduct.org>:


 Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:

 On 12/30/13 1:00 PM, Walter Bright wrote:

 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:

 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.


 As I mentioned elsewhere, I dislike the requirement to use the inverse
 feature. Consider maintenance - I add a symbol, and then have to grep
 the rest of the module for any inadvertent collisions? Features should
 be positive ones, not negative.


=20
 Your arguments are convincing.
=20
 Andrei
=20


=20
 The argument in isolation is sound, no doubt. But alone it doesn't lead
 to a solution*. There are a lot of arguments regarding the current
 situation and if the decision is not to use the existing _negative rule,
 then all of those should be carefully taken into account.
=20
  - Visual noise and effort when using $(XREF) style macros
  - $(XREF) isn't sufficient! A multitude of macros is actually needed
  - There is no standard set of macros, it's project specific
  -> The whole documentation needs to be revised when moving code between
     projects!
  - Macros need to be carefully crafted so that they don't encode
    knowledge about the final documentation/file structure
  - Macros alone can't cope with nested directory structures
  - The cognitive load for multiple such macros would very likely cause
    them to go unused in many cases
=20
 My favorite solution would be to introduce a simple #identifier.chain or
  identifier.chain syntax to let the compiler or documentation generator
 insert the proper links or macro calls. Using a sepecial $(REF
 identifier.chain) pseudo macro, like Jacob suggested, would work as
 well, but I can't see how the additional syntax/visual/typing overhead
 is justified for such a common feature - at least when trying to find an
 optimal solution, compared to the current macro based system it would of
 course still be a big improvement.
=20
 But still, considering that there is precedence for the _negative
 approach (i.e. the problems of it are already there), I find using that
 acceptable, too, even if the concept is not as pretty/clean.
=20
=20
 * *Some* solution is needed. The different structure of the DDOX
 documentation for example makes the $(XREF) macros worthless, so they
 are currently completely disabled and automatic link generation is used
 instead.


I'm fully with you on that one and I would like to add that
DDoc's macro system reverses the responsibilities. Instead of
writing easy to read documentation that is agnostic to the
target output, you have to take into account any potential
output format's peculiarities.
This starts with the list of escaped characters in your
custom template and ends with structuring elements for lists
and tables. Without abstraction of structuring elements in
DDoc lists and tables look like HTML and are likely not
portable to other formats that use a different or implicit
structuring scheme. E.g. if you wanted to support MarkDown with
its explicit numbers for ordered lists, you'd have to replace
all $(LI) with $(LI1), $(LI2), ...
I guess it is a little too late to ask for a complete
reimplementation of DDoc?

--=20
Marco
Dec 31 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/31/13 1:28 AM, Sönke Ludwig wrote:

 Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:

 On 12/30/13 1:00 PM, Walter Bright wrote:

 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:

 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.


 As I mentioned elsewhere, I dislike the requirement to use the inverse
 feature. Consider maintenance - I add a symbol, and then have to grep
 the rest of the module for any inadvertent collisions? Features should
 be positive ones, not negative.


 Your arguments are convincing.

 Andrei


 The argument in isolation is sound, no doubt. But alone it doesn't lead
 to a solution*. There are a lot of arguments regarding the current
 situation and if the decision is not to use the existing _negative rule,
 then all of those should be carefully taken into account.

   - Visual noise and effort when using $(XREF) style macros
   - $(XREF) isn't sufficient! A multitude of macros is actually needed
   - There is no standard set of macros, it's project specific
   -> The whole documentation needs to be revised when moving code between
      projects!
   - Macros need to be carefully crafted so that they don't encode
     knowledge about the final documentation/file structure
   - Macros alone can't cope with nested directory structures
   - The cognitive load for multiple such macros would very likely cause
     them to go unused in many cases


In a nutshell, "good cross-referencing is hard". That's exactly the 
experience of anyone who's created an index. In publishing there are 
automatic index generators and they're universally known to be vastly 
inferior to professional-produced ones. I've created my own index by 
hand for TDPL and it's been difficult, but probably the result is better 
than an automated index.

Andrei
Dec 31 2013
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 31.12.2013 17:48, schrieb Andrei Alexandrescu:

 On 12/31/13 1:28 AM, Sönke Ludwig wrote:

 Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:

 On 12/30/13 1:00 PM, Walter Bright wrote:

 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:

 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.


 As I mentioned elsewhere, I dislike the requirement to use the inverse
 feature. Consider maintenance - I add a symbol, and then have to grep
 the rest of the module for any inadvertent collisions? Features should
 be positive ones, not negative.


 Your arguments are convincing.

 Andrei


 The argument in isolation is sound, no doubt. But alone it doesn't lead
 to a solution*. There are a lot of arguments regarding the current
 situation and if the decision is not to use the existing _negative rule,
 then all of those should be carefully taken into account.

   - Visual noise and effort when using $(XREF) style macros
   - $(XREF) isn't sufficient! A multitude of macros is actually needed
   - There is no standard set of macros, it's project specific
   -> The whole documentation needs to be revised when moving code between
      projects!
   - Macros need to be carefully crafted so that they don't encode
     knowledge about the final documentation/file structure
   - Macros alone can't cope with nested directory structures
   - The cognitive load for multiple such macros would very likely cause
     them to go unused in many cases


 In a nutshell, "good cross-referencing is hard".


I don't think so at all (on the technical level). What is the issue with 
using the #identifier.chain pattern for example? It seems simple enough 
and I can't imagine a more natural thing than to reference symbols in 
documentation the same way as in code.


 That's exactly the
 experience of anyone who's created an index. In publishing there are
 automatic index generators and they're universally known to be vastly
 inferior to professional-produced ones. I've created my own index by
 hand for TDPL and it's been difficult, but probably the result is better
 than an automated index.


But index generation is a higher level issue. The basis for it is to 
have some kind of universal anchor that can be used as a reference. 
#identifier.chain or $(REF identifier.chain) is nothing more than that 
and just solves the issue of supporting arbitrary nested scopes and 
different file system or document hierarchies.
Dec 31 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/31/13 9:39 AM, Sönke Ludwig wrote:

 Am 31.12.2013 17:48, schrieb Andrei Alexandrescu:

 In a nutshell, "good cross-referencing is hard".


 I don't think so at all (on the technical level). What is the issue with
 using the #identifier.chain pattern for example? It seems simple enough
 and I can't imagine a more natural thing than to reference symbols in
 documentation the same way as in code.


I see #identifier.chain similar to e.g. $(X identifier.chain) so no 
issue with that. It just adds more syntax without necessity. I'd instead 
work on improving macros to generate proper xrefs out of $(X ...) rather 
than adding new syntax. But either way is fine.


 That's exactly the
 experience of anyone who's created an index. In publishing there are
 automatic index generators and they're universally known to be vastly
 inferior to professional-produced ones. I've created my own index by
 hand for TDPL and it's been difficult, but probably the result is better
 than an automated index.


 But index generation is a higher level issue. The basis for it is to
 have some kind of universal anchor that can be used as a reference.
 #identifier.chain or $(REF identifier.chain) is nothing more than that
 and just solves the issue of supporting arbitrary nested scopes and
 different file system or document hierarchies.


Agreed. In all likelihood I misunderstood something because I thought 
you were referring to cross-referencing text without any user intervention.


Andrei
Dec 31 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-31 19:16, Andrei Alexandrescu wrote:


 Agreed. In all likelihood I misunderstood something because I thought
 you were referring to cross-referencing text without any user intervention.


I made two suggestions, one which is completely automatic and one which 
requires some syntax. It seems most posts here only address the first 
suggestion.

-- 
/Jacob Carlborg
Dec 31 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/29/13 10:35 AM, Sönke Ludwig wrote:

 Am 29.12.2013 18:38, schrieb Jacob Carlborg:

 A. Automatic cross reference


 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:


It's past time we do this. So the code is in there, we need to build it 
appropriately. Where do I start reading?

Andrei
Dec 30 2013
parent =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 30.12.2013 17:31, schrieb Andrei Alexandrescu:

 On 12/29/13 10:35 AM, Sönke Ludwig wrote:

 Am 29.12.2013 18:38, schrieb Jacob Carlborg:

 A. Automatic cross reference


 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:


 
 It's past time we do this. So the code is in there, we need to build it
 appropriately. Where do I start reading?
 
 Andrei
 


I've just added a small alternative link below the normal Phobos
documentation and added it to the "all" target of posix.mak. It will
automatically call dub to build "dpl-docs" in the tools repository.

https://github.com/D-Programming-Language/dlang.org/pull/462

If there are no more issues with it, it should be the first step that
actually gets something online.
Dec 30 2013
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Dec 29, 2013 at 06:38:55PM +0100, Jacob Carlborg wrote:

 If nothing has happened recently the current situation of cross
 referencing in Ddoc sucks. What's currently being used in the Phobos
 documentation is the XREF, CXREF and ECXREF ddoc macros. These
 macros take two arguments, append "std", "core" or "etc" and form a
 link of the arguments. The problem with this is that it doesn't work
 so good for referring to symbols in a deeper package hierarchy.


[...]

Not only so, even without cross-referencing, the way Ddoc currently does
referencing *within* a module is faulty, because it does not take
symbols declared in nested scopes into account. For example:

	module mymodule;

	/// docs here
	struct S {
		/// docs here
		void func() { ... }
	}

	/// docs here
	void func() { ... }

Ddoc will use "func" as the identifier for *both* mymodule.func and
mymodule.S.func, so any hyperlinks to "func" will likely point to the
wrong overload of func (depending on declaration order).

This problem is actually being exhibited on dlang.org at this very
moment, in std.algorithm. Look for 'remove' in the navigation table at
the top of the page, and click on it; you'll see it jumps to the wrong
place because there's a member function called 'remove' in a struct that
took precedence over the module-level 'remove' function. (Not to mention
it breaks HTML compliance because it gives multiple elements the same
ID.)


T

-- 
People tell me that I'm paranoid, but they're just out to get me.
Dec 29 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 05:05, H. S. Teoh wrote:


 Not only so, even without cross-referencing, the way Ddoc currently does
 referencing *within* a module is faulty, because it does not take
 symbols declared in nested scopes into account. For example:

 	module mymodule;

 	/// docs here
 	struct S {
 		/// docs here
 		void func() { ... }
 	}

 	/// docs here
 	void func() { ... }

 Ddoc will use "func" as the identifier for *both* mymodule.func and
 mymodule.S.func, so any hyperlinks to "func" will likely point to the
 wrong overload of func (depending on declaration order).

 This problem is actually being exhibited on dlang.org at this very
 moment, in std.algorithm. Look for 'remove' in the navigation table at
 the top of the page, and click on it; you'll see it jumps to the wrong
 place because there's a member function called 'remove' in a struct that
 took precedence over the module-level 'remove' function. (Not to mention
 it breaks HTML compliance because it gives multiple elements the same
 ID.)


Yeah, that's another problem.

-- 
/Jacob Carlborg
Dec 30 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
30-Dec-2013 14:19, Jacob Carlborg пишет:

 On 2013-12-30 05:05, H. S. Teoh wrote:


 Not only so, even without cross-referencing, the way Ddoc currently does
 referencing *within* a module is faulty, because it does not take
 symbols declared in nested scopes into account. For example:

     module mymodule;

     /// docs here
     struct S {
         /// docs here
         void func() { ... }
     }

     /// docs here
     void func() { ... }

 Ddoc will use "func" as the identifier for *both* mymodule.func and
 mymodule.S.func, so any hyperlinks to "func" will likely point to the
 wrong overload of func (depending on declaration order).

 This problem is actually being exhibited on dlang.org at this very
 moment, in std.algorithm. Look for 'remove' in the navigation table at
 the top of the page, and click on it; you'll see it jumps to the wrong
 place because there's a member function called 'remove' in a struct that
 took precedence over the module-level 'remove' function. (Not to mention
 it breaks HTML compliance because it gives multiple elements the same
 ID.)


 Yeah, that's another problem.


And couple of years old at that...

-- 
Dmitry Olshansky
Dec 30 2013
prev sibling next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Sunday, 29 December 2013 at 17:38:55 UTC, Jacob Carlborg wrote:

 If nothing has happened recently the current situation of cross 
 referencing in Ddoc sucks. What's currently being used in the 
 Phobos documentation is the XREF, CXREF and ECXREF ddoc macros. 
 These macros take two arguments, append "std", "core" or "etc" 
 and form a link of the arguments. The problem with this is that 
 it doesn't work so good for referring to symbols in a deeper 
 package hierarchy.


bootDoc[1] fixes this (thanks to Denis) with a number of standard 
macros that are reasonably easy to use[2].

It also uses JavaScript to "fix" (insofar as JS is a fix for 
anything) the issue of page anchors not supporting nested symbols 
properly.

[1] https://github.com/JakobOvrum/bootDoc

[2] https://github.com/JakobOvrum/bootDoc/wiki/Macros
Dec 30 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 10:43, Jakob Ovrum wrote:


 bootDoc[1] fixes this (thanks to Denis) with a number of standard macros
 that are reasonably easy to use[2].


Based on the documentation it looks like it has the same problems as the 
current approach, that is:

* Multiple macros are needed for something that at most one single macro 
should be needed

* Doesn't work for arbitrary deep hierarchies

Also, it feels like a workaround for something that should be fixed in 
the compiler.

-- 
/Jacob Carlborg
Dec 30 2013
parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 30 December 2013 at 10:23:25 UTC, Jacob Carlborg wrote:

 Based on the documentation it looks like it has the same 
 problems as the current approach, that is:


Yep, its only benefit is that it's a standardized set of macros 
and works for any reasonably deep hierarchy.

bootDoc is about doing the best that is possible within the 
limitations of DDoc. I think it is a better example of DDoc's 
best than Phobos, as it's much cleaner and actually has 
documentation.

That said, DDoc's best is hardly impressive.
Dec 30 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-30 11:26, Jakob Ovrum wrote:


 Yep, its only benefit is that it's a standardized set of macros and
 works for any reasonably deep hierarchy.


Not as far as I can see. What I can see it only supports one level of 
packages. That means basically a flat hierarchy like Phobos. All of my 
projects have a deeper hierarchy than that, usually one extra level of 
packages. I would consider a nested package to be within the limit of a 
reasonably deep hierarchy.

-- 
/Jacob Carlborg
Dec 30 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/29/2013 9:38 AM, Jacob Carlborg wrote:

 A. Automatic cross reference

 Automatically create links for all matching symbols in the current scope. That
 means to create a link to a symbol in the current scope it's enough to mention
 it's name in the documentation.


This will create many unwanted links, in three scenarios:

1. In the description for S, there should not be hyperlinks to S. These would
be 
annoying - when you click on them, nothing will happen.

2. Using the word S as a word, not in reference to symbol S, would generate a 
hyperlink which would not make sense.

3. If there are multiple symbols S, hyperlinks to the wrong one would be 
created. This is worse than useless.

In my adding of such hyperlinks, 1..3 happen regularly, this is not a "may" 
happen. It's a "will" happen, a lot.
Dec 30 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 20:08, Walter Bright wrote:


 This will create many unwanted links, in three scenarios:


I don't think so, see below.


 1. In the description for S, there should not be hyperlinks to S. These
 would be annoying - when you click on them, nothing will happen.


The compiler knows which symbol it currently generating documentation 
for, just exclude that.


 2. Using the word S as a word, not in reference to symbol S, would
 generate a hyperlink which would not make sense.


I've have seen that happened a couple of times with documentation 
generators that support it, it has never bother me. As Sönke has said, 
you can solve that by appending an underscore and the compiler would not 
create a link of the symbol.


 3. If there are multiple symbols S, hyperlinks to the wrong one would be
 created. This is worse than useless.


Most of the times there are not multiple symbols with the same name, the 
language make sure of that. Although there will be a problem with 
function overloading.


 In my adding of such hyperlinks, 1..3 happen regularly, this is not a
 "may" happen. It's a "will" happen, a lot.


I don't agree, I don't think it's a problem. It's far better than having 
to add macros everywhere instead.

-- 
/Jacob Carlborg
Dec 30 2013
next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am I slow... ;)
Dec 30 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 12:14 PM, Jacob Carlborg wrote:

 On 2013-12-30 20:08, Walter Bright wrote:


 This will create many unwanted links, in three scenarios:


 I don't think so, see below.


 1. In the description for S, there should not be hyperlinks to S. These
 would be annoying - when you click on them, nothing will happen.


 The compiler knows which symbol it currently generating documentation for, just
 exclude that.


That's true.



 2. Using the word S as a word, not in reference to symbol S, would
 generate a hyperlink which would not make sense.


 I've have seen that happened a couple of times with documentation generators
 that support it, it has never bother me. As Sönke has said, you can solve that
 by appending an underscore and the compiler would not create a link of the
symbol.


While possible, I'm not too enamored with this. It makes for a maintenance 
problem. Add a new symbol to a module, and then you have to manually search for 
any other instances of the word. Naturally, the latter won't get done, and 
you're left with nonsense links in the doc.



 3. If there are multiple symbols S, hyperlinks to the wrong one would be
 created. This is worse than useless.


 Most of the times there are not multiple symbols with the same name, the
 language make sure of that. Although there will be a problem with function
 overloading.


I've run into this a few times, and it cannot be dismissed easily. Worse, there 
is the issue of how one overrides the auto-link generation to be the right link.



 It's far better than having to add
 macros everywhere instead.


I'll have to disagree on that. I just finished doing it with std.datetime, and 
it didn't take more than a few minutes with global query-search-replace.
Dec 30 2013
next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 30.12.2013 21:58, schrieb Walter Bright:

 2. Using the word S as a word, not in reference to symbol S, would
 generate a hyperlink which would not make sense.


 I've have seen that happened a couple of times with documentation
 generators
 that support it, it has never bother me. As Sönke has said, you can
 solve that
 by appending an underscore and the compiler would not create a link of
 the symbol.


 
 While possible, I'm not too enamored with this. It makes for a
 maintenance problem. Add a new symbol to a module, and then you have to
 manually search for any other instances of the word. Naturally, the
 latter won't get done, and you're left with nonsense links in the doc.
 


True, but I'd argue that this is really the exception. New symbols also
influence just the documentation of the current scope (i.e. a new class
member won't affect the documentation of other classes in the same module).


 

 3. If there are multiple symbols S, hyperlinks to the wrong one would be
 created. This is worse than useless.


 Most of the times there are not multiple symbols with the same name, the
 language make sure of that. Although there will be a problem with
 function
 overloading.


 
 I've run into this a few times, and it cannot be dismissed easily.
 Worse, there is the issue of how one overrides the auto-link generation
 to be the right link.


Qualified names. The qualifying prefix could be hidden from the user or
displayed as is. But after all it also provides useful information to
the reader, as it's now possible to tell which symbol is meant without
following the link first.
Dec 30 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 21:58, Walter Bright wrote:


 I've run into this a few times, and it cannot be dismissed easily.
 Worse, there is the issue of how one overrides the auto-link generation
 to be the right link.


I guess that would require a macro.


 I'll have to disagree on that. I just finished doing it with
 std.datetime, and it didn't take more than a few minutes with global
 query-search-replace.


How did you limit that search to only the documentation?

-- 
/Jacob Carlborg
Dec 30 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 1:31 PM, Jacob Carlborg wrote:

 On 2013-12-30 21:58, Walter Bright wrote:

 I'll have to disagree on that. I just finished doing it with
 std.datetime, and it didn't take more than a few minutes with global
 query-search-replace.


 How did you limit that search to only the documentation?


The "query" part, meaning it highlighted each one and asked for yes/no.

http://www.gnu.org/software/emacs/manual/html_node/emacs/Query-Replace.html
Dec 30 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 23:10, Walter Bright wrote:


 The "query" part, meaning it highlighted each one and asked for yes/no.

 http://www.gnu.org/software/emacs/manual/html_node/emacs/Query-Replace.html


That doesn't sound very efficient. Regardless of that, the current 
macros still don't work. How do you do cross referencing for something 
that has a deeper hierarchy and what most of Phobos has?

Just take a look at the documentation of std.digest.crc[1]. None of the 
cross referencing to std.digest.digest.toHexString work. It's due to a
misconception of how the $(XREF) works.

[1] http://dlang.org/phobos/std_digest_crc.html

-- 
/Jacob Carlborg
Dec 31 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/31/2013 4:37 AM, Jacob Carlborg wrote:

 That doesn't sound very efficient.


You don't need an efficient solution for ddoc files of readable size. It only 
takes a few moments. I've spent far, far more time typing about it here than it 
took to do. Some solutions are simply not worth the bother.


 Regardless of that, the current macros still
 don't work. How do you do cross referencing for something that has a deeper
 hierarchy and what most of Phobos has?

 Just take a look at the documentation of std.digest.crc[1]. None of the cross
 referencing to std.digest.digest.toHexString work. It's due to a
 misconception of how the $(XREF) works.


Using the macros incorrectly is not necessarily an indictment of macros.
Dec 31 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 21:52, Walter Bright wrote:


 You don't need an efficient solution for ddoc files of readable size. It
 only takes a few moments. I've spent far, far more time typing about it
 here than it took to do. Some solutions are simply not worth the bother.


I've probably spent at least as much time on the documentation as on the 
code for std.serialization/Orange. The documentation could still be 
improved a lot, like more cross referencing. That's why I started this 
thread.


 Using the macros incorrectly is not necessarily an indictment of macros.


So which macro should I use to make cross referencing for arbitrary deep 
hierarchies?

-- 
/Jacob Carlborg
Dec 31 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/31/2013 1:17 PM, Jacob Carlborg wrote:

 So which macro should I use to make cross referencing for arbitrary deep
 hierarchies?


I used this one in std.datetime:
Dec 31 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 23:27, Walter Bright wrote:


 I used this one in std.datetime:




That would require specific knowledge about how Ddoc generate the HTML 
and file structure. Like replaces dots with an underscore in the module 
name for the file name. std.datetime becomes std_datetime.htm. Is that wise?

-- 
/Jacob Carlborg
Jan 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/1/2014 4:43 AM, Jacob Carlborg wrote:

 On 2013-12-31 23:27, Walter Bright wrote:


 I used this one in std.datetime:




 That would require specific knowledge about how Ddoc generate the HTML and file
 structure.



reference, no file structure required.
Jan 01 2014
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 January 2014 at 20:38:43 UTC, Walter Bright wrote:

 Ddoc doesn't generate HTML (the macros do)


Ddoc doesn't generate HTML at all, not even with macros.

/// Returns true iff a > b
bool greaterThan(int a, int b);

Ddoc does /not/ do the right thing there. The best you can do is 
write $(GT), which is stupid and insufficient or &gt; which is 
wrong.
Jan 01 2014
prev sibling next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 01.01.2014 21:38, schrieb Walter Bright:




A declaration being inside the *same* file is *also* part of the 
file/directory structure. I could also be in different files. The point 
is that if this kind of knowledge is encoded in the documentation, or it 
is tied to that particular structure. This is the case for the Phobos 
documentation and is the reason why the DDOX based docs have to 
completely ignore the manual cross references.
Jan 02 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-01 21:38, Walter Bright wrote:



 reference, no file structure required.


Right, I failed to see that. But then that macro won't work for symbols 
in other modules.

The file structure I was referring to was that for the Phobos 
documentation the HTML files are created with the same name as the fully 
qualified module name but with dots replaced with underscores. If I want 
to create a link to a symbol in an other module I need to know about that.

-- 
/Jacob Carlborg
Jan 02 2014
prev sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 30.12.2013 20:08, schrieb Walter Bright:

 On 12/29/2013 9:38 AM, Jacob Carlborg wrote:

 A. Automatic cross reference

 Automatically create links for all matching symbols in the current
 scope. That
 means to create a link to a symbol in the current scope it's enough to
 mention
 it's name in the documentation.


 
 This will create many unwanted links, in three scenarios:
 
 1. In the description for S, there should not be hyperlinks to S. These
 would be annoying - when you click on them, nothing will happen.


This one can easily be implemented by the documentation generator.


 
 2. Using the word S as a word, not in reference to symbol S, would
 generate a hyperlink which would not make sense.


The _S scheme can be used to avoid the hyperlink.


 
 3. If there are multiple symbols S, hyperlinks to the wrong one would be
 created. This is worse than useless.


If the same lookup rules are used as for the D code itself, the results
should turn out pretty much as expected. DDOX currently doesn't follow
imports correctly, yet, but even with just a simple hierarchical lookup
in the current scope and a fallback to a global lookup (across all
symbols) this seems to work pretty well.


 
 In my adding of such hyperlinks, 1..3 happen regularly, this is not a
 "may" happen. It's a "will" happen, a lot.
 


It will happen, but you have to weigh disabling these places manually
against having to add an explicit link every time. From what I've seen
so far, the latter means a *lot* more work.

Personally, I'd prefer an explicit, but unobtrusive, marker like Doxygen
uses (e.g. "func()", "::func" or "#func" are recognized as cross
references). But since the _ rule is already there, it seems to be a
natural choice.
Dec 30 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 12:33 PM, Sönke Ludwig wrote:

 [...]


I've pretty much replied to these points in my response to Jacob and Andrei.
Dec 30 2013