www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Documentation generation

reply bearophile <bearophileHUGS lycos.com> writes:
The purpose of a static language compiler is to help the programmer spot the
bugs in the code very well and to compile it. It's not the job of a compiler to
create nice HTML/XML documentation files.

So I suggest to:
1) DMD can copy all documentation strings inside the JSON file generated with
the -X option. DMD has just to recognize the /** */ and /// inside the D code
and put only such comments inside the JSON (the JSON has to contain all the
information necessary to create the HTML pages).
2) Move the code that understands DDoc syntax and generates the HTML docs
inside an external utility written in D or Python, and keep such utility inside
the zip of the standard DMD distribution. This program can ask DMD to generate
the JSON files, can read them and generate the HTML pages. 
3) Deprecate and then remove the -D option of DMD2.

Removing that HTML ddoc parsing & HTML generation from DMD allows such code to
evolve faster and to be debugged more efficiently. Today some people are using 

[Mostly unrelated: once xfbuild is well debugged it can be added inside the zip
of the standard DMD distribution.]

Bye,
bearophile
Aug 01 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
bearophile wrote:

 Removing that HTML ddoc parsing & HTML generation from DMD allows such code
 to evolve faster and to be debugged more efficiently. Today some people are
 using


There are very good reasons why ddoc is part of dmd.

1. Being a defined part of D means it's ALWAYS there. That means there won't be 
D compilers without ddoc. That means that people can rely on it being there,
and 
use it with confidence.

2. Please do not underestimate the power of something reliably being there. In 
my experience, it makes it FAR more likely that people will actually use it,
and 
there's the big win. If it's a separate utility, I guarantee you that usage of 
ddoc will drop probably by 90%.

3. A separate ddoc utility means that, inevitably, dmd and ddoc will get out of 
sync with each other. People will update dmd and not ddoc, and vice versa.
There 
is quite enough grief with people mixing one version of dmd with the wrong 
version of phobos and then being mystified by the results. I can't tell you how 
many times I've helped someone who INSISTED they were using the correct 
libphobos.a and there were no others on their machine, only to eventually 
sheepishly discover another old libphobos.a on their path that was being used.

4. Making it a separate utility means others will invent "improved" ddocs, with 
the resulting balkanization and non-portability of documentation comments. See 
C++ for the affect this has.

5. The documentation of Phobos took a HUGE leap forward with the integration of 
ddoc.

6. Similar advantages can be found with the integration of unittesting in D,
and 
the integration of the preprocessor with C/C++ (I'm old enough to remember when 
that was a separate program).

7. Contrary to what you state, making ddoc a separate utility will make it far 
harder to maintain, not easier. Just outputting JSON files is not enough, as 
ddoc is also able to pretty-print the declarations and do color syntax 
highlighting of coding examples. If you look at the dmd source code changes, 
you'll find that ddoc tracks changes in D's grammar and syntax with rarely any 
changes at all in the ddoc code.
Aug 01 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:

 There are very good reasons why ddoc is part of dmd.


Thank you for your many answers. I think you can even add them to the D2 FAQ :-)

Bye,
bearophile
Aug 01 2010
parent Brad Roberts <braddr puremagic.com> writes:
On 8/1/2010 1:53 PM, bearophile wrote:

 
 I think you can...
 
 Bye,
 bearophile


I've been avoiding saying this for a while, but I suspect a lot of people have
thought it at one time or another.  You probably ought to find a word other than
'can' to use.  Of course he's capable of... whatever.  Perhaps you meant to make
it a suggestion?
Aug 01 2010
prev sibling next sibling parent Lurker <lurk lurkers.org> writes:
Walter Bright Wrote:


 bearophile wrote:

 Removing that HTML ddoc parsing & HTML generation from DMD allows such code
 to evolve faster and to be debugged more efficiently. Today some people are
 using


 
 There are very good reasons why ddoc is part of dmd.
 
 1. Being a defined part of D means it's ALWAYS there. That means there won't
be 
 D compilers without ddoc. That means that people can rely on it being there,
and 
 use it with confidence.
 
 2. Please do not underestimate the power of something reliably being there. In 
 my experience, it makes it FAR more likely that people will actually use it,
and 
 there's the big win. If it's a separate utility, I guarantee you that usage of 
 ddoc will drop probably by 90%.


That's just silly. People using 15 different good document generators is much
better than one s**y one with s**y templates you need to dig from old newsgroup
posts.


 
 3. A separate ddoc utility means that, inevitably, dmd and ddoc will get out
of 
 sync with each other. People will update dmd and not ddoc, and vice versa.
There 
 is quite enough grief with people mixing one version of dmd with the wrong 
 version of phobos and then being mystified by the results. I can't tell you
how 
 many times I've helped someone who INSISTED they were using the correct 
 libphobos.a and there were no others on their machine, only to eventually 
 sheepishly discover another old libphobos.a on their path that was being used.


His idea was probably to separate the tools and still provide them both as part
of the standard distribution. It would be your duty to keep both up to date.


 
 4. Making it a separate utility means others will invent "improved" ddocs,
with 
 the resulting balkanization and non-portability of documentation comments. See 
 C++ for the affect this has.


D already has several document generators because ddoc is way too primitive.
Luckily the main competition (Tango) died because of an organized FUD attack.


 
 5. The documentation of Phobos took a HUGE leap forward with the integration
of 
 ddoc.


The output is still a huge joke compared to tools like doxygen and javadoc. No
offense. And those are rather basic document generation tools -- better ones
exist.


 
 6. Similar advantages can be found with the integration of unittesting in D,
and 
 the integration of the preprocessor with C/C++ (I'm old enough to remember
when 
 that was a separate program).


The integrated unit test is a joke. Spend 30 minutes with Java 6 someday and
try JUnit. Or maybe few hours. The more you use, the more you learn. If you
open your eyes, you might actually see how crippled your vision of unit testing
is. The integrated unittest stuff brings almost zero value when doing serious
software development.


 
 7. Contrary to what you state, making ddoc a separate utility will make it far 
 harder to maintain, not easier.


One of the main reasons why 3rd party tool support sucks is that YOU have zero
ability communicating technical details to the community. We can see it in
documentation. Almost none exists. People make lots of questions here and
instead of answering you always focus on bikeshedding.


 Just outputting JSON files is not enough, as 
 ddoc is also able to pretty-print the declarations and do color syntax 
 highlighting of coding examples. If you look at the dmd source code changes, 
 you'll find that ddoc tracks changes in D's grammar and syntax with rarely any 
 changes at all in the ddoc code.


That's not a surprise. If a docgen tool and a compiler frontend coming from a
single vendor didn't share any code, I wouldn't call it software engineering.
It would be a horrible toy project written by an amateur.
Aug 01 2010
prev sibling next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Walter Bright wrote:


 bearophile wrote:

 Removing that HTML ddoc parsing & HTML generation from DMD allows such code
 to evolve faster and to be debugged more efficiently. Today some people are
 using


 
 There are very good reasons why ddoc is part of dmd.
 
 1. Being a defined part of D means it's ALWAYS there. That means there won't
 be D compilers without ddoc. That means that people can rely on it being
 there, and use it with confidence.


...

I agree, though the standard output is not so pleasant. When the new website 
design is adopted, would you please consider shipping a .ddoc plus the css 
needed in a similar style for user documentation? No doubt people are willing
to 
help with this, I certainly would.
Aug 01 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Lutger wrote:

 I agree, though the standard output is not so pleasant. When the new website 
 design is adopted, would you please consider shipping a .ddoc plus the css 
 needed in a similar style for user documentation? No doubt people are willing
to 
 help with this, I certainly would.



Sure.
Aug 02 2010
prev sibling parent reply Bernard Helyer <b.helyer gmail.com> writes:
On Sun, 01 Aug 2010 13:22:53 -0700, Walter Bright wrote:

 1. Being a defined part of D means it's ALWAYS there. That means there
 won't be D compilers without ddoc.


/me waves.

I'm writing a D compiler, and have zero plans to add DDoc support. I 
couldn't do it well enough for me to waste my time on it.
Aug 06 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Bernard Helyer:

 I'm writing a D compiler, and have zero plans to add DDoc support. I 
 couldn't do it well enough for me to waste my time on it.  


Walter has said that putting ddoc inside the compiler helps keep ddoc uniform,
so people don't invent/use something different. In theory this is a nice
purpose, but in practice I have seen lot of people use other documentation
means for D, Doxygen. Walter seems to ignore how much often such other systems
are used instead of ddoc.

Bye,
bearophile
Aug 07 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
bearophile wrote:

 Walter has said that putting ddoc inside the compiler helps keep ddoc
 uniform, so people don't invent/use something different. In theory this is a
 nice purpose, but in practice I have seen lot of people use other
 documentation means for D, Doxygen. Walter seems to ignore how much often
 such other systems are used instead of ddoc.


It's fine if people choose to use other documentation systems. Ddoc, though, 
sets a minimum standard, and it's always there, requires no additional
installs, 
is always synced to the compiler, is always on every platform D is, etc.

My experience with languages that do not have built in doc abilities is, by and 
large, only a tiny minority of programmers use a doc system. Having to
research, 
download, install, read the manual, address incompatibilities, etc., takes 
effort and the reality is that by making such effort close to zero it greatly 
encourages people to use it.

The same goes for unit tests.

For another example, back in the 80's, it was commonplace for people to dis the 
C text preprocessor as "not a real macro system". Those who wanted a better one 
were advised to "just use m4". How many people using C or C++ today use m4 as 
the preprocessor? Zilch. The C preprocessor was good enough to get the job
done, 
it was always there, always ready, and that was that.

(Note that I'm not defending the preprocessor, I'm just illustrating the power 
of having something built in as opposed to being a separate tool.)
Aug 07 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/07/2010 02:53 PM, Walter Bright wrote:

 bearophile wrote:

 Walter has said that putting ddoc inside the compiler helps keep ddoc
 uniform, so people don't invent/use something different. In theory
 this is a
 nice purpose, but in practice I have seen lot of people use other
 documentation means for D, Doxygen. Walter seems to ignore how much often
 such other systems are used instead of ddoc.


 It's fine if people choose to use other documentation systems. Ddoc,
 though, sets a minimum standard, and it's always there, requires no
 additional installs, is always synced to the compiler, is always on
 every platform D is, etc.

 My experience with languages that do not have built in doc abilities is,
 by and large, only a tiny minority of programmers use a doc system.
 Having to research, download, install, read the manual, address
 incompatibilities, etc., takes effort and the reality is that by making
 such effort close to zero it greatly encourages people to use it.

 The same goes for unit tests.

 For another example, back in the 80's, it was commonplace for people to
 dis the C text preprocessor as "not a real macro system". Those who
 wanted a better one were advised to "just use m4". How many people using
 C or C++ today use m4 as the preprocessor? Zilch. The C preprocessor was
 good enough to get the job done, it was always there, always ready, and
 that was that.

 (Note that I'm not defending the preprocessor, I'm just illustrating the
 power of having something built in as opposed to being a separate tool.)


Boost's preprocessor library is a powerful exhibit in favor of your 
argument. I think I wouldn't be mistaken to say that everything it does 
can be done much easier and much better, plus with m4 you can do a lot 
other things of interest. But the Boost preprocessor library wins 
because "it's there".


Andrei
Aug 07 2010
prev sibling parent reply Jonathan M Davis <jmdavisprog gmail.com> writes:
On Friday 06 August 2010 20:06:59 Bernard Helyer wrote:

 On Sun, 01 Aug 2010 13:22:53 -0700, Walter Bright wrote:

 1. Being a defined part of D means it's ALWAYS there. That means there
 won't be D compilers without ddoc.


 
 /me waves.
 
 I'm writing a D compiler, and have zero plans to add DDoc support. I
 couldn't do it well enough for me to waste my time on it.


Well, that's your choice, but then you haven't followed the D spec in that 
regard, so your compiler wouldn't be properly standard D. It is the sort of 
thing that I would expect to be somewhat lower in priority than other features 
though.

- Jonathan M Davis
Aug 07 2010
parent reply Bernard Helyer <b.helyer gmail.com> writes:
On Sat, 07 Aug 2010 01:55:35 -0700, Jonathan M Davis wrote:


 On Friday 06 August 2010 20:06:59 Bernard Helyer wrote:

 On Sun, 01 Aug 2010 13:22:53 -0700, Walter Bright wrote:

 1. Being a defined part of D means it's ALWAYS there. That means
 there won't be D compilers without ddoc.


 
 /me waves.
 
 I'm writing a D compiler, and have zero plans to add DDoc support. I
 couldn't do it well enough for me to waste my time on it.


 
 Well, that's your choice, but then you haven't followed the D spec in
 that regard, so your compiler wouldn't be properly standard D. 


Nope. But it doesn't matter. It doesn't affect the code built or 
generated, and that's my focus -- at least for the foreseeable future.

I'm writing a compiler, not a half-arsed documentation generator (because 
if I wrote the DDoc stuff, that's exactly what it would be).
Aug 07 2010
next sibling parent reply div0 <div0 sourceforge.net> writes:
On 08/08/2010 01:32, Bernard Helyer wrote:

 Nope. But it doesn't matter. It doesn't affect the code built or
 generated, and that's my focus -- at least for the foreseeable future.

 I'm writing a compiler, not a half-arsed documentation generator (because
 if I wrote the DDoc stuff, that's exactly what it would be).


Sounds cool. What are you writing it in and is it going to be public?

-- 
My enormous talent is exceeded only by my outrageous laziness.
http://www.ssTk.co.uk
Aug 07 2010
parent Bernard Helyer <b.helyer gmail.com> writes:
On Sun, 08 Aug 2010 02:02:28 +0100, div0 wrote:


 On 08/08/2010 01:32, Bernard Helyer wrote:

 Nope. But it doesn't matter. It doesn't affect the code built or
 generated, and that's my focus -- at least for the foreseeable future.

 I'm writing a compiler, not a half-arsed documentation generator
 (because if I wrote the DDoc stuff, that's exactly what it would be).


 
 Sounds cool. What are you writing it in and is it going to be public?


D. It's already public[1], but there's not a lot to show yet. You can 
compile `int main() { return 42; }`, but that's about it, so far. :P

[1]:http://github.com/bhelyer/SDC
Aug 07 2010
prev sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Bernard Helyer (b.helyer gmail.com)'s article

 I'm writing a compiler, not a half-arsed documentation generator (because
 if I wrote the DDoc stuff, that's exactly what it would be).


I'll concur with Walter.  I think that sometimes power users of a given
tool/feature fail to realize that very often simple, good enough and works out
of
the box beats well-designed, feature-packed, efficient, but a PITA to set up and
use, overkill for simple cases, and not available out of the box.  I understand
treating ddoc as a low priority feature, but **please** support it eventually,
as
the whole point of it is that it's just there on any implementation (nothing
separate to download/configure) and just works for the simplest 80-90% of cases
even if its deficiencies become obvious in the last 10-20%.

In general I also think that DDoc's evolution is actually good design strategy.
 I
wish more tools were designed by pinning down how the simplest 80-90% of cases
should work first, treating that as a constraint and then figuring out how to
make
the most complicated 10-20% work within the constraint.  Designing with
"everything must be possible" as the first goal encourages overengineering such
that users with simple needs end up reinventing the wheel because it's the path
of
least resistance.  For example, I'd probably never use Ddoc if I had to use XML
for it.
Aug 07 2010
next sibling parent Bernard Helyer <b.helyer gmail.com> writes:
On Sun, 08 Aug 2010 02:22:30 +0000, dsimcha wrote:


 == Quote from Bernard Helyer (b.helyer gmail.com)'s article

 I'm writing a compiler, not a half-arsed documentation generator
 (because if I wrote the DDoc stuff, that's exactly what it would be).


 I understand treating ddoc as a low priority feature,
 but **please** support it eventually,


Well, sure, once basically *everything* else is done, I'll probably poke 
at it. But it is a very low priority.
Aug 07 2010
prev sibling parent retard <re tard.com.invalid> writes:
Sun, 08 Aug 2010 02:22:30 +0000, dsimcha wrote:


 == Quote from Bernard Helyer (b.helyer gmail.com)'s article

 I'm writing a compiler, not a half-arsed documentation generator
 (because if I wrote the DDoc stuff, that's exactly what it would be).


 
 I'll concur with Walter.  I think that sometimes power users of a given
 tool/feature fail to realize that very often simple, good enough and
 works out of the box beats well-designed, feature-packed, efficient, but
 a PITA to set up and use, overkill for simple cases, and not available
 out of the box.  I understand treating ddoc as a low priority feature,
 but **please** support it eventually, as the whole point of it is that
 it's just there on any implementation (nothing separate to
 download/configure) and just works for the simplest 80-90% of cases even
 if its deficiencies become obvious in the last 10-20%.
 
 In general I also think that DDoc's evolution is actually good design
 strategy.  I wish more tools were designed by pinning down how the
 simplest 80-90% of cases should work first, treating that as a
 constraint and then figuring out how to make the most complicated 10-20%
 work within the constraint.  Designing with "everything must be
 possible" as the first goal encourages overengineering such that users
 with simple needs end up reinventing the wheel because it's the path of
 least resistance.  For example, I'd probably never use Ddoc if I had to
 use XML for it.


The problem with ddoc is that even though 1000+ documentation formats 
already existed, he wanted to reinvent his own. The ddoc syntax is 
terrible ad-hoc crap and geared towards html (3.2/4). Has anyone 
seriously tried to generate any non-html documents with ddoc which look 
as good as better than those created with e.g. doxygen?! I've personally 
just filtered out the bad constructs from the sources and rendered with 
the buggy doxygen. Much better. I get nice pdf/tex/man/xml output.
Aug 14 2010
prev sibling parent bearophile <bearophileHUGS lycos.com> writes:
Brad Roberts:

 I've been avoiding saying this for a while, but I suspect a lot of people have
 thought it at one time or another.  You probably ought to find a word other
than
 'can' to use.  Of course he's capable of... whatever.  Perhaps you meant to
make
 it a suggestion?


You are right, I am sorry. I will have to work on my grammar.

Bye,
bearophile
Aug 04 2010