www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - json and ddoc

reply "Adam D. Ruppe" <destructionator gmail.com> writes:
I'm (finally) updating my dpldocs.info website again, and
before I just pulled names from the generated Phobos html.

This time, I want to use dmd's json output to avoid dependency
on the specific html layout of std.ddoc.


Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
one big .json file with all the info.

But, the comments aren't parsed at all.



So, my question is: do we have a ddoc parser that stands alone?


BTW: this is the complete listing from json so far:
http://dpldocs.info/search/awesome

I'm actually impressed with the speed, considering it is 
re-parsing
3 MB of generated json per load. std.variant and std.json might 
not
be speed demons, but they are plenty good enough for me.

The ultimate goal here is to revitalize my D search engine and to
make it as a nice index for all functions, with brief 
descriptions,
etc., hopefully for all D libs everywhere.

With the html, dealing with different people's macros to parse it
is a pain. But with the json, dmd always does it the same way, so
with some luck, I can just add a box "upload your project's json
file" and allow easy instant submissions for other libs.
Aug 01 2012
next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Wed, Aug 1, 2012 at 4:07 PM, Adam D. Ruppe <destructionator gmail.com> wrote:

 I'm (finally) updating my dpldocs.info website again, and



Yes!


 Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
 one big .json file with all the info.

 But, the comments aren't parsed at all.


Well, the comments are discarded as soon as lexical analysis is done,
I guess. I suppose the DMD part that does the json output never ever
sees any comment, just code tokens (see the discussion on lexing D)? I
wonder how documentation generation is done with -D?

Since with the -D flag, DMD correctly 'attach' doc comments on code
part, (this is a modul-level doc, this is a field documentation, ...)
I guess the problem is to somehow transport this information to the
json generator. This generator does not give much information btw, but
my impression is that people do not use it that much and,
consequently, no feature request is emitted.
Aug 01 2012
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 August 2012 at 14:36:26 UTC, Philippe Sigaud 
wrote:

 Well, the comments are discarded as soon as lexical analysis is 
 done, I guess.


They are actually there if you do both -X and -D together.
(-X alone doesn't show it; it must be discarded then, but
-D keeps it).


The problem is though that it is the text from the comment
without any organization or macro expansion.

For example:

{
"name" : "map(fun...) if (fun.length >= 1)",
"kind" : "template",
"protection" : "public",
"comment" : "\nImplements the homonym function (also known as $(D 
transform)) present\nin many languages of    functional flavor. 
The call $(D map!(fun)(range))\nreturns a range of which elements 
are obtained by applying  $(D fun(x))\nleft to right for all $(D 
x) in $(D range). The original ranges are\nnot changed. 
Evaluation is   done lazily. The range returned by $(D 
map)\ncaches the last value such that evaluating $(D front) 
multiple    times\ndoes not result in multiple calls to $(D 
fun).\n\nExample:\n----\nint[] arr1 = [ 1, 2, 3, 4 ];\nint[]   
arr2 = [ 5, 6 ];\nauto squares = map!(\"a * a\")(chain(arr1, 
arr2));\nassert(equal(squares, [ 1, 4, 9, 16, 25, 36 
]));\n----\n\nMultiple functions can be passed to $(D map). In 
that case, the\nelement type of $(D map) is  a tuple containing 
one element for each\nfunction.\n\nExample:\n\n----\nauto arr1 = 
[ 1, 2, 3, 4 ];\nforeach   (e; map!(\"a + a\", \"a * 
a\")(arr1))\n{\n    writeln(e[0], \" \", e[1]);\n}\n----\n\nYou 
may alias $(D map)   with some function(s) to a symbol and 
use\nit separately:\n\n----\nalias map!(to!string) stringize;     
        \nassert(equal(stringize([ 1, 2, 3, 4 ]), [ \"1\", \"2\", 
\"3\", \"4\" ]));\n----\n",
"line" : 367,




I'd like to get the sections broken out and as many macros 
expanded
as possible, so the summary text in search results can look 
pretty.


One option is doing my own parsing, but Walter said the recursive
expansion in the macros was actually very tricky to get right.. so
I'd prefer to skip doing that again myself if I can.



 This generator does not give much information btw, but
 my impression is that people do not use it that much and,
 consequently, no feature request is emitted.


Yeah. It isn't bad though; more detailed than dpldocs.info
version 1.0 which did nothing more than grep '<a name' in
the generated html!
Aug 01 2012
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Wed, Aug 1, 2012 at 4:46 PM, Adam D. Ruppe <destructionator gmail.com> wrote:

 On Wednesday, 1 August 2012 at 14:36:26 UTC, Philippe Sigaud wrote:

 Well, the comments are discarded as soon as lexical analysis is done, I
 guess.



 They are actually there if you do both -X and -D together.
 (-X alone doesn't show it; it must be discarded then, but
 -D keeps it).


Ah, dammit. I was at work far from a D compiler. I never used -X *and*
-D at the same time.


 The problem is though that it is the text from the comment
 without any organization or macro expansion.


OK.


 This generator does not give much information btw, but
 my impression is that people do not use it that much and,
 consequently, no feature request is emitted.



 Yeah. It isn't bad though; more detailed than dpldocs.info
 version 1.0 which did nothing more than grep '<a name' in
 the generated html!


No no no, I wasn't talking about your app! dpldocs is cool!
I was talking about the json generator in DMD, which, to my eyes, does
not give enough information yet to be truly useful.
Aug 01 2012
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 August 2012 at 17:40:13 UTC, Philippe Sigaud 
wrote:

 No no no, I wasn't talking about your app! dpldocs is cool!


Thanks! I think the new version will be even better.
Preview:

http://dpldocs.info/search/search-2?searchTerm=post
http://dpldocs.info/search/search-2?searchTerm=get

I'm now extending the search to be able to look at
the type names from the json and do more inter-linking.
It also shows the brief description, if present, and
will filter out private members for you.

And, of course, it will be able to look at submitted
libraries too.

The little breadcrumbs I intend to make into nice indexes,
giving some nice sorting that most ddoc sites don't currently
do.


Still a ways to go, but I think it will end up really nice.


 I was talking about the json generator in DMD, which, to my 
 eyes, does not give enough information yet to be truly useful.


Yea, but it's still better than nothing at least.
Aug 01 2012
prev sibling next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 01.08.2012 16:07, schrieb Adam D. Ruppe:

 I'm (finally) updating my dpldocs.info website again, and
 before I just pulled names from the generated Phobos html.

 This time, I want to use dmd's json output to avoid dependency
 on the specific html layout of std.ddoc.


 Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
 one big .json file with all the info.

 But, the comments aren't parsed at all.



 So, my question is: do we have a ddoc parser that stands alone?


 BTW: this is the complete listing from json so far:
 http://dpldocs.info/search/awesome

 I'm actually impressed with the speed, considering it is re-parsing
 3 MB of generated json per load. std.variant and std.json might not
 be speed demons, but they are plenty good enough for me.

 The ultimate goal here is to revitalize my D search engine and to
 make it as a nice index for all functions, with brief descriptions,
 etc., hopefully for all D libs everywhere.

 With the html, dealing with different people's macros to parse it
 is a pain. But with the json, dmd always does it the same way, so
 with some luck, I can just add a box "upload your project's json
 file" and allow easy instant submissions for other libs.


I have a simple DDOC parser in vibe.d (*). No guarantees that the output 
is always acting the same as DMD (e.g. there is no support for special 
sections yet), but it was good enough for me so far.

Something like this should work:

import vibe.vibe;

void main(string[] args)
{
	auto fin = openFile(args[1], FileMode.Read);
	auto ddoc = cast(string)fin.readAll();
	auto html = appender!string();
	filterDdocComment(html, ddoc);

	auto fout = openFile(args[2], FileMode.CreateTrunc);
	fout.write(formatDdocComment(html.data));
}

It doesn't really have dependencies to vibe.d so ripping it out should 
also be easy.

(*) http://vibed.org/api/vibe.textfilter.ddoc#filterDdocComment

...that module is not really documented atm
Aug 01 2012
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 August 2012 at 15:09:45 UTC, Sönke Ludwig wrote:

 It doesn't really have dependencies to vibe.d so ripping it out 
 should also be easy.


Thanks! It looks like that will do the job.
Aug 01 2012
parent Brad Roberts <braddr puremagic.com> writes:
On 8/1/2012 8:20 AM, Adam D. Ruppe wrote:

 On Wednesday, 1 August 2012 at 15:09:45 UTC, Sönke Ludwig wrote:

 It doesn't really have dependencies to vibe.d so ripping it out should also be
easy.


 
 Thanks! It looks like that will do the job.


I'd almost pay money for the ddoc support that's inside dmd to be pulled into a
separate process that just uses the json
output.  I suspect it wouldn't be that hard to separate them, just never have
had the time to try it.
Aug 01 2012
prev sibling parent reply Brad Anderson <eco gnuk.net> writes:
On Wed, Aug 1, 2012 at 8:07 AM, Adam D. Ruppe <destructionator gmail.com>wrote:


 I'm (finally) updating my dpldocs.info website again, and
 before I just pulled names from the generated Phobos html.

 This time, I want to use dmd's json output to avoid dependency
 on the specific html layout of std.ddoc.


 Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
 one big .json file with all the info.

 But, the comments aren't parsed at all.



 So, my question is: do we have a ddoc parser that stands alone?


 BTW: this is the complete listing from json so far:
 http://dpldocs.info/search/**awesome <http://dpldocs.info/search/awesome>

 I'm actually impressed with the speed, considering it is re-parsing
 3 MB of generated json per load. std.variant and std.json might not
 be speed demons, but they are plenty good enough for me.

 The ultimate goal here is to revitalize my D search engine and to
 make it as a nice index for all functions, with brief descriptions,
 etc., hopefully for all D libs everywhere.

 With the html, dealing with different people's macros to parse it
 is a pain. But with the json, dmd always does it the same way, so
 with some luck, I can just add a box "upload your project's json
 file" and allow easy instant submissions for other libs.


There is already a pull request to add (at least some) ddoc to json:

http://forum.dlang.org/post/ycfyhcsbciguzaxwuppv forum.dlang.org
Aug 01 2012
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 August 2012 at 17:52:02 UTC, Brad Anderson wrote:

 There is already a pull request to add (at least some) ddoc to 
 json:

 http://forum.dlang.org/post/ycfyhcsbciguzaxwuppv forum.dlang.org


Well, we already have some... but that request looks like it
does a lot of good stuff.
Aug 01 2012