www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Automatic documentation builds

reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
I've put together a CI system of sorts that builds the 
documentation for all pull requests. Hopefully this should avoid 
the dlang.org build breaking again in the future. It integrates 
with GitHub as well, and on repositories covered by Brad's 
auto-tester, will create a second status for the documentation. 
The whole thing is Digger-powered, so uses the same code as 
Digger and http://digger.k3.1azy.net/trend/.

Sample page: 
http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/

Source code: https://github.com/CyberShadow/DAutoTest

Cool fact: When you request a built file (e.g. a .html page), 
it'll fetch the data directly from the Git cache repository, 
through a named pipe (bypassing disk) connected to a long-living 
Git process. This is also done transparently for included 
resources (images, CSS etc.)
Jun 11 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 12/06/2015 12:21 a.m., Vladimir Panteleev wrote:

 I've put together a CI system of sorts that builds the documentation for
 all pull requests. Hopefully this should avoid the dlang.org build
 breaking again in the future. It integrates with GitHub as well, and on
 repositories covered by Brad's auto-tester, will create a second status
 for the documentation. The whole thing is Digger-powered, so uses the
 same code as Digger and http://digger.k3.1azy.net/trend/.

 Sample page:
 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/


 Source code: https://github.com/CyberShadow/DAutoTest

 Cool fact: When you request a built file (e.g. a .html page), it'll
 fetch the data directly from the Git cache repository, through a named
 pipe (bypassing disk) connected to a long-living Git process. This is
 also done transparently for included resources (images, CSS etc.)


So for a PR it puts up a URL on the request?
Because this would be pretty awesome in that context!
Jun 11 2015
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 11 June 2015 at 12:27:29 UTC, Rikki Cattermole wrote:

 So for a PR it puts up a URL on the request?
 Because this would be pretty awesome in that context!


Yes, just click "Details" at the bottom of the PR page to go to 
the test result page.
Jun 11 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 12/06/2015 12:28 a.m., Vladimir Panteleev wrote:

 On Thursday, 11 June 2015 at 12:27:29 UTC, Rikki Cattermole wrote:

 So for a PR it puts up a URL on the request?
 Because this would be pretty awesome in that context!


 Yes, just click "Details" at the bottom of the PR page to go to the test
 result page.


Cool just tested it out!

Small thoughts, have some "goto" links e.g. phobos prerelease. It'll 
make it slightly easier to move around.
Otherwise I'm very impressed.
Jun 11 2015
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 11 June 2015 at 12:37:28 UTC, Rikki Cattermole wrote:

 On 12/06/2015 12:28 a.m., Vladimir Panteleev wrote:

 On Thursday, 11 June 2015 at 12:27:29 UTC, Rikki Cattermole 
 wrote:

 So for a PR it puts up a URL on the request?
 Because this would be pretty awesome in that context!


 Yes, just click "Details" at the bottom of the PR page to go 
 to the test
 result page.


 Cool just tested it out!

 Small thoughts, have some "goto" links e.g. phobos prerelease. 
 It'll make it slightly easier to move around.
 Otherwise I'm very impressed.


OK, added. But for Phobos changes, you should find the relevant 
module quickly in the list of changed files.
Jun 11 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/11/15 8:21 AM, Vladimir Panteleev wrote:

 I've put together a CI system of sorts that builds the documentation for
 all pull requests. Hopefully this should avoid the dlang.org build
 breaking again in the future. It integrates with GitHub as well, and on
 repositories covered by Brad's auto-tester, will create a second status
 for the documentation. The whole thing is Digger-powered, so uses the
 same code as Digger and http://digger.k3.1azy.net/trend/.

 Sample page:
 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/


 Source code: https://github.com/CyberShadow/DAutoTest

 Cool fact: When you request a built file (e.g. a .html page), it'll
 fetch the data directly from the Git cache repository, through a named
 pipe (bypassing disk) connected to a long-living Git process. This is
 also done transparently for included resources (images, CSS etc.)


This sounds really cool!

So given I have a PR, for example, 
https://github.com/D-Programming-Language/druntime/pull/1301

How do I find the ddoc build for it?

-Steve
Jun 11 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 12/06/2015 12:44 a.m., Steven Schveighoffer wrote:

 On 6/11/15 8:21 AM, Vladimir Panteleev wrote:

 I've put together a CI system of sorts that builds the documentation for
 all pull requests. Hopefully this should avoid the dlang.org build
 breaking again in the future. It integrates with GitHub as well, and on
 repositories covered by Brad's auto-tester, will create a second status
 for the documentation. The whole thing is Digger-powered, so uses the
 same code as Digger and http://digger.k3.1azy.net/trend/.

 Sample page:
 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/



 Source code: https://github.com/CyberShadow/DAutoTest

 Cool fact: When you request a built file (e.g. a .html page), it'll
 fetch the data directly from the Git cache repository, through a named
 pipe (bypassing disk) connected to a long-living Git process. This is
 also done transparently for included resources (images, CSS etc.)


 This sounds really cool!

 So given I have a PR, for example,
 https://github.com/D-Programming-Language/druntime/pull/1301

 How do I find the ddoc build for it?

 -Steve


Doesn't look like it has yet been added. Phobos works.
Jun 11 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/11/15 8:47 AM, Rikki Cattermole wrote:

 Doesn't look like it has yet been added. Phobos works.


Nice! I see it now. Also enabled for dlang.org.

Very cool, exactly the right UI IMO :)

-Steve
Jun 11 2015
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 11 June 2015 at 12:44:32 UTC, Steven Schveighoffer 
wrote:

 On 6/11/15 8:21 AM, Vladimir Panteleev wrote:

 [...]


 This sounds really cool!

 So given I have a PR, for example, 
 https://github.com/D-Programming-Language/druntime/pull/1301

 How do I find the ddoc build for it?

 -Steve


it doesn't seem to have triggered/finished yet for that one, but 
older pulls have a link in them
Jun 11 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/11/15 9:04 AM, John Colvin wrote:

 On Thursday, 11 June 2015 at 12:44:32 UTC, Steven Schveighoffer wrote:

 On 6/11/15 8:21 AM, Vladimir Panteleev wrote:

 [...]


 This sounds really cool!

 So given I have a PR, for example,
 https://github.com/D-Programming-Language/druntime/pull/1301

 How do I find the ddoc build for it?


 it doesn't seem to have triggered/finished yet for that one, but older
 pulls have a link in them


Yeah, hah! I seemed to have picked the only one that doesn't :)

It was the siren of seeing "DDOC:" in the title that drew me to it :P

Vladimir, this isn't a bug is it?

-Steve
Jun 11 2015
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 11 June 2015 at 13:12:17 UTC, Steven Schveighoffer 
wrote:

 On 6/11/15 9:04 AM, John Colvin wrote:

 it doesn't seem to have triggered/finished yet for that one, 
 but older
 pulls have a link in them


 Yeah, hah! I seemed to have picked the only one that doesn't :)

 It was the siren of seeing "DDOC:" in the title that drew me to 
 it :P

 Vladimir, this isn't a bug is it?


The order in which it picks PRs to test is a bit naive, I'll have 
to tweak it a bit I guess. Right now it goes through each D 
component, and for it goes through all open PRs. This takes a few 
hours. If a PR has been added after it grabbed the PR list, it 
won't get to it until the next iteration.
Jun 11 2015
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 11 June 2015 at 19:36:02 UTC, Vladimir Panteleev 
wrote:

 On Thursday, 11 June 2015 at 13:12:17 UTC, Steven Schveighoffer 
 wrote:

 Yeah, hah! I seemed to have picked the only one that doesn't :)

 It was the siren of seeing "DDOC:" in the title that drew me 
 to it :P

 Vladimir, this isn't a bug is it?


 The order in which it picks PRs to test is a bit naive, I'll 
 have to tweak it a bit I guess. Right now it goes through each 
 D component, and for it goes through all open PRs. This takes a 
 few hours. If a PR has been added after it grabbed the PR list, 
 it won't get to it until the next iteration.


It's all caught up now. I've tweaked the PR test order as well.
Jun 11 2015
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 11-Jun-2015 15:21, Vladimir Panteleev wrote:

 I've put together a CI system of sorts that builds the documentation for
 all pull requests. Hopefully this should avoid the dlang.org build
 breaking again in the future. It integrates with GitHub as well, and on
 repositories covered by Brad's auto-tester, will create a second status
 for the documentation. The whole thing is Digger-powered, so uses the
 same code as Digger and http://digger.k3.1azy.net/trend/.

 Sample page:
 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/


 Source code: https://github.com/CyberShadow/DAutoTest

 Cool fact: When you request a built file (e.g. a .html page), it'll
 fetch the data directly from the Git cache repository, through a named
 pipe (bypassing disk) connected to a long-living Git process. This is
 also done transparently for included resources (images, CSS etc.)


Awesome. This is going to be a huge time saver.


-- 
Dmitry Olshansky
Jun 11 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/11/15 8:21 AM, Vladimir Panteleev wrote:

 I've put together a CI system of sorts that builds the documentation for
 all pull requests. Hopefully this should avoid the dlang.org build
 breaking again in the future. It integrates with GitHub as well, and on
 repositories covered by Brad's auto-tester, will create a second status
 for the documentation. The whole thing is Digger-powered, so uses the
 same code as Digger and http://digger.k3.1azy.net/trend/.

 Sample page:
 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/


 Source code: https://github.com/CyberShadow/DAutoTest

 Cool fact: When you request a built file (e.g. a .html page), it'll
 fetch the data directly from the Git cache repository, through a named
 pipe (bypassing disk) connected to a long-living Git process. This is
 also done transparently for included resources (images, CSS etc.)


On a PR I was doing for ddoc, the "old" and "new" seem to be reversed:

http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/6d3c95ecc5422ae1d5bd6f3308da78b733175ba7/

-Steve
Jun 11 2015
parent reply David Gileadi <gileadis NSPMgmail.com> writes:
On 6/11/15 7:04 AM, Steven Schveighoffer wrote:

 On 6/11/15 8:21 AM, Vladimir Panteleev wrote:

 I've put together a CI system of sorts that builds the documentation for
 all pull requests. Hopefully this should avoid the dlang.org build
 breaking again in the future. It integrates with GitHub as well, and on
 repositories covered by Brad's auto-tester, will create a second status
 for the documentation. The whole thing is Digger-powered, so uses the
 same code as Digger and http://digger.k3.1azy.net/trend/.

 Sample page:
 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/



 Source code: https://github.com/CyberShadow/DAutoTest

 Cool fact: When you request a built file (e.g. a .html page), it'll
 fetch the data directly from the Git cache repository, through a named
 pipe (bypassing disk) connected to a long-living Git process. This is
 also done transparently for included resources (images, CSS etc.)


 On a PR I was doing for ddoc, the "old" and "new" seem to be reversed:

 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/6d3c95ecc5422ae1d5bd6f3308da78b733175ba7/


While randomly browsing through for a PR to see this magic on, I ran 
across an error page:

http://dtest.thecybershadow.net/artifact/website-9bbe7b16aaa20f28b879722609308a024a12ae52-005ef14643f10a0919ace4bd80bcfa44/web/library-prerelease/std/range/merge.html

I get it when I click on the New link for merge.html on this page:

http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/82878d02c06ec112663fd953e78d541dda69a38e/

I believe that, like Steven says, the Old and New links are reversed 
(since the diff shows that merge.html to be new in this PR but the new 
page shows up under the Old link), and perhaps the tool should better 
handle added/deleted files.

In any case this is a seriously awesome tool!
Jun 11 2015
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 11 June 2015 at 14:45:54 UTC, David Gileadi wrote:

 I believe that, like Steven says, the Old and New links are 
 reversed (since the diff shows that merge.html to be new in 
 this PR but the new page shows up under the Old link), and 
 perhaps the tool should better handle added/deleted files.


Yeah, I got the order wrong. Fixed now.
Jun 11 2015
prev sibling next sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
This looks like a rather good start.  Thanks for taking on this task.

On 6/11/2015 5:21 AM, Vladimir Panteleev via Digitalmars-d wrote:

 I've put together a CI system of sorts that builds the documentation 
 for all pull requests. Hopefully this should avoid the dlang.org build 
 breaking again in the future. It integrates with GitHub as well, and 
 on repositories covered by Brad's auto-tester, will create a second 
 status for the documentation. The whole thing is Digger-powered, so 
 uses the same code as Digger and http://digger.k3.1azy.net/trend/.

 Sample page: 
 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/

 Source code: https://github.com/CyberShadow/DAutoTest

 Cool fact: When you request a built file (e.g. a .html page), it'll 
 fetch the data directly from the Git cache repository, through a named 
 pipe (bypassing disk) connected to a long-living Git process. This is 
 also done transparently for included resources (images, CSS etc.)
Jun 11 2015
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, 11 June 2015 at 20:35:10 UTC, Brad Roberts wrote:

 This looks like a rather good start.  Thanks for taking on this 
 task.


Indeed. Cool stuff. Thanks for the effort.

- Jonathan M Davis
Jun 11 2015
prev sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 11.06.2015 um 14:21 schrieb Vladimir Panteleev:

 I've put together a CI system of sorts that builds the documentation for
 all pull requests. Hopefully this should avoid the dlang.org build
 breaking again in the future. It integrates with GitHub as well, and on
 repositories covered by Brad's auto-tester, will create a second status
 for the documentation. The whole thing is Digger-powered, so uses the
 same code as Digger and http://digger.k3.1azy.net/trend/.

 Sample page:
 http://dtest.thecybershadow.net/results/a6e6883454a4d62c8d4b41819b9a7e33899b6222/31744d2fb90cf25ef1778f52ef640abc56c494c2/


 Source code: https://github.com/CyberShadow/DAutoTest


This is really great to have, thanks a lot for making this happen! (I'm 
really looking forward to not getting Ddox related e-mail before each 
release ;) But especially the preview feature should be really useful 
for doing visual reviews.


 Cool fact: When you request a built file (e.g. a .html page), it'll
 fetch the data directly from the Git cache repository, through a named
 pipe (bypassing disk) connected to a long-living Git process. This is
 also done transparently for included resources (images, CSS etc.)


Have you thought about using dlibgit [1] instead? Well, I guess that 
because a separate thread is necessary to not block the event loop, the 
performance benefit wouldn't really be a compelling argument... but at 
least it should be more convenient to use.

[1]: http://code.dlang.org/packages/dlibgit
Jun 11 2015
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 11 June 2015 at 20:41:14 UTC, Sönke Ludwig wrote:

 Have you thought about using dlibgit [1] instead? Well, I guess 
 that because a separate thread is necessary to not block the 
 event loop, the performance benefit wouldn't really be a 
 compelling argument... but at least it should be more 
 convenient to use.


Ha, yes, you're right. I should've been using it since a long 
time ago.
Jun 11 2015