www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Auto-testing of GitHub pull requests

reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
Hello all,

GitHub very nicely ensures that all pull requests to dmd, druntime and phobos 
are sent to the auto-tester and that success/failure reports are sent back.

But ... how is this done?  Is the auto-tester entirely custom written or does
it 
use some standard tools?  And how is it hooked into GitHub?

Thanks & best wishes,

     -- Joe
Sep 24 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-24 12:25, Joseph Rushton Wakeling wrote:

 Hello all,

 GitHub very nicely ensures that all pull requests to dmd, druntime and
 phobos are sent to the auto-tester and that success/failure reports are
 sent back.

 But ... how is this done?  Is the auto-tester entirely custom written or
 does it use some standard tools?  And how is it hooked into GitHub?


It's custom. I would guess it either uses a git hook or it uses some API 
provided by Github.

-- 
/Jacob Carlborg
Sep 24 2013
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 24/09/13 13:01, Jacob Carlborg wrote:

 It's custom. I would guess it either uses a git hook or it uses some API
 provided by Github.


Ahh, OK.  I was wondering if it used some standard tool to manage starting the 
test jobs and reporting back the tests -- something like Gerrit, Jenkins, etc. 
I don't have any personal experience of these so couldn't tell on the basis of 
the test website.

Thanks for the info! :-)
Sep 24 2013
prev sibling next sibling parent reply Brad Roberts <braddr puremagic.com> writes:
On 9/24/13 4:29 AM, Joseph Rushton Wakeling wrote:

 On 24/09/13 13:01, Jacob Carlborg wrote:

 It's custom. I would guess it either uses a git hook or it uses some API
 provided by Github.


 Ahh, OK.  I was wondering if it used some standard tool to manage starting the
test jobs and
 reporting back the tests -- something like Gerrit, Jenkins, etc. I don't have
any personal
 experience of these so couldn't tell on the basis of the test website.

 Thanks for the info! :-)


https://github.com/braddr/d-tester

I'd use a standard tool (or would have, silly to switch when this one does the
job) but none of them 
support the combination of features used.  Primarily: github, testing on
multiple platforms, testing 
pulls as merges into the target branch (rather than just the pull itself),
multiple repositories in 
the same build.  There's probably more requirements, it's been a while since I
researched the state 
of the ecosystem and exactly what the gaps are.

But now that it's written, it's just fundamentally not that much code, nor is
it actually difficult 
code.
Sep 24 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-24 18:25, Brad Roberts wrote:


 https://github.com/braddr/d-tester


Could you add a link to this on the test results page? I always forget 
where this code is located.

-- 
/Jacob Carlborg
Sep 24 2013
parent reply "simendsjo" <simendsjo gmail.com> writes:
On Wednesday, 25 September 2013 at 06:51:44 UTC, Jacob Carlborg 
wrote:

 On 2013-09-24 18:25, Brad Roberts wrote:


 https://github.com/braddr/d-tester


 Could you add a link to this on the test results page? I always 
 forget where this code is located.


http://d.puremagic.com/test-results/
Sep 25 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-25 09:30, simendsjo wrote:


 http://d.puremagic.com/test-results/


I know where the test results are. But, what I'm forgetting is where the 
code is which implements these test results. I want this page:

http://d.puremagic.com/test-results/

To have a link to this:

https://github.com/braddr/d-tester

-- 
/Jacob Carlborg
Sep 25 2013
parent "simendsjo" <simendsjo gmail.com> writes:
On Wednesday, 25 September 2013 at 11:29:16 UTC, Jacob Carlborg 
wrote:

 On 2013-09-25 09:30, simendsjo wrote:


 http://d.puremagic.com/test-results/


 I know where the test results are. But, what I'm forgetting is 
 where the code is which implements these test results. I want 
 this page:

 http://d.puremagic.com/test-results/

 To have a link to this:

 https://github.com/braddr/d-tester


Seems like I didn't actually read you post :)
Sep 25 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 24/09/13 18:25, Brad Roberts wrote:

 I'd use a standard tool (or would have, silly to switch when this one does the
 job) but none of them support the combination of features used.  Primarily:
 github, testing on multiple platforms, testing pulls as merges into the target
 branch (rather than just the pull itself), multiple repositories in the same
 build.  There's probably more requirements, it's been a while since I
researched
 the state of the ecosystem and exactly what the gaps are.

 But now that it's written, it's just fundamentally not that much code, nor is
it
 actually difficult code.


Thanks very much for pointing me to that.  I'm looking into auto-testing stuff 
that would work with a project hosted on Gitorious (GitHub is unsatisfactory
for 
political reasons) -- I'm working on the assumption that one of the standard 
testing frameworks will play nice with that (I've seen a writeup on Gerrit + 
Gitorious), but assuming anything custom is required, this could be a very 
useful template.
Sep 25 2013
prev sibling parent Brad Roberts <braddr puremagic.com> writes:
On 9/25/13 2:58 PM, Joseph Rushton Wakeling wrote:

 On 24/09/13 18:25, Brad Roberts wrote:

 I'd use a standard tool (or would have, silly to switch when this one does the
 job) but none of them support the combination of features used.  Primarily:
 github, testing on multiple platforms, testing pulls as merges into the target
 branch (rather than just the pull itself), multiple repositories in the same
 build.  There's probably more requirements, it's been a while since I
researched
 the state of the ecosystem and exactly what the gaps are.

 But now that it's written, it's just fundamentally not that much code, nor is
it
 actually difficult code.


 Thanks very much for pointing me to that.  I'm looking into auto-testing stuff
that would work with
 a project hosted on Gitorious (GitHub is unsatisfactory for political reasons)
-- I'm working on the
 assumption that one of the standard testing frameworks will play nice with
that (I've seen a writeup
 on Gerrit + Gitorious), but assuming anything custom is required, this could
be a very useful template.


The only parts that really hook into github are:

1) serverd/update_pulls.d -- That's a cronjob that polls github ever few
minutes for the current set 
of pull requests.  It's pretty ghetto due to evolution and lack of time
investment.  It really ought 
to be replaced with a webhook that github posts to and just left around as a
backup in case a hook 
call gets lost.

2) server/add/github-post.ghtml -- That's called by a github web hook when the
master branches are 
checked into.  Ignore the language it's written in.  The gist is easy to
understand: grab what's 
been updated, flush old state in the db.

Unless I'm forgetting something, all the rest of the code is the tester itself:

a) client -- the code that executes on the clients

b) server -- the web ui, written in an ancient scripting language that I still
use from a previous 
job.  Don't use as is unless you enjoy reverse engineering interpreters.

c) serverd -- the apis that the client code calls to ask about work and report
status.

And there's a massive lack of ui surrounding the configuration tables.  I keep
putting off working 
on them since it's awfully rare that any of them need to be updated, but I
always hate running sql 
directly against a live data store.


In case stats are of interesting to anyone (they are to me):


    741000 pull          (first one on Dec 14, 2011)
     51000 master branch (first one on Sept 5th, 2010 -- just over 3 years ago)

Users with 100 or more closed pull requests (note, data doesn't cover a small
number pull requests 
that pre-dates the tester)
+---------+----------------+----------+
| user_id | username       | count(*) |
+---------+----------------+----------+
|  254394 | 9rnsr          |     1068 |
|  288976 | dawgfoto       |      328 |
|  580100 | yebblies       |      306 |
|  567837 | donc           |      283 |
|  568298 | WalterBright   |      229 |
|  360179 | AndrejMitrovic |      226 |
|  579956 | jmdavis        |      167 |
|   44076 | alexrp         |      152 |
| 1911406 | monarchdodra   |      129 |
|   40242 | braddr         |      123 |
+---------+----------------+----------+
Sep 25 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 24 September 2013 at 10:25:21 UTC, Joseph Rushton 
Wakeling wrote:

 Hello all,

 GitHub very nicely ensures that all pull requests to dmd, 
 druntime and phobos are sent to the auto-tester and that 
 success/failure reports are sent back.

 But ... how is this done?  Is the auto-tester entirely custom 
 written or does it use some standard tools?  And how is it 
 hooked into GitHub?

 Thanks & best wishes,

     -- Joe


D one is custom but most major CI tools have such functionality 
(I personally have used Jenkins). It uses GitHub application API 
and needs to be explicitly allowed by repository owner.
Sep 24 2013