www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Build farm(er)

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Hello,


I was talking yesterday with Walter and Brad about enacting a build farm 
and the build master who would tend to it.

The basic idea is to amass a number of remote machines (real and/or 
virtual) running various OSs and use them for building and testing 
recent commits of dmd and phobos. The website will have a "cutting-edge" 
downloadable package that contains the most recent successful build. At 
the end of the day I want us to get to the point where I type:

% make world

and everything is built from source all the way to uploading the 
installers for all OSs to the website and updating html files accordingly.

Walter and I agreed that wine should be adequate for building and 
testing on Windows, which is helpful because we can use the Unix command 
line tools throughout.

The build master will act as a coordinator of check ins, reviews, 
builds, test harnesses, and probably more. Walter is currently handling 
these roles, and he would be relieved to pass them on to someone reliable.

Any ideas, proposals, contributions and candidates are welcome!


Andrei
Aug 05 2010
next sibling parent reply BCS <none anon.com> writes:
Hello Andrei,


 Hello,
 
 I was talking yesterday with Walter and Brad about enacting a build
 farm and the build master who would tend to it.


I'd be interested in partaking in such an endeavor. I was actually thinking 
of doing something related (automatic regression testing). I don't think 
I'd be a good candidate for maintaining the whole things but I'd be willing 
to help.
 

 The basic idea is to amass a number of remote machines (real and/or
 virtual) running various OSs and use them for building and testing
 recent commits of dmd and phobos. The website will have a
 "cutting-edge" downloadable package that contains the most recent
 successful build. At the end of the day I want us to get to the point
 where I type:
 
 % make world
 
 and everything is built from source all the way to uploading the
 installers for all OSs to the website and updating html files
 accordingly.
 
 Walter and I agreed that wine should be adequate for building and
 testing on Windows, which is helpful because we can use the Unix
 command line tools throughout.
 
 The build master will act as a coordinator of check ins, reviews,
 builds, test harnesses, and probably more. Walter is currently
 handling these roles, and he would be relieved to pass them on to
 someone reliable.
 
 Any ideas, proposals, contributions and candidates are welcome!
 
 Andrei
 


-- 
... <IXOYE><
Aug 05 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/06/2010 12:00 AM, BCS wrote:

 Hello Andrei,


 Hello,

 I was talking yesterday with Walter and Brad about enacting a build
 farm and the build master who would tend to it.


 I'd be interested in partaking in such an endeavor. I was actually
 thinking of doing something related (automatic regression testing). I
 don't think I'd be a good candidate for maintaining the whole things but
 I'd be willing to help.


That's great, thanks for volunteering. Automatic regression testing 
sounds like a good starting point, feel free to start on it as you find 
fit and post your progress. You may want to get familiar with the build 
and test processes of dmd and phobos if you aren't.

In terms of machines, if anyone could volunteer ssh accounts to stable 
machines (e.g. preferably not laptops and not connected via dynamic IP, 
though the latter is satisfactorily fixed by dyndns.org), please chime in!

Andrei
Aug 06 2010
next sibling parent reply BCS <none anon.com> writes:
Hello Andrei,


 On 08/06/2010 12:00 AM, BCS wrote:
 

 Hello Andrei,
 

 Hello,
 
 I was talking yesterday with Walter and Brad about enacting a build
 farm and the build master who would tend to it.
 


 I'd be interested in partaking in such an endeavor. I was actually
 thinking of doing something related (automatic regression testing). I
 don't think I'd be a good candidate for maintaining the whole things
 but I'd be willing to help.
 


 That's great, thanks for volunteering. Automatic regression testing
 sounds like a good starting point, feel free to start on it as you
 find fit and post your progress. You may want to get familiar with the
 build and test processes of dmd and phobos if you aren't.
 
 In terms of machines, if anyone could volunteer ssh accounts to stable
 machines (e.g. preferably not laptops and not connected via dynamic
 IP, though the latter is satisfactorily fixed by dyndns.org), please
 chime in!


Where will the build system reside? in the same repo and DMD or Phobos or 
in it's own?

-- 
... <IXOYE><
Aug 06 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
BCS wrote:

 Hello Andrei,
 

 On 08/06/2010 12:00 AM, BCS wrote:


 Hello Andrei,


 Hello,

 I was talking yesterday with Walter and Brad about enacting a build
 farm and the build master who would tend to it.


 I'd be interested in partaking in such an endeavor. I was actually
 thinking of doing something related (automatic regression testing). I
 don't think I'd be a good candidate for maintaining the whole things
 but I'd be willing to help.


 That's great, thanks for volunteering. Automatic regression testing
 sounds like a good starting point, feel free to start on it as you
 find fit and post your progress. You may want to get familiar with the
 build and test processes of dmd and phobos if you aren't.

 In terms of machines, if anyone could volunteer ssh accounts to stable
 machines (e.g. preferably not laptops and not connected via dynamic
 IP, though the latter is satisfactorily fixed by dyndns.org), please
 chime in!


 
 Where will the build system reside? in the same repo and DMD or Phobos 
 or in it's own?


I suggest you start experimenting in a separate project.

Andrei
Aug 06 2010
prev sibling next sibling parent reply Tomek =?UTF-8?B?U293acWEc2tp?= <just ask.me> writes:
Andrei Alexandrescu napisał:


 In terms of machines, if anyone could volunteer ssh accounts to stable
 machines (e.g. preferably not laptops and not connected via dynamic IP,
 though the latter is satisfactorily fixed by dyndns.org), please chime in!


It would be a shame to leave out laptops. Ideally if someone lights up a
computer with a build slave on it, the master could automatically 
delegate two or three builds on their lousy laptop while they're reading some
article. Does anyone know a build system that supports that 
kind of dynamic work distribution? I know Hudson has slaves but AFAIK you have
to register each one manually on the master. For 
thousands of personal comps going on- and offline we need automization.


Tomek
Aug 06 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/07/2010 01:40 AM, Tomek Sowiński wrote:

 Andrei Alexandrescu napisał:


 In terms of machines, if anyone could volunteer ssh accounts to stable
 machines (e.g. preferably not laptops and not connected via dynamic IP,
 though the latter is satisfactorily fixed by dyndns.org), please chime in!


 It would be a shame to leave out laptops. Ideally if someone lights up a
computer with a build slave on it, the master could automatically
 delegate two or three builds on their lousy laptop while they're reading some
article. Does anyone know a build system that supports that
 kind of dynamic work distribution? I know Hudson has slaves but AFAIK you have
to register each one manually on the master. For
 thousands of personal comps going on- and offline we need automization.


When we'll have thousands, sure, laptops would be great. So far we're 
keen about the 0->1 transition :o).

Andrei
Aug 07 2010
prev sibling parent Lionello Lunesu <lio lunesu.remove.com> writes:
On 2010-08-06 5:44, Andrei Alexandrescu wrote:

 On 08/06/2010 12:00 AM, BCS wrote:

 Hello Andrei,


 Hello,

 I was talking yesterday with Walter and Brad about enacting a build
 farm and the build master who would tend to it.


 I'd be interested in partaking in such an endeavor. I was actually
 thinking of doing something related (automatic regression testing). I
 don't think I'd be a good candidate for maintaining the whole things but
 I'd be willing to help.


 That's great, thanks for volunteering. Automatic regression testing
 sounds like a good starting point, feel free to start on it as you find
 fit and post your progress. You may want to get familiar with the build
 and test processes of dmd and phobos if you aren't.

 In terms of machines, if anyone could volunteer ssh accounts to stable
 machines (e.g. preferably not laptops and not connected via dynamic IP,
 though the latter is satisfactorily fixed by dyndns.org), please chime in!

 Andrei


I can offer shell access to my VPS: Ubuntu 8.04 LTS with static IP. 
Regular user access is enough?

L.
Aug 07 2010
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-08-06 03:59, Andrei Alexandrescu wrote:

 Hello,


 I was talking yesterday with Walter and Brad about enacting a build farm
 and the build master who would tend to it.

 The basic idea is to amass a number of remote machines (real and/or
 virtual) running various OSs and use them for building and testing
 recent commits of dmd and phobos. The website will have a "cutting-edge"
 downloadable package that contains the most recent successful build. At
 the end of the day I want us to get to the point where I type:

 % make world

 and everything is built from source all the way to uploading the
 installers for all OSs to the website and updating html files accordingly.

 Walter and I agreed that wine should be adequate for building and
 testing on Windows, which is helpful because we can use the Unix command
 line tools throughout.

 The build master will act as a coordinator of check ins, reviews,
 builds, test harnesses, and probably more. Walter is currently handling
 these roles, and he would be relieved to pass them on to someone reliable.

 Any ideas, proposals, contributions and candidates are welcome!


 Andrei


Sounds like a good idea.

-- 
/Jacob Carlborg
Aug 06 2010
prev sibling next sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
Andrei Alexandrescu wrote:


 Hello,
 
 
 I was talking yesterday with Walter and Brad about enacting a build farm
 and the build master who would tend to it.
 
 The basic idea is to amass a number of remote machines (real and/or
 virtual) running various OSs and use them for building and testing
 recent commits of dmd and phobos. The website will have a "cutting-edge"
 downloadable package that contains the most recent successful build. At
 the end of the day I want us to get to the point where I type:
 
 % make world
 
 and everything is built from source all the way to uploading the
 installers for all OSs to the website and updating html files accordingly.
 
 Walter and I agreed that wine should be adequate for building and
 testing on Windows, which is helpful because we can use the Unix command
 line tools throughout.
 
 The build master will act as a coordinator of check ins, reviews,
 builds, test harnesses, and probably more. Walter is currently handling
 these roles, and he would be relieved to pass them on to someone reliable.
 
 Any ideas, proposals, contributions and candidates are welcome!
 
 
 Andrei


Have you considered going with an existing continuous integration package, such 
as Hudson? I am in the process of setting one up (but using CruiseControl.NET), 
it does offer some extra value at the expense of an intial setup cost. The 
benefit of such system are better reporting, automated builds and
notifications. 
Hudson should be able to integrate information from svn and bugzilla for 
example.

http://hudson-ci.org/
Aug 06 2010
prev sibling next sibling parent reply Tomek =?UTF-8?B?U293acWEc2tp?= <just ask.me> writes:
Andrei Alexandrescu napisał:


 Hello,
 
 
 I was talking yesterday with Walter and Brad about enacting a build farm
 and the build master who would tend to it.
 
 The basic idea is to amass a number of remote machines (real and/or
 virtual) running various OSs and use them for building and testing
 recent commits of dmd and phobos. The website will have a "cutting-edge"
 downloadable package that contains the most recent successful build. At
 the end of the day I want us to get to the point where I type:
 
 % make world
 
 and everything is built from source all the way to uploading the
 installers for all OSs to the website and updating html files accordingly.
 
 Walter and I agreed that wine should be adequate for building and
 testing on Windows, which is helpful because we can use the Unix command
 line tools throughout.
 
 The build master will act as a coordinator of check ins, reviews,
 builds, test harnesses, and probably more. Walter is currently handling
 these roles, and he would be relieved to pass them on to someone reliable.
 
 Any ideas, proposals, contributions and candidates are welcome!


An idea: could the assembled farm be reused to build other D projects? Being
able to easily run thorough unittests on a variety of 
machines would be a massive boost in quality of D's software ecosystem.


Tomek
Aug 06 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Tomek Sowiński wrote:

 Andrei Alexandrescu napisał:
 

 Hello,


 I was talking yesterday with Walter and Brad about enacting a build farm
 and the build master who would tend to it.

 The basic idea is to amass a number of remote machines (real and/or
 virtual) running various OSs and use them for building and testing
 recent commits of dmd and phobos. The website will have a "cutting-edge"
 downloadable package that contains the most recent successful build. At
 the end of the day I want us to get to the point where I type:

 % make world

 and everything is built from source all the way to uploading the
 installers for all OSs to the website and updating html files accordingly.

 Walter and I agreed that wine should be adequate for building and
 testing on Windows, which is helpful because we can use the Unix command
 line tools throughout.

 The build master will act as a coordinator of check ins, reviews,
 builds, test harnesses, and probably more. Walter is currently handling
 these roles, and he would be relieved to pass them on to someone reliable.

 Any ideas, proposals, contributions and candidates are welcome!


 
 An idea: could the assembled farm be reused to build other D projects? Being
able to easily run thorough unittests on a variety of 
 machines would be a massive boost in quality of D's software ecosystem.


I think that's a great idea. We only need the appropriate people power 
and computer power. In an ideal world dsource.org would grow to provide 
that, but my perception is that Brad A. can only put this much time into 
it (which is a lot btw).

Andrei
Aug 06 2010
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Tomek Sowiński wrote:

 An idea: could the assembled farm be reused to build other D projects? Being
 able to easily run thorough unittests on a variety of machines would be a
 massive boost in quality of D's software ecosystem.


I agree. We have a constant problem with fixes on one platform breaking on 
others because the developer doesn't have immediate access to those others.
Aug 06 2010
prev sibling next sibling parent SK <sk metrokings.com> writes:
Hello Andrei,

On Thu, Aug 5, 2010 at 6:59 PM, Andrei Alexandrescu wrote:

 and everything is built from source all the way to uploading the installers
 for all OSs to the website and updating html files accordingly.


How about CDash?  http://www.cdash.org/cdash/project/about.html
CMake+CTest+CDash provide integrated build, test and reporting.  For
example, here the dashboard for CDash itself:
http://www.cdash.org/CDash/index.php?project=CDash

That said, I have not tried CDash myself, though I'm a user and fan of
the first two pieces: CMake and CTest.  Speaking of which, I'm working
on CMake for D2, which seems to have lost its maintainer.

Also, I highly recommend using virtual machines rather than bare metal
for a build farm -- easier and much more economical.
Regards,
-steve
Aug 06 2010
prev sibling next sibling parent reply sybrandy <sybrandy gmail.com> writes:
I can't help with hardware, but if you want to virtualize some of your 
operating systems, VirtualBox is very nice and freely available.  Also, 
and this is just me throwing out something new, you could write some 
small Erlang programs that are linked to each other to manage the 
distribution.  Toss a web interface built using Yaws or Erlyweb and you 
can have a web client that is built in and talks Erlang.

Of course, if something already exists that does it, that we should 
probably use it unless it's just too bad to use.

Casey
Aug 06 2010
parent Jacob Carlborg <doob me.com> writes:
On 2010-08-06 23:10, sybrandy wrote:

 I can't help with hardware, but if you want to virtualize some of your
 operating systems, VirtualBox is very nice and freely available. Also,
 and this is just me throwing out something new, you could write some
 small Erlang programs that are linked to each other to manage the
 distribution. Toss a web interface built using Yaws or Erlyweb and you
 can have a web client that is built in and talks Erlang.

 Of course, if something already exists that does it, that we should
 probably use it unless it's just too bad to use.

 Casey


If I recall correctly, according to Apple's licenses it's only legal to 
virtualize Mac OS X server.

-- 
/Jacob Carlborg
Aug 07 2010
prev sibling next sibling parent Byron Heads <bheads emich.edu> writes:
On Thu, 05 Aug 2010 20:59:28 -0500, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> wrote:

 The basic idea is to amass a number of remote machines (real and/or 
 virtual) running various OSs and use them for building and testing 
 recent commits of dmd and phobos. The website will have a 


"cutting-edge" 

 downloadable package that contains the most recent successful 


build. At 

 the end of the day I want us to get to the point where I type:


I can give you a regular ssh account on an debian server I have setup 
for students here at EMU.  It is dell power edge.

--
Sent from my droid.
Aug 10 2010
prev sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 06/08/2010 02:59, Andrei Alexandrescu wrote:

 Hello,


 I was talking yesterday with Walter and Brad about enacting a build farm
 and the build master who would tend to it.

 The basic idea is to amass a number of remote machines (real and/or
 virtual) running various OSs and use them for building and testing
 recent commits of dmd and phobos. The website will have a "cutting-edge"
 downloadable package that contains the most recent successful build. At
 the end of the day I want us to get to the point where I type:

 % make world

 and everything is built from source all the way to uploading the
 installers for all OSs to the website and updating html files accordingly.

 Walter and I agreed that wine should be adequate for building and
 testing on Windows, which is helpful because we can use the Unix command
 line tools throughout.

 The build master will act as a coordinator of check ins, reviews,
 builds, test harnesses, and probably more. Walter is currently handling
 these roles, and he would be relieved to pass them on to someone reliable.

 Any ideas, proposals, contributions and candidates are welcome!


 Andrei


Continuous integration, pretty nice! I hope someone takes up on that task.

I myself want to learn and gain expertise (hopefully on the near future) 
on that general area: automated building and testing, across multiple 
platforms.

-- 
Bruno Medeiros - Software Engineer
Sep 30 2010