www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [upforgrabs] tools/update

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
We need a script to update all of dmd's paraphernalia. By this I'm 
putting the task up for grabs and am describing it.

The script would require git to be installed, and would orchestrate git 
commands to freshen the installation. Either a tag can be given, or the 
script would automatically detect the latest "semistable" release, or 
even the script can be told to get the latest and greatest. The exact 
tagging convention is to be defined later.

The trick with such tools is they must "just work". That means wherever 
and however you installed dmd, the script should be able to detect 
everything it needs for a successful update. This means: dmd and rdmd 
binary locations, druntime and phobos library locations, and druntime 
and phobos import locations.

The way I'm thinking this all can be done is by running dmd -v against a 
small do-nothing program that imports one std module, for example 
std.stdio. The run will print (among other things):

binary    dmd
version   v2.054
config    /Users/john/dmd.conf
parse     test
importall test
import    object	(/Users/john/code/dmd/druntime/import/object.di)
import    std.stdio	(/Users/john/code/dmd/phobos/std/stdio.d)
...

Simple parsing reveals the location of the configuration file and the 
locations of druntime and std imports. Further simple parsing of the 
conf file reveals the location of libraries.

With this information in hand, the script goes and fetches the 
appropriate code in the appropriate places.

Also, the tool needs to do all of its work transactionally, i.e. first 
download everything next to the target with an added suffix (e.g. 
".tmp"). Only after a full successful download, the tool would rename 
everything .tmp to its final name (an operation that can't fail under 
most circumstances).

If anyone is willing to take up working on this, it would be great. D 
itself is a great candidate as the language to write the tool in, and 
has the advantage of being more portable than shell scripts.

We'll add the tool to the tools/ repository and offer it with the 
standard distribution.


Thanks,

Andrei
Jun 10 2011
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I thought this is what DVM already does? Or are you talking about
fetching unstable branches and compiling/installing them?
Jun 10 2011
prev sibling next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 10/06/2011 15:14, Andrei Alexandrescu wrote:

 We need a script to update all of dmd's paraphernalia. By this I'm
 putting the task up for grabs and am describing it.

 The script would require git to be installed, and would orchestrate git
 commands to freshen the installation. Either a tag can be given, or the
 script would automatically detect the latest "semistable" release, or
 even the script can be told to get the latest and greatest. The exact
 tagging convention is to be defined later.

 The trick with such tools is they must "just work". That means wherever
 and however you installed dmd, the script should be able to detect
 everything it needs for a successful update. This means: dmd and rdmd
 binary locations, druntime and phobos library locations, and druntime
 and phobos import locations.

 The way I'm thinking this all can be done is by running dmd -v against a
 small do-nothing program that imports one std module, for example
 std.stdio. The run will print (among other things):

 binary dmd
 version v2.054
 config /Users/john/dmd.conf
 parse test
 importall test
 import object (/Users/john/code/dmd/druntime/import/object.di)
 import std.stdio (/Users/john/code/dmd/phobos/std/stdio.d)
 ...

 Simple parsing reveals the location of the configuration file and the
 locations of druntime and std imports. Further simple parsing of the
 conf file reveals the location of libraries.

 With this information in hand, the script goes and fetches the
 appropriate code in the appropriate places.

 Also, the tool needs to do all of its work transactionally, i.e. first
 download everything next to the target with an added suffix (e.g.
 ".tmp"). Only after a full successful download, the tool would rename
 everything .tmp to its final name (an operation that can't fail under
 most circumstances).

 If anyone is willing to take up working on this, it would be great. D
 itself is a great candidate as the language to write the tool in, and
 has the advantage of being more portable than shell scripts.

 We'll add the tool to the tools/ repository and offer it with the
 standard distribution.


 Thanks,

 Andrei


This sounds like a rather limited tool, why limit it to dmd? The tool 
you describe is just a watered down version of a full package manager.

Also, as already mentioned, dvm does this.

-- 
Robert
http://octarineparrot.com/
Jun 10 2011
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Plus thanks to Nick dvm now works on Windoze.
Jun 10 2011
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/10/11 12:34 PM, Robert Clipsham wrote:

 This sounds like a rather limited tool, why limit it to dmd? The tool
 you describe is just a watered down version of a full package manager.


Then define a full package manager and have the tool use it.


 Also, as already mentioned, dvm does this.


 From what I saw it doesn't do all that.


Andrei
Jun 10 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-06-10 20:09, Andrei Alexandrescu wrote:

 On 6/10/11 12:34 PM, Robert Clipsham wrote:

 This sounds like a rather limited tool, why limit it to dmd? The tool
 you describe is just a watered down version of a full package manager.


 Then define a full package manager and have the tool use it.


 Also, as already mentioned, dvm does this.


  From what I saw it doesn't do all that.


 Andrei


What is it that you need it to do?

-- 
/Jacob Carlborg
Jun 17 2011
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message 
news:ist957$19nj$1 digitalmars.com...

 We need a script to update all of dmd's paraphernalia. By this I'm putting 
 the task up for grabs and am describing it.

 The script would require git to be installed, and would orchestrate git 
 commands to freshen the installation. Either a tag can be given, or the 
 script would automatically detect the latest "semistable" release, or even 
 the script can be told to get the latest and greatest. The exact tagging 
 convention is to be defined later.

 The trick with such tools is they must "just work". That means wherever 
 and however you installed dmd, the script should be able to detect 
 everything it needs for a successful update. This means: dmd and rdmd 
 binary locations, druntime and phobos library locations, and druntime and 
 phobos import locations.


I've already been planning to add the features of "recompile DMD" and "check 
out DMD from git and compile it" to DVM (unless Jacob plans to beat me to 
it). You do have good suggestions on some of the implementation details. If 
there's still another feature in your proposal that I've overlooked here, 
let me know.

I was going to wait to do this until Jacob merged in my Windows pull request 
(he said he'd get to it end of next week, IIRC), because I want to base it 
off a DVM that already has Windows support, but maybe I shouldn't wait and 
should just fork my "windows fork" and get to it? (I'm still fairly new to 
DVCS and "proper DVCS procedures".)

DVM is written in D1/Tango, though. Would that be a problem? If so, and if 
Jacob doesn't have any objections to it being ported to D2/Phobos, then I'd 
volunteer to do the port. I already have experience porting a bunch of my 
own code from D1/Tango to D2/Phobos. But I don't want to be hasty about 
porting it: I assume Jacob had a reason for using D1/Tango. I don't know how 
important the reason was.
Jun 10 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-06-10 22:09, Nick Sabalausky wrote:

 "Andrei Alexandrescu"<SeeWebsiteForEmail erdani.org>  wrote in message
 news:ist957$19nj$1 digitalmars.com...

 We need a script to update all of dmd's paraphernalia. By this I'm putting
 the task up for grabs and am describing it.

 The script would require git to be installed, and would orchestrate git
 commands to freshen the installation. Either a tag can be given, or the
 script would automatically detect the latest "semistable" release, or even
 the script can be told to get the latest and greatest. The exact tagging
 convention is to be defined later.

 The trick with such tools is they must "just work". That means wherever
 and however you installed dmd, the script should be able to detect
 everything it needs for a successful update. This means: dmd and rdmd
 binary locations, druntime and phobos library locations, and druntime and
 phobos import locations.


 I've already been planning to add the features of "recompile DMD" and "check
 out DMD from git and compile it" to DVM (unless Jacob plans to beat me to
 it). You do have good suggestions on some of the implementation details. If
 there's still another feature in your proposal that I've overlooked here,
 let me know.


This has been on my long-term-todo list since I started the project. 
Although I have no idea if this is possible, because, as far as I know, 
there is no the "connection" between the DMD, druntime and Phobos 
repositories. How do I know if a particular revision/hash from one 
repository will work with one from the other repositories?


 I was going to wait to do this until Jacob merged in my Windows pull request
 (he said he'd get to it end of next week, IIRC), because I want to base it
 off a DVM that already has Windows support, but maybe I shouldn't wait and
 should just fork my "windows fork" and get to it? (I'm still fairly new to
 DVCS and "proper DVCS procedures".)

 DVM is written in D1/Tango, though. Would that be a problem? If so, and if
 Jacob doesn't have any objections to it being ported to D2/Phobos, then I'd
 volunteer to do the port. I already have experience porting a bunch of my
 own code from D1/Tango to D2/Phobos. But I don't want to be hasty about
 porting it: I assume Jacob had a reason for using D1/Tango. I don't know how
 important the reason was.


I would prefer to use Tango. The reasons I chose Tango were/are:

* I'm used to D1/Tango
* Every time I try to use D2 something isn't working
* I still think Tango is more superior than Phobos, although Phobos is 
doing a great job of filling in the feature holes
* A particular feature and the most important that I needed was to easy 
download files via HTTP, Tango makes this very easy. Now Phobos is 
getting a curl wrapper which should make it as easy, don't know the 
status of that though. I don't want to and don't know how to use a low 
level socket to do this.

-- 
/Jacob Carlborg
Jun 17 2011
next sibling parent reply Jose Armando Garcia <jsancio gmail.com> writes:
On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg <doob me.com> wrote:

 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is getting a
 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to do
 this.


You can always exec a program. You can just call the wget or curl or even scp!
Jun 17 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-06-17 18:15, Jose Armando Garcia wrote:

 On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg<doob me.com>  wrote:

 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is getting a
 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to do
 this.


 You can always exec a program. You can just call the wget or curl or even scp!


That just creates additional dependencies and these tools are not 
available on Windows. Creating a new process also takes additional 
resources, time, memory and so on.

-- 
/Jacob Carlborg
Jun 17 2011
next sibling parent Jose Armando Garcia <jsancio gmail.com> writes:
On Fri, Jun 17, 2011 at 4:32 PM, Jacob Carlborg <doob me.com> wrote:

 On 2011-06-17 18:15, Jose Armando Garcia wrote:

 On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg<doob me.com> =A0wrote:

 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is
 getting a
 curl wrapper which should make it as easy, don't know the status of tha=






t

 though. I don't want to and don't know how to use a low level socket to
 do
 this.


 You can always exec a program. You can just call the wget or curl or eve=




n

 scp!


 That just creates additional dependencies and these tools are not availab=


le

 on Windows. Creating a new process also takes additional resources, time,
 memory and so on.


Windows? Who programs for that thing? Just kidding. I know of the
problems. My comment was meant as a obscure comment about that aspect
of the Windows platform.
Jun 17 2011
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:itgadv$2ggr$3 digitalmars.com...

 On 2011-06-17 18:15, Jose Armando Garcia wrote:

 On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg<doob me.com>  wrote:

 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is 
 getting a
 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to 
 do
 this.


 You can always exec a program. You can just call the wget or curl or even 
 scp!


 That just creates additional dependencies and these tools are not 
 available on Windows. Creating a new process also takes additional 
 resources, time, memory and so on.


wget is available in Windows (I have it installed), but 
finding/downloading/installing any of the GNU command line tools on Windows 
is a royal pain. And even if it weren't, I agree, it's better to just not 
have that dependency anyway.
Jun 18 2011
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Holy shit look how complicated:
http://gnuwin32.sourceforge.net/packages/wget.htm
http://downloads.sourceforge.net/gnuwin32/wget-1.11.4-1-bin.zip
Jun 18 2011
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
P.S. as you can see I'm a little jumpy today. Phobos bugs got to me a little. :)

On 6/18/11, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:

 Holy shit look how complicated:
 http://gnuwin32.sourceforge.net/packages/wget.htm
 http://downloads.sourceforge.net/gnuwin32/wget-1.11.4-1-bin.zip
Jun 18 2011
parent "Nick Sabalausky" <a a.a> writes:
Heh, well, my statement was perhaps a bit inaccurate.

"Andrej Mitrovic" <andrej.mitrovich gmail.com> wrote in message 
news:mailman.1002.1308435182.14074.digitalmars-d puremagic.com...

 P.S. as you can see I'm a little jumpy today. Phobos bugs got to me a 
 little. :)

 On 6/18/11, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:

 Holy shit look how complicated:
 http://gnuwin32.sourceforge.net/packages/wget.htm
 http://downloads.sourceforge.net/gnuwin32/wget-1.11.4-1-bin.zip
Jun 18 2011
prev sibling next sibling parent Brad Anderson <eco gnuk.net> writes:
On Fri, Jun 17, 2011 at 10:15 AM, Jose Armando Garcia <jsancio gmail.com>wrote:


 On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg <doob me.com> wrote:

 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is


 getting a

 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to


 do

 this.


 You can always exec a program. You can just call the wget or curl or even
 scp!


Windows complicates this as neither are available by typically.
Jun 17 2011
prev sibling parent reply Jimmy Cao <jcao219 gmail.com> writes:
On Fri, Jun 17, 2011 at 11:07 AM, Jacob Carlborg <doob me.com> wrote:


 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is getting a
 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to do
 this.


etc.c.curl seems to be working great on Windows.  It does create a
dependency on libcurl.dll, libeay32.dll, libidn-11.dll, and libssl32.dll
though.
Jun 17 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-06-17 22:12, Jimmy Cao wrote:

 On Fri, Jun 17, 2011 at 11:07 AM, Jacob Carlborg <doob me.com
 <mailto:doob me.com>> wrote:


     * A particular feature and the most important that I needed was to
     easy download files via HTTP, Tango makes this very easy. Now Phobos
     is getting a curl wrapper which should make it as easy, don't know
     the status of that though. I don't want to and don't know how to use
     a low level socket to do this.


 etc.c.curl seems to be working great on Windows.  It does create a
 dependency on libcurl.dll, libeay32.dll, libidn-11.dll, and libssl32.dll
 though.


Yet another dependency to worry about. This is not as easy to distribute 
as zlib due to the license.

-- 
/Jacob Carlborg
Jun 17 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-06-10 16:14, Andrei Alexandrescu wrote:

 We need a script to update all of dmd's paraphernalia. By this I'm
 putting the task up for grabs and am describing it.

 The script would require git to be installed, and would orchestrate git
 commands to freshen the installation. Either a tag can be given, or the
 script would automatically detect the latest "semistable" release, or
 even the script can be told to get the latest and greatest. The exact
 tagging convention is to be defined later.


You don't need git to just do a simple clone. Have you seen the big 
"Downloads" button? You can download HEAD, tags and (I assume) branches.

-- 
/Jacob Carlborg
Jun 17 2011