www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dsss / scons

reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Anyone given any thought to building a dsss-like system on top of SCONS? 
  Maybe even one compatible with the real dsss?  Maybe even mr. Gregor 
Richards has considered this?

Just a thought that popped in my head after watching my app's resources 
get needlessly compiled for the umpteenth time because of the lack of 
dependency tracking for anything but D code in DSSS.

--bb
Jan 01 2008
next sibling parent Lars Ivar Igesund <larsivar igesund.net> writes:
Bill Baxter wrote:


 Anyone given any thought to building a dsss-like system on top of SCONS?
   Maybe even one compatible with the real dsss?  Maybe even mr. Gregor
 Richards has considered this?
 
 Just a thought that popped in my head after watching my app's resources
 get needlessly compiled for the umpteenth time because of the lack of
 dependency tracking for anything but D code in DSSS.
 
 --bb


A few years back (before build) someone added D support to SCONS, and I
added it to A-A-P. Neither gained much momentum as the python dependency
for the build system seemed to put most people off.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource, #d.tango & #D: larsivi
Dancing the Tango
Jan 01 2008
prev sibling parent reply =?ISO-8859-1?Q?=22J=E9r=F4me_M=2E_Berger=22?= <jeberger free.fr> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bill Baxter wrote:

 Anyone given any thought to building a dsss-like system on top of SCONS?
  Maybe even one compatible with the real dsss?  Maybe even mr. Gregor
 Richards has considered this?
 
 Just a thought that popped in my head after watching my app's resources
 get needlessly compiled for the umpteenth time because of the lack of
 dependency tracking for anything but D code in DSSS.
 


	I have "thought about it" recently, but I haven't actually done
much for it. What I have however is a system built on top of SCons
which you can point at a folder and say (among other things): "pick
every D source file in here and build a program / a lib with them".
I can make it available if you'd like.

		Jerome
- --
+------------------------- Jerome M. BERGER ---------------------+
|    mailto:jeberger free.fr      | ICQ:    238062172            |
|    http://jeberger.free.fr/     | Jabber: jeberger jabber.fr   |
+---------------------------------+------------------------------+
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFHejXUd0kWM4JG3k8RAjRwAKCIOfkgD62xGME1iMEBzv/m5tcYvACfdLzx
ODaA23OZluOWuTHhCogJqZ8=
=cxmH
-----END PGP SIGNATURE-----
Jan 01 2008
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Jérôme M. Berger wrote:

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Bill Baxter wrote:

 Anyone given any thought to building a dsss-like system on top of SCONS?
  Maybe even one compatible with the real dsss?  Maybe even mr. Gregor
 Richards has considered this?

 Just a thought that popped in my head after watching my app's resources
 get needlessly compiled for the umpteenth time because of the lack of
 dependency tracking for anything but D code in DSSS.


 	I have "thought about it" recently, but I haven't actually done
 much for it. What I have however is a system built on top of SCons
 which you can point at a folder and say (among other things): "pick
 every D source file in here and build a program / a lib with them".
 I can make it available if you'd like.


How is that different from the built-in D builder in SCons?  Honest 
question.  I've never tried to build D code with SCons before, so I 
really don't know what the current state of the art is.  If your code is 
better, maybe it should go upstream to be included with SCons itself.

--bb
Jan 01 2008
parent reply =?ISO-8859-1?Q?=22J=E9r=F4me_M=2E_Berger=22?= <jeberger free.fr> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bill Baxter wrote:

 Jérôme M. Berger wrote:

 Bill Baxter wrote:

 Anyone given any thought to building a dsss-like system on top of SCONS?
  Maybe even one compatible with the real dsss?  Maybe even mr. Gregor
 Richards has considered this?

 Just a thought that popped in my head after watching my app's resources
 get needlessly compiled for the umpteenth time because of the lack of
 dependency tracking for anything but D code in DSSS.


     I have "thought about it" recently, but I haven't actually done
 much for it. What I have however is a system built on top of SCons
 which you can point at a folder and say (among other things): "pick
 every D source file in here and build a program / a lib with them".
 I can make it available if you'd like.


 
 How is that different from the built-in D builder in SCons?  Honest
 question.  I've never tried to build D code with SCons before, so I
 really don't know what the current state of the art is.  If your code is
 better, maybe it should go upstream to be included with SCons itself.
 


	With SCons, you need to list all source files by hand. I've added
three things:
 - A 10-lines python function that scans a folder and returns the
list of files matching a given regexp. This list can then be given
to the standard SCons builder to generate a lib or a program;
 - An improved dependency scanner which recognizes the '.di'
extension and multiple imports (eg "import a, b;"). That one was
included upstream a couple of weeks ago and should be available in
the next development release of SCons;
 - Support for gdc (since dmd doesn't work on 64bits linux).
Unfortunately, that one is not properly integrated with the standard
SCons and cannot be sent upstream as is.

	What I've thought about doing is a function that would take a
program (or lib) name and a list of source files (possibly including
only a single file) and automatically compile all the dependencies.

	Here's an example. Let's assume you want to build a program named
"foo" and you have the following source hierarchy:

src
 `- main.d
 `- module1.d
 `- module2.d

	And main.d imports both module1 and module2.

	With standard SCons, you need to do:
env.Program ("foo", Split ("main.d module1.d module2.d"))

	Which can become pretty cumbersome if you have lots of source files.

	With my current system, you can do:
env.Program ("foo", scanFolder ("src", re.compile (r".*\.d$")))

	Which means that each program or lib must have its own folder
independent of the others.

	With what I've thought to do when I get the time, you would only
have to say:
env.DProgram ("foo", "main.d")

		Jerome
- --
+------------------------- Jerome M. BERGER ---------------------+
|    mailto:jeberger free.fr      | ICQ:    238062172            |
|    http://jeberger.free.fr/     | Jabber: jeberger jabber.fr   |
+---------------------------------+------------------------------+
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFHekHWd0kWM4JG3k8RApAEAJ9VTBiFrYzLxwBVp5Qa56UHTFAh6ACfU1uo
P1JUE82H/VALE7PxbV3OUXY=
=P1oP
-----END PGP SIGNATURE-----
Jan 01 2008
next sibling parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Jérôme M. Berger wrote:

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Bill Baxter wrote:

 Jérôme M. Berger wrote:

 Bill Baxter wrote:

 Anyone given any thought to building a dsss-like system on top of SCONS?
  Maybe even one compatible with the real dsss?  Maybe even mr. Gregor
 Richards has considered this?

 Just a thought that popped in my head after watching my app's resources
 get needlessly compiled for the umpteenth time because of the lack of
 dependency tracking for anything but D code in DSSS.


     I have "thought about it" recently, but I haven't actually done
 much for it. What I have however is a system built on top of SCons
 which you can point at a folder and say (among other things): "pick
 every D source file in here and build a program / a lib with them".
 I can make it available if you'd like.


 How is that different from the built-in D builder in SCons?  Honest
 question.  I've never tried to build D code with SCons before, so I
 really don't know what the current state of the art is.  If your code is
 better, maybe it should go upstream to be included with SCons itself.


 	With SCons, you need to list all source files by hand. I've added
 three things:
  - A 10-lines python function that scans a folder and returns the
 list of files matching a given regexp. This list can then be given
 to the standard SCons builder to generate a lib or a program;
  - An improved dependency scanner which recognizes the '.di'
 extension and multiple imports (eg "import a, b;"). That one was
 included upstream a couple of weeks ago and should be available in
 the next development release of SCons;
  - Support for gdc (since dmd doesn't work on 64bits linux).
 Unfortunately, that one is not properly integrated with the standard
 SCons and cannot be sent upstream as is.
 
 	What I've thought about doing is a function that would take a
 program (or lib) name and a list of source files (possibly including
 only a single file) and automatically compile all the dependencies.
 
 	Here's an example. Let's assume you want to build a program named
 "foo" and you have the following source hierarchy:
 
 src
  `- main.d
  `- module1.d
  `- module2.d
 
 	And main.d imports both module1 and module2.
 
 	With standard SCons, you need to do:
 env.Program ("foo", Split ("main.d module1.d module2.d"))
 
 	Which can become pretty cumbersome if you have lots of source files.
 
 	With my current system, you can do:
 env.Program ("foo", scanFolder ("src", re.compile (r".*\.d$")))
 
 	Which means that each program or lib must have its own folder
 independent of the others.
 
 	With what I've thought to do when I get the time, you would only
 have to say:
 env.DProgram ("foo", "main.d")


I see.  Thanks for the explanation.
So the "what you've thought to do" part is to basically make a 
bud/rebuild-like builder?

I think the compiler's "-v -o-" flags can help get a simple prototype 
working pretty quickly, but I guess there must be more to it that that 
or else rebuild wouldn't be lugging around the entire front end source 
code just to find dependencies.  And maybe those flags only work for dmd?

--bb
Jan 01 2008
next sibling parent =?ISO-8859-1?Q?=22J=E9r=F4me_M=2E_Berger=22?= <jeberger free.fr> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Bill Baxter wrote:

 Jérôme M. Berger wrote:

 Bill Baxter wrote:

 Jérôme M. Berger wrote:

 Bill Baxter wrote:

 Anyone given any thought to building a dsss-like system on top of
 SCONS?
  Maybe even one compatible with the real dsss?  Maybe even mr. Gregor
 Richards has considered this?

 Just a thought that popped in my head after watching my app's
 resources
 get needlessly compiled for the umpteenth time because of the lack of
 dependency tracking for anything but D code in DSSS.


     I have "thought about it" recently, but I haven't actually done
 much for it. What I have however is a system built on top of SCons
 which you can point at a folder and say (among other things): "pick
 every D source file in here and build a program / a lib with them".
 I can make it available if you'd like.


 How is that different from the built-in D builder in SCons?  Honest
 question.  I've never tried to build D code with SCons before, so I
 really don't know what the current state of the art is.  If your code is
 better, maybe it should go upstream to be included with SCons itself.


     With SCons, you need to list all source files by hand. I've added
 three things:
  - A 10-lines python function that scans a folder and returns the
 list of files matching a given regexp. This list can then be given
 to the standard SCons builder to generate a lib or a program;
  - An improved dependency scanner which recognizes the '.di'
 extension and multiple imports (eg "import a, b;"). That one was
 included upstream a couple of weeks ago and should be available in
 the next development release of SCons;
  - Support for gdc (since dmd doesn't work on 64bits linux).
 Unfortunately, that one is not properly integrated with the standard
 SCons and cannot be sent upstream as is.

     What I've thought about doing is a function that would take a
 program (or lib) name and a list of source files (possibly including
 only a single file) and automatically compile all the dependencies.

     Here's an example. Let's assume you want to build a program named
 "foo" and you have the following source hierarchy:

 src
  `- main.d
  `- module1.d
  `- module2.d

     And main.d imports both module1 and module2.

     With standard SCons, you need to do:
 env.Program ("foo", Split ("main.d module1.d module2.d"))

     Which can become pretty cumbersome if you have lots of source files.

     With my current system, you can do:
 env.Program ("foo", scanFolder ("src", re.compile (r".*\.d$")))

     Which means that each program or lib must have its own folder
 independent of the others.

     With what I've thought to do when I get the time, you would only
 have to say:
 env.DProgram ("foo", "main.d")


 
 I see.  Thanks for the explanation.
 So the "what you've thought to do" part is to basically make a
 bud/rebuild-like builder?
 


	More or less, yes.


 I think the compiler's "-v -o-" flags can help get a simple prototype
 working pretty quickly, but I guess there must be more to it that that
 or else rebuild wouldn't be lugging around the entire front end source
 code just to find dependencies. 


	I'd guess that the reason they need the entire front-end is because
imports may be emitted from string mixins with the strings generated
through CTFE and in order to handle conditional compilation.

	The system I had in mind would be simpler than that: it would only
handle basic imports (but that should cover 95+% of user needs).
Moreover, I don't know how much support there would be for
conditional compilation (no "static if" and I don't know how much
"version").


 And maybe those flags only work for dmd?
 


	No, there are equivalent flags for gdc (-fd-verbose -fsyntax-only).
This might indeed be a way to look.

		Jerome
- --
+------------------------- Jerome M. BERGER ---------------------+
|    mailto:jeberger free.fr      | ICQ:    238062172            |
|    http://jeberger.free.fr/     | Jabber: jeberger jabber.fr   |
+---------------------------------+------------------------------+
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFHek22d0kWM4JG3k8RAs11AJ9Z31SDTB8jjzYa7cA4Pefu39wb1gCfQ1So
in1Jdkm3+Wcr5Ls5ivbANi4=
=o1N3
-----END PGP SIGNATURE-----
Jan 01 2008
prev sibling parent Lars Ivar Igesund <larsivar igesund.net> writes:
Bill Baxter wrote:

 
 I think the compiler's "-v -o-" flags can help get a simple prototype
 working pretty quickly, but I guess there must be more to it that that
 or else rebuild wouldn't be lugging around the entire front end source
 code just to find dependencies.  And maybe those flags only work for dmd?


It is fairly easy detecting imports correctly (except possibly for the
string mixin stuff) without the frontend, but the frontend _do_ give
rebuild a whole lot of potential power, for instance the ddoc generation is
enabled there. Partially I guess it is mostly done because Gregor have
several dmd frontend projects, and using it for rebuild was convenient.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource, #d.tango & #D: larsivi
Dancing the Tango
Jan 01 2008
prev sibling parent Jay Norwood <jayn io.com> writes:
Jérôme M. Berger Wrote:


 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Bill Baxter wrote:

 Jérôme M. Berger wrote:

 Bill Baxter wrote:

 Anyone given any thought to building a dsss-like system on top of SCONS?
  Maybe even one compatible with the real dsss?  Maybe even mr. Gregor
 Richards has considered this?

 Just a thought that popped in my head after watching my app's resources
 get needlessly compiled for the umpteenth time because of the lack of
 dependency tracking for anything but D code in DSSS.


     I have "thought about it" recently, but I haven't actually done
 much for it. What I have however is a system built on top of SCons
 which you can point at a folder and say (among other things): "pick
 every D source file in here and build a program / a lib with them".
 I can make it available if you'd like.


 
 How is that different from the built-in D builder in SCons?  Honest
 question.  I've never tried to build D code with SCons before, so I
 really don't know what the current state of the art is.  If your code is
 better, maybe it should go upstream to be included with SCons itself.
 


 	With SCons, you need to list all source files by hand. I've added
 three things:
  - A 10-lines python function that scans a folder and returns the
 list of files matching a given regexp. This list can then be given
 to the standard SCons builder to generate a lib or a program;



 scons has something similar now with glob().  Was that your contribution?

http://www.scons.org/doc/0.97.0d20071212/HTML/scons-api/index.html


glob(self, pathname, ondisk=True, source=False, strings=False) 
Returns a list of Nodes (or strings) matching a specified pathname pattern.

Pathname patterns follow UNIX shell semantics: * matches any-length strings of
any characters, ? matches any character, and [] can enclose lists or ranges of
characters. Matches do not span directory separators.

The matches take into account Repositories, returning local Nodes if a
corresponding entry exists in a Repository (either an in-memory Node or
something on disk).

By defafult, the glob() function matches entries that exist on-disk, in
addition to in-memory Nodes. Setting the "ondisk" argument to False (or some
other non-true value) causes the glob() function to only match in-memory Nodes.
The default behavior is to return both the on-disk and in-memory Nodes.

The "source" argument, when true, specifies that corresponding source Nodes
must be returned if you're globbing in a build directory (initialized with
BuildDir()). The default behavior is to return Nodes local to the BuildDir().

The "strings" argument, when true, returns the matches as strings, not Nodes.
The strings are path names relative to this directory.

The underlying algorithm is adapted from the glob.glob() function in the Python
library (but heavily modified), and uses fnmatch() under the covers.
Jan 06 2008