www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - resurrecting Bud and Rebuild

reply Walter Bright <newshound digitalmars.com> writes:
It was (correctly) pointed out that textual import breaks Bud and 
Rebuild. It's also been pointed out that textual import may be an 
unexpected vector for security problems.

Both can be resolved by only allowing textual import if a command 
switch, say, -Jpath, is given. 'path' gives the location of where to 
look for the file; and the file will be restricted to being under that 
path. No -Jpath, and textual import won't be allowed.

For Bud and Rebuild, if there's no -J, they know there are no textual 
imports, so they work as before. With -J, they always recompile.
Feb 07 2007
next sibling parent "Andrei Alexandrescu (See Website For Email)" <SeeWebsiteForEmail erdani.org> writes:
Walter Bright wrote:

 It was (correctly) pointed out that textual import breaks Bud and 
 Rebuild. It's also been pointed out that textual import may be an 
 unexpected vector for security problems.
 
 Both can be resolved by only allowing textual import if a command 
 switch, say, -Jpath, is given. 'path' gives the location of where to 
 look for the file; and the file will be restricted to being under that 
 path. No -Jpath, and textual import won't be allowed.
 
 For Bud and Rebuild, if there's no -J, they know there are no textual 
 imports, so they work as before. With -J, they always recompile.


I think that's great. Many advanced/powerful language features are 
unusual and may make tools unhappy. A classic is Java's runtime 
reflection, which wreaks havoc with pretty much all static analysis tools.


Andrei
Feb 07 2007
prev sibling next sibling parent reply Gregor Richards <Richards codu.org> writes:
Walter Bright wrote:

 It was (correctly) pointed out that textual import breaks Bud and 
 Rebuild. It's also been pointed out that textual import may be an 
 unexpected vector for security problems.
 
 Both can be resolved by only allowing textual import if a command 
 switch, say, -Jpath, is given. 'path' gives the location of where to 
 look for the file; and the file will be restricted to being under that 
 path. No -Jpath, and textual import won't be allowed.
 
 For Bud and Rebuild, if there's no -J, they know there are no textual 
 imports, so they work as before. With -J, they always recompile.


I can't speak to the security aspect, but I don't think that -J would be 
helpful to rebuild. For example, if you have a file a.d:

mixin(SomeRidiculouslyComplicatedTemplate!(WithRidiculousArguments));


It imports foo/b.d, foo/c.d or foo/d.d depending on some bizarrely 
complex situation. Each of them will only work in some scenario.

What rebuild sees is just a.d .  -Jpath would suggest that a.d depends 
on at least one of foo/b.d, foo/c.d or foo/d.d, but there's no way for 
it to know which short of actually resolving the templates.


The solution? Right now, there's no way to get the list of dependant 
files without compiling one. Since rebuild gets meta-data out of files, 
it would then have to compile it again, which is why I'm not doing it 
that way. I'd like to see something like this:

$ [g]dmd -p -files foo.d
file foo.d foo
file /opt/something/src/phobos/object.d object
file /opt/something/src/phobos/std/path.d std.path


That way, I could simply run the compiler, and get the list of actual 
files (not just imports) from its output. However, the compiler wouldn't 
actually compile anything (-p meaning parse-only), and it would list all 
the /files/ used (not just the imports).


Any way you'd add this feature? :)

  - Gregor Richards
Feb 07 2007
next sibling parent reply Derek Parnell <derek nomail.afraid.org> writes:
On Wed, 07 Feb 2007 15:38:40 -0800, Gregor Richards wrote:


 Walter Bright wrote:

 It was (correctly) pointed out that textual import breaks Bud and 
 Rebuild. It's also been pointed out that textual import may be an 
 unexpected vector for security problems.
 
 Both can be resolved by only allowing textual import if a command 
 switch, say, -Jpath, is given. 'path' gives the location of where to 
 look for the file; and the file will be restricted to being under that 
 path. No -Jpath, and textual import won't be allowed.
 
 For Bud and Rebuild, if there's no -J, they know there are no textual 
 imports, so they work as before. With -J, they always recompile.


 
 I can't speak to the security aspect, but I don't think that -J would be 
 helpful to rebuild.
 ...
 The solution? Right now, there's no way to get the list of dependant 
 files without compiling one. Since rebuild gets meta-data out of files, 
 it would then have to compile it again, which is why I'm not doing it 
 that way. I'd like to see something like this:
 
 $ [g]dmd -p -files foo.d
 file foo.d foo
 file /opt/something/src/phobos/object.d object
 file /opt/something/src/phobos/std/path.d std.path
 
 That way, I could simply run the compiler, and get the list of actual 
 files (not just imports) from its output. However, the compiler wouldn't 
 actually compile anything (-p meaning parse-only), and it would list all 
 the /files/ used (not just the imports).
 
 Any way you'd add this feature? :)


I totally agree with Gregor. The -J switch might give us a hint that the
files /on the command line/ must be recompiled, but it still doesn't make
it easier to find out what other files are needed in the application. One
purpose of Bud, and I assume Rebuild, is to allow the coder to just enter
one name on the command line and get all the other required files magically
included in the compilation process. The alternative is to require the
coder to maintain makefiles and/or type every required file onto the
command line.

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocrity!"
8/02/2007 10:42:00 AM
Feb 07 2007
parent BCS <BCS pathlink.com> writes:
Derek Parnell wrote:

 
 
 I totally agree with Gregor. The -J switch might give us a hint that the
 files /on the command line/ must be recompiled, but it still doesn't make
 it easier to find out what other files are needed in the application. One
 purpose of Bud, and I assume Rebuild, is to allow the coder to just enter
 one name on the command line and get all the other required files magically
 included in the compilation process. The alternative is to require the
 coder to maintain makefiles and/or type every required file onto the
 command line.
 



All that build tool needs is to maintain a list of *everything* that was 
looked at. The first time around (clean build) everything has to be 
built anyway so you just walk around building things. until everything 
is built, then link every .o you just built. After that you look at the 
list of things that you used last time, and if any of them change you 
rebuild everything that got you there.

the only thing hindering this I can think of is an apparent desirer to 
have the build process be state-less (it isn't now as we leave around 
lots of .o files).
Feb 07 2007
prev sibling parent reply Walter Bright <newshound digitalmars.com> writes:
Gregor Richards wrote:

 I can't speak to the security aspect, but I don't think that -J would be 
 helpful to rebuild. For example, if you have a file a.d:
 
 mixin(SomeRidiculouslyComplicatedTemplate!(WithRidiculousArguments));
 
 
 It imports foo/b.d, foo/c.d or foo/d.d depending on some bizarrely 
 complex situation. Each of them will only work in some scenario.
 
 What rebuild sees is just a.d .  -Jpath would suggest that a.d depends 
 on at least one of foo/b.d, foo/c.d or foo/d.d, but there's no way for 
 it to know which short of actually resolving the templates.


That's right, so when it sees -J, it just always rebuilds it (not just 
if one of its dependencies is newer).



 The solution? Right now, there's no way to get the list of dependant 
 files without compiling one. Since rebuild gets meta-data out of files, 
 it would then have to compile it again, which is why I'm not doing it 
 that way. I'd like to see something like this:
 
 $ [g]dmd -p -files foo.d
 file foo.d foo
 file /opt/something/src/phobos/object.d object
 file /opt/something/src/phobos/std/path.d std.path
 
 
 That way, I could simply run the compiler, and get the list of actual 
 files (not just imports) from its output. However, the compiler wouldn't 
 actually compile anything (-p meaning parse-only), and it would list all 
 the /files/ used (not just the imports).
 
 
 Any way you'd add this feature? :)


It's a good idea.
Feb 07 2007
next sibling parent Gregor Richards <Richards codu.org> writes:
Walter Bright wrote:

 Gregor Richards wrote:
 

 I can't speak to the security aspect, but I don't think that -J would 
 be helpful to rebuild. For example, if you have a file a.d:

 mixin(SomeRidiculouslyComplicatedTemplate!(WithRidiculousArguments));


 It imports foo/b.d, foo/c.d or foo/d.d depending on some bizarrely 
 complex situation. Each of them will only work in some scenario.

 What rebuild sees is just a.d .  -Jpath would suggest that a.d depends 
 on at least one of foo/b.d, foo/c.d or foo/d.d, but there's no way for 
 it to know which short of actually resolving the templates.


 
 
 That's right, so when it sees -J, it just always rebuilds it (not just 
 if one of its dependencies is newer).


I think we're talking about slightly different scenarios. If it simply 
mixes in the file, then this would work perfectly. So would the below if 
it listed files that were imported with import(filename)

But, if it mixes in "import <module name>;", then I'll need not only to 
compile a.d, but whatever module is imported via mixin("import").  To 
compile this next module, the feature I listed below would be necessary.



 
 

 The solution? Right now, there's no way to get the list of dependant 
 files without compiling one. Since rebuild gets meta-data out of 
 files, it would then have to compile it again, which is why I'm not 
 doing it that way. I'd like to see something like this:

 $ [g]dmd -p -files foo.d
 file foo.d foo
 file /opt/something/src/phobos/object.d object
 file /opt/something/src/phobos/std/path.d std.path


 That way, I could simply run the compiler, and get the list of actual 
 files (not just imports) from its output. However, the compiler 
 wouldn't actually compile anything (-p meaning parse-only), and it 
 would list all the /files/ used (not just the imports).


 Any way you'd add this feature? :)


 
 
 It's a good idea.


Fantastic :)

  - Gregor Richards
Feb 07 2007
prev sibling parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Walter Bright wrote:

 Gregor Richards wrote:

 I can't speak to the security aspect, but I don't think that -J would 
 be helpful to rebuild. For example, if you have a file a.d:

 mixin(SomeRidiculouslyComplicatedTemplate!(WithRidiculousArguments));


 It imports foo/b.d, foo/c.d or foo/d.d depending on some bizarrely 
 complex situation. Each of them will only work in some scenario.

 What rebuild sees is just a.d .  -Jpath would suggest that a.d depends 
 on at least one of foo/b.d, foo/c.d or foo/d.d, but there's no way for 
 it to know which short of actually resolving the templates.


 
 That's right, so when it sees -J, it just always rebuilds it (not just 
 if one of its dependencies is newer).


That still doesn't tell it what the dependencies *are*. In the above 
scenario, one of foo/{b,c,d}.d needs to be built & linked as well (as 
well as all modules it depends on, and so on).
Which, I presume, the feature below was requested.


 The solution? Right now, there's no way to get the list of dependant 
 files without compiling one. Since rebuild gets meta-data out of 
 files, it would then have to compile it again, which is why I'm not 
 doing it that way. I'd like to see something like this:

 $ [g]dmd -p -files foo.d
 file foo.d foo
 file /opt/something/src/phobos/object.d object
 file /opt/something/src/phobos/std/path.d std.path


 That way, I could simply run the compiler, and get the list of actual 
 files (not just imports) from its output. However, the compiler 
 wouldn't actually compile anything (-p meaning parse-only), and it 
 would list all the /files/ used (not just the imports).


 Any way you'd add this feature? :)


 
 It's a good idea.


Good to hear :)
Feb 07 2007
prev sibling parent reply Max Samukha <samukha voliacable.com> writes:
On Wed, 07 Feb 2007 13:40:16 -0800, Walter Bright
<newshound digitalmars.com> wrote:


It was (correctly) pointed out that textual import breaks Bud and 
Rebuild. It's also been pointed out that textual import may be an 
unexpected vector for security problems.

Both can be resolved by only allowing textual import if a command 
switch, say, -Jpath, is given. 'path' gives the location of where to 
look for the file; and the file will be restricted to being under that 
path. No -Jpath, and textual import won't be allowed.

For Bud and Rebuild, if there's no -J, they know there are no textual 
imports, so they work as before. With -J, they always recompile.


Will relative paths to subdirectories of 'path' be allowed in the
import expression (a behavior similar to ordinary imports)?

import ("file") - 'path'/file
import ("subdir/file") - 'path'/subdir/file
import ("/etc/passwd") - error
import ("../file") - error
Feb 11 2007
parent reply Walter Bright <newshound digitalmars.com> writes:
Max Samukha wrote:

 On Wed, 07 Feb 2007 13:40:16 -0800, Walter Bright
 <newshound digitalmars.com> wrote:

 Both can be resolved by only allowing textual import if a command 
 switch, say, -Jpath, is given. 'path' gives the location of where to 
 look for the file; and the file will be restricted to being under that 
 path. No -Jpath, and textual import won't be allowed.

 For Bud and Rebuild, if there's no -J, they know there are no textual 
 imports, so they work as before. With -J, they always recompile.


 
 Will relative paths to subdirectories of 'path' be allowed in the
 import expression (a behavior similar to ordinary imports)?
 
 import ("file") - 'path'/file
 import ("subdir/file") - 'path'/subdir/file
 import ("/etc/passwd") - error
 import ("../file") - error


No paths at all will be allowed in the import string. They'll have to be 
supplied via the -Jpath command line switch. It's probably more 
conservative than necessary, but:

1) introducing host system dependent path separators makes the source 
code non-portable

2) it gives the 'buildmaster' an easy way to find out and control what 
files are imported

3) I think it's best to err on the side of conservatism here
Feb 13 2007
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Walter Bright wrote:

 No paths at all will be allowed in the import string. They'll have to be 
 supplied via the -Jpath command line switch. It's probably more 
 conservative than necessary, but:
 
 1) introducing host system dependent path separators makes the source 
 code non-portable


Don't all systems in any amount of use these days accept '/'? (Yes, that 
includes Windows)


 2) it gives the 'buildmaster' an easy way to find out and control what 
 files are imported
 
 3) I think it's best to err on the side of conservatism here


Well, at least it won't break any code if you later decide to allow paths...
Feb 13 2007
parent Sean Kelly <sean f4.ca> writes:
Frits van Bommel wrote:

 Walter Bright wrote:

 No paths at all will be allowed in the import string. They'll have to 
 be supplied via the -Jpath command line switch. It's probably more 
 conservative than necessary, but:

 1) introducing host system dependent path separators makes the source 
 code non-portable


 
 Don't all systems in any amount of use these days accept '/'? (Yes, that 
 includes Windows)


Sort of.  The windows command shell is supposed to, but it's not usable. 
  For example "cd /" doesn't change to the root directory and "more 
/autoexec.bat" won't dump the contents of that file to the screen.  But 
we're talking about path separators in source code anyway, and '/' has 
been accepted by compilers for as long as I can remember.


Sean
Feb 13 2007