www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compile time executable calling?

reply "Tofu Ninja" <emmons0 purdue.edu> writes:
So I had an idea recently, wouldn't it be cool to have the 
ability to call an executable at compile time and capture its 
output. Something like the string imports but instead of opening 
and reading a text file, it run an executable, waits for it to 
finish, and grabs its output.

It would get really cool if you could pass this executable some 
args and then mix in its out put into your own code. It could be 
used similarly to how CTFE are used but with out the overhead of 
trying to compile that function and what not and with out the 
limitations on what it can do.

I could imagine all sorts of things that would be possible with 
this that is currently not.

Not sure if this is something that could be implemented easily, 
but seems like something that could be done and something that 
would be really cool.
Jul 12 2013
next sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
On Fri, Jul 12, 2013 at 1:42 PM, Tofu Ninja <emmons0 purdue.edu> wrote:


 So I had an idea recently, wouldn't it be cool to have the ability to call
 an executable at compile time and capture its output. Something like the
 string imports but instead of opening and reading a text file, it run an
 executable, waits for it to finish, and grabs its output.

 It would get really cool if you could pass this executable some args and
 then mix in its out put into your own code. It could be used similarly to
 how CTFE are used but with out the overhead of trying to compile that
 function and what not and with out the limitations on what it can do.

 I could imagine all sorts of things that would be possible with this that
 is currently not.

 Not sure if this is something that could be implemented easily, but seems
 like something that could be done and something that would be really cool.


I would definitely love that, and I don't see why this should be hard.
It should be enabled by a compiler switch (dmd -enable_exec) for obvious
safety reasons though (just as string import requires -J).

This enables or simplifies a number of things:

* ctfeWriteln becomes trivial (this feature has been asked a while ago)
* CTFE in situations where current CTFE fails
* do things that would temprarily require malloc
* easily customize compilation process (user would be able to add logging /
profile info without touching the dmd compiler)
* accessing environment variables during compilation

seems to me all that is needed is optionally redirecting stdin/stdout of a
forked process inside dmd compilation, with the exec C function family.
Jul 12 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/12/2013 1:42 PM, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the ability to call an
 executable at compile time and capture its output. Something like the string
 imports but instead of opening and reading a text file, it run an executable,
 waits for it to finish, and grabs its output.

 It would get really cool if you could pass this executable some args and then
 mix in its out put into your own code. It could be used similarly to how CTFE
 are used but with out the overhead of trying to compile that function and what
 not and with out the limitations on what it can do.

 I could imagine all sorts of things that would be possible with this that is
 currently not.

 Not sure if this is something that could be implemented easily, but seems like
 something that could be done and something that would be really cool.


This is actually done in the makefile that builds DMD. A program (optabgen) is 
compiled and then run. Optabgen's output is several .c files which are then 
compiled into DMD's binary. The programs impcnvgen and idgen do similar things.

It is a very powerful technique.
Jul 12 2013
next sibling parent reply Timothee Cour <thelastmammoth gmail.com> writes:
On Fri, Jul 12, 2013 at 2:54 PM, Walter Bright
<newshound2 digitalmars.com>wrote:


 On 7/12/2013 1:42 PM, Tofu Ninja wrote:


 So I had an idea recently, wouldn't it be cool to have the ability to
 call an
 executable at compile time and capture its output. Something like the
 string
 imports but instead of opening and reading a text file, it run an
 executable,
 waits for it to finish, and grabs its output.

 It would get really cool if you could pass this executable some args and
 then
 mix in its out put into your own code. It could be used similarly to how
 CTFE
 are used but with out the overhead of trying to compile that function and
 what
 not and with out the limitations on what it can do.

 I could imagine all sorts of things that would be possible with this that
 is
 currently not.

 Not sure if this is something that could be implemented easily, but seems
 like
 something that could be done and something that would be really cool.


 This is actually done in the makefile that builds DMD. A program
 (optabgen) is compiled and then run. Optabgen's output is several .c files
 which are then compiled into DMD's binary. The programs impcnvgen and idgen
 do similar things.

 It is a very powerful technique.


I think the OP was refering to something different:
ability to call an arbitrary executable / shell command during compile time
of a D function, whereas optabgen is during compiling dmd itself:

what we want to have is this:

----
import("some_source_file.txt"); //currently this is our only interaction
with outside world during compile time
import std.process; //or std.ctfeprocess ?

string getStringAtCompileTime(string command){
  if(!__ctfe) assert(0);
  //below is currently impossible, but would be great to have
  Tuple!(int,string,string) result=systemCaptureStdinStdout(command);
//likewise without redirecting stdin/stdout
  assert(!result[0]);
  return result[1];
}

void main(){
  enum host = getStringAtCompileTime("env | grep HOST");
}
----
Jul 12 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/12/2013 3:12 PM, Timothee Cour wrote:

 I think the OP was refering to something different:
 ability to call an arbitrary executable / shell command during compile time of
a
 D function, whereas optabgen is during compiling dmd itself:


It's still the same idea - using external programs to generate source code.



 what we want to have is this:


I do understand that. I'm just saying that this can currently (but awkwardly)
be 
done in the makefile.
Jul 12 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jul 12, 2013 at 03:35:30PM -0700, Walter Bright wrote:

 On 7/12/2013 3:12 PM, Timothee Cour wrote:

I think the OP was refering to something different: ability to call
an arbitrary executable / shell command during compile time of a D
function, whereas optabgen is during compiling dmd itself:


 
 It's still the same idea - using external programs to generate source
 code.


This idea isn't new. lex/yacc (or their modern incarnations flex/bison)
come to mind. The usage is a bit clunky, but the essence is the same. 



what we want to have is this:


 
 I do understand that. I'm just saying that this can currently (but
 awkwardly) be done in the makefile.


At what point does the balance shift from having such an ability
built-in, vs. just using OS-level facilities for combining different
programs? For example, one could pipe source through a program that
performs arbitrary transformations on it, then pipe the result through
the compiler. Or one can write a program that generates arbitrary source
code and pipe that into the compiler.

Presently, such things are easily handled by a modern build system (of
which makefiles are a rather clunky implementation thereof).


T

-- 
This is not a sentence.
Jul 12 2013
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 13.07.2013 01:43, schrieb H. S. Teoh:

 On Fri, Jul 12, 2013 at 03:35:30PM -0700, Walter Bright wrote:

 On 7/12/2013 3:12 PM, Timothee Cour wrote:

 I think the OP was refering to something different: ability to call
 an arbitrary executable / shell command during compile time of a D
 function, whereas optabgen is during compiling dmd itself:


 It's still the same idea - using external programs to generate source
 code.


 This idea isn't new. lex/yacc (or their modern incarnations flex/bison)
 come to mind. The usage is a bit clunky, but the essence is the same.



 what we want to have is this:


 I do understand that. I'm just saying that this can currently (but
 awkwardly) be done in the makefile.


 At what point does the balance shift from having such an ability
 built-in, vs. just using OS-level facilities for combining different
 programs? For example, one could pipe source through a program that
 performs arbitrary transformations on it, then pipe the result through
 the compiler. Or one can write a program that generates arbitrary source
 code and pipe that into the compiler.

 Presently, such things are easily handled by a modern build system (of
 which makefiles are a rather clunky implementation thereof).


 T


In most modern build systems you can easily achieve that via plugins.

Does DUB accept plugins?

--
Paulo
Jul 13 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 12 July 2013 at 21:54:02 UTC, Walter Bright wrote:

 On 7/12/2013 1:42 PM, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the 
 ability to call an
 executable at compile time and capture its output. Something 
 like the string
 imports but instead of opening and reading a text file, it run 
 an executable,
 waits for it to finish, and grabs its output.

 It would get really cool if you could pass this executable 
 some args and then
 mix in its out put into your own code. It could be used 
 similarly to how CTFE
 are used but with out the overhead of trying to compile that 
 function and what
 not and with out the limitations on what it can do.

 I could imagine all sorts of things that would be possible 
 with this that is
 currently not.

 Not sure if this is something that could be implemented 
 easily, but seems like
 something that could be done and something that would be 
 really cool.


 This is actually done in the makefile that builds DMD. A 
 program (optabgen) is compiled and then run. Optabgen's output 
 is several .c files which are then compiled into DMD's binary. 
 The programs impcnvgen and idgen do similar things.

 It is a very powerful technique.


I did similar stuff for a mixed D/C++ program. The build 
contained some intermediary utilities used to spit out infos 
about the C++ code (typically sizeof and other characteristics 
like that) that could then be used to generate part of the 
binding in D.
Jul 15 2013
prev sibling next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the 
 ability to call an executable at compile time and capture its 
 output.


How long until D compilers are able to read mail? :-)

There's many obvious applications of this proposed feature, but I 
say such things should be delegated to the build process. Just 
run those executables using a makefile, or whatever build system 
you use.

If you need compile time inputs from your D code then just run a 
separate build to extract them. Yes it's more work and less 
convenient, but I think we need to be careful with how much is 
added to the D compilers. We don't want to turn them into 
operating systems.
Jul 12 2013
next sibling parent reply Justin Whear <justin economicmodeling.com> writes:
On Sat, 13 Jul 2013 00:36:21 +0200, Peter Alexander wrote:


 On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the ability to
 call an executable at compile time and capture its output.


 
 How long until D compilers are able to read mail? :-)
 
 There's many obvious applications of this proposed feature, but I say
 such things should be delegated to the build process. Just run those
 executables using a makefile, or whatever build system you use.
 
 If you need compile time inputs from your D code then just run a
 separate build to extract them. Yes it's more work and less convenient,
 but I think we need to be careful with how much is added to the D
 compilers. We don't want to turn them into operating systems.


Other compilers allow symbols/macros to be defined on the command line, 
e.g. gcc's `-D` flag;  it'd be nice for DMD to define a similar mechanism 
for defining enums, e.g.

---D code---
  void main() { writeln(VERSION_NUMBER); }
------------

dmd main.d -DVERSION_NUMBER=1.2

Currently I have to use my makefiles to output a whole file which defines 
the various build constants.
Jul 12 2013
parent reply "Daniel Kozak" <kozzi11 gmail.com> writes:
On Friday, 12 July 2013 at 22:46:39 UTC, Justin Whear wrote:

 On Sat, 13 Jul 2013 00:36:21 +0200, Peter Alexander wrote:


 On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the 
 ability to
 call an executable at compile time and capture its output.


 
 How long until D compilers are able to read mail? :-)
 
 There's many obvious applications of this proposed feature, 
 but I say
 such things should be delegated to the build process. Just run 
 those
 executables using a makefile, or whatever build system you use.
 
 If you need compile time inputs from your D code then just run 
 a
 separate build to extract them. Yes it's more work and less 
 convenient,
 but I think we need to be careful with how much is added to 
 the D
 compilers. We don't want to turn them into operating systems.


 Other compilers allow symbols/macros to be defined on the 
 command line,
 e.g. gcc's `-D` flag;  it'd be nice for DMD to define a similar 
 mechanism
 for defining enums, e.g.

 ---D code---
   void main() { writeln(VERSION_NUMBER); }
 ------------

 dmd main.d -DVERSION_NUMBER=1.2

 Currently I have to use my makefiles to output a whole file 
 which defines
 the various build constants.


A few minutes ago, I have a same issue. I need add some enum 
values as a compile option.

Now I have something like this:
version (Parser) {
	enum Application = "parser";
}
version (Report) {
	enum Application = "report";
}

But would be nice to have posibility to add same value as a 
compile parametr, so I could have code like this:

enum Application = __traits(getParametrValue, 
"MY_APPLICATION_NAME");

and compile code with some args for eg.: 
-VMY_APPLICATION_NAME=parser
Jul 13 2013
parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 13 July 2013 at 11:19:49 UTC, Daniel Kozak wrote:

 A few minutes ago, I have a same issue. I need add some enum 
 values as a compile option.

 Now I have something like this:
 version (Parser) {
 	enum Application = "parser";
 }
 version (Report) {
 	enum Application = "report";
 }

 But would be nice to have posibility to add same value as a 
 compile parametr, so I could have code like this:

 enum Application = __traits(getParametrValue, 
 "MY_APPLICATION_NAME");

 and compile code with some args for eg.: 
 -VMY_APPLICATION_NAME=parser


You can solve this using import expressions.

---parser/config.include---
enum Application = "parser";

---report/config.include---
enum Application = "report";

---main.d---
mixin(import("config.include"));

---Makefile---
parser:
     dmd main.d -Jparser

report:
     dmd main.d -Jreport
Jul 13 2013
parent "Daniel Kozak" <kozzi11 gmail.com> writes:
On Saturday, 13 July 2013 at 11:25:05 UTC, Peter Alexander wrote:

 On Saturday, 13 July 2013 at 11:19:49 UTC, Daniel Kozak wrote:

 A few minutes ago, I have a same issue. I need add some enum 
 values as a compile option.

 Now I have something like this:
 version (Parser) {
 	enum Application = "parser";
 }
 version (Report) {
 	enum Application = "report";
 }

 But would be nice to have posibility to add same value as a 
 compile parametr, so I could have code like this:

 enum Application = __traits(getParametrValue, 
 "MY_APPLICATION_NAME");

 and compile code with some args for eg.: 
 -VMY_APPLICATION_NAME=parser


 You can solve this using import expressions.

 ---parser/config.include---
 enum Application = "parser";

 ---report/config.include---
 enum Application = "report";

 ---main.d---
 mixin(import("config.include"));

 ---Makefile---
 parser:
     dmd main.d -Jparser

 report:
     dmd main.d -Jreport


Thanks, this is usefull :)
Jul 13 2013
prev sibling next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:

 On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the 
 ability to call an executable at compile time and capture its 
 output.


 How long until D compilers are able to read mail? :-)

 There's many obvious applications of this proposed feature, but 
 I say such things should be delegated to the build process. 
 Just run those executables using a makefile, or whatever build 
 system you use.

 If you need compile time inputs from your D code then just run 
 a separate build to extract them. Yes it's more work and less 
 convenient, but I think we need to be careful with how much is 
 added to the D compilers. We don't want to turn them into 
 operating systems.


I think this is in line with what you want though, it allows the 
things that really should be separate to be separate in another 
executable, the only thing that it is really bring into the 
language is that you wont have to rely on a makefile as much, 
which in my opinion is a good thing.
Jul 12 2013
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:

 On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:

On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

So I had an idea recently, wouldn't it be cool to have the
ability to call an executable at compile time and capture its
output.


How long until D compilers are able to read mail? :-)




Zawinski's law of software envelopment:

	Every program attempts to expand until it can read mail. Those
	programs which cannot so expand are replaced by ones which can.

:-)



There's many obvious applications of this proposed feature, but I
say such things should be delegated to the build process. Just run
those executables using a makefile, or whatever build system you
use.

If you need compile time inputs from your D code then just run a
separate build to extract them. Yes it's more work and less
convenient, but I think we need to be careful with how much is
added to the D compilers. We don't want to turn them into
operating systems.


 
 I think this is in line with what you want though, it allows the
 things that really should be separate to be separate in another
 executable, the only thing that it is really bring into the language
 is that you wont have to rely on a makefile as much, which in my
 opinion is a good thing.


Makefiles are really a poor example of build systems. A proper build
system can do such things in a much better, consistent way.

Makefiles, for example, can't correctly recompile your program if you
change compiler flags. And complex interdependencies, like you said,
require painful manual maintenance, otherwise they tend to screw up and
produce inconsistent results.

For an example of a modern build system, see http://gittup.org/tup/.


T

-- 
Ph.D. = Permanent head Damage
Jul 12 2013
prev sibling next sibling parent reply Timothee Cour <thelastmammoth gmail.com> writes:
On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:


 On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:

 On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:

On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

So I had an idea recently, wouldn't it be cool to have the
ability to call an executable at compile time and capture its
output.


How long until D compilers are able to read mail? :-)




 Zawinski's law of software envelopment:

         Every program attempts to expand until it can read mail. Those
         programs which cannot so expand are replaced by ones which can.



forget about reading mail, we currently don't even have hello world! (we
have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)




 :-)



There's many obvious applications of this proposed feature, but I
say such things should be delegated to the build process. Just run
those executables using a makefile, or whatever build system you
use.

If you need compile time inputs from your D code then just run a
separate build to extract them. Yes it's more work and less
convenient, but I think we need to be careful with how much is
added to the D compilers. We don't want to turn them into
operating systems.


 I think this is in line with what you want though, it allows the
 things that really should be separate to be separate in another
 executable, the only thing that it is really bring into the language
 is that you wont have to rely on a makefile as much, which in my
 opinion is a good thing.


 Makefiles are really a poor example of build systems. A proper build
 system can do such things in a much better, consistent way.

 Makefiles, for example, can't correctly recompile your program if you
 change compiler flags. And complex interdependencies, like you said,
 require painful manual maintenance, otherwise they tend to screw up and
 produce inconsistent results.

 For an example of a modern build system, see http://gittup.org/tup/.


Using a build system would likely be non-portable (see current mess even in
dmd source code, which requires posix.ma / win32.mak / win64.mak which is
not DRY and hard to maintain), and suffer from the points I mentioned in my
previous post.

We don't need yet another build system if we have "exec" available at CT.

Let's put it another way: if I or someone else made a pull request for CTFE
"exec", would it have a chance of being accepted?
Jul 12 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/12/2013 5:00 PM, Timothee Cour wrote:

 Let's put it another way: if I or someone else made a pull request for CTFE
 "exec", would it have a chance of being accepted?


A big problem with it would be the equivalent of the "SQL Injection Exploit". 
Since the compiler can now execute arbitrary code, someone passing around 
malicious source code could do anything to your system.
Jul 12 2013
next sibling parent reply "BLM768" <blm768 gmail.com> writes:
On Saturday, 13 July 2013 at 04:23:56 UTC, Walter Bright wrote:

 A big problem with it would be the equivalent of the "SQL 
 Injection Exploit". Since the compiler can now execute 
 arbitrary code, someone passing around malicious source code 
 could do anything to your system.


Assuming that the user is compiling the code in order to run it 
(which does seem to be the most common case, at least in my 
experience), the user is already running arbitrary code. I don't 
really see how this would create a greater security risk than 
what already exists.

That said, I'm not completely sold on this idea, either, at least 
not while there are more important issues to solve. It could be 
nice at times, but I'm having a hard time coming up with a usage 
case where this would really be much more convenient than just 
using the build system.
Jul 12 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/12/2013 11:52 PM, BLM768 wrote:

 On Saturday, 13 July 2013 at 04:23:56 UTC, Walter Bright wrote:

 A big problem with it would be the equivalent of the "SQL Injection Exploit".
 Since the compiler can now execute arbitrary code, someone passing around
 malicious source code could do anything to your system.


 Assuming that the user is compiling the code in order to run it (which does
seem
 to be the most common case, at least in my experience), the user is already
 running arbitrary code. I don't really see how this would create a greater
 security risk than what already exists.


People can be endlessly creative at this kind of thing. I'm not at all sure you 
and I have thought of every possible exploit.
Jul 13 2013
next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Saturday, 13 July 2013 at 07:13:37 UTC, Walter Bright wrote:

 On 7/12/2013 11:52 PM, BLM768 wrote:

 On Saturday, 13 July 2013 at 04:23:56 UTC, Walter Bright wrote:

 A big problem with it would be the equivalent of the "SQL 
 Injection Exploit".
 Since the compiler can now execute arbitrary code, someone 
 passing around
 malicious source code could do anything to your system.


 Assuming that the user is compiling the code in order to run 
 it (which does seem
 to be the most common case, at least in my experience), the 
 user is already
 running arbitrary code. I don't really see how this would 
 create a greater
 security risk than what already exists.


 People can be endlessly creative at this kind of thing. I'm not 
 at all sure you and I have thought of every possible exploit.


Any idea how difficult such a thing would be to implement? Any 
one willing to work on something like this? If something like 
this was made, would it be included?
Jul 13 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/13/2013 1:34 AM, Tofu Ninja wrote:

 Any idea how difficult such a thing would be to implement? Any one willing to
 work on something like this? If something like this was made, would it be
included?


It would be easy to implement. I don't know if it should be included or not,
though.
Jul 13 2013
prev sibling parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
W dniu 13.07.2013 09:13, Walter Bright pisze:

 On 7/12/2013 11:52 PM, BLM768 wrote:

 On Saturday, 13 July 2013 at 04:23:56 UTC, Walter Bright wrote:

 A big problem with it would be the equivalent of the "SQL Injection
 Exploit".
 Since the compiler can now execute arbitrary code, someone passing
 around
 malicious source code could do anything to your system.


 Assuming that the user is compiling the code in order to run it (which
 does seem
 to be the most common case, at least in my experience), the user is
 already
 running arbitrary code. I don't really see how this would create a
 greater
 security risk than what already exists.


 People can be endlessly creative at this kind of thing. I'm not at all
 sure you and I have thought of every possible exploit.


Use sandboxing. On Linux it's easy: 
http://en.wikipedia.org/wiki/Seccomp. But, it could be difficult to 
create cross-platform solution.
Jul 13 2013
prev sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
On Fri, Jul 12, 2013 at 9:23 PM, Walter Bright
<newshound2 digitalmars.com>wrote:


 On 7/12/2013 5:00 PM, Timothee Cour wrote:


 Let's put it another way: if I or someone else made a pull request for
 CTFE
 "exec", would it have a chance of being accepted?


 A big problem with it would be the equivalent of the "SQL Injection
 Exploit". Since the compiler can now execute arbitrary code, someone
 passing around malicious source code could do anything to your system.


Which is why I suggested in my first post above that it should be enabled
by a compiler switch (eg: dmd -enable_exec) for safety reasons (just as
string import requires -J).
At the very least it is useful for debugging personal projects.
Jul 14 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jul 12, 2013 at 05:00:29PM -0700, Timothee Cour wrote:

 On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:
 

 On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:

 On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:

On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

So I had an idea recently, wouldn't it be cool to have the
ability to call an executable at compile time and capture its
output.


How long until D compilers are able to read mail? :-)




 Zawinski's law of software envelopment:

         Every program attempts to expand until it can read mail. Those
         programs which cannot so expand are replaced by ones which can.


 
 
 forget about reading mail, we currently don't even have hello world! (we
 have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)


Why should the compiler turn into a hello world program?


[...]

 Makefiles, for example, can't correctly recompile your program if you
 change compiler flags. And complex interdependencies, like you said,
 require painful manual maintenance, otherwise they tend to screw up and
 produce inconsistent results.

 For an example of a modern build system, see http://gittup.org/tup/.


 Using a build system would likely be non-portable (see current mess
 even in dmd source code, which requires posix.ma / win32.mak /
 win64.mak which is not DRY and hard to maintain), and suffer from the
 points I mentioned in my previous post.


Again, makefiles are a poor example of build systems. There *are* build
systems that allow you to write portable build scripts, such as SCons.
They're not perfect, but makefiles are a far cry from what they can do.



 We don't need yet another build system if we have "exec" available at
 CT.


That doesn't replace a proper build system. What if your external
program itself is a D program that needs to be compiled first? What if
you need to run an external program like povray in order to generate
images (say, animation frames) that are string-imported into your
program? Should DMD have the capability of running arbitrary programs
that produce arbitrary output and be able to sort out which programs
must run first and which output should be imported into the D program?

So we're no longer talking about the D compiler, but about the D
operating system. The real "DOS". :-P Which, mind you, would be a good
thing IMO, but I think it's a bit of a stretch from where we are
currently.



 Let's put it another way: if I or someone else made a pull request for
 CTFE "exec", would it have a chance of being accepted?


I'm not one to decide, you have to convince Walter.

And I'm not saying this idea is good or bad, I'm just saying that you
can already do such things without compiler support. At this point, I'd
rather we focus on stabilizing the language and improving its
implementation, than to continue adding new features that reinvent the
wheel.


T

-- 
For every argument for something, there is always an equal and opposite
argument against it. Debates don't give answers, only wounded or
inflated egos.
Jul 12 2013
prev sibling next sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
On Fri, Jul 12, 2013 at 5:10 PM, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:


 On Fri, Jul 12, 2013 at 05:00:29PM -0700, Timothee Cour wrote:

 On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh quickfur.ath.cx>


 wrote:

 On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:

 On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:

On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

So I had an idea recently, wouldn't it be cool to have the
ability to call an executable at compile time and capture its
output.


How long until D compilers are able to read mail? :-)




 Zawinski's law of software envelopment:

         Every program attempts to expand until it can read mail. Those
         programs which cannot so expand are replaced by ones which can.



 forget about reading mail, we currently don't even have hello world! (we
 have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)


 Why should the compiler turn into a hello world program?



For logging / debugging purposes, we often need to be able to print at
compile time some variables, but ctfeWriteln doesn't even work. And again,
pragma(msg) can't replace it.




 [...]

 Makefiles, for example, can't correctly recompile your program if you
 change compiler flags. And complex interdependencies, like you said,
 require painful manual maintenance, otherwise they tend to screw up and
 produce inconsistent results.

 For an example of a modern build system, see http://gittup.org/tup/.


 Using a build system would likely be non-portable (see current mess
 even in dmd source code, which requires posix.ma / win32.mak /
 win64.mak which is not DRY and hard to maintain), and suffer from the
 points I mentioned in my previous post.


 Again, makefiles are a poor example of build systems. There *are* build
 systems that allow you to write portable build scripts, such as SCons.
 They're not perfect, but makefiles are a far cry from what they can do.



 We don't need yet another build system if we have "exec" available at
 CT.


 That doesn't replace a proper build system.



I wrote a D based makefile-like tool in D (which uses direct acyclic graph
to find compilation order). It's much cleaner in D for obvious reasons.



 What if your external program itself is a D program that needs to be
 compiled first?



call exec("dmd mydeps.d"); (or exec("make -all") or the D based makefile)



 What if you need to run an external program like povray in order to
 generate
 images (say, animation frames) that are string-imported into your
 program?



I don't see how that's different:
void main(){
enum s=exec("program generating images");
import(s);
//use the string import
//or even simply: use s directly, depending on conditions
}




 Should DMD have the capability of running arbitrary programs
 that produce arbitrary output and be able to sort out which programs
 must run first and which output should be imported into the D program?


The order would be simply program order, as in the example above: "enum
s=exec(...)" is run during CTFE before import(s), as it comes first.



 So we're no longer talking about the D compiler, but about the D
 operating system. The real "DOS". :-P Which, mind you, would be a good
 thing IMO, but I think it's a bit of a stretch from where we are
 currently.


No, all it requires in terms of dmd modification is CTFE exec.
The user can use it to do complex stuff if he so wishes.

CTFE already runs very complex program (involving std.algorithm and
whatnot), so there's already a justification for running complex programs
during CTFE. Allowing that one simple function will enable many more
things.



 Let's put it another way: if I or someone else made a pull request for
 CTFE "exec", would it have a chance of being accepted?


 I'm not one to decide, you have to convince Walter.

 And I'm not saying this idea is good or bad, I'm just saying that you
 can already do such things without compiler support.



we can't even print variables during CTFE. Using a build system to do that
is way overkill.
Jul 12 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jul 12, 2013 at 05:39:45PM -0700, Timothee Cour wrote:

 On Fri, Jul 12, 2013 at 5:10 PM, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:
 

 On Fri, Jul 12, 2013 at 05:00:29PM -0700, Timothee Cour wrote:

 On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh quickfur.ath.cx>


 wrote:

 On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:

 On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:

On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

So I had an idea recently, wouldn't it be cool to have the
ability to call an executable at compile time and capture its
output.


How long until D compilers are able to read mail? :-)




 Zawinski's law of software envelopment:

         Every program attempts to expand until it can read mail.
         Those programs which cannot so expand are replaced by
         ones which can.



 forget about reading mail, we currently don't even have hello world! (we
 have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)


 Why should the compiler turn into a hello world program?


 
 
 For logging / debugging purposes, we often need to be able to print at
 compile time some variables, but ctfeWriteln doesn't even work. And
 again, pragma(msg) can't replace it.


This is a bug / limitation in CTFE, not a justification for invoking
arbitrary programs from the compiler.


[...]

 We don't need yet another build system if we have "exec" available
 at CT.


 That doesn't replace a proper build system.


 
 
 I wrote a D based makefile-like tool in D (which uses direct acyclic
 graph to find compilation order). It's much cleaner in D for obvious
 reasons.


I'm interested. Is the code on github?



 What if your external program itself is a D program that needs to be
 compiled first?


 
 
 call exec("dmd mydeps.d"); (or exec("make -all") or the D based makefile)


So basically you're saying DMD should be expanded into a D shell?


[...]

 So we're no longer talking about the D compiler, but about the D
 operating system. The real "DOS". :-P Which, mind you, would be a good
 thing IMO, but I think it's a bit of a stretch from where we are
 currently.


 
 No, all it requires in terms of dmd modification is CTFE exec.
 The user can use it to do complex stuff if he so wishes.
 
 CTFE already runs very complex program (involving std.algorithm and
 whatnot), so there's already a justification for running complex
 programs during CTFE. Allowing that one simple function will enable
 many more things.


In the old days, somebody wrote a 1-line universal C program that could
do literally *anything*. All you have to do is to specify, in C syntax,
what you want the program to do. The program's source code is:

	#include "/dev/tty"

:-)  (Sadly, this no longer works on modern systems.)

Seriously, though, I think adding an exec function is total overkill for
working around CTFE limitations. If that's what we're trying to solve,
we should be looking at how to improve CTFE, not adding a nuclear
warhead to dmd.



 Let's put it another way: if I or someone else made a pull request
 for CTFE "exec", would it have a chance of being accepted?


 I'm not one to decide, you have to convince Walter.

 And I'm not saying this idea is good or bad, I'm just saying that
 you can already do such things without compiler support.


 
 
 we can't even print variables during CTFE. Using a build system to do
 that is way overkill.


If that's the root problem, then we should fix/improve CTFE. I'm not
convinced that this should be grounds for adding an exec function to
dmd. It seems to be complete overkill just for working around CTFE
limitations.

But maybe Walter has a different opinion about this. :)


T

-- 
Let's call it an accidental feature. -- Larry Wall
Jul 12 2013
prev sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
On Fri, Jul 12, 2013 at 6:14 PM, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:


 On Fri, Jul 12, 2013 at 05:39:45PM -0700, Timothee Cour wrote:

 On Fri, Jul 12, 2013 at 5:10 PM, H. S. Teoh <hsteoh quickfur.ath.cx>


 wrote:

 On Fri, Jul 12, 2013 at 05:00:29PM -0700, Timothee Cour wrote:

 On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh quickfur.ath.cx>


 wrote:

 On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:

 On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:

On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

So I had an idea recently, wouldn't it be cool to have the
ability to call an executable at compile time and capture its
output.


How long until D compilers are able to read mail? :-)




 Zawinski's law of software envelopment:

         Every program attempts to expand until it can read mail.
         Those programs which cannot so expand are replaced by
         ones which can.



 forget about reading mail, we currently don't even have hello world!






 (we

 have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)


 Why should the compiler turn into a hello world program?



 For logging / debugging purposes, we often need to be able to print at
 compile time some variables, but ctfeWriteln doesn't even work. And
 again, pragma(msg) can't replace it.


 This is a bug / limitation in CTFE, not a justification for invoking
 arbitrary programs from the compiler.


there would be so many other things to fix (what if I want to access
environment variables? or know whether a program / library is available?).
Whereas simply allowing CTFE exec unleashes arbitrarily complex use cases.
The choice would be up to the user to make use of it or not.


[...]

 We don't need yet another build system if we have "exec" available
 at CT.


 That doesn't replace a proper build system.



 I wrote a D based makefile-like tool in D (which uses direct acyclic
 graph to find compilation order). It's much cleaner in D for obvious
 reasons.


 I'm interested. Is the code on github?



Not yet, code isn't very presentable at the moment but API is easy to use;
user can mix both pure D and a makefile-like DSL syntax, so it makes simple
things simple and complex things possible. It's DRY (nothing has to be
repeated twice, which makes maintenance easy). I can push it to github if
there's interest, but my point is that doing such things in D is easier
than one might think.


 What if your external program itself is a D program that needs to be

 compiled first?



 call exec("dmd mydeps.d"); (or exec("make -all") or the D based makefile)


 So basically you're saying DMD should be expanded into a D shell?


No I never intended that. the call to exec("dmd...") would be a regular
statement in a regular D file to be compiled. All I'm suggesting is making
at least some of the functions in std.process CTFE-able.



 [...]

 So we're no longer talking about the D compiler, but about the D
 operating system. The real "DOS". :-P Which, mind you, would be a good
 thing IMO, but I think it's a bit of a stretch from where we are
 currently.


 No, all it requires in terms of dmd modification is CTFE exec.
 The user can use it to do complex stuff if he so wishes.

 CTFE already runs very complex program (involving std.algorithm and
 whatnot), so there's already a justification for running complex
 programs during CTFE. Allowing that one simple function will enable
 many more things.


 In the old days, somebody wrote a 1-line universal C program that could
 do literally *anything*. All you have to do is to specify, in C syntax,
 what you want the program to do. The program's source code is:

         #include "/dev/tty"

 :-)  (Sadly, this no longer works on modern systems.)

 Seriously, though, I think adding an exec function is total overkill for
 working around CTFE limitations. If that's what we're trying to solve,
 we should be looking at how to improve CTFE, not adding a nuclear
 warhead to dmd.


I disagree. From the compiler implementation perspective, it's very easy
(only requires making some of std.process CTFE-able), yet unleashes lots of
potential use cases. Versus what you're suggesting which is to provide
implementation for each individual CTFE limitation one at a time.

Being lazy is good :-)

A similar argument could be made for runtime: if we didn't have
std.process, we'd have to reimplement or wrap "in D" all missing features.
But thanks to std.process I can easily add many missing D functions, eg for
readLink:
----
string realPath(string file){
import std.process;
return runPython(`import os;
a=os.path.realpath(`~file.escapeShellFileName~`); print(a);`);
}
string runPython(string code){
import std.process;
string command=`python -c `~code.escapeShellFileName;
return command.executeShell.outputThrows;
}
----

bam, that was easy.

I see no point in arbitrarily impose restrictions on future use cases.




 Let's put it another way: if I or someone else made a pull request
 for CTFE "exec", would it have a chance of being accepted?


 I'm not one to decide, you have to convince Walter.

 And I'm not saying this idea is good or bad, I'm just saying that
 you can already do such things without compiler support.



 we can't even print variables during CTFE. Using a build system to do
 that is way overkill.


 If that's the root problem, then we should fix/improve CTFE. I'm not
 convinced that this should be grounds for adding an exec function to
 dmd. It seems to be complete overkill just for working around CTFE
 limitations.

 But maybe Walter has a different opinion about this. :)


ping, Walter ?
Jul 12 2013
prev sibling parent reply Timothee Cour <thelastmammoth gmail.com> writes:
On Fri, Jul 12, 2013 at 3:36 PM, Peter Alexander <
peter.alexander.au gmail.com> wrote:


 On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:


 So I had an idea recently, wouldn't it be cool to have the ability to
 call an executable at compile time and capture its output.


 How long until D compilers are able to read mail? :-)

 There's many obvious applications of this proposed feature, but I say such
 things should be delegated to the build process. Just run those executables
 using a makefile, or whatever build system you use.

 If you need compile time inputs from your D code then just run a separate
 build to extract them. Yes it's more work and less convenient, but I think
 we need to be careful with how much is added to the D compilers. We don't
 want to turn them into operating systems.


regarding added complexity: the only thing this adds is 1 function (calling
an executable, with option to redirect stdin/out/err). And yes, that could
read mail as you joked if the user called such a program inside his D
function, but that would require ZERO change in compiler, apart from that
ONE function to support calling external programs.

we need this for the same reason we need CTFE: try using makefiles to
achieve what CTFE does. Using a separate build is way less convenient and
doesn't allow complex interactions, as it requires the process to be
sequential: do other stuff THEN compile with dmd. Whereas integrating it
inside compilation would allow interdependent computations offloaded to an
external process.

With makefile-like build, you'd have to have potentially many layers of
interaction between compiling and calling external programs. Not only would
that be much slower (dmd has overhead), but would also require parsing
files each time.
Jul 12 2013
parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Friday, 12 July 2013 at 23:34:05 UTC, Timothee Cour wrote:

 regarding added complexity: the only thing this adds is 1 
 function (calling
 an executable, with option to redirect stdin/out/err). And yes, 
 that could
 read mail as you joked if the user called such a program inside 
 his D
 function, but that would require ZERO change in compiler, apart 
 from that
 ONE function to support calling external programs.


It's one function now. Like CTFE, it won't be long before 
everyone is using it and we see its limitations, or we see more 
opportunities for further enhancement.



 we need this for the same reason we need CTFE: try using 
 makefiles to
 achieve what CTFE does. Using a separate build is way less 
 convenient and
 doesn't allow complex interactions, as it requires the process 
 to be
 sequential: do other stuff THEN compile with dmd. Whereas 
 integrating it
 inside compilation would allow interdependent computations 
 offloaded to an
 external process.


Let's be clear here: we don't "need" this we "want" this.

There's an endless number of features we want, but we have to 
draw the line somewhere. This is classic feature creep. As Andrei 
said at DConf, we need to be more professional in the development 
of D. Endlessly adding features is not a professional approach to 
software development.

In my opinion, the only features that should be added at this 
point are to fix language problems. For example, some form of 
copy constructors will be necessary to fix the const postblit 
problem. Adding features for convenience should be postponed for 
D3, or at the very least postponed until all major language 
issues are resolved.
Jul 13 2013
prev sibling next sibling parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
On 7/12/13 5:42 PM, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the ability to
 call an executable at compile time and capture its output. Something
 like the string imports but instead of opening and reading a text file,
 it run an executable, waits for it to finish, and grabs its output.

 It would get really cool if you could pass this executable some args and
 then mix in its out put into your own code. It could be used similarly
 to how CTFE are used but with out the overhead of trying to compile that
 function and what not and with out the limitations on what it can do.

 I could imagine all sorts of things that would be possible with this
 that is currently not.

 Not sure if this is something that could be implemented easily, but
 seems like something that could be done and something that would be
 really cool.


What would actually be cool is to run any kind of code at compile time 
(within a sandbox, if you want). For example reading a mysql database 
and generating classes for the tables at compile time. No need to run a 
separate executable and remember to run it before compiling your program.

But this has been discussed many times in this list, and it won't 
happen, because compile time functions are not jitted and run, they are 
just interpreted.

You ask for executing a program at compile time. Somebody will ask 
reading a file at compile time. Another one will ask reading a schema 
from a database. Yet another will want to do a web scraper at compile 
time to generate efficient code from that (not sure that's useful :-P)
Jul 13 2013
next sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
On Sat, Jul 13, 2013 at 9:05 AM, Ary Borenszweig <ary esperanto.org.ar>wrote:


 On 7/12/13 5:42 PM, Tofu Ninja wrote:


 So I had an idea recently, wouldn't it be cool to have the ability to
 call an executable at compile time and capture its output. Something
 like the string imports but instead of opening and reading a text file,
 it run an executable, waits for it to finish, and grabs its output.

 It would get really cool if you could pass this executable some args and
 then mix in its out put into your own code. It could be used similarly
 to how CTFE are used but with out the overhead of trying to compile that
 function and what not and with out the limitations on what it can do.

 I could imagine all sorts of things that would be possible with this
 that is currently not.

 Not sure if this is something that could be implemented easily, but
 seems like something that could be done and something that would be
 really cool.


 What would actually be cool is to run any kind of code at compile time
 (within a sandbox, if you want). For example reading a mysql database and
 generating classes for the tables at compile time. No need to run a
 separate executable and remember to run it before compiling your program.

 But this has been discussed many times in this list, and it won't happen,
 because compile time functions are not jitted and run, they are just
 interpreted.

 You ask for executing a program at compile time. Somebody will ask reading
 a file at compile time. Another one will ask reading a schema from a
 database. Yet another will want to do a web scraper at compile time to
 generate efficient code from that (not sure that's useful :-P)


reading a file at compile time is already possible with string import.
Jul 14 2013
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/13/2013 09:05 AM, Ary Borenszweig wrote:


 What would actually be cool is to run any kind of code at compile time
 (within a sandbox, if you want). For example reading a mysql database
 and generating classes for the tables at compile time. No need to run a
 separate executable and remember to run it before compiling your program.


Humans make mistakes all the time, especially with repetitive tasks. 
That's why such tasks are handled by tools like 'make'.

Ali
Jul 15 2013
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
On 7/15/13 2:03 PM, Ali Çehreli wrote:

 On 07/13/2013 09:05 AM, Ary Borenszweig wrote:

  > What would actually be cool is to run any kind of code at compile time
  > (within a sandbox, if you want). For example reading a mysql database
  > and generating classes for the tables at compile time. No need to run a
  > separate executable and remember to run it before compiling your
 program.

 Humans make mistakes all the time, especially with repetitive tasks.
 That's why such tasks are handled by tools like 'make'.

 Ali


Why have CTFE at all if you can produce the desired code with an 
external program, output it to some file and then use it with 
import("file")? Just use 'make'.

I believe a programming language should be as self-contained as possible 
for building an executable out of it. The more you can do within the 
language itself, minimizing external dependencies, the better. Because 
users need only to understand the language. They don't need to 
understand make, build systems and whatnot.

But that's just my opinion...
Jul 15 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jul 15, 2013 at 02:06:47PM -0300, Ary Borenszweig wrote:

 On 7/15/13 2:03 PM, Ali Çehreli wrote:

On 07/13/2013 09:05 AM, Ary Borenszweig wrote:


 What would actually be cool is to run any kind of code at compile
 time (within a sandbox, if you want). For example reading a mysql
 database and generating classes for the tables at compile time. No
 need to run a separate executable and remember to run it before
 compiling your program.


Humans make mistakes all the time, especially with repetitive tasks.
That's why such tasks are handled by tools like 'make'.

Ali


 
 Why have CTFE at all if you can produce the desired code with an
 external program, output it to some file and then use it with
 import("file")? Just use 'make'.


That's an invalid argument. CTFE allows compile-time introspection of
types and declarations within the language. To move this outside the
compiler would require essentially re-implementing the compiler so that
you can parse your code to extract types, declarations, etc., in order
to perform CTFE. This means it's better for it to be in the compiler.

OTOH, executing arbitrary programs has nothing to do with the compiling
of D source code. That belongs to the OS level of operations, not inside
the compiler.



 I believe a programming language should be as self-contained as
 possible for building an executable out of it. The more you can do
 within the language itself, minimizing external dependencies, the
 better. Because users need only to understand the language. They don't
 need to understand make, build systems and whatnot.


[...]

I don't understand this reluctance to learn how to use build systems.
Why insist on building a house with only a hammer, and refuse to learn
how to use a screwdriver and wrench? Sure, you *can* build a house with
only a hammer if you try hard enough, but why not use tools that already
exist?

(Of course, the problem is exacerbated by make being a really poor
example of a build system, but also the one most people know about.
Perhaps that explains part of the bad rep of build systems.)


T

-- 
What are you when you run out of Monet? Baroque.
Jul 15 2013
prev sibling next sibling parent reply "Don" <prosthetictelevisions teletubby.medical.com> writes:
On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the 
 ability to call an executable at compile time and capture its 
 output. Something like the string imports but instead of 
 opening and reading a text file, it run an executable, waits 
 for it to finish, and grabs its output.

 It would get really cool if you could pass this executable some 
 args and then mix in its out put into your own code. It could 
 be used similarly to how CTFE are used but with out the 
 overhead of trying to compile that function and what not and 
 with out the limitations on what it can do.

 I could imagine all sorts of things that would be possible with 
 this that is currently not.

 Not sure if this is something that could be implemented easily, 
 but seems like something that could be done and something that 
 would be really cool.


I personally think it's a *horrible* idea. It's one of those 
things which looks good in small cases but doesn't scale.

Don't underestimate the fact that the compiler is a deterministic 
program at present. When compiled with the same flags, on the 
same set of source files, the results should always be the same. 
It's a classic filter program.

input -> compiler -> output

Losing that purity is a HUGE thing to give up.
Jul 15 2013
next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 15 July 2013 at 13:26:13 UTC, Don wrote:

 On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the 
 ability to call an executable at compile time and capture its 
 output. Something like the string imports but instead of 
 opening and reading a text file, it run an executable, waits 
 for it to finish, and grabs its output.

 It would get really cool if you could pass this executable 
 some args and then mix in its out put into your own code. It 
 could be used similarly to how CTFE are used but with out the 
 overhead of trying to compile that function and what not and 
 with out the limitations on what it can do.

 I could imagine all sorts of things that would be possible 
 with this that is currently not.

 Not sure if this is something that could be implemented 
 easily, but seems like something that could be done and 
 something that would be really cool.


 I personally think it's a *horrible* idea. It's one of those 
 things which looks good in small cases but doesn't scale.

 Don't underestimate the fact that the compiler is a 
 deterministic program at present. When compiled with the same 
 flags, on the same set of source files, the results should 
 always be the same. It's a classic filter program.

 input -> compiler -> output

 Losing that purity is a HUGE thing to give up.


Ehh... to say its deterministic is kinda a stretch, for one, 
compiling on different systems could result in different output 
with the ability to version different sections of code. Also as 
has been mentioned before, string imports already make the same 
kind of break in purity.

If the programmer wants to use it then they also should be well 
aware of the consequences of such a decision, which can be 
considered on a case by case basis.

We should be as flexible as possible and leave the decision up to 
the programmer. We shouldn't restrict a feature based on a 
fictional idea of purity or vague worry of feature bloat(as 
others have mentioned) especially in a case like this where the 
implementation is fairly simple and the use cases are immense.
Jul 15 2013
next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Monday, 15 July 2013 at 15:05:39 UTC, Tofu Ninja wrote:

 On Monday, 15 July 2013 at 13:26:13 UTC, Don wrote:

 Don't underestimate the fact that the compiler is a 
 deterministic program at present. When compiled with the same 
 flags, on the same set of source files, the results should 
 always be the same. It's a classic filter program.

 input -> compiler -> output

 Losing that purity is a HUGE thing to give up.


 Ehh... to say its deterministic is kinda a stretch, for one, 
 compiling on different systems could result in different output 
 with the ability to version different sections of code. Also as 
 has been mentioned before, string imports already make the same 
 kind of break in purity.


If you want to be technical then yes, the purity takes the 
operating system as an input.

String imports don't break purity, they just add an input.

By allowing arbitrary code execution, you essentially require the 
whole state of the host machine as an input.


 If the programmer wants to use it then they also should be well 
 aware of the consequences of such a decision, which can be 
 considered on a case by case basis.

 We should be as flexible as possible and leave the decision up 
 to the programmer. We shouldn't restrict a feature based on a 
 fictional idea of purity or vague worry of feature bloat(as 
 others have mentioned) especially in a case like this where the 
 implementation is fairly simple and the use cases are immense.


No, this is a bad attitude to have. If you add a feature that is 
easy to misuse then it WILL be misused. Just looks at macros in C.

Any program that uses compile time executable calling will almost 
certainly become non-portable. This is not a good thing.
Jul 15 2013
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/15/2013 08:05 AM, Tofu Ninja wrote:


 We should be as flexible as possible and leave the decision up to the
 programmer. We shouldn't restrict a feature based on a fictional idea of
 purity or vague worry of feature bloat(as others have mentioned)


My worry is not vague at all. What stops one from adding emailing 
ability to the compiler then? That would be immensely useful too becaues 
the compiler could email me. (!) Where is the line?

A compiler is a tool that compiles source code in a preset environment. 
Simple is better.


 especially in a case like this where the implementation is fairly simple
 and the use cases are immense.


The use cases are not immense compared to the current situation. There 
are lots of free tools that can set the environment for compilation and 
the compiler compiles.

Ali
Jul 15 2013
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 15 July 2013 at 17:08:48 UTC, Ali Çehreli wrote:

 On 07/15/2013 08:05 AM, Tofu Ninja wrote:


 We should be as flexible as possible and leave the decision


 up to the

 programmer. We shouldn't restrict a feature based on a


 fictional idea of

 purity or vague worry of feature bloat(as others have


 mentioned)

 My worry is not vague at all. What stops one from adding 
 emailing ability to the compiler then? That would be immensely 
 useful too becaues the compiler could email me. (!) Where is 
 the line?

 A compiler is a tool that compiles source code in a preset 
 environment. Simple is better.


 especially in a case like this where the implementation is


 fairly simple

 and the use cases are immense.


 The use cases are not immense compared to the current 
 situation. There are lots of free tools that can set the 
 environment for compilation and the compiler compiles.

 Ali


The use cases I am more interested in are not possible with make. 
Having the ability to pass the arguments from within the language 
itself allows you to define your use cases inline instead of 
having to separately define the use case outside. Something where 
you would have many different points in the code that needed 
different input from what ever executable you were calling. With 
out ctec you would have to maintain a list of every time you 
needed something generated from the outside and add that to your 
make which is error prone. I think it is in these cases where the 
ctec would be the most useful as it would remove a lot of the 
burden of keeping track of all the pregenerated code.
Jul 15 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jul 15, 2013 at 07:35:44PM +0200, Tofu Ninja wrote:
[...]

 The use cases I am more interested in are not possible with make.
 Having the ability to pass the arguments from within the language
 itself allows you to define your use cases inline instead of having
 to separately define the use case outside. Something where you would
 have many different points in the code that needed different input
 from what ever executable you were calling. With out ctec you would
 have to maintain a list of every time you needed something generated
 from the outside and add that to your make which is error prone. I
 think it is in these cases where the ctec would be the most useful
 as it would remove a lot of the burden of keeping track of all the
 pregenerated code.


You mean, you want to do something like this?

	static inputValues = [
		"data1", "data2", "data3"
	];

	static foreach (val; inputValues) {
		import exec("prog", val);
	}

I can see why that could be useful.


T

-- 
By understanding a machine-oriented language, the programmer will tend
to use a much more efficient method; it is much closer to reality. -- D.
Knuth
Jul 15 2013
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 15 July 2013 at 17:49:04 UTC, H. S. Teoh wrote:

 On Mon, Jul 15, 2013 at 07:35:44PM +0200, Tofu Ninja wrote:
 [...]

 The use cases I am more interested in are not possible with 
 make.
 Having the ability to pass the arguments from within the 
 language
 itself allows you to define your use cases inline instead of 
 having
 to separately define the use case outside. Something where you 
 would
 have many different points in the code that needed different 
 input
 from what ever executable you were calling. With out ctec you 
 would
 have to maintain a list of every time you needed something 
 generated
 from the outside and add that to your make which is error 
 prone. I
 think it is in these cases where the ctec would be the most 
 useful
 as it would remove a lot of the burden of keeping track of all 
 the
 pregenerated code.


 You mean, you want to do something like this?

 	static inputValues = [
 		"data1", "data2", "data3"
 	];

 	static foreach (val; inputValues) {
 		import exec("prog", val);
 	}

 I can see why that could be useful.


 T


  Or something more complex where a template generates its code 
based on the output of an executable, so in that case the 
template args would be passed into the executable call.
Jul 15 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jul 15, 2013 at 07:54:56PM +0200, Tofu Ninja wrote:

 On Monday, 15 July 2013 at 17:49:04 UTC, H. S. Teoh wrote:

On Mon, Jul 15, 2013 at 07:35:44PM +0200, Tofu Ninja wrote:
[...]

The use cases I am more interested in are not possible with make.
Having the ability to pass the arguments from within the language
itself allows you to define your use cases inline instead of having
to separately define the use case outside. Something where you would
have many different points in the code that needed different input
from what ever executable you were calling. With out ctec you would
have to maintain a list of every time you needed something generated
from the outside and add that to your make which is error prone.  I
think it is in these cases where the ctec would be the most useful
as it would remove a lot of the burden of keeping track of all the
pregenerated code.


You mean, you want to do something like this?

	static inputValues = [
		"data1", "data2", "data3"
	];

	static foreach (val; inputValues) {
		import exec("prog", val);
	}

I can see why that could be useful.


T


 Or something more complex where a template generates its code based
 on the output of an executable, so in that case the template args
 would be passed into the executable call.


Hmm. Let's see:

	module dmd;
	static grammar = import exec("wget", "http://dlang.org/spec");
	static c_code = import exec("yacc", grammar);
	void main() {
		import exec("d-ify", c_code);
	}

Hey look, we just solved the problem of rewriting DMD in D, *and* the
problem of the spec being out-of-sync with the compiler! :-P

On a more serious note, though, if I ever needed a template of the sort
you're describing, I'd prefer to use CTFE to generate it (and
fix/improve CTFE if for whatever reason it can't yet do what I want it
to do).

The problem with invoking an external program to produce template code
is that you have to somehow serialize the template arguments so that you
can pass them to the program, and the program has to somehow deserialize
them and reconstruct what the compiler has already constructed. Whereas
if you did everything in CTFE, you have access to whatever the compiler
has internally built to describe the types and values.  Serializing /
deserializing these structures generally are a big hiding place for
bugs; I'd imagine that CTFE, being part of the compiler, is far more
reliable at interpreting these structures correctly.

Besides, I'd hate for my D code to have opaque blocks of import exec()
where I have to examine another program to figure out exactly what's
being compiled, instead of just reading the source code.

So I'm still kinda on the fence about this whole thing (ignoring, of
course, the major security concerns involving having import exec() in
the first place).

What I really want to see, is for CTFE to improve to the point where you
can essentially write an arbitrary program that runs in CTFE to produce
whatever code you wish to produce for the executable, thus making import
exec() redundant.


T

-- 
MASM = Mana Ada Sistem, Man!
Jul 15 2013
prev sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
On Mon, Jul 15, 2013 at 10:35 AM, Tofu Ninja <emmons0 purdue.edu> wrote:


 On Monday, 15 July 2013 at 17:08:48 UTC, Ali =C7ehreli wrote:


 On 07/15/2013 08:05 AM, Tofu Ninja wrote:


 We should be as flexible as possible and leave the decision


 up to the

 programmer. We shouldn't restrict a feature based on a


 fictional idea of

 purity or vague worry of feature bloat(as others have


 mentioned)

 My worry is not vague at all. What stops one from adding emailing abilit=




y

 to the compiler then? That would be immensely useful too becaues the
 compiler could email me. (!) Where is the line?

 A compiler is a tool that compiles source code in a preset environment.
 Simple is better.


 especially in a case like this where the implementation is


 fairly simple

 and the use cases are immense.


 The use cases are not immense compared to the current situation. There
 are lots of free tools that can set the environment for compilation and =




the

 compiler compiles.

 Ali


 The use cases I am more interested in are not possible with make. Having
 the ability to pass the arguments from within the language itself allows
 you to define your use cases inline instead of having to separately defin=


e

 the use case outside. Something where you would have many different point=


s

 in the code that needed different input from what ever executable you wer=


e

 calling. With out ctec you would have to maintain a list of every time yo=


u

 needed something generated from the outside and add that to your make whi=


ch

 is error prone. I think it is in these cases where the ctec would be the
 most useful as it would remove a lot of the burden of keeping track of al=


l

 the pregenerated code.


Instead of debating forever we should implement it and make a pull request
for it. People can test it individually and see whether it creates havoc or
enables interesting scenarios. Then we can decide after some time.
Jul 15 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/15/13 6:26 AM, Don wrote:

 On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:

 So I had an idea recently, wouldn't it be cool to have the ability to
 call an executable at compile time and capture its output. Something
 like the string imports but instead of opening and reading a text
 file, it run an executable, waits for it to finish, and grabs its output.

 It would get really cool if you could pass this executable some args
 and then mix in its out put into your own code. It could be used
 similarly to how CTFE are used but with out the overhead of trying to
 compile that function and what not and with out the limitations on
 what it can do.

 I could imagine all sorts of things that would be possible with this
 that is currently not.

 Not sure if this is something that could be implemented easily, but
 seems like something that could be done and something that would be
 really cool.


 I personally think it's a *horrible* idea. It's one of those things
 which looks good in small cases but doesn't scale.


I think the right direction here is to factor compiler parts as a 
library. Then user code may insert external artifacts (notably REPL) on 
its own.

Andrei
Jul 15 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 16 July 2013 at 04:10:35 UTC, Andrei Alexandrescu 
wrote:

 I think the right direction here is to factor compiler parts as 
 a library. Then user code may insert external artifacts 
 (notably REPL) on its own.

 Andrei


SDC is already based on that principle, but still far from 
release state.
Jul 15 2013
prev sibling next sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
On Mon, Jul 15, 2013 at 9:10 PM, Andrei Alexandrescu <
SeeWebsiteForEmail erdani.org> wrote:


 On 7/15/13 6:26 AM, Don wrote:


 On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:


 So I had an idea recently, wouldn't it be cool to have the ability to
 call an executable at compile time and capture its output. Something
 like the string imports but instead of opening and reading a text
 file, it run an executable, waits for it to finish, and grabs its output.

 It would get really cool if you could pass this executable some args
 and then mix in its out put into your own code. It could be used
 similarly to how CTFE are used but with out the overhead of trying to
 compile that function and what not and with out the limitations on
 what it can do.

 I could imagine all sorts of things that would be possible with this
 that is currently not.

 Not sure if this is something that could be implemented easily, but
 seems like something that could be done and something that would be
 really cool.


 I personally think it's a *horrible* idea. It's one of those things
 which looks good in small cases but doesn't scale.


 I think the right direction here is to factor compiler parts as a library.
 Then user code may insert external artifacts (notably REPL) on its own.


CTFE exec would still be needed (see yet another use case in EMAIL:"Proof
of concept: automatically import C header files"), but having a
compiler-as-a-library would make implementation easier and more
customizable, eg by redirecting certain symbols  such as 'CTFE exec' to a
given delegate.

myfile.d:
import std.process;
void main(){enum HOME=exec("echo $HOME", Redirect.all);}

driver.d:
import std.compiler;
void exec_dg(){...}
void main(){"myfile.d".readText.compile(&exec_dg);}
Jul 16 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-07-16 06:10, Andrei Alexandrescu wrote:


 I think the right direction here is to factor compiler parts as a
 library.


Yes, I agree. I've been wanting this for a long time.

-- 
/Jacob Carlborg
Jul 17 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jul 17, 2013 at 10:34:32AM +0200, Jacob Carlborg wrote:

 On 2013-07-16 06:10, Andrei Alexandrescu wrote:
 

I think the right direction here is to factor compiler parts as a
library.


 
 Yes, I agree. I've been wanting this for a long time.


[...]

We've been talking about this for a while. Have we made any progress in
this direction since?


T

-- 
What do you get if you drop a piano down a mineshaft? A flat minor.
Jul 17 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-07-17 16:55, H. S. Teoh wrote:


 We've been talking about this for a while. Have we made any progress in
 this direction since?


There's been a bunch of pull requests merged related to adopting the DMD 
source code to make it work with the tool, I think it was, Daniel Murphy 
has created. This tool will be used to automatically port DMD to D. It's 
designed specifically for the DMD code base.

Look for pull request tagged with [DDMD] in the title.

Then I think the idea is refactor DMD to make it usable as a library.

-- 
/Jacob Carlborg
Jul 17 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jul 17, 2013 at 05:40:03PM +0200, Jacob Carlborg wrote:

 On 2013-07-17 16:55, H. S. Teoh wrote:
 

We've been talking about this for a while. Have we made any progress
in this direction since?


 
 There's been a bunch of pull requests merged related to adopting the
 DMD source code to make it work with the tool, I think it was,
 Daniel Murphy has created. This tool will be used to automatically
 port DMD to D. It's designed specifically for the DMD code base.
 
 Look for pull request tagged with [DDMD] in the title.
 
 Then I think the idea is refactor DMD to make it usable as a library.


[...]

Wow. That sounds like it's still a long ways off. But at least we're
making progress.


T

-- 
The volume of a pizza of thickness a and radius z can be described by
the following formula: pi zz a. -- Wouter Verhelst
Jul 17 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-07-18 00:15, H. S. Teoh wrote:


 Wow. That sounds like it's still a long ways off. But at least we're
 making progress.


Oh, forgot to mention. There are several other people here that are 
working on implementing a D front end. These are mostly lexers. There's 
one in the review queue, std.d.lexer, by Brian Schott:

http://wiki.dlang.org/Review_Queue

-- 
/Jacob Carlborg
Jul 17 2013
prev sibling parent "Tofu Ninja" <emmons0 purdue.edu> writes:
So if some one were to implement this, any ideas on the preferred 
syntax for such a feature?
Jul 17 2013