www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Running external program from a D program [D2]

reply Jonathan M Davis <jmdavisProg gmail.com> writes:
Okay, I'm looking to be able to run an external program from within my D 
program. The library functions for this appear to be in std.process. You 
have system() and various versions of execv(). system() makes a call in 
shell. execv() executes the program without a shell.

What I'd _like_ to be able to do is run the program without a shell so that 
I don't have to worry about escaping special characters in file names, and I 
want to have access to the output from the program. This poses two problems.

1. Is there a way to run a program without the shell and without the call 
terminating the program (per bugzilla 3158, execv takes over your program 
and terminates it when it's done, in spite of what the documentation says - 
it's certainly been killing my programs when I've tried)? The only way that 
I see to do that at the moment is to spawn a separate thread and run execv 
in it. I'd prefer to just make the call and wait for it to return. Is there 
a way to do so?

2. Is there a way to get at what the called program sends to stdout? I'm 
afraid that I don't have a clue how to get at that.

Any help would be appreciated. Thanks.

- Jonathan M Davis
Feb 21 2010
next sibling parent Jonathan M Davis <jmdavisProg gmail.com> writes:
Jonathan M Davis wrote:


 Okay, I'm looking to be able to run an external program from within my D
 program. The library functions for this appear to be in std.process. You
 have system() and various versions of execv(). system() makes a call in
 shell. execv() executes the program without a shell.
 
 What I'd _like_ to be able to do is run the program without a shell so
 that I don't have to worry about escaping special characters in file
 names, and I want to have access to the output from the program. This
 poses two problems.
 
 1. Is there a way to run a program without the shell and without the call
 terminating the program (per bugzilla 3158, execv takes over your program
 and terminates it when it's done, in spite of what the documentation says
 - it's certainly been killing my programs when I've tried)? The only way
 that I see to do that at the moment is to spawn a separate thread and run
 execv in it. I'd prefer to just make the call and wait for it to return.
 Is there a way to do so?


Well, upon actually trying execv() within a separate thread, it _still_ 
kills the program, so that doesn't seem to fly for some reason, which means 
that I don't even have a poor solution for successively calling a program 
from within D without using the shell. At the moment, it looks like phobos 
could really use such a function, but I don't know enough about the 
underlying system calls available to know how easy that is if the system's 
execv() always terminates the program.


 
 2. Is there a way to get at what the called program sends to stdout? I'm
 afraid that I don't have a clue how to get at that.


Upon closer examination, it looks like someone asked this question fairly 
recently on this list and didn't really get a useful response, so maybe 
there isn't a good way to do this. If not, then it would be _really_ nice if 
it were to be added to phobos.


 
 Any help would be appreciated. Thanks.
 
 - Jonathan M Davis
Feb 21 2010
prev sibling next sibling parent reply Jesse Phillips <jessekphillips+D gmail.com> writes:
Jonathan M Davis wrote:


 Okay, I'm looking to be able to run an external program from within my D 
 program. The library functions for this appear to be in std.process. You 
 have system() and various versions of execv(). system() makes a call in 
 shell. execv() executes the program without a shell.

 What I'd _like_ to be able to do is run the program without a shell so that 
 I don't have to worry about escaping special characters in file names, and I 
 want to have access to the output from the program. This poses two problems.

 1. Is there a way to run a program without the shell and without the call 
 terminating the program (per bugzilla 3158, execv takes over your program 
 and terminates it when it's done, in spite of what the documentation says - 
 it's certainly been killing my programs when I've tried)? The only way that 
 I see to do that at the moment is to spawn a separate thread and run execv 
 in it. I'd prefer to just make the call and wait for it to return. Is there 
 a way to do so?

 2. Is there a way to get at what the called program sends to stdout? I'm 
 afraid that I don't have a clue how to get at that.

 Any help would be appreciated. Thanks.

 - Jonathan M Davis


As you may have noticed by the comments to on bug 3158. exec()[1] calls
replace your process, this means it will not continue your program. To
get arround this you find that people will first fork()[2] and exec on
the child process.

In order to get your standard output you would use the dup2()[3] call.

Please note that I do not know how any of this will related to practices
on Windows. And sadly this is not D specific and actually C stuff.

1. http://www.opengroup.org/onlinepubs/007908799/xsh/exec.html
2. http://www.yolinux.com/TUTORIALS/ForkExecProcesses.html
3. http://www.mkssoftware.com/docs/man3/dup2.3.asp
Feb 21 2010
parent reply Jonathan M Davis <jmdavisProg gmail.com> writes:
Jesse Phillips wrote:

 As you may have noticed by the comments to on bug 3158. exec()[1] calls
 replace your process, this means it will not continue your program. To
 get arround this you find that people will first fork()[2] and exec on
 the child process.


Ah yes. Threads are all part of the same process, so you have to fork rather 
than create a new thread. Whoops. I should have thought of that. But I guess 
that comes from my almost never using fork and even only using threads when 
I have to. Of course, it would be nice if there were a function in phobos to 
abstract that out, but there's enough work to do on phobos that that's 
probably not at the top of the list by any means.


 
 In order to get your standard output you would use the dup2()[3] call.
 
 Please note that I do not know how any of this will related to practices
 on Windows. And sadly this is not D specific and actually C stuff.
 
 1. http://www.opengroup.org/onlinepubs/007908799/xsh/exec.html
 2. http://www.yolinux.com/TUTORIALS/ForkExecProcesses.html
 3. http://www.mkssoftware.com/docs/man3/dup2.3.asp


Well, it's good info. So, thanks. But it would be nice if that too were 
nicely abstracted in phobos. The undocumented shell() command works, but it 
uses the shell, so you have to worry about escaping characters. Bleh. In any 
case, thanks for the info.

- Jonathan M Davis
Feb 21 2010
next sibling parent reply =?ISO-8859-1?Q?=22J=E9r=F4me_M=2E_Berger=22?= <jeberger free.fr> writes:
Jonathan M Davis wrote:

 Jesse Phillips wrote:

 As you may have noticed by the comments to on bug 3158. exec()[1] call=




s

 replace your process, this means it will not continue your program. To=





 get arround this you find that people will first fork()[2] and exec on=





 the child process.


=20
 Ah yes. Threads are all part of the same process, so you have to fork r=


ather=20

 than create a new thread. Whoops. I should have thought of that. But I =


guess=20

 that comes from my almost never using fork and even only using threads =


when=20

 I have to. Of course, it would be nice if there were a function in phob=


os to=20

 abstract that out, but there's enough work to do on phobos that that's =



 probably not at the top of the list by any means.
=20

 In order to get your standard output you would use the dup2()[3] call.=





 Please note that I do not know how any of this will related to practic=




es

 on Windows. And sadly this is not D specific and actually C stuff.

 1. http://www.opengroup.org/onlinepubs/007908799/xsh/exec.html
 2. http://www.yolinux.com/TUTORIALS/ForkExecProcesses.html
 3. http://www.mkssoftware.com/docs/man3/dup2.3.asp


=20
 Well, it's good info. So, thanks. But it would be nice if that too were=


=20

 nicely abstracted in phobos. The undocumented shell() command works, bu=


t it=20

 uses the shell, so you have to worry about escaping characters. Bleh. I=


n any=20

 case, thanks for the info.
=20


	In C, you have popen which allows to start a child process and
capture either the standard input or the standard output. It is
originally a POSIX function, but AFAIK it exists also on Windows
(although it might be called _popen there).

		Jerome
--=20
mailto:jeberger free.fr
http://jeberger.free.fr
Jabber: jeberger jabber.fr
Feb 22 2010
parent Lutger <lutger.blijdestijn gmail.com> writes:
"Jérôme M. Berger" wrote:


 Jonathan M Davis wrote:

 Jesse Phillips wrote:

 As you may have noticed by the comments to on bug 3158. exec()[1] calls
 replace your process, this means it will not continue your program. To
 get arround this you find that people will first fork()[2] and exec on
 the child process.


 
 Ah yes. Threads are all part of the same process, so you have to fork
 rather than create a new thread. Whoops. I should have thought of that.
 But I guess that comes from my almost never using fork and even only
 using threads when I have to. Of course, it would be nice if there were a
 function in phobos to abstract that out, but there's enough work to do on
 phobos that that's probably not at the top of the list by any means.
 

 In order to get your standard output you would use the dup2()[3] call.

 Please note that I do not know how any of this will related to practices
 on Windows. And sadly this is not D specific and actually C stuff.

 1. http://www.opengroup.org/onlinepubs/007908799/xsh/exec.html
 2. http://www.yolinux.com/TUTORIALS/ForkExecProcesses.html
 3. http://www.mkssoftware.com/docs/man3/dup2.3.asp


 
 Well, it's good info. So, thanks. But it would be nice if that too were
 nicely abstracted in phobos. The undocumented shell() command works, but
 it uses the shell, so you have to worry about escaping characters. Bleh.
 In any case, thanks for the info.
 


 In C, you have popen which allows to start a child process and
 capture either the standard input or the standard output. It is
 originally a POSIX function, but AFAIK it exists also on Windows
 (although it might be called _popen there).
 
 Jerome


While convenient, popen also invokes the shell to do it's bidding so you 
will get shell expansion and some performance penalty.
Feb 23 2010
prev sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Jonathan M Davis Wrote:


 Jesse Phillips wrote:

 Please note that I do not know how any of this will related to practices
 on Windows. And sadly this is not D specific and actually C stuff.
 
 1. http://www.opengroup.org/onlinepubs/007908799/xsh/exec.html
 2. http://www.yolinux.com/TUTORIALS/ForkExecProcesses.html
 3. http://www.mkssoftware.com/docs/man3/dup2.3.asp


 
 Well, it's good info. So, thanks. But it would be nice if that too were 
 nicely abstracted in phobos. The undocumented shell() command works, but it 
 uses the shell, so you have to worry about escaping characters. Bleh. In any 
 case, thanks for the info.
 
 - Jonathan M Davis


Agreed, but hopefully it is enough to write your own wrapper. If you do,
submitting a patch would be good; but as Lars pointed out, could be superseded
anyway. Good luck.
Feb 22 2010
prev sibling parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
Jonathan M Davis wrote:

 Okay, I'm looking to be able to run an external program from within my D 
 program. The library functions for this appear to be in std.process. You 
 have system() and various versions of execv(). system() makes a call in 
 shell. execv() executes the program without a shell.
 
 What I'd _like_ to be able to do is run the program without a shell so that 
 I don't have to worry about escaping special characters in file names, and I 
 want to have access to the output from the program. This poses two problems.
 
 1. Is there a way to run a program without the shell and without the call 
 terminating the program (per bugzilla 3158, execv takes over your program 
 and terminates it when it's done, in spite of what the documentation says - 
 it's certainly been killing my programs when I've tried)? The only way that 
 I see to do that at the moment is to spawn a separate thread and run execv 
 in it. I'd prefer to just make the call and wait for it to return. Is there 
 a way to do so?
 
 2. Is there a way to get at what the called program sends to stdout? I'm 
 afraid that I don't have a clue how to get at that.
 
 Any help would be appreciated. Thanks.
 
 - Jonathan M Davis



std.process has an undocumented function spawnvp(), which I believe does 
the same as the C function _spawnvp(), i.e. it forks and executes a 
child process, optionally waiting for it to finish.  The signature is:

   int spawnvp(int mode, string pathname, string[] argv);

where 'mode' is one of P_WAIT or P_NOWAIT.  You can use this, but be 
aware that there is most likely a reason for it being undocumented. :)


When the new concurrency framework is in place, I assume this will 
improve.  To begin with, the D developers are focusing on multithreading 
with message passing, such as:

   auto threadID = spawn(&threadFunction);
   threadID.send(message);

But they have said that this will be extended to multiprocess message 
passing, and then I almost expect something like this:

   auto procID = spawnProcess("firefox");
   procID.send(message);

It's some time into the future, though. ;)

-Lars
Feb 22 2010