digitalmars.D.learn - std.process: how to process stdout chunk by chunk without waiting for
- Timothee Cour (20/20) Jun 18 2013 I'd like to do the following:
- Steven Schveighoffer (16/38) Jun 18 2013 I think the issue is on the child process side. If you are using buffer...
- Timothee Cour (19/66) Jun 19 2013 yes
-
Kevin Lamonte
(7/92)
Jun 24 2013
http://dlang.org/phobos/std_**stdio.html#.File.setvbuf
I'd like to do the following: auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr); while(true){ version(A1) string line=pipes.stdout.readln; version(A2) auto line=pipes.stdout.readChunk(10); version(A3) auto line=pipes.stdout.readChar(); // do something with line if(tryWait(pipes.pid).terminated) break; } The problem is that 'string line=pipes.stdout.readln;' seems to block until the process is terminated, ie if the command is a long running command that prints a line every 1 second for 10 seconds, this program will wait 10 seconds before starting the processing. I also tried with rawRead, readf, fgetc but couldn't make it work. I'm on OSX, if that matters. Is there any way to achieve this?
Jun 18 2013
On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour <thelastmammoth gmail.com> wrote:I'd like to do the following: auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr); while(true){ version(A1) string line=pipes.stdout.readln; version(A2) auto line=pipes.stdout.readChunk(10); version(A3) auto line=pipes.stdout.readChar(); // do something with line if(tryWait(pipes.pid).terminated) break; } The problem is that 'string line=pipes.stdout.readln;' seems to block until the process is terminated, ie if the command is a long running command that prints a line every 1 second for 10 seconds, this program will wait 10 seconds before starting the processing. I also tried with rawRead, readf, fgetc but couldn't make it work. I'm on OSX, if that matters. Is there any way to achieve this?I think the issue is on the child process side. If you are using buffered I/O you have to flush the buffer. For instance, if the child is using D writeln or C printf, and you are using stdout, then it will only flush after writing 4096 bytes. You can flush early by calling flush on stdout, or fflush in C. Note that C will auto-detect if it is an interactive console, and flush via newlines instead. So running the same program from the console will flush every line! Alternatively, you can set the flush policy to flush after every line. See here: https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html And here: -Steve
Jun 18 2013
On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer <schveiguy yahoo.com>wrote:On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour < thelastmammoth gmail.com> wrote: I'd like to do the following:yesauto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr); while(true){ version(A1) string line=pipes.stdout.readln; version(A2) auto line=pipes.stdout.readChunk(**10); version(A3) auto line=pipes.stdout.readChar(); // do something with line if(tryWait(pipes.pid).**terminated) break; } The problem is that 'string line=pipes.stdout.readln;' seems to block until the process is terminated, ie if the command is a long running command that prints a line every 1 second for 10 seconds, this program will wait 10 seconds before starting the processing. I also tried with rawRead, readf, fgetc but couldn't make it work. I'm on OSX, if that matters. Is there any way to achieve this?I think the issue is on the child process side. If you are using buffered I/O you have to flush the buffer.For instance, if the child is using D writeln or C printf, and you are using stdout, then it will only flush after writing 4096 bytes. You can flush early by calling flush on stdout, or fflush in C. Note that C will auto-detect if it is an interactive console, and flush via newlines instead. So running the same program from the console will flush every line! Alternatively, you can set the flush policy to flush after every line. See here: https://developer.apple.com/**library/ios/#documentation/** System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html> And here: -SteveThanks, that does indeed work if I have source code for the child program and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after main. However I want to make it work without modifying source of child program. I tried http://stackoverflow.com/questions/1401002/trick-an-application-into-thinking-its-stdin-is-interactive-not-a-pipewith script: auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout | Redirect.stderr); that works but has issues : only buffers stdout, not stderr; and I may not want to redirect stderr to stdout; also it won't work in more complex cases, eg if program contains '|' etc, and it requires replacing \r\n by \n. I tried replacing fork with forkpty inside std.process. That doesn't seem to work: calling isatty(1),isatty(2) on child process still returns 0. Not sure why. I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process creation in std.process (after the fork with case 0); doesn't work either.
Jun 19 2013
Timothee Cour wrote:On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer <schveiguy yahoo.com>wrote:System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour < thelastmammoth gmail.com> wrote: I'd like to do the following:yesauto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr); while(true){ version(A1) string line=pipes.stdout.readln; version(A2) auto line=pipes.stdout.readChunk(**10); version(A3) auto line=pipes.stdout.readChar(); // do something with line if(tryWait(pipes.pid).**terminated) break; } The problem is that 'string line=pipes.stdout.readln;' seems to block until the process is terminated, ie if the command is a long running command that prints a line every 1 second for 10 seconds, this program will wait 10 seconds before starting the processing. I also tried with rawRead, readf, fgetc but couldn't make it work. I'm on OSX, if that matters. Is there any way to achieve this?I think the issue is on the child process side. If you are using buffered I/O you have to flush the buffer.For instance, if the child is using D writeln or C printf, and you are using stdout, then it will only flush after writing 4096 bytes. You can flush early by calling flush on stdout, or fflush in C. Note that C will auto-detect if it is an interactive console, and flush via newlines instead. So running the same program from the console will flush every line! Alternatively, you can set the flush policy to flush after every line. See here: https://developer.apple.com/**library/ios/#documentation/**And here:thinking-its-stdin-is-interactive-not-a-pipewith-SteveThanks, that does indeed work if I have source code for the child program and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after main. However I want to make it work without modifying source of child program. I tried http://stackoverflow.com/questions/1401002/trick-an-application-into-script: auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout | Redirect.stderr); that works but has issues : only buffers stdout, not stderr; and I may not want to redirect stderr to stdout; also it won't work in more complex cases, eg if program contains '|' etc, and it requires replacing \r\n by \n. I tried replacing fork with forkpty inside std.process. That doesn't seem to work: calling isatty(1),isatty(2) on child process still returns 0. Not sure why. I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process creation in std.process (after the fork with case 0); doesn't work either.I ran into this also. My solution is the makeShell() function at https://github.com/klamonte/d-tui/blob/master/tterminal.d . I'd like to see forkpty in phobos myself, so I just opened bug 10464 for that request.
Jun 24 2013