• Carl Sturtivant (19/19) Jun 01 2018 A computationally intensive process run from the command line
• Basile B. (15/34) Jun 01 2018 1/ Maybe something with stdin
• Steven Schveighoffer (7/27) Jun 01 2018 Are the sub-processes outputting anything? Because that is what can

Carl Sturtivant <sturtivant gmail.com> writes:

A computationally intensive process run from the command line 
works fine, runs to completion after several minutes, writing a 
few hundred lines of text to standard output and creating, 
writing to and closing around 200 files of size around 20KB.

Now run from std.process.pipeProcess and periodically tested for 
completion with tryWait interleaved with sleep everything seems 
fine for a while. htop reveals that one core is running this at 
100% CPU, and the first 76 files appear one after another, but 
the 77th file is opened and nothing is written to it, and htop 
reveals that the CPU usage has dropped to zero, and yet the 
process is still running according to ps, and this continues 
indefinitely, no error message, no indication from tryWait that 
it is done. htop does not reveal at any point that memory use is 
even half of what is available.

Previously with similar processes that are a somewhat scaled back 
version of the one that fails as above, there's been no 
difference between what happened at the command line and what's 
happening here.

Any ideas or suggestions?

Basile B. <b2.temp gmx.com> writes:

On Friday, 1 June 2018 at 13:40:51 UTC, Carl Sturtivant wrote:
 A computationally intensive process run from the command line 
 works fine, runs to completion after several minutes, writing a 
 few hundred lines of text to standard output and creating, 
 writing to and closing around 200 files of size around 20KB.

 Now run from std.process.pipeProcess and periodically tested 
 for completion with tryWait interleaved with sleep everything 
 seems fine for a while. htop reveals that one core is running 
 this at 100% CPU, and the first 76 files appear one after 
 another, but the 77th file is opened and nothing is written to 
 it, and htop reveals that the CPU usage has dropped to zero, 
 and yet the process is still running according to ps, and this 
 continues indefinitely, no error message, no indication from 
 tryWait that it is done. htop does not reveal at any point that 
 memory use is even half of what is available.

 Previously with similar processes that are a somewhat scaled 
 back version of the one that fails as above, there's been no 
 difference between what happened at the command line and what's 
 happening here.

 Any ideas or suggestions?

1/ Maybe something with stdin

I don't believe this is the issue but if one proc expects input 
and if its stdin is not closed then it'll stuck in the code where 
input is read. E.g maybe something like

     ProcessPipes pp = pipeProcess(...);
     pp.stdin.close;

is needed somewhere ?

2/ Maybe something with stdout

If not all the output is read then the proc doesn't finish.
So something like this is needed maybe ?

     ProcessPipes pp = pipeProcess(...);
     foreach(c; pp.stdout.byChunk(4096)) {...}


These are blind ideas.
It would better if you can show some code.

Steven Schveighoffer <schveiguy yahoo.com> writes:

On 6/1/18 9:40 AM, Carl Sturtivant wrote:
 A computationally intensive process run from the command line works 
 fine, runs to completion after several minutes, writing a few hundred 
 lines of text to standard output and creating, writing to and closing 
 around 200 files of size around 20KB.
 
 Now run from std.process.pipeProcess and periodically tested for 
 completion with tryWait interleaved with sleep everything seems fine for 
 a while. htop reveals that one core is running this at 100% CPU, and the 
 first 76 files appear one after another, but the 77th file is opened and 
 nothing is written to it, and htop reveals that the CPU usage has 
 dropped to zero, and yet the process is still running according to ps, 
 and this continues indefinitely, no error message, no indication from 
 tryWait that it is done. htop does not reveal at any point that memory 
 use is even half of what is available.
 
 Previously with similar processes that are a somewhat scaled back 
 version of the one that fails as above, there's been no difference 
 between what happened at the command line and what's happening here.
 
 Any ideas or suggestions?

Are the sub-processes outputting anything? Because that is what can 
cause issues like this -- you have to process the pipe output or else it 
stalls when the pipe buffer gets full. It seems like you aren't 
processing the output from your description, but hard to tell without 
more context.

-Steve