www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.process.execute performance without Config.inheritFDs

reply Jonathan Shamir <jonathan weka.io> writes:
Hey,

This code is from std.process:

         if (!(config & Config.inheritFDs))
         {
             import core.sys.posix.sys.resource;
             rlimit r;
             getrlimit(RLIMIT_NOFILE, &r);
             foreach (i; 3 .. cast(int) r.rlim_cur) close(i);
         }

This is a close-loop to make sure no fds are leaked before 
execve'ing, and can be disabled using Config.inheritFDs.

Here's a program I used to profile:

void main() {
     StopWatch sw;

     import core.sys.posix.sys.resource;
     rlimit r;
     errnoEnforce(getrlimit(RLIMIT_NOFILE, &r) == 0);
     writefln("The rlimit is %d (hard: %s)", r.rlim_cur, 
r.rlim_max);
     //r.rlim_cur = 1024 * 1024;
     //errnoEnforce(setrlimit(RLIMIT_NOFILE, &r) == 0);

     sw.start();
     foreach (_; 0 .. 100) {
         executeShell("echo hello", null, 
std.process.Config.inheritFDs);
     }
     sw.stop();

     writefln("Total time: %s", sw.peek().to!Duration.to!string);
}

These are the results I got:

// On a typical linux machine, without inheritFDs
The rlimit is 524288 (hard: 1048576)
Total time: 3 secs, 875 ms, 759 μs, and 7 hnsecs

// with inheritFDs
The rlimit is 524288 (hard: 1048576)
Total time: 80 ms, 549 μs, and 2 hnsecs

// osx, without inheritFDs
The rlimit is 7168 (hard: 9223372036854775807)
Total time: 476 ms, 483 μs, and 5 hnsecs

// with inheritFDs
The rlimit is 7168 (hard: 9223372036854775807)
Total time: 352 ms, 637 μs, and 7 hnsecs

So obviously this becomes a problem if the rlimit is high.

A few suggestions:
1. Make inheritFDs the default, since most people aren't aware of 
the performance cost. Also most programs will run just fine if a 
few fds are leaked.
1.1. Or, by default, close up to min(rlim_cur, 1024) - this 
should be enough for most processes. (If I'm not mistaken, that's 
what they do in python). Also this change won't break existing 
code.
2. All phobos fds should be opened with O_CLOEXEC, as intended. 
This eliminates the need to close all the fds when execve'ing.
3. Optimization for linux - use /proc to check which fds are 
actually opened, and close them, instead of making thousands of 
syscalls!
Jul 03 2017
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 3 July 2017 at 14:48:38 UTC, Jonathan Shamir wrote:

         if (!(config & Config.inheritFDs))
         {
             import core.sys.posix.sys.resource;
             rlimit r;
             getrlimit(RLIMIT_NOFILE, &r);
             foreach (i; 3 .. cast(int) r.rlim_cur) close(i);
         }


That code is pretty old. std.process was changed to use poll when 
available over a year ago:

https://github.com/dlang/phobos/pull/4114

Update your D installation?
Jul 03 2017
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 07/03/2017 07:53 AM, Vladimir Panteleev wrote:


 That code is pretty old. std.process was changed to use poll when
 available over a year ago:

 https://github.com/dlang/phobos/pull/4114

 Update your D installation?


Jonathan is hailing from Weka. :) As evidenced from Johan's recent 
messages, they're in the process of updating their ldc.

Ali
Jul 03 2017
parent bauss <jj_1337 live.dk> writes:
On Monday, 3 July 2017 at 20:34:00 UTC, Ali Çehreli wrote:

 On 07/03/2017 07:53 AM, Vladimir Panteleev wrote:


 That code is pretty old. std.process was changed to use poll


 when

 available over a year ago:

 https://github.com/dlang/phobos/pull/4114

 Update your D installation?


 Jonathan is hailing from Weka. :) As evidenced from Johan's 
 recent messages, they're in the process of updating their ldc.

 Ali


But then what's the point of this thread?
Jul 03 2017