digitalmars.D - Why are the exec* functions deprecated in std.process?
- Andrei Alexandrescu (17/17) Oct 28 2013 I am using the new std.process and am disappointed with it.
- Lars T. Kyllingstad (22/40) Oct 29 2013 Since I was the one who added the deprecation notices, I guess
- Andrei Alexandrescu (26/46) Oct 29 2013 That would be reason to not add, not to remove. They're there, and I
- Lars T. Kyllingstad (36/72) Oct 29 2013 Even if you don't buy my arguments, I think Vladimir's point
- Andrei Alexandrescu (26/66) Oct 29 2013 I think that's the weakest argument of the lot, see the rebuttal in my
- Lars T. Kyllingstad (9/47) Oct 29 2013 I've rebutted your rebuttal in another post.
- Vladimir Panteleev (7/10) Oct 30 2013 Windows does not have exec! This page is part of Microsoft's
- Andrei Alexandrescu (9/14) Oct 29 2013 Replacing
- Lars T. Kyllingstad (16/23) Oct 29 2013 That's really weird. I've tried and failed to reproduce it with
- Kagamin (10/14) Oct 30 2013 I think, a more appropriate name is std.native
- Jonathan M Davis (5/21) Oct 30 2013 Then you have to worry about which functions in a module work on which O...
- Kagamin (8/13) Oct 30 2013 posix and windows each end up as a soup of OS-specific feature
- Vladimir Panteleev (17/21) Oct 29 2013 In addition to what Lars said, I'd like to add that replacing the
- Andrei Alexandrescu (5/22) Oct 29 2013 That argument doesn't seem to hold water. So if Windows does it less
- Lars T. Kyllingstad (29/48) Oct 29 2013 It's not just that it does it less efficiently. What's worse is
- Andrei Alexandrescu (5/9) Oct 29 2013 Well argued. I'm okay with moving this to a posix-specific module and
- Marco Leise (38/50) Oct 31 2013 That was an interesting discussion. With every new post I was
- Lars T. Kyllingstad (7/8) Oct 29 2013 Going OT here, but being somewhat of a pedant when it comes to
- Andrei Alexandrescu (6/12) Oct 29 2013 The meaning of "homonym" is more narrow:
I am using the new std.process and am disappointed with it. There is no convenient function to replace the running process with a new one. There used to be the exec() family of functions, which conveniently use string[] for arguments etc., but now the doc says: ============ These functions are scheduled for deprecation. Please use spawnShell instead (or, alternatively, the homonymous C functions declared in std.c.process.) ============ The problems are: 1. spawnShell does NOT do the same thing as exec*. The former waits for the spawned process and then continues the calling process. The latter _replaces_ the running process with the new process. 2. The homonym (not homonymous btw) functions in std.c.process use in char*, not string, which makes them unpleasant to use. So... what to do here? Andrei
Oct 28 2013
On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:I am using the new std.process and am disappointed with it. There is no convenient function to replace the running process with a new one. There used to be the exec() family of functions, which conveniently use string[] for arguments etc., but now the doc says: ============ These functions are scheduled for deprecation. Please use spawnShell instead (or, alternatively, the homonymous C functions declared in std.c.process.) ============ The problems are: 1. spawnShell does NOT do the same thing as exec*. The former waits for the spawned process and then continues the calling process. The latter _replaces_ the running process with the new process. 2. The homonym (not homonymous btw) functions in std.c.process use in char*, not string, which makes them unpleasant to use. So... what to do here?Since I was the one who added the deprecation notices, I guess I'll have to answer for it. :) The main rationale for removing them was that I considered them to be low-level, rarely used functionality. Most applications *by far* will use the "spawn" style of process creation. While I have no statistics to back this statement with, the very fact that std.process went through several reviews on this forum and the Phobos mailing list, and even went into active use by some people (and one notable project, vibe.d) long before its inclusion in Phobos, without these functions being mentioned even *once*, is testament to its accuracy. To me, these functions simply fall on the wrong side of the usefulness/bloat line. And to quote http://dlang.org/phobos/, "No pointless wrappers around C runtime library functions or OS API functions. [...] Pointless D wrappers around those functions just adds blather, bloat, baggage and bugs." That said, they are not actually deprecated yet (as in marked with the "deprecated" attribute), so if the community wants them back, it is a simple matter of removing the deprecation notices. -Lars
Oct 29 2013
On 10/29/13 11:16 AM, Lars T. Kyllingstad wrote:Since I was the one who added the deprecation notices, I guess I'll have to answer for it. :)Great, thanks.The main rationale for removing them was that I considered them to be low-level, rarely used functionality. Most applications *by far* will use the "spawn" style of process creation.That would be reason to not add, not to remove. They're there, and I must use them, otherwise my D code is 10% slower than the equivalent C++ code, in a project at work in which speed is of the essence. Why do I need to resort to the C functions?While I have no statistics to back this statement with, the very fact that std.process went through several reviews on this forum and the Phobos mailing list, and even went into active use by some people (and one notable project, vibe.d) long before its inclusion in Phobos, without these functions being mentioned even *once*, is testament to its accuracy.It's been indeed reviewed, and I followed the reviews and was glad to see people liked it. My understanding was it was a backward compatible (within reason) revamp of std.process. It honestly didn't occur to me to go and see whether essential existing functionality has been deemed deprecated.To me, these functions simply fall on the wrong side of the usefulness/bloat line.On usefulness: "Objection, your honor" :o). Forwarding from one process to another is an essential part of process control. Loading another one and waiting until it's done is a very wasteful way to replace it. And I do recall the std.process revamping took pride in being more efficient than the old one. On bloating: the module std.process defines a couple dozen symbols. It's very far from bloating, and unlikely to grow because OSs are fairly stable when it comes to process control. Bloating doesn't strike me as a valid argument for removing existing useful and relevant functionality.And to quote http://dlang.org/phobos/, "No pointless wrappers around C runtime library functions or OS API functions. [...] Pointless D wrappers around those functions just adds blather, bloat, baggage and bugs."This quote is thoroughly misapplied. A pointless wrapper is a one-liner. Writing a passable wrapper for exec* is quite a bit more involved.That said, they are not actually deprecated yet (as in marked with the "deprecated" attribute), so if the community wants them back, it is a simple matter of removing the deprecation notices.I hope I provided compelling arguments. Andrei P.S. Also found this: "abstract final class environment". What happened to the naming conventions? Shouldn't that be capitalized?
Oct 29 2013
On Tuesday, 29 October 2013 at 19:13:28 UTC, Andrei Alexandrescu wrote:On 10/29/13 11:16 AM, Lars T. Kyllingstad wrote: [...]Even if you don't buy my arguments, I think Vladimir's point about exec*() basically being POSIX specific, and nothing more than a hack on Windows, is an even stronger argument. You are doing platform-specific stuff, so you should resort to platform-specific libraries. Were it up to me, I'd remove them from std.c too, forcing users to import core.sys.posix.unistd instead.The main rationale for removing them was that I considered them to be low-level, rarely used functionality. Most applications *by far* will use the "spawn" style of process creation.That would be reason to not add, not to remove. They're there, and I must use them, otherwise my D code is 10% slower than the equivalent C++ code, in a project at work in which speed is of the essence. Why do I need to resort to the C functions?[...] On usefulness: "Objection, your honor" :o). Forwarding from one process to another is an essential part of process control.It is apparently not essential enough for it to be natively supported on Windows.Loading another one and waiting until it's done is a very wasteful way to replace it. [...]I'm not suggesting this as a solution for you, but let me just point out that you wouldn't have to wait until it's done. You'd spawn it and exit. On Windows this is, as Vladimir points out, exactly what happens when you call exec*(). On POSIX it comes at the cost of a fork() (which I completely agree is unacceptable in some situations).[...]Ok, I'll give you that.And to quote http://dlang.org/phobos/, "No pointless wrappers around C runtime library functions or OS API functions. [...] Pointless D wrappers around those functions just adds blather, bloat, baggage and bugs."This quote is thoroughly misapplied. A pointless wrapper is a one-liner. Writing a passable wrapper for exec* is quite a bit more involved.Nope, I don't think so. :) That said, I know that there are completely legitimate uses for exec*(), and I have no doubt yours is one. I also agree that the fact that the functionality is already there is an argument against removing it. Therefore, I would like to suggest a compromise: I propose we move the functions into an std.posix.process module. (There is currently no std.posix package, but we do have std.windows, so I don't see why we can't add it.)That said, they are not actually deprecated yet (as in marked with the "deprecated" attribute), so if the community wants them back, it is a simple matter of removing the deprecation notices.I hope I provided compelling arguments.P.S. Also found this: "abstract final class environment". What happened to the naming conventions? Shouldn't that be capitalized?Seems I'm the culprit again. :) The rationale is: It walks like a variable, it talks like a variable, so let's name it like a variable. The semantics of "environment" are those of a global variable (of a noncopyable associative-array-like type), and the fact that it's actually a class is an implementation detail. Btw, environment has been in std.process for ages; it was not part of the revamp. (Or rather, it was a part of the revamp that got integrated years before the rest.) Lars
Oct 29 2013
On 10/29/13 1:25 PM, Lars T. Kyllingstad wrote:Even if you don't buy my arguments, I think Vladimir's point about exec*() basically being POSIX specific, and nothing more than a hack on Windows, is an even stronger argument.I think that's the weakest argument of the lot, see the rebuttal in my answer to it.You are doing platform-specific stuff, so you should resort to platform-specific libraries. Were it up to me, I'd remove them from std.c too, forcing users to import core.sys.posix.unistd instead.It's Posix, and Windows also implements the family. Not an argument for removal that Windows implements it in a suboptimal manner.That's a good point. Nevertheless there are deeper reasons for that. Far as I can tell functions like fork() and exec() are tenuous on Windows due to the way it's architected, so they favor other ways to go about things. Then they also took the time to implement exec(). No fault there. Just don't make it difficult to get to it.On usefulness: "Objection, your honor" :o). Forwarding from one process to another is an essential part of process control.It is apparently not essential enough for it to be natively supported on Windows.Interesting, I'll do that. But my point here remains: there is no lower bound in the speed I need.Loading another one and waiting until it's done is a very wasteful way to replace it. [...]I'm not suggesting this as a solution for you, but let me just point out that you wouldn't have to wait until it's done. You'd spawn it and exit. On Windows this is, as Vladimir points out, exactly what happens when you call exec*(). On POSIX it comes at the cost of a fork() (which I completely agree is unacceptable in some situations).Then we have a problem. Because I am convinced I am copiously right, and you failed to make any comparable argument to the contrary. To me the only matter to deal with is my being annoying when I know I'm right. (That may, in fact, be the bigger problem because I can be mightily annoying.)Nope, I don't think so. :)That said, they are not actually deprecated yet (as in marked with the "deprecated" attribute), so if the community wants them back, it is a simple matter of removing the deprecation notices.I hope I provided compelling arguments.That said, I know that there are completely legitimate uses for exec*(), and I have no doubt yours is one. I also agree that the fact that the functionality is already there is an argument against removing it. Therefore, I would like to suggest a compromise: I propose we move the functions into an std.posix.process module. (There is currently no std.posix package, but we do have std.windows, so I don't see why we can't add it.)I can only assume Windows made a bona fide effort to make their _exec() implementation (http://msdn.microsoft.com/en-us/library/vstudio/431x4c1w.aspx) as good as possible within the design of that environment. I find it very forced to insist on letting that go to waste and moving exec() to Posix just because... I honestly don't know why. There was no good argument in favor of going any other way but fixing the mistake of deprecation.Well that's a bummer but not the focus of my ire. AndreiP.S. Also found this: "abstract final class environment". What happened to the naming conventions? Shouldn't that be capitalized?Seems I'm the culprit again. :) The rationale is: It walks like a variable, it talks like a variable, so let's name it like a variable. The semantics of "environment" are those of a global variable (of a noncopyable associative-array-like type), and the fact that it's actually a class is an implementation detail. Btw, environment has been in std.process for ages; it was not part of the revamp. (Or rather, it was a part of the revamp that got integrated years before the rest.)
Oct 29 2013
On Tuesday, 29 October 2013 at 20:44:50 UTC, Andrei Alexandrescu wrote:On 10/29/13 1:25 PM, Lars T. Kyllingstad wrote:I've rebutted your rebuttal in another post.Even if you don't buy my arguments, I think Vladimir's point about exec*() basically being POSIX specific, and nothing more than a hack on Windows, is an even stronger argument.I think that's the weakest argument of the lot, see the rebuttal in my answer to it.Like I said in the other post, suboptimal is one thing, *wrong* is another.You are doing platform-specific stuff, so you should resort to platform-specific libraries. Were it up to me, I'd remove them from std.c too, forcing users to import core.sys.posix.unistd instead.It's Posix, and Windows also implements the family. Not an argument for removal that Windows implements it in a suboptimal manner.I've suggested a way to make it easy to get to, but which discourages its use on Windows: Put it in std.posix.That's a good point. Nevertheless there are deeper reasons for that. Far as I can tell functions like fork() and exec() are tenuous on Windows due to the way it's architected, so they favor other ways to go about things. Then they also took the time to implement exec(). No fault there. Just don't make it difficult to get to it.On usefulness: "Objection, your honor" :o). Forwarding from one process to another is an essential part of process control.It is apparently not essential enough for it to be natively supported on Windows.[...]Good thing I'm halfway around the globe, then. :) -LarsThen we have a problem. Because I am convinced I am copiously right, and you failed to make any comparable argument to the contrary. To me the only matter to deal with is my being annoying when I know I'm right. (That may, in fact, be the bigger problem because I can be mightily annoying.)I hope I provided compelling arguments.Nope, I don't think so. :)
Oct 29 2013
On Tuesday, 29 October 2013 at 20:44:50 UTC, Andrei Alexandrescu wrote:I can only assume Windows made a bona fide effort to make their _exec() implementation (http://msdn.microsoft.com/en-us/library/vstudio/431x4c1w.aspx)Windows does not have exec! This page is part of Microsoft's implementation of the standard C library used by their C compilers, which DMD32 doesn't even use! There is an important distinction between capabilities of an operating system, and the runtime of one implementation of C for that operating system.
Oct 30 2013
On 10/29/13 1:25 PM, Lars T. Kyllingstad wrote:I'm not suggesting this as a solution for you, but let me just point out that you wouldn't have to wait until it's done. You'd spawn it and exit. On Windows this is, as Vladimir points out, exactly what happens when you call exec*(). On POSIX it comes at the cost of a fork() (which I completely agree is unacceptable in some situations).Replacing return wait(spawnShell(cmd)) != 0; with spawnShell(cmd); return 0; as the last line of main() produces weird errors, including intermittent segfaults. Thoughts? Andrei
Oct 29 2013
On Tuesday, 29 October 2013 at 20:55:22 UTC, Andrei Alexandrescu wrote:Replacing return wait(spawnShell(cmd)) != 0; with spawnShell(cmd); return 0; as the last line of main() produces weird errors, including intermittent segfaults. Thoughts?That's really weird. I've tried and failed to reproduce it with this minimal test program on my Linux box: int main(string[] args) { import std.process; spawnShell(args[1]); return 0; } This worked fine with a variety of app names passed in args[1]. Are you able to provide a bit more context? Such as: - Which platform are you on? - What happens before spawnShell()? - Which program is spawned? Lars
Oct 29 2013
On Tuesday, 29 October 2013 at 20:25:16 UTC, Lars T. Kyllingstad wrote:Therefore, I would like to suggest a compromise: I propose we move the functions into an std.posix.process module. (There is currently no std.posix package, but we do have std.windows, so I don't see why we can't add it.)I think, a more appropriate name is std.native (std.native.process etc). This package would strive to be platform-specific for the sake of efficiency rather than cross-platform. If someone doesn't need to be cross-platform, he needs an efficient interface to the native efficient capabilities of the current platform. That said, the content of std.native modules will be platform-specific and will provide comprehensive D-ified API for the platform features.
Oct 30 2013
On Wednesday, October 30, 2013 17:42:36 Kagamin wrote:On Tuesday, 29 October 2013 at 20:25:16 UTC, Lars T. Kyllingstad wrote:Then you have to worry about which functions in a module work on which OS. It's generally a far cleaner separation to separate the OSes via the module system, which is what druntime does for posix and windows. - Jonathan M DavisTherefore, I would like to suggest a compromise: I propose we move the functions into an std.posix.process module. (There is currently no std.posix package, but we do have std.windows, so I don't see why we can't add it.)I think, a more appropriate name is std.native (std.native.process etc). This package would strive to be platform-specific for the sake of efficiency rather than cross-platform. If someone doesn't need to be cross-platform, he needs an efficient interface to the native efficient capabilities of the current platform. That said, the content of std.native modules will be platform-specific and will provide comprehensive D-ified API for the platform features.
Oct 30 2013
On Thursday, 31 October 2013 at 02:25:11 UTC, Jonathan M Davis wrote:Then you have to worry about which functions in a module work on which OS. It's generally a far cleaner separation to separate the OSes via the module system, which is what druntime does for posix and windows.posix and windows each end up as a soup of OS-specific feature sets too, and you still have to worry about which functions work on which OS. Is it a clear enough separation? And posix doesn't mean native, it means portable: windows implements it to some extent, so one might want to add windows support to posix modules or implement missing posix functions from scratch.
Oct 30 2013
On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:There is no convenient function to replace the running process with a new one. There used to be the exec() family of functions, which conveniently use string[] for arguments etc., but now the doc says:In addition to what Lars said, I'd like to add that replacing the current process is a rather platform-dependent trick. While process creation in POSIX is done by forking then replacing the forked process with a new one, on Windows it's the other way around - created processes are always isolated from the current one, and Windows implementations of this function simply emulate the behavior by creating a new process, then terminating the current one. I believe one of the goals of designing the new std.process is to create an interface that would be as platform-independent as possible. This meant that platform-specific functionality (which, incidentally, seems to be rarely used in practice) was abstracted away or scheduled for removal, delegating the task to such platform-specific (= low-level) tasks to C bindings in std.c.*, core.sys.* and core.stdc.*.
Oct 29 2013
On 10/29/13 11:34 AM, Vladimir Panteleev wrote:On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:That argument doesn't seem to hold water. So if Windows does it less efficiently than Linux, we should... remove the option altogether so we level the field? AndreiThere is no convenient function to replace the running process with a new one. There used to be the exec() family of functions, which conveniently use string[] for arguments etc., but now the doc says:In addition to what Lars said, I'd like to add that replacing the current process is a rather platform-dependent trick. While process creation in POSIX is done by forking then replacing the forked process with a new one, on Windows it's the other way around - created processes are always isolated from the current one, and Windows implementations of this function simply emulate the behavior by creating a new process, then terminating the current one. I believe one of the goals of designing the new std.process is to create an interface that would be as platform-independent as possible. This meant that platform-specific functionality (which, incidentally, seems to be rarely used in practice) was abstracted away or scheduled for removal, delegating the task to such platform-specific (= low-level) tasks to C bindings in std.c.*, core.sys.* and core.stdc.*.
Oct 29 2013
On Tuesday, 29 October 2013 at 19:29:12 UTC, Andrei Alexandrescu wrote:On 10/29/13 11:34 AM, Vladimir Panteleev wrote:It's not just that it does it less efficiently. What's worse is that it does it *differently*. While I don't have a Windows machine available to test this, I've done some research, and it seems the effect of calling _exec*() on Windows is *exactly* as if a new process was created and the parent process exited. In other words, the child process starts in the "background" (with a new PID), and the parent process returns control to the program which called it (such as cmd.exe). Furthermore, Windows' exec*() functions do not handle program names with spaces well. Apparently, Python has included support for the exec*() functionality on all platforms, which has led to several bug reports: http://bugs.python.org/issue9148 http://bugs.python.org/issue19066 http://bugs.python.org/issue19124 These were only the ones that came up in my first Google search. The answer from the Python team to all of them seems to be "don't use exec on Windows". StackOverflow users agree: http://stackoverflow.com/q/7004687 http://stackoverflow.com/q/7264571 I suspect that these functions were only included in Windows to satisfy POSIX requirements (which they fail to do). We should discourage their use on Windows, and one way to do this is to *not* have a nice cross-platform interface to them in Phobos. LarsOn Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei In addition to what Lars said, I'd like to add that replacing the current process is a rather platform-dependent trick. While process creation in POSIX is done by forking then replacing the forked process with a new one, on Windows it's the other way around - created processes are always isolated from the current one, and Windows implementations of this function simply emulate the behavior by creating a new process, then terminating the current one. [...]That argument doesn't seem to hold water. So if Windows does it less efficiently than Linux, we should... remove the option altogether so we level the field?
Oct 29 2013
On 10/29/13 3:07 PM, Lars T. Kyllingstad wrote: ...I suspect that these functions were only included in Windows to satisfy POSIX requirements (which they fail to do). We should discourage their use on Windows, and one way to do this is to *not* have a nice cross-platform interface to them in Phobos.Well argued. I'm okay with moving this to a posix-specific module and mentioning it in std.process. Andrei
Oct 29 2013
Am Tue, 29 Oct 2013 16:42:08 -0700 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:On 10/29/13 3:07 PM, Lars T. Kyllingstad wrote: ...That was an interesting discussion. With every new post I was switching sides. The fact that on Windows it is just emulated with CreateProcess reminded me how Phobos still uses "emulation" layers for I/O, namely C files in one or two places. If at some point Phobos used OS functionality exclusively instead of D wrappers around C wrappers, we could fix the O_CLOEXEC issue. By default Posix keeps file descriptors from the original process open after an exec(), whereas Windows asks you for 'inheritance' at file creation time and when you spawn a subprocess. We had a discussion about it when std.process was ready for review, so I'll cut it short. C doesn't offer a standard mode flag for file inheritance, so every C library does it differently. If Phobos is to offer a portable file handling, it needs to have platform specific code and default to not inheriting file descriptors. I think it was Steven, who argued that with everyone using std.process, no open file handles would ever slip through an exec() call (since std.process closes them all manually), but Andrei's almost use of bare bones C exec() shows once again that the issue is not mitigated. I had a patch for Phobos in the pipe, that ensures, O_CLOEXEC is set everywhere a file is opened (e.g. file streams), but I had to stop at std.stdio, where the C API is exposed in the form of mode flag strings - which already offer the non-portable way of setting O_CLOEXEC through different flags for each C library. So next you would see me argue for a replacement of zero-terminated mode strings with something more D-ish and at that point we are in a full blown debate about a std.stdio rewrite... So for now just remember to open all files with O_CLOEXEC (available as the "e" mode flag for glibc) on Linux unless you-know-what-you-are-doing=E2=84=A2. --=20 MarcoI suspect that these functions were only included in Windows to satisfy POSIX requirements (which they fail to do). We should discourage their use on Windows, and one way to do this is to *not* have a nice cross-platform interface to them in Phobos.=20 Well argued. I'm okay with moving this to a posix-specific module and=20 mentioning it in std.process. =20 Andrei
Oct 31 2013
On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:2. The homonym (not homonymous btw) functions [...]Going OT here, but being somewhat of a pedant when it comes to grammar myself, I can't just let this go. :) What is wrong with this use of "homonymous"? What would be an example of appropriate use? Lars
Oct 29 2013
On 10/29/13 12:06 PM, Lars T. Kyllingstad wrote:On Tuesday, 29 October 2013 at 03:44:37 UTC, Andrei Alexandrescu wrote:The meaning of "homonym" is more narrow: http://www.merriam-webster.com/dictionary/homonym http://www.merriam-webster.com/dictionary/homonymous But I think both work, so my mistake. Andrei2. The homonym (not homonymous btw) functions [...]Going OT here, but being somewhat of a pedant when it comes to grammar myself, I can't just let this go. :) What is wrong with this use of "homonymous"? What would be an example of appropriate use? Lars
Oct 29 2013