digitalmars.D - [WEKA] Calling fork() and D runtime
- Jonathan Shamir (89/89) May 15 2017 Hey,
- Daniel Kozak via Digitalmars-d (4/86) May 15 2017 This is how I use fork on one of my project
- Jonathan Shamir (2/2) May 15 2017 OK well, seems like calling _exit() (instead of exit()) did the
- Kagamin (2/2) May 16 2017 D runtime is incompatible with fork, enforce and catch won't work
- Petar Kirov [ZombineDev] (8/98) May 21 2017 Some links to related discussions:
Hey,
This is my first time writing in the D forums!
I have an application written in D that runs as a linux daemon
(some python service script is in charge of running and
daemonizing it).
This "agent" works similar to docker - a service that accepts
commands and runs in the background, and starts our application
container. The container is itself a daemon (ppid = 1) with
unshared namespaces etc.
So, normally, implementing such an application would look
something like:
1. Main "agent" process runs fork() to create a child process
(child_1). All memory is copy-on-write.
2. Child_1 malloc()s a stack, and calls clone() to create yet
another child (child_2), which will eventually become the
container pid 1.
3. Child_2 initializes the container (mounts, unshare, chroot,
etc) then eventually exec()s into the container init process.
4. child_1 exit()s, which causes child_2 to become a daemon.
5. The agent main process should wait() on the forked pid since
it's impolite to leave zombies (I do this in a thread).
The problem I encounter is with the forked process (child_1).
Here is the code I wrote handling the fork() (Note: this
functionality should really be provided by core.threads or
something, for unix environments).
```private void deferToForkProcess(int delegate() entryPoint,
Timeout timeout = Timeout.infinite) {
import core.runtime : Runtime;
import core.sys.posix.unistd : fork, pid_t;
import core.sys.posix.sys.wait;
import core.stdc.stdlib : exit;
int rc = theReactor.deferToThread({
pid_t pid = fork();
errnoEnforce(pid >= 0, "Fork failed");
// child process
if (pid == 0) {
try {
int rc = entryPoint();
exit(rc);
} catch (Throwable ex) {
try {
LOG_ERROR(ex.toString);
} catch (Throwable) {}
exit(1);
}
//assert(false);
}
// parent - wait for child to exit
int status = 0;
do {
errnoEnforce(waitpid(pid, &status, 0) != -1, "Waitpid
failed");
} while (!WIFEXITED(status));
int rc = WEXITSTATUS(status);
return rc;
}, timeout);
enforce(rc == 0, "Child process failed (rc = %d)".format(rc));
}
```
entryPoint() returns 0, but the exit(0) raises an
OutOfMemoryError:
```0x4e6472
exit
??:0
0x4e6428
__run_exit_handlers
??:0
0x4df976
__libc_csu_fini
??:0
0x40327e
ldc.register_dso
crtstuff.c:0
0x4caee4
_d_dso_registry
??:0
0x4ccdba
_D2rt4util9container6common8xreallocFNbNiPvmZPv
??:0
0x4b873d
onOutOfMemoryError
??:0```
I tried to call Runtime.initialize() and Runtime.terminate()
surrounding the call to entryPoint(), but this didn't help. I
suspect calling initialize() was a no-op since the forked process
shares (copy-on-write) the VM space with it's parent, that
already initialized the runtime. (Note: confirmed, initialize()
returns true indicating it was already inited).
What is the correct way to handle fork() with the D runtime?
May 15 2017
This is how I use fork on one of my project
http://gitlab.eurosat.cz/ekj/esatd/raw/master/source/app.d
On Mon, May 15, 2017 at 1:33 PM, Jonathan Shamir via Digitalmars-d <
digitalmars-d puremagic.com> wrote:
Hey,
This is my first time writing in the D forums!
I have an application written in D that runs as a linux daemon (some
python service script is in charge of running and daemonizing it).
This "agent" works similar to docker - a service that accepts commands and
runs in the background, and starts our application container. The container
is itself a daemon (ppid = 1) with unshared namespaces etc.
So, normally, implementing such an application would look something like:
1. Main "agent" process runs fork() to create a child process (child_1).
All memory is copy-on-write.
2. Child_1 malloc()s a stack, and calls clone() to create yet another
child (child_2), which will eventually become the container pid 1.
3. Child_2 initializes the container (mounts, unshare, chroot, etc) then
eventually exec()s into the container init process.
4. child_1 exit()s, which causes child_2 to become a daemon.
5. The agent main process should wait() on the forked pid since it's
impolite to leave zombies (I do this in a thread).
The problem I encounter is with the forked process (child_1).
Here is the code I wrote handling the fork() (Note: this functionality
should really be provided by core.threads or something, for unix
environments).
```private void deferToForkProcess(int delegate() entryPoint, Timeout
timeout = Timeout.infinite) {
import core.runtime : Runtime;
import core.sys.posix.unistd : fork, pid_t;
import core.sys.posix.sys.wait;
import core.stdc.stdlib : exit;
int rc = theReactor.deferToThread({
pid_t pid = fork();
errnoEnforce(pid >= 0, "Fork failed");
// child process
if (pid == 0) {
try {
int rc = entryPoint();
exit(rc);
} catch (Throwable ex) {
try {
LOG_ERROR(ex.toString);
} catch (Throwable) {}
exit(1);
}
//assert(false);
}
// parent - wait for child to exit
int status = 0;
do {
errnoEnforce(waitpid(pid, &status, 0) != -1, "Waitpid failed");
} while (!WIFEXITED(status));
int rc = WEXITSTATUS(status);
return rc;
}, timeout);
enforce(rc == 0, "Child process failed (rc = %d)".format(rc));
}
```
entryPoint() returns 0, but the exit(0) raises an OutOfMemoryError:
```0x4e6472
exit
??:0
0x4e6428
__run_exit_handlers
??:0
0x4df976
__libc_csu_fini
??:0
0x40327e
ldc.register_dso
crtstuff.c:0
0x4caee4
_d_dso_registry
??:0
0x4ccdba
_D2rt4util9container6common8xreallocFNbNiPvmZPv
??:0
0x4b873d
onOutOfMemoryError
??:0```
I tried to call Runtime.initialize() and Runtime.terminate() surrounding
the call to entryPoint(), but this didn't help. I suspect calling
initialize() was a no-op since the forked process shares (copy-on-write)
the VM space with it's parent, that already initialized the runtime. (Note:
confirmed, initialize() returns true indicating it was already inited).
What is the correct way to handle fork() with the D runtime?
May 15 2017
Jonathan Shamir <jonathan weka.io> OK well, seems like calling _exit() (instead of exit()) did the trick. Thanks for the help!
May 15 2017
D runtime is incompatible with fork, enforce and catch won't work in the child process after fork, stick with betterC style.
May 16 2017
On Monday, 15 May 2017 at 11:33:29 UTC, Jonathan Shamir wrote:
Hey,
This is my first time writing in the D forums!
I have an application written in D that runs as a linux daemon
(some python service script is in charge of running and
daemonizing it).
This "agent" works similar to docker - a service that accepts
commands and runs in the background, and starts our application
container. The container is itself a daemon (ppid = 1) with
unshared namespaces etc.
So, normally, implementing such an application would look
something like:
1. Main "agent" process runs fork() to create a child process
(child_1). All memory is copy-on-write.
2. Child_1 malloc()s a stack, and calls clone() to create yet
another child (child_2), which will eventually become the
container pid 1.
3. Child_2 initializes the container (mounts, unshare, chroot,
etc) then eventually exec()s into the container init process.
4. child_1 exit()s, which causes child_2 to become a daemon.
5. The agent main process should wait() on the forked pid since
it's impolite to leave zombies (I do this in a thread).
The problem I encounter is with the forked process (child_1).
Here is the code I wrote handling the fork() (Note: this
functionality should really be provided by core.threads or
something, for unix environments).
```private void deferToForkProcess(int delegate() entryPoint,
Timeout timeout = Timeout.infinite) {
import core.runtime : Runtime;
import core.sys.posix.unistd : fork, pid_t;
import core.sys.posix.sys.wait;
import core.stdc.stdlib : exit;
int rc = theReactor.deferToThread({
pid_t pid = fork();
errnoEnforce(pid >= 0, "Fork failed");
// child process
if (pid == 0) {
try {
int rc = entryPoint();
exit(rc);
} catch (Throwable ex) {
try {
LOG_ERROR(ex.toString);
} catch (Throwable) {}
exit(1);
}
//assert(false);
}
// parent - wait for child to exit
int status = 0;
do {
errnoEnforce(waitpid(pid, &status, 0) != -1,
"Waitpid failed");
} while (!WIFEXITED(status));
int rc = WEXITSTATUS(status);
return rc;
}, timeout);
enforce(rc == 0, "Child process failed (rc =
%d)".format(rc));
}
```
entryPoint() returns 0, but the exit(0) raises an
OutOfMemoryError:
```0x4e6472
exit
??:0
0x4e6428
__run_exit_handlers
??:0
0x4df976
__libc_csu_fini
??:0
0x40327e
ldc.register_dso
crtstuff.c:0
0x4caee4
_d_dso_registry
??:0
0x4ccdba
_D2rt4util9container6common8xreallocFNbNiPvmZPv
??:0
0x4b873d
onOutOfMemoryError
??:0```
I tried to call Runtime.initialize() and Runtime.terminate()
surrounding the call to entryPoint(), but this didn't help. I
suspect calling initialize() was a no-op since the forked
process shares (copy-on-write) the VM space with it's parent,
that already initialized the runtime. (Note: confirmed,
initialize() returns true indicating it was already inited).
What is the correct way to handle fork() with the D runtime?
Some links to related discussions:
https://issues.dlang.org/show_bug.cgi?id=14205
http://forum.dlang.org/thread/ksqubftqniwznqbmurrk forum.dlang.org
https://issues.dlang.org/show_bug.cgi?id=14770
https://issues.dlang.org/show_bug.cgi?id=16006
https://github.com/dlang/phobos/pull/4294
https://github.com/dlang/druntime/pull/1569
May 21 2017