www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Terminating multiple processes

reply Russel Winder <russel winder.org.uk> writes:
So, I have an application which has a sort of nano-services
architecture, basically it is a set of communicating processes.
Terminating those processes blocked on an input channel is quite easy,
send a terminate message on the input channel. But what about a process
that has no input channel, one that is blocked on OS events?

Is there a way of forcibly, but nicely, terminating a spawned process
that never executes `receive()`?

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Jan 31 2018
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/31/18 12:44 PM, Russel Winder wrote:

 So, I have an application which has a sort of nano-services
 architecture, basically it is a set of communicating processes.
 Terminating those processes blocked on an input channel is quite easy,
 send a terminate message on the input channel. But what about a process
 that has no input channel, one that is blocked on OS events?
 
 Is there a way of forcibly, but nicely, terminating a spawned process
 that never executes `receive()`?
 


You talking about processes or threads? `receive` I believe is an 
inter-thread channel, no?

Terminating processes is generally done via signals or in the case of 
windows, calling the right system call.

Threads are another story. Typically, you need to have the thread check 
periodically for a termination event. There's no "nice" way to do it out 
of band.

In my experience, the best way to do it is to never block, but use some 
sort of "wait on input" for any would-be-blocking operation. You can use 
a large timeout, like 1 second, if immediate termination isn't important.

If you are using Fibers, and all your i/o is done using some event-based 
system (e.g. vibe.d), then things can be easier.

-Steve
Jan 31 2018
parent Russel Winder <russel winder.org.uk> writes:
On Wed, 2018-01-31 at 13:26 -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

=20


[=E2=80=A6]

 You talking about processes or threads? `receive` I believe is an=20
 inter-thread channel, no?


These are threads with no shared memory, just message passing via
channels. I think of them as processes, but not OS processes. My poor
choose of names, I should just have said threads.


 Terminating processes is generally done via signals or in the case
 of=20
 windows, calling the right system call.


Indeed. usually SIGHUP or SIGTERM, but never SIGKILL.


 Threads are another story. Typically, you need to have the thread
 check=20
 periodically for a termination event. There's no "nice" way to do it
 out=20
 of band.


This is what I was fearing. Of course from a theoretical standpoint it
is the right way to do things.


 In my experience, the best way to do it is to never block, but use
 some=20
 sort of "wait on input" for any would-be-blocking operation. You can
 use=20
 a large timeout, like 1 second, if immediate termination isn't
 important.


The thread in question is usually blocked in an inotify wait state
having no input channel. It seems I will have to manufacture a select.

I think this is what is missing in D, or at least my knowledge of it.
In Go, I can set up a select between channels and block, no need for a
timeout, an input on either channel causes a wakeup. The code then has
to work out which channel caused the wakeup. On the other hand DInotify
doesn't provide a channel style blocking wait, so select isn't an
option.


 If you are using Fibers, and all your i/o is done using some event-
 based=20
 system (e.g. vibe.d), then things can be easier.


I am not sure I want to get into single-threaded event queues, but
maybe I should. I like the dataflow approach, it is simpler for me to
reason about.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Feb 01 2018
prev sibling parent reply Arek <arychlinski gmail.com> writes:
On Wednesday, 31 January 2018 at 17:44:37 UTC, Russel Winder 
wrote:

 So, I have an application which has a sort of nano-services 
 architecture, basically it is a set of communicating processes. 
 Terminating those processes blocked on an input channel is 
 quite easy, send a terminate message on the input channel. But 
 what about a process that has no input channel, one that is 
 blocked on OS events?

 Is there a way of forcibly, but nicely, terminating a spawned 
 process that never executes `receive()`?


Assuming your're talking about threads: there's no secure method 
of forcing the thread to stop. Threads share the state (eg. can 
hold the locks) and killing them is always risky.

If your threads are blocked reading the socket, you probably can 
close these sockets and exit after the read error.

Another way is to use atomic flag indicating that thread needs to 
be interrupted.
After any blocking operation, the thread have to check this flag 
and finish the job.
It's good to use timeouts (eg socket timeout) in such scenario 
(if possible).

Arek
Jan 31 2018
parent reply Russel Winder <russel winder.org.uk> writes:
On Wed, 2018-01-31 at 22:15 +0000, Arek via Digitalmars-d-learn wrote:

=20


[=E2=80=A6]

 Assuming your're talking about threads: there's no secure method=20
 of forcing the thread to stop. Threads share the state (eg. can=20
 hold the locks) and killing them is always risky.


I am indeed talking threads not OS processes. I just like working with
(processes|tasks) submitted to a thread pool, but this can lead to
jargon use problems.


 If your threads are blocked reading the socket, you probably can=20
 close these sockets and exit after the read error.


The problem is actually a thread blocked in an inotify blocking read.
As both Steven and yourself have pointed out I am going to have to use
a timeout to check the state of the application.=20


 Another way is to use atomic flag indicating that thread needs to=20
 be interrupted.
 After any blocking operation, the thread have to check this flag=20
 and finish the job.
 It's good to use timeouts (eg socket timeout) in such scenario=20
 (if possible).


I guess there is a choice here between shared memory to set the
termination flag, or using an input channel and sending the termination
message. I think the latter may be preferable, and certainly more
consistent with how the other threads terminate.

Thanks to you and Steven for confirming what I had been fearing, but
wondered if there was an easier (albeit less theoretically sound) one.
=20
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Feb 01 2018
next sibling parent reply Arek <arychlinski gmail.com> writes:
On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote:

 On Wed, 2018-01-31 at 22:15 +0000, Arek via Digitalmars-d-learn 
 wrote:

 


 […]

 The problem is actually a thread blocked in an inotify blocking 
 read. As both Steven and yourself have pointed out I am going 
 to have to use a timeout to check the state of the application.


Try to use inotify in non-blocking mode (an example here: 
https://gist.github.com/pkrnjevic/6016356) with select or epoll 
and timeouts.


 I guess there is a choice here between shared memory to set the 
 termination flag, or using an input channel and sending the 
 termination message. I think the latter may be preferable, and 
 certainly more consistent with how the other threads terminate.


I would use shared memory here (eg. atomic bool) because any 
communication channel introduces possibility of further blocking 
problems.

Arek
Feb 01 2018
parent reply Russel Winder <russel winder.org.uk> writes:
On Thu, 2018-02-01 at 12:15 +0000, Arek via Digitalmars-d-learn wrote:

=20


[=E2=80=A6]

 Try to use inotify in non-blocking mode (an example here:=20
 https://gist.github.com/pkrnjevic/6016356) with select or epoll=20
 and timeouts.


Isn't there a C++ binding for the C API?

I am using DInotify which is a D binding. I will be checking soon but I
am assuming there is a timeout version so I can loop to check the
application state.


=20
 I guess there is a choice here between shared memory to set the=20
 termination flag, or using an input channel and sending the=20
 termination message. I think the latter may be preferable, and=20
 certainly more consistent with how the other threads terminate.
=20


=20
 I would use shared memory here (eg. atomic bool) because any=20
 communication channel introduces possibility of further blocking=20
 problems.


A priori I am not convinced. I have used a state variable in C++ and
Python code where there is no channel system, but in Go, Groovy/GPars,
using channels is always preferable. Given the channel has a "read if
there is something to read" there can't be a blocking problem =E2=80=93 if =
the
channel system is a good one.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Feb 01 2018
parent reply Arek <arychlinski gmail.com> writes:
On Thursday, 1 February 2018 at 12:30:24 UTC, Russel Winder wrote:

 On Thu, 2018-02-01 at 12:15 +0000, Arek via Digitalmars-d-learn 
 wrote:

 


 […]

 Try to use inotify in non-blocking mode (an example here: 
 https://gist.github.com/pkrnjevic/6016356) with select or 
 epoll and timeouts.


 Isn't there a C++ binding for the C API?

 I am using DInotify which is a D binding. I will be checking 
 soon but I am assuming there is a timeout version so I can loop 
 to check the application state.


DInitify doesn't cover full capabilities of the inotify API.
Especially it doesn't utilize newer inotify_init1 syscall and 
doesn't expose 'select' interface.
This C++ wrapper may be interesting for you: 
https://github.com/erikzenker/inotify-cpp
But I haven't used it.


 
 I would use shared memory here (eg. atomic bool) because any 
 communication channel introduces possibility of further 
 blocking problems.


 A priori I am not convinced. I have used a state variable in 
 C++ and Python code where there is no channel system, but in 
 Go, Groovy/GPars, using channels is always preferable. Given 
 the channel has a "read if there is something to read" there 
 can't be a blocking problem – if the channel system is a good 
 one.


You may be right. But interrupting the thread is very low level 
mechanism.
Eg in Java, it is incorporated into Thread class: thread has 
method 'interrupt()' which
sets the flag, and there is property 'isInterrupted()'. Usually 
your thread code have something like  
while(!Thread.currentThread().isInterrupted()) { /* do 
something*/ }
I would use similar pattern.


Arek
Feb 01 2018
parent Russel Winder <russel winder.org.uk> writes:
On Thu, 2018-02-01 at 20:13 +0000, Arek via Digitalmars-d-learn wrote:

=20


[=E2=80=A6]

 DInitify doesn't cover full capabilities of the inotify API.
 Especially it doesn't utilize newer inotify_init1 syscall and=20
 doesn't expose 'select' interface.
 This C++ wrapper may be interesting for you:=20
 https://github.com/erikzenker/inotify-cpp
 But I haven't used it.


Hummm=E2=80=A6 sounds like I need to look to create a pull request to get t=
o
dinotify 0.2.3 or even 0.3.0.


=20






[=E2=80=A6]

 You may be right. But interrupting the thread is very low level=20
 mechanism.
 Eg in Java, it is incorporated into Thread class: thread has=20
 method 'interrupt()' which
 sets the flag, and there is property 'isInterrupted()'. Usually=20
 your thread code have something like =20
 while(!Thread.currentThread().isInterrupted()) { /* do=20
 something*/ }
 I would use similar pattern.


I do want to avoid such low-level things, they may be required for
building libraries, but they seem totally the wrong level of concept
for an application.

I am definitely headed to the (select|poll|epoll) on the inotify file
descriptor to get the timeout, and then either read or check the input
channel for termination or an atomic Boolean state variable. Channels
seem most likely since parent threads send a terminating token.


--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Feb 02 2018
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote:

 The problem is actually a thread blocked in an inotify blocking 
 read. As both Steven and yourself have pointed out I am going 
 to have to use a timeout to check the state of the application.


There are better solutions (select/...), But couldn't you in 
theory just send a custom signal to the thread (which you 
ignore), and then check for the exit flag after the syscall 
returned EINTR?

The DInotify wrapper might of course have the retry loop 
hardcoded; I didn't check.

  —David
Feb 01 2018
next sibling parent Russel Winder <russel winder.org.uk> writes:
On Fri, 2018-02-02 at 01:09 +0000, David Nadlinger via Digitalmars-d-
learn wrote:

 On Thursday, 1 February 2018 at 11:42:32 UTC, Russel Winder wrote:

 The problem is actually a thread blocked in an inotify blocking=20
 read. As both Steven and yourself have pointed out I am going=20
 to have to use a timeout to check the state of the application.


=20
 There are better solutions (select/...), But couldn't you in=20
 theory just send a custom signal to the thread (which you=20
 ignore), and then check for the exit flag after the syscall=20
 returned EINTR?


I think I am going to go the (select|poll|epoll) on the file descriptor
with a timeout and then the read if that is appropriate or check the
input channel for the terminate token if that is appropriate.

The question whether to do this in my code or create a "read with
timeout" in dinotify


 The DInotify wrapper might of course have the retry loop=20
 hardcoded; I didn't check.


Dinotify adds the tree monitoring as an extra over the inotify API but
U do not see any timeout related things.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Feb 02 2018
prev sibling parent Russel Winder <russel winder.org.uk> writes:
On Fri, 2018-02-02 at 20:14 +0000, David Nadlinger wrote:

=20


[=E2=80=A6]

 That sounds entirely sensible. Your original question was whether it
 was=20
 possible to terminate threads blocked in a syscall, though. Signals=20
 allow you to do that on POSIX for many "slow" syscalls, by making it=20
 return EINTR. You would check whatever channel/atomic flag/=E2=80=A6
 mechanism=20
 you use to signal termination before continuing to block by
 reissuing=20
 the syscall.
=20
 I haven't tested how `read` behaves for inotify myself, but
 inotify(7)=20
 suggests that it indeed handles signals this way.


[=E2=80=A6]

After some discussions, Dmitry acted very quickly and added a timeout
read to the blocking read in dinotify.

This solves my problem "at a stroke" as I can now block waiting for a
specific time on inotify and then check the input channel for a time
and repeat. Not really busy wait just a sequence of blocking reads with
timeouts. The important thing is that With very little latency, the
thread is checking two things.

DInotify 0.3.0 will likely appear soon in Dub. It would be good to get
it packaged for Debian. I will take up an invitation to submit a ready
made package for the Debian D packaging team.


--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Feb 05 2018