www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - threading issues with D -> C -> Python

reply "Michael" <triorph gmail.com> writes:
Hi. I'm new here and this is my first post. I'm not sure this is 
the right subforum for it, but wasn't sure where else to put it 
either.

I've written a library to talk to some external hardware using a 
socket. It uses the std.concurrency threads to send messages 
between the main D-object for the hardware and the D-object for 
the sockets. I then wanted to be able to call these functions 
from Python. PyD appeared to be out of date, so I've been using a 
D -> C interface, and a C -> Python interface. The python code 
will often run from different python threads, so I then added yet 
another message-passing layer between the D->C interface and the 
D->hardware interface.

My problem is that this code routinely causes segmentation 
faults. I've spent a long time going through trying to figure out 
exactly what the causes are. I think there have been some related 
to D-exceptions not being handled gracefully by the C/Python 
code. Some more by stdout writing from multiple threads (which 
surprised me).

I'm fairly sure I have tackled both of these issues, but it still 
seems like Python threads and D threads don't mix well. When 
running the same functions from D, I am able to get no errors, 
but when run from Python/C it causes segfaults reliably.

Sorry for the large exposition. I am currently at the point of 
suspecting bugs in Phobos, but I am unskilled enough to tell for 
sure, and would appreciate any help.

The latest core dump gives a backtrace of almost entirely phobos 
commands:


/lib/libphobos2.so.0.66

/lib/libphobos2.so.0.66

/lib/libphobos2.so.0.66

/lib/libphobos2.so.0.66

/lib/libphobos2.so.0.66

/lib/libphobos2.so.0.66

std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from 
/usr/lib/libv5camera.so

std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from 
/usr/lib/libv5camera.so

std.array.__T8AppenderTAaZ.Appender.__T3putTAaZ.put() () from 
/usr/lib/libv5camera.so

std.array.__T8AppenderTAaZ.Appender.__T3putTxwZ.put() () from 
/usr/lib/libv5camera.so

std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() 
() from /usr/lib/libv5camera.so

std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.formatChar() 
() from /usr/lib/libv5camera.so

std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so

std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so

std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T6__ctorTS3std8typecons24__T5TupleTAyaTiTG655
6kZ5TupleZ.__ctor() 
()
    from /usr/lib/libv5camera.so

std.variant.__T8VariantNVmi32Z.VariantN.__T7handlerTS3std8typecons24__T5TupleTAyaTiTG6553
kZ5TupleZ.handler() 
()
    from /usr/lib/libv5camera.so

std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T8opEqualsTS3std8typecons24__T5TupleTAyaTiTG65536
Z5TupleZ.opEquals() 
()
    from /usr/lib/libv5camera.so

std.typecons.__T5TupleTAyaTiTG65536kZ.injectNamedFields() () from 
/usr/lib/libv5camera.so

std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so

std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so

std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ.get() 
() from /usr/lib/libv5camera.so

/lib/libphobos2.so.0.66

/lib/libphobos2.so.0.66

pthread_create.c:312

../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Cheers,
Michael.
Dec 02 2014
next sibling parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 01:07:42 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:


 I'm fairly sure I have tackled both of these issues, but it still=20
 seems like Python threads and D threads don't mix well. When=20
 running the same functions from D, I am able to get no errors,=20
 but when run from Python/C it causes segfaults reliably.


you are right, D threads and other language/library threads aren't mix
well. at least you have to use `thread_attachThis()` and
`thread_detachThis()` from core.threads module to make sure that GC is
aware of "alien" threads. and i assume that calling this functions
from python will not be very easy.

but it's better to not mix 'em at all if it is possible.
Dec 02 2014
parent reply "Michael" <triorph gmail.com> writes:
Thanks for this. Its definitely a step in the right direction. 
Would you mind explaining a bit more about the problem here, if 
you can? I don't fully understand why the garbage collector needs 
to know about the threads, and if so for how long does it need to 
know? If I put in 
"thread_attachThis();scope(exit)thread_detachThis();" it doesn't 
appear to fix my problems, so I'm definitely curious as to what 
is going on under the hood.

Cheers,
Michael.

On Wednesday, 3 December 2014 at 01:17:43 UTC, ketmar via 
Digitalmars-d-learn wrote:

 On Wed, 03 Dec 2014 01:07:42 +0000
 Michael via Digitalmars-d-learn 
 <digitalmars-d-learn puremagic.com>
 wrote:


 I'm fairly sure I have tackled both of these issues, but it 
 still seems like Python threads and D threads don't mix well. 
 When running the same functions from D, I am able to get no 
 errors, but when run from Python/C it causes segfaults 
 reliably.


 you are right, D threads and other language/library threads 
 aren't mix
 well. at least you have to use `thread_attachThis()` and
 `thread_detachThis()` from core.threads module to make sure 
 that GC is
 aware of "alien" threads. and i assume that calling this 
 functions
 from python will not be very easy.

 but it's better to not mix 'em at all if it is possible.
Dec 02 2014
parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 02:21:45 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:


 Thanks for this. Its definitely a step in the right direction.=20
 Would you mind explaining a bit more about the problem here, if=20
 you can? I don't fully understand why the garbage collector needs=20
 to know about the threads, and if so for how long does it need to=20
 know? If I put in=20
 "thread_attachThis();scope(exit)thread_detachThis();" it doesn't=20
 appear to fix my problems, so I'm definitely curious as to what=20
 is going on under the hood.


you have to call `thread_attachThis();` in "alien" thread, not in D
thread. i.e. if you created thread from python code, you have to call
`thread_attachThis();` in that python thread (i don't know how you'll
do that, but you must ;-). and you must call `thread_detachThis();`
from the same python thread before exiting from it.

garbage collector must know about all running threads so it can scan
their stacks, variables and so on. as there is no portable way to set
application-wide hooks on thread creation and termination, you must
inform GC about that events manually.

the other thing you can do is to not use any D allocated data in
"alien" threads. i.e. don't pass anything that was allocated by D code
to python thread and vice versa. if you want to pass some data to
"alien" thread, `malloc()` the necessary space, copy data to it and
pass malloc'ed pointer. don't forget to free that data in "alien"
thread. but i think that this is not what you really want, as it means
alot of allocations and copying, and complicates the whole thing alot.


"alien" is the thread that was created outside of D code.
Dec 02 2014
parent reply "Michael" <triorph gmail.com> writes:
On Wednesday, 3 December 2014 at 02:41:11 UTC, ketmar via 
Digitalmars-d-learn wrote:

 On Wed, 03 Dec 2014 02:21:45 +0000
 Michael via Digitalmars-d-learn 
 <digitalmars-d-learn puremagic.com>
 wrote:


 Thanks for this. Its definitely a step in the right direction. 
 Would you mind explaining a bit more about the problem here, 
 if you can? I don't fully understand why the garbage collector 
 needs to know about the threads, and if so for how long does 
 it need to know? If I put in 
 "thread_attachThis();scope(exit)thread_detachThis();" it 
 doesn't appear to fix my problems, so I'm definitely curious 
 as to what is going on under the hood.


 you have to call `thread_attachThis();` in "alien" thread, not 
 in D
 thread. i.e. if you created thread from python code, you have 
 to call
 `thread_attachThis();` in that python thread (i don't know how 
 you'll
 do that, but you must ;-). and you must call 
 `thread_detachThis();`
 from the same python thread before exiting from it.

 garbage collector must know about all running threads so it can 
 scan
 their stacks, variables and so on. as there is no portable way 
 to set
 application-wide hooks on thread creation and termination, you 
 must
 inform GC about that events manually.

 the other thing you can do is to not use any D allocated data in
 "alien" threads. i.e. don't pass anything that was allocated by 
 D code
 to python thread and vice versa. if you want to pass some data 
 to
 "alien" thread, `malloc()` the necessary space, copy data to it 
 and
 pass malloc'ed pointer. don't forget to free that data in 
 "alien"
 thread. but i think that this is not what you really want, as 
 it means
 alot of allocations and copying, and complicates the whole 
 thing alot.


 "alien" is the thread that was created outside of D code.


Okay. Well I am already not passing any D-allocated data. I'm 
specifically creating variables/arrays on the C-stack, and then 
passing the pointer of that to D and overwriting the data of the 
C-stack pointer for any return values. I was worried about that 
specific problem and I thought this would be a solution. I am 
then able to tell python to use the C-stack variable without 
having to worry about D trying to run any garbage collection on 
it.

Going the other way, I probably am passing some python strings 
etc.. into D, but I would assume they are valid for the lifetime 
of the function call, and that D would have no reason to try and 
perform any garbage collection on them.
Dec 02 2014
next sibling parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 02:52:27 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:


 Okay. Well I am already not passing any D-allocated data. I'm=20
 specifically creating variables/arrays on the C-stack, and then=20
 passing the pointer of that to D and overwriting the data of the=20
 C-stack pointer for any return values. I was worried about that=20
 specific problem and I thought this would be a solution. I am=20
 then able to tell python to use the C-stack variable without=20
 having to worry about D trying to run any garbage collection on=20
 it.


if D code has any pointer to that data stored anywhere, GC will walk it
and hit another thread's stack. and now it doesn't know where it is,
and it can't pause that uknown thread so it will not mutate the area GC
is scanning now. this *may* work, but it will segfault sooner or later.


 Going the other way, I probably am passing some python strings=20
 etc.. into D, but I would assume they are valid for the lifetime=20
 of the function call, and that D would have no reason to try and=20
 perform any garbage collection on them.


D has conservative GC, so it will try to walk with the unknown data
just in case that data contains some pointers. and GC can hit "false
positives" there (something that *looks* like a pointer to some area)
and other things.

so to make the long story short: you should either register and
deregister *all* your threads in GC (for D threads it's automatic
process; for other threads you must do it manually), or don't use GC at
all.

besides, if you are using your D library from C code, you must call
`rt_init()` once before calling any D code. this function will
initialize D runtime. and you have to call `rt_term()` before exiting
your program to deinitialize D runtime.
Dec 02 2014
prev sibling parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 02:52:27 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

by "using from C code" i mean that your main program is not written in
D, it has no D `main()` and so on. i.e. you wrote, for example, some
.a library in D and now you want to use that library in C code.
Dec 02 2014
prev sibling next sibling parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 01:07:42 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

all in all, you'd better not mixing D code with "alien" mulththreaded
code and not using .a/.so libraries written in D in another language
until you are familiar with D runtime and GC. those mixes are very
fragile.
Dec 02 2014
prev sibling next sibling parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 01:07:42 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

btw, Adam Ruppe's "D Cookbook" has a chapter which describes how to
call D library from C code. don't remember if it describes threading,
though.
Dec 02 2014
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/02/2014 05:07 PM, Michael wrote:

 Hi. I'm new here and this is my first post. I'm not sure this is the
 right subforum for it, but wasn't sure where else to put it either.

 I've written a library to talk to some external hardware using a socket.
 It uses the std.concurrency threads to send messages between the main
 D-object for the hardware and the D-object for the sockets. I then
 wanted to be able to call these functions from Python. PyD appeared to
 be out of date, so I've been using a D -> C interface, and a C -> Python
 interface. The python code will often run from different python threads,
 so I then added yet another message-passing layer between the D->C
 interface and the D->hardware interface.


are you looking at this pyd: https://bitbucket.org/ariovistus/pyd
Dec 02 2014
parent reply "Michael" <triorph gmail.com> writes:
On Wednesday, 3 December 2014 at 06:11:56 UTC, Ellery Newcomer 
wrote:

 are you looking at this pyd: 
 https://bitbucket.org/ariovistus/pyd


I'm looking at this one, which is what came up when googling 
"python to D"
http://pyd.dsource.org/
Dec 03 2014
parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 03 Dec 2014 20:41:46 +0000
Michael via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:


 On Wednesday, 3 December 2014 at 06:11:56 UTC, Ellery Newcomer=20
 wrote:

 are you looking at this pyd:=20
 https://bitbucket.org/ariovistus/pyd


=20
 I'm looking at this one, which is what came up when googling=20
 "python to D"
 http://pyd.dsource.org/


ah, dsource strikes back! that vile site keep biting us again and
again. let's hope that new admins will kill it for good.
Dec 03 2014
parent reply "Michael" <triorph gmail.com> writes:
On Wednesday, 3 December 2014 at 21:35:48 UTC, ketmar via 
Digitalmars-d-learn wrote:

 ah, dsource strikes back! that vile site keep biting us again 
 and
 again. let's hope that new admins will kill it for good.


Yeah. I've got the new PyD and it compiles and does everything I 
want much nicer, but it appears to have the exact same problems. 
When calling a python thread to my code, it can cause segfaults 
and hanging issues.

Cheers,
Michael.
Dec 03 2014
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/03/2014 04:43 PM, Michael wrote:

 On Wednesday, 3 December 2014 at 21:35:48 UTC, ketmar via
 Digitalmars-d-learn wrote:

 ah, dsource strikes back! that vile site keep biting us again and
 again. let's hope that new admins will kill it for good.


 Yeah. I've got the new PyD and it compiles and does everything I want
 much nicer, but it appears to have the exact same problems. When calling
 a python thread to my code, it can cause segfaults and hanging issues.

 Cheers,
 Michael.


okay. that's not too surprising.

If you can get me a minimal example, I'd be happy to have a look since 
pyd should probably support this case.
Dec 03 2014
parent reply "Michael" <triorph gmail.com> writes:
On Thursday, 4 December 2014 at 02:31:51 UTC, Ellery Newcomer 
wrote:

 okay. that's not too surprising.

 If you can get me a minimal example, I'd be happy to have a 
 look since pyd should probably support this case.


Cool. Unfortunately most of the times I've attempted to reduce 
this down it always seems to work, but I think that's because I 
often did the example code in D. I'll play around with it and try 
to send you an example.

Cheers,
Michael.
Dec 03 2014
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/03/2014 06:56 PM, Michael wrote:

 On Thursday, 4 December 2014 at 02:31:51 UTC, Ellery Newcomer wrote:

 okay. that's not too surprising.

 If you can get me a minimal example, I'd be happy to have a look since
 pyd should probably support this case.


 Cool. Unfortunately most of the times I've attempted to reduce this down
 it always seems to work, but I think that's because I often did the
 example code in D. I'll play around with it and try to send you an example.

 Cheers,
 Michael.


dustmite?
Dec 03 2014
parent reply "Michael" <triorph gmail.com> writes:
On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer 
wrote:

 dustmite?


Not sure what went wrong with dustmite, but every time I tried it 
it just started deleting all the files in the directory and 
setup.py would give errors. I manually deleted a reasonable chunk 
of the code and I'm left with these files which still seem to 
cause segfaults:

Main code: http://pastebin.com/zqgNTk9w
PyD definitions: http://pastebin.com/6mRH3KZZ
setup.py: http://pastebin.com/i9Ph78UC
test code that causes segfaults: http://pastebin.com/1ukzShVh

Cheers,
Michael.
Dec 04 2014
next sibling parent "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Thu, Dec 04, 2014 at 10:11:53PM +0000, Michael via Digitalmars-d-learn wrote:

 On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote:

dustmite?


 
 Not sure what went wrong with dustmite, but every time I tried it it
 just started deleting all the files in the directory and setup.py
 would give errors.


[...]

Nothing is wrong with dustmite; one of its standard techniques to reduce
code is to delete source files and see if the problem still happens
(i.e., it's independent of that file). What you need to do is to craft a
test script such that only the specific error you're looking for will
return a success status, and everything else won't. Then when dustmite
deletes an essential file, it will know that that was a wrong step and
backtrack.


T

-- 
Food and laptops don't mix.
Dec 04 2014
prev sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/04/2014 02:11 PM, Michael wrote:

 On Thursday, 4 December 2014 at 03:22:05 UTC, Ellery Newcomer wrote:

 dustmite?


 Not sure what went wrong with dustmite, but every time I tried it it
 just started deleting all the files in the directory and setup.py would
 give errors. I manually deleted a reasonable chunk of the code and I'm
 left with these files which still seem to cause segfaults:

 Main code: http://pastebin.com/zqgNTk9w
 PyD definitions: http://pastebin.com/6mRH3KZZ
 setup.py: http://pastebin.com/i9Ph78UC
 test code that causes segfaults: http://pastebin.com/1ukzShVh

 Cheers,
 Michael.


hmm.. looks like here it originates in python when it tries to acquire 
the GIL. specifically, pthread_cond_timedwait is segfaulting.

in your code, execution inside a python thread makes it to 
receiveTimeout in get_image. it made it past receiveTimeout in acquire.

then I guess there is a context switch. the main python thread throws an 
exception, but a number of things trigger the segfault. I think it's 
just the interpreter loop calling RestoreThread. backtrace looks like


     at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238

     mut=0x379a0631e0 <gil_mutex>, us=5000)
     at /usr/src/debug/Python-3.3.2/Python/condvar.h:103

     at /usr/src/debug/Python-3.3.2/Python/ceval_gil.h:224

(tstate=tstate entry=0x604410)
...

It looks like this is the python main thread.


to be your listener thread. std.concurrency.send seems to have gotten it 
into a gc_malloc, but it looks like it's just waiting. it's currently in 



its stack trace, receiveTimeout is just waiting.


I guess tomorrow I can try messing around with thread_attachThis, as the 

aren't really passing anything from python to d or vice versa, so I'm 
not sure why the gc would need to know about the python threads.

not exactly my area of expertise, this.
Dec 04 2014
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/04/2014 10:55 PM, Ellery Newcomer wrote:


 I guess tomorrow I can try messing around with thread_attachThis, as the

 aren't really passing anything from python to d or vice versa, so I'm
 not sure why the gc would need to know about the python threads.


by gum, thread_attachThis and thread_detachThis fix the issue!

now to figure out how to use them in the general case.
Dec 05 2014
parent reply "Michael" <triorph gmail.com> writes:
On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer 
wrote:

 On 12/04/2014 10:55 PM, Ellery Newcomer wrote:


 I guess tomorrow I can try messing around with 
 thread_attachThis, as the

 data. But you
 aren't really passing anything from python to d or vice versa, 
 so I'm
 not sure why the gc would need to know about the python 
 threads.


 by gum, thread_attachThis and thread_detachThis fix the issue!

 now to figure out how to use them in the general case.


This is great.. Thank you. I'm looking forward to being able to 
try the finished result.
Dec 07 2014
next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/07/2014 03:12 PM, Michael wrote:

 On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer wrote:

 On 12/04/2014 10:55 PM, Ellery Newcomer wrote:


 I guess tomorrow I can try messing around with thread_attachThis, as the

 aren't really passing anything from python to d or vice versa, so I'm
 not sure why the gc would need to know about the python threads.


 by gum, thread_attachThis and thread_detachThis fix the issue!

 now to figure out how to use them in the general case.


 This is great.. Thank you. I'm looking forward to being able to try the
 finished result.


It would be great if there were some convenient hook in python to stick 
these calls. Near as I can tell, there isn't. That leaves you with 
either explicitly calling attach and detach with an exposed api, or pyd 
obsessively checking whether the current thread is registered.

Actually, I suppose with a thread local flag the latter wouldn't be too bad.

Mind if I incorporate your example into pyd's test suite?
Dec 07 2014
parent "Michael" <triorph gmail.com> writes:
On Monday, 8 December 2014 at 01:17:16 UTC, Ellery Newcomer wrote:

 On 12/07/2014 03:12 PM, Michael wrote:

 On Saturday, 6 December 2014 at 00:40:49 UTC, Ellery Newcomer 
 wrote:

 On 12/04/2014 10:55 PM, Ellery Newcomer wrote:


 I guess tomorrow I can try messing around with 
 thread_attachThis, as the

 data. But you
 aren't really passing anything from python to d or vice 
 versa, so I'm
 not sure why the gc would need to know about the python 
 threads.


 by gum, thread_attachThis and thread_detachThis fix the issue!

 now to figure out how to use them in the general case.


 This is great.. Thank you. I'm looking forward to being able 
 to try the
 finished result.


 It would be great if there were some convenient hook in python 
 to stick these calls. Near as I can tell, there isn't. That 
 leaves you with either explicitly calling attach and detach 
 with an exposed api, or pyd obsessively checking whether the 
 current thread is registered.

 Actually, I suppose with a thread local flag the latter 
 wouldn't be too bad.

 Mind if I incorporate your example into pyd's test suite?


Not at all. Go for it.
Dec 08 2014
prev sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 12/07/2014 03:12 PM, Michael wrote:

 now to figure out how to use them in the general case.


 This is great.. Thank you. I'm looking forward to being able to try the
 finished result.


My build servers are broken at the moment, but I think I have this 
fixed, on linux at least.
Dec 17 2014
prev sibling parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 2014-12-03 at 01:07 +0000, Michael via Digitalmars-d-learn
wrote:

 Hi. I'm new here and this is my first post. I'm not sure this is=20
 the right subforum for it, but wasn't sure where else to put it=20
 either.
=20
 I've written a library to talk to some external hardware using a=20
 socket. It uses the std.concurrency threads to send messages=20
 between the main D-object for the hardware and the D-object for=20
 the sockets. I then wanted to be able to call these functions=20
 from Python. PyD appeared to be out of date, so I've been using a=20


As far as I can tell PyD is still active, but in a non-funded FOSS way,
i.e. work happens as and when volunteers put time and effort in. I
haven't tried PyD recently but it worked fine last time I did. If can
set out what you tried and what didn't work, maybe there is a PyD
solution, or a fix to PyD to give a solution?


 D -> C interface, and a C -> Python interface. The python code=20
 will often run from different python threads, so I then added yet=20
 another message-passing layer between the D->C interface and the=20
 D->hardware interface.


D's "big problem" is shared objects/dynamic link libraries. Without them
you cannot interwork with Python at all. I have tried experiments on
Linux creating shared libraries from D code with C linkage entry points
to create classic Python extensions, and it appears to work fine. Except
for having to start up the D heap and thread  management, should they be
needed. But that is what PyD is there for. If I took my experiments any
further I would end up recreating PyD or something like it.

It sounds like you are in a similar situation except that you appear to
have an extra layer of C code. I am not sure a layer of C is needed
between Python and D, it would be good to know more about why you seem
to need it.

Your use case is interesting as I have more or less given up on using D
in a Python context. CPython naturally requires C linkage shared objects
for non-Python code so C, C++ or D would be fine except that everyone
does it in C or C++, very few people in the arena have even heard of D.
PyPy brings it's own issues with non-Python code but the now have a C
capability. With Cython, Pythran, and more recently Numba there is
increasing less and less need for any user written non-Python code.
Hardware control would be done in C and PyPy now has a C linkage
mechanism.

Of course for Python networking there is Twisted, Asyncio and Tornado so
no Python folk are using non-Python code for handling networking.


 My problem is that this code routinely causes segmentation=20
 faults. I've spent a long time going through trying to figure out=20
 exactly what the causes are. I think there have been some related=20
 to D-exceptions not being handled gracefully by the C/Python=20
 code. Some more by stdout writing from multiple threads (which=20
 surprised me).
=20
 I'm fairly sure I have tackled both of these issues, but it still=20
 seems like Python threads and D threads don't mix well. When=20
 running the same functions from D, I am able to get no errors,=20
 but when run from Python/C it causes segfaults reliably.


Without seeing your code, it is difficult to say what may or may not be
the problem, but I would guess it is about D infrastructure start up. I
recollect being able to reliably get segfaults this way.

Are you using ctypes or CFFI on the Python side?


 Sorry for the large exposition. I am currently at the point of=20
 suspecting bugs in Phobos, but I am unskilled enough to tell for=20
 sure, and would appreciate any help.
=20
 The latest core dump gives a backtrace of almost entirely phobos=20
 commands:
=20

 /lib/libphobos2.so.0.66

 /lib/libphobos2.so.0.66

 /lib/libphobos2.so.0.66

 /lib/libphobos2.so.0.66

 /lib/libphobos2.so.0.66

 /lib/libphobos2.so.0.66

 std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from=20
 /usr/lib/libv5camera.so

 std.array.__T8AppenderTAaZ.Appender.__T3putTAxaZ.put() () from=20
 /usr/lib/libv5camera.so

 std.array.__T8AppenderTAaZ.Appender.__T3putTAaZ.put() () from=20
 /usr/lib/libv5camera.so

 std.array.__T8AppenderTAaZ.Appender.__T3putTxwZ.put() () from=20
 /usr/lib/libv5camera.so

 std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.format=


Char()=20

 () from /usr/lib/libv5camera.so

 std.format.__T10formatCharTS3std5array16__T8AppenderTAaZ8AppenderZ.format=


Char()=20

 () from /usr/lib/libv5camera.so

 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()=


=20

 () from /usr/lib/libv5camera.so

 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ=


.get()=20

 () from /usr/lib/libv5camera.so

 std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T6__ctorTS3std8typecons24__=


T5TupleTAyaTiTG65536kZ5TupleZ.__ctor()=20

 ()
     from /usr/lib/libv5camera.so

 std.variant.__T8VariantNVmi32Z.VariantN.__T7handlerTS3std8typecons24__T5T=


upleTAyaTiTG65536kZ5TupleZ.handler()=20

 ()
     from /usr/lib/libv5camera.so

 std.typecons.__T5TupleTAyaTiTG65536kZ.Tuple.__T8opEqualsTS3std8typecons24=


__T5TupleTAyaTiTG65536kZ5TupleZ.opEquals()=20

 ()
     from /usr/lib/libv5camera.so

 std.typecons.__T5TupleTAyaTiTG65536kZ.injectNamedFields() () from=20
 /usr/lib/libv5camera.so

 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()=


=20

 () from /usr/lib/libv5camera.so

 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNfAyaiZvZ.get()=


=20

 () from /usr/lib/libv5camera.so

 std.concurrency.MessageBox.__T3getTS4core4time8DurationTDFNaNbNiNfAyaiZvZ=


.get()=20

 () from /usr/lib/libv5camera.so

 /lib/libphobos2.so.0.66

 /lib/libphobos2.so.0.66

 pthread_create.c:312

 ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


My guess would be not properly initializing the D infrastructure from
the incoming Python thread.

I would suggest that you want to avoid threads crossing the boundaries
and just pass data via a shared channel. Unix pipes seem to work well in
this context since they provide a language independent data channel. =20

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Dec 02 2014
parent "Michael" <triorph gmail.com> writes:
On Wednesday, 3 December 2014 at 06:30:07 UTC, Russel Winder via 
Digitalmars-d-learn wrote:

 As far as I can tell PyD is still active, but in a non-funded 
 FOSS way,
 i.e. work happens as and when volunteers put time and effort 
 in. I
 haven't tried PyD recently but it worked fine last time I did. 
 If can
 set out what you tried and what didn't work, maybe there is a 
 PyD
 solution, or a fix to PyD to give a solution?


Yeah apparently I might have used the wrong PyD. This might end 
up being all I need to do to fix my problem.



 D's "big problem" is shared objects/dynamic link libraries. 
 Without them
 you cannot interwork with Python at all. I have tried 
 experiments on
 Linux creating shared libraries from D code with C linkage 
 entry points
 to create classic Python extensions, and it appears to work 
 fine. Except
 for having to start up the D heap and thread  management, 
 should they be
 needed. But that is what PyD is there for. If I took my 
 experiments any
 further I would end up recreating PyD or something like it.


This is what I'm doing. I'm using the rt_init() function to setup 
the heap/thread management. Am I missing anything else here? It 
seems the issue is definitely more complicated than that.

I'll also point out that: http://dlang.org/interfaceToC had no 
information on calling rt_init first, although it seems like it 
should.



 It sounds like you are in a similar situation except that you 
 appear to
 have an extra layer of C code. I am not sure a layer of C is 
 needed
 between Python and D, it would be good to know more about why 
 you seem
 to need it.


Well I wanted to compile D code and directly call it in Python, 
and since PyD didn't work for me, I instead tried the python-> C 
interface I already knew 
(https://docs.python.org/2/c-api/index.html) and worked my way 
through a C->D interface, which I understood to be relatively 
simple.


 My guess would be not properly initializing the D 
 infrastructure from
 the incoming Python thread.

 I would suggest that you want to avoid threads crossing the 
 boundaries
 and just pass data via a shared channel. Unix pipes seem to 
 work well in
 this context since they provide a language independent data 
 channel.


Yeah I'm leaning in that direction myself, although I might try 
the other PyD library first. I wanted to be able to use the D 
message-passing libraries to do the thread-safety stuff at first, 
because it was much easier than the alternative, but I'm not sure 
that's true anymore.
Dec 03 2014