• John McFarlane (36/36) Oct 27 2014 Hi,
• Chris (11/47) Oct 28 2014 I had a similar problem with a DLL for Python. The reason my DLL
• John McFarlane (10/64) Nov 03 2014 Thanks for the suggestion. I looked into removing all of my

"John McFarlane" <dlang john.mcfarlane.name> writes:

Hi,

I've written a modest shared library in D that I'd like to call 
directly from a Python web server (Linux/OS X, Apache, WSGI, 
Pyramid). I can call it reliably from within Python unit tests 
but on a running server, the library causes a SIGSEGV as soon as 
I try anything as complicated as a writeln call. Under Linux, 
I've tried using Pyd, CFFI and ctypes to bind the native .so to 
Python with the same result. I've tried calling 
Runtime.initialize();from an init function on server startup and 
from the D function being called as part of the web request. I've 
even tried writing a shim in C to make these calls and call 
through to the D.

I've compiled with DMD and GDC with a variety of switches. 
Currently my build command looks like:

     dmd ./d/*.d -version=library -release -shared -O -inline 
-defaultlib=libphobos2.so -fPIC -oflibgusteau.so

The extern(C) function takes a string and returns a string. I've 
tried holding onto a pointer to the returned string and passing 
in an adequately sized byte array from Python. Smaller strings 
seem to cause a crash with lower probability but typically the 
input and output strings are a few KB in size and consistently 
crash. I taken measures to ensure that they are correctly encoded 
and null terminated. I've also tried disabling the GC, calling 
terminate on function exit (after copying result into received 
buffer) and various other measures in an attempt to get something 
working.

I'm guessing that the web server spawns/relies on a thread or 
process for each request and I've read that runtime 
initialization should be invoked from the main thread. Does this 
always mean the first thread that was created on process start up 
or does it just have to be consistent? If calling from a thread 
other than the one used to initialize, is that a problem? Are 
other threads created when GC is invoked which might last past 
the extern function being called and if so, can I prevent this by 
controlling collection explicitly?

Thanks, John

"Chris" <wendlec tcd.ie> writes:

On Tuesday, 28 October 2014 at 00:16:03 UTC, John McFarlane wrote:
 Hi,

 I've written a modest shared library in D that I'd like to call 
 directly from a Python web server (Linux/OS X, Apache, WSGI, 
 Pyramid). I can call it reliably from within Python unit tests 
 but on a running server, the library causes a SIGSEGV as soon 
 as I try anything as complicated as a writeln call. Under 
 Linux, I've tried using Pyd, CFFI and ctypes to bind the native 
 .so to Python with the same result. I've tried calling 
 Runtime.initialize();from an init function on server startup 
 and from the D function being called as part of the web 
 request. I've even tried writing a shim in C to make these 
 calls and call through to the D.

 I've compiled with DMD and GDC with a variety of switches. 
 Currently my build command looks like:

     dmd ./d/*.d -version=library -release -shared -O -inline 
 -defaultlib=libphobos2.so -fPIC -oflibgusteau.so

 The extern(C) function takes a string and returns a string. 
 I've tried holding onto a pointer to the returned string and 
 passing in an adequately sized byte array from Python. Smaller 
 strings seem to cause a crash with lower probability but 
 typically the input and output strings are a few KB in size and 
 consistently crash. I taken measures to ensure that they are 
 correctly encoded and null terminated. I've also tried 
 disabling the GC, calling terminate on function exit (after 
 copying result into received buffer) and various other measures 
 in an attempt to get something working.

 I'm guessing that the web server spawns/relies on a thread or 
 process for each request and I've read that runtime 
 initialization should be invoked from the main thread. Does 
 this always mean the first thread that was created on process 
 start up or does it just have to be consistent? If calling from 
 a thread other than the one used to initialize, is that a 
 problem? Are other threads created when GC is invoked which 
 might last past the extern function being called and if so, can 
 I prevent this by controlling collection explicitly?

 Thanks, John

I had a similar problem with a DLL for Python. The reason my DLL 
would crash were some writeln statements. After removing them it 
worked fine. I suppose the writeln messes things up, be it 
because of threads or because of conflicts in the file system 
(remember writeln is a file system operation). I suggest you have 
your lib not print things to console, but use a different 
mechanism instead, e.g. pass a string back to Python and have 
Python do the printing (if needs be).

You could also try to set your lib up as a small socket server 
that waits for input and answers. That usually works for me.

"John McFarlane" <dlang john.mcfarlane.name> writes:

On Tuesday, 28 October 2014 at 09:34:33 UTC, Chris wrote:
 On Tuesday, 28 October 2014 at 00:16:03 UTC, John McFarlane 
 wrote:
 Hi,

 I've written a modest shared library in D that I'd like to 
 call directly from a Python web server (Linux/OS X, Apache, 
 WSGI, Pyramid). I can call it reliably from within Python unit 
 tests but on a running server, the library causes a SIGSEGV as 
 soon as I try anything as complicated as a writeln call. Under 
 Linux, I've tried using Pyd, CFFI and ctypes to bind the 
 native .so to Python with the same result. I've tried calling 
 Runtime.initialize();from an init function on server startup 
 and from the D function being called as part of the web 
 request. I've even tried writing a shim in C to make these 
 calls and call through to the D.

 I've compiled with DMD and GDC with a variety of switches. 
 Currently my build command looks like:

    dmd ./d/*.d -version=library -release -shared -O -inline 
 -defaultlib=libphobos2.so -fPIC -oflibgusteau.so

 The extern(C) function takes a string and returns a string. 
 I've tried holding onto a pointer to the returned string and 
 passing in an adequately sized byte array from Python. Smaller 
 strings seem to cause a crash with lower probability but 
 typically the input and output strings are a few KB in size 
 and consistently crash. I taken measures to ensure that they 
 are correctly encoded and null terminated. I've also tried 
 disabling the GC, calling terminate on function exit (after 
 copying result into received buffer) and various other 
 measures in an attempt to get something working.

 I'm guessing that the web server spawns/relies on a thread or 
 process for each request and I've read that runtime 
 initialization should be invoked from the main thread. Does 
 this always mean the first thread that was created on process 
 start up or does it just have to be consistent? If calling 
 from a thread other than the one used to initialize, is that a 
 problem? Are other threads created when GC is invoked which 
 might last past the extern function being called and if so, 
 can I prevent this by controlling collection explicitly?

 Thanks, John

 I had a similar problem with a DLL for Python. The reason my 
 DLL would crash were some writeln statements. After removing 
 them it worked fine. I suppose the writeln messes things up, be 
 it because of threads or because of conflicts in the file 
 system (remember writeln is a file system operation). I suggest 
 you have your lib not print things to console, but use a 
 different mechanism instead, e.g. pass a string back to Python 
 and have Python do the printing (if needs be).

 You could also try to set your lib up as a small socket server 
 that waits for input and answers. That usually works for me.

Thanks for the suggestion. I looked into removing all of my 
writeln calls and
found the I still had the same problem. Simply a call to 
to!string with the
input char* is enough to cause a crash. I will go with the socket 
server as a
last resort but it would be much easier (especially from a 
testing POV) if I
could call the native D code directly from the web server.