digitalmars.D.learn - building shared library from D code to import into cython
- Laeeth Isharc (83/84) Oct 07 2014 Hi.
- Freddy (20/104) Oct 07 2014 Since when does gcc have a -defaultlib option?
- Laeeth Isharc (55/187) Oct 07 2014 Hi.
- Ellery Newcomer (8/14) Oct 09 2014 I managed to get it to compile with default dmd rpm:
- Laeeth Isharc (13/31) Oct 12 2014 Thanks for this.
- Ellery Newcomer (13/15) Oct 12 2014 pyd is basically just a convenience layer on top of the C
Hi. I am trying to create a shared library in D linked against phobos so that I may use this in a cython extension module for Python. Ultimately I would like to be able to use a D class or struct (via the C++ interface) and call it from within cython, since cython classes cannot be instantiated without the gil (and this prevents easy parallelisation). I feel a bit foolish asking the question as there is a nice example here for working with plain C using dmd as the linker, and using dmd and gcc to create a DMD shared library statically linked to phobos. However, I have not succeeded in creating a D library statically linked to phobos that works with cython and python, http://dlang.org/dll-linux.html#dso7 I tried it first with test C code to make sure I am able to get the C library/cython/Python interaction working. pytest.c: #include <stdio.h> long pytest(long a) { return a+1; } int main() { long a =pytest(100); printf("%ld",a); return 0; } pytestpy.pyx: cdef extern long pytest(long a) cpdef pytestpy(): return pytest(109) setup.py: from distutils.core import setup from distutils.extension import Extension from Cython.Build import cythonize setup( ext_modules = cythonize([ Extension("pytestpy", ["pytestpy.pyx"], libraries=["pytest"], ) ])) command line: gcc -shared -o libpytest.so pytest.o python setup.py build_ext -i <copied libpytest.so to /usr/local/lib> python import pytestpy pytestpy.pytestpy() <it works> ---- now try pytest.d import std.stdio; extern (C) long pytest(long a) { return a*2; } void main() { auto a =pytest(100); writefln("%d",a); } command line: rm pytestd.o rm libpytest.so rm /usr/local/lib/libpytest.so dmd -c pytest.d -fPIC gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so -L-rpath=/usr/local/lib pythonTraceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /usr/local/lib/libpytest.so: undefined symbol: _D3std5stdio12__ModuleInfoZ I guess it is not linking to the D runtime, but I am not sure what I should be doing to fix. Any thoughts appreciated. (The next step I was going to try when this works was C++ interface vs importing as a Cython class, but I thought best to start simple). I am running this on 64 bit Fedora 20. Thanks. Laeeth.import pytestpy
Oct 07 2014
On Tuesday, 7 October 2014 at 20:55:59 UTC, Laeeth Isharc wrote:Hi. I am trying to create a shared library in D linked against phobos so that I may use this in a cython extension module for Python. Ultimately I would like to be able to use a D class or struct (via the C++ interface) and call it from within cython, since cython classes cannot be instantiated without the gil (and this prevents easy parallelisation). I feel a bit foolish asking the question as there is a nice example here for working with plain C using dmd as the linker, and using dmd and gcc to create a DMD shared library statically linked to phobos. However, I have not succeeded in creating a D library statically linked to phobos that works with cython and python, http://dlang.org/dll-linux.html#dso7 I tried it first with test C code to make sure I am able to get the C library/cython/Python interaction working. pytest.c: #include <stdio.h> long pytest(long a) { return a+1; } int main() { long a =pytest(100); printf("%ld",a); return 0; } pytestpy.pyx: cdef extern long pytest(long a) cpdef pytestpy(): return pytest(109) setup.py: from distutils.core import setup from distutils.extension import Extension from Cython.Build import cythonize setup( ext_modules = cythonize([ Extension("pytestpy", ["pytestpy.pyx"], libraries=["pytest"], ) ])) command line: gcc -shared -o libpytest.so pytest.o python setup.py build_ext -i <copied libpytest.so to /usr/local/lib> python import pytestpy pytestpy.pytestpy() <it works> ---- now try pytest.d import std.stdio; extern (C) long pytest(long a) { return a*2; } void main() { auto a =pytest(100); writefln("%d",a); } command line: rm pytestd.o rm libpytest.so rm /usr/local/lib/libpytest.so dmd -c pytest.d -fPIC gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so -L-rpath=/usr/local/lib pythonSince when does gcc have a -defaultlib option? --- $ man gcc | grep defaultlib object-file-name -llibrary -nostartfiles -nodefaultlibs This is useful when you use -nostdlib or -nodefaultlibs but you do -nodefaultlibs is used. -nodefaultlibs -nodefaultlibs is libgcc.a, a library of internal subroutines which -nodefaultlibs you should usually specify -lgcc as well. This --- and why do you have a main function when compiling a shared library? Anyway, I think the problem is that "libpytest.so" can't find "libphobos2.so". Try adding "libphobos2.so"'s location (/usr/lib/x86_64-linux-gnu/libphobos2.so on my linux mint) to your -rpath .Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /usr/local/lib/libpytest.so: undefined symbol: _D3std5stdio12__ModuleInfoZ I guess it is not linking to the D runtime, but I am not sure what I should be doing to fix. Any thoughts appreciated. (The next step I was going to try when this works was C++ interface vs importing as a Cython class, but I thought best to start simple). I am running this on 64 bit Fedora 20. Thanks. Laeeth.import pytestpy
Oct 07 2014
Hi. Thanks for the quick response. The -defaultlib was left around from trying all kinds of combinations of dmd and gcc. I am not used to gcc, and it will take me some time to become properly acquainted with all the options. I simply could not get it to recognize libphobos no matter what path settings I tried passing. http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=4 I don't know whether this was necessary, but I did manage to get it to work after looking at a thread from a couple of years back. I recompiled the D runtime and Phobos with position independent code (PIC) and pointed everything to that version instead, and it worked fine. (Just export PIC=True, rename the posix.mak to Makefile, change the DMD path to /usr/sbin/dmd - in my case, and it all worked). Next step is to try calling D interface from Cython. Laeeth In case anyone else should struggle with this in future. Makefile here: PHOBOS_PATH=/opt/dmd2/src/phobos/generated/linux/release/64 pytestpy.so: pytest.d pytestpy.o -L-rpath=.:${PHOBOS_PATH} gcc -shared pytest.o pytestpy.o -o pytest.so -lphobos2 -lpthread -lrt -L. -L${PHOBOS_PATH} -Wl,-rpath=.:${PHOBOS_PATH} -o pytestpy.so -defaultlib=libphobos2.so -L-rpath=.:${PHOBOS_PATH} pytest.o: %.d dmd -c $< pytestpy.o: pytestpy.pyx cython pytestpy.pyx gcc -fPIC -c pytestpy.c -o pytestpy.o -I/usr/include/python2.7 clean: rm -f pytestpy.c pytest.o pytestpy.o pytest.so pytest.d extern (C) long pytest(long a) { return a*2; } pytestpy.pyx cdef extern long pytest(long a) cpdef pytestpy(): return pytest(109) print pytest(102) setup.py from distutils.core import setup from distutils.extension import Extension from Cython.Build import cythonize setup( ext_modules = cythonize([ Extension("pytestpy", ["pytestpy.pyx"], libraries=["pytest"], ) ])) On Tuesday, 7 October 2014 at 22:46:09 UTC, Freddy wrote:On Tuesday, 7 October 2014 at 20:55:59 UTC, Laeeth Isharc wrote:Hi. I am trying to create a shared library in D linked against phobos so that I may use this in a cython extension module for Python. Ultimately I would like to be able to use a D class or struct (via the C++ interface) and call it from within cython, since cython classes cannot be instantiated without the gil (and this prevents easy parallelisation). I feel a bit foolish asking the question as there is a nice example here for working with plain C using dmd as the linker, and using dmd and gcc to create a DMD shared library statically linked to phobos. However, I have not succeeded in creating a D library statically linked to phobos that works with cython and python, http://dlang.org/dll-linux.html#dso7 I tried it first with test C code to make sure I am able to get the C library/cython/Python interaction working. pytest.c: #include <stdio.h> long pytest(long a) { return a+1; } int main() { long a =pytest(100); printf("%ld",a); return 0; } pytestpy.pyx: cdef extern long pytest(long a) cpdef pytestpy(): return pytest(109) setup.py: from distutils.core import setup from distutils.extension import Extension from Cython.Build import cythonize setup( ext_modules = cythonize([ Extension("pytestpy", ["pytestpy.pyx"], libraries=["pytest"], ) ])) command line: gcc -shared -o libpytest.so pytest.o python setup.py build_ext -i <copied libpytest.so to /usr/local/lib> python import pytestpy pytestpy.pytestpy() <it works> ---- now try pytest.d import std.stdio; extern (C) long pytest(long a) { return a*2; } void main() { auto a =pytest(100); writefln("%d",a); } command line: rm pytestd.o rm libpytest.so rm /usr/local/lib/libpytest.so dmd -c pytest.d -fPIC gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so -L-rpath=/usr/local/lib pythonSince when does gcc have a -defaultlib option? --- $ man gcc | grep defaultlib object-file-name -llibrary -nostartfiles -nodefaultlibs This is useful when you use -nostdlib or -nodefaultlibs but you do -nodefaultlibs is used. -nodefaultlibs -nodefaultlibs is libgcc.a, a library of internal subroutines which -nodefaultlibs you should usually specify -lgcc as well. This --- and why do you have a main function when compiling a shared library? Anyway, I think the problem is that "libpytest.so" can't find "libphobos2.so". Try adding "libphobos2.so"'s location (/usr/lib/x86_64-linux-gnu/libphobos2.so on my linux mint) to your -rpath .Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /usr/local/lib/libpytest.so: undefined symbol: _D3std5stdio12__ModuleInfoZ I guess it is not linking to the D runtime, but I am not sure what I should be doing to fix. Any thoughts appreciated. (The next step I was going to try when this works was C++ interface vs importing as a Cython class, but I thought best to start simple). I am running this on 64 bit Fedora 20. Thanks. Laeeth.import pytestpy
Oct 07 2014
On Wednesday, 8 October 2014 at 00:25:57 UTC, Laeeth Isharc wrote:Hi. Thanks for the quick response. The -defaultlib was left around from trying all kinds of combinations of dmd and gcc. I am not used to gcc, and it will take me some time to become properly acquainted with all the options.I managed to get it to compile with default dmd rpm: https://github.com/ariovistus/cythonic_d fedora 20, x86_64 Are you aware of pyd? (https://bitbucket.org/ariovistus/pyd) It knows how to build shared libraries, so I recommend you play with it, if only to watch how it does that. Biggest gotcha is starting up druntime.
Oct 09 2014
Thanks for this. I am aware of pyd and will take a look at source/build process. Any thoughts on speed in 2014 of pyd vs using cython to talk to D directly via C/C++ interface? I saw this old coment here: prabhuramachandran.blogspot.co.uk/2008/09/python-vs-cython-vs-d-pyd-vs-c-swig " predict that the D version's relative slowness might have something to do with Pyd's somewhat awful handling of arrays (at least in part). More low-level and verbose, but possibly faster code could be written to compensate for this if this is indeed the problem. However, this is not a very attractive solution. (Optimally, Pyd should be capable of directly pointing D arrays at numpy arrays, but this is not actually implemented.)" On Friday, 10 October 2014 at 02:19:17 UTC, Ellery Newcomer wrote:On Wednesday, 8 October 2014 at 00:25:57 UTC, Laeeth Isharc wrote:Hi. Thanks for the quick response. The -defaultlib was left around from trying all kinds of combinations of dmd and gcc. I am not used to gcc, and it will take me some time to become properly acquainted with all the options.I managed to get it to compile with default dmd rpm: https://github.com/ariovistus/cythonic_d fedora 20, x86_64 Are you aware of pyd? (https://bitbucket.org/ariovistus/pyd) It knows how to build shared libraries, so I recommend you play with it, if only to watch how it does that. Biggest gotcha is starting up druntime.
Oct 12 2014
On Sunday, 12 October 2014 at 16:07:19 UTC, Laeeth Isharc wrote:Any thoughts on speed in 2014 of pyd vs using cython to talk to D directly via C/C++ interface? I saw this old coment here:pyd is basically just a convenience layer on top of the C interface. The part that would most likely screw with performance would be conversion from D objects to python objects and vice versa since that normally does a value copy. At the time of that comment, pyd's strategy for arrays was just iterate and convert each element. Now pyd can take advantage of the buffer protocol if it is exposed (2.7+), so converting e.g. numpy arrays should be just a memcopy. If you want by ref semantics instead of by value, use PydObject instead of a D type in your function signature. You can also turn off all conveniences and just use the C api with raw_only (https://bitbucket.org/ariovistus/pyd/wiki/CeleriD)
Oct 12 2014