www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Is D a good choice for embedding python/octave/julia

reply Bastien <bDOTqueste ueaDOTac.uk> writes:
Hi, apologies for what may be a fairly obvious question to some.

## The background:
I have been tasked with building software to process data output 
by scientific instruments for non-experts - basically with GUI, 
menus, easy config files (JSON or similar) - and the ability to 
do some serious number crunching.

My background is python/octave and would be happy building it in 
python (or god forbid, even octave), but it would end up clunky 
and slow once ported to a standalone executable. Hence why I'm 
looking at other languages. D caught my eye.

## The problem:
The sticking point is unless I commit the rest of my life to 
maintaining this software, I can't write it all in D. The 
algorithms change/are improved yearly; the output format from the 
instrument changes once in a while and therefore these need to be 
easily scripted/modified by other (non-programming) scientists 
and the community that only really know python and octave.

Essentially I'd like a D front end, and a D back-end that does 
most of the memory and data management but calls and interprets 
.py, .m and/or .jl scripts (python, matlab, julia) to know how to 
treat the data. This leaves the py/m/jl scripts visible to be 
edited by the end user.

## The question:
Can it be done?
Does this entirely defeat the point of using D and I should just 
code it in python because of the added overheads?


Thanks for your help!
B
Mar 13 2016
next sibling parent Lass Safin <lasssafin gmail.com> writes:
On Sunday, 13 March 2016 at 13:02:16 UTC, Bastien wrote:

 Hi, apologies for what may be a fairly obvious question to some.

 ## The background:
 I have been tasked with building software to process data 
 output by scientific instruments for non-experts - basically 
 with GUI, menus, easy config files (JSON or similar) - and the 
 ability to do some serious number crunching.

 My background is python/octave and would be happy building it 
 in python (or god forbid, even octave), but it would end up 
 clunky and slow once ported to a standalone executable. Hence 
 why I'm looking at other languages. D caught my eye.

 ## The problem:
 The sticking point is unless I commit the rest of my life to 
 maintaining this software, I can't write it all in D. The 
 algorithms change/are improved yearly; the output format from 
 the instrument changes once in a while and therefore these need 
 to be easily scripted/modified by other (non-programming) 
 scientists and the community that only really know python and 
 octave.

 Essentially I'd like a D front end, and a D back-end that does 
 most of the memory and data management but calls and interprets 
 .py, .m and/or .jl scripts (python, matlab, julia) to know how 
 to treat the data. This leaves the py/m/jl scripts visible to 
 be edited by the end user.

 ## The question:
 Can it be done?
 Does this entirely defeat the point of using D and I should 
 just code it in python because of the added overheads?


 Thanks for your help!
 B


I REALLY don't think you should use _any_ scripted language, if 
what you're looking for is speed.
Now for your main question:
It can be done.
An incomplete list of libraries and bindings for D: 
http://wiki.dlang.org/List_of_Libraries_and_Frameworks.
It includes tools such as GTK.

And I'm very sure that it will be faster than writing it 
completely in python.

Another thing: I myself find D *much* easier to program in than 
python (having experience in both). The many meta-programming 
tools in D and the nice syntactic features of D really make-up 
for the increased complexity of the language compared to Python.


Somethings I'd like to recommend: OpenCL. For algorithms and 
such, using the GPU is much much faster than using the CPU.
Mar 13 2016
prev sibling next sibling parent Chris Wright <dhasenan gmail.com> writes:
On Sun, 13 Mar 2016 13:02:16 +0000, Bastien wrote:

 The sticking point is unless I commit the rest of my life to maintaining
 this software, I can't write it all in D. The algorithms change/are
 improved yearly; the output format from the instrument changes once in a
 while and therefore these need to be easily scripted/modified by other
 (non-programming) scientists and the community that only really know
 python and octave.


http://code.dlang.org/packages/pyd has you covered.

The degree to which it has you covered is based on how much you can wrap 
expensive operations up opaquely. If it turns out next year that someone 
has a new algorithm they need for something you implemented in D, it's 
going to be more work overall.
Mar 13 2016
prev sibling next sibling parent reply cym13 <cpicard openmailbox.org> writes:
On Sunday, 13 March 2016 at 13:02:16 UTC, Bastien wrote:

 Hi, apologies for what may be a fairly obvious question to some.

 ## The background:
 I have been tasked with building software to process data 
 output by scientific instruments for non-experts - basically 
 with GUI, menus, easy config files (JSON or similar) - and the 
 ability to do some serious number crunching.

 My background is python/octave and would be happy building it 
 in python (or god forbid, even octave), but it would end up 
 clunky and slow once ported to a standalone executable. Hence 
 why I'm looking at other languages. D caught my eye.

 ## The problem:
 The sticking point is unless I commit the rest of my life to 
 maintaining this software, I can't write it all in D. The 
 algorithms change/are improved yearly; the output format from 
 the instrument changes once in a while and therefore these need 
 to be easily scripted/modified by other (non-programming) 
 scientists and the community that only really know python and 
 octave.

 Essentially I'd like a D front end, and a D back-end that does 
 most of the memory and data management but calls and interprets 
 .py, .m and/or .jl scripts (python, matlab, julia) to know how 
 to treat the data. This leaves the py/m/jl scripts visible to 
 be edited by the end user.

 ## The question:
 Can it be done?
 Does this entirely defeat the point of using D and I should 
 just code it in python because of the added overheads?


 Thanks for your help!
 B



I don't have much experience in mixing python and D but here's my 
take on it:

D is a great language but it's not a great glue language. I know 
of no
binding to Julia but good bindings to python exist (pyd as said 
above).
However, if what you want to keep in python is the algorithms 
themselves then
I don't see the point. If I were to mix the two languages I'd use 
python to
do the user interface, some module interface in order to link the 
tool to
others maybe, but the algorithm would definitely be the one thing 
I would do
in D because that's what D is for.
Mar 13 2016
parent reply Bastien <bDOTqueste ueaDOTac.uk> writes:
On Sunday, 13 March 2016 at 18:12:07 UTC, cym13 wrote:

 On Sunday, 13 March 2016 at 13:02:16 UTC, Bastien wrote:

 Hi, apologies for what may be a fairly obvious question to 
 some.

 ## The background:
 I have been tasked with building software to process data 
 output by scientific instruments for non-experts - basically 
 with GUI, menus, easy config files (JSON or similar) - and the 
 ability to do some serious number crunching.

 My background is python/octave and would be happy building it 
 in python (or god forbid, even octave), but it would end up 
 clunky and slow once ported to a standalone executable. Hence 
 why I'm looking at other languages. D caught my eye.

 ## The problem:
 The sticking point is unless I commit the rest of my life to 
 maintaining this software, I can't write it all in D. The 
 algorithms change/are improved yearly; the output format from 
 the instrument changes once in a while and therefore these 
 need to be easily scripted/modified by other (non-programming) 
 scientists and the community that only really know python and 
 octave.

 Essentially I'd like a D front end, and a D back-end that does 
 most of the memory and data management but calls and 
 interprets .py, .m and/or .jl scripts (python, matlab, julia) 
 to know how to treat the data. This leaves the py/m/jl scripts 
 visible to be edited by the end user.

 ## The question:
 Can it be done?
 Does this entirely defeat the point of using D and I should 
 just code it in python because of the added overheads?


 Thanks for your help!
 B



 I don't have much experience in mixing python and D but here's 
 my take on it:

 D is a great language but it's not a great glue language. I 
 know of no
 binding to Julia but good bindings to python exist (pyd as said 
 above).
 However, if what you want to keep in python is the algorithms 
 themselves then
 I don't see the point. If I were to mix the two languages I'd 
 use python to
 do the user interface, some module interface in order to link 
 the tool to
 others maybe, but the algorithm would definitely be the one 
 thing I would do
 in D because that's what D is for.


Thanks for all the very useful replies!
Overall seems that D on its own may be better. May not be such a 
bad thing in the end if it moves the scientists away from 
commerical matlab and the great python 2/3 schism.

I guess my resilience to using D for the algorithms is because 
with python, I have access to numpy and matplotlib. There do seem 
to be some ongoing developments though:
http://forum.dlang.org/post/mailman.4923.1434903477.7663.digitalmars-d puremagic.com

So maybe that will all change. I've just ordered a couple books 
which will hopefully give me a bit more insight into the 
feasibility of this project. Otherwise, I'll fall back on 
python...
Mar 13 2016
parent reply Laeeth Isharc <nospamlaeeth nospamlaeeth.com> writes:
On Sunday, 13 March 2016 at 18:42:59 UTC, Bastien wrote:

 The sticking point is unless I commit the rest of my life to 
 maintaining this software, I can't write it all in D. The 
 algorithms change/are improved yearly; the output format from 
 the instrument changes once in a while and therefore these need 
 to be easily scripted/modified by other (non-programming) 
 scientists and the community that only really know python and 
 octave.


It's pretty easy to use D from python and python from D.  The 
documentation for PyD isn't so great, but once you have figured 
it out, it's easy.  Start with something very small (the examples 
and tests in the PyD repo are the best documentation) and work up 
from there.

You can even embed D in a Jupyter/iPython notebook, and write 
some cells in D and some in python and have them call each other. 
  See PydMagic by John Colvin.
https://github.com/DlangScience/PydMagic

D can talk to Julia via C linkage already.  Just extern(C) when 
you declare your D function, and I guess use C style arrays.

I started porting julia.h to D, but didn't get have time to 
finish.  Ilya Yaroshenko, who created the ndslice library in 
std.experimental, will be working on julia integration in coming 
months.

By the way, you can also embed R in D and call D from R - see 
work by bachmeier on bitbucket.  That's very important because of 
the huge numbers of R libraries.  Although R is slow, there 
apparently shouldn't be so much overhead in calling a C library 
written for R from D.

 I guess my resilience to using D for the algorithms is because 
 with python, I have access to numpy and matplotlib. There do 
 seem to be some ongoing developments though:
 http://forum.dlang.org/post/mailman.4923.1434903477.7663.digitalmars-d puremagic.com


You already have access to matplotlib from D - see here (and 
scroll down or search for matplotlib):
https://d.readthedocs.org/en/latest/examples.html


And you can call numpy from D, but there is some overhead, and I 
wouldn't want to do it inside a tight loop.

 From PyD unit tests:

InterpContext context = new InterpContext();

     context.py_stmts(outdent("
         import numpy
         a = numpy.eye(2, dtype='complex128')
     "));

     context.a.to_d!(Complex!double[][] )();
Mar 13 2016
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 03/13/2016 02:36 PM, Laeeth Isharc wrote:

 InterpContext context = new InterpContext();

      context.py_stmts(outdent("
          import numpy
          a = numpy.eye(2, dtype='complex128')
      "));

      context.a.to_d!(Complex!double[][] )();


nitpicking, but the outdent is unnecessary, py_stmts calls it.

hm. maybe that should be documented..
Mar 14 2016
parent Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Tuesday, 15 March 2016 at 05:56:36 UTC, Ellery Newcomer wrote:

 On 03/13/2016 02:36 PM, Laeeth Isharc wrote:

 InterpContext context = new InterpContext();

      context.py_stmts(outdent("
          import numpy
          a = numpy.eye(2, dtype='complex128')
      "));

      context.a.to_d!(Complex!double[][] )();


 nitpicking, but the outdent is unnecessary, py_stmts calls it.

 hm. maybe that should be documented..


I was in a hurry so just copy and pasted from the unit test...  I 
never do that myself.
Mar 15 2016
prev sibling parent John Colvin <john.loughran.colvin gmail.com> writes:
On Sunday, 13 March 2016 at 13:02:16 UTC, Bastien wrote:

 Hi, apologies for what may be a fairly obvious question to some.

 ## The background:
 I have been tasked with building software to process data 
 output by scientific instruments for non-experts - basically 
 with GUI, menus, easy config files (JSON or similar) - and the 
 ability to do some serious number crunching.

 [...]


If the other language has some C api that can be called to 
interpret code then you can do so from D as well. See also e.g. 
https://github.com/ariovistus/pyd to make this easier for python.
Mar 15 2016