www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Using python in D

reply rnd <r_narang yahoo.com> writes:
I have a simple python script file which contains following 3 
statements:

import pandas
df = pandas.read_csv('testfile.csv')
print(df[0:3])

Can I incorporate above in a D program?
I see there is pyd package for using python in D: 
https://code.dlang.org/packages/pyd
Will following program work:


import std.stdio;
import pyd.embedded;

void main(){
	py_import("pandas");
	py_stmts("df = pandas.read_csv('testfile.csv')"); 	
	py_stmts("print(df[0:3])");
}

Thanks for your insight.
Jun 06 2019
parent reply rnd <r_narang yahoo.com> writes:
Edit:

I tried above D program and got following error:

core.exception.AssertError /home/abcde/.dub/packages/pyd-0.10.5/pyd/infrastructure
pyd/embedded.d(53): python not initialized
----------------
??:? _d_assert_msg [0x5562d3f3c466]
/home/abcde/.dub/packages/pyd-0.10.5/pyd/infrastructure/pyd/embedded.d:53
pyd.pydobject.PydObject pyd.embedded.py_import(immutable(char)[])
[0x5562d3f0b2f1]
source/main.d:6 _Dmain [0x5562d3f0adeb]


How should I 'initialize python' ?
Jun 06 2019
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 07/06/2019 3:54 PM, rnd wrote:

 Edit:
 
 I tried above D program and got following error:
 
 core.exception.AssertError /home/abcde/.dub/packages/pyd-0.10.5/pyd/infrastructure
pyd/embedded.d(53): 
 python not initialized
 ----------------
 ??:? _d_assert_msg [0x5562d3f3c466]
 /home/abcde/.dub/packages/pyd-0.10.5/pyd/infrastructure/pyd/embedded.d:53
pyd.pydobject.PydObject 
 pyd.embedded.py_import(immutable(char)[]) [0x5562d3f0b2f1]
 source/main.d:6 _Dmain [0x5562d3f0adeb]
 
 
 How should I 'initialize python' ?


The example is probably a good place to begin.

https://github.com/ariovistus/pyd/blob/master/examples/simple_embedded/hello.d
Jun 06 2019
parent reply rnd <r_narang yahoo.com> writes:
On Friday, 7 June 2019 at 04:39:14 UTC, rikki cattermole wrote:

 On 07/06/2019 3:54 PM, rnd wrote:

 
 How should I 'initialize python' ?


 The example is probably a good place to begin.

 https://github.com/ariovistus/pyd/blob/master/examples/simple_embedded/hello.d


Thanks for the link. After putting in missing statements, it 
started running but reported:

      exceptions.ImportError: No module named pandas

I believe pyd is working on Python2 while I have pandas installed 
in Python3.

How can I specify Python version 3 in pyd?
Jun 06 2019
parent reply JN <666total wp.pl> writes:
On Friday, 7 June 2019 at 05:04:30 UTC, rnd wrote:

 On Friday, 7 June 2019 at 04:39:14 UTC, rikki cattermole wrote:

 On 07/06/2019 3:54 PM, rnd wrote:

 
 How should I 'initialize python' ?


 The example is probably a good place to begin.

 https://github.com/ariovistus/pyd/blob/master/examples/simple_embedded/hello.d


 Thanks for the link. After putting in missing statements, it 
 started running but reported:

      exceptions.ImportError: No module named pandas

 I believe pyd is working on Python2 while I have pandas 
 installed in Python3.

 How can I specify Python version 3 in pyd?


https://github.com/ariovistus/pyd

"To use with dub, either specify the relevant subConfiguration 
for your python version, or run source pyd_set_env_vars.sh <your 
python> on linux or pyd_set_env_vars.bat <your python> on windows 
to set the relevant environment variables and use the env 
subConfiguration."
Jun 07 2019
next sibling parent Rnd <r_narang yahoo.com> writes:
On Friday, 7 June 2019 at 10:55:22 UTC, JN wrote:

 On Friday, 7 June 2019 at 05:04:30 UTC, rnd wrote:

 On Friday, 7 June 2019 at 04:39:14 UTC, rikki cattermole wrote:

 On 07/06/2019 3:54 PM, rnd wrote:


 How can I specify Python version 3 in pyd?


 https://github.com/ariovistus/pyd

 "To use with dub, either specify the relevant subConfiguration 
 for your python version, or run source pyd_set_env_vars.sh 
 <your python> on linux or pyd_set_env_vars.bat <your python> on 
 windows to set the relevant environment variables and use the 
 env subConfiguration."


After executable is created, will it work on systems where Python 
and pandas are not installed?
Jun 07 2019
prev sibling parent reply rnd <r_narang yahoo.com> writes:
On Friday, 7 June 2019 at 10:55:22 UTC, JN wrote:

 On Friday, 7 June 2019 at 05:04:30 UTC, rnd wrote:

 On Friday, 7 June 2019 at 04:39:14 UTC, rikki cattermole wrote:

 On 07/06/2019 3:54 PM, rnd wrote:

 
 How should I 'initialize python' ?


 The example is probably a good place to begin.

 https://github.com/ariovistus/pyd/blob/master/examples/simple_embedded/hello.d


 Thanks for the link. After putting in missing statements, it 
 started running but reported:

      exceptions.ImportError: No module named pandas

 I believe pyd is working on Python2 while I have pandas 
 installed in Python3.

 How can I specify Python version 3 in pyd?


 https://github.com/ariovistus/pyd

 "To use with dub, either specify the relevant subConfiguration 
 for your python version, or run source pyd_set_env_vars.sh 
 <your python> on linux or pyd_set_env_vars.bat <your python> on 
 windows to set the relevant environment variables and use the 
 env subConfiguration."


I tried following in dub.selections.json file:

{
	"fileVersion": 1,
	"versions": {
		"pyd": "0.10.5"
	},
	"subConfigurations": {
		"pyd": "python35"
	}
}

I also tried python3.5 and python3 in place of python35, but 
everytime python27 is being used (where pandas is not installed). 
The error is:

$ dub run
Performing "debug" build using /usr/bin/dmd for x86_64.
pyd 0.10.5: target for configuration "python27" is up to date.   
<<< NOTE VERSON HERE;
rntestpy ~master: building configuration "application"...
Linking...
To force a rebuild of up-to-date targets, run again with --force.
Running ./rntestpy
pyd.exception.PythonException /home/cardio/.dub/packages/pyd-0.10.5/pyd/infrastructure/pyd/pydobject.d(59):
exceptions.ImportError: No module named pandas
----------------
/home/cardio/.dub/packages/pyd-0.10.5/pyd/infrastructure/pyd/exception.d:46
void pyd.exception.handle_exception(immutable(char)[], ulong) [0x56438721e9e8]
/home/cardio/.dub/packages/pyd-0.10.5/pyd/infrastructure/pyd/pydobject.d:59
pyd.pydobject.PydObject
pyd.pydobject.PydObject.__ctor(deimos.python.object.PyObject*) [0x56438721f2e8]
/home/cardio/.dub/packages/pyd-0.10.5/pyd/infrastructure/pyd/embedded.d:54
pyd.pydobject.PydObject pyd.embedded.py_import(immutable(char)[])
[0x56438721e501]
source/main.d:6 _Dmain [0x56438721ded0]
Program exited with code 1

I also tried command "dub run --force" but still the error 
persists.

I also tried to give command "source pyd_set_env_vars.sh 
/usr/bin/python3" but still it is not working.

Another command which I tried produced following output but still 
the problem persisted:

$ python3 pyd_get_env_set_text.py
export PYD_D_VERSION_1=Python_2_4_Or_Later
export PYD_D_VERSION_2=Python_2_5_Or_Later
export PYD_D_VERSION_3=Python_2_6_Or_Later
export PYD_D_VERSION_4=Python_2_7_Or_Later
export PYD_D_VERSION_5=Python_3_0_Or_Later
export PYD_D_VERSION_6=Python_3_1_Or_Later
export PYD_D_VERSION_7=Python_3_2_Or_Later
export PYD_D_VERSION_8=Python_3_3_Or_Later
export PYD_D_VERSION_9=Python_3_4_Or_Later
export PYD_D_VERSION_10=Python_3_5_Or_Later
export PYD_D_VERSION_11=__PYD__DUMMY__
export PYD_D_VERSION_12=__PYD__DUMMY__
export PYD_D_VERSION_13=__PYD__DUMMY__
export PYD_LIBPYTHON_DIR=/usr/lib
export PYD_LIBPYTHON=python3.5m

Where could be the problem and how can it be solved?
Jun 07 2019
parent reply Russel Winder <russel winder.org.uk> writes:
On Fri, 2019-06-07 at 14:02 +0000, rnd via Digitalmars-d-learn wrote:

This being D meets Python, I thought I'd take a look at this one.


 I tried following in dub.selections.json file:
=20
 {
 	"fileVersion": 1,
 	"versions": {
 		"pyd": "0.10.5"
 	},
 	"subConfigurations": {
 		"pyd": "python35"
 	}
 }


I do not use JSON for human constructed files, and whilst I think SDL
is inferior to TOML for these sorts of files, I am using the Dub
specification:

   name "pyd_test"
   description "A PyD project from rnd on the email list."
   dependency "pyd" version=3D"*"
   subConfiguration "pyd" "python37"

which seems to do the job.


 I also tried python3.5 and python3 in place of python35, but=20
 everytime python27 is being used (where pandas is not installed).=20
 The error is:


I only have Python 3.7, I deleted Python 2.7 long ago, and always use
the latest Python on Debian (which may get a bit old during the stupid
process called Debian release freeze.


 $ dub run
 Performing "debug" build using /usr/bin/dmd for x86_64.
 pyd 0.10.5: target for configuration "python27" is up to date.  =20
 <<< NOTE VERSON HERE;
 rntestpy ~master: building configuration "application"...
 Linking...
 To force a rebuild of up-to-date targets, run again with --force.
 Running ./rntestpy
 pyd.exception.PythonException /home/cardio/.dub/packages/pyd-
 0.10.5/pyd/infrastructure/pyd/pydobject.d(59):
 exceptions.ImportError: No module named pandas
 ----------------
 /home/cardio/.dub/packages/pyd-
 0.10.5/pyd/infrastructure/pyd/exception.d:46 void
 pyd.exception.handle_exception(immutable(char)[], ulong)
 [0x56438721e9e8]
 /home/cardio/.dub/packages/pyd-
 0.10.5/pyd/infrastructure/pyd/pydobject.d:59 pyd.pydobject.PydObject
 pyd.pydobject.PydObject.__ctor(deimos.python.object.PyObject*)
 [0x56438721f2e8]
 /home/cardio/.dub/packages/pyd-
 0.10.5/pyd/infrastructure/pyd/embedded.d:54 pyd.pydobject.PydObject
 pyd.embedded.py_import(immutable(char)[]) [0x56438721e501]
 source/main.d:6 _Dmain [0x56438721ded0]
 Program exited with code 1
=20
 I also tried command "dub run --force" but still the error=20
 persists.


I am getting what appears to be the exact same problem:

   |> dub
   Performing "debug" build using /usr/bin/ldc2 for x86_64.
   pyd 0.10.5: target for configuration "python37" is up to date.
   pyd_test ~master: building configuration "application"...
   To force a rebuild of up-to-date targets, run again with --force.
   Running ./pyd_test=20
   pyd.exception.PythonException source/main.d(10):=20
   NameError: name 'pandas' is not defined
   ----------------
   exception.d:46 [0x564147a72be6]
   embedded.d:283 [0x564147a72a23]
   main.d:10 [0x564147a6fce1]
   ??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(cha=
r[][])*).runAll() [0x7f8116bff12f]
   ??:? _d_run_main [0x7f8116bfef45]
   __entrypoint.d:8 [0x564147a70314]
   ??:? __libc_start_main [0x7f81167d909a]
   ??:? [0x564147a6fb99]
   Program exited with code 1

But I am quietly confident that pandas is installed into the standard
place:

|> aptitude search python3-pandas
i   python3-pandas                                                         =
                        - data structures for "relational" or "labeled" dat=
a - Python 3                                          =20
i A python3-pandas-lib                                                     =
                        - low-level implementations and bindings for pandas=
 - Python 3                                           =20

So, a priori, this would seem to be a fault within PyD for not looking
up the right place for Python packages.

Having looked at the dub.json ( :-( ) file for PyD it seems that it is
looking in /usr/lib/python3.7m whereas on Debian the directory should
be /usr/lib/python3.7. I have no idea how to control this from a user
dub.sdl.

Of course the installed Python packages are in /usr/lib/python3/dist-
packages, at least on Debian, so the above shouldn't apply!


 I also tried to give command "source pyd_set_env_vars.sh=20
 /usr/bin/python3" but still it is not working.
=20
 Another command which I tried produced following output but still=20
 the problem persisted:
=20
 $ python3 pyd_get_env_set_text.py
 export PYD_D_VERSION_1=3DPython_2_4_Or_Later
 export PYD_D_VERSION_2=3DPython_2_5_Or_Later
 export PYD_D_VERSION_3=3DPython_2_6_Or_Later
 export PYD_D_VERSION_4=3DPython_2_7_Or_Later
 export PYD_D_VERSION_5=3DPython_3_0_Or_Later
 export PYD_D_VERSION_6=3DPython_3_1_Or_Later
 export PYD_D_VERSION_7=3DPython_3_2_Or_Later
 export PYD_D_VERSION_8=3DPython_3_3_Or_Later
 export PYD_D_VERSION_9=3DPython_3_4_Or_Later
 export PYD_D_VERSION_10=3DPython_3_5_Or_Later
 export PYD_D_VERSION_11=3D__PYD__DUMMY__
 export PYD_D_VERSION_12=3D__PYD__DUMMY__
 export PYD_D_VERSION_13=3D__PYD__DUMMY__
 export PYD_LIBPYTHON_DIR=3D/usr/lib
 export PYD_LIBPYTHON=3Dpython3.5m


This last one may well be the problem. Try manually adjusting it to
python3.5?

Having said this I tried export PYD_LIBPYTHON=3Dpython3.7 but it seems to
make no difference to not finding pandas.


 Where could be the problem and how can it be solved?


The problem seems to be that PyD is not looking inthe right place for
importing packages, but this is pure speculation.

Perhaps this merits a bug report against the PyD source repository?


--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Jun 08 2019
parent reply rnd <r_narang yahoo.com> writes:
On Saturday, 8 June 2019 at 09:11:48 UTC, Russel Winder wrote:
...
...

 The problem seems to be that PyD is not looking in the right 
 place for importing packages, but this is pure speculation.

 Perhaps this merits a bug report against the PyD source 
 repository?


Thanks for your time, effort and a detailed analysis of this 
problem.

I did not delete Python2.7 since I thought there could be some 
part of Debian/installed package that may depend on it. But I 
will try to do it now.

I am not well versed with the procedure of submitting bug report 
and will appreciate if you/some other expert on this forum does 
it.

best wishes,
rnd.
Jun 08 2019
parent reply Russel Winder <russel winder.org.uk> writes:
On Sat, 2019-06-08 at 16:53 +0000, rnd via Digitalmars-d-learn wrote:

 [=E2=80=A6]
 I did not delete Python2.7 since I thought there could be some=20
 part of Debian/installed package that may depend on it. But I=20
 will try to do it now.


I overstated the case a bit. I do not use Python 2.7 but there are a
few packages that are LaTeX, Samba, etc. related that still require
Python 2. This is clearly an insane position but the Debian packages
and/or the upstream applications have not been properly managed.  There
are one or two situations where Python 2 should still be used, but for
99.99% of situations Python 3 is the only choice.


 I am not well versed with the procedure of submitting bug report=20
 and will appreciate if you/some other expert on this forum does=20
 it.


PyD is hosted on GitHub so it is a question of putting in an issue
there:

    https://github.com/ariovistus/pyd/issues

The best bet is to write up what you have presented on email here, and
then let the PyD developers guide you into providing the data they need
to decide what the actual problem is and how it might be fixed.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Jun 08 2019
parent reply rnd <r_narang yahoo.com> writes:
On Saturday, 8 June 2019 at 19:35:00 UTC, Russel Winder wrote:
....

 PyD is hosted on GitHub so it is a question of putting in an 
 issue there:

     https://github.com/ariovistus/pyd/issues

 The best bet is to write up what you have presented on email 
 here, and then let the PyD developers guide you into providing 
 the data they need to decide what the actual problem is and how 
 it might be fixed.


Thanks for guidance.

I also wanted to know: Once executable is successfully created, 
will it work on systems where Python and pandas are not installed?
Jun 08 2019
next sibling parent evilrat <evilrat666 gmail.com> writes:
On Sunday, 9 June 2019 at 03:42:23 UTC, rnd wrote:

 Thanks for guidance.

 I also wanted to know: Once executable is successfully created, 
 will it work on systems where Python and pandas are not 
 installed?


That is unlikely. PyD is just a D package, it doesn't have the 
ability to mess up with your project files to pack proper 
shippable app.

On linux it also adds complexity due the way how dynamic 
libraries are handled (i.e. need to set this fancy RPATH, 
LD_LIBRARY_PATH, whatever else necessary), so I would first make 
isolated python copy using venv(and install required packages) 
and then copy it to your final redistributable dir, might also 
require messing a bit with linker flags (to tell where to look 
for shared libs), but anyway it all depends on what prebuilt 
python depends, for example it will likely fail if run on older 
distro or just distro with older libc, or if any of its other 
dependencies being older or missing.


As for py_import issue you can also try do import within py_stmt 
and see if it works(assuming python has all relevant paths set 
up).
Jun 09 2019
prev sibling parent Russel Winder <russel winder.org.uk> writes:
On Sun, 2019-06-09 at 03:42 +0000, rnd via Digitalmars-d-learn wrote:

=20


[=E2=80=A6]

 I also wanted to know: Once executable is successfully created,=20
 will it work on systems where Python and pandas are not installed?


I suspect not, but then I do not use Windows.

I would be surprised if the executable was self contained since that
would imply loading the Python VM, it's standard library, and the used
extenal packages.

py2exe does though do this sort of thing so maybe PyD can. I do not
know, sorry.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Jun 09 2019