www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A new Python header

reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
In wrapping Python, I have had cause to update David Rushby's version of 
the Python API header for D.

Aside from some bug fixes, one thing I've been trying to do is remove 
the need to call _loadPythonSupport() at the start of every module. This 
is not much of a burden with the wrapper I'm writing (it can simply call 
it in the module_init function), but I have an idea which I think would 
work better. The trouble is, it seems to cause Python to crash, which 
always seems to put a damper on things.

The _loadPythonSupport function just loads a whole mess of singleton 
objects. Py_None, Py_True, Py_False, all of the type objects, and all of 
the exception types, and some others. Rather than load all of these 
things when the module is loaded (as a given module will only need to
use a few of them), I am trying to load them lazily:

(Excuse the formatting, I didn't pay close attention to width when I 
wrote it.) :-)

private {

PyObject* m_Py_None;
PyObject* m_Py_True;
// ... and so on for all of the singletons

PyObject* m_builtins, m_types, m_weakref;

// Note the new function template syntax. Ohh, ahh.
typeof(Ptr) lazy_sys(alias Ptr, char[] name) () {
     if (Ptr is null) {
         PyObject* sys_modules = PyImport_GetModuleDict();
         Ptr = PyDict_GetItemString(sys_modules, name ~ \0);
         Py_DECREF(sys_modules);
         assert (Ptr !is null,
             "python.d couldn't load " ~ name ~ " attribute!");
     }
     return Ptr;
}

alias lazy_sys!(m_builtins, "__builtin__") builtins;
alias lazy_sys!(m_types, "types") types;
alias lazy_sys!(m_weakref, "weakref") weakref;

typeof(Ptr) lazy_load(alias from, alias Ptr, char[] name) () {
     if (Ptr is null) {
         Ptr = cast(typeof(Ptr)) PyObject_GetAttrString(from(),
             name ~ \0);
         assert (Ptr !is null,
             "python.d couldn't load " ~ name ~ " attribute!");
     }
     return Ptr;
}

} /* end private */

alias lazy_load!(builtins, m_Py_None, "None") Py_None;
alias lazy_load!(builtins, m_Py_True, "True") Py_True;
// ... and so on for all of the singletons

The loading code is based on David Rushby's.

Note that, though these are functions, you don't actually need to call 
them. Viz. you don't need to say "Py_None()", just "Py_None", as D seems 
to treat them like properties, or something. I'm still not totally clear 
on why that works, I only know that it does. (The "def" template in my 
Python wrapper makes use of the same oddity.)

Thus, the singletons are loaded the first time they are needed, and only 
the first time they are needed. There is no need to call Py_DECREF on 
these references in (say) a static destructor, as Python cannot "unload" 
a module; and besides, they cannot be deallocated by Python anyway.

This does, in fact, work, except for the wee problem that Python, once 
it has loaded a module compiled with this header, crashes horribly when 
you try to quit it. This makes me suspect something very wrong is 
happening when Python tries to unload the module. I'm not sure precisely 
what I did here that would cause this, however, and so I throw this out 
for ideas.

-Kirk McDonald
Jun 20 2006
parent reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Ha! I fixed it. It was a stupid problem. I managed to decrement a 
reference count when I shouldn't have, causing Python to free() 
something twice when the interpreter closed down.

Kirk McDonald wrote:

 // Note the new function template syntax. Ohh, ahh.
 typeof(Ptr) lazy_sys(alias Ptr, char[] name) () {
     if (Ptr is null) {
         PyObject* sys_modules = PyImport_GetModuleDict();
         Ptr = PyDict_GetItemString(sys_modules, name ~ \0);


This one right here; GetModuleDict returns a borrowed reference!

         Py_DECREF(sys_modules);
         assert (Ptr !is null,
             "python.d couldn't load " ~ name ~ " attribute!");
     }
     return Ptr;
 }
 


Anyway, the updated (and now working) header can be found at:

http://216.190.88.10/media/python.d

-Kirk McDonald
Jun 20 2006
parent reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Kirk McDonald wrote:

 Ha! I fixed it. It was a stupid problem. I managed to decrement a 
 reference count when I shouldn't have, causing Python to free() 
 something twice when the interpreter closed down.
 
 Kirk McDonald wrote:
 

 // Note the new function template syntax. Ohh, ahh.
 typeof(Ptr) lazy_sys(alias Ptr, char[] name) () {
     if (Ptr is null) {
         PyObject* sys_modules = PyImport_GetModuleDict();
         Ptr = PyDict_GetItemString(sys_modules, name ~ \0);


 
 This one right here; GetModuleDict returns a borrowed reference!
 

         Py_DECREF(sys_modules);
         assert (Ptr !is null,
             "python.d couldn't load " ~ name ~ " attribute!");
     }
     return Ptr;
 }


 
 Anyway, the updated (and now working) header can be found at:
 
 http://216.190.88.10/media/python.d
 
 -Kirk McDonald


Durrrr, that's not the right URL. It's this:

http://216.190.88.10:8087/media/python.d

-Kirk McDonald
Jun 21 2006
parent reply Brad Anderson <brad dsource.org> writes:
Kirk McDonald wrote:

 Durrrr, that's not the right URL. It's this:
 
 http://216.190.88.10:8087/media/python.d



Kirk,

Do you need a place to host this project of yours?  Might I offer up
dsource.org for you?  I would need a project name (hopefully more exciting
than Dython, but if that's all you got...) and a short description to get you
set up.

It seems as if it's gone beyond just being a binding, but if not, and you
don't want a full-blown project, you can always use the Bindings project to
get SVN source control on your files.

http://www.dsource.org/projects/bindings

Cheers,
Brad
Jun 21 2006
parent reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Brad Anderson wrote:

 Kirk McDonald wrote:
 

Durrrr, that's not the right URL. It's this:

http://216.190.88.10:8087/media/python.d


 
 
 
 Kirk,
 
 Do you need a place to host this project of yours?  Might I offer up
 dsource.org for you?  I would need a project name (hopefully more exciting
 than Dython, but if that's all you got...) and a short description to get you
 set up.
 
 It seems as if it's gone beyond just being a binding, but if not, and you
 don't want a full-blown project, you can always use the Bindings project to
 get SVN source control on your files.
 
 http://www.dsource.org/projects/bindings
 
 Cheers,
 Brad


I've been calling it "Pyd" (pronounced "pied"), but I am open to 
suggestions. :-)

I would certainly enjoy some dsource space. My webserver is just my poor 
little Linux box. (Although my logs show only 10 people have grabbed 
that file so far, so I think it's been up to the task.) :-)

The long-term goal is something analagous to Boost.Python. At the 
moment, all I have is a nifty function wrapper and a class that manages 
a PyObject*, overloading very nearly every single overloadable operator, 
passing them along to various Python API functions. (The class, 
DPyObject, is largely working, but I need to clean up the opCall 
overload somewhat.) I also have a couple functions that can convert 
values to and from arbitrary Python objects, which support both the 
function wrapper and the DPyObject class. Some next steps include better 
support for lists, tuples, and dicts. (An array or AA of a convertable 
type can easily be converted to a Python list or dict. A Python list or 
dict can easily be converted to an array or AA of DPyObjects. It would 
also be nice to have a Boost.Python-style make_tuple function.)

So! A short description might be: "An object-oriented wrapper around the 
Python API."

-Kirk McDonald
Jun 21 2006
parent reply Brad Anderson <brad dsource.org> writes:
Kirk McDonald wrote:

 I've been calling it "Pyd" (pronounced "pied"), but I am open to
 suggestions. :-)
 
 I would certainly enjoy some dsource space. My webserver is just my poor
 little Linux box. (Although my logs show only 10 people have grabbed
 that file so far, so I think it's been up to the task.) :-)
 
 The long-term goal is something analagous to Boost.Python. At the
 moment, all I have is a nifty function wrapper and a class that manages
 a PyObject*, overloading very nearly every single overloadable operator,
 passing them along to various Python API functions. (The class,
 DPyObject, is largely working, but I need to clean up the opCall
 overload somewhat.) I also have a couple functions that can convert
 values to and from arbitrary Python objects, which support both the
 function wrapper and the DPyObject class. Some next steps include better
 support for lists, tuples, and dicts. (An array or AA of a convertable
 type can easily be converted to a Python list or dict. A Python list or
 dict can easily be converted to an array or AA of DPyObjects. It would
 also be nice to have a Boost.Python-style make_tuple function.)
 
 So! A short description might be: "An object-oriented wrapper around the
 Python API."
 
 -Kirk McDonald


You need a username at dsource (register in the Forums section of the site)
and then I'll get it set up.

BA
Jun 22 2006
parent Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Brad Anderson wrote:

 Kirk McDonald wrote:

So! A short description might be: "An object-oriented wrapper around the
Python API."

-Kirk McDonald


 
 
 You need a username at dsource (register in the Forums section of the site)
 and then I'll get it set up.
 
 BA


Done! I'm "KirkMcDonald".
Jun 22 2006