digitalmars.D - Python integration going well
- David Rushby (51/51) Oct 19 2005 D's a breeze. To fuse link-level C compatibility and high performance i...
- Walter Bright (1/1) Oct 19 2005 This is great stuff! Thanks for taking charge of this and getting it don...
- David Rushby (77/79) Oct 23 2005 You're welcome.
- Walter Bright (8/10) Oct 25 2005 http://kinterbasdb.sourceforge.net/other/d/
- David Rushby (21/31) Oct 25 2005 The only reason I'm currently placing the files at
- John Demme (9/50) Oct 25 2005 In regards to #1 (the shared libraries), I'm really looking forward to
- Walter Bright (11/26) Oct 25 2005 That is
- J C Calvarese (9/18) Oct 25 2005 I don't recall anyone saying it was bad. Sometimes people contain there'...
- J C Calvarese (4/18) Oct 26 2005 I guess I meant "complain" rather than "contain". Perhaps it's time for ...
D's a breeze. To fuse link-level C compatibility and high performance into a reasonably high-level language *without descending into a cesspit of complexity* is a major accomplishment. I'm grateful to Deja for his/her translation of the Python header files; that saved me a lot of work. Having said that, the translation was clearly in a pre-alpha state. Initally, I couldn't even create a Python extension module containing a function that "did nothing", because the global variables that Python expects extensions to manipulate weren't exposed to D. The initial translation had fallen into the trap of trying to provide access to global variables in a C library, while in fact creating a distinct set of variables with the same names in D. I tried the "place the declarations in a secondary D module and import that module, but don't link it" workaround suggested in numerous past threads on this forum, but it didn't work. The D pointers were initialized to *something* non-null, but Python segfaulted upon any attempt to use them. Eventually I gave up and wrote code in the python.d module's static initializer that traverses the Python module structure via the Python/C API and extracts pointers to the requisite global variables. That works fine, though it'll be a bit of a maintenance burden. Another problem with the initial translation was that it assumed that Python's internal Unicode representation is UCS-2. In fact, Python can be compiled in any of the following configs: 1) without any Unicode support 2) using UCS-2 as the internal repr 3) using UCS-4 as the internal repr The most common config, including that of the standard Windows binaries and many Also, the initial translation used the D type int in areas of the Python/C API that expect a C long. That's fine on x86, and on x86_64-Windows, but is not portable to most other 64-bit operating systems, including x86_64-Linux. This problem should be easy to fix with a conditional typedef and minor tweaks. --- All things considered, I'm astonished at how easy it was to go from near-total ignorance of D to a working--albeit trivial--Python extension written in D. I plan to address the issues I outlined above, then have a go at writing Python distutils support for extension modules written in D. (distutils is the standard Python infrastructure for the compilation and distributable-package-creation of extension modules.) After that, I plan to write a tutorial on the subject. In the not-too-distant future, I expect to have polished the infrastructural aspects of writing Python extension modules in D nearly to the extent that they are today for C (ugh, what a sentence... need sleep). Thanks to Deja Augustine for performing the initial translation of the Python header files to D, to Justin Calvarese for digging up that translation after it had disappeared from the web, and most of all to Walter for designing and implementing D. The language has great potential in general, and in particular, as an extension language for Python. Just in case I get run over by a truck tomorrow, here's a snapshot of what I have so far: http://kinterbasdb.sourceforge.net/other/d/python-extension-written-in-d-2005_10_20_00_38.zip
Oct 19 2005
This is great stuff! Thanks for taking charge of this and getting it done.
Oct 19 2005
Walter Bright wrote:This is great stuff! Thanks for taking charge of this and getting it done.You're welcome. I think it's wise of you to be so supportive of efforts to integrate D with the so-called "scripting languages". Although a few of those languages (Python and Ruby foremost among them) provide good support for the construction of non-trivial programs, they're all plagued by slow execution speed, which creates strong demand for high-performance extensions. Folks who're already using a so-called scripting language for serious programming have recognized the value of most of the features that differentiate D from C++, so I'd expect them to be receptive to D. Furthermore, those of us who're using a so-called scripting over our choice of languages, so there are fewer bureaucratic barriers to D adoption among the "scripting language" community. It's hard to tell whether D can be made approachable by Python programmers who've never written a C extension. They still have to use a C API to interact with Python, after all. But Pyrex (http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ ), a language that "lets you write code that mixes Python and C data types any way you want, and compiles it into a C extension for Python", seems to be popular among Python programmers who don't know C. Since I know C and the Python/C API well, I'll probably settle for finishing (and maintaining) the header translation, writing distutils (build infrastructure) support, and creating a basic tutorial. It'd be interesting to see whether a creative person could make it comfortable for Python programmers to write extensions in D without using the Python/C API at all. --- Here's what I've done since my last message on the 20th: - Annotated each section of the python.d header translation to make it easy to trace the D header code back to the original Python C header file. This will make maintenance of the D header file easier. - Updated the translation to reflect additions made to the Python/C API in Python 2.4, which is the current version of Python. The initial translation was based on Python 2.3. - The following header files were missing from the initial translation, but have now been translated: - cStringIO.h - datetime.h - eval.h - genobject.h - marshal.h - pystrtod.h - pythread.h - setobject.h - structmember.h - structseq.h As far as I know, all Python/C API header content that could conceivably be useful in D programmer has now been translated. Some areas of the API are not covered because it seems likely that any extensions that would use these advanced facilities would be written in C. For example, a non-Python-core program that creates and manipulates parse trees for Python code would almost certainly be written in Python itself; only the Python core is likely to implement such functionality in a native extension, and that would be in C, not D. - Used D's mixin feature instead of manually expanding the field header macros _PyObject_HEAD_EXTRA, PyObject_HEAD, PyObject_VAR_HEAD, and PyDescr_COMMON. In addition to easing maintenance, this will make it possible for extensions written in D to run under a debug build of the Python interpreter, which adds extra fields to every PyObject for diagnostic purposes. - Applied a consistent renaming scheme where it was necessary to circumvent D keywords (I used the renaming convention that's most popular in the Python community, which is to add a trailing underscore). - Fixed all problems with 64-bit portability. - Implemented dynamic type checking macros as D functions. These tend to be used heavily in Python/C API code, but the intial translation omitted them. - Fixed numerous other omissions and a few errors. The following issues remain unresolved, though I plan to deal with them in due course (after I've written distutils support for D, so that there's a build framework within which to operate): - Need to conditionalize UCS2-versus-UCS4 Python Unicode API functions based on Python distutils control over the D compilation process. - During this step, need to review Python's unicodeobject.h to ensure that no members were omitted from the Deja's initial translation. - In a build process controlled by Python distutils, need to detect whether the Python interpreter was built in debug build mode, and if so, make the appropriate adjustments to the header mixins. I'll continue to post snapshots at http://kinterbasdb.sourceforge.net/other/d/ in case I get run over by the proverbial truck.
Oct 23 2005
"David Rushby" <David_member pathlink.com> wrote in message news:djhir0$2uuk$1 digitaldaemon.com...I'll continue to post snapshots athttp://kinterbasdb.sourceforge.net/other/d/in case I get run over by the proverbial truck.Your post is full of great information. When you are ready, could you distill it all into a "How To" document and tutorial, and then we can put it on the D web site? That would give it much more exposure, as a random project on sourceforge isn't going to be run across by the people who need it :-(
Oct 25 2005
In article <djlu53$agr$7 digitaldaemon.com>, Walter Bright says..."David Rushby" <David_member pathlink.com> wrote in message news:djhir0$2uuk$1 digitaldaemon.com...The only reason I'm currently placing the files at http://kinterbasdb.sourceforge.net/other/d/ is that I have ready access to that web space. kinterbasdb is not related to D in any way. I believe in the value of documentation, and have always planned to document Python/D integration when I am confident that the infrastructure is sufficiently mature. (I'd like to compliment you on the quality of D's documentation. Not only is it broad in its coverage, but it's written in fluent English. That is atypical and quite refreshing.) Before I reach the stage where I consider Python/D integration worth documenting or publicizing, though, I need to: 1) Make sure there are no fundamental stumbling blocks (such as http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29399 ). 2) Have implemented the infrastructure that Python programmers expect (chiefly distutils support). stumbling blocks. I couldn't seriously advocate D as an extension language for Python when there's no (reliable) way to use it as such under Linux. I'm willing to put forth effort myself to correct this, assuming that it doesn't require months of preparatory study.I'll continue to post snapshots athttp://kinterbasdb.sourceforge.net/other/d/in case I get run over by the proverbial truck.Your post is full of great information. When you are ready, could you distill it all into a "How To" document and tutorial, and then we can put it on the D web site? That would give it much more exposure, as a random project on sourceforge isn't going to be run across by the people who need it :-(
Oct 25 2005
this D integration as well as shared library support (keeping in mind that I don't even have a copy of Windows on my computer). Walter, since you seem interested in this project, this should be a good motivation to get the shared library support working under Linux. Right? BTW, did you read Georg's recent "Policy opinion" post? I agree with it, although I'm pretty biased. ~John Demme On Tue, 25 Oct 2005 19:35:05 +0000, David Rushby wrote:In article <djlu53$agr$7 digitaldaemon.com>, Walter Bright says..."David Rushby" <David_member pathlink.com> wrote in message news:djhir0$2uuk$1 digitaldaemon.com...The only reason I'm currently placing the files at http://kinterbasdb.sourceforge.net/other/d/ is that I have ready access to that web space. kinterbasdb is not related to D in any way. I believe in the value of documentation, and have always planned to document Python/D integration when I am confident that the infrastructure is sufficiently mature. (I'd like to compliment you on the quality of D's documentation. Not only is it broad in its coverage, but it's written in fluent English. That is atypical and quite refreshing.) Before I reach the stage where I consider Python/D integration worth documenting or publicizing, though, I need to: 1) Make sure there are no fundamental stumbling blocks (such as http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29399 ). 2) Have implemented the infrastructure that Python programmers expect (chiefly distutils support). stumbling blocks. I couldn't seriously advocate D as an extension language for Python when there's no (reliable) way to use it as such under Linux. I'm willing to put forth effort myself to correct this, assuming that it doesn't require months of preparatory study.I'll continue to post snapshots athttp://kinterbasdb.sourceforge.net/other/d/in case I get run over by the proverbial truck.Your post is full of great information. When you are ready, could you distill it all into a "How To" document and tutorial, and then we can put it on the D web site? That would give it much more exposure, as a random project on sourceforge isn't going to be run across by the people who need it :-(
Oct 25 2005
"David Rushby" <David_member pathlink.com> wrote in message news:djm1d9$nq7$1 digitaldaemon.com...(I'd like to compliment you on the quality of D's documentation. Not only is it broad in its coverage, but it's written in fluent English.That isatypical and quite refreshing.)Thanks for the kind words, most people say the documentation stinks <g>. I actually agree with them, and making the documentation better was the inspiration for ddoc.Before I reach the stage where I consider Python/D integration worthdocumentingor publicizing, though, I need to: 1) Make sure there are no fundamental stumbling blocks (such as http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29399 ). 2) Have implemented the infrastructure that Python programmers expect(chieflydistutils support). stumbling blocks. I couldn't seriously advocate D as an extensionlanguage forPython when there's no (reliable) way to use it as such under Linux. I'm willing to put forth effort myself to correct this, assuming that itdoesn'trequire months of preparatory study.
Oct 25 2005
In article <djn1el$2v5n$2 digitaldaemon.com>, Walter Bright says..."David Rushby" <David_member pathlink.com> wrote in message news:djm1d9$nq7$1 digitaldaemon.com...I don't recall anyone saying it was bad. Sometimes people contain there's just not enough of it. ;) But I guess Ddoc should definitely help with that. I think having Ddoc built into the compiler is smart and it's going to be a terrific boon for D and D users. I suppose you might be thinking of when people complained about the formating of the web pages, but you'll probably have a lot fewer complaints now. I think they look pretty slick these days! Thanks for fixing them up. jcc7(I'd like to compliment you on the quality of D's documentation. Not only is it broad in its coverage, but it's written in fluent English.That isatypical and quite refreshing.)Thanks for the kind words, most people say the documentation stinks <g>. I actually agree with them, and making the documentation better was the inspiration for ddoc.
Oct 25 2005
In article <djn7ug$8h8$1 digitaldaemon.com>, J C Calvarese says...In article <djn1el$2v5n$2 digitaldaemon.com>, Walter Bright says...I guess I meant "complain" rather than "contain". Perhaps it's time for me to go to bed? jcc7"David Rushby" <David_member pathlink.com> wrote in message news:djm1d9$nq7$1 digitaldaemon.com...I don't recall anyone saying it was bad. Sometimes people contain there's just(I'd like to compliment you on the quality of D's documentation. Not only is it broad in its coverage, but it's written in fluent English.That isatypical and quite refreshing.)Thanks for the kind words, most people say the documentation stinks <g>. I actually agree with them, and making the documentation better was the inspiration for ddoc.
Oct 26 2005