www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Recommendations on porting Python to D

reply Chris Piker <chris hoopjump.com> writes:
Hi D

I have a somewhat extensive CGI based web service written in 
Python and I'd like to port it to D.  I can do this manually of 
course, and maybe that's the best way, but for a rough start, is 
anyone aware of any tools that generate an abstract syntax tree 
which could then be converted to somewhat equivalent D code?  
This might give me a jump-start on the manual conversion process. 
  Then later I can work on removing the CGI dependency.

I'm aware that this wouldn't work in general due to all the third 
party modules typically used in python, but most of this code is 
self contained Python2 and doesn't depend on many imports.

I can just call my old C code from D, but the old Python is 
another story.

Thanks for any advice you may have,
Apr 24
next sibling parent Tim <tim.dlang t-online.de> writes:
On Wednesday, 24 April 2024 at 19:50:45 UTC, Chris Piker wrote:

 I have a somewhat extensive CGI based web service written in 
 Python and I'd like to port it to D.  I can do this manually of 
 course, and maybe that's the best way, but for a rough start, 
 is anyone aware of any tools that generate an abstract syntax 
 tree which could then be converted to somewhat equivalent D 
 code?


My parser generator has an example with a grammar for Python: 
https://github.com/tim-dlang/dparsergen/tree/master/examples/python
Apr 24
prev sibling next sibling parent reply Lance Bachmeier <no spam.net> writes:
On Wednesday, 24 April 2024 at 19:50:45 UTC, Chris Piker wrote:


 is anyone aware of any tools that generate an abstract syntax 
 tree which could then be converted to somewhat equivalent D 
 code?  This might give me a jump-start on the manual conversion 
 process.  Then later I can work on removing the CGI dependency.


I haven't used Python much in recent years, but my recollection 
is that Python 2 had an ast module that would spit out the ast 
for you.
Apr 24
parent reply Chris Piker <chris hoopjump.com> writes:
On Wednesday, 24 April 2024 at 20:13:26 UTC, Lance Bachmeier 
wrote:

 I haven't used Python much in recent years, but my recollection 
 is that Python 2 had an ast module that would spit out the ast 
 for you.


Thanks for the pointer! So I ran one of my modules through and 
generated an AST, and get results similar to:
```
Module(
    body=[
       Import(
          names=[
             alias(name='sys')]),
       FunctionDef(
          name='pout',
          args=arguments(
             posonlyargs=[],
             args=[
                arg(arg='item')],
             kwonlyargs=[],
             kw_defaults=[],
             defaults=[]),
          body=[
```
etc.

I presume I'll now need to write something that parses this into 
D source (maybe with the assistance of a module provided above).  
Before I do that, is this syntax general enough that a Python-AST 
to D source converter may already exist?  Obvious searches in 
google and the D package index didn't turn up anything.

I have no background at all in working with ASTs, in fact my 
formal education is not even in CS, so I'm way outside my 
wheelhouse at this point.
Apr 24
next sibling parent reply Sergey <kornburn yandex.ru> writes:
On Wednesday, 24 April 2024 at 22:07:41 UTC, Chris Piker wrote:


 Python-AST to D source converter may already exist?


Another possible way maybe is using C :)
Python -> C -> D
https://wiki.python.org/moin/PythonImplementations#Compilers
Apr 25
parent Chris Piker <chris hoopjump.com> writes:
On Thursday, 25 April 2024 at 07:04:13 UTC, Sergey wrote:

 On Wednesday, 24 April 2024 at 22:07:41 UTC, Chris Piker wrote:


 Python-AST to D source converter may already exist?


 Another possible way maybe is using C :)
 Python -> C -> D
 https://wiki.python.org/moin/PythonImplementations#Compilers


Thanks for the info, though I think going to a low-level language 
in the middle will make code that has almost no high level 
structure.  Converting that back in to class-style D would likely 
take a while, in which case a manual port is a better option.  
Both python and D offer garbage collection so going through a 
non-GC language first would probably obfuscate the intended 
structure.

Maybe converting this AST to another AST format and then using a 
D code generator would be a better route.

So backing up, are there any AST -> D source generators in 
existence?
Apr 25
prev sibling parent reply mw <mingwu gmail.com> writes:
On Wednesday, 24 April 2024 at 22:07:41 UTC, Chris Piker wrote:


 Python-AST to D source converter may already exist?


https://github.com/joortcom/eiffel_rename/tree/main/yi

A rudimentary converter from (extended) Python to D. Maybe you 
can use it as a starting point.

It uses: PEG parser generator for (standard) Python (to extend 
Python syntax):

https://github.com/we-like-parsers/pegen



Another thing you can try (but both the Python-like syntax and 
parser is home-made I think):

dmt is a converter (offline or auto-invoking compiler after 
conversion) from Python-like indention style to curly braces for 
D programming language.

https://github.com/baryluk/dmt

ref:

https://forum.dlang.org/thread/vtftlolshtrtwhlhgubn forum.dlang.org?page=1
Apr 25
next sibling parent mw <mw g.c> writes:
BTW, maybe you can also try Mojo:

https://github.com/modularml/mojo
Apr 25
prev sibling parent reply Chris Piker <chris hoopjump.com> writes:
On Thursday, 25 April 2024 at 16:57:53 UTC, mw wrote:

 On Wednesday, 24 April 2024 at 22:07:41 UTC, Chris Piker wrote:


 Python-AST to D source converter may already exist?


 https://github.com/joortcom/eiffel_rename/tree/main/yi

 A rudimentary converter from (extended) Python to D. Maybe you 
 can use it as a starting point.


Thanks for the suggestions.  I put the question aside for a bit, 
but yesterday ran across a python transpiler here:

   https://github.com/py2many/py2many

It already has support for C++, Go and others.  Since I have 
mountains of python code created over many years, maybe it would 
be worth contributing to this project out of self interest.

Can you take a look at py2many and see what you think about it?  
Getting D on the support list might be good.
May 03
next sibling parent reply mw <mingwu gmail.com> writes:
On Friday, 3 May 2024 at 17:38:10 UTC, Chris Piker wrote:

 On Thursday, 25 April 2024 at 16:57:53 UTC, mw wrote:

 On Wednesday, 24 April 2024 at 22:07:41 UTC, Chris Piker wrote:


 Python-AST to D source converter may already exist?


 https://github.com/joortcom/eiffel_rename/tree/main/yi

 A rudimentary converter from (extended) Python to D. Maybe you 
 can use it as a starting point.


 Thanks for the suggestions.  I put the question aside for a 
 bit, but yesterday ran across a python transpiler here:

   https://github.com/py2many/py2many

 It already has support for C++, Go and others.  Since I have 
 mountains of python code created over many years, maybe it 
 would be worth contributing to this project out of self 
 interest.

 Can you take a look at py2many and see what you think about it? 
  Getting D on the support list might be good.


(Haven't checked its own implementation and output code quality.)

But it says has output for Kotlin, Dart, these two languages are 
similar to D syntactically, so will be a good start.
May 03
parent mw <mw g.c> writes:
On Friday, 3 May 2024 at 17:53:41 UTC, mw wrote:

 On Friday, 3 May 2024 at 17:38:10 UTC, Chris Piker wrote:

 On Thursday, 25 April 2024 at 16:57:53 UTC, mw wrote:

 [...]


 Thanks for the suggestions.  I put the question aside for a 
 bit, but yesterday ran across a python transpiler here:

   https://github.com/py2many/py2many

 It already has support for C++, Go and others.  Since I have 
 mountains of python code created over many years, maybe it 
 would be worth contributing to this project out of self 
 interest.

 Can you take a look at py2many and see what you think about 
 it?  Getting D on the support list might be good.


 (Haven't checked its own implementation and output code 
 quality.)

 But it says has output for Kotlin, Dart, these two languages 
 are similar to D syntactically, so will be a good start.


I took another quick look of the project, it uses the Python 
builtin `ast` module as parser, and visitor design pattern to 
implement the transcompiler, so I think it's of decent quality.

HTH.
May 23
prev sibling parent reply mw <mingwu gmail.com> writes:
On Friday, 3 May 2024 at 17:38:10 UTC, Chris Piker wrote:

 On Thursday, 25 April 2024 at 16:57:53 UTC, mw wrote:

 On Wednesday, 24 April 2024 at 22:07:41 UTC, Chris Piker wrote:


 Python-AST to D source converter may already exist?


 https://github.com/joortcom/eiffel_rename/tree/main/yi

 A rudimentary converter from (extended) Python to D. Maybe you 
 can use it as a starting point.


 Thanks for the suggestions.  I put the question aside for a 
 bit, but yesterday ran across a python transpiler here:

   https://github.com/py2many/py2many

 It already has support for C++, Go and others.  Since I have 
 mountains of python code created over many years, maybe it 
 would be worth contributing to this project out of self 
 interest.

 Can you take a look at py2many and see what you think about it? 
  Getting D on the support list might be good.


Hi,

I have made basic py2many.pyd work at language/syntax level in my 
dlang fork:

https://github.com/mw66/py2many/tree/dlang

The following examples works now:

https://github.com/mw66/py2many/tree/dlang/tests/expected

py2many/ 13:56:23$ ls ./tests/expected/*.d
./tests/expected/bubble_sort.d
./tests/expected/cls.d
./tests/expected/fib.d
./tests/expected/import_tests.d
./tests/expected/classes.d
./tests/expected/dict.d
./tests/expected/hello_world.d
./tests/expected/nested_dict.d


I haven't created PR to be merged into the main branch, since 
it's better to pass all the tests.

All the remaining work is to make Python's specific feature (e.g. 
async), library (e.g. complex number, NamedTemporaryFile) work in 
D. There are many things need to be done, if you have time, you 
can pick up from my fork, and work from there. (E.g. you can 
create PR to my branch, and when everything is ready, we submit 
to the main py2many all together).


HTH.
Jul 12
next sibling parent mw <mingwu gmail.com> writes:
On Friday, 12 July 2024 at 18:07:50 UTC, mw wrote:

 On Friday, 3 May 2024 at 17:38:10 UTC, Chris Piker wrote:


...


 Hi,

 I have made basic py2many.pyd work at language/syntax level in 
 my dlang fork:

 https://github.com/mw66/py2many/tree/dlang

 The following examples works now:

 https://github.com/mw66/py2many/tree/dlang/tests/expected

 py2many/ 13:56:23$ ls ./tests/expected/*.d
 ./tests/expected/bubble_sort.d
 ./tests/expected/cls.d
 ./tests/expected/fib.d
 ./tests/expected/import_tests.d
 ./tests/expected/classes.d
 ./tests/expected/dict.d
 ./tests/expected/hello_world.d
 ./tests/expected/nested_dict.d


 I haven't created PR to be merged into the main branch, since 
 it's better to pass all the tests.

 All the remaining work is to make Python's specific feature 
 (e.g. async), library (e.g. complex number, NamedTemporaryFile) 
 work in D. There are many things need to be done, if you have 
 time, you can pick up from my fork, and work from there. (E.g. 
 you can create PR to my branch, and when everything is ready, 
 we submit to the main py2many all together).


 HTH.



Please use this Makefile for local setup and run tests:

https://github.com/mw66/py2many/blob/dlang/Makefile
Jul 12
prev sibling next sibling parent reply mw <mingwu gmail.com> writes:
On Friday, 12 July 2024 at 18:07:50 UTC, mw wrote:

 On Friday, 3 May 2024 at 17:38:10 UTC, Chris Piker wrote:

 On Thursday, 25 April 2024 at 16:57:53 UTC, mw wrote:

 On Wednesday, 24 April 2024 at 22:07:41 UTC, Chris Piker 
 wrote:


 Python-AST to D source converter may already exist?


 https://github.com/joortcom/eiffel_rename/tree/main/yi

 A rudimentary converter from (extended) Python to D. Maybe 
 you can use it as a starting point.


 Thanks for the suggestions.  I put the question aside for a 
 bit, but yesterday ran across a python transpiler here:

   https://github.com/py2many/py2many

 It already has support for C++, Go and others.  Since I have 
 mountains of python code created over many years, maybe it 
 would be worth contributing to this project out of self 
 interest.

 Can you take a look at py2many and see what you think about 
 it?  Getting D on the support list might be good.


 Hi,

 I have made basic py2many.pyd work at language/syntax level in 
 my dlang fork:

 https://github.com/mw66/py2many/tree/dlang

 The following examples works now:

 https://github.com/mw66/py2many/tree/dlang/tests/expected

 py2many/ 13:56:23$ ls ./tests/expected/*.d
 ./tests/expected/bubble_sort.d
 ./tests/expected/cls.d
 ./tests/expected/fib.d
 ./tests/expected/import_tests.d
 ./tests/expected/classes.d
 ./tests/expected/dict.d
 ./tests/expected/hello_world.d
 ./tests/expected/nested_dict.d


 I haven't created PR to be merged into the main branch, since 
 it's better to pass all the tests.

 All the remaining work is to make Python's specific feature 
 (e.g. async), library (e.g. complex number, NamedTemporaryFile) 
 work in D. There are many things need to be done, if you have 
 time, you can pick up from my fork, and work from there. (E.g. 
 you can create PR to my branch, and when everything is ready, 
 we submit to the main py2many all together).


 HTH.


FYI, now merged into the main branch:

https://github.com/py2many/py2many/tree/main/pyd
Jul 15
parent rkompass <rkompass gmx.de> writes:
On Monday, 15 July 2024 at 19:40:01 UTC, mw wrote:

 On Friday, 12 July 2024 at 18:07:50 UTC, mw wrote:

 [...]


 FYI, now merged into the main branch:

 https://github.com/py2many/py2many/tree/main/pyd


This is great and certainly deserves an own discussion 
contribution in General.

Did you try to convert any of the pystone programs?
This would allow for benchmarking comparisons with e.g. nuitka or 
other approaches of compiled Python.
Jul 17
prev sibling parent reply Chris Piker <chris hoopjump.com> writes:
On Friday, 12 July 2024 at 18:07:50 UTC, mw wrote:

 I have made basic py2many.pyd work at language/syntax level in 
 my dlang fork:

 https://github.com/mw66/py2many/tree/dlang

 The following examples works now:

 https://github.com/mw66/py2many/tree/dlang/tests/expected

 py2many/ 13:56:23$ ls ./tests/expected/*.d
 ./tests/expected/bubble_sort.d
 ...


Outstanding! Thanks for the work! (Apologies for my slow 
response. I've been back on a C project for a while and didn't 
notice your posts.)

I will definitely make use of this when D is back atop the stack, 
should be about three weeks from now. One of my core support 
services still has about 20% python (the rest is D).  Will be 
nice to convert the remaining 20%.
Aug 08
next sibling parent reply Sergey <kornburn yandex.ru> writes:
On Thursday, 8 August 2024 at 20:20:11 UTC, Chris Piker wrote:

 (the rest is D).


D in space when? :)
Aug 08
parent Chris Piker <chris hoopjump.com> writes:
On Thursday, 8 August 2024 at 20:23:02 UTC, Sergey wrote:

 On Thursday, 8 August 2024 at 20:20:11 UTC, Chris Piker wrote:

 (the rest is D).


 D in space when? :)


Unfortunately I'm only the ground segment, so D's not going to 
space today.  But you never know what the future holds.  (I did 
notice this the other night, https://wiki.osdev.org/D_Bare_Bones 
...interesting.)

Back on Earth, I do hope my D based telemetry parser helps out a 
lot of missions over time.  Btw, std.sumtype and dpq2 were 
notably useful in the work. Shout-outs to Paul and Denis are 
definitely in order.  They get free beer from me if I'm ever 
lucky enough to run into either of them.
Aug 08
prev sibling parent mw <mw gmail.com> writes:
FYI, the code has been merged into the main branch already:

https://github.com/py2many/py2many/tree/main/pyd


On Thursday, 8 August 2024 at 20:20:11 UTC, Chris Piker wrote:

 On Friday, 12 July 2024 at 18:07:50 UTC, mw wrote:

 I have made basic py2many.pyd work at language/syntax level in 
 my dlang fork:

 https://github.com/mw66/py2many/tree/dlang

 The following examples works now:

 https://github.com/mw66/py2many/tree/dlang/tests/expected

 py2many/ 13:56:23$ ls ./tests/expected/*.d
 ./tests/expected/bubble_sort.d
 ...


 Outstanding! Thanks for the work! (Apologies for my slow 
 response. I've been back on a C project for a while and didn't 
 notice your posts.)

 I will definitely make use of this when D is back atop the 
 stack, should be about three weeks from now. One of my core 
 support services still has about 20% python (the rest is D).  
 Will be nice to convert the remaining 20%.
Aug 08
prev sibling next sibling parent Sergey <kornburn yandex.ru> writes:
On Wednesday, 24 April 2024 at 19:50:45 UTC, Chris Piker wrote:

 Hi D

 I have a somewhat extensive CGI based web service written in


There is also https://code.dlang.org/packages/arsd-official%3Acgi
Apr 24
prev sibling next sibling parent max haughton <maxhaton gmail.com> writes:
On Wednesday, 24 April 2024 at 19:50:45 UTC, Chris Piker wrote:

 Hi D

 I have a somewhat extensive CGI based web service written in 
 Python and I'd like to port it to D.  I can do this manually of 
 course, and maybe that's the best way, but for a rough start, 
 is anyone aware of any tools that generate an abstract syntax 
 tree which could then be converted to somewhat equivalent D 
 code?  This might give me a jump-start on the manual conversion 
 process.  Then later I can work on removing the CGI dependency.

 I'm aware that this wouldn't work in general due to all the 
 third party modules typically used in python, but most of this 
 code is self contained Python2 and doesn't depend on many 
 imports.

 I can just call my old C code from D, but the old Python is 
 another story.

 Thanks for any advice you may have,


A strategy roughly along the lines of:

Test what you can, then port the tests, then just let chatgpt 
have at it can go further than one might reasonably expect (I 
have used chatgpt to convert to and from languages that don't 
even exist in public and it can basically get the gist of most 
things).

Treat it interactively rather than like a CLI tool, it must be 
said.
Apr 25
prev sibling next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Wednesday, 24 April 2024 at 19:50:45 UTC, Chris Piker wrote:

 I can just call my old C code from D, but the old Python is 
 another story.

 Thanks for any advice you may have,


You could also try some AI solution
Jul 12
prev sibling parent reply IchorDev <zxinsworld gmail.com> writes:
On Wednesday, 24 April 2024 at 19:50:45 UTC, Chris Piker wrote:

 is anyone aware of any tools that generate an abstract syntax 
 tree which could then be converted to somewhat equivalent D 
 code?


Just keep in mind that dependence on such a method might cause 
you to overlook potential higher level transformations between 
the two languages; especially since they have vastly different 
feature-sets, and D is a language where you have to consider 
value vs reference semantics, whereas Python… uhh… makes my head 
hurt.

Does any of this code happen to be open-source?
Aug 10
parent Chris Piker <chris hoopjump.com> writes:
On Saturday, 10 August 2024 at 11:10:01 UTC, IchorDev wrote:


 Does any of this code happen to be open-source?


The initial code I'd like to convert isn't open-source yet, 
though it will have to be soon.  NASA is big on open source these 
days. The logic being, the public paid for development, so the 
public should get to see it, and in the case of analysis 
software, verify it.

Looking down the road, the second and more difficult conversion 
on the horizon is currently still a CGI web-service:  
[dasFlex](https://github.com/das-developers/das2py-server/tree/dasflex).  That
one will take more careful attention as I'll likely go with a fiber based model
and will likely require a fully manual re-write, though it is an example of
work-a-day application code for a converter to chew on.
Aug 15