www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Expose the CTFE interpter in Phobos?

reply bearophile <bearophileHUGS lycos.com> writes:
(This post is related to something I have suggested time ago, to offer some
parts of the D compiler through the D standard library itself (to use the

do.)

There is a part of the D compiler that to me seems more useful than other ones,
I mean the D interpreter used for CTFE. People often add Lua, Python, MiniD,
JavaScript to large C/C++/D programs (I think most video games contain some
kind of interpreter). So is it possible to offer this part alone to the D
programmer? With it you are allowed to create at runtime small strings of D
code and interpret it. No need to learn and add a second scripting language to
do it. Just import a function execute() from a Phobos module :-)

The (hopefully) introduction of some writing function
(http://d.puremagic.com/issues/show_bug.cgi?id=3952 
https://github.com/D-Programming-Language/dmd/pull/237 ) will make run-time D
interpretation even more useful.

There are disadvantages:
- Even with some printing function, CTFE interpreter is limited. Don will
remove some more CTFE limitations like allowing classes and exceptions, but a
"scripting language" that can't read and save files is less useful; so maybe it
can't fully replace a Lua interpreter.
- If you want to use this feature you need the whole D compiler at run time
too. The D compiler is probably much bigger than a Lua interpreter. On the
other hand maybe it's possible to push the CTFE interpreter in a DLL that is
normally used by the D compiler, that the D standard library uses if you want
to interpret D code at run time. I don't know if this DLL is going to be small
enough.
- CTFE is currently much slower than dynamic/scripting languages like LuaJIT
(that are becoming almost as fast as well compiled D). But in LDC with the LLVM
back-end you have all the tools to create a JIT for interpted D code too :-)
LLVM is not a compiler, it's an aggregate of parts.
- D language is not as simple as a scripting language. In video games the
people that write the 3D engine in C++ are not the same people that program
game logic in Lua. The second group of people has different skills, and they
often are not good programmers able to write C++ code too. So among other
things Lua is used to allow a large number of people to write how the game has
to act, not just hard-core C++ programmers.

Do you know/have use cases for running D code (with current or near-future CTFE
limitations) at run-time?

Bye,
bearophile
Jul 25 2011
next sibling parent reply Ary Manzana <ary esperanto.org.ar> writes:
On 7/25/11 12:55 PM, bearophile wrote:

 (This post is related to something I have suggested time ago, to offer some
parts of the D compiler through the D standard library itself (to use the

do.)

 There is a part of the D compiler that to me seems more useful than other
ones, I mean the D interpreter used for CTFE. People often add Lua, Python,
MiniD, JavaScript to large C/C++/D programs (I think most video games contain
some kind of interpreter). So is it possible to offer this part alone to the D
programmer? With it you are allowed to create at runtime small strings of D
code and interpret it. No need to learn and add a second scripting language to
do it. Just import a function execute() from a Phobos module :-)

 The (hopefully) introduction of some writing function
(http://d.puremagic.com/issues/show_bug.cgi?id=3952 
https://github.com/D-Programming-Language/dmd/pull/237 ) will make run-time D
interpretation even more useful.

 There are disadvantages:
 - Even with some printing function, CTFE interpreter is limited. Don will
remove some more CTFE limitations like allowing classes and exceptions, but a
"scripting language" that can't read and save files is less useful; so maybe it
can't fully replace a Lua interpreter.
 - If you want to use this feature you need the whole D compiler at run time
too. The D compiler is probably much bigger than a Lua interpreter. On the
other hand maybe it's possible to push the CTFE interpreter in a DLL that is
normally used by the D compiler, that the D standard library uses if you want
to interpret D code at run time. I don't know if this DLL is going to be small
enough.
 - CTFE is currently much slower than dynamic/scripting languages like LuaJIT
(that are becoming almost as fast as well compiled D). But in LDC with the LLVM
back-end you have all the tools to create a JIT for interpted D code too :-)
LLVM is not a compiler, it's an aggregate of parts.
 - D language is not as simple as a scripting language. In video games the
people that write the 3D engine in C++ are not the same people that program
game logic in Lua. The second group of people has different skills, and they
often are not good programmers able to write C++ code too. So among other
things Lua is used to allow a large number of people to write how the game has
to act, not just hard-core C++ programmers.

 Do you know/have use cases for running D code (with current or near-future
CTFE limitations) at run-time?

 Bye,
 bearophile


That's why I suggest abstracting away the CTFE to the backend. DMD's 
backend could implement it as now, just interpreting things in memory 
and basically implementing an interpreter from scratch. For LDC the 
backend could just JIT-compile the functions and execute them without 
any restriction at all...
Jul 25 2011
parent Don <nospam nospam.com> writes:
Ary Manzana wrote:

 On 7/25/11 12:55 PM, bearophile wrote:

 (This post is related to something I have suggested time ago, to offer 
 some parts of the D compiler through the D standard library itself (to 
 use the compiler at run-time for some purposes), as recent versions of 


 There is a part of the D compiler that to me seems more useful than 
 other ones, I mean the D interpreter used for CTFE. People often add 
 Lua, Python, MiniD, JavaScript to large C/C++/D programs (I think most 
 video games contain some kind of interpreter). So is it possible to 
 offer this part alone to the D programmer? With it you are allowed to 
 create at runtime small strings of D code and interpret it. No need to 
 learn and add a second scripting language to do it. Just import a 
 function execute() from a Phobos module :-)

 The (hopefully) introduction of some writing function 
 (http://d.puremagic.com/issues/show_bug.cgi?id=3952  
 https://github.com/D-Programming-Language/dmd/pull/237 ) will make 
 run-time D interpretation even more useful.

 There are disadvantages:
 - Even with some printing function, CTFE interpreter is limited. Don 
 will remove some more CTFE limitations like allowing classes and 
 exceptions, but a "scripting language" that can't read and save files 
 is less useful; so maybe it can't fully replace a Lua interpreter.
 - If you want to use this feature you need the whole D compiler at run 
 time too. The D compiler is probably much bigger than a Lua 
 interpreter. On the other hand maybe it's possible to push the CTFE 
 interpreter in a DLL that is normally used by the D compiler, that the 
 D standard library uses if you want to interpret D code at run time. I 
 don't know if this DLL is going to be small enough.
 - CTFE is currently much slower than dynamic/scripting languages like 
 LuaJIT (that are becoming almost as fast as well compiled D). But in 
 LDC with the LLVM back-end you have all the tools to create a JIT for 
 interpted D code too :-) LLVM is not a compiler, it's an aggregate of 
 parts.
 - D language is not as simple as a scripting language. In video games 
 the people that write the 3D engine in C++ are not the same people 
 that program game logic in Lua. The second group of people has 
 different skills, and they often are not good programmers able to 
 write C++ code too. So among other things Lua is used to allow a large 
 number of people to write how the game has to act, not just hard-core 
 C++ programmers.

 Do you know/have use cases for running D code (with current or 
 near-future CTFE limitations) at run-time?

 Bye,
 bearophile


 
 That's why I suggest abstracting away the CTFE to the backend. DMD's 
 backend could implement it as now, just interpreting things in memory 
 and basically implementing an interpreter from scratch. For LDC the 
 backend could just JIT-compile the functions and execute them without 
 any restriction at all...


There's certainly a lot of potential in that direction, but it's not 
simple. For example:
* Cross compilation. The target CPU is not necessarily the same as the 
one which the compiler is running on. (Even the 64 bit compiler is 
executed in 32 bit mode, so this isn't just a theoretical problem).
* The environment in which the user code runs (eg, user account, 
directory being run from, etc) isn't generally the same as the one which 
runs the build system.
Jul 25 2011
prev sibling parent reply Peter Alexander <peter.alexander.au gmail.com> writes:
CTFE isn't really an interpreter in the traditional sense. I believe it 
just uses the parsed source in its IR form, which is different from 
normal byte code interpreters (and also slower). I'm guessing here 
though, so I could be wrong.

Whether I'm right or wrong though, I don't see much use for this. As you 
said:

- CTFE is limited.
- You'd need nearly the whole D compiler, as well as the standard 
library and runtime.
- It would be slower than other scripting languages (unless given lots 
of attention).
- D wasn't made to be a scripting language.

The most important thing is simply that CTFE is a function evaluator. 
For scripting, that's pretty much useless. You need an entire virtual 
machine that you can call into, one that has state and can bind into 
your main application.

I think this would be a lot of work for little gain.
Jul 25 2011
parent "Paulo Pinto" <pjmlp progtools.org> writes:
Hi,

at least in .Net case there is no interpretation going on. All code is
JITted, the bytecodes are only used as file representation.

--
Paulo

"Peter Alexander" <peter.alexander.au gmail.com> wrote in message 
news:j0k8pm$29e3$1 digitalmars.com...

 CTFE isn't really an interpreter in the traditional sense. I believe it 
 just uses the parsed source in its IR form, which is different from normal 
 byte code interpreters (and also slower). I'm guessing here though, so I 
 could be wrong.

 Whether I'm right or wrong though, I don't see much use for this. As you 
 said:

 - CTFE is limited.
 - You'd need nearly the whole D compiler, as well as the standard library 
 and runtime.
 - It would be slower than other scripting languages (unless given lots of 
 attention).
 - D wasn't made to be a scripting language.

 The most important thing is simply that CTFE is a function evaluator. For 
 scripting, that's pretty much useless. You need an entire virtual machine 
 that you can call into, one that has state and can bind into your main 
 application.

 I think this would be a lot of work for little gain.
Jul 26 2011