www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D run-time interpretation

reply Jacob <JJ12343 gmail.com> writes:
There are many programs out there that use some sort of scripting 
capabilities.

I've been wanting to write an app that exposes a scripting like 
environment D, lua, or some other fast language to the user.

But there are two requirements:

1. The user has access to all objects created in app. e.g., if I 
create a class X in the app then the user can instantiate X in 
the script without having to recreate it. (the script then, is 
sort of an extension of the app)

I don't mind actually having to expose X to the script, but it 
should be easy.

2. It should be fast. When it is "recompiled" it shouldn't take 
more than a few seconds. It also shouldn't require a 
re-initialization of everything.

Can D do this easily?
Oct 06 2015
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, 6 October 2015 at 19:42:08 UTC, Jacob wrote:

 There are many programs out there that use some sort of 
 scripting capabilities.

 I've been wanting to write an app that exposes a scripting like 
 environment D, lua, or some other fast language to the user.

 But there are two requirements:

 1. The user has access to all objects created in app. e.g., if 
 I create a class X in the app then the user can instantiate X 
 in the script without having to recreate it. (the script then, 
 is sort of an extension of the app)

 I don't mind actually having to expose X to the script, but it 
 should be easy.

 2. It should be fast. When it is "recompiled" it shouldn't take 
 more than a few seconds. It also shouldn't require a 
 re-initialization of everything.

 Can D do this easily?


You know, you can use D as a scripting language. On Linux, you do 
something like put



at the beginning of the file, mark it is executable, and then you 
can run it. I don't know what you need to put on the top to make 
it work on Windows.

These days, I write most of my scripts in D.

- Jonathan M Davis
Oct 06 2015
parent reply Jacob <JJ12343 gmail.com> writes:
On Tuesday, 6 October 2015 at 20:02:48 UTC, Jonathan M Davis 
wrote:

 On Tuesday, 6 October 2015 at 19:42:08 UTC, Jacob wrote:

 There are many programs out there that use some sort of 
 scripting capabilities.

 I've been wanting to write an app that exposes a scripting 
 like environment D, lua, or some other fast language to the 
 user.

 But there are two requirements:

 1. The user has access to all objects created in app. e.g., if 
 I create a class X in the app then the user can instantiate X 
 in the script without having to recreate it. (the script then, 
 is sort of an extension of the app)

 I don't mind actually having to expose X to the script, but it 
 should be easy.

 2. It should be fast. When it is "recompiled" it shouldn't 
 take more than a few seconds. It also shouldn't require a 
 re-initialization of everything.

 Can D do this easily?


 You know, you can use D as a scripting language. On Linux, you 
 do something like put



 at the beginning of the file, mark it is executable, and then 
 you can run it. I don't know what you need to put on the top to 
 make it work on Windows.

 These days, I write most of my scripts in D.

 - Jonathan M Davis



But this isn't what I want, only part.

Essentially I want to write an app that allows the user to 
"inteface" with the internals of the app, to control things and 
such.

Half the app is graphics and provides the backbone, sets up the 
graphics, deals with all that mess that the user doesn't need to 
deal with. The user, though, can control specific things such as 
the colors, objects, and such in the graphics scene. But instead 
of providing relatively static settings for the user(mess with 
sliders and such), I would like them to be able to specify what 
they want in "code".

e.g.,
User script code:

Graphics.Objects["Ship"].X = 10*cos(time);
Graphics.Objects["Ship"].Y = 10*sin(time);


...

So the user has the ability to access the "internals". Here 
Graphics is a sort of container for all the graphics, Objects is 
a map of all the objects in the scene.

But ultimately the code should be "compiled" in to the backbone 
of the app so it runs as fast as possible(not re-interpreted each 
scene).

One can think of it like this: The user provides code, the code 
is compiled as a function and the function called by the app. As 
long as one can link up the objects this should not be difficult.

If I were to do this by hand I'd have to write or use an 
interpreter or compiler(probably too slow) and provide a mapping 
of all the objects I want to expose(I'll have to "export" them 
from the app).

I could use something like LuaD, as I think it has the ability to 
parse strings as lua code, and the lua code can directly 
interface with the app. So this would not be that difficult to 
implement. The questions here are, is it fast enough and can it 
provide the simplicity I'd want.


time, to re-compile source code chunks and execute them as part 
of the program(not as separate entities that have no access to 
the underlying code).

This process would work but surely is too slow:

1. Treat the script code as pure D code.
2. Provide object references(pointers) to the objects I want to 
expose to the D code(as function arguments or through a lookup 
table or whatever)
3. compile the code into a dll.
4. Call the code in the dll per frame(or how ever I need to).


This would work great except the recompiling part and dll 
interfacing would surely be too slow. Since the user code could 
be modified often(modify a line, recompile), I need the changes 
to happen as fast as possible. If one wanted to modify the above 
code to

Graphics.Objects["Ship"].X = 10*sin(time);
Graphics.Objects["Ship"].Y = 10*cos(time);

I don't want the user to wait 5 mins while the app does all the 
work listed. Even, if it's 50 seconds it's still too slow.

I'd also like to use D as the scripting language since it would 
probably flow better(although, maybe lua and python could work 
too).

My app essentially has a visual scene representing the 3d 
graphics, and a "script editor" that allows the user to interact 
and modify the scene using code.

The scripting is crucial as it what makes the app what it is.
Oct 06 2015
parent reply Laeeth Isharc <laeethnospam nospamlaeeth.com> writes:
On Tuesday, 6 October 2015 at 21:41:18 UTC, Jacob wrote:

 On Tuesday, 6 October 2015 at 19:42:08 UTC, Jacob wrote:

 
 I've been wanting to write an app that exposes a scripting 
 like environment D, lua, or some other fast language to the 
 user.






PyD works nicely and I have used it enough to feel comfortable, 
but fast it isn't.

Isn't LuaJIT the obvious answer for you ?  You can use the FFI 
and generate C headers, or use LuaD.  I am struggling a bit with 
LuaD at the moment, so I can't tell you for sure it's perfectly 
usable, but you have the backdrop of knowing that the worst case 
of dropping down to C style is still pretty easy.  Note that the 
FFI allows you to wrap foreign methods nicely for constructors 
etc.  it even deals with templated types !

LuaJIT is shockingly fast, and Lua itself seems the perfect 
scripting language.

I am replying now as I have the exact same problem in a different 
domain myself, and tentatively going with Lua even though I know 
Python and PyD better.  LuaD was complaining and trying to wrap 
some private methods, enums etc.  my meta programming is at an 
early stage, and I am trying to fix these teething problems.  But 
it shouldn't be too bad - I am not far off, I think.

Also it currently harmlessly segfaults on exit due to a change in 
compiler behaviour interacting with LuaObject destructor (Ponce 
may have the answer) and creating a D module for Lua doesn't work 
for me (segfaults) on Arch Linux 64 although embedding Lua is 
fine.  I am not worried about it because I know I have C API as 
last resort, and it's coming along nicely anyway.


 1. The user has access to all objects created in app. e.g., 
 if I create a class X in the app then the user can 
 instantiate X in the script without having to recreate it. 
 (the script then, is sort of an extension of the app)

 I don't mind actually having to expose X to the script, but 
 it should be easy.






Should be easy.

 2. It should be fast. When it is "recompiled" it shouldn't 
 take more than a few seconds. It also shouldn't require a 
 re-initialization of everything.

 Can D do this easily?






You won't need to precompile Lua but if you do it will be quick.


 Half the app is graphics and provides the backbone, sets up the 
 graphics, deals with all that mess that the user doesn't need 
 to deal with. The user, though, can control specific things 
 such as the colors, objects, and such in the graphics scene. 
 But instead of providing relatively static settings for the 
 user(mess with sliders and such), I would like them to be able 
 to specify what they want in "code".


Yes - exactly...  Back end power, correctness, efficiency with a 
light scripting interface.



 But ultimately the code should be "compiled" in to the backbone 
 of the app so it runs as fast as possible(not re-interpreted 
 each scene).


LuaJIT will be very fast.  I think calling back to C might not be 
something you want to do in an inner loop though, so structure 
accordingly.  It's a drop in replacement for Lua 5.1, which is 
the version supported by LuaD so just change the library you link 
against.  Beware that you might need to compile LuaJIT  on Linux 
with don't disable stack frame and some other weirdness on 
Windows.  And that LuaD is usable but docs could be more complete 
and it's a little rough around edges.


 I could use something like LuaD, as I think it has the ability 
 to parse strings as lua code, and the lua code can directly 
 interface with the app. So this would not be that difficult to 
 implement. The questions here are, is it fast enough and can it 
 provide the simplicity I'd want.


I think yes and yes.  Can, but you might spend an afternoon or 
two grumbling at it before you figure it out.


 1. Treat the script code as pure D code.
 2. Provide object references(pointers) to the objects I want to 
 expose to the D code(as function arguments or through a lookup 
 table or whatever)
 3. compile the code into a dll.
 4. Call the code in the dll per frame(or how ever I need to).


 This would work great except the recompiling part and dll 
 interfacing would surely be too slow. Since the user code could 
 be modified often(modify a line, recompile), I need the changes 
 to happen as fast as possible. If one wanted to modify the 
 above code to


Have a look at D REPL or Jupyter notebook for what's possible 
with compiling and dynamically loading libraries.  Strikes me as 
a no brainer to use Lua.  But not because of compilation speed - 
for D that should be a handful of seconds at most (Phobos 
compiles in four seconds)


 I'd also like to use D as the scripting language since it would 
 probably flow better(although, maybe lua and python could work 
 too).

 My app essentially has a visual scene representing the 3d 
 graphics, and a "script editor" that allows the user to 
 interact and modify the scene using code.

 The scripting is crucial as it what makes the app what it is.


Let us know what you decide and how it goes.



 Oct 06 2015



  parent  Laeeth Isharc <laeethnospam nospamlaeeth.com>  writes:


On Tuesday, 6 October 2015 at 22:05:59 UTC, Laeeth Isharc wrote:

 LuaJIT will be very fast.  I think calling back to C might not 
 be something you want to do in an inner loop though, so 
 structure accordingly.


I meant other way around.  You don't want to call out to Lua 
there, but I don't think that's your intent anyway.

https://gist.github.com/spion/3049314

it took a lot of work just for C++ to beat LuaJIT on this little 
benchmark.



 Oct 06 2015



prev sibling  parent  Vladimir Panteleev <thecybershadow.lists gmail.com>  writes:


On Tuesday, 6 October 2015 at 19:42:08 UTC, Jacob wrote:

 There are many programs out there that use some sort of 
 scripting capabilities.

 I've been wanting to write an app that exposes a scripting like 
 environment D, lua, or some other fast language to the user.

 But there are two requirements:

 1. The user has access to all objects created in app. e.g., if 
 I create a class X in the app then the user can instantiate X 
 in the script without having to recreate it. (the script then, 
 is sort of an extension of the app)

 I don't mind actually having to expose X to the script, but it 
 should be easy.

 2. It should be fast. When it is "recompiled" it shouldn't take 
 more than a few seconds. It also shouldn't require a 
 re-initialization of everything.

 Can D do this easily?


I can see two ways about this:

1. Use D

Can you include your application's source code with the 
application?

If so, what I think could work is simply import modules belonging 
to your app from your plugin's source code, but don't link with 
the app object files.

The main complication with this approach is that you would need 
to either include a D compiler with your app, or expect the user 
to have one installed on their system. Note that DMD is not 
redistributable (you can't include DMD with your program), but 
this doesn't apply to LDC/GDC.

This might also get tricky on Windows, as you'll probably need to 
create or generate a .DEF file (and accompanying import library) 
which lists all exports/imports shared between your program and 
plugins. This will include all class methods as mangled names.

D compile times are quite low already, so I don't think this 
would be an issue.

2. Use a scripting language, e.g. Lua

D's compile-time reflection and code-generation abilities 
certainly make it possible to expose functions and classes to 
scripts. You could add a mixin to an exposed class, which would 
generate all the necessary run-time type information to expose 
the class to the scripting language. I don't know if any existing 
library already provides this in an easy-to-use manner, though.

Perhaps a good starting point would be LuaD:
https://github.com/JakobOvrum/LuaD



 Oct 06 2015