www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Idea to make D great again!

reply PiSquared <PiSquared gmail.com> writes:
D has the ability to embed most languages in to it through 
scripting. There is matplotlibD already, I have modified the code 
to be able to run arbitrary scripts and also work with Lua. Any 
language can be used through the method of:

1. Writing a D string representing the target language's program 
to a file or stdin of the interpreter.

2. Before executing above, first modify the input to replace 
special tokens with D data. This allows one to insert D data in 
to the target program. This allows one to use D to generate the 
data which may be much faster and can be used in other ways.

[See below for an example]

These methods should work with all languages since it simply 
emulates coding manually. It becomes a powerful feature because 
one and leverage the target language as if one was writing 
directly in the languages.

More importantly, Visual D and Visual Studio allow one to load 
many target languages in to a D project and get almost full 
benefit such as intelligence(debugging does not work but I 
imagine it could with some work).


Furthermore, with some work, interopt can be made to work really 
well by transferring data between the programs using a client 
server protocol which can be done through files or global 
memory(depending on the languages level of OS access).


With the proper design 1. D can be used along side and as a host 
for most languages. The most common will work fine. With a bit of 
work it should be work extremely well. This allows D to be 
leverage and piggy back on these languages and will bring users 
of those languages to D. D can be used for what it is good at and 
the holes it has can be filled by some other language. 2. The 
design is not specific to D as most languages have this ability, 
but with D it generally can be quite easy.

The only downside is that of "marshaling" the data, this can be 
reduced significantly by not using text to transfer data but by 
having appropriate data marshalers that can work in memory and 
relate data between D and the target language correctly.


Further more, the target language needs some capacity too send 
data back to the D program.

I have created a server and a client that do this. It is used for 
writing latex code in D. This allows me to also use python's 
scientific plotting libraries along with tikz by using a common D 
library. Tikz is slow, D+mlab is fast.

The latex code, for example, can send data back to the server 
using a client proxy and this can trigger different results when 
the latex file is being compiled by latex.

It is all quite simple but it allows me to debug D programs in D 
without issue as all the plumbing on the target language is 
relatively hidden.


I would like you guys to consider taking this proposal seriously: 
Create a proper D design that allows one to leverage any target 
language in a unified way. Make it robust, powerful, feature rich.

It will bring many people to use D if done properly. With a 
proper IDE D could act as a centerpiece to use many languages 
together.

One could use a gui in lua, a plotting library in python, and 
latex dox generation.

All done rather seamless.



here is a lua target, bare bones with no fancy features and 
essentially ripped from matplotlibD(which essentially does the 
same with python, of which I have a version that is essentially 
the same as below):


module LuaD;
import std.stdio;

enum DDataToken = `" D$`;

alias immutable bool LuaBool;
alias immutable (void*) LuaNone;

LuaBool False = false;
LuaBool True = true;
LuaNone None = null;

// A header to use for common lua files
string luaHeader = `
`;

string luaFooter = ``;


string d2lua(T)(T v)
{

	int x = 3242;
     import std.conv, std.traits, std.array, std.algorithm, 
std.format: format;
	static if (is(T : LuaNone))
		return "None";

     else static if (is(T : bool))
         return v ? "True" : "False";

     else static if (is(T : string))
         return format("\"%s\"", v);

     else
	{
		static if (isArray!T)
			return v.map!(a=>to!string(a)~",").join;

         return format("%s", v);
	}
}



string lua_path = `C:\Lua\lua.exe`;

// executes lua on the program
void lua(string program)
{	

     import std.process: environment, pipeProcess, wait, Redirect;

	if (!lua_path)
		lua_path =  environment.get("LuaD", "lua");
	
     auto pipes = pipeProcess(lua_path, Redirect.stdin | 
Redirect.stderr);

     pipes.stdin.writeln(program);
     pipes.stdin.writeln("exit()");
     pipes.stdin.close();

     auto result = wait(pipes.pid);

     if (result != 0) {
         string error;
         foreach (line; pipes.stderr.byLine)
             error ~= line ~ "\n";
         throw new Exception("\n\nERROR occurred in Lua:\n" ~ 
error);
     }
}



// converts all " $..." strings to D variables by converting
auto luaD(bool ret = false)(string program)
{	

	if (program.length < 5) return "";

	import std.string, std.range, std.algorithm;


	string r;
	for(int i = 0; i < program.length; i++)
	{
		if (i < program.length - 5)
		{
			if (program[i..i+DDataToken.length] == DDataToken)
			{
				i += DDataToken.length;
				string id;
				while(i < program.length && program[i] != '"')
					id ~= program[i++];

				r ~= `"~d2lua(`~id~`)~"`;
				continue;
			}
		}
		if (program[i] == '"') r ~= "\\";
		r ~= program[i];
	}

	static if (ret) { return `"`~r~`"`; }

	return `lua("`~r~`");`;
}



Then a simple lua program `test.lua`:

     print(" D$TestString")



one calls this program in D:

     auto TestString = "This is a D string";

     // Will automatically substitute TestString
     mixin(luaD(import(test.lua));





and the lua program will be executed with the D data.


I have used such things for computing data in D and then plotting 
them using some of pythons plotting libraries(matplotlib, mlab, 
mayavi, etc).


It is not as optimal since large datasets can cause problems 
since they can take up so much textual space but this can be 
alleviated by writing them to binary files, or ideally, using 
memory transfers.


With a good robust design D can be the centerpiece of language 
hosting bringing a very large number of users. Python, for some 
odd reason, has a lot of scientific plotting libraries but, as 
most people know, is not very fast. Imagine using D for the 
algorithms and python for the plotting. That is precisely what I 
have done and it works well(debugging errors is the hardest since 
no IDE but that could change). In visual D I get full syntax 
coloring, intellisense, and documentation for python and lua.

Imagine F#, C#, vbs, C++, Haskell, even assembly and many other 
languages being usable with this method. With a good "marshaling" 
library it one would could have limited performance degradation. 
For apps that do not have performance issues, say typical gui's 
one can use a language with good gui frameworks and D for the 
business end. With D's meta capabilities one could probably 
export most of the bridging automatically in to the gui.


Done well it could be a boon to D and bring in many new people 
and $$$. As far as I'm aware, no language or framework exists to 
do this well and easily yet it is not a difficult problem.

Also it lets a person become "multi-lingual" in real time. I know 
quite a few languages but rarely use them because I typically use 
only one at a time unless required.  But being able to 
essentially choose the best language for the job would be nice.

Diet templates sort of approach the idea the same but for html. 
The above approach could allow embedding D code in to other 
languages too by having "code strings".

By having a universal design it could be done quite effectively. 
I've already been able to do far more in D by leveraging python 
than I could do otherwise... but by leveraging I mean having D 
automatically transfer in to the python program rather than 
having to do complicated coding.


My code is not robust as it is proof of concept and meets my 
limited needs. I'd like to see a more robust framework for doing 
this... I'd do it myself but I simply do not have the time to do 
it all and make it as good as it should be.
Oct 06
next sibling parent reply Chris <wendlec tcd.ie> writes:
On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:
 My code is not robust as it is proof of concept and meets my 
 limited needs. I'd like to see a more robust framework for 
 doing this... I'd do it myself but I simply do not have the 
 time to do it all and make it as good as it should be.
Would you have time to put that on Github with a few examples? Maybe someone could pick it up from there.
Oct 08
parent PiSquared <PiSquared gmail.com> writes:
On Tuesday, 8 October 2019 at 10:02:08 UTC, Chris wrote:
 On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:
 My code is not robust as it is proof of concept and meets my 
 limited needs. I'd like to see a more robust framework for 
 doing this... I'd do it myself but I simply do not have the 
 time to do it all and make it as good as it should be.
Would you have time to put that on Github with a few examples? Maybe someone could pick it up from there.
I don't have the code in an organized state yet to make it valuable. What I posted is pretty much what I have(I just have the python version and some server/client stuff but is part of something separate and not finished). I might get around to it at some point but it will be slow. You should be able to take that code, and on windows at least, get it to work pretty easily. My main goal would be to be able to optimize memory transfer and enable some type of bi-directional interaction that is very simple but fast and robust. I just haven't though about it enough. I don't seem much interest in it since you are the only one who responded so I guess the D community doesn't really care about such things.
Oct 09
prev sibling parent Laeeth Isharc <Laeeth laeeth.com> writes:
On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:
 D has the ability to embed most languages in to it through 
 scripting. There is matplotlibD already, I have modified the 
 code to be able to run arbitrary scripts and also work with 
 Lua. Any language can be used through the method of:

 1. Writing a D string representing the target language's 
 program to a file or stdin of the interpreter.

 2. Before executing above, first modify the input to replace 
 special tokens with D data. This allows one to insert D data in 
 to the target program. This allows one to use D to generate the 
 data which may be much faster and can be used in other ways.

 [See below for an example]

 These methods should work with all languages since it simply 
 emulates coding manually. It becomes a powerful feature because 
 one and leverage the target language as if one was writing 
 directly in the languages.

 More importantly, Visual D and Visual Studio allow one to load 
 many target languages in to a D project and get almost full 
 benefit such as intelligence(debugging does not work but I 
 imagine it could with some work).


 Furthermore, with some work, interopt can be made to work 
 really well by transferring data between the programs using a 
 client server protocol which can be done through files or 
 global memory(depending on the languages level of OS access).


 With the proper design 1. D can be used along side and as a 
 host for most languages. The most common will work fine. With a 
 bit of work it should be work extremely well. This allows D to 
 be leverage and piggy back on these languages and will bring 
 users of those languages to D. D can be used for what it is 
 good at and the holes it has can be filled by some other 
 language. 2. The design is not specific to D as most languages 
 have this ability, but with D it generally can be quite easy.

 The only downside is that of "marshaling" the data, this can be 
 reduced significantly by not using text to transfer data but by 
 having appropriate data marshalers that can work in memory and 
 relate data between D and the target language correctly.


 Further more, the target language needs some capacity too send 
 data back to the D program.

 I have created a server and a client that do this. It is used 
 for writing latex code in D. This allows me to also use 
 python's scientific plotting libraries along with tikz by using 
 a common D library. Tikz is slow, D+mlab is fast.

 The latex code, for example, can send data back to the server 
 using a client proxy and this can trigger different results 
 when the latex file is being compiled by latex.

 It is all quite simple but it allows me to debug D programs in 
 D without issue as all the plumbing on the target language is 
 relatively hidden.


 I would like you guys to consider taking this proposal 
 seriously: Create a proper D design that allows one to leverage 
 any target language in a unified way. Make it robust, powerful, 
 feature rich.

 It will bring many people to use D if done properly. With a 
 proper IDE D could act as a centerpiece to use many languages 
 together.

 One could use a gui in lua, a plotting library in python, and 
 latex dox generation.

 All done rather seamless.



 here is a lua target, bare bones with no fancy features and 
 essentially ripped from matplotlibD(which essentially does the 
 same with python, of which I have a version that is essentially 
 the same as below):


 module LuaD;
 import std.stdio;

 enum DDataToken = `" D$`;

 alias immutable bool LuaBool;
 alias immutable (void*) LuaNone;

 LuaBool False = false;
 LuaBool True = true;
 LuaNone None = null;

 // A header to use for common lua files
 string luaHeader = `
 `;

 string luaFooter = ``;


 string d2lua(T)(T v)
 {

 	int x = 3242;
     import std.conv, std.traits, std.array, std.algorithm, 
 std.format: format;
 	static if (is(T : LuaNone))
 		return "None";

     else static if (is(T : bool))
         return v ? "True" : "False";

     else static if (is(T : string))
         return format("\"%s\"", v);

     else
 	{
 		static if (isArray!T)
 			return v.map!(a=>to!string(a)~",").join;

         return format("%s", v);
 	}
 }



 string lua_path = `C:\Lua\lua.exe`;

 // executes lua on the program
 void lua(string program)
 {	

     import std.process: environment, pipeProcess, wait, 
 Redirect;

 	if (!lua_path)
 		lua_path =  environment.get("LuaD", "lua");
 	
     auto pipes = pipeProcess(lua_path, Redirect.stdin | 
 Redirect.stderr);

     pipes.stdin.writeln(program);
     pipes.stdin.writeln("exit()");
     pipes.stdin.close();

     auto result = wait(pipes.pid);

     if (result != 0) {
         string error;
         foreach (line; pipes.stderr.byLine)
             error ~= line ~ "\n";
         throw new Exception("\n\nERROR occurred in Lua:\n" ~ 
 error);
     }
 }



 // converts all " $..." strings to D variables by converting
 auto luaD(bool ret = false)(string program)
 {	

 	if (program.length < 5) return "";

 	import std.string, std.range, std.algorithm;


 	string r;
 	for(int i = 0; i < program.length; i++)
 	{
 		if (i < program.length - 5)
 		{
 			if (program[i..i+DDataToken.length] == DDataToken)
 			{
 				i += DDataToken.length;
 				string id;
 				while(i < program.length && program[i] != '"')
 					id ~= program[i++];

 				r ~= `"~d2lua(`~id~`)~"`;
 				continue;
 			}
 		}
 		if (program[i] == '"') r ~= "\\";
 		r ~= program[i];
 	}

 	static if (ret) { return `"`~r~`"`; }

 	return `lua("`~r~`");`;
 }



 Then a simple lua program `test.lua`:

     print(" D$TestString")



 one calls this program in D:

     auto TestString = "This is a D string";

     // Will automatically substitute TestString
     mixin(luaD(import(test.lua));





 and the lua program will be executed with the D data.


 I have used such things for computing data in D and then 
 plotting them using some of pythons plotting 
 libraries(matplotlib, mlab, mayavi, etc).


 It is not as optimal since large datasets can cause problems 
 since they can take up so much textual space but this can be 
 alleviated by writing them to binary files, or ideally, using 
 memory transfers.


 With a good robust design D can be the centerpiece of language 
 hosting bringing a very large number of users. Python, for some 
 odd reason, has a lot of scientific plotting libraries but, as 
 most people know, is not very fast. Imagine using D for the 
 algorithms and python for the plotting. That is precisely what 
 I have done and it works well(debugging errors is the hardest 
 since no IDE but that could change). In visual D I get full 
 syntax coloring, intellisense, and documentation for python and 
 lua.

 Imagine F#, C#, vbs, C++, Haskell, even assembly and many other 
 languages being usable with this method. With a good 
 "marshaling" library it one would could have limited 
 performance degradation. For apps that do not have performance 
 issues, say typical gui's one can use a language with good gui 
 frameworks and D for the business end. With D's meta 
 capabilities one could probably export most of the bridging 
 automatically in to the gui.


 Done well it could be a boon to D and bring in many new people 
 and $$$. As far as I'm aware, no language or framework exists 
 to do this well and easily yet it is not a difficult problem.
Yes, and for example in C# and F# I think Roslyn could take you a long way since you can compile dotnet code at D runtime. As is well known, C++ is an interpreted language where you can instantiate templates at runtime. I'm joking, but it can be if you want it to be and CERN do this at considerable scale with Cling, which allows you to instantiate templates at runtime and introspect, create instance of classes and call them. Some interesting early results in calling cling from D, but I was waiting till things were a bit more polished to talk about them. The next stage might be to run cling on clang to help with dpp development. Libclang looks amazing at first sight but it is missing a lot. I think there are two reasons why interoperability matters. One is external libraries but another is that when you are exploring using D then it's often a bit challenging if you have C++ code and need to use it or implement part of a new feature in D. If replacing working code you don't ideally want to do a complete rewrite but replace small pieces at a time and have working code all along the way. That might not be easy in many cases. I think RoI is often a less important barrier than limits to upfront pain. If you use D you are already an unusual sort of person because the discomfort is front loaded and how do you know it will be worth it to persevere? Atila downloaded the compiler and back then it segfaulted when he tried to compile hello world! Most reasonable people would have given up then. But the dollar return on being reasonable is time varying and for some people at some times it's not necessarily a bad thing to hire people who are a bit unreasonable. Perhaps all progress depends on people not being reasonable. So if it becomes easier to use libraries and internal code written in other languages then it's easy to make small beginnings. Lots of large projects begin as little beginnings and so making it easy to start work has value for the ecosystem I think. I agree that sometimes you don't need to do things that are that clever to be quite productive. But one probably can do a bit better than what you suggest even if for any individual it's more than good enough. It's almost easy to use python, C, Lua,R libraries from D but not quite. Pyd is showing its age right now and LuaD isn't that well maintained. It's not super easy to find embedr and it's not trivially easy to get up and running. I think lots of that is documentation, maintenance,small improvements and blog posts. Dpp is pretty solid a lot of the time. On Linux anyway but on Windows not yet, judging by a conversation I had yesterday. I asked Adam Ruppe to look into that and I think in time it will just work for what it works for. It's a shame it doesn't dependably yet at least handle Linux headers without choking. I think that's an SAoC project if I recall. We could do a bit more to programmatically just try wrapping every header we can get our hands on and then try to programmatically analyse the results. Ultimately there is a limit with macros, maybe with using libclang and maybe how far you can get with generating strings in D rather than AST to AST transformation. The latter for C isn't a trivial project - DoD spent millions on C2Rust though on the other hand they are not going to be able to do things on the cheap. C to D ought to be quite doable though. Our focus has been more on the cpp side for now but in time could do a bit more with C. With PyD under autowrap there is a project called PyNIH. Steady progress with focus first on Python calling D but that helps with the reverse. Embedding C# in D has stalled a bit - see dflat and a project to do so via Mono also. It's not difficult work technically but just a bit of a grind. We did some very basic work on embedding Julia in D. That also isn't difficult but time is limited. The basic idea is to use Symmetry Integration Language as a substrate where you can write plugins in any sensible language and call it from any sensible language for us. We have to solve our own problems first but in the process of doing so probably it will help others. One more thing - work Sebastian Koppe did shows a proof of concept of D compiling to web assembly and using Javascript libraries. He will generate the wrapping code from Typescript bindings in time. From work I did before I don't think embedding Java would be so difficult but we just don't use Java that much. So I think the biggest constraint right now is the ability to organise and direct the work in some cases (like C#) and in others for other people to submit improvements. But in time I think interoperability will just keep getting better.
Oct 09