www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Dynamic binding to the Mono runtime API

reply Jakub Szewczyk <kubasz51 gmail.com> writes:
Mono runtime is a cross-platform, open-source alternative to 
Microsoft's .NET framework [1], and it can be embedded in other 
applications as a "scripting" VM, but with JIT-compilation 

or IronPython [2].
It provides a C API, so I've bound it to D as a Derelict-based 
project, available at https://github.com/kubasz/derelict-mono, 
and as a DUB package 
(http://code.dlang.org/packages/derelict-mono). It currently 
wraps the Mono 5.0 API.



PS: Because I don't own a Mac I have no idea what the correct 
paths to the Mono shared library are, so it'd be great if someone 
could post/create a PR of them.

[1] http://www.mono-project.com/
[2] http://www.mono-project.com/docs/advanced/embedding/scripting/
Jun 03 2017
next sibling parent reply extrawurst <stephan extrawurst.org> writes:
On Saturday, 3 June 2017 at 17:30:05 UTC, Jakub Szewczyk wrote:

 Mono runtime is a cross-platform, open-source alternative to 
 Microsoft's .NET framework [1], and it can be embedded in other 
 applications as a "scripting" VM, but with JIT-compilation 


 It provides a C API, so I've bound it to D as a Derelict-based 
 project, available at https://github.com/kubasz/derelict-mono, 
 and as a DUB package 
 (http://code.dlang.org/packages/derelict-mono). It currently 
 wraps the Mono 5.0 API.



 [...]


Cool work! Thanks! Did u use dstep for the translation or manual 
labor?

I just checked on my mac and it works from the get go!

Cheers,
Stephan
Jun 03 2017
parent Jacob Carlborg <doob me.com> writes:
On 2017-06-03 23:44, extrawurst wrote:

 On Saturday, 3 June 2017 at 17:30:05 UTC, Jakub Szewczyk wrote:

 Mono runtime is a cross-platform, open-source alternative to
 Microsoft's .NET framework [1], and it can be embedded in other
 applications as a "scripting" VM, but with JIT-compilation enhanced

 [2].
 It provides a C API, so I've bound it to D as a Derelict-based
 project, available at https://github.com/kubasz/derelict-mono, and as
 a DUB package (http://code.dlang.org/packages/derelict-mono). It
 currently wraps the Mono 5.0 API.

 code calling a native function implemented in D.

 [...]


 Cool work! Thanks! Did u use dstep for the translation or manual labor?


I'm guessing no, since these bindings are using function pointers while 
DStep will translate function declarations to function declarations in D 
as well, not function pointers.

-- 
/Jacob Carlborg
Jun 04 2017
prev sibling next sibling parent Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Saturday, 3 June 2017 at 17:30:05 UTC, Jakub Szewczyk wrote:

 Mono runtime is a cross-platform, open-source alternative to 
 Microsoft's .NET framework [1], and it can be embedded in other 
 applications as a "scripting" VM, but with JIT-compilation 


 It provides a C API, so I've bound it to D as a Derelict-based 
 project, available at https://github.com/kubasz/derelict-mono, 
 and as a DUB package 
 (http://code.dlang.org/packages/derelict-mono). It currently 
 wraps the Mono 5.0 API.



 PS: Because I don't own a Mac I have no idea what the correct 
 paths to the Mono shared library are, so it'd be great if 
 someone could post/create a PR of them.

 [1] http://www.mono-project.com/
 [2] 
 http://www.mono-project.com/docs/advanced/embedding/scripting/



This is very cool - thank you for doing this.  It could prove 
very helpful.

Have you thought of/any interest in looking at automatically 

myself mostly in nested templated structs/arrays rather than 
classes).  It may not be relevant for you, but if it is please 
drop me an email on laeeth

... at kaleidic.io

Thanks.


Laeeth.
Jun 03 2017
prev sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On 6/3/17 10:30, Jakub Szewczyk wrote:

 Mono runtime is a cross-platform, open-source alternative to Microsoft's
 .NET framework [1], and it can be embedded in other applications as a
 "scripting" VM, but with JIT-compilation enhanced performance and

 It provides a C API, so I've bound it to D as a Derelict-based project,
 available at https://github.com/kubasz/derelict-mono, and as a DUB
 package (http://code.dlang.org/packages/derelict-mono). It currently
 wraps the Mono 5.0 API.

 code calling a native function implemented in D.

 PS: Because I don't own a Mac I have no idea what the correct paths to
 the Mono shared library are, so it'd be great if someone could
 post/create a PR of them.

 [1] http://www.mono-project.com/
 [2] http://www.mono-project.com/docs/advanced/embedding/scripting/



Thank you for this!

I've looked over the code a bit and I have a couple of questions.
This appears to be an interface to the runtime itself, not a BCL 
interface correct?
It looks like this could be used to could this be used to read into a 
Mono Class Libraries, and if so would so some sort of automated code 
generation tool be required? It looks to me like the binding will be 
non-trivial, with GC, exceptions, etc. that all need to be handled at 
the call-site.
Can we get a static library version of this, or is there a dependency on 
dynamic libraries?

I have to admit I am very impressed. I have spent a lot of time building 
code generators before and I have to admit that the idea of binding to 
arbitrary .NET libraries via code generation is extremely appealing to 
me. I am seriously tempted to take this and start building a binding 
generator...

I seriously need more free time! Way too many cool and useful things 
happening in D for my limited free time. A D binding for XAML ... THAT 
would sight to behold!

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Jun 03 2017
parent reply Jakub Szewczyk <kubasz51 gmail.com> writes:
On Sunday, 4 June 2017 at 05:47:32 UTC, Adam Wilson wrote:

 On 6/3/17 10:30, Jakub Szewczyk wrote:

 ...



 work. Thank you for this!


Thank you all!


 I've looked over the code a bit and I have a couple of 
 questions.
 This appears to be an interface to the runtime itself, not a 
 BCL interface correct?
 It looks like this could be used to could this be used to read 
 into a Mono Class Libraries, and if so would so some sort of 
 automated code generation tool be required? It looks to me like 
 the binding will be non-trivial, with GC, exceptions, etc. that 
 all need to be handled at the call-site.
 Can we get a static library version of this, or is there a 
 dependency on dynamic libraries?

 I have to admit I am very impressed. I have spent a lot of time 
 building code generators before and I have to admit that the 
 idea of binding to arbitrary .NET libraries via code generation 
 is extremely appealing to me. I am seriously tempted to take 
 this and start building a binding generator...

 I seriously need more free time! Way too many cool and useful 
 things happening in D for my limited free time. A D binding for 
 XAML ... THAT would sight to behold!


This is an interface to the Mono libraries, D/CLI would require 
quite a lot of compiler changes, both on the front-end and 
back-end side, but thanks to metaprogramming a wrapper library 
can get very close to such an interface.
I plan on making an automated D to Mono bridge, akin to LuaD, so 
that it can be used as a scripting platform for e.g. games. I 
haven't thought about doing it the other way around, but now it 
seems like a very interesting idea! Unfortunately no XAML 
(http://www.mono-project.com/docs/gui/wpf/), but many other 
libraries, such as XWT (https://github.com/mono/xwt) could be 
ported this way, I'll certainly look into it.

Mono actually supports some kind of GC bridging as far as I 
understand, as there is a sgen-bridge.h header just for that, and 

Android. As for exceptions - D functions have to be wrapped in a 
nothrow wrapper, but that can be completely automated with 
templates. The other way around is also quite simple - when 
invoking a Mono function a pointer can be given that will set an 
exception reference if one is thrown from the .NET side, and a 
wrapper can easily rethrow it back to D.

I can make a static library version, it's just some regex 
substitutions done on the functions file actually, most probably 
I'll publish it today. It will be a dub subconfiguration like it 
is the case for GLFW3. Btw, I've manually ported the basic and 
configuration headers, so that no mistakes are made, and then 
used DStep and a modified DStep to generate the rest of the 
headers - my modification was only to change the way function 
declarations are generated, to make them in derelict form of 
alias da_function = void function(...);, and the rest was done 
with quite a lot of editor(VSCode) shortcuts.
Jun 04 2017
next sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On 6/4/17 01:18, Jakub Szewczyk wrote:

 This is an interface to the Mono libraries, D/CLI would require quite a
 lot of compiler changes, both on the front-end and back-end side, but
 thanks to metaprogramming a wrapper library can get very close to such
 an interface.
 I plan on making an automated D to Mono bridge, akin to LuaD, so that it
 can be used as a scripting platform for e.g. games. I haven't thought
 about doing it the other way around, but now it seems like a very
 interesting idea! Unfortunately no XAML
 (http://www.mono-project.com/docs/gui/wpf/), but many other libraries,
 such as XWT (https://github.com/mono/xwt) could be ported this way, I'll
 certainly look into it.


My interest is less in code ports than bindings to the actual code. My 
experience with code ports or translations is that often subtle bugs 
creep in during translation due to the fact that each language has 
different idioms.

What I am thinking about is a tool that loads an assembly, examines it's 
types and methods via this API and emits D code that directly interfaces 
into the .NET types via this API. The tricky part here is mapping the 
.NET dependencies into D. The moment the library exposes a type from a 
dependency, that dependency ALSO needs to be included somehow. All 
libraries reference "mscorlib", AKA the BCL, so we'd have to provide a 
"mono-bcl" package on DUB.

One solution is to simply include the exposed dependency in the 
generated code. This would work because while the D code would have 
distinct types for the same class, the underlying .NET types is the 
same. The drawback with this approach is that you can't share the 
instances across D interfaces as the types are different. For example, 
Library1.B and Library2.C both rely on Dependency.A. In D you would have 
Library1.B.A and Library2.C.A and these two types, while the same in 
practice, are different types to the compiler. Maybe a clever use of 
alias can solve this at code-gen time... more research is required there.

The other solution is too have the code-gen throw an error when it 
encounters a type from a dependency. This ensures that the types are the 
same across libraries in D, but at the cost of increased complexity for 
the developer when running the tool (specifying extra deps) and when 
building their code (ensuring all relevant packages are built/referenced).

I'm not sold on either one. And it'd be best if it's possible to support 
both somehow.

As for the WPF remark, my brain immediately jump to trying to interface 
to .NET/Core using a similar mechanism, but alas some research time 
indicates that this is not possible outside of COM. *le sigh* My 
apologize for the random side-track.


 Mono actually supports some kind of GC bridging as far as I understand,
 as there is a sgen-bridge.h header just for that, and it has apparently

 - D functions have to be wrapped in a nothrow wrapper, but that can be
 completely automated with templates. The other way around is also quite
 simple - when invoking a Mono function a pointer can be given that will
 set an exception reference if one is thrown from the .NET side, and a
 wrapper can easily rethrow it back to D.


On the GC side I was mostly thinking about GC Handles so that the 
objects don't get collected out from underneath us. That is something is 
trivial to code-gen.

As for exceptions, I like the catch->translate->rethrow mechanism. And 
if the exception is unknown we could simply throw a generic exception. 
The important thing is to get close to the D experience, not try to map 
it perfectly.


 I can make a static library version, it's just some regex substitutions
 done on the functions file actually, most probably I'll publish it
 today. It will be a dub subconfiguration like it is the case for GLFW3.
 Btw, I've manually ported the basic and configuration headers, so that
 no mistakes are made, and then used DStep and a modified DStep to
 generate the rest of the headers - my modification was only to change
 the way function declarations are generated, to make them in derelict
 form of alias da_function = void function(...);, and the rest was done
 with quite a lot of editor(VSCode) shortcuts.


Thank you for this! I find static libraries easier to deal with. I'm 
sure other people have differing opinions, so having both would make 
everyone happy.

I am very excited about this!

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Jun 04 2017
parent reply Jakub Szewczyk <kubasz51 gmail.com> writes:
On Sunday, 4 June 2017 at 09:43:23 UTC, Adam Wilson wrote:

 On 6/4/17 01:18, Jakub Szewczyk wrote:

 This is an interface to the Mono libraries, D/CLI would [...]


 My interest is less in code ports than bindings to the actual 
 code. My experience with code ports or translations is that 
 often subtle bugs creep in during translation due to the fact 
 that each language has different idioms.

 What I am thinking about is a tool that loads an assembly, 
 examines it's types and methods via this API and emits D code 
 that directly interfaces into the .NET types via this API. The 
 tricky part here is mapping the .NET dependencies into D. The 
 moment the library exposes a type from a dependency, that 
 dependency ALSO needs to be included somehow. All libraries 
 reference "mscorlib", AKA the BCL, so we'd have to provide a 
 "mono-bcl" package on DUB.


That's what I actually meant, "porting" was a misused term on my 
part, "binding" would be a better word, sorry for that.
As for the dependency problem - I think that a linking layer 
generator would accept a list of input assemblies (and 
optionally, specific classes) to which it should generate 
bindings, the core Mono types could be automatically translated 
to D equivalents, and the rest could be left as an opaque 
reference, like MonoObject* in C, also providing support for very 
basic reflection through the Mono methods if it turned out to be 
useful for anyone.


 Mono actually supports some kind of GC bridging as far as I 
 understand, [...]


 On the GC side I was mostly thinking about GC Handles so that 
 the objects don't get collected out from underneath us. That is 
 something is trivial to code-gen.

 As for exceptions, I like the catch->translate->rethrow 
 mechanism. And if the exception is unknown we could simply 
 throw a generic exception. The important thing is to get close 
 to the D experience, not try to map it perfectly.


Yes, GCHandles to keep Mono objects in D and a wrapper based on 
that GC bridge to keep D references from being collected by Mono. 
I have previously implemented a very similar mechanism for Lua in 
a small wrapper layer, and it worked perfectly.


 I can make a static library version, [...]


 Thank you for this! I find static libraries easier to deal 
 with. I'm sure other people have differing opinions, so having 
 both would make everyone happy.


It's now public as v1.1.0, I've tested that it works with the 
tiny sample, the only important part is that the library to link 
must be specified by the project using this binding, because 
those paths may vary across systems, and they cannot be specified 
in code like the dynamic link ones. However, a simple 
"libs":["mono-2.0"] entry in dub.json should be enough for most 
use cases.
Jun 04 2017
parent Adam Wilson <flyboynw gmail.com> writes:
On 6/4/17 04:15, Jakub Szewczyk wrote:

 On Sunday, 4 June 2017 at 09:43:23 UTC, Adam Wilson wrote:

 On 6/4/17 01:18, Jakub Szewczyk wrote:

 This is an interface to the Mono libraries, D/CLI would [...]


 My interest is less in code ports than bindings to the actual code. My
 experience with code ports or translations is that often subtle bugs
 creep in during translation due to the fact that each language has
 different idioms.

 What I am thinking about is a tool that loads an assembly, examines
 it's types and methods via this API and emits D code that directly
 interfaces into the .NET types via this API. The tricky part here is
 mapping the .NET dependencies into D. The moment the library exposes a
 type from a dependency, that dependency ALSO needs to be included
 somehow. All libraries reference "mscorlib", AKA the BCL, so we'd have
 to provide a "mono-bcl" package on DUB.


 That's what I actually meant, "porting" was a misused term on my part,
 "binding" would be a better word, sorry for that.
 As for the dependency problem - I think that a linking layer generator
 would accept a list of input assemblies (and optionally, specific
 classes) to which it should generate bindings, the core Mono types could
 be automatically translated to D equivalents, and the rest could be left
 as an opaque reference, like MonoObject* in C, also providing support
 for very basic reflection through the Mono methods if it turned out to
 be useful for anyone.


 Mono actually supports some kind of GC bridging as far as I
 understand, [...]


 On the GC side I was mostly thinking about GC Handles so that the
 objects don't get collected out from underneath us. That is something
 is trivial to code-gen.

 As for exceptions, I like the catch->translate->rethrow mechanism. And
 if the exception is unknown we could simply throw a generic exception.
 The important thing is to get close to the D experience, not try to
 map it perfectly.


 Yes, GCHandles to keep Mono objects in D and a wrapper based on that GC
 bridge to keep D references from being collected by Mono. I have
 previously implemented a very similar mechanism for Lua in a small
 wrapper layer, and it worked perfectly.


 I can make a static library version, [...]


 Thank you for this! I find static libraries easier to deal with. I'm
 sure other people have differing opinions, so having both would make
 everyone happy.


 It's now public as v1.1.0, I've tested that it works with the tiny
 sample, the only important part is that the library to link must be
 specified by the project using this binding, because those paths may
 vary across systems, and they cannot be specified in code like the
 dynamic link ones. However, a simple "libs":["mono-2.0"] entry in
 dub.json should be enough for most use cases.


Thank you for this, I've tried it and it works!

I did some in depth research and prototyping in D, and it looks like the 
only way to enumerate the types in an assembly is to use the Metadata 
Table API's map everything that way. That's a little beyond the scope of 
my free time so I'll have to shelve the idea from now. :(

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Jun 07 2017
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-06-04 10:18, Jakub Szewczyk wrote:


 Btw, I've manually ported the basic and configuration headers, so that
 no mistakes are made, and then used DStep and a modified DStep to
 generate the rest of the headers - my modification was only to change
 the way function declarations are generated, to make them in derelict
 form of alias da_function = void function(...);, and the rest was done
 with quite a lot of editor(VSCode) shortcuts.


Would you like to contribute your changes under a flag to DStep?

-- 
/Jacob Carlborg
Jun 04 2017
parent reply Jakub Szewczyk <kubasz51 gmail.com> writes:
On Sunday, 4 June 2017 at 15:43:09 UTC, Jacob Carlborg wrote:

 On 2017-06-04 10:18, Jakub Szewczyk wrote:


 Btw, I've manually ported the basic and configuration headers, 
 so that
 no mistakes are made, and then used DStep and a modified DStep 
 to
 generate the rest of the headers - my modification was only to 
 change
 the way function declarations are generated, to make them in 
 derelict
 form of alias da_function = void function(...);, and the rest 
 was done
 with quite a lot of editor(VSCode) shortcuts.


 Would you like to contribute your changes under a flag to DStep?


The problem is, it emits completely wrong code whereever a 
function is
necessary, like in function pointers. I can try to isolate the 
change to
global-level function declarations only, to make it generate 
correct
code that doesn't require running two versions of DStep in 
parallel to
extract all the information, when I do that I'll submit a PR.
Jun 04 2017
parent Jacob Carlborg <doob me.com> writes:
On 2017-06-04 20:19, Jakub Szewczyk wrote:


 The problem is, it emits completely wrong code whereever a function is
 necessary, like in function pointers. I can try to isolate the change to
 global-level function declarations only, to make it generate correct
 code that doesn't require running two versions of DStep in parallel to
 extract all the information, when I do that I'll submit a PR.


Cool, that would be great.

-- 
/Jacob Carlborg
Jun 04 2017