www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Allocating Executable Memory

reply "Maxime Chevalier" <maximechevalierb gmail.com> writes:
New to the D language here. I'm working on a tracing JIT compiler 
and will need to allocate chunks of memory that are marked as 
executable.

Is there a standard way of doing this in D, or do I need to 
directly call into mprotect? If I'm going to be calling mprotect, 
what's the cleanest way to do that, do I need to declare my own 
prototype for the function and its flags, or should I write some 
C code that does the call?

Thanks for your help.
Jul 31 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 31-07-2012 19:23, Maxime Chevalier wrote:

 New to the D language here. I'm working on a tracing JIT compiler and
 will need to allocate chunks of memory that are marked as executable.

 Is there a standard way of doing this in D, or do I need to directly
 call into mprotect? If I'm going to be calling mprotect, what's the
 cleanest way to do that, do I need to declare my own prototype for the
 function and its flags, or should I write some C code that does the call?

 Thanks for your help.


There's no standard way to do it. You'll have to either use mprotect or 
just pass the relevant flags when you call mmap.

The functions and constants related to this are all in core.sys.posix.mman.

See also:

* https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71 
(and further down)
* https://github.com/lycus/mci/blob/master/src/mci/vm/code.d

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org
Jul 31 2012
parent reply "Maxime Chevalier" <maximechevalierb gmail.com> writes:
 The functions and constants related to this are all in 
 core.sys.posix.mman.

 See also:

 * 
 https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71 
 (and further down)
 * https://github.com/lycus/mci/blob/master/src/mci/vm/code.d


Thanks. Forgive my ignorance though, but does 
core.sys.posix.sys.mman ship standard with D compilers?
Jul 31 2012
next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
On Tue, Jul 31, 2012 at 9:36 PM, Maxime Chevalier <
maximechevalierb gmail.com> wrote:


 The functions and constants related to this are all in core.sys.posix.mman.

 See also:

 * https://github.com/lycus/mci/**blob/master/src/mci/core/**memory.d#L71<https://github.com/lycus/mci/blob/master/src/mci/co
e/memory.d#L71>(and further down)
 * https://github.com/lycus/mci/**blob/master/src/mci/vm/code.d<https://github.com/lycus/mci/blob/master/src/mci/vm/code.d>


 Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship
 standard with D compilers?


It's part of druntime. Druntime is D's runtime library, without which D
code above version 2.0 can't work.

-- 
Bye,
Gor Gyolchanyan.
Jul 31 2012
prev sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 31-07-2012 19:36, Maxime Chevalier wrote:

 The functions and constants related to this are all in
 core.sys.posix.mman.

 See also:

 * https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71
 (and further down)
 * https://github.com/lycus/mci/blob/master/src/mci/vm/code.d


 Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman
 ship standard with D compilers?


Yep. Anything core.* and std.* is part of druntime and phobos which both 
ship with any D 2.0 compiler.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org
Jul 31 2012
parent reply "Maxime Chevalier" <maximechevalierb gmail.com> writes:
 Yep. Anything core.* and std.* is part of druntime and phobos 
 which both ship with any D 2.0 compiler.


Thanks very much for the quick responses.

My tracing JIT will likely need to have the compiled code write 
into data structures of the host VM. This will mean the compiled 
code holds pointers to host data.

How well does the D GC deal with immovable objects. Would it be a 
problem if I allocated many chunks of immovable data?

Would it be preferable to minimize immovable data by having 
compiled code refer to an (immovable) table with pointers to 
(movable) host data?

Also, if you don't mind me asking, which D compiler do you guys 
prefer? Which one is most widely supported, most up to date?
Jul 31 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 31-07-2012 20:06, Maxime Chevalier wrote:

 Yep. Anything core.* and std.* is part of druntime and phobos which
 both ship with any D 2.0 compiler.


 Thanks very much for the quick responses.

 My tracing JIT will likely need to have the compiled code write into
 data structures of the host VM. This will mean the compiled code holds
 pointers to host data.

 How well does the D GC deal with immovable objects. Would it be a
 problem if I allocated many chunks of immovable data?


Can you clarify what you mean by immovable data? I don't think I follow.

In any case, if the compiled code's executable memory regions aren't 
added as root ranges to D's GC (which is entirely reasonable to not do), 
then you can keep the data alive by simply keeping pointers to the D 
GC-managed data in the host environment. D doesn't use a moving collector.


 Would it be preferable to minimize immovable data by having compiled
 code refer to an (immovable) table with pointers to (movable) host data?

 Also, if you don't mind me asking, which D compiler do you guys prefer?
 Which one is most widely supported, most up to date?


DMD: Latest updates, fixes, and enhancements. Very fast compilation. Not 
that great optimization.
GDC: Best optimization and fantastic debug info generation. Somewhat 
slow at compilation.
LDC: Fast compilation and good optimization. Sits somewhere between DMD 
and GDC.

Note that DMD can only target x86 while LDC and GDC (due to using LLVM 
and GCC respectively) can target many other 32-bit and 64-bit architectures.

Also, LDC and GDC tend to lag one release behind DMD sometimes, since 
they rely on the DMD front end code.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org
Jul 31 2012
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 07/31/2012 08:06 PM, Maxime Chevalier wrote:

 How well does the D GC deal with immovable objects. Would it be a
 problem if I allocated many chunks of immovable data?


The current GC in druntime never moves data.
Jul 31 2012
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Maxime Chevalier:


 How well does the D GC deal with immovable objects. Would it be 
 a problem if I allocated many chunks of immovable data?


The GC supports flags to define memory as movable or immovable,
but I think at the moment it performs no memory movements.



 Also, if you don't mind me asking, which D compiler do you guys 
 prefer? Which one is most widely supported, most up to date?


DMD has a back-end that produces less efficient binaries compared
to GDC and LDC. But DMD is more up to date (but not a lot), and
it's the most supported, it's the reference implementation. DMD
works well on Windows too, GDC works on Windows too. I think LDC
doesn't work well on Windows.

Bye,
bearophile
Jul 31 2012
parent "bearophile" <bearophileHUGS lycos.com> writes:
bearophile:


 DMD has a back-end that produces less efficient binaries 
 compared
 to GDC and LDC. But DMD is more up to date (but not a lot), and
 it's the most supported, it's the reference implementation. DMD
 works well on Windows too, GDC works on Windows too. I think LDC
 doesn't work well on Windows.


An DMD doesn't support 64 bit on Windows.

Bye,
bearophile
Jul 31 2012
prev sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Tuesday, 31 July 2012 at 18:06:28 UTC, Maxime Chevalier wrote:

 How well does the D GC deal with immovable objects. Would it be 
 a problem if I allocated many chunks of immovable data?


D's GC currently doesn't move memory at all, so this should not 
be a problem. If there is a chance that all references to the 
object in question in D-managed memory (i.e. stack, GC heap) go 
away, but the memory is still used, you might need to manually 
mark the object as live, though (see core.memory.GC.addRoot).


 Also, if you don't mind me asking, which D compiler do you guys 
 prefer? Which one is most widely supported, most up to date?


DMD is the official reference compiler and compiles faster than 
the others, but is x86-only (32 bit only on Windows), and the 
performance of the generated code is sometimes significantly 
worse than with the other compilers. GDC and LDC also use the 
same frontend as DMD, but it usually takes them a few weeks to 
catch up after a new version of it is released. LDC currently 
doesn't work on Windows if your application uses exception 
handling (which most applications do).

I personally use LDC as my default compiler, but I might be 
biased… ;)

David
Jul 31 2012
prev sibling next sibling parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:

 New to the D language here. I'm working on a tracing JIT 
 compiler and will need to allocate chunks of memory that are 
 marked as executable.

 Is there a standard way of doing this in D, or do I need to 
 directly call into mprotect? If I'm going to be calling 
 mprotect, what's the cleanest way to do that, do I need to 
 declare my own prototype for the function and its flags, or 
 should I write some C code that does the call?


  The x86 chip it's a simple flag that the OS can set. For use 
with like UPX, the whole section is marked read, write & execute 
I believe (since it has to expand it first before it can execute 
the code; That is of course for loaded memory, not 
allocating...). I would say check their sources, may prove 
interesting.
Jul 31 2012
parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 31-07-2012 20:30, Era Scarecrow wrote:

 On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:

 New to the D language here. I'm working on a tracing JIT compiler and
 will need to allocate chunks of memory that are marked as executable.

 Is there a standard way of doing this in D, or do I need to directly
 call into mprotect? If I'm going to be calling mprotect, what's the
 cleanest way to do that, do I need to declare my own prototype for the
 function and its flags, or should I write some C code that does the call?


   The x86 chip it's a simple flag that the OS can set. For use with like
 UPX, the whole section is marked read, write & execute I believe (since
 it has to expand it first before it can execute the code; That is of
 course for loaded memory, not allocating...). I would say check their
 sources, may prove interesting.


UPX simply uses VirtualProtect/mprotect. It's the only way you _can_ do 
this in ring 3 on any OS with sane security.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org
Jul 31 2012
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:

 New to the D language here. I'm working on a tracing JIT 
 compiler and will need to allocate chunks of memory that are 
 marked as executable.

 Is there a standard way of doing this in D, or do I need to 
 directly call into mprotect? If I'm going to be calling 
 mprotect, what's the cleanest way to do that, do I need to 
 declare my own prototype for the function and its flags, or 
 should I write some C code that does the call?

 Thanks for your help.


Hope this helps:
http://stackoverflow.com/a/8656294/21501
Jul 31 2012