www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Re: Memory Mapped File Access

reply Sean Kelly <sean invisibleduck.org> writes:
Robert Wrote:


 Hi, has anyone played around with D and memory mapped files on Windows / Linux?
 
 A friend of mine and I want to use D to develop a D native 
 database-system. Yes, sounds crazy and it will take long and we haven't 
 done a lot yet. So don't expect anything to look at soon :-)



Andrei and I had talked a while back about adding memory-mapped file support to
the GC and then it fell off the radar while we worked on other things.  I'll
see if I can remember how it was to work.
May 29 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 05/29/2010 10:40 AM, Sean Kelly wrote:

 Robert Wrote:


 Hi, has anyone played around with D and memory mapped files on
 Windows / Linux?

 A friend of mine and I want to use D to develop a D native
 database-system. Yes, sounds crazy and it will take long and we
 haven't done a lot yet. So don't expect anything to look at soon
 :-)



 Andrei and I had talked a while back about adding memory-mapped file
 support to the GC and then it fell off the radar while we worked on
 other things.  I'll see if I can remember how it was to work.



The basic idea is that the only way to handle memory-mapped files safely 
is to let the garbage collector close them. This is because in any other 
case you'd have dangling pointers.

So the idea is that druntime should provide a safe means for mapping a 
file to memory and an unsafe means of closing a file. Safe code should 
be able to count on the garbage collector to close memory-mapped files 
that have no pointers referring to them.


Andrei
May 29 2010
parent reply Robert <robert.muench robertmuench.de> writes:
On 2010-05-29 18:30:13 +0200, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 Andrei and I had talked a while back about adding memory-mapped file
 support to the GC and then it fell off the radar while we worked on
 other things.  I'll see if I can remember how it was to work.



 The basic idea is that the only way to handle memory-mapped files 
 safely is to let the garbage collector close them. This is because in 
 any other case you'd have dangling pointers.



Ok, that makes sense. On the other hand I will use a very simple 
rule-of-thumb: As long as the app runs the file is open. Only if the 
app terminates the file gets closed.

Which implies that the reference to the MMF is a global but this 
shouldn't be a problem.


 So the idea is that druntime should provide a safe means for mapping a 
 file to memory and an unsafe means of closing a file.



Why should it provide an unsafe way of closing a MMF file?


 Safe code should be able to count on the garbage collector to close 
 memory-mapped files that have no pointers referring to them.



Have you sketched any ideas how to use the GC for MMF?

--
Robert M. Münch
http://www.robertmuench.de
May 29 2010
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 05/29/2010 02:52 PM, Robert wrote:

 On 2010-05-29 18:30:13 +0200, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:


 Andrei and I had talked a while back about adding memory-mapped file
 support to the GC and then it fell off the radar while we worked on
 other things. I'll see if I can remember how it was to work.



 The basic idea is that the only way to handle memory-mapped files
 safely is to let the garbage collector close them. This is because in
 any other case you'd have dangling pointers.



 Ok, that makes sense. On the other hand I will use a very simple
 rule-of-thumb: As long as the app runs the file is open. Only if the app
 terminates the file gets closed.



That might be the case if no collection ensues, but if a collection 
does, the runtime should attempt to reclaim available resources.


 Which implies that the reference to the MMF is a global but this
 shouldn't be a problem.


 So the idea is that druntime should provide a safe means for mapping a
 file to memory and an unsafe means of closing a file.



 Why should it provide an unsafe way of closing a MMF file?



For applications concerned with deterministic closing of a file (e.g. 
following writing) and that are willing to take the unsafety risk.


 Safe code should be able to count on the garbage collector to close
 memory-mapped files that have no pointers referring to them.



 Have you sketched any ideas how to use the GC for MMF?



There's nothing to sketch, really. The runtime tracks the opened 
memory-mapped files and the memory ranges associated with them. Upon a 
collections, if a file's memory has been successfully freed, the file 
can be safely closed.


Andrei



Andrei
May 29 2010
parent Robert <robert.muench robertmuench.de> writes:
On 2010-05-29 23:23:06 +0200, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 There's nothing to sketch, really. The runtime tracks the opened 
 memory-mapped files and the memory ranges associated with them. Upon a 
 collections, if a file's memory has been successfully freed, the file 
 can be safely closed.



Is this something that needs to be integrated into the deeper D levels 
or is this something I could plug on-top if it? I'm not yet that 
familiar with the GC internals.

--
Robert M. Münch
http://www.robertmuench.de
May 29 2010
prev sibling parent BLS <windevguy hotmail.de> writes:
On 29/05/2010 21:52, Robert wrote:

 Ok, that makes sense. On the other hand I will use a very simple
 rule-of-thumb: As long as the app runs the file is open. Only if the app
 terminates the file gets closed.

 Which implies that the reference to the MMF is a global but this
 shouldn't be a problem.




 From the docs..
---
File is closed when the object instance is deleted.
---
Maybe we can have a more precise doc. please.
May 29 2010