digitalmars.D.announce - perpetual: new module for mapping file into memory
- Sergei Degtiarev (9/9) Aug 17 2015 I suggests a new module for mapping files into memory. Instead
- Liam McSherry (4/5) Aug 17 2015 There is already this in Phobos:
- Sergei Degtiarev (6/7) Aug 17 2015 Implementation at system level is similar, the approach is
- Rikki Cattermole (14/23) Aug 17 2015 What I like about this approach is:
- Dmitry Olshansky (22/27) Aug 17 2015 Hm it depends - one thing is that after mmaping a file the type may
- Sergei Degtiarev (8/20) Aug 18 2015 Why do you think the destructor may not be called? It is, and it
- Dmitry Olshansky (6/15) Aug 18 2015 Class is allocated on GC heap dtor is called on collection in that sense...
- Laeeth Isharc (6/17) Aug 19 2015 It might be run, but there is no guarantee a class destructor
- Sergei Degtiarev (20/29) Aug 18 2015 I plan to implement arguments forwarding to constructor, just
- Sergei Degtiarev (4/5) Aug 18 2015 I added helper functions with argument forwarding for
I suggests a new module for mapping files into memory. Instead of viewing the file as array of bytes, it directly maps the object onto file allowing to handle it as regular in-memory object. The data remain persistent after the program exits and may be re-opened and used again or shared between processes. Dynamic arrays are also supported with some limitations. The code is at https://github.com/sdegtiarev/phobos/tree/master/std/experimental/perpetual Is anybody interested?
Aug 17 2015
On Monday, 17 August 2015 at 18:27:48 UTC, Sergei Degtiarev wrote:[...]There is already this in Phobos: http://dlang.org/phobos/std_mmfile.html Is there any difference between `std.mmfile` and your solution?
Aug 17 2015
On Monday, 17 August 2015 at 19:00:37 UTC, Liam McSherry wrote:Is there any difference between `std.mmfile` and your solution?Implementation at system level is similar, the approach is different. std.mmfile returns uninitialized chunk of memory, std.perpetual returns an object which can be viewed and manipulated like regular object of the underlying type. This is kind of OOP approach to shared memory.
Aug 17 2015
On 18/08/2015 6:27 a.m., Sergei Degtiarev wrote:I suggests a new module for mapping files into memory. Instead of viewing the file as array of bytes, it directly maps the object onto file allowing to handle it as regular in-memory object. The data remain persistent after the program exits and may be re-opened and used again or shared between processes. Dynamic arrays are also supported with some limitations. The code is at https://github.com/sdegtiarev/phobos/tree/master/std/experimental/perpetual Is anybody interested?What I like about this approach is: - No memory allocation, yes yuppie! - Uses what ever your type is you want! - The type you want is initialized, yuppie What I don't like: - Does not allow for calling the constructor on the type should it have one - Does not map to std.mmfile in any way - No way to say an offset in the file to start working from (also maybe an upper limit for arrays) - You implement basically everything in std.mmfile all over again Fix up the things above and maybe with a bit more cleaning up it can go into std.mmfile. It's basically a poor mans (de/)serializer which is quite a nice thing to have.
Aug 17 2015
On 18-Aug-2015 07:45, Rikki Cattermole wrote:What I don't like: - Does not allow for calling the constructor on the type should it have oneHm it depends - one thing is that after mmaping a file the type may already be constructed, if it wasn't then emplacement should be the right thing to do. That is there should be a default/string value.- Does not map to std.mmfile in any wayWhich is a good thing. std.mmfile is crappy 8 year old ported crap.- No way to say an offset in the file toThat may be useful.- You implement basically everything in std.mmfile all over againLow-level API might be in order but it's not std.mmfile which is not low-level enough. Separating out a set of portable mmaping primitives would be awesome. What I don't like is perpetual being a class - too bad as it won't honor scoping and destructor may or may not be called. Just make it a struct. Also type should be Perputal, then perpetual is helper function that creates it then by means of IFTI type may be deduced. I suggest the following API: // all parameters after file are either T itself auto p = perpetual("file-name", 42);// would deduce T=int // or args to T: struct RGB{ int r,g,b; } // if not in file use RGB(1,2,3) auto p = perpetual!RGB("file-name", 1, 2, 3); -- Dmitry Olshansky
Aug 17 2015
On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky wrote:What I don't like is perpetual being a class - too bad as it won't honor scoping and destructor may or may not be called. Just make it a struct.Why do you think the destructor may not be called? It is, and it syncs memory and unmaps the file. Making perpetual a struct would require implementing reference counting or disabling copying, and all such staff. In my opinion, perpetual should definitely have reference semantics, this is exactly a point where D shines.Also type should be Perputal, then perpetual is helper function that creates it then by means of IFTI type may be deduced. I suggest the following API: // all parameters after file are either T itself auto p = perpetual("file-name", 42);// would deduce T=int // or args to T: struct RGB{ int r,g,b; } // if not in file use RGB(1,2,3) auto p = perpetual!RGB("file-name", 1, 2, 3);Right, just forgot. Will do, thanks.
Aug 18 2015
On 18-Aug-2015 17:32, Sergei Degtiarev wrote:On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky wrote:Class is allocated on GC heap dtor is called on collection in that sense it is a finalizer. Just use struct if you don't need inheritance and/or do resource management. -- Dmitry OlshanskyWhat I don't like is perpetual being a class - too bad as it won't honor scoping and destructor may or may not be called. Just make it a struct.Why do you think the destructor may not be called? It is, and it syncs memory and unmaps the file. Making perpetual a struct would require implementing reference counting or disabling copying, and all such staff. In my opinion, perpetual should definitely have reference semantics, this is exactly a point where D shines.
Aug 18 2015
On Tuesday, 18 August 2015 at 14:32:18 UTC, Sergei Degtiarev wrote:On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky wrote:It might be run, but there is no guarantee a class destructor will ever be called. A struct destructor will be when it goes out of scope. Hence Scoped! http://forum.dlang.org/thread/ikuomwcxhydluojkhjqy forum.dlang.orgWhat I don't like is perpetual being a class - too bad as it won't honor scoping and destructor may or may not be called. Just make it a struct.Why do you think the destructor may not be called? It is, and it syncs memory and unmaps the file. Making perpetual a struct would require implementing reference counting or disabling copying, and all such staff. In my opinion, perpetual should definitely have reference semantics, this is exactly a point where D shines.
Aug 19 2015
On Tuesday, 18 August 2015 at 04:46:10 UTC, Rikki Cattermole wrote:- Does not allow for calling the constructor on the type should it have oneI plan to implement arguments forwarding to constructor, just decided that the concept would be more clear with minimal code. Technically, the object is initialized with .init and is in valid state.- Does not map to std.mmfile in any way - You implement basically everything in std.mmfile all over againDoes not need to, there are only two system calls involved and it makes no sense to reuse such amount of code.- No way to say an offset in the file to start working from (also maybe an upper limit for arrays)This might be useful in some cases, but in general from my experience (I used this utility for several years in C++), it downgrades the interface, makes you regarding the data as set of bytes rather than an object. It is always possible to wrap up everything in a struct, so that working portion is at right offset.it can go into std.mmfile.The concept is very different from plain shared memory, std.mmfile can be improved with a few lines of code, but it wouldn't change the way it's used.It's basically a poor mans (de/)serializer.Not at all, it allows fast access to the data, so it might be used as effectively as regular values, and insurance from data loss at crash or stop.
Aug 18 2015
On Tuesday, 18 August 2015 at 14:16:30 UTC, Sergei Degtiarev wrote:Right, just forgot. Will do, thanks.I added helper functions with argument forwarding for initialization.
Aug 18 2015