www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Binary data-structure serialization

reply Robert <robert.muench robertmuench.de> writes:
Hi, are there are any projectes / snippets / prototypes where someone 
tried to serializa D run-time data-structures (like an associative 
array) into a file?

--
Robert M. Münch
http://www.robertmuench.de
May 30 2010
next sibling parent reply Robert <robert.muench robertmuench.de> writes:
On 2010-05-30 12:38:10 +0200, Robert <robert.muench robertmuench.de> said:


 Hi, are there are any projectes / snippets / prototypes where someone 
 tried to serializa D run-time data-structures (like an associative 
 array) into a file?


Thinking further about this: How about a way to change the underlying 
memory manager from RAM to a MMF? If I could specify which memory 
allocator to use when creating a class, The class could be created in a 
MMF (which is mapped to main memory and hence there should be no 
difference in speed) and all references would go to the MMF as well.

With such a low-level switch persistent data-structures would be for 
free. The memory allocator could be optimized to yield minimal MMF.

Any comments?

-- 
Robert M. Münch
http://www.robertmuench.de
May 30 2010
parent reply Trass3r <un known.com> writes:
 If I could specify which memory allocator to use when creating a class,  
 The class could be created in a MMF (which is mapped to main memory and  
 hence there should be no difference in speed) and all references would  
 go to the MMF as well.


Isn't http://digitalmars.com/d/2.0/class.html#ClassAllocator what you want?
May 31 2010
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-05-31 17:01:16 +0200, Trass3r said:


 Isn't http://digitalmars.com/d/2.0/class.html#ClassAllocator what you want?


Ah, that's cool. Didn't saw it. This looks very good. Will those clases 
be GCed?

-- 
Robert M. Münch
www.robertmuench.de
Jun 01 2010
parent reply "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Wed, 02 Jun 2010 06:29:12 +0900, Robert M. Münch
<robert.muench robertmuench.de> wrote:


 On 2010-05-31 17:01:16 +0200, Trass3r said:


 Isn't http://digitalmars.com/d/2.0/class.html#ClassAllocator what you  
 want?


 Ah, that's cool. Didn't saw it. This looks very good. Will those clases  
 be GCed?


Default no. Documentation is here:

http://digitalmars.com/d/2.0/memory.html#newdelete

NOTE: This info is a bit outdated. std.gc was moved to core.memory and
std.outofmemory was moved to onOutOfMemoryError in core.exception.
Jun 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/01/2010 06:41 PM, Masahiro Nakagawa wrote:

 On Wed, 02 Jun 2010 06:29:12 +0900, Robert M. Münch
 <robert.muench robertmuench.de> wrote:


 On 2010-05-31 17:01:16 +0200, Trass3r said:


 Isn't http://digitalmars.com/d/2.0/class.html#ClassAllocator what you
 want?


 Ah, that's cool. Didn't saw it. This looks very good. Will those
 clases be GCed?


 Default no. Documentation is here:

 http://digitalmars.com/d/2.0/memory.html#newdelete

 NOTE: This info is a bit outdated. std.gc was moved to core.memory and
 std.outofmemory was moved to onOutOfMemoryError in core.exception.


Sorry, class allocators are gone.

Andrei
Jun 01 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 Sorry, class allocators are gone.


I must have missed the memo :o) Can you explain a bit?

Bye,
bearophile
Jun 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/01/2010 07:35 PM, bearophile wrote:

 Andrei Alexandrescu:

 Sorry, class allocators are gone.


 I must have missed the memo :o) Can you explain a bit?


It was an ill-defined misfeature that didn't deserve to stay in the 
language. Walter and I agreed to let it go, but somehow forgot to 
announce it.

Andrei
Jun 01 2010
next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 06/01/2010 08:17 PM, Andrei Alexandrescu wrote:

 On 06/01/2010 07:35 PM, bearophile wrote:

 Andrei Alexandrescu:

 Sorry, class allocators are gone.


 I must have missed the memo :o) Can you explain a bit?


 It was an ill-defined misfeature that didn't deserve to stay in the
 language. Walter and I agreed to let it go, but somehow forgot to
 announce it.

 Andrei


replacement?
Jun 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/01/2010 08:29 PM, Ellery Newcomer wrote:

 On 06/01/2010 08:17 PM, Andrei Alexandrescu wrote:

 On 06/01/2010 07:35 PM, bearophile wrote:

 Andrei Alexandrescu:

 Sorry, class allocators are gone.


 I must have missed the memo :o) Can you explain a bit?


 It was an ill-defined misfeature that didn't deserve to stay in the
 language. Walter and I agreed to let it go, but somehow forgot to
 announce it.

 Andrei


 replacement?


TightArray.

Andrei
Jun 01 2010
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-02 03:58:57 +0200, Andrei Alexandrescu said:


 TightArray.


What's that? Any link / reference, I haven't found it in the docs.

I was quite happy to see a class allocator because it gives flexibility 
in how & where class objects are allocated. That would have matched my 
requirements perfectly.

I need to somehow get control of the allocation procedure but would 
like to let the GC take care of the returned memory.

-- 
Robert M. Münch
www.robertmuench.de
Jun 02 2010
parent reply "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
On Wed, 02 Jun 2010 10:47:37 +0200, Robert M. Münch wrote:


 On 2010-06-02 03:58:57 +0200, Andrei Alexandrescu said:
 

 TightArray.


 
 What's that? Any link / reference, I haven't found it in the docs.


It's part of his proposal for a container library, which he has presented 
in some other threads:

http://erdani.com/d/phobos/std_container.html
http://erdani.com/d/phobos/container.d

In particular, see

http://erdani.com/d/phobos/std_container.html#TightArray

-Lars
Jun 02 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Lars T. Kyllingstad:

 In particular, see
 http://erdani.com/d/phobos/std_container.html#TightArray


The TightArray seems a reference-counted array of items, where items can be
references of objects allocated on the heap. This doesn't solve the problem of
avoiding heap allocations when you want to use objects. How can you allocate
objects themselves in an arena, a stack, or a pool?

Bye,
bearophile
Jun 02 2010
prev sibling parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-02 11:37:18 +0200, Lars T. Kyllingstad said:


 It's part of his proposal for a container library, which he has presented
 in some other threads:
 
 http://erdani.com/d/phobos/std_container.html
 http://erdani.com/d/phobos/container.d
 
 In particular, see
 
 http://erdani.com/d/phobos/std_container.html#TightArray


Thanks for the link.

Is this stuff a phobos fork? An extension to phobos? Is this related to 
D2? I'm a bit confused how to relate this work to the runtime lib.

-- 
Robert M. Münch
www.robertmuench.de
Jun 02 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Robert M. Münch:

 Is this stuff a phobos fork? An extension to phobos? Is this related to 
 D2? I'm a bit confused how to relate this work to the runtime lib.


It's just a module that will be added to Phobos.

Bye,
bearophile
Jun 02 2010
prev sibling parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
On Wed, 02 Jun 2010 15:15:25 +0200, Robert M. Münch wrote:


 On 2010-06-02 11:37:18 +0200, Lars T. Kyllingstad said:
 

 It's part of his proposal for a container library, which he has
 presented in some other threads:
 
 http://erdani.com/d/phobos/std_container.html
 http://erdani.com/d/phobos/container.d
 
 In particular, see
 
 http://erdani.com/d/phobos/std_container.html#TightArray


 
 Thanks for the link.
 
 Is this stuff a phobos fork? An extension to phobos? Is this related to
 D2? I'm a bit confused how to relate this work to the runtime lib.



It's a work in progress, supposed to be included in Phobos for D2 at some 
point.  It was discussed in these threads:

http://www.digitalmars.com/d/archives/digitalmars/D/
container_stuff_110423.html

http://www.digitalmars.com/d/archives/digitalmars/D/
std.container_update_110509.html

http://www.digitalmars.com/d/archives/digitalmars/D/std.container_update_-
_now_Array_is_in_110574.html


-Lars
Jun 02 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to 
 announce it.


Are stack-allocated (scoped) classes too gone?

Bye,
bearophile
Jun 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/01/2010 08:53 PM, bearophile wrote:

 Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to
 announce it.


 Are stack-allocated (scoped) classes too gone?


De facto no, de jure yes.

Andrei
Jun 01 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 De facto no, de jure yes.


Walter is listening you (see comment 6):
http://d.puremagic.com/issues/show_bug.cgi?id=2070

I have used a good number of "scope" in classes for my small programs that I am
slowly porting from D1 to D2. A higher level language, that omits manual
low-level optimizations requires a more powerful compiler, able to optimize
away some abstractions by itself. So I hope D will soon use a good escape
analysis to stack-allocate classes (and dynamic arrays), as recent versions of
HotSpot do (but even with an escape analysis, the objects inside objects get
allocated elsewhere anyway. Maybe this requires another kind of escape
analysis?). (I have said "use" because I think DMD is already able to perform
some escape analysis).


http://java.sun.com/javase/6/webnotes/6u14.html

Optimization Using Escape Analysis: The -XX:+DoEscapeAnalysis option directs
HotSpot to look for objects that are created and referenced by a single thread
within the scope of a method compilation. Allocation is omitted for such
non-escaping objects, and their fields are treated as local variables, often
residing in machine registers. Synchronization on non-escaping objects is also
elided.<



EA, from page 52:
http://webdocs.cs.ualberta.ca/~amaral/IBM-Stoodley-talks/UofAKASWorkshop.pdf


To focus it's better to start with a very simple example. So at first dmd can
remove the heap allocation in this basic case:


import std.stdio: writeln;

final class Rectangle {
    int x, y;

    this(int xx, int yy) {
        this.x = xx;
        this.y = yy;
    }

    int area() {
        return this.x * this.y;
    }
}

void main () {
    enum N = 100;

    ulong tot = 0;
    foreach (x; 0 .. N)
        foreach (y; 0 .. N) {
            Rectangle rectangle = new Rectangle(x, y);
            tot += rectangle.area();
        }

    writeln(tot);
}



So the optimization phase can turn that into something equivalent to:

import std.stdio: writeln;

void main () {
    enum N = 100;
    ulong tot = 0;
    foreach (x; 0 .. N)
        foreach (y; 0 .. N)
            tot += x * y;
    writeln(tot);
}


That was a naive/toy example, but you can't start from very complex things.
Later more complex cases can be added, like dynamic arrays, synchronized
methods, etc.

Bye,
bearophile
Jun 02 2010
prev sibling next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 De facto no, de jure yes.


Maybe the "Typesafe Variadic Functions For class objects" can be removed from
D2.

Bye,
bearophile
Jun 02 2010
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 06/01/2010 08:53 PM, bearophile wrote:

 Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to
 announce it.


 Are stack-allocated (scoped) classes too gone?


 De facto no, de jure yes.


I don't really understand this.  Are they going to be gone or not?

In any case, can we at least keep them for unsafe D?  Stack allocation is  
not easy to do via placement new, the compiler is much better suited for  
it, and stack allocating classes can be a huge performance gain.  Until we  
get full escape analysis, scope classes are the only option.

-Steve
Jun 02 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:

 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/01/2010 08:53 PM, bearophile wrote:

 Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to
 announce it.


 Are stack-allocated (scoped) classes too gone?


 De facto no, de jure yes.


 I don't really understand this. Are they going to be gone or not?

 In any case, can we at least keep them for unsafe D? Stack allocation is
 not easy to do via placement new, the compiler is much better suited for
 it, and stack allocating classes can be a huge performance gain. Until
 we get full escape analysis, scope classes are the only option.


There's no need for a language feature. I am implementing stack 
allocation for classes in library code; I'm just undecided on where to 
put it.

Andrei
Jun 02 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 I am implementing stack allocation for classes in library code;
 I'm just undecided on where to put it.


A module that contains code to manage memory, create memory pools, allocate a
whole 2D dynamic array out of a single chunck of memory, and so on.

Bye,
bearophile
Jun 02 2010
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:

 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/01/2010 08:53 PM, bearophile wrote:

 Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to
 announce it.


 Are stack-allocated (scoped) classes too gone?


 De facto no, de jure yes.


 I don't really understand this. Are they going to be gone or not?

 In any case, can we at least keep them for unsafe D? Stack allocation is
 not easy to do via placement new, the compiler is much better suited for
 it, and stack allocating classes can be a huge performance gain. Until
 we get full escape analysis, scope classes are the only option.


 There's no need for a language feature. I am implementing stack  
 allocation for classes in library code; I'm just undecided on where to  
 put it.


I think it belongs in druntime, since it's an allocation feature.  How  
easy is it to use?

-Steve
Jun 02 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2010 02:27 PM, Steven Schveighoffer wrote:

 On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:

 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/01/2010 08:53 PM, bearophile wrote:

 Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to
 announce it.


 Are stack-allocated (scoped) classes too gone?


 De facto no, de jure yes.


 I don't really understand this. Are they going to be gone or not?

 In any case, can we at least keep them for unsafe D? Stack allocation is
 not easy to do via placement new, the compiler is much better suited for
 it, and stack allocating classes can be a huge performance gain. Until
 we get full escape analysis, scope classes are the only option.


 There's no need for a language feature. I am implementing stack
 allocation for classes in library code; I'm just undecided on where to
 put it.


 I think it belongs in druntime, since it's an allocation feature. How
 easy is it to use?


The signature for non-class types is:

T* emplace(T, Args...)(void * address, Args args);

The function creates an object of type T at location address and 
initializes it with args. Returns a pointer to the just-created value. I 
needed this function for TightArray.

I haven't yet defined the signature for class types, but probably it 
would look like this:

T emplace(T, Args...)(void * address, Args args);

To obtain stack allocation, you may therefore write something like:

auto obj = emplace(alloca(_traits(classInstanceSize, T)), stuff);

Although it uses the obscure classInstanceSize, I think it's near the 
desired balance between terseness and making the programmer sweat for a 
dangerous achievement.


Andrei
Jun 02 2010
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 02 Jun 2010 15:34:32 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:27 PM, Steven Schveighoffer wrote:

 On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:

 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/01/2010 08:53 PM, bearophile wrote:

 Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to
 announce it.


 Are stack-allocated (scoped) classes too gone?


 De facto no, de jure yes.


 I don't really understand this. Are they going to be gone or not?

 In any case, can we at least keep them for unsafe D? Stack allocation  
 is
 not easy to do via placement new, the compiler is much better suited  
 for
 it, and stack allocating classes can be a huge performance gain. Until
 we get full escape analysis, scope classes are the only option.


 There's no need for a language feature. I am implementing stack
 allocation for classes in library code; I'm just undecided on where to
 put it.


 I think it belongs in druntime, since it's an allocation feature. How
 easy is it to use?


 The signature for non-class types is:

 T* emplace(T, Args...)(void * address, Args args);

 The function creates an object of type T at location address and  
 initializes it with args. Returns a pointer to the just-created value. I  
 needed this function for TightArray.

 I haven't yet defined the signature for class types, but probably it  
 would look like this:

 T emplace(T, Args...)(void * address, Args args);

 To obtain stack allocation, you may therefore write something like:

 auto obj = emplace(alloca(_traits(classInstanceSize, T)), stuff);

 Although it uses the obscure classInstanceSize, I think it's near the  
 desired balance between terseness and making the programmer sweat for a  
 dangerous achievement.


This makes it better than scope classes for one reason -- it's an  
expression instead of a declaration.  That means I can use it to create a  
temporary.

But I have some qualms here -- when is the object's destructor called?  I  
know it probably cannot be done any other way unless we had something like  
macros, but it's not easy for humans to parse this mess.  Sure, it should  
be somewhat difficult for the developer to write, but what about the  
reader?  I suppose there just isn't an easy way to do this.

Also, wouldn't it have to be something like this:

auto obj = emplace!T(alloca(_traits(classInstanceSize, T)), stuff);

And finally, is there any possible way to write an emplace that uses an  
allocator alias instead of the actual memory location?  i.e.

T emplace2(T, alias alloc, Args...)(Args args)
{
    return emplace!(T, Args)(alloc(_traits(classInstanceSize, T)), args);
}

Anything to avoid the double-specification of T would be nice.  This of  
course would be impossible to use with alloca without macros.

-Steve
Jun 02 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2010 02:49 PM, Steven Schveighoffer wrote:

 On Wed, 02 Jun 2010 15:34:32 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:27 PM, Steven Schveighoffer wrote:

 On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:

 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/01/2010 08:53 PM, bearophile wrote:

 Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to
 announce it.


 Are stack-allocated (scoped) classes too gone?


 De facto no, de jure yes.


 I don't really understand this. Are they going to be gone or not?

 In any case, can we at least keep them for unsafe D? Stack
 allocation is
 not easy to do via placement new, the compiler is much better
 suited for
 it, and stack allocating classes can be a huge performance gain. Until
 we get full escape analysis, scope classes are the only option.


 There's no need for a language feature. I am implementing stack
 allocation for classes in library code; I'm just undecided on where to
 put it.


 I think it belongs in druntime, since it's an allocation feature. How
 easy is it to use?


 The signature for non-class types is:

 T* emplace(T, Args...)(void * address, Args args);

 The function creates an object of type T at location address and
 initializes it with args. Returns a pointer to the just-created value.
 I needed this function for TightArray.

 I haven't yet defined the signature for class types, but probably it
 would look like this:

 T emplace(T, Args...)(void * address, Args args);

 To obtain stack allocation, you may therefore write something like:

 auto obj = emplace(alloca(_traits(classInstanceSize, T)), stuff);

 Although it uses the obscure classInstanceSize, I think it's near the
 desired balance between terseness and making the programmer sweat for
 a dangerous achievement.


 This makes it better than scope classes for one reason -- it's an
 expression instead of a declaration. That means I can use it to create a
 temporary.

 But I have some qualms here -- when is the object's destructor called? I
 know it probably cannot be done any other way unless we had something
 like macros, but it's not easy for humans to parse this mess. Sure, it
 should be somewhat difficult for the developer to write, but what about
 the reader? I suppose there just isn't an easy way to do this.

 Also, wouldn't it have to be something like this:

 auto obj = emplace!T(alloca(_traits(classInstanceSize, T)), stuff);

 And finally, is there any possible way to write an emplace that uses an
 allocator alias instead of the actual memory location? i.e.

 T emplace2(T, alias alloc, Args...)(Args args)
 {
 return emplace!(T, Args)(alloc(_traits(classInstanceSize, T)), args);
 }

 Anything to avoid the double-specification of T would be nice. This of
 course would be impossible to use with alloca without macros.

 -Steve


Good points.

To ensure the destructor is called, I guess we need this:

 system struct StackAllocated(T)
{
     private ubyte buffer[__traits(classInstanceSize, T)] = void;
      property T payload() { return cast(T) buffer.ptr; }
     alias payload this;
      disable this(); // meh
      disable this(this);
     this(Args...)(auto ref Args args)
     {
         emplace!T(buffer.ptr, buffer.length);
     }
     ~this()
     {
         clear(payload);
     }
}


Andrei
Jun 02 2010
next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 02 Jun 2010 15:57:42 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:



       disable this(); // meh


Is this wishful thinking or going to be a reality? ;)

-Steve
Jun 02 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 02 Jun 2010 15:57:42 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:49 PM, Steven Schveighoffer wrote:

 On Wed, 02 Jun 2010 15:34:32 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:27 PM, Steven Schveighoffer wrote:

 On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:

 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 06/01/2010 08:53 PM, bearophile wrote:

 Andrei Alexandrescu Wrote:

 Walter and I agreed to let it go, but somehow forgot to
 announce it.


 Are stack-allocated (scoped) classes too gone?


 De facto no, de jure yes.


 I don't really understand this. Are they going to be gone or not?

 In any case, can we at least keep them for unsafe D? Stack
 allocation is
 not easy to do via placement new, the compiler is much better
 suited for
 it, and stack allocating classes can be a huge performance gain.  
 Until
 we get full escape analysis, scope classes are the only option.


 There's no need for a language feature. I am implementing stack
 allocation for classes in library code; I'm just undecided on where  
 to
 put it.


 I think it belongs in druntime, since it's an allocation feature. How
 easy is it to use?


 The signature for non-class types is:

 T* emplace(T, Args...)(void * address, Args args);

 The function creates an object of type T at location address and
 initializes it with args. Returns a pointer to the just-created value.
 I needed this function for TightArray.

 I haven't yet defined the signature for class types, but probably it
 would look like this:

 T emplace(T, Args...)(void * address, Args args);

 To obtain stack allocation, you may therefore write something like:

 auto obj = emplace(alloca(_traits(classInstanceSize, T)), stuff);

 Although it uses the obscure classInstanceSize, I think it's near the
 desired balance between terseness and making the programmer sweat for
 a dangerous achievement.


 This makes it better than scope classes for one reason -- it's an
 expression instead of a declaration. That means I can use it to create a
 temporary.

 But I have some qualms here -- when is the object's destructor called? I
 know it probably cannot be done any other way unless we had something
 like macros, but it's not easy for humans to parse this mess. Sure, it
 should be somewhat difficult for the developer to write, but what about
 the reader? I suppose there just isn't an easy way to do this.

 Also, wouldn't it have to be something like this:

 auto obj = emplace!T(alloca(_traits(classInstanceSize, T)), stuff);

 And finally, is there any possible way to write an emplace that uses an
 allocator alias instead of the actual memory location? i.e.

 T emplace2(T, alias alloc, Args...)(Args args)
 {
 return emplace!(T, Args)(alloc(_traits(classInstanceSize, T)), args);
 }

 Anything to avoid the double-specification of T would be nice. This of
 course would be impossible to use with alloca without macros.

 -Steve


 Good points.

 To ensure the destructor is called, I guess we need this:

  system struct StackAllocated(T)
 {
      private ubyte buffer[__traits(classInstanceSize, T)] = void;
       property T payload() { return cast(T) buffer.ptr; }
      alias payload this;
       disable this(); // meh
       disable this(this);
      this(Args...)(auto ref Args args)
      {
          emplace!T(buffer.ptr, buffer.length);
      }
      ~this()
      {
          clear(payload);
      }
 }


Hm... is this valid usage then?

foo(StackAllocated!MyClass(a, b, c));

:)

-Steve
Jun 02 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 To ensure the destructor is called, I guess we need this:
 
  system struct StackAllocated(T)
 {
      private ubyte buffer[__traits(classInstanceSize, T)] = void;
       property T payload() { return cast(T) buffer.ptr; }
      alias payload this;
       disable this(); // meh
       disable this(this);
      this(Args...)(auto ref Args args)
      {
          emplace!T(buffer.ptr, buffer.length);
      }
      ~this()
      {
          clear(payload);
      }
 }


Is this useful?

static if (__traits(hasMember, T, "__dtor")) {
     ~this() {
         clear(payload);
     }
}

Bye,
bearophile
Jun 02 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2010 03:34 PM, bearophile wrote:

 Andrei Alexandrescu:

 To ensure the destructor is called, I guess we need this:

  system struct StackAllocated(T)
 {
       private ubyte buffer[__traits(classInstanceSize, T)] = void;
        property T payload() { return cast(T) buffer.ptr; }
       alias payload this;
        disable this(); // meh
        disable this(this);
       this(Args...)(auto ref Args args)
       {
           emplace!T(buffer.ptr, buffer.length);
       }
       ~this()
       {
           clear(payload);
       }
 }


 Is this useful?

 static if (__traits(hasMember, T, "__dtor")) {
       ~this() {
           clear(payload);
       }
 }

 Bye,
 bearophile


Absolutely!

Andrei
Jun 02 2010
prev sibling parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-02 21:34:32 +0200, Andrei Alexandrescu said:


 The signature for non-class types is:
 
 T* emplace(T, Args...)(void * address, Args args);
 
 The function creates an object of type T at location address and 
 initializes it with args. Returns a pointer to the just-created value. 
 I needed this function for TightArray.


Hi, does this already work? This would make it possible to allocate T 
within a memory-mapped file. I would like to give it a try.

Not sure how to deal with re-allocation problem as pointers within the 
MMF need to be MMF relative and not absolut. IMO via operator 
overloading this should be possible to handle in the background.


-- 
Robert M. Münch
www.robertmuench.de
Jun 03 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/03/2010 02:31 AM, Robert M. Münch wrote:

 On 2010-06-02 21:34:32 +0200, Andrei Alexandrescu said:


 The signature for non-class types is:

 T* emplace(T, Args...)(void * address, Args args);

 The function creates an object of type T at location address and
 initializes it with args. Returns a pointer to the just-created value.
 I needed this function for TightArray.


 Hi, does this already work? This would make it possible to allocate T
 within a memory-mapped file. I would like to give it a try.


I didn't have much time to work on it lately, but I'll check it in soon.


 Not sure how to deal with re-allocation problem as pointers within the
 MMF need to be MMF relative and not absolut. IMO via operator
 overloading this should be possible to handle in the background.


Once created, a file mapping is at a constant address. If you remap a 
file (an unsafe operation), you can memcpy the data around. All D 
objects are assumed to be relocatable.


Andrei
Jun 03 2010
parent Robert <robert.muench robertmuench.de> writes:
On 2010-06-03 15:45:11 +0200, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 Once created, a file mapping is at a constant address.


Hi, sorry but don't get this. This is true for one session. But the 
next time you run the prog the mapping can be on an other address.


 If you remap a file (an unsafe operation),


What do you mean by "remaping a file"?


  you can memcpy the data around. All D objects are assumed to be relocatable.


Still don't get it.

If a MMF is mapped into VM at 0x1000 and you create objects and 
pointers, the pointers within the MMF need to get 0x1000 substracted. 
Only in this case the pointer within the MMF references an address = 
the index within the file.

-- 
Robert M. Münch
http://www.robertmuench.de
Jun 03 2010
prev sibling next sibling parent reply "Masahiro Nakagawa" <repeatedly gmail.com> writes:
Hi Robert,

On Sun, 30 May 2010 19:38:10 +0900, Robert <robert.muench robertmuench.de>  
wrote:


 Hi, are there are any projectes / snippets / prototypes where someone  
 tried to serializa D run-time data-structures (like an associative  
 array) into a file?


I may not be right about your question, but I am writing msgpack4d.

http://www.bitbucket.org/repeatedly/msgpack4d

This library will be available in Phobos(see d.announce).
Now, I fix bugs and improve some features.

Hope this helps.


Masahiro
May 31 2010
parent reply Robert <robert.muench robertmuench.de> writes:
On 2010-05-31 11:09:54 +0200, "Masahiro Nakagawa" <repeatedly gmail.com> said:


 I may not be right about your question, but I am writing msgpack4d.
 
 http://www.bitbucket.org/repeatedly/msgpack4d
 
 This library will be available in Phobos(see d.announce).
 Now, I fix bugs and improve some features.


Hi, does it support data-structures like trees, lists etc. or is the 
focus to transfer plain types?

I'm looking for a way, where I can save and load complex 
data-structures like graphs, b-trees etc. to a file.

-- 
Robert M. Münch
http://www.robertmuench.de
May 31 2010
parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Mon, 31 May 2010 22:36:13 +0900, Robert <robert.muench robertmuench.de>  
wrote:


 On 2010-05-31 11:09:54 +0200, "Masahiro Nakagawa" <repeatedly gmail.com>  
 said:


 I may not be right about your question, but I am writing msgpack4d.
  http://www.bitbucket.org/repeatedly/msgpack4d
  This library will be available in Phobos(see d.announce).
 Now, I fix bugs and improve some features.


 Hi, does it support data-structures like trees, lists etc. or is the  
 focus to transfer plain types?

 I'm looking for a way, where I can save and load complex data-structures  
 like graphs, b-trees etc. to a file.


You can define mp_pack / mp_unpack if need(Complex data-structure requires  
specified routines).
I added simple b-tree example to example/custom.d.

If you want more efficient serialization library for complex  
data-structures, you will need to create like existing DBMs.


Masahiro
Jun 01 2010
prev sibling next sibling parent reply Eric Poggel <dnewsgroup yage3d.net> writes:
On 5/30/2010 6:38 AM, Robert wrote:

 Hi, are there are any projectes / snippets / prototypes where someone
 tried to serializa D run-time data-structures (like an associative
 array) into a file?

 --
 Robert M. Münch
 http://www.robertmuench.de


After having difficulty getting ddbg to work, I decided to write a dump 
function so I could easily see my data structures at runtime.  The 
biggest part of it is a json encoder which can handle most D structures. 
  If I recall, it only has trouble with enum's and pointers (since 
there's no length info).

It's available in Yage's source here 
(http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d) 
Yage is licensed LGPL but I grant permission to use this under the Boost 
1.0 license if you or anyone else needs it.

However, for now it's only one-way and there's no deserialization part 
to it.
May 31 2010
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-01 02:13:18 +0200, Eric Poggel said:


 After having difficulty getting ddbg to work, I decided to write a dump 
 function so I could easily see my data structures at runtime.  The 
 biggest part of it is a json encoder which can handle most D 
 structures.   If I recall, it only has trouble with enum's and pointers 
 (since there's no length info).
 
 It's available in Yage's source here 
 (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d) 
 Yage is licensed LGPL but I grant permission to use this under the 
 Boost 1.0 license if you or anyone else needs it.


Thanks for this. I will take a look.


 However, for now it's only one-way and there's no deserialization part to it.


Well, this part is left for the reader as an exercise ;-).

-- 
Robert M. Münch
www.robertmuench.de
Jun 01 2010
parent reply Eric Poggel <dnewsgroup yage3d.net> writes:
On 6/1/2010 5:31 PM, Robert M. Münch wrote:

 On 2010-06-01 02:13:18 +0200, Eric Poggel said:


 After having difficulty getting ddbg to work, I decided to write a
 dump function so I could easily see my data structures at runtime. The
 biggest part of it is a json encoder which can handle most D
 structures. If I recall, it only has trouble with enum's and pointers
 (since there's no length info).

 It's available in Yage's source here
 (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d)
 Yage is licensed LGPL but I grant permission to use this under the
 Boost 1.0 license if you or anyone else needs it.


 Thanks for this. I will take a look.


 However, for now it's only one-way and there's no deserialization part
 to it.


 Well, this part is left for the reader as an exercise ;-).


It seems it also doesn't serialize members of parent classes, which can 
be a major caveat.  I took a look at it yesterday but unfortunately 
wasn't able to figure it out.
Jun 06 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-06-06 21:32, Eric Poggel wrote:

 On 6/1/2010 5:31 PM, Robert M. Münch wrote:

 On 2010-06-01 02:13:18 +0200, Eric Poggel said:


 After having difficulty getting ddbg to work, I decided to write a
 dump function so I could easily see my data structures at runtime. The
 biggest part of it is a json encoder which can handle most D
 structures. If I recall, it only has trouble with enum's and pointers
 (since there's no length info).

 It's available in Yage's source here
 (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d)
 Yage is licensed LGPL but I grant permission to use this under the
 Boost 1.0 license if you or anyone else needs it.


 Thanks for this. I will take a look.


 However, for now it's only one-way and there's no deserialization part
 to it.


 Well, this part is left for the reader as an exercise ;-).


 It seems it also doesn't serialize members of parent classes, which can
 be a major caveat. I took a look at it yesterday but unfortunately
 wasn't able to figure it out.


You can have a look at my serialization library: 
http://dsource.org/projects/orange/ it could use some testing.

* It handles both serializing and deserializing
* It automatically serializes the base classes
* It supports events (before and after (de)serializing)
* It supports non-serialized fields (you can say that some fields in a 
class should not be serialized)
* It's licensed under the Boost license
* It's fairly std/runtime library independent (hopefully it's only the 
XMLArchive that is dependent on the runtime library, I've only tested it 
with Tango)
* You can create new archive types and use them with the existing serializer

* It currently only supports XML as the archive type but the library is 
built so you can create new archive types and use them with the existing 
serializer
* If you want to serialize objects through base class references you 
need to register a serialize function, everything else should be handled 
automatically

You can have a look at an example of usage at the project site.

-- 
/Jacob Carlborg
Jun 07 2010
parent Eric Poggel <dnewsgroup yage3d.net> writes:
On 6/7/2010 7:37 AM, Jacob Carlborg wrote:

 On 2010-06-06 21:32, Eric Poggel wrote:

 On 6/1/2010 5:31 PM, Robert M. Münch wrote:

 On 2010-06-01 02:13:18 +0200, Eric Poggel said:


 After having difficulty getting ddbg to work, I decided to write a
 dump function so I could easily see my data structures at runtime. The
 biggest part of it is a json encoder which can handle most D
 structures. If I recall, it only has trouble with enum's and pointers
 (since there's no length info).

 It's available in Yage's source here
 (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d)
 Yage is licensed LGPL but I grant permission to use this under the
 Boost 1.0 license if you or anyone else needs it.


 Thanks for this. I will take a look.


 However, for now it's only one-way and there's no deserialization part
 to it.


 Well, this part is left for the reader as an exercise ;-).


 It seems it also doesn't serialize members of parent classes, which can
 be a major caveat. I took a look at it yesterday but unfortunately
 wasn't able to figure it out.


 You can have a look at my serialization library:
 http://dsource.org/projects/orange/ it could use some testing.

 * It handles both serializing and deserializing
 * It automatically serializes the base classes
 * It supports events (before and after (de)serializing)
 * It supports non-serialized fields (you can say that some fields in a
 class should not be serialized)
 * It's licensed under the Boost license
 * It's fairly std/runtime library independent (hopefully it's only the
 XMLArchive that is dependent on the runtime library, I've only tested it
 with Tango)
 * You can create new archive types and use them with the existing
 serializer

 * It currently only supports XML as the archive type but the library is
 built so you can create new archive types and use them with the existing
 serializer
 * If you want to serialize objects through base class references you
 need to register a serialize function, everything else should be handled
 automatically

 You can have a look at an example of usage at the project site.


Thanks.  This looks pretty nice!
Jun 07 2010
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 30 May 2010 06:38:10 -0400, Robert <robert.muench robertmuench.de>  
wrote:


 Hi, are there are any projectes / snippets / prototypes where someone  
 tried to serializa D run-time data-structures (like an associative  
 array) into a file?

 --
 Robert M. Münch
 http://www.robertmuench.de


I've written a JSON serializer/de-serializer which uses compile time type  
reflection. Doesn't handle cycles or polymorphism yet, but I'd gladly  
share it if you (or others) are interested.
Jun 03 2010
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-04 02:27:56 +0200, Robert Jacques said:


 I've written a JSON serializer/de-serializer which uses compile time 
 type reflection. Doesn't handle cycles or polymorphism yet, but I'd 
 gladly share it if you (or others) are interested.


Thanks for the offer. AFAIK JSON is focused for wire-transfer, what I'm 
looking for is a way to efficiently write such structures into a file 
in binary form and re-create the run-time structure fast.

What I need is directed to a database / transactional system where 
millions of in/out operations happen.

-- 
Robert M. Münch
http://www.robertmuench.de
Jun 04 2010
parent "Robert Jacques" <sandford jhu.edu> writes:
On Fri, 04 Jun 2010 08:43:11 -0400, Robert M. Münch  
<robert.muench robertmuench.de> wrote:


 On 2010-06-04 02:27:56 +0200, Robert Jacques said:


 I've written a JSON serializer/de-serializer which uses compile time  
 type reflection. Doesn't handle cycles or polymorphism yet, but I'd  
 gladly share it if you (or others) are interested.


 Thanks for the offer. AFAIK JSON is focused for wire-transfer, what I'm  
 looking for is a way to efficiently write such structures into a file in  
 binary form and re-create the run-time structure fast.

 What I need is directed to a database / transactional system where  
 millions of in/out operations happen.


Pretty much all serialization methods have some amount of verification  
overhead, which it sounds like you neither want nor need. I'd recommend  
directly using .tupleof for structs/classes. Here's some simplified code  
snippets from my JSON library:

         // From toJson(T)(T value)
         static if(is(T == struct) || is(T == class)) {
             foreach(i,v;value.tupleof) {
                 result[getVariableName!(T,i)] = toJson(v);
             }
         }


         // From fromJson(T)(ref T dst)
         static if(is(T == struct) || is(T == class)) {
             static if(__traits(compiles, dst = new T() ))
                    if(dst is null)       dst = new T();
             static if(is(T == class)) enforce(dst !is null);
             foreach(i,v;dst.tupleof) { // Warning V is read-only because  
ref v doesn't compile
                 auto pJson = getVariableName!(T,i) in obj;
                 if(!pJson)   continue;
                 static if( is(typeof(v)==ulong) ) {
                     dst.tupleof[i] = roundTo!(long)(pJson.number);         
// Hack around bug 3418
                 } else {
                     pJson.fromJson( dst.tupleof[i] );
                 }
             }
         }
Jun 04 2010