www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - how to allocate large number of small object?

reply newbie d.com writes:
I need to allocate a large number (>10K) of small objects.  

class Small {
some field ...
}

Right now I call "new Small()" for each allocation, the overall performance of
the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.

char *pool = new char[ N * Small.size_of ];
char *next = pool;

Small myNew() {
Small obj = cast(Small)next;
next += Small.size_of;
}

is there any problem with it?  What's the correct way to do this in D?

Thanks.
Oct 04 2005
next sibling parent reply xs0 <xs0 xs0.com> writes:
newbie d.com wrote:

 I need to allocate a large number (>10K) of small objects.  
 
 class Small {
 some field ...
 }
 
 Right now I call "new Small()" for each allocation, the overall performance of
 the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.
 
 char *pool = new char[ N * Small.size_of ];
 char *next = pool;
 
 Small myNew() {
 Small obj = cast(Small)next;
 next += Small.size_of;
 }
 
 is there any problem with it?  What's the correct way to do this in D?
 
 Thanks.


You can write a custom allocator without any "hacks":
http://www.digitalmars.com/d/memory.html#newdelete

BTW, I think it should be Small.init.sizeof, Small.sizeof returns the 
size of a reference, not an instance..


xs0
Oct 05 2005
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <di05kl$a40$1 digitaldaemon.com>, xs0 says...

BTW, I think it should be Small.init.sizeof, Small.sizeof returns the 
size of a reference, not an instance..


Really?  But Small is the type name.  Classes are just a special case all around
it seems.


Sean
Oct 05 2005
prev sibling parent "Ben Hinkle" <bhinkle mathworks.com> writes:
 BTW, I think it should be Small.init.sizeof, Small.sizeof returns the size 
 of a reference, not an instance..


I think you mean Small.classinfo.init.length
Small.init.sizeof == Small.sizeof
Oct 05 2005
prev sibling next sibling parent pragma <pragma_member pathlink.com> writes:
In article <dhvr8g$vg$1 digitaldaemon.com>, newbie d.com says...

I need to allocate a large number (>10K) of small objects.  

class Small {
some field ...
}

Right now I call "new Small()" for each allocation, the overall performance of
the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.

char *pool = new char[ N * Small.size_of ];
char *next = pool;

Small myNew() {
Small obj = cast(Small)next;
next += Small.size_of;
}

is there any problem with it?  What's the correct way to do this in D?


Another solution apart from what has already been suggested:


 const uint POOL_SIZE = 1024*1024*5; // 5MB
 ubyte[POOL_SIZE] pool = void;
 ubyte* next = pool.ptr;
 Small obj = cast(Small)next;
 next += Small.size_of;


The above will get you a 5MB pool of memory, allocated to a ubyte array (which
is slightly better form than char[]).  The 'void' initalizer instructs the
compiler to not initalize the entire array to zero, which can cost you time for
big chunks of memory.

- EricAnderton at yahoo
Oct 05 2005
prev sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
<newbie d.com> wrote in message news:dhvr8g$vg$1 digitaldaemon.com...

I need to allocate a large number (>10K) of small objects.

 class Small {
 some field ...
 }

 Right now I call "new Small()" for each allocation, the overall 
 performance of
 the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.

 char *pool = new char[ N * Small.size_of ];
 char *next = pool;

 Small myNew() {
 Small obj = cast(Small)next;
 next += Small.size_of;
 }

 is there any problem with it?  What's the correct way to do this in D?

 Thanks.


Be careful about casting a random pointer to a Small. The resulting "object" 
won't have a vtable or monitor or be initialized so you won't be able to 
call any methods or synchronize on the object. If you don't care about 
either of those I suggesting making Small a struct.
Oct 05 2005
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <di0m9b$p23$1 digitaldaemon.com>, Ben Hinkle says...

<newbie d.com> wrote in message news:dhvr8g$vg$1 digitaldaemon.com...

I need to allocate a large number (>10K) of small objects.

 class Small {
 some field ...
 }

 Right now I call "new Small()" for each allocation, the overall 
 performance of
 the program is slow.  I'm wondering if I can use the C++ idiom in D, e.g.

 char *pool = new char[ N * Small.size_of ];
 char *next = pool;

 Small myNew() {
 Small obj = cast(Small)next;
 next += Small.size_of;
 }

 is there any problem with it?  What's the correct way to do this in D?

 Thanks.


Be careful about casting a random pointer to a Small. The resulting "object" 
won't have a vtable or monitor or be initialized so you won't be able to 
call any methods or synchronize on the object. If you don't care about 
either of those I suggesting making Small a struct. 


It should be simple enough to do a placement new on the memory block though...


Sean
Oct 05 2005
prev sibling parent reply newbie d.com writes:
In article <di0m9b$p23$1 digitaldaemon.com>, Ben Hinkle says...

Be careful about casting a random pointer to a Small. The resulting "object" 
won't have a vtable or monitor or be initialized so you won't be able to 
call any methods or synchronize on the object. If you don't care about 
either of those I suggesting making Small a struct. 


I cannot use struct.  So under this constraint, can I do something as I showed
earlier?  And what's the best way to make sure the *object* I got from the pool
is properly initialized?

In article <di05kl$a40$1 digitaldaemon.com>, xs0 says...

You can write a custom allocator without any "hacks":
http://www.digitalmars.com/d/memory.html#newdelete


I checked that page, same question: what's the magic to make sure the *object*
is properly initialize?

new() {
p = std.c.stdlib.malloc(sz);
gc.addRange(p, p + sz);
return p;
}

Thanks.
Oct 05 2005
parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
 I checked that page, same question: what's the magic to make sure the 
 *object*
 is properly initialize?

 new() {
 p = std.c.stdlib.malloc(sz);
 gc.addRange(p, p + sz);
 return p;
 }


I haven't had to ever use custom allocators so I haven't tried it myself but 
presumably from the description on 
http://www.digitalmars.com/d/class.html#allocators the result of new() is 
initialized.
Oct 05 2005