www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - data.d

reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
I am very bad at naming things.

This is a module containing classes to manage raw data in external memory.  
It provides semantics similar to built-in void[] arrays, but has the  
following advantages:

* Faster allocation and deallocation, since memory is requested from the  
OS directly as whole pages
* Greatly reduced chance of memory leaks due to stray pointers
* Overall improved GC performance due to reduced size of managed heap
* Memory is immediately returned to the OS when data is deallocated

I've been using it in my own programs, and it saved me from many headaches.
Personally, I think that it is extremely useful for any applications that  
deal with non-trivial amounts of raw data - web/file servers, multimedia  
applications, video games, etc., and belongs in the standard library.

Currently D1/Phobos, but should be easy to port to D2 or Tango.
I believe it can be made memory-safe by removing deleteContents and fixing  
the append clobber issue.

Source and more info here: http://github.com/CyberShadow/data.d

If you see anything that can be improved, feel free to fork the github  
repo, or post patches, or just comment on it.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 14 2010
parent reply "JimBob" <jim bob.com> writes:
"Vladimir Panteleev" <vladimir thecybershadow.net> wrote in message 
news:op.vfvdtwgwtuzx1w 89-28-59-99.starnet.md...

 If you see anything that can be improved, feel free to fork the github 
 repo, or post patches, or just comment on it.


VirtualAlloc returns chunks that have 'dwAllocationGranularity' granularity, 
which is 64K on every Windows OS I've used. So allocating a page, 4K, will 
actualy get you 64K.

So using VirtualAlloc as a replacement for malloc is very wasteful unless 
the allocations are larger that 64K.

You might want to look at HeapCreate and it's siblings. (on windows anyway)
Jul 15 2010
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thu, 15 Jul 2010 11:03:57 +0300, JimBob <jim bob.com> wrote:


 VirtualAlloc returns chunks that have 'dwAllocationGranularity'  
 granularity,
 which is 64K on every Windows OS I've used. So allocating a page, 4K,  
 will
 actualy get you 64K.

 So using VirtualAlloc as a replacement for malloc is very wasteful unless
 the allocations are larger that 64K.

 You might want to look at HeapCreate and it's siblings. (on windows  
 anyway)


dwAllocationGranularity only controls the granularity of the allocation  
address, but not of the allocation size. A simple program that calls  
VirtualAlloc 16384 times uses up 1 GB of the address space, but only 64 MB  
of actual memory. So, while it is a waste of address space, it's not a  
waste of system memory.

I think that using heaps of any kind defeats part of this module's  
purpose, for the reason that due to heap fragmentation it is not possible  
to return memory to the OS directly (which is only possible to do in whole  
pages). There is also a small performance cost involved with using heaps.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 15 2010
next sibling parent "JimBob" <jim bob.com> writes:
"Vladimir Panteleev" <vladimir thecybershadow.net> wrote in message 
news:op.vfvnf1bvtuzx1w 89-28-59-99.starnet.md...

 On Thu, 15 Jul 2010 11:03:57 +0300, JimBob <jim bob.com> wrote:


 VirtualAlloc returns chunks that have 'dwAllocationGranularity' 
 granularity,
 which is 64K on every Windows OS I've used. So allocating a page, 4K, 
 will
 actualy get you 64K.

 So using VirtualAlloc as a replacement for malloc is very wasteful unless
 the allocations are larger that 64K.

 You might want to look at HeapCreate and it's siblings. (on windows 
 anyway)


 dwAllocationGranularity only controls the granularity of the allocation 
 address, but not of the allocation size. A simple program that calls 
 VirtualAlloc 16384 times uses up 1 GB of the address space, but only 64 MB 
 of actual memory. So, while it is a waste of address space, it's not a 
 waste of system memory.


Appologies, msdn says you're right.

Which sucks cause I recently rewrote a bunch of code because of that 
misunderstanding.

Thanks.
Jul 15 2010
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Vladimir Panteleev wrote:

 On Thu, 15 Jul 2010 11:03:57 +0300, JimBob <jim bob.com> wrote:
 

 VirtualAlloc returns chunks that have 'dwAllocationGranularity' 
 granularity,
 which is 64K on every Windows OS I've used. So allocating a page, 4K, 
 will
 actualy get you 64K.

 So using VirtualAlloc as a replacement for malloc is very wasteful unless
 the allocations are larger that 64K.

 You might want to look at HeapCreate and it's siblings. (on windows 
 anyway)


 
 dwAllocationGranularity only controls the granularity of the allocation 
 address, but not of the allocation size. A simple program that calls 
 VirtualAlloc 16384 times uses up 1 GB of the address space, but only 64 
 MB of actual memory. So, while it is a waste of address space, it's not 
 a waste of system memory.


In a 32-bit process, the waste of address space is sometimes more 
critical nowadays. There is only 2GB virtual memory available (3GB with 
some tweaks), and with the example above, VirtualAlloc fails well before 
allocating 128 MB.
Jul 15 2010
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Rainer Schuetze (r.sagitario gmx.de)'s article

 Vladimir Panteleev wrote:

 On Thu, 15 Jul 2010 11:03:57 +0300, JimBob <jim bob.com> wrote:


 VirtualAlloc returns chunks that have 'dwAllocationGranularity'
 granularity,
 which is 64K on every Windows OS I've used. So allocating a page, 4K,
 will
 actualy get you 64K.

 So using VirtualAlloc as a replacement for malloc is very wasteful unless
 the allocations are larger that 64K.

 You might want to look at HeapCreate and it's siblings. (on windows
 anyway)


 dwAllocationGranularity only controls the granularity of the allocation
 address, but not of the allocation size. A simple program that calls
 VirtualAlloc 16384 times uses up 1 GB of the address space, but only 64
 MB of actual memory. So, while it is a waste of address space, it's not
 a waste of system memory.


 In a 32-bit process, the waste of address space is sometimes more
 critical nowadays. There is only 2GB virtual memory available (3GB with
 some tweaks), and with the example above, VirtualAlloc fails well before
 allocating 128 MB.


Yea, but I wonder how much longer it is going to be before 32-bit is dead as a
dodo except on things like netbooks.  Frankly, it's about time for it to die,
because dealing w/ address space limitations when plenty of physical memory is
available just plain sucks.  DMD is in the process of being ported to 64-bit as
we
speak.  Almost any Linux distro is available in 64-bit.  IIRC (I can't remember
for sure where I heard this) Windows 7 will be the last 32-bit version, and it
seems like most people who have migrated use the 64-bit version.

On the hardware end, 64-bit is now 6-7 years old, which has to be a standard
deviation or two older than the average age at which computers get replaced.
Jul 15 2010
next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 16 Jul 2010 01:58:27 +0300, dsimcha <dsimcha yahoo.com> wrote:


 On the hardware end, 64-bit is now 6-7 years old, which has to be a  
 standard  deviation or two older than the average age at which computers  
 get replaced.


FWIW, AFAIK data.d is 64-bit ready :D

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 15 2010
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"dsimcha" <dsimcha yahoo.com> wrote in message 
news:i1o3qj$13b4$1 digitalmars.com...

 Yea, but I wonder how much longer it is going to be before 32-bit is dead 
 as a
 dodo except on things like netbooks.  Frankly, it's about time for it to 
 die,
 because dealing w/ address space limitations when plenty of physical 
 memory is
 available just plain sucks.


I have 1GB. (And I get by just fine.)
Jul 15 2010
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 16 Jul 2010 02:41:17 +0300, Nick Sabalausky <a a.a> wrote:


 I have 1GB. (And I get by just fine.)


The discussion wasn't about physical memory, but address space. Due to the  
discussed limitation, you won't be able to fill all of that 1 GB with  
small VirtualAlloc'd objects because you'll run out of address space much  
earlier before you run out of memory. Having more address space than  
physical memory can thus sometimes be beneficial (also e.g. for virtual  
memory).

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 15 2010
prev sibling parent dennis luehring <dl.soluz gmx.net> writes:
 Yea, but I wonder how much longer it is going to be before 32-bit is dead as a
 dodo except on things like netbooks.


it "consumes" (leaves holes useable by others) to much memory because of 
the allocation strategie - that is also a problem under 64bit
Jul 15 2010
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 16 Jul 2010 01:20:07 +0300, Rainer Schuetze <r.sagitario gmx.de>  
wrote:


 There is only 2GB virtual memory available (3GB with some tweaks)


The allocation granularity doesn't affect virtual memory either (at least  
according to all Process Explorer indications).

Does this 2 or 3GB limitation only affect 32-bit operating systems? On my  
64-bit Windows, with /LARGEADDRESSAWARE, a simple program can do close to  
64K (65062 for me) 1-byte VirtualAllocs.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 15 2010
next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 16 Jul 2010 02:20:16 +0300, Vladimir Panteleev  
<vladimir thecybershadow.net> wrote:


 Does this 2 or 3GB limitation only affect 32-bit operating systems? On  
 my 64-bit Windows, with /LARGEADDRESSAWARE, a simple program can do  
 close to 64K (65062 for me) 1-byte VirtualAllocs.


Said simple program in case anyone wants to test, excuse the C:

#include <windows.h>
#include <stdio.h>

#define N (64*1024)

void main()
{
	int i;
	char s[10];

	gets(s);
	for (i=0; i<N; i++)
	{
		void *p = VirtualAlloc(NULL, 1, MEM_COMMIT, PAGE_READWRITE);
		if (p==NULL)
		{
			printf("Allocation failed at %d\n", i);
			break;
		}
		*(char*)p = 17;
		if (i==0 || i==N-1 || p==NULL)
			printf("%p\n", p);
	}
	gets(s);
}

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 15 2010
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Vladimir Panteleev wrote:

 On Fri, 16 Jul 2010 01:20:07 +0300, Rainer Schuetze <r.sagitario gmx.de> 
 wrote:
 

 There is only 2GB virtual memory available (3GB with some tweaks)


 
 The allocation granularity doesn't affect virtual memory either (at 
 least according to all Process Explorer indications).


But the alignment requirements disallow allocating memory that fills the 
wasted address space. I guess even HeapAllocate will use VirtualAlloc as 
its back end.


 
 Does this 2 or 3GB limitation only affect 32-bit operating systems? On 
 my 64-bit Windows, with /LARGEADDRESSAWARE, a simple program can do 
 close to 64K (65062 for me) 1-byte VirtualAllocs.
 


Normal 32-bit processes have 2 GB address space. Processes with 
/LARGEADDRESSAWARE set in the executable get 3GB on 32-bit OS (XP needs 
to boot with the /3GB option), and almost 4GB on 64-bit OS.
Jul 15 2010
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 16 Jul 2010 09:19:09 +0300, Rainer Schuetze <r.sagitario gmx.de>  
wrote:


 But the alignment requirements disallow allocating memory that fills the  
 wasted address space. I guess even HeapAllocate will use VirtualAlloc as  
 its back end.


Doesn't this apply only to virtual (user-space) memory? If so, then the  
problem is address space exhaustion as already discussed. The difference  
is whether any physical memory is wasted, and it doesn't seem to be the  
case.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 16 2010