www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Getting core.exception.OutOfMemoryError error on allocating large

reply "Sparsh Mittal" <sparsh0mittal gmail.com> writes:
I am running


enum long DIM = 1024L * 1024L * 1024L* 8L ;
void main() {
   auto signal = new double[DIM];
}


and getting core.exception.OutOfMemoryError error.  One option is 
to use short/int, but I need to use double. Also, on using large 
arrays, computer becomes slow.

Is there no workaround at all, so that I can work on large 
arrays? Please let me know.
Mar 03 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 3 March 2013 at 14:05:02 UTC, Sparsh Mittal wrote:

 I am running


 enum long DIM = 1024L * 1024L * 1024L* 8L ;
 void main() {
   auto signal = new double[DIM];
 }


 and getting core.exception.OutOfMemoryError error.  One option 
 is to use short/int, but I need to use double. Also, on using 
 large arrays, computer becomes slow.

 Is there no workaround at all, so that I can work on large 
 arrays? Please let me know.


Assuming double.sizeof==8 on your machine, You're requesting 
1024*1024*1024*8*8 bytes = 68GB, do you have that much RAM 
available?
Mar 03 2013
next sibling parent reply "Sparsh Mittal" <sparsh0mittal gmail.com> writes:
 Assuming double.sizeof==8 on your machine, You're requesting 
 1024*1024*1024*8*8 bytes = 68GB, do you have that much RAM 
 available?


You are completely correct, however in C, one could do:

const long DIM = 1024L * 1024L * 1024L* 8L ;
int  main() {
    double signal[DIM];
  }

which runs fine. So, I was sort of looking for some solution like 
this.
Mar 03 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 3 March 2013 at 16:02:31 UTC, Sparsh Mittal wrote:

 Assuming double.sizeof==8 on your machine, You're requesting 
 1024*1024*1024*8*8 bytes = 68GB, do you have that much RAM 
 available?


 You are completely correct, however in C, one could do:

 const long DIM = 1024L * 1024L * 1024L* 8L ;
 int  main() {
    double signal[DIM];
  }

 which runs fine. So, I was sort of looking for some solution 
 like this.


For a start, that's not equivalent. The C version is creating a 
static array, the D version is a dynamic one. The C version is 
allocated on the stack, the D on the heap.

I'm pretty certain the C version doesn't actually work as 68GB 
will have completely clobbered the stack.
Mar 03 2013
parent reply "zorran" <zorran tut.by> writes:
on my machine (core i7, 16 gb ram, win7/64)
next code written core.exception.OutOfMemoryError:
enum long size= 1300_000_000;
auto arr = new byte[size];
		
but next code work fine:
enum long size= 1300_000_000;
byte * p = cast(byte *) malloc(size);
		
i compiled in 64 bit mode
i use keys: "dmc -c -m64 test.d"
Aug 18 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 18 August 2013 at 12:07:02 UTC, zorran wrote:

 on my machine (core i7, 16 gb ram, win7/64)
 next code written core.exception.OutOfMemoryError:
 enum long size= 1300_000_000;
 auto arr = new byte[size];
 		
 but next code work fine:
 enum long size= 1300_000_000;
 byte * p = cast(byte *) malloc(size);
 		
 i compiled in 64 bit mode
 i use keys: "dmc -c -m64 test.d"


Interesting... What happens if you use core.memory.GC.malloc?
Aug 18 2013
parent reply "zorran" <zorran tut.by> writes:
 Interesting... What happens if you use core.memory.GC.malloc?


enum long size= 1300_000_000;		
byte * p = cast(byte *) malloc(size);
	
for(int i=0; i<size; i++)
	p[i]=1;

ulong sum=0;	
for(int i=0; i<size; i++)
    sum += p[i];		
		
writef("%d ", sum); // here written 1300000000
Aug 18 2013
next sibling parent "zorran" <zorran tut.by> writes:
On Sunday, 18 August 2013 at 12:40:42 UTC, zorran wrote:

 Interesting... What happens if you use core.memory.GC.malloc?




i am using in sample import std.c.stdlib;
GC.malloc also written core.exception.OutOfMemoryError
Aug 18 2013
prev sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 18 August 2013 at 12:40:42 UTC, zorran wrote:

 Interesting... What happens if you use core.memory.GC.malloc?


 enum long size= 1300_000_000;		
 byte * p = cast(byte *) malloc(size);
 	
 for(int i=0; i<size; i++)
 	p[i]=1;

 ulong sum=0;	
 for(int i=0; i<size; i++)
    sum += p[i];		
 		
 writef("%d ", sum); // here written 1300000000
 		


Well that proves malloc is actually allocating the memory.

I'd say file a bug report. This should definitely work.
Aug 18 2013
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 3 March 2013 at 15:52:49 UTC, John Colvin wrote:

 On Sunday, 3 March 2013 at 14:05:02 UTC, Sparsh Mittal wrote:

 I am running


 enum long DIM = 1024L * 1024L * 1024L* 8L ;
 void main() {
  auto signal = new double[DIM];
 }


 and getting core.exception.OutOfMemoryError error.  One option 
 is to use short/int, but I need to use double. Also, on using 
 large arrays, computer becomes slow.

 Is there no workaround at all, so that I can work on large 
 arrays? Please let me know.


 Assuming double.sizeof==8 on your machine, You're requesting 
 1024*1024*1024*8*8 bytes = 68GB, do you have that much RAM 
 available?


Depending on your OS, you will of course have to take in to 
account page file limits and swap partition sizes.    
Nonetheless, 68GB is a huge amount of memory to have in a single 
array, that's 8.6 billion numbers. In reality I doubt you will 
need that many.
Mar 03 2013
parent "Sparsh Mittal" <sparsh0mittal gmail.com> writes:
Thanks. Yes, you are right. I will change my program.
Mar 03 2013