www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Allocating many object

reply Frank Benoit <frank nix.de> writes:
class Test
{
    int[10] values;
}


void main()
{
    Test[1000_000] arr;
    printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
    printf( "sizeof Array: %d\n\x00", arr.sizeof );
    for( int j = 0; j < 1000; j++ )
    {
        for( int i = 0; i < 1000; i++ )
        {
            arr[j*1000+i] = new Test;
        }
        printf( "loop %d\n\x00", j );
    }
}

The program runs and gets slower and slower.
After 8min at loop 700 I canceled.
Is this speed acceptable for allocating 48MB (+4MB for the array)?

I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take
endless.

Is there a workaround? e.g. preallocating memory in one piece.
Mar 27 2006
next sibling parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Frank Benoit wrote:

 class Test
 {
     int[10] values;
 }
 
 
 void main()
 {
     Test[1000_000] arr;
     printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
     printf( "sizeof Array: %d\n\x00", arr.sizeof );
     for( int j = 0; j < 1000; j++ )
     {
         for( int i = 0; i < 1000; i++ )
         {
             arr[j*1000+i] = new Test;
         }
         printf( "loop %d\n\x00", j );
     }
 }
 
 The program runs and gets slower and slower.
 After 8min at loop 700 I canceled.
 Is this speed acceptable for allocating 48MB (+4MB for the array)?
 
 I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take
 endless.
 
 Is there a workaround? e.g. preallocating memory in one piece.


I used the following re-write of your program:





















Do note I dropped the size of 'arr' from 1000000 to 100000.  This has nothing
to do with 
with the issue, so far as I know, and everything to do with a stack size
problem on the 
machine I'm sitting at right now.  Don't ask... pos.

Anyhow, the Timer reports from about thirty or so runs of that program spanned
between 
0.66 seconds and 0.77 seconds total, so I shouldn't expect your original to
take more than 
a second or two.

-- Chris Nicholson-Sauls
Mar 27 2006
prev sibling next sibling parent reply Frank Benoit <frank nix.de> writes:
To see the difference to Java...

public class Alloc {

	int[] val;
	public Alloc()
	{
		val = new int[10];
	}
	
	public static void main(String[] args) {
		Alloc[] arr = new Alloc[1000000];
		for( int j = 0; j < 1000; j++ )
		{
	        	for( int i = 0; i < 1000; i++ )
	        	{
	            		arr[j*1000+i] = new Alloc();
	        	}
	        	System.out.println( "loop "+j );
	    	}
		System.out.println( "complete " );
	}
}


Note, this Java program make a second allocation for the array and does
also initialize the memory.

The program was finished after ONE second.
This 1000 times faster like the D program.

So I have to say that again... The GC is a show stopper and need to be
changed.
Mar 27 2006
parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Frank Benoit wrote:

 	int[] val;
 	public Alloc()
 	{
 		val = new int[10];
 	}


Just for the record, you can use the same construct above in D.  In fact, you
could 
technically do such silliness as:


Madness.  But occasionally useful madness.

-- Chris Nicholson-Sauls
Mar 27 2006
prev sibling next sibling parent reply Frank Benoit <frank nix.de> writes:
I am completely sorry.
I used a wrong verson of libphobos.
With the original one it works fast.

Frank
Mar 27 2006
parent reply Frank Benoit <frank nix.de> writes:
Frank Benoit schrieb:

 I am completely sorry.
 I used a wrong verson of libphobos.
 With the original one it works fast.
 
 Frank


BTW: The only difference was in the makefiles:

original: DFLAGS = -O -inline -release
My one  : DFLAGS = -g -debug

With the original options and an own compiled libphobos.a it is also
completelly OK. ~500ms
Mar 27 2006
parent reply "Andrew Fedoniouk" <news terrainformatica.com> writes:
"Frank Benoit" <frank nix.de> wrote in message 
news:e09ju8$28nc$1 digitaldaemon.com...

 Frank Benoit schrieb:

 I am completely sorry.
 I used a wrong verson of libphobos.
 With the original one it works fast.

 Frank


 BTW: The only difference was in the makefiles:

 original: DFLAGS = -O -inline -release
 My one  : DFLAGS = -g -debug

 With the original options and an own compiled libphobos.a it is also
 completelly OK. ~500ms


By the way:
SGML tokenizer (aka HTML/XML pull parser) in Harmonia is
capable to scan 30mb XML file in one second. It depends on hardware but
order of magnitude is like this.
It does not allocate any memory while parsing.

Andrew.
Mar 27 2006
parent reply ZZ <ZZ zz.com> writes:
Andrew Fedoniouk wrote:

 By the way:
 SGML tokenizer (aka HTML/XML pull parser) in Harmonia is
 capable to scan 30mb XML file in one second. It depends on hardware but
 order of magnitude is like this.
 It does not allocate any memory while parsing.
 
 Andrew. 
 
 


Just looked at it briefly and I noticed that Scanner seems to only take 
in a string. Do you plan on adding a file stream as input?

Zz
Mar 28 2006
parent "Andrew Fedoniouk" <news terrainformatica.com> writes:
"ZZ" <ZZ zz.com> wrote in message news:e0ba8l$20k5$1 digitaldaemon.com...

 Andrew Fedoniouk wrote:

 By the way:
 SGML tokenizer (aka HTML/XML pull parser) in Harmonia is
 capable to scan 30mb XML file in one second. It depends on hardware but
 order of magnitude is like this.
 It does not allocate any memory while parsing.

 Andrew.


 Just looked at it briefly and I noticed that Scanner seems to only take in 
 a string. Do you plan on adding a file stream as input?


If you need to scan file then you can use MMFile.
module harmonia.io.mmfile;
( http://harmonia.terrainformatica.com )

or the one from phobos.

For stream parsing you need to create your own scanner and override

StreamScanner:Scanner
{
   get_char()
   switch_input()
   currentFragment() (optional, error reporting)
}


 Zz
Mar 28 2006
prev sibling parent Derek Parnell <derek psych.ward> writes:
On Mon, 27 Mar 2006 21:05:55 +0200, Frank Benoit wrote:


 class Test
 {
     int[10] values;
 }
 
 void main()
 {
     Test[1000_000] arr;
     printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
     printf( "sizeof Array: %d\n\x00", arr.sizeof );
     for( int j = 0; j < 1000; j++ )
     {
         for( int i = 0; i < 1000; i++ )
         {
             arr[j*1000+i] = new Test;
         }
         printf( "loop %d\n\x00", j );
     }
 }
 
 The program runs and gets slower and slower.
 After 8min at loop 700 I canceled.
 Is this speed acceptable for allocating 48MB (+4MB for the array)?
 
 I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take
 endless.
 
 Is there a workaround? e.g. preallocating memory in one piece.


I made a slight change to the code and it worked well.

I moved 'Test[1000_000] arr;' to the module level so it wouldn't be stack
allocated, and I changed it to 'Test[1000_000] arr = void;' to avoid
preinitializing it.

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocracy!"
28/03/2006 10:30:30 AM
Mar 27 2006