www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Clear big AAs

reply useo <useo6 start.bg> writes:
Hi,

I'm trying to clear big associative arrays, but I always get an
object error - what I currently doing is:

private string[uint] myAA;

void main() {

   fill(myAA);
   myAA.clear();
   fill(myAA); // object.Error: Access Violation
}

void fill(string[uint] aa) {
   for (uint i = 0; i < 10_000_000_000; i++) {
      myAA[i] = std.conv.to!(string)(i);
   }
}

I already saw the bug report at http://www.digitalmars.com/d/archives/
digitalmars/D/bugs/
Issue_5683_New_Calling_.clear_on_a_fresh_associative_array_causes_subsequent_segfault_28632.html
and I'm also using 2.052 - as I saw at the change-log of 2.053 it's
also unfixed. So... is there any solution to clear associative arrays
without memory leaking?
May 31 2011
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
useo:


 So... is there any solution to clear associative arrays
 without memory leaking?


What about remove an item at a time inside a loop (iterations on the keys
array), and then doing a rehash?

Bye,
bearophile
May 31 2011
prev sibling parent reply David Nadlinger <see klickverbot.at> writes:
I realize that this might sound strange, but try setting myAA to null 
after clear(), this should fix the crash.

David


On 5/31/11 4:00 PM, useo wrote:

 Hi,

 I'm trying to clear big associative arrays, but I always get an
 object error - what I currently doing is:

 private string[uint] myAA;

 void main() {

     fill(myAA);
     myAA.clear();
     fill(myAA); // object.Error: Access Violation
 }

 void fill(string[uint] aa) {
     for (uint i = 0; i<  10_000_000_000; i++) {
        myAA[i] = std.conv.to!(string)(i);
     }
 }

 I already saw the bug report at http://www.digitalmars.com/d/archives/
 digitalmars/D/bugs/
 Issue_5683_New_Calling_.clear_on_a_fresh_associative_array_causes_subsequent_segfault_28632.html
 and I'm also using 2.052 - as I saw at the change-log of 2.053 it's
 also unfixed. So... is there any solution to clear associative arrays
 without memory leaking?
May 31 2011
parent reply useo <useo6 start.bg> writes:
== Auszug aus David Nadlinger (see klickverbot.at)'s Artikel

 I realize that this might sound strange, but try setting myAA to


null

 after clear(), this should fix the crash.
 David
 On 5/31/11 4:00 PM, useo wrote:

 Hi,

 I'm trying to clear big associative arrays, but I always get an
 object error - what I currently doing is:

 private string[uint] myAA;

 void main() {

     fill(myAA);
     myAA.clear();
     fill(myAA); // object.Error: Access Violation
 }

 void fill(string[uint] aa) {
     for (uint i = 0; i<  10_000_000_000; i++) {
        myAA[i] = std.conv.to!(string)(i);
     }
 }

 I already saw the bug report at http://www.digitalmars.com/d/




archives/

 digitalmars/D/bugs/




Issue_5683_New_Calling_.clear_on_a_fresh_associative_array_causes_subsequent_segfault_28632.html

 and I'm also using 2.052 - as I saw at the change-log of 2.053




it's

 also unfixed. So... is there any solution to clear associative




arrays

 without memory leaking?




Thanks a lot,

I solved it by using

pGraphics.clear();
pGraphics = null;
// pGraphics.rehash();

I thought this'll solve my problem, but it doesn't... I've to fill my
AA every 500 ms with around 250 values (in principle much less than
10_000_000_000). When I start my application (in this context, my
game), it works as needed (around 2000 fps/s) but after some seconds,
the fps-rate drops down to 1/s and back to normal (and again, and
again...). When I remove the command to clear the array, it works
without dropping down to 1 fps. I this case I can't update my screen
as needed. I also tried to delete the class instance which contains
the array and create an new but it's the same as before (it drops
down to 1 fps). Is there anything I forgot to consider?
Jun 01 2011
parent reply bearophile <bearophileHUGS lycos.com> writes:
useo:


 Is there anything I forgot to consider?


If the key and values are primitive values or structs of primitive values then
you may try another AA implementation that doesn't use the GC.

Bye,
bearophile
Jun 01 2011
parent reply useo <useo6 start.bg> writes:
 useo:

 Is there anything I forgot to consider?


 If the key and values are primitive values or structs of primitive


values then you may try another AA implementation that doesn't use
the GC.

 Bye,
 bearophile


I tried some different implementations and I also reduced the size of
my arrays without any success :(. It always drops down to 1 FPS for 1
second and after that it runs for some seconds normal until next 1
FPS-second.
I also tried to do the work within a separate thread, also without
success. My current code looks like:

toRender.clear();
toRender = null;

for (int y = pFromY; y < pToY && y < pTiles.length; y++) {
	for (int x = pFromX; x < pToX && x < pTiles[y].length; x++) {
		toRender[pTiles[y][x].texture] ~= pTiles[y][x];
	}
}

toRender is my AA which contains the textures as key (instances of my
class "Texture") and the tiles (position of them) as values. When I
remove the first two lines (clear and set null) it doesn't drops down
to 1 FPS, it runs normal.

I hope anyone know a solution :)
Jun 12 2011
parent bearophile <bearophileHUGS lycos.com> writes:
useo:


 toRender is my AA which contains the textures as key (instances of my
 class "Texture") and the tiles (position of them) as values. When I
 remove the first two lines (clear and set null) it doesn't drops down
 to 1 FPS, it runs normal.
 
 I hope anyone know a solution :)


Try to disable the GC before that double loop and enable after it. Generally
those appends create lot of garbage, so it's not something to do in a
performance critical part of the program (try to use fixed-sized arrays, or
structs with a fixed-sized array plus a length). You can also try an AA of
appender. Also try to profile your code, and take a look at GC stats.

Bye,
bearophile
Jun 13 2011