www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Clear big AAs

reply useo <useo6 start.bg> writes:
Hi,

I'm trying to clear big associative arrays, but I always get an
object error - what I currently doing is:

private string[uint] myAA;

void main() {

   fill(myAA);
   myAA.clear();
   fill(myAA); // object.Error: Access Violation
}

void fill(string[uint] aa) {
   for (uint i = 0; i < 10_000_000_000; i++) {
      myAA[i] = std.conv.to!(string)(i);
   }
}

I already saw the bug report at http://www.digitalmars.com/d/archives/
digitalmars/D/bugs/
Issue_5683_New_Calling_.clear_on_a_fresh_associative_array_causes_subsequent_segfault_28632.html
and I'm also using 2.052 - as I saw at the change-log of 2.053 it's
also unfixed. So... is there any solution to clear associative arrays
without memory leaking?
May 31 2011
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
useo:

 So... is there any solution to clear associative arrays
 without memory leaking?
What about remove an item at a time inside a loop (iterations on the keys array), and then doing a rehash? Bye, bearophile
May 31 2011
prev sibling parent reply David Nadlinger <see klickverbot.at> writes:
I realize that this might sound strange, but try setting myAA to null 
after clear(), this should fix the crash.

David


On 5/31/11 4:00 PM, useo wrote:
 Hi,

 I'm trying to clear big associative arrays, but I always get an
 object error - what I currently doing is:

 private string[uint] myAA;

 void main() {

     fill(myAA);
     myAA.clear();
     fill(myAA); // object.Error: Access Violation
 }

 void fill(string[uint] aa) {
     for (uint i = 0; i<  10_000_000_000; i++) {
        myAA[i] = std.conv.to!(string)(i);
     }
 }

 I already saw the bug report at http://www.digitalmars.com/d/archives/
 digitalmars/D/bugs/
 Issue_5683_New_Calling_.clear_on_a_fresh_associative_array_causes_subsequent_segfault_28632.html
 and I'm also using 2.052 - as I saw at the change-log of 2.053 it's
 also unfixed. So... is there any solution to clear associative arrays
 without memory leaking?
May 31 2011
parent reply useo <useo6 start.bg> writes:
== Auszug aus David Nadlinger (see klickverbot.at)'s Artikel
 I realize that this might sound strange, but try setting myAA to
null
 after clear(), this should fix the crash.
 David
 On 5/31/11 4:00 PM, useo wrote:
 Hi,

 I'm trying to clear big associative arrays, but I always get an
 object error - what I currently doing is:

 private string[uint] myAA;

 void main() {

     fill(myAA);
     myAA.clear();
     fill(myAA); // object.Error: Access Violation
 }

 void fill(string[uint] aa) {
     for (uint i = 0; i<  10_000_000_000; i++) {
        myAA[i] = std.conv.to!(string)(i);
     }
 }

 I already saw the bug report at http://www.digitalmars.com/d/
archives/
 digitalmars/D/bugs/
Issue_5683_New_Calling_.clear_on_a_fresh_associative_array_causes_subsequent_segfault_28632.html
 and I'm also using 2.052 - as I saw at the change-log of 2.053
it's
 also unfixed. So... is there any solution to clear associative
arrays
 without memory leaking?
Thanks a lot, I solved it by using pGraphics.clear(); pGraphics = null; // pGraphics.rehash(); I thought this'll solve my problem, but it doesn't... I've to fill my AA every 500 ms with around 250 values (in principle much less than 10_000_000_000). When I start my application (in this context, my game), it works as needed (around 2000 fps/s) but after some seconds, the fps-rate drops down to 1/s and back to normal (and again, and again...). When I remove the command to clear the array, it works without dropping down to 1 fps. I this case I can't update my screen as needed. I also tried to delete the class instance which contains the array and create an new but it's the same as before (it drops down to 1 fps). Is there anything I forgot to consider?
Jun 01 2011
parent reply bearophile <bearophileHUGS lycos.com> writes:
useo:

 Is there anything I forgot to consider?
If the key and values are primitive values or structs of primitive values then you may try another AA implementation that doesn't use the GC. Bye, bearophile
Jun 01 2011
parent reply useo <useo6 start.bg> writes:
 useo:
 Is there anything I forgot to consider?
If the key and values are primitive values or structs of primitive
values then you may try another AA implementation that doesn't use the GC.
 Bye,
 bearophile
I tried some different implementations and I also reduced the size of my arrays without any success :(. It always drops down to 1 FPS for 1 second and after that it runs for some seconds normal until next 1 FPS-second. I also tried to do the work within a separate thread, also without success. My current code looks like: toRender.clear(); toRender = null; for (int y = pFromY; y < pToY && y < pTiles.length; y++) { for (int x = pFromX; x < pToX && x < pTiles[y].length; x++) { toRender[pTiles[y][x].texture] ~= pTiles[y][x]; } } toRender is my AA which contains the textures as key (instances of my class "Texture") and the tiles (position of them) as values. When I remove the first two lines (clear and set null) it doesn't drops down to 1 FPS, it runs normal. I hope anyone know a solution :)
Jun 12 2011
parent bearophile <bearophileHUGS lycos.com> writes:
useo:

 toRender is my AA which contains the textures as key (instances of my
 class "Texture") and the tiles (position of them) as values. When I
 remove the first two lines (clear and set null) it doesn't drops down
 to 1 FPS, it runs normal.
 
 I hope anyone know a solution :)
Try to disable the GC before that double loop and enable after it. Generally those appends create lot of garbage, so it's not something to do in a performance critical part of the program (try to use fixed-sized arrays, or structs with a fixed-sized array plus a length). You can also try an AA of appender. Also try to profile your code, and take a look at GC stats. Bye, bearophile
Jun 13 2011