www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Cleaning/Releasing large arrays from memory

reply Clinton <clinton.d.skakun gmail.com> writes:
Hi guys, I have a question on how to free large arrays in D after 
they're no longer needed.

Let's say I have this:


SomeKey[] getKeys() {
   SomeKey[] n;

   foreach(batching stuff...) {
    SomeKey newstuff = db.select!(SomeKey[])(...); // gets around 
6M of these from db

    redis.send("newItems", newStuff.length);

    n ~= newStuff;

    destroy(newStuff);
   }

   return n;
}

void main() {
   SomeKey[] myarray = getKeys();
   ulong[string] mappedValues;

   foreach(i, item; myArray) {
    mappedValues[item.name] = i;
   }

   // No longer need myarray, and we need the memory back
   destory(myarray);
   GC.collect;

   // Another memory intensive operation below
}


This seems to work for me to a point, but I notice my program 
still holds a lot in memory even after certain memory intensive 
arrays should have been collected.

Is this just a hit and miss with the GC or am I doing something 
wrong? Uses around 40% less memory forcing the GC and cleaning 
arrays but seems like a lot is left over, especially for threads 
that have already finished(I've expect a thread to free all it's 
memory). I'm not sharing anything across threads besides a config 
struct.

This is a simplified version of my app. At the moment when at 
around 9GB at it's peak instead of 19GB like before.
Jul 24 2017
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
Stuff the GC.
You don't need it to care about collecting (or destroying for that matter).

Tell it to free[0] the array directly.

```D
T[] array;

GC.free(array.ptr);
```

Normally I would suggest to create your own buffer, but because of the 
DB library probably doesn't support that, no point trying to force it.
Jul 24 2017
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
rikki cattermole wrote:


 Tell it to free[0] the array directly.

 ```D
 T[] array;

 GC.free(array.ptr);


or just `delete arr;`. it is marked as "deprecated" in changelog, but who 
cares? it works.
Jul 24 2017
prev sibling next sibling parent Clinton <clinton.d.skakun gmail.com> writes:
On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:

 Stuff the GC.
 You don't need it to care about collecting (or destroying for 
 that matter).

 Tell it to free[0] the array directly.

 ```D
 T[] array;

 GC.free(array.ptr);
 ```

 Normally I would suggest to create your own buffer, but because 
 of the DB library probably doesn't support that, no point 
 trying to force it.




Thanks! I didn't know I could do that with the GC enabled.
Jul 24 2017
prev sibling parent reply Clinton <clinton.d.skakun gmail.com> writes:
On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:

 Stuff the GC.
 You don't need it to care about collecting (or destroying for 
 that matter).

 Tell it to free[0] the array directly.

 ```D
 T[] array;

 GC.free(array.ptr);
 ```

 Normally I would suggest to create your own buffer, but because 
 of the DB library probably doesn't support that, no point 
 trying to force it.




BTW, how would I do this with associative arrays?
Jul 24 2017
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
Clinton wrote:


 On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole wrote:

 Stuff the GC.
 You don't need it to care about collecting (or destroying for that 
 matter).

 Tell it to free[0] the array directly.

 ```D
 T[] array;

 GC.free(array.ptr);
 ```

 Normally I would suggest to create your own buffer, but because of the 
 DB library probably doesn't support that, no point trying to force it.




 BTW, how would I do this with associative arrays?


you can't. you can `.clear` 'em, though, but this won't immediately free 
the internal storage.
Jul 24 2017
parent Clinton <clinton.d.skakun gmail.com> writes:
On Monday, 24 July 2017 at 18:01:19 UTC, ketmar wrote:

 Clinton wrote:


 On Monday, 24 July 2017 at 14:51:04 UTC, rikki cattermole 
 wrote:

 Stuff the GC.
 You don't need it to care about collecting (or destroying for 
 that matter).

 Tell it to free[0] the array directly.

 ```D
 T[] array;

 GC.free(array.ptr);
 ```

 Normally I would suggest to create your own buffer, but 
 because of the DB library probably doesn't support that, no 
 point trying to force it.




 BTW, how would I do this with associative arrays?


 you can't. you can `.clear` 'em, though, but this won't 
 immediately free the internal storage.


Meh.. Okay, well using these two should make things a lot better. 
Thanks for the help and clearing up my confusion.
Jul 24 2017