digitalmars.D.learn - How to reuse the space of RAM of an array of strings?
- Giovanni Di Maria (47/47) Dec 21 2018 Hi. Can you help me please?
- H. S. Teoh (49/66) Dec 21 2018 [...]
- Giovanni Di Maria (5/6) Dec 21 2018 Thank you very much for your replay.
- Steven Schveighoffer (13/57) Dec 21 2018 Note: alloca is a builtin intrinsic, so I wouldn't use that as a
- Giovanni Di Maria (6/22) Dec 22 2018 Hi Steve
Hi. Can you help me please?
I asked also to "General Group" but i haven't solved my problem.
I have tried many experiments but without success.
I have a dynamic array of strings:
string[] vec;
After to have inserted dynamically 5_000_000 strings to array,
for six times, i measure the RAM with an utility (for example
Wise Memory Optimizer), deleting the array every time.
Every time I destroy the array, but the RAM is always less.
In particular:
- at beginning the FREE RAM is 1564 MB;
- after first loop the FREE RAM IS 1480 MB
- after second loop the FREE RAM IS 1415 MB
- after third loop the FREE RAM IS 1402 MB
- after forth loop the FREE RAM IS 1338 MB
- after fifth loop the FREE RAM IS 1280 MB
- after sixth loop the FREE RAM IS 1200 MB
- at end the FREE RAM returns to 1564 MB
I want to reuse the dynamic array.
This is the program:
import std.stdio;
string[] vec;
void main()
{
alloca();
alloca();
alloca();
alloca();
alloca();
alloca();
}
void alloca()
{
vec.destroy;
writeln("Filling .....");
for (int i = 0; i < 5000000; i++)
vec ~= "1234567890ABCDEFGHIL123456";
writeln("Array occupation: ", vec.capacity);
writeln("Press <ENTER>");
readln();
}
P.S.
I have tried: free(), destroy(), minimize(), GC, collect(), etc
but without success.
Can you help me please?
Thank you very much
Giovanni Di Maria
Dec 21 2018
On Fri, Dec 21, 2018 at 08:41:10PM +0000, Giovanni Di Maria via Digitalmars-d-learn wrote: [...]After to have inserted dynamically 5_000_000 strings to array, for six times, i measure the RAM with an utility (for example Wise Memory Optimizer), deleting the array every time. Every time I destroy the array, but the RAM is always less. In particular: - at beginning the FREE RAM is 1564 MB; - after first loop the FREE RAM IS 1480 MB - after second loop the FREE RAM IS 1415 MB - after third loop the FREE RAM IS 1402 MB - after forth loop the FREE RAM IS 1338 MB - after fifth loop the FREE RAM IS 1280 MB - after sixth loop the FREE RAM IS 1200 MB - at end the FREE RAM returns to 1564 MB I want to reuse the dynamic array.[...] You need to understand that generally speaking, the GC does not return memory to the OS; it retains that memory for future use, e.g., when you allocate a new object. Also, not all objects may be freed immediately if the GC can still find references to it somewhere, including on the stack. And if you're in a 32-bit environment, there's a higher chance you might run into the problem of false pointers (non-pointer values that look like a valid pointer into an allocated object, causing the GC to think the object is still live when it's actually already dead). Furthermore, when allocating large numbers of objects the GC may create additional internal data structures to keep track of said objects, and may continue to retain these structures after the objects have been collected. If you want deterministic deallocation, your best bet is to use malloc/free instead of the GC (that means you'll need to avoid allocating operators like ~, ~=, array literals, and so on). Using nogc may help here. A more reliable test is if you let your loop run forever, and see if the memory usage eventually plateaus. It should stop increasing at a certain point once you've reached steady state; if not, that would be stronger proof of memory leakage. Furthermore, if you want to reuse existing memory for the array, you should just write to it directly instead of deallocating / reallocating. I.e., instead of doing this: foreach (i; 0 .. 1_000_000) { auto arr = new int[1_000_000]; ... // do stuff with arr arr = null; // or arr.destroy or whatever } do this: auto arr = new int[1_000_000]; // only allocate once foreach (i; 0 .. 1_000_000) { ... // overwrite any previous contents of arr with new data ... // do stuff with arr } For your array of strings, you probably want to use a string buffer / pool somewhere, i.e., a contiguous block of memory that you allocate once, and take slices of as you populate your string array, instead of allocating once per array element. This will give you better control over your memory usage, and also reduce GC load (so collection cycles will generally be faster, since there's less stuff that needs to be marked and collected). T -- Give a man a fish, and he eats once. Teach a man to fish, and he will sit forever.
Dec 21 2018
Giovanni Di Maria <calimero22 yahoo.it> On Friday, 21 December 2018 at 21:28:14 UTC, H. S. Teoh wrote:[...]Thank you very much for your replay. I will meditate about your words. Thank you Giovanni
Dec 21 2018
On 12/21/18 3:41 PM, Giovanni Di Maria wrote:Hi. Can you help me please? I asked also to "General Group" but i haven't solved my problem. I have tried many experiments but without success. I have a dynamic array of strings: string[] vec; After to have inserted dynamically 5_000_000 strings to array, for six times, i measure the RAM with an utility (for example Wise Memory Optimizer), deleting the array every time. Every time I destroy the array, but the RAM is always less. In particular: - at beginning the FREE RAM is 1564 MB; - after first loop the FREE RAM IS 1480 MB - after second loop the FREE RAM IS 1415 MB - after third loop the FREE RAM IS 1402 MB - after forth loop the FREE RAM IS 1338 MB - after fifth loop the FREE RAM IS 1280 MB - after sixth loop the FREE RAM IS 1200 MB - at end the FREE RAM returns to 1564 MB I want to reuse the dynamic array. This is the program: import std.stdio; string[] vec; void main() { alloca(); alloca(); alloca(); alloca(); alloca(); alloca(); } void alloca()Note: alloca is a builtin intrinsic, so I wouldn't use that as a function name. Don't think it's affecting your program, but I wanted to point that out.{ vec.destroy;This does NOT free the ram, it simply resets vec to null. If you want to free the memory, use GC.free(vec.ptr); vec = null;writeln("Filling ....."); for (int i = 0; i < 5000000; i++) vec ~= "1234567890ABCDEFGHIL123456";Note, this allocates a lot of smaller arrays on its way up to the really big array. You are better off doing: vec.reserve(5_000_000); which will pre-allocate the capacity needed. This will make you only allocate once. -Steve
Dec 21 2018
On Friday, 21 December 2018 at 22:31:26 UTC, Steven Schveighoffer wrote:On 12/21/18 3:41 PM, Giovanni Di Maria wrote:Hi Steve Ok, thank you very much. Ciao Giovanni[...]Note: alloca is a builtin intrinsic, so I wouldn't use that as a function name. Don't think it's affecting your program, but I wanted to point that out.[...]This does NOT free the ram, it simply resets vec to null. If you want to free the memory, use GC.free(vec.ptr); vec = null;[...]Note, this allocates a lot of smaller arrays on its way up to the really big array. You are better off doing: vec.reserve(5_000_000); which will pre-allocate the capacity needed. This will make you only allocate once. -Steve
Dec 22 2018