digitalmars.D.learn - Deallocate array?
- Matic Kukovec (24/24) May 07 2013 Hi
- =?UTF-8?B?QWxpIMOHZWhyZWxp?= (3/8) May 07 2013 GC.minimize() may work.
- Matic Kukovec (3/15) May 07 2013 Thanks for the quick reply, Ali.
- =?UTF-8?B?QWxpIMOHZWhyZWxp?= (4/7) May 07 2013 Works for your test program under Linux but as the documentation says,
- Matic Kukovec (8/16) May 07 2013 I found this problem with a program that reads a large xml file
- =?UTF-8?B?QWxpIMOHZWhyZWxp?= (7/25) May 07 2013 Is this a 32-bit platform? If so, the reason may be the conservative GC
- Matic Kukovec (2/36) May 07 2013 The system is Windows Vista 64bit. DMD is 2.062.
- bearophile (6/7) May 07 2013 DMD doesn't yet produce 64 bit binaries on Windows.
- Juan Manuel Cabo (11/18) May 07 2013 Try the following before getting rid of your arrays:
- Juan Manuel Cabo (12/13) May 07 2013 The memory might be free, but still not released to the OS.
- Steven Schveighoffer (14/37) May 07 2013 This is very inefficient, use temp_array ~= "aaaaaaaaaaaaaaaaaaaaa";
- Matic Kukovec (44/44) May 08 2013 Thanks you guys for all the suggestions.
- Minas Mina (4/28) May 08 2013 I guess the memory isn't freed because there is no need to do so.
- Matic Kukovec (14/50) May 08 2013 I don't think that is the case, because if i copy/paste:
- Matic Kukovec (26/26) May 08 2013 Was playing around with the code and found that this WORKS:
- Steven Schveighoffer (21/26) May 08 2013 [snip]
- Matic Kukovec (8/35) May 08 2013 Thanks for the explanation Steve.
- Steven Schveighoffer (15/20) May 08 2013 auto arr = new char[N];
- Matic Kukovec (66/66) May 09 2013 Hi again Steve, or anyone else whose reading this.
- =?UTF-8?B?QWxpIMOHZWhyZWxp?= (41/48) May 09 2013 directly?
- Steven Schveighoffer (19/56) May 09 2013 y =
- =?UTF-8?B?QWxpIMOHZWhyZWxp?= (10/15) May 09 2013 You are right. They start at pointing to the same string but as soon as
- Matic Kukovec (17/17) May 10 2013 Thanks for the explanation guys.
- Steven Schveighoffer (10/24) May 10 2013 Delete is unsafe. It is being deprecated actually (not sure when).
Hi I'm running Windows Vista 64 with dmd 2.062. I have a simple program: import std.stdio, core.memory, std.cstream; void main() { string[] temp_array; for(int i=0;i<5000000;i++) { ++temp_array.length; temp_array[temp_array.length - 1] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; GC.collect(); writeln("end"); din.getc(); } When the program waits at "din.getc();", memory usage in the Task Manager is 150MB. Why isn't the memory deallocating? P.S.; I tried temp_array.clear() and destroy(temp_array), but nothing changed.
May 07 2013
On 05/07/2013 04:09 PM, Matic Kukovec wrote:> HiWhen the program waits at "din.getc();", memory usage in the Task Manager is 150MB. Why isn't the memory deallocating? P.S.; I tried temp_array.clear() and destroy(temp_array), but nothing changed.GC.minimize() may work. Ali
May 07 2013
On Tuesday, 7 May 2013 at 23:14:20 UTC, Ali Çehreli wrote:On 05/07/2013 04:09 PM, Matic Kukovec wrote:> HiThanks for the quick reply, Ali. Tried it, no changes.When the program waits at "din.getc();", memory usage in theTaskManager is 150MB. Why isn't the memory deallocating? P.S.; I tried temp_array.clear() and destroy(temp_array), butnothing changed. GC.minimize() may work. Ali
May 07 2013
On 05/07/2013 04:18 PM, Matic Kukovec wrote:On Tuesday, 7 May 2013 at 23:14:20 UTC, Ali Çehreli wrote:GC.minimize() may work.Tried it, no changes.Works for your test program under Linux but as the documentation says, it is not guaranteed to have any effect at all. Ali
May 07 2013
On Tuesday, 7 May 2013 at 23:31:41 UTC, Ali Çehreli wrote:On 05/07/2013 04:18 PM, Matic Kukovec wrote:I found this problem with a program that reads a large xml file (250000+ lines), then stores the lines in a string[], does a comparison with another array and finally clears the original array. On the second or third file I always get an OutOfMemoryError, when the Task Manager shows about 1.3GB memory usage. Is this a Windows specific thing or am I doing something wrong?On Tuesday, 7 May 2013 at 23:14:20 UTC, Ali Çehreli wrote:GC.minimize() may work.Tried it, no changes.Works for your test program under Linux but as the documentation says, it is not guaranteed to have any effect at all. Ali
May 07 2013
On 05/07/2013 04:42 PM, Matic Kukovec wrote:On Tuesday, 7 May 2013 at 23:31:41 UTC, Ali Çehreli wrote:You don't need to clear the original array but it should be harmless.On 05/07/2013 04:18 PM, Matic Kukovec wrote:I found this problem with a program that reads a large xml file (250000+ lines), then stores the lines in a string[], does a comparison with another array and finally clears the original array.On Tuesday, 7 May 2013 at 23:14:20 UTC, Ali Çehreli wrote:GC.minimize() may work.Tried it, no changes.Works for your test program under Linux but as the documentation says, it is not guaranteed to have any effect at all. AliOn the second or third file I always get an OutOfMemoryError, when the Task Manager shows about 1.3GB memory usage. Is this a Windows specific thing or am I doing something wrong?Is this a 32-bit platform? If so, the reason may be the conservative GC that dmd uses. What happens is, unrelated 32-bit values in other parts of the program may look like pointers into the allocated space and the GC thinks that they are still in use. Ali
May 07 2013
On Tuesday, 7 May 2013 at 23:58:53 UTC, Ali Çehreli wrote:On 05/07/2013 04:42 PM, Matic Kukovec wrote:The system is Windows Vista 64bit. DMD is 2.062.On Tuesday, 7 May 2013 at 23:31:41 UTC, Ali Çehreli wrote:documentation says,On 05/07/2013 04:18 PM, Matic Kukovec wrote:On Tuesday, 7 May 2013 at 23:14:20 UTC, Ali Çehreli wrote:GC.minimize() may work.Tried it, no changes.Works for your test program under Linux but as thefile (250000+it is not guaranteed to have any effect at all. AliI found this problem with a program that reads a large xmllines), then stores the lines in a string[], does acomparison withanother array and finally clears the original array.You don't need to clear the original array but it should be harmless.On the second or third file I always get an OutOfMemoryError,when theTask Manager shows about 1.3GB memory usage. Is this a Windows specific thing or am I doing somethingwrong? Is this a 32-bit platform? If so, the reason may be the conservative GC that dmd uses. What happens is, unrelated 32-bit values in other parts of the program may look like pointers into the allocated space and the GC thinks that they are still in use. Ali
May 07 2013
Matic Kukovec:The system is Windows Vista 64bit. DMD is 2.062.DMD doesn't yet produce 64 bit binaries on Windows. I have tried to solve your problem using GC.free, but I am not seeing good results... Bye, bearophile
May 07 2013
I found this problem with a program that reads a large xml file (250000+ lines), then stores the lines in a string[], does a comparison with another array and finally clears the original array. On the second or third file I always get an OutOfMemoryError, when the Task Manager shows about 1.3GB memory usage. Is this a Windows specific thing or am I doing something wrong?Try the following before getting rid of your arrays: myarray[] = null; //if it is an array of classes or strings myIntArray[] = 0; //if it is an array of ints myBigString[] = '\0'; etc. This clears the contents of the arrays, and helps the garbage collector, so that it doesn't confuse data with pointers. Also, avoid growing arrays little by little. This is VERY bad for garbage accumulation. Use instead an appender!(string[])() for string arrays for instance. --jm
May 07 2013
Why isn't the memory deallocating?The memory might be free, but still not released to the OS. Especially in windows, when you free memory it still isn't freed from your process. But you can reuse it in your program if the GC has collected it. The correct test would be to copy and paste the same code below, and see if it stays at 150Mb or it jumps to 300Mb. If it jumps to 300Mb, then the GC hasn't collected it. If it stays at 150Mb, the GC had collected it but windows didn't claim it. --jm
May 07 2013
On Tue, 07 May 2013 19:09:28 -0400, Matic Kukovec <matic.kukovec pametnidom.si> wrote:Hi I'm running Windows Vista 64 with dmd 2.062. I have a simple program: import std.stdio, core.memory, std.cstream; void main() { string[] temp_array; for(int i=0;i<5000000;i++) {++temp_array.length; temp_array[temp_array.length - 1] = "aaaaaaaaaaaaaaaaaaaaa";This is very inefficient, use temp_array ~= "aaaaaaaaaaaaaaaaaaaaa";} temp_array = null; GC.collect(); writeln("end"); din.getc(); } When the program waits at "din.getc();", memory usage in the Task Manager is 150MB. Why isn't the memory deallocating?The GC does not return free memory to the OS, just to free memory pools/free-lists. GC.minimize may or may not help. But your code may not have freed that memory anyway. It is not really possible to ensure that temp_array isn't referred to. For example, the compiler could keep temp_array in a register.P.S.; I tried temp_array.clear() and destroy(temp_array), but nothing changed.Neither of these will deallocate memory. All are equivalent to setting temp_array = null. If you want to ensure deallocation, you need to free it using GC.free(temp_array.ptr). A very dangerous operation, use with caution, make sure there are no other references to that data. -Steve
May 07 2013
Thanks you guys for all the suggestions. bearophile: I am not creating 64 bit binaries. Tried compiling with or without the "-m32" flag. I have played around with GC.free, see below for more details. Juan Manuel Cabo: I tried your methods, but no change. The original program I described earlier used an Appender, but the result is the same. I also copy/pasted the code twice (with or without GC.minimize()), to see if D was just reusing the memory, but the memory jumped to 290MB so it is not being reused or freed. Steven Schveighoffer: I used the "++temp_array.length;" example, because it was straight from the dlang.org docs. In my original program i tried the "~=" first then an Appender. In this example, if i use the "temp_array ~= "aaaaaaaaaaaaaaaaaaaaa";" the memory usage goes down from 150MB to about 80MB. Then i added: temp_array = null; GC.free(temp_array.ptr); GC.minimize(); The memory goes down to about 40MB! But why not lower? It also shrinks from 150MB to 100MB with the original "++temp_array.length;" apending to the array. The program in now: import std.stdio, core.memory, std.cstream; void main() { string[] temp_array; for(int i=0;i<5000000;i++) { temp_array ~= "aaaaaaaaaaaaaaaaaaaaa"; } temp_array[] = null; GC.free(temp_array.ptr); GC.minimize(); writeln("end"); din.getc(); } Any ideas?
May 08 2013
On Tuesday, 7 May 2013 at 23:09:29 UTC, Matic Kukovec wrote:Hi I'm running Windows Vista 64 with dmd 2.062. I have a simple program: import std.stdio, core.memory, std.cstream; void main() { string[] temp_array; for(int i=0;i<5000000;i++) { ++temp_array.length; temp_array[temp_array.length - 1] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; GC.collect(); writeln("end"); din.getc(); } When the program waits at "din.getc();", memory usage in the Task Manager is 150MB. Why isn't the memory deallocating? P.S.; I tried temp_array.clear() and destroy(temp_array), but nothing changed.I guess the memory isn't freed because there is no need to do so. A garbage collector kicks in when there is not enough memory. In this case there is no need to.
May 08 2013
On Wednesday, 8 May 2013 at 08:59:57 UTC, Minas Mina wrote:On Tuesday, 7 May 2013 at 23:09:29 UTC, Matic Kukovec wrote:I don't think that is the case, because if i copy/paste: for(int i=0;i<5000000;i++) { ++temp_array.length; temp_array[temp_array.length - 1] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; GC.collect(); 15 times and run the code, i get a core.exception.OutOfMemoryError. I don't know enough about D's garbage collection to figure out why the memory is not getting released/reused?Hi I'm running Windows Vista 64 with dmd 2.062. I have a simple program: import std.stdio, core.memory, std.cstream; void main() { string[] temp_array; for(int i=0;i<5000000;i++) { ++temp_array.length; temp_array[temp_array.length - 1] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; GC.collect(); writeln("end"); din.getc(); } When the program waits at "din.getc();", memory usage in the Task Manager is 150MB. Why isn't the memory deallocating? P.S.; I tried temp_array.clear() and destroy(temp_array), but nothing changed.I guess the memory isn't freed because there is no need to do so. A garbage collector kicks in when there is not enough memory. In this case there is no need to.
May 08 2013
Was playing around with the code and found that this WORKS: (note the "app_temp_array.reserve(50000000);" at the top, without this line it doesn't work) import std.stdio, core.memory, std.cstream, std.array; void main() { string[] temp_array; Appender!(string[]) app_temp_array = appender(temp_array); app_temp_array.reserve(50000000); for(int i=0;i<50000000;i++) { app_temp_array.put("aaaaaaaaaaaaaaaaaaaaa"); } temp_array = app_temp_array.data; app_temp_array.clear(); GC.free(app_temp_array.data); or GC.free(temp_array.ptr); GC.collect(); GC.minimize(); writeln("end"); din.getc(); } The memory usage shrinks from 400MB to 2MB, which is right! Why?
May 08 2013
On Wed, 08 May 2013 06:20:45 -0400, Matic Kukovec <matic.kukovec pametnidom.si> wrote:Was playing around with the code and found that this WORKS: (note the "app_temp_array.reserve(50000000);" at the top, without this line it doesn't work)[snip]The memory usage shrinks from 400MB to 2MB, which is right! Why?This is the conservative garbage collector at work. What happens is there is a "false pointer", or a word-aligned 32-bit value on the stack, or in global registers, that points at one of the arrays that you allocate. As you append, and the GC needs to allocate larger and larger arrays to hold your new data, the old ones are left behind for the GC to collect. But some random integer somewhere happens to "point" at one of those arrays. It is then kept in memory. It doesn't have to be a large array. GC.minimize can only shrink the OS-provided memory block, which is sequential. If any data is still valid, even if there is a bunch of free pages below that data, it cannot be returned to the OS. When you reserve, however, one, and only one, memory block is allocated. This block you are explictly freeing at the end, so it will be collected. Then minimize can do a good job. Not saying you should do things this way, just explaining the behavior. arr.reserve is a good tool to use when you know how much data you will need. But it can't fix every situation. -Steve
May 08 2013
On Wednesday, 8 May 2013 at 12:03:08 UTC, Steven Schveighoffer wrote:On Wed, 08 May 2013 06:20:45 -0400, Matic Kukovec <matic.kukovec pametnidom.si> wrote:Thanks for the explanation Steve. Can you please give me a correct example of: - creating a dynamic string array - looping over it and adding values - discarding/releasing the array from memory Thanks, MaticWas playing around with the code and found that this WORKS: (note the "app_temp_array.reserve(50000000);" at the top, without this line it doesn't work)[snip]The memory usage shrinks from 400MB to 2MB, which is right! Why?This is the conservative garbage collector at work. What happens is there is a "false pointer", or a word-aligned 32-bit value on the stack, or in global registers, that points at one of the arrays that you allocate. As you append, and the GC needs to allocate larger and larger arrays to hold your new data, the old ones are left behind for the GC to collect. But some random integer somewhere happens to "point" at one of those arrays. It is then kept in memory. It doesn't have to be a large array. GC.minimize can only shrink the OS-provided memory block, which is sequential. If any data is still valid, even if there is a bunch of free pages below that data, it cannot be returned to the OS. When you reserve, however, one, and only one, memory block is allocated. This block you are explictly freeing at the end, so it will be collected. Then minimize can do a good job. Not saying you should do things this way, just explaining the behavior. arr.reserve is a good tool to use when you know how much data you will need. But it can't fix every situation. -Steve
May 08 2013
On Wed, 08 May 2013 08:28:50 -0400, Matic Kukovec <matic.kukovec pametnidom.si> wrote:Thanks for the explanation Steve. Can you please give me a correct example of: - creating a dynamic string arrayauto arr = new char[N]; N can be a runtime value.- looping over it and adding valuesNot sure what you mean here.- discarding/releasing the array from memorycorrect: arr = null; // GC should collect dangerous: GC.free(arr.ptr); Due to conservative GC, you may need to explicitly free memory for practical reasons (e.g. if GC doesn't seem to collect the data you release). This problem is MUCH less prevalent on 64-bit, since the memory space is quite sparse. -Steve
May 08 2013
Hi again Steve, or anyone else whose reading this. array stuff to work. My D program: import std.stdio, std.cstream, std.array; void main() { string[] temp_array = new string[100000000]; for(int i=0;i<100000000;i++) { temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; temp_array = new string[100000000]; for(int i=0;i<100000000;i++) { temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; writeln("end"); din.getc(); } This program causes an OutOfMemoryError. using System; using System.Collections.Generic; using System.Linq; using System.Text; namespace testing { class Program { static void Main(string[] args) { string[] temp_array = new string[100000000]; for (int i = 0; i < 100000000; i++) { temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; temp_array = new string[100000000]; for (int i = 0; i < 100000000; i++) { temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; temp_array = new string[100000000]; for (int i = 0; i < 100000000; i++) { temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa"; } temp_array = null; Console.WriteLine("end"); Console.ReadKey(true); } } } It's the same program, but doesn't cause and OutOfMemory exception, even if i double the times i run the loop part. D without running out of memory and preferably without using the GC directly? I'm running Windows Vista 64bit, DMD 2.062 fresh install, compiling with "-m32" flag.
May 09 2013
On 05/09/2013 05:58 PM, Matic Kukovec wrote:Hi again Steve, or anyone else whose reading this. stuff to work.[...]without running out of memory and preferably without using the GCdirectly?I'm running Windows Vista 64bit, DMD 2.062 fresh install, compiling with "-m32" flag.You have a string array of 100_000_000 elements. (You can insert underscores in literals to make code more readable.) On a 32-bit system string is 8 bytes (e.g. by pragma(msg, string.sizeof)). So you have a single array of 800 million bytes. Then, each of those strings point at 22 bytes of individully allocated memory areas. Note that 800 million is about 20% of the entire addressable memory of that system. If you have a single value that happens to look like a pointer into that memory, that huge block will remain in memory forever. The funny thing about dmd's current conservative GC is that even an integer can be mistaken to be a pointer into that memory. So that's the problem today. I don't know the exact requirement here but if you really must have 100 million strings at one time, then you may want to do your own memory management. The following should work: import std.stdio; import core.memory; void main() { enum size_t times = 10; enum size_t stringCount = 100_000_000; foreach (i; 0 .. times) { auto rawMemory = cast(string*)GC.calloc(string.sizeof * stringCount); // D's cool feature of making a slice from raw pointer auto tempArray = rawMemory[0 .. stringCount]; foreach (ref s; tempArray) { s = "aaaaaaaaaaaaaaaaaaaaa"; } GC.free(rawMemory); // You may want to set tempArray to null at this point to prevent // using it accidentally but it is not necessary. writefln("Done with allocation %s/%s; please press Enter", i + 1, times); readln(); } } Ali
May 09 2013
On Thu, 09 May 2013 22:14:41 -0400, Ali =C3=87ehreli <acehreli yahoo.com=wrote:On a 32-bit system string is 8 bytes (e.g. by pragma(msg, =string.sizeof)). So you have a single array of 800 million bytes. Then=, =each of those strings point at 22 bytes of individully allocated memor=y =areas.No, each array points at static data. Strings are immutables stored in = = the data segment.Note that 800 million is about 20% of the entire addressable memory of==that system. If you have a single value that happens to look like a =pointer into that memory, that huge block will remain in memory foreve=r. =The funny thing about dmd's current conservative GC is that even an =integer can be mistaken to be a pointer into that memory. So that's the problem today.Correct, the only solution at the moment is to deallocate explicitly. r, = D = at GC as long as this is true. 64 bit address space also helps tremendously.I don't know the exact requirement here but if you really must have 10=0 =million strings at one time, then you may want to do your own memory =management. The following should work: import std.stdio; import core.memory; void main() { enum size_t times =3D 10; enum size_t stringCount =3D 100_000_000; foreach (i; 0 .. times) { auto rawMemory =3D cast(string*)GC.calloc(string.sizeof * =stringCount); // D's cool feature of making a slice from raw pointer auto tempArray =3D rawMemory[0 .. stringCount];You can allocate normally, and just use free on the pointerforeach (ref s; tempArray) { s =3D "aaaaaaaaaaaaaaaaaaaaa"; } GC.free(rawMemory);GC.free(tempArray.ptr); // also works// You may want to set tempArray to null at this point to =prevent // using it accidentally but it is not necessary. writefln("Done with allocation %s/%s; please press Enter", i + 1, times); readln(); } }-Steve
May 09 2013
On 05/09/2013 07:43 PM, Steven Schveighoffer wrote:On Thu, 09 May 2013 22:14:41 -0400, Ali Çehreli <acehreli yahoo.com>wrote:You are right. They start at pointing to the same string but as soon as each string is used in a special way, say each gets appended a char, then they own their special characters: s = "aaaaaaaaaaaaaaaaaaaaa"; // Some time later: s ~= 'x'; Now each s.ptr is different. AliThen, each of those strings point at 22 bytes of individully allocated memory areas.No, each array points at static data. Strings are immutables stored in the data segment.
May 09 2013
Thanks for the explanation guys. Hope there will be improvements in D's GC in the future. Also "delete temp_array" also works instead of "GC.free(temp_array.ptr)". And you also need to call "GC.minimize()" after a delete/GC.free() if you want to shrink the memory use after the for loop: temp_array = new string[100000000]; for(int i=0;i<100000000;i++) { temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa"; } delete temp_array; or GC.free(temp_array.ptr); temp_array = null; GC.minimize(); //releases the memory, works without this line Is "delete" safe for this kind of use? Thanks guys
May 10 2013
On Fri, 10 May 2013 04:44:14 -0400, Matic Kukovec <matic.kukovec pametnidom.si> wrote:Thanks for the explanation guys. Hope there will be improvements in D's GC in the future. Also "delete temp_array" also works instead of "GC.free(temp_array.ptr)". And you also need to call "GC.minimize()" after a delete/GC.free() if you want to shrink the memory use after the for loop: temp_array = new string[100000000]; for(int i=0;i<100000000;i++) { temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa"; } delete temp_array; or GC.free(temp_array.ptr); temp_array = null; GC.minimize(); //releases the memory, works without this line Is "delete" safe for this kind of use?Delete is unsafe. It is being deprecated actually (not sure when). Becuase it cannot zero out all pointers that reference that data, you can have dangling pointers to the data. delete is replaced by the function destroy, which finalizes data, and GC.free to deallocate. Only the GC.free part is unsafe, destroy is OK to use. However, for arrays, destroy is equivalent to setting the value to null. -Steve
May 10 2013