digitalmars.D - how to use GC as a leak detector? i.e. get some help info from GC?
- nobody (14/14) May 24 2009 Hi,
- Jason House (2/21) May 24 2009 Why not use valgrind? With the GC disabled, it should give accurate resu...
- nobody (6/7) May 24 2009 Strange enough, indeed I have tried valgrind with the GC disabled versio...
- Frits van Bommel (10/17) May 24 2009 The GC allocates memory directly from the OS, it doesn't use malloc/free...
- Leandro Lucarella (15/25) May 24 2009 I guess that with such amount of memory used, your program can greatly
- nobody (14/18) May 25 2009 No, my data are classes (not structs), and they need to be class by some...
- Leandro Lucarella (9/32) May 25 2009 There are request for them, but not for now...
- Unknown W. Brackets (10/29) May 24 2009 Theoretically, you could recompile the GC to write to a log file any
- nobody (3/5) May 24 2009 Is it possible to recompile Phobos to let the GC write to a log whenever...
- Nick Sabalausky (9/29) May 24 2009 Depending how exactly your program is working, another common thing that...
- dsimcha (25/39) May 24 2009 I've dealt with a bunch of somewhat similar situations in code I've writ...
- nobody (9/21) May 24 2009 I have tried all these: with GC enabled only periodically runs in the ma...
- dsimcha (9/30) May 24 2009 I personally have not experienced this. Please be more specific:
- nobody (4/12) May 24 2009 It's hard to isolate the code, and since the program is non-trivial I'm ...
- Brad Roberts (5/31) May 24 2009 After enabling the gc, did you force a collection? Just enabling it won...
- nobody (5/7) May 24 2009 Yes, I called:
- Leandro Lucarella (16/33) May 24 2009 As other asked, are you using D1 Tango/Phobos? D2? In Tango/D2 you can
- nobody (3/6) May 24 2009 Sorry, no.
- Leandro Lucarella (10/20) May 25 2009 =(
- nobody (5/18) May 25 2009 But I have posted a simple example earlier, it's silly, but disable GC g...
- Leandro Lucarella (13/34) May 25 2009 I think libphobos2.a contains libdruntime.a, but I'm not sure. You bette...
Hi, I'm writing a data processing program in D, which deals with large amounts of small objects. One of the thing I found is that D's GC is horribly slow in such situation. I tried my program with gc enable & disabled (with some manual deletes). The GC disabled version (2 min) is ~100 times faster than the GC enabled version (4 hours)! But of course the GC disabled version still leak memory, it soon exceeds the machine memory limit when I try to process more data; while the GC enabled version don't have such problem. So my plan is to use the GC disabled version with manual deletes. But it was very hard to find all the memory leaks. I'm wondering: is there anyway to use GC as a leak detector? can the GC enabled version give me some help information on which objects get collected, so I can manually delete them in my GC disabled version? Thanks!
May 24 2009
nobody Wrote:Hi, I'm writing a data processing program in D, which deals with large amounts of small objects. One of the thing I found is that D's GC is horribly slow in such situation. I tried my program with gc enable & disabled (with some manual deletes). The GC disabled version (2 min) is ~100 times faster than the GC enabled version (4 hours)! But of course the GC disabled version still leak memory, it soon exceeds the machine memory limit when I try to process more data; while the GC enabled version don't have such problem. So my plan is to use the GC disabled version with manual deletes. But it was very hard to find all the memory leaks. I'm wondering: is there anyway to use GC as a leak detector? can the GC enabled version give me some help information on which objects get collected, so I can manually delete them in my GC disabled version? Thanks!Why not use valgrind? With the GC disabled, it should give accurate results.
May 24 2009
== Quote from Jason House (jason.james.house gmail.com)'s articleWhy not use valgrind? With the GC disabled, it should give accurate results.Strange enough, indeed I have tried valgrind with the GC disabled version. It didn't report anything useful. That's why I'm puzzled, does D's GC do something special? The GC disabled version run out of 3G memory; but the GC enabled version stays at ~800M throughout the run.
May 24 2009
nobody wrote:== Quote from Jason House (jason.james.house gmail.com)'s articleThe GC allocates memory directly from the OS, it doesn't use malloc/free and friends. It does this even when the GC is "disabled", which just means the collections won't happen. (Disabling the GC doesn't change the method of allocation) Valgrind probably doesn't detect those OS calls (and almost certainly doesn't know about the GC calls). If you're using Tango, you can link to the 'stub' GC instead of the normal ('basic') one. The stub GC doesn't actually collect, it passes calls on to malloc/calloc/realloc/free instead. That should make Valgrind work. (something similar probably applies if you're using D2 with druntime)Why not use valgrind? With the GC disabled, it should give accurate results.Strange enough, indeed I have tried valgrind with the GC disabled version. It didn't report anything useful. That's why I'm puzzled, does D's GC do something special?
May 24 2009
nobody, el 24 de mayo a las 20:03 me escribiste:== Quote from Jason House (jason.james.house gmail.com)'s articleI guess that with such amount of memory used, your program can greatly benefit from using NO_SCAN if your 800M of data are plain old data. Did you tried it? And if you never have interior pointers to that data, your program can possibly avoid a lot of false positives due to the conservativism if you use NO_INTERIOR (this is only available if you patch the GC with David Simcha's patch[1]). [1] http://d.puremagic.com/issues/show_bug.cgi?id=2927 -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- This is what you get, when you mess with us.Why not use valgrind? With the GC disabled, it should give accurate results.Strange enough, indeed I have tried valgrind with the GC disabled version. It didn't report anything useful. That's why I'm puzzled, does D's GC do something special? The GC disabled version run out of 3G memory; but the GC enabled version stays at ~800M throughout the run.
May 24 2009
== Quote from Leandro Lucarella (llucax gmail.com)'s articlebenefit from using NO_SCAN if your 800M of data are plain old data. Did you tried it? And if you never have interior pointers to that data, your program can possibly avoid a lot of false positives due to the conservativism if you use NO_INTERIOR (this is only available if you patchNo, my data are classes (not structs), and they need to be class by some other design considerations; and worse they contain pointers to other data, e.g. class SmallDataA { // need to be class } class SmallDataB { // need to be class SmallDataA a; // in D 'a' is a reference, or 'pointer' } I have thought about use POD. I think the above code in C++ will be more what I want: i.e. the 'a' object (not the reference) is embedded directly into SmallDataB. I guess when I have millions of such SmallDataB objects, it will make the GC busy in D since 'a' is reference. So question: can we have such expanded class objects in D?
May 25 2009
nobody, el 25 de mayo a las 07:37 me escribiste:== Quote from Leandro Lucarella (llucax gmail.com)'s articleThere are request for them, but not for now... -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- CAYO HUGO CONZI --- TENIA PUESTA PELUCA -- Crónica TVbenefit from using NO_SCAN if your 800M of data are plain old data. Did you tried it? And if you never have interior pointers to that data, your program can possibly avoid a lot of false positives due to the conservativism if you use NO_INTERIOR (this is only available if you patchNo, my data are classes (not structs), and they need to be class by some other design considerations; and worse they contain pointers to other data, e.g. class SmallDataA { // need to be class } class SmallDataB { // need to be class SmallDataA a; // in D 'a' is a reference, or 'pointer' } I have thought about use POD. I think the above code in C++ will be more what I want: i.e. the 'a' object (not the reference) is embedded directly into SmallDataB. I guess when I have millions of such SmallDataB objects, it will make the GC busy in D since 'a' is reference. So question: can we have such expanded class objects in D?
May 25 2009
Theoretically, you could recompile the GC to write to a log file any time it frees anything. For data processing, though, you really want to try to have a fixed memory buffer. You've got to be hurting from the allocations and frees, which if at all possible you should get rid of. Also, if you're allocating buffers of memory (e.g. for the data), you can tell the GC not to scan them. This will probably solve the problem of the GC being so slow. -[Unknown] nobody wrote:Hi, I'm writing a data processing program in D, which deals with large amounts of small objects. One of the thing I found is that D's GC is horribly slow in such situation. I tried my program with gc enable & disabled (with some manual deletes). The GC disabled version (2 min) is ~100 times faster than the GC enabled version (4 hours)! But of course the GC disabled version still leak memory, it soon exceeds the machine memory limit when I try to process more data; while the GC enabled version don't have such problem. So my plan is to use the GC disabled version with manual deletes. But it was very hard to find all the memory leaks. I'm wondering: is there anyway to use GC as a leak detector? can the GC enabled version give me some help information on which objects get collected, so I can manually delete them in my GC disabled version? Thanks!
May 24 2009
== Quote from Unknown W. Brackets (unknown simplemachines.org)'s articleTheoretically, you could recompile the GC to write to a log file any time it frees anything.Is it possible to recompile Phobos to let the GC write to a log whenever it frees? I guess I also need the type info of the object being freed.
May 24 2009
"nobody" <no where.com> wrote in message news:gvc5q7$2bc3$1 digitalmars.com...Hi, I'm writing a data processing program in D, which deals with large amounts of small objects. One of the thing I found is that D's GC is horribly slow in such situation. I tried my program with gc enable & disabled (with some manual deletes). The GC disabled version (2 min) is ~100 times faster than the GC enabled version (4 hours)! But of course the GC disabled version still leak memory, it soon exceeds the machine memory limit when I try to process more data; while the GC enabled version don't have such problem. So my plan is to use the GC disabled version with manual deletes. But it was very hard to find all the memory leaks. I'm wondering: is there anyway to use GC as a leak detector? can the GC enabled version give me some help information on which objects get collected, so I can manually delete them in my GC disabled version? Thanks!Depending how exactly your program is working, another common thing that might help is to manually manage free pools. Ie, allocate a bunch up-front, and instead of letting one get GCed when done with it, hold on to it, make note of it being available for re-use, and then reuse it instead of allocating a new one. Or, allocate one big chuck of memory and stick your small objects in there. They typically do this sort of thing for particle systems.
May 24 2009
== Quote from nobody (no where.com)'s articleHi, I'm writing a data processing program in D, which deals with large amounts of small objects. One of the thing I found is that D's GC is horribly slow in such situation. I tried my program with gc enable & disabled (with some manual deletes). The GC disabled version (2 min) is ~100 times faster than the GC enabled version (4 hours)! But of course the GC disabled version still leak memory, it soon exceeds the machine memory limit when I try to process more data; while the GC enabled version don't have such problem. So my plan is to use the GC disabled version with manual deletes. But it was very hard to find all the memory leaks. I'm wondering: is there anyway to use GC as a leak detector? can the GC enabled version give me some help information on which objects get collected, so I can manually delete them in my GC disabled version? Thanks!I've dealt with a bunch of somewhat similar situations in code I've written, here are some tips that others have not already mentioned, and that might be less drastic than going with fully manual memory management: One thing you could try is disabling the GC (this really just disables automatic running of the collector) and run it manually at points that you know make sense. For example, you could just insert a GC.collect() statement at the end of every run of your main loop. Another thing to try is avoiding appending to arrays. If you know the length in advance, you can get pretty good speedups by pre-allocating the array instead of appending using the ~= operator. You can safely delete specific objects manually even when the GC is enabled. For very large objects with trivial lifetimes, this is probably worth doing. First of all, the GC will run less frequently. Secondly, D's GC is partially conservative, meaning that occasionally memory will not be freed when it should be. The probability of this happening is proportional to the size of the memory block. Lastly, I've been working on a generic second stack/mark-release allocator for D2, called TempAlloc. It's useful for when you need to temporarily allocate memory in a last in, first out order, but you can't use the call stack for whatever reason. I've also implemented a few basic data structures (hash tables and hash sets) that are specifically designed for this allocator. Right now, it's coevolving with my dstats statistics lib, but if you want to try it or at least look at it and give me some feedback, I'd like to eventually get it to the point where it can be added to Phobos and/or Tango. See http://svn.dsource.org/projects/dstats/docs/alloc.html .
May 24 2009
One thing you could try is disabling the GC (this really just disables automatic running of the collector) and run it manually at points that you know make sense. For example, you could just insert a GC.collect() statement at the end of every run of your main loop. Another thing to try is avoiding appending to arrays. If you know the length in advance, you can get pretty good speedups by pre-allocating the array instead of appending using the ~= operator. You can safely delete specific objects manually even when the GC is enabled. For very large objects with trivial lifetimes, this is probably worth doing. First of all, the GC will run less frequently. Secondly, D's GC is partially conservative, meaning that occasionally memory will not be freed when it should be. The probability of this happening is proportional to the size of the memory block.I have tried all these: with GC enabled only periodically runs in the main loop, however the memory still grows faster than I expected when I feed more data into the program. Then I manually delete some specific objects. However the program start to fail randomly. Has anyone experienced similar issues: i.e. with GC on, you defined you own dtor for certain class, and called delete manually on certain objects. The program fails at random stages, with some stack trace showing some GC calls like: 0x0821977a in _D2gc3gcx3Gcx16fullcollectshellMFZk () I suspected the GC is buggy when mixed with manual deletes.
May 24 2009
== Quote from nobody (no where.com)'s articlelike:One thing you could try is disabling the GC (this really just disables automatic running of the collector) and run it manually at points that you know make sense. For example, you could just insert a GC.collect() statement at the end of every run of your main loop. Another thing to try is avoiding appending to arrays. If you know the length in advance, you can get pretty good speedups by pre-allocating the array instead of appending using the ~= operator. You can safely delete specific objects manually even when the GC is enabled. For very large objects with trivial lifetimes, this is probably worth doing. First of all, the GC will run less frequently. Secondly, D's GC is partially conservative, meaning that occasionally memory will not be freed when it should be. The probability of this happening is proportional to the size of the memory block.I have tried all these: with GC enabled only periodically runs in the main loop, however the memory still grows faster than I expected when I feed more data into the program. Then I manually delete some specific objects. However the program start to fail randomly. Has anyone experienced similar issues: i.e. with GC on, you defined you own dtor for certain class, and called delete manually on certain objects. The program fails at random stages, with some stack trace showing some GC calls0x0821977a in _D2gc3gcx3Gcx16fullcollectshellMFZk () I suspected the GC is buggy when mixed with manual deletes.I personally have not experienced this. Please be more specific: D1 or D2? If D1, Phobos or Tango? DMD, LDC, or GDC? Compiler version? Also, please file a bug report, especially if you can create a concise, reproducible test case.
May 24 2009
D2.I suspected the GC is buggy when mixed with manual deletes.I personally have not experienced this. Please be more specific: D1 or D2?If D1, Phobos or Tango? DMD, LDC, or GDC?DMD v2.030Compiler version? Also, please file a bug report, especially if you can create a concise, reproducible test case.It's hard to isolate the code, and since the program is non-trivial I'm not 100% sure, as it could be my bug.
May 24 2009
nobody wrote:After enabling the gc, did you force a collection? Just enabling it won't cause one to occur. Later, BradOne thing you could try is disabling the GC (this really just disables automatic running of the collector) and run it manually at points that you know make sense. For example, you could just insert a GC.collect() statement at the end of every run of your main loop. Another thing to try is avoiding appending to arrays. If you know the length in advance, you can get pretty good speedups by pre-allocating the array instead of appending using the ~= operator. You can safely delete specific objects manually even when the GC is enabled. For very large objects with trivial lifetimes, this is probably worth doing. First of all, the GC will run less frequently. Secondly, D's GC is partially conservative, meaning that occasionally memory will not be freed when it should be. The probability of this happening is proportional to the size of the memory block.I have tried all these: with GC enabled only periodically runs in the main loop, however the memory still grows faster than I expected when I feed more data into the program. Then I manually delete some specific objects. However the program start to fail randomly. Has anyone experienced similar issues: i.e. with GC on, you defined you own dtor for certain class, and called delete manually on certain objects. The program fails at random stages, with some stack trace showing some GC calls like: 0x0821977a in _D2gc3gcx3Gcx16fullcollectshellMFZk () I suspected the GC is buggy when mixed with manual deletes.
May 24 2009
== Quote from Brad Roberts (braddr puremagic.com)'s articleAfter enabling the gc, did you force a collection? Just enabling it won't cause one to occur.Yes, I called: core.memory.GC.enable(); core.memory.GC.collect(); core.memory.GC.disable();
May 24 2009
nobody, el 24 de mayo a las 19:05 me escribiste:Hi, I'm writing a data processing program in D, which deals with large amounts of small objects. One of the thing I found is that D's GC is horribly slow in such situation. I tried my program with gc enable & disabled (with some manual deletes). The GC disabled version (2 min) is ~100 times faster than the GC enabled version (4 hours)! But of course the GC disabled version still leak memory, it soon exceeds the machine memory limit when I try to process more data; while the GC enabled version don't have such problem. So my plan is to use the GC disabled version with manual deletes. But it was very hard to find all the memory leaks. I'm wondering: is there anyway to use GC as a leak detector? can the GC enabled version give me some help information on which objects get collected, so I can manually delete them in my GC disabled version? Thanks!As other asked, are you using D1 Tango/Phobos? D2? In Tango/D2 you can enable logging in the GC (using the LOGGING version identifier). Is your program source available? I'm gathering programs to make a D GC benchmark suite an your programs seems like a good candidate for measuring the GC performance. Thank you. -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- Que importante, entonces en estos días de globalización refregar nuestras almas, pasarle el lampazo a nuestros corazones para alcanzar un verdadero estado de babia peperianal. -- Peperino Pómoro
May 24 2009
As other asked, are you using D1 Tango/Phobos? D2? In Tango/D2 you canDMD v2.030 on Linux.enable logging in the GC (using the LOGGING version identifier).How to do it in D2?Is your program source available? I'm gathering programs to make a D GCSorry, no.
May 24 2009
nobody, el 25 de mayo a las 03:24 me escribiste:You should recompile Druntime's GC with -version=LOGGING.As other asked, are you using D1 Tango/Phobos? D2? In Tango/D2 you canDMD v2.030 on Linux.enable logging in the GC (using the LOGGING version identifier).How to do it in D2?=( -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- Es más probable que el tomate sea perita, a que la pera tomatito. -- Peperino PómoroIs your program source available? I'm gathering programs to make a D GCSorry, no.
May 25 2009
== Quote from Leandro Lucarella (llucax gmail.com)'s articlenobody, el 25 de mayo a las 03:24 me escribiste:When linking does DMD link against libphobos2.a, libdruntime.a or both?You should recompile Druntime's GC with -version=LOGGING.As other asked, are you using D1 Tango/Phobos? D2? In Tango/D2 you canDMD v2.030 on Linux.enable logging in the GC (using the LOGGING version identifier).How to do it in D2?But I have posted a simple example earlier, it's silly, but disable GC get 5x speed up: http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=90983=(Is your program source available? I'm gathering programs to make a D GCSorry, no.
May 25 2009
nobody, el 25 de mayo a las 18:31 me escribiste:== Quote from Leandro Lucarella (llucax gmail.com)'s articleI think libphobos2.a contains libdruntime.a, but I'm not sure. You better check the sources.nobody, el 25 de mayo a las 03:24 me escribiste:When linking does DMD link against libphobos2.a, libdruntime.a or both?You should recompile Druntime's GC with -version=LOGGING.As other asked, are you using D1 Tango/Phobos? D2? In Tango/D2 you canDMD v2.030 on Linux.enable logging in the GC (using the LOGGING version identifier).How to do it in D2?Oh, I saw that, thank you. -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- Robar un alfajor es revolucionario, pues rompe con la idea de propiedad, incluso más que si se tratara de dinero. -- publikazion anarkista mdp (hablando de los destrozos de la Cumbre de las Americas en Mar del Plata, 2005)But I have posted a simple example earlier, it's silly, but disable GC get 5x speed up: http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=90983=(Is your program source available? I'm gathering programs to make a D GCSorry, no.
May 25 2009