D - dynamic array problems
- Alex A. B. (83/83) Jan 29 2004 I have numerous problems with the program I wrote to test and benchmark ...
- Paul Runde (11/44) Jan 29 2004 This is probably the memory allocation issue that been reported
- Ben Hinkle (14/58) Jan 29 2004 Anyone else know what is going on with this? Has someone tried it who
- Alex A. B. (8/13) Jan 29 2004 much
- Matthew (3/16) Jan 29 2004 Maybe it's time to consider that collecting background thread?
- Ben Hinkle (17/39) Jan 29 2004 Glancing over the test code it doesn't look like it is doing much
- Ben Hinkle (60/73) Jan 30 2004 I've attached a small program to ferret out some GC behaviors. Playing
I have numerous problems with the program I wrote to test and benchmark D's
dynamic arrays.
The source code is attached. It compiles with D 0.79 and D 0.75(with minor
modification).
What it tries to do is to read file contents and fill dynamic array of
arrays(each of random length(1-20 bytes),
then reassemble the data back and write it to another file.
It seems to work Ok, but only on small(really small) files. On my system
under Win98 it
stops working with files bigger than 50kb. ;) Under XP it manages to work
with any size
I tried, but consuming catastrophic amounts of memory(up to 90mb per 800kb
file!) and time.
Of course program does nothing useful. ;) But it was intended to be
benchmark for memory
allocation and management speed.
Just out of curiousity I have implemented the same program in
Prolog(SWI-Prolog) and ran
it on the same files. Well the speed was just about the same and memory
consumption was
twice as small. This is unbelievable results, because prolog is fully
interpreted, linked-lists
are used instead of arrays and even file ops are done on element-by-element
basis
through tail recursion!
Probably my code is wrong somewhere or D's run-time is doing very bad job.
Please prove me wrong, because I was(and still) going to use D extensively.
begin 666 test_arrays.d
M<')I;G0H*3L-"GT-" T*:6YT(')U;BAC:&%R6UU;72!A<F=S*0T*>PT*"6-H
M+R!T;R!T<GD ,FYD(&UE=&AO9 T*"0D)"2\O* T*"0D)"69R86=S('X](&YE
M9W,N;&5N9W1H("L ,3L-" D)"0EF<F%G<UMF<F%G<RYL96YG=& M,5TN;&5N
M9W1H(#T 8VYT.PT*"0D)"2\O*B\)"2 -" D)"7T-" D)"6-A=&-H("A/8FIE
M=&8H(F9R86=S+FQE;F=T:" ]/2 E9"P 8VYT(#T]("5D7&XB+&9R86=S+FQE
M;F=T:"QC;G0I.PT*"0D)"7)E='5R;B M,CL-" D)"7T-" D)?2 -" D)=')Y
M;&5N9W1H+3%=+FQE;F=T:"UC;G1=(#T
M"6-A=&-H("A/8FIE8W0 ;V)J*2 -" D)>R -" D)"65X7W!R:6YT*&]B:BD[
M86=S+FQE;F=T:"TQ72YL96YG=& /2!F<F%G<UMF<F%G<RYL96YG=& M,5TN
M;&5N9W1H+6-N=#L-" T*"69O<F5A8V H8VAA<EM=(&9R86<[(&9R86=S*0T*
M" D)"7)E='5R;B M-#L-" D)?0T*"7T-" T*"71R>0T*"7L-" D)<W1D+F9I
M;&4N=W)I=&4H87)G<ULR72QC87-T*'9O:61;72EO=71P=70I.PT*"7T-" EC
?;B(L<G5N*&%R9W,I*3L-" T*"7)E='5R;B P.PT*?0``
`
end
begin 666 test_arrays.pl
M<F%G*%M)8WQ)72Q;26-\1ETL23$L0RDZ+0T*0S$ :7, 0RTQ+&5N9G)A9RA)
M9&5F<F%G*%M=+%M=*2X-" T*9&5F<F%G*%M&<F%G;65N='Q&72Q/=71P=70I
M1BQ/=71P=70Q*2X-" T*8V]P>5]F:6QE*%-R8T9I;&4L1'-T1FEL92DZ+0T*
M;W!E;BA3<F-&:6QE+')E860L24Y&24Q%+%MT>7!E*&)I;F%R>2E=*2P-"F]P
M96XH1'-T1FEL92QW<FET92Q/551&24Q%+%MT>7!E*&)I;F%R>2E=*2P-"G)E
M95]C;V1E<U]T;U]S=')E86TH3U541DE,12Q/=71P=70I+ T*8VQO<V4H24Y&
424Q%*2QC;&]S92A/551&24Q%*2X`
`
end
Jan 29 2004
Alex A. B. wrote:I have numerous problems with the program I wrote to test and benchmark D's dynamic arrays. The source code is attached. It compiles with D 0.79 and D 0.75(with minor modification). What it tries to do is to read file contents and fill dynamic array of arrays(each of random length(1-20 bytes), then reassemble the data back and write it to another file. It seems to work Ok, but only on small(really small) files. On my system under Win98 it stops working with files bigger than 50kb. ;) Under XP it manages to work with any size I tried, but consuming catastrophic amounts of memory(up to 90mb per 800kb file!) and time. Of course program does nothing useful. ;) But it was intended to be benchmark for memory allocation and management speed. Just out of curiousity I have implemented the same program in Prolog(SWI-Prolog) and ran it on the same files. Well the speed was just about the same and memory consumption was twice as small. This is unbelievable results, because prolog is fully interpreted, linked-lists are used instead of arrays and even file ops are done on element-by-element basis through tail recursion! Probably my code is wrong somewhere or D's run-time is doing very bad job. Please prove me wrong, because I was(and still) going to use D extensively.This is probably the memory allocation issue that been reported recently. Here is the fix originally suggested by yaneurao. In std.thread add foward declaration: extern(Windows) export thread_hdl GetCurrentProcess(); In Thread.getCurrentThreadHandle() in std.thread change the line: thread_hdl currentProcess = cast(thread_hdl)-1; to thread_hdl currentProcess = GetCurrentProcess(); Then rebuild phobos.lib This fixed the problems I was encountering. Paul
Jan 29 2004
Anyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking up this much memory. thanks, -Ben "Paul Runde" <prunde consolidated.net> wrote in message news:bvb1v6$1l8o$1 digitaldaemon.com...Alex A. B. wrote:D'sI have numerous problems with the program I wrote to test and benchmarkminordynamic arrays. The source code is attached. It compiles with D 0.79 and D 0.75(withworkmodification). What it tries to do is to read file contents and fill dynamic array of arrays(each of random length(1-20 bytes), then reassemble the data back and write it to another file. It seems to work Ok, but only on small(really small) files. On my system under Win98 it stops working with files bigger than 50kb. ;) Under XP it manages to800kbwith any size I tried, but consuming catastrophic amounts of memory(up to 90mb perelement-by-elementfile!) and time. Of course program does nothing useful. ;) But it was intended to be benchmark for memory allocation and management speed. Just out of curiousity I have implemented the same program in Prolog(SWI-Prolog) and ran it on the same files. Well the speed was just about the same and memory consumption was twice as small. This is unbelievable results, because prolog is fully interpreted, linked-lists are used instead of arrays and even file ops are done onjob.basis through tail recursion! Probably my code is wrong somewhere or D's run-time is doing very badextensively.Please prove me wrong, because I was(and still) going to use DThis is probably the memory allocation issue that been reported recently. Here is the fix originally suggested by yaneurao. In std.thread add foward declaration: extern(Windows) export thread_hdl GetCurrentProcess(); In Thread.getCurrentThreadHandle() in std.thread change the line: thread_hdl currentProcess = cast(thread_hdl)-1; to thread_hdl currentProcess = GetCurrentProcess(); Then rebuild phobos.lib This fixed the problems I was encountering. Paul
Jan 29 2004
Ben Hinkle <bhinkle4 juno.com> сообщил в новостях следующее:bvbsbv$90$1 digitaldaemon.com...Anyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking up thismuchmemory. thanks, -BenYes, I have recompiled phobos, while memory allocation problem with Windows 98 was gone, it still does consume HUGE amounts of memory. Even SWI-Prologs' GC does a much better job at the same task.
Jan 29 2004
Maybe it's time to consider that collecting background thread? "Alex A. B." <freshmind fromru.com> wrote in message news:bvc9eo$lj8$1 digitaldaemon.com...Ben Hinkle <bhinkle4 juno.com> сообщил в новостях следующее:bvbsbv$90$1 digitaldaemon.com...Anyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking up thismuchmemory. thanks, -BenYes, I have recompiled phobos, while memory allocation problem with Windows 98 was gone, it still does consume HUGE amounts of memory. Even SWI-Prologs' GC does a much better job at the same task.
Jan 29 2004
"Ben Hinkle" <bhinkle4 juno.com> Glancing over the test code it doesn't look like it is doing much collecting, only lots and lots of small allocation. There are some reallocs being called by ~= but in general it looks like there are small (1 to 20 bytes) chunks and few 800K chunks (for an 800K input). Is it overhead on small objects? Is it poor recylcing of realloced arrays? I notice the last loop as it catenates all the small chunks back together the GC never has to alloc a small chunk - it is always resizing the output array to something larger so if the old array was just put on the free list it would never get reused). Does the GC ever merge free'd blocks? I can't tell from glancing at the gcx.d what is causing this code to perform like it is. It would be a pity to get D 1.0 and have it dismissed because of GC corner cases that make it look bad. -Ben "Matthew" <matthew.hat stlsoft.dot.org> wrote in message news:bvcaoe$nri$1 digitaldaemon.com...Maybe it's time to consider that collecting background thread? "Alex A. B." <freshmind fromru.com> wrote in message news:bvc9eo$lj8$1 digitaldaemon.com...thisBen Hinkle <bhinkle4 juno.com> сообщил в новостях следующее:bvbsbv$90$1 digitaldaemon.com...Anyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking upmuchmemory. thanks, -BenYes, I have recompiled phobos, while memory allocation problem with Windows 98 was gone, it still does consume HUGE amounts of memory. Even SWI-Prologs' GC does a much better job at the same task.
Jan 29 2004
"Alex A. B." <freshmind fromru.com> wrote in message news:bvc9eo$lj8$1 digitaldaemon.com...Ben Hinkle <bhinkle4 juno.com> сообщил в новостях следующее:bvbsbv$90$1 digitaldaemon.com...I've attached a small program to ferret out some GC behaviors. Playing around you'll see that 1) the smallest allocable chunks come in sizes 16, 32, 64 etc bytes 2) resizing an array dynamically one element at a time is very inefficient. I think these two factors contribute to the memory usage you saw with your test program. I haven't tried modifying your code exactly, but see what happens when you replace the "rand()%20+1" in your program with numbers around 16 or 32. Also try getting rid of the "~=" and replace it with explicitly setting the length so some large numbers. At one point Walter recommended setting the length to a large number and then back to zero in order to "preallocate" space. This didn't completely fix the problem but it helped a little. -Ben begin 666 test_small.d M+R]V97)S:6]N(#T M8SL-" T*+R]V97)S:6]N(#T 9'EN86UI8U]R97-I>F5?;VYL>3L-" T*+R]V M97)S:6]N(#T 9'EN86UI8U]P<F5A;&QO8SL-"B\O=F5R<VEO;B ](&1Y;F%M M('9E<G-I;VX *'!R:6YT7VEN7VQO;W I('L-"B (" (&=E=%-T871S*'-T M871S*3L-"B (" ('!R:6YT9B B<&]O;" E=2!U<V5D("5U(&9R964 )74 M<&%G92 E=5QN(BP-" D (" ('-T871S+G!O;VQS:7IE+'-T871S+G5S961S M:7IE+ T*"2 (" <W1A=',N9G)E96)L;V-K<RQS=&%T<RYP86=E8FQO8VMS M*3L-"B ('T-"GT-" T*:6YT(&UA:6XH8VAA<EM=6UT 87)G<RD-"GL-"B M(&-O;G-T('5I;G0 34%8(#T M=6EN="!K/3 [(&L\34%8.R!K*RLI('L-" D =F5R<VEO;B H<W1A=&EC7W)A M;F1O;2D >PT*"2 ("!O8FIS6VM=(#T ;F5W(&-H87);<F%N9" I)3(P*S%= M6VM=(#T ;F5W(&-H87);,3!=.R O+R!T<GD ,C L(&]R(&]T:&5R(&YU;6)E M('9E<G-I;VX *&1Y;F%M:6,I('L-" T*(" (" 9F]R("AU:6YT(&L],#L M:SQ-05 [(&LK*RD >PT*"2!D;V)J<R!^/2!N97< 8VAA<EMR86YD*"DE,C K M<VEO;B H9'EN86UI8U]R97-I>F5?;VYL>2D >PT*(" (" +R\ =F%R>2 Q M:G,N;&5N9W1H(#T 9&]B:G,N;&5N9W1H("L ,3 P,#L-" D 9'5M<%]S=&%T M;W( *'5I;G0 :STP.R!K/$U!6#L M(&-H87);<F%N9" I)3(P*S%=.PT*"2 O+PD 9&]B:G-;:UT /2!N97< 8VAA M=F5R<VEO;B H9'EN86UI8U]P<F5A;&QO8S(I('L-"B (" (&1O8FIS+FQE M;F=T:" ]($U!6#L- M;W( *'5I;G0 :STP.R!K/$U!6#L M:&%R6W)A;F0H*24R,"LQ73L-" D 9'5M<%]S=&%T<R I.PT*(" (" ?0T* M.B <&]O;" E=2!U<V5D("5U(&9R964 )74 <&%G92 E=5QN(BP-" D ('-T M871S+G!O;VQS:7IE+'-T871S+G5S961S:7IE+ T*"2 <W1A=',N9G)E96)L M;V-K<RQS=&%T<RYP86=E8FQO8VMS*3L-" T*(" <')I;G1F*")S=&%T:6, M8F%S93H )7 9'EN86UI8R!B87-E.B E<%QN(BPH=F]I9"HI;V)J<RPH=F]I ` endAnyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking up thismuchmemory. thanks, -BenYes, I have recompiled phobos, while memory allocation problem with Windows 98 was gone, it still does consume HUGE amounts of memory. Even SWI-Prologs' GC does a much better job at the same task.
Jan 30 2004