www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - More threads -> Slower program ??

reply "Yuxuan Shui" <yshuiv7 gmail.com> writes:
Here is a small program 
(https://gist.github.com/yshui/a426f73be77d1d699555) that uses 
taskPool to parallely reading from /proc/<pid>/ and sum the swap 
usage.

Individual tasks has zero dependency between each other, but when 
I remove the 'defaultPoolThreads(1)' line, the programs takes 8x 
more CPU time and also runs longer in total (I have 12 cores).

What is wrong here?
Aug 12 2015
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 12 August 2015 at 23:06:32 UTC, Yuxuan Shui wrote:
 What is wrong here?
I didn't look too closely, but there's some memory allocations going on there which have the potential of locking all the threads any time one of them tries to allocate. Parallelism's benefits are largely erased by the memory allocator lock and can be set back by the cache being invalidated as it jumps around that allocated memory, so you generally want to make sure the threads are doing work on local variables only. This restricts what you can do with strings, since most the std.string functions allocate new strings for their return values...
Aug 12 2015
parent reply "Yuxuan Shui" <yshuiv7 gmail.com> writes:
On Wednesday, 12 August 2015 at 23:15:48 UTC, Adam D. Ruppe wrote:
 On Wednesday, 12 August 2015 at 23:06:32 UTC, Yuxuan Shui wrote:
 What is wrong here?
I didn't look too closely, but there's some memory allocations going on there which have the potential of locking all the threads any time one of them tries to allocate. Parallelism's benefits are largely erased by the memory allocator lock and can be set back by the cache being invalidated as it jumps around that allocated memory, so you generally want to make sure the threads are doing work on local variables only. This restricts what you can do with strings, since most the std.string functions allocate new strings for their return values...
Is there a way to do thread-local allocations?
Aug 12 2015
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 12 August 2015 at 23:23:03 UTC, Yuxuan Shui wrote:
 Is there a way to do thread-local allocations?
Yes, either manually or just putting things on the stack, but the std.string functions you use doesn't try to do them. You might try replacing format() with formattedWrite(), which calls a function you provide to put the data where it needs to be. I tried running your program on my computer and it was too fast to measure so i'm not sure if this would make a difference or not.
Aug 14 2015