• Yuxuan Shui (8/8) Aug 12 2015 Here is a small program
• Adam D. Ruppe (11/12) Aug 12 2015 I didn't look too closely, but there's some memory allocations
• Yuxuan Shui (2/15) Aug 12 2015 Is there a way to do thread-local allocations?
• Adam D. Ruppe (7/8) Aug 14 2015 Yes, either manually or just putting things on the stack, but the

"Yuxuan Shui" <yshuiv7 gmail.com> writes:

Here is a small program 
(https://gist.github.com/yshui/a426f73be77d1d699555) that uses 
taskPool to parallely reading from /proc/<pid>/ and sum the swap 
usage.

Individual tasks has zero dependency between each other, but when 
I remove the 'defaultPoolThreads(1)' line, the programs takes 8x 
more CPU time and also runs longer in total (I have 12 cores).

What is wrong here?

"Adam D. Ruppe" <destructionator gmail.com> writes:

On Wednesday, 12 August 2015 at 23:06:32 UTC, Yuxuan Shui wrote:
 What is wrong here?

I didn't look too closely, but there's some memory allocations 
going on there which have the potential of locking all the 
threads any time one of them tries to allocate.

Parallelism's benefits are largely erased by the memory allocator 
lock and can be set back by the cache being invalidated as it 
jumps around that allocated memory, so you generally want to make 
sure the threads are doing work on local variables only.

This restricts what you can do with strings, since most the 
std.string functions allocate new strings for their return 
values...

"Yuxuan Shui" <yshuiv7 gmail.com> writes:

On Wednesday, 12 August 2015 at 23:15:48 UTC, Adam D. Ruppe wrote:
 On Wednesday, 12 August 2015 at 23:06:32 UTC, Yuxuan Shui wrote:
 What is wrong here?

 I didn't look too closely, but there's some memory allocations 
 going on there which have the potential of locking all the 
 threads any time one of them tries to allocate.

 Parallelism's benefits are largely erased by the memory 
 allocator lock and can be set back by the cache being 
 invalidated as it jumps around that allocated memory, so you 
 generally want to make sure the threads are doing work on local 
 variables only.

 This restricts what you can do with strings, since most the 
 std.string functions allocate new strings for their return 
 values...

Is there a way to do thread-local allocations?

"Adam D. Ruppe" <destructionator gmail.com> writes:

On Wednesday, 12 August 2015 at 23:23:03 UTC, Yuxuan Shui wrote:
 Is there a way to do thread-local allocations?

Yes, either manually or just putting things on the stack, but the 
std.string functions you use doesn't try to do them.

You might try replacing format() with formattedWrite(), which 
calls a function you provide to put the data where it needs to be.

I tried running your program on my computer and it was too fast 
to measure so i'm not sure if this would make a difference or not.