www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Distributed work load

reply Jacob Carlborg <doob me.com> writes:
I'm working on a tool which reads a an unknown number of files, does 
some processing on the content and writes out the result to new files on 
disk. The processing of the content is completely independent of any 
other processing, therefore I thought it might be a good idea to do this 
in parallel. So what I basically want is something like this:

foreach (file ; files)
{
     executInParallel((file) {
         auto content = read(file);
         process(content);
         write(newFile);
     });
}

"executInParallel" would then distribute this on an appropriate amount 
of threads and cores.

Is this what std.parallelism does ?

-- 
/Jacob Carlborg
May 18 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 18.05.2012 17:16, Jacob Carlborg wrote:

 I'm working on a tool which reads a an unknown number of files, does
 some processing on the content and writes out the result to new files on
 disk. The processing of the content is completely independent of any
 other processing, therefore I thought it might be a good idea to do this
 in parallel. So what I basically want is something like this:

 foreach (file ; files)
 {
 executInParallel((file) {
 auto content = read(file);
 process(content);
 write(newFile);
 });
 }

 "executInParallel" would then distribute this on an appropriate amount
 of threads and cores.

 Is this what std.parallelism does ?


Yes. Start with:
foreach (file; parallel(files))
{
	...
}

-- 
Dmitry Olshansky
May 18 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-05-18 16:56, Dmitry Olshansky wrote:


 Yes. Start with:
 foreach (file; parallel(files))
 {
 ...
 }


Aha, I was looking at "task". Thanks.

-- 
/Jacob Carlborg
May 18 2012
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 05/18/2012 06:16 AM, Jacob Carlborg wrote:

 I'm working on a tool which reads a an unknown number of files, does
 some processing on the content and writes out the result to new files on
 disk. The processing of the content is completely independent of any
 other processing, therefore I thought it might be a good idea to do this
 in parallel. So what I basically want is something like this:

 foreach (file ; files)
 {
 executInParallel((file) {
 auto content = read(file);
 process(content);
 write(newFile);
 });
 }

 "executInParallel" would then distribute this on an appropriate amount
 of threads and cores.

 Is this what std.parallelism does ?


I have tried to summarize what parallel() and its sisters do here:

   http://ddili.org/ders/d.en/parallelism.html

I hope at least the Summary section at the end is a useful cheat sheet.

Ali

-- 
D Programming Language Tutorial: http://ddili.org/ders/d.en/index.html
May 18 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-05-18 20:44, Ali Çehreli wrote:


 I have tried to summarize what parallel() and its sisters do here:

 http://ddili.org/ders/d.en/parallelism.html

 I hope at least the Summary section at the end is a useful cheat sheet.

 Ali


I'lL have to take a look at that.

-- 
/Jacob Carlborg
May 18 2012