www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - object splitting in multiple threads

reply yes <yes no.com> writes:
How bad is the following idea?


class Calc
{

void addThread()
{
 Data data;
 data = new Data();
}

void run()
{
if ( hardware threads > current threadcount)
{
 addThread();
}

//wait for some signal
//run calculations on data / threads

}

}

Calc mainCalc;
mainCalc = new Calc();
mainCalc.run();
Jan 09 2009
next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Sat, 10 Jan 2009 05:44:20 +0300, yes <yes no.com> wrote:


 How bad is the following idea?


 class Calc
 {

 void addThread()
 {
  Data data;
  data = new Data();
 }

 void run()
 {
 if ( hardware threads > current threadcount)
 {
  addThread();
 }

 //wait for some signal
 //run calculations on data / threads

 }

 }

 Calc mainCalc;
 mainCalc = new Calc();
 mainCalc.run();


What does this code do? It looks like a ThreadPool use case but I might be
wrong.
Jan 09 2009
prev sibling parent reply yes <yes no.com> writes:
 
 What does this code do? It looks like a ThreadPool use case but I might be
wrong.
 


I tried to recursively make an object distribute its calculations.
The calculations should take at least a minute.
Jan 09 2009
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Sat, 10 Jan 2009 07:32:43 +0300, yes <yes no.com> wrote:


 What does this code do? It looks like a ThreadPool use case but I might  
 be wrong.


 I tried to recursively make an object distribute its calculations.
 The calculations should take at least a minute.



Hm... I guess I understand now, you need a task parallelization, right?

In order to do this, you usually separate the task into several independent
sub-tasks, put into a queue and wait until it is finished.

For example, we have a list of files to download from remote server. For each
file, we create a new connection and retrieve it independently (and, possibly,
out of order):

import tango.core.ThreadPool;

void execute()
{
    auto pool = new ThreadPool!(string)(10); // create a thread pool with 10
threads
    string[] fileList = loadFileList();
    foreach (string filePath; fileList) {
        pool.append(&download, filePath);   // append the task
    }

    pool.finish(); // wait until all the tasks complete
}

void download(string fileName)
{
    // ...
}

In this example, I create a lot of threads (more than hardware runs
concurrently) because the CPU is not a bottleneck.
You may experience different results in different CPU workaload, though.

Hope that helps.
Jan 09 2009
parent reply yes <yes no.com> writes:
Does Phobos also do threadpools?

Somehow I liked the idea of an Object like agent smith, just duplicate yourself
when the task is too big. ( ^_^ )


 
 
 Hm... I guess I understand now, you need a task parallelization, right?
 
 In order to do this, you usually separate the task into several independent
sub-tasks, put into a queue and wait until it is finished.
 
 For example, we have a list of files to download from remote server. For each
file, we create a new connection and retrieve it independently (and, possibly,
out of order):
 
 import tango.core.ThreadPool;
 
 void execute()
 {
     auto pool = new ThreadPool!(string)(10); // create a thread pool with 10
threads
     string[] fileList = loadFileList();
     foreach (string filePath; fileList) {
         pool.append(&download, filePath);   // append the task
     }
 
     pool.finish(); // wait until all the tasks complete
 }
 
 void download(string fileName)
 {
     // ...
 }
 
 In this example, I create a lot of threads (more than hardware runs
concurrently) because the CPU is not a bottleneck.
 You may experience different results in different CPU workaload, though.
 
 Hope that helps.
Jan 09 2009
next sibling parent reply "Jarrett Billingsley" <jarrett.billingsley gmail.com> writes:
On Sat, Jan 10, 2009 at 12:30 AM, Jarrett Billingsley
<jarrett.billingsley gmail.com> wrote:

 On Sat, Jan 10, 2009 at 12:18 AM, yes <yes no.com> wrote:

 Does Phobos also do threadpools?


 From what I understand, not without modification.  At least, not
 efficiently.  Downs has implemented such a thing in scrapple.tools,
 but it requires a patch to Phobos to make it run at a reasonable
 speed.


Ffffffff.. ignore this, I was thinking about fiber pools.  I don't
think there's any technical reason why a ThreadPool couldn't be
written for Phobos, but there isn't one provided by default.
Jan 09 2009
parent reply downs <default_357-line yahoo.de> writes:
Jarrett Billingsley wrote:

 On Sat, Jan 10, 2009 at 12:30 AM, Jarrett Billingsley
 <jarrett.billingsley gmail.com> wrote:

 On Sat, Jan 10, 2009 at 12:18 AM, yes <yes no.com> wrote:

 Does Phobos also do threadpools?


 From what I understand, not without modification.  At least, not
 efficiently.  Downs has implemented such a thing in scrapple.tools,
 but it requires a patch to Phobos to make it run at a reasonable
 speed.


 
 Ffffffff.. ignore this, I was thinking about fiber pools.  I don't
 think there's any technical reason why a ThreadPool couldn't be
 written for Phobos, but there isn't one provided by default.


Scrapple.Tools has a Threadpool too :)
Jan 10 2009
parent "Jarrett Billingsley" <jarrett.billingsley gmail.com> writes:
On Sat, Jan 10, 2009 at 6:39 AM, downs <default_357-line yahoo.de> wrote:

 Jarrett Billingsley wrote:

 On Sat, Jan 10, 2009 at 12:30 AM, Jarrett Billingsley
 <jarrett.billingsley gmail.com> wrote:

 On Sat, Jan 10, 2009 at 12:18 AM, yes <yes no.com> wrote:

 Does Phobos also do threadpools?


 From what I understand, not without modification.  At least, not
 efficiently.  Downs has implemented such a thing in scrapple.tools,
 but it requires a patch to Phobos to make it run at a reasonable
 speed.


 Ffffffff.. ignore this, I was thinking about fiber pools.  I don't
 think there's any technical reason why a ThreadPool couldn't be
 written for Phobos, but there isn't one provided by default.


 Scrapple.Tools has a Threadpool too :)


NO!  *SLAM*
Jan 10 2009
prev sibling parent "Jarrett Billingsley" <jarrett.billingsley gmail.com> writes:
On Sat, Jan 10, 2009 at 12:18 AM, yes <yes no.com> wrote:

 Does Phobos also do threadpools?


From what I understand, not without modification.  At least, not
efficiently.  Downs has implemented such a thing in scrapple.tools,
but it requires a patch to Phobos to make it run at a reasonable
speed.


 Somehow I liked the idea of an Object like agent smith, just duplicate
yourself when the task is too big. ( ^_^ )


Hehe.
Jan 09 2009