www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Parallelism Map and Reduce

reply "Zardoz" <luis.panadero gmail.com> writes:
Hi! Now I have this code :

       auto acelByObjs = map!( (Entity o) {
           Vector3 r = o.pos[0] - pos[0];
           return r * (o.mass / pow((r.sq_length + epsilon2), 
1.5));
           } )(objects);
       newAcel = reduce!("a + b")(acelByObjs);

It works very well with the std.algorithm Map and Reduce but when 
I try to use std.parallelism versions of it, parallel Map give me 
this compilataion errors :

entity.d(63): Error: template instance map!(delegate  system 
Vector3(Entity o)
{
Vector3 r = o.pos[0LU].opBinary(this.pos[0LU]);
return r.opBinary(o.mass / pow(r.sq_length() + epsilon2,1.5));
}
) cannot use local '__lambda3' as parameter to non-global 
template map(functions...)
/usr/include/dmd/phobos/std/parallelism.d(1969): Error: function 
std.parallelism.TaskPool.map!(delegate  system Vector3(Entity o)
{
Vector3 r = o.pos[0LU].opBinary(this.pos[0LU]);
return r.opBinary(o.mass / pow(r.sq_length() + epsilon2,1.5));
}
).map!(Entity[]).map.Map.fillBuf cannot access frame of function 
entity.Entity.CalcAcel
/usr/include/dmd/phobos/std/parallelism.d(1974): Error: template 
instance amap!(delegate  system Vector3(Entity o)
{
Vector3 r = o.pos[0LU].opBinary(this.pos[0LU]);
return r.opBinary(o.mass / pow(r.sq_length() + epsilon2,1.5));
}
) cannot use local '__lambda3' as parameter to non-global 
template amap(functions...)
/usr/include/dmd/phobos/std/parallelism.d(1675): Error: function 
std.parallelism.TaskPool.amap!(delegate  system Vector3(Entity o)
{
Vector3 r = o.pos[0LU].opBinary(this.pos[0LU]);
return r.opBinary(o.mass / pow(r.sq_length() + epsilon2,1.5));
}
).amap!(Entity[],ulong,Vector3[]).amap cannot access frame of 
function entity.Entity.CalcAcel
/usr/include/dmd/phobos/std/parallelism.d(1706): Error: function 
std.parallelism.TaskPool.amap!(delegate  system Vector3(Entity o)
{
Vector3 r = o.pos[0LU].opBinary(this.pos[0LU]);
return r.opBinary(o.mass / pow(r.sq_length() + epsilon2,1.5));
}
).amap!(Entity[],ulong,Vector3[]).amap.doIt cannot access frame 
of function entity.Entity.CalcAcel
/usr/include/dmd/phobos/std/parallelism.d(1974): Error: template 
instance std.parallelism.TaskPool.amap!(delegate  system 
Vector3(Entity o)
{
Vector3 r = o.pos[0LU].opBinary(this.pos[0LU]);
return r.opBinary(o.mass / pow(r.sq_length() + epsilon2,1.5));
}
).amap!(Entity[],ulong,Vector3[]) error instantiating
entity.d(66):        instantiated from here: map!(Entity[])
entity.d(66): Error: template instance 
std.parallelism.TaskPool.map!(delegate  system Vector3(Entity o)
{
Vector3 r = o.pos[0LU].opBinary(this.pos[0LU]);
return r.opBinary(o.mass / pow(r.sq_length() + epsilon2,1.5));
}
).map!(Entity[]) error instantiating
make: *** [predSim] Error 1

Plus , if  I only the parallel Reduce, It takes more around 9 
times more that the serial versión for an array of 30 elements!
I tested before parallel reduction with the example code of 
calculating PI, in my quad core machine taking 4 times less that 
the serial versión... I will try with a bigger array but using 
parallel foreach takes less times that serial versión, so I don't 
think that the these big extra time comes for making the theads. 
Any other idea of what can be hapening here ?
Dec 11 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 12/11/2012 02:53 AM, Zardoz wrote:


 auto acelByObjs = map!( (Entity o) {
 Vector3 r = o.pos[0] - pos[0];
 return r * (o.mass / pow((r.sq_length + epsilon2), 1.5));
 } )(objects);
 newAcel = reduce!("a + b")(acelByObjs);

 It works very well with the std.algorithm Map and Reduce but when I try
 to use std.parallelism versions of it, parallel Map give me this
 compilataion errors :

 entity.d(63): Error: template instance map!(delegate  system
 Vector3(Entity o)
 {
 Vector3 r = o.pos[0LU].opBinary(this.pos[0LU]);
 return r.opBinary(o.mass / pow(r.sq_length() + epsilon2,1.5));
 }
 ) cannot use local '__lambda3' as parameter to non-global template
 map(functions...)


That used to work a couple of dmd versions ago. I think it was a bug 
that it worked, so it stopped working after bug fixes.

If I'm not mistaken this is actually related to a compiler 
implementation issue: Lambda's have a single pointer to store the 
context that they have been started in.

When a lambda is a free-standing function (aka "module function" or 
"global function") then there is only the context to deal with. When the 
template is a member function (taskPool.map is) then there is also the 
object that the function is started on.

The single pointer of the lambda is not sufficient to store both without 
big changes in the compiler.

(I may be off with that description above. e.g. there may be two 
pointers when three are actually needed, etc.)

I had to change following chapter after dmd's behavior had changed:

   http://ddili.org/ders/d.en/parallelism.html

--- Quoting ---
import std.parallelism;
// ...
double averageGrade(Student student)
{
     return student.averageGrade;
}
// ...
     auto results = taskPool.map!averageGrade(students, 3);

Note: The free-standing averageGrade() function above is needed due to a 
limitation that involves using local delegates with member function 
templates like TaskPool.map:

   auto results = taskPool.map!(a => a.averageGrade)(students, 3); 
// ← compilation ERROR
----------

As you see above, the solution is to use a function with taskPool.map, 
not a lambda.

Ali
Dec 11 2012
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Ali Çehreli:


 The single pointer of the lambda is not sufficient to store 
 both without big changes in the compiler.


I think adding a heavier 3-word delegate is not too much hard to 
do. But it makes the language more complex, so so far Walter is 
not willing to introduce them.

But in the end introducing them may become inevitable :-) I think 
such hypothetical 3-word delegates need to be discussed in the 
main D newsgroup.

Bye,
bearophile
Dec 11 2012
prev sibling parent reply "Zardoz" <luis.panadero gmail.com> writes:
On Tuesday, 11 December 2012 at 15:22:49 UTC, Ali Çehreli wrote:

 That used to work a couple of dmd versions ago. I think it was 
 a bug that it worked, so it stopped working after bug fixes.

 If I'm not mistaken this is actually related to a compiler 
 implementation issue: Lambda's have a single pointer to store 
 the context that they have been started in.

 When a lambda is a free-standing function (aka "module 
 function" or "global function") then there is only the context 
 to deal with. When the template is a member function 
 (taskPool.map is) then there is also the object that the 
 function is started on.

 The single pointer of the lambda is not sufficient to store 
 both without big changes in the compiler.

 (I may be off with that description above. e.g. there may be 
 two pointers when three are actually needed, etc.)

 I had to change following chapter after dmd's behavior had 
 changed:

   http://ddili.org/ders/d.en/parallelism.html

 --- Quoting ---
 import std.parallelism;
 // ...
 double averageGrade(Student student)
 {
     return student.averageGrade;
 }
 // ...
     auto results = taskPool.map!averageGrade(students, 3);

 Note: The free-standing averageGrade() function above is needed 
 due to a limitation that involves using local delegates with 
 member function templates like TaskPool.map:

   auto results = taskPool.map!(a => a.averageGrade)(students, 
 3); // ← compilation ERROR
 ----------

 As you see above, the solution is to use a function with 
 taskPool.map, not a lambda.

 Ali


I try to use a function instead of a lambda function I'm keep 
getting compiler errors. Code :
       Entity MapIntegrator (ref Entity me) {
         me.Integrador3Orden (iDeltaT);
         return me;
       }
       objects = array( taskPool.map!MapIntegrator(objects) );

With taskPool.Map I get this errors :
simulator.d(196): Error: template instance map!(MapIntegrator) 
cannot use local 'MapIntegrator' as parameter to non-global 
template map(functions...)
/usr/include/dmd/phobos/std/parallelism.d(1969): Error: function 
std.parallelism.TaskPool.map!(MapIntegrator).map!(Entity[]).map.Map.fillBuf 
cannot access frame of function D main
/usr/include/dmd/phobos/std/parallelism.d(1974): Error: template 
instance amap!(MapIntegrator) cannot use local 'MapIntegrator' as 
parameter to non-global template amap(functions...)
/usr/include/dmd/phobos/std/parallelism.d(1675): Error: function 
std.parallelism.TaskPool.amap!(MapIntegrator).amap!(Entity[],
long,Entity[]).amap 
cannot access frame of function D main
/usr/include/dmd/phobos/std/parallelism.d(1706): Error: function 
std.parallelism.TaskPool.amap!(MapIntegrator).amap!(Entity[],ulong
Entity[]).amap.doIt 
cannot access frame of function D main
/usr/include/dmd/phobos/std/parallelism.d(1974): Error: template 
instance 
std.parallelism.TaskPool.amap!(MapIntegrator).amap!(Entity[],ulong,Entity[]) 
error instantiating
simulator.d(196):        instantiated from here: map!(Entity[])
simulator.d(196): Error: template instance 
std.parallelism.TaskPool.map!(MapIntegrator).map!(Entity[]) error 
instantiating
make: *** [predSim] Error 1

But again, with std.algorthim Map it don give any error and works 
fine.
Dec 11 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 12/11/2012 08:12 AM, Zardoz wrote:


 I try to use a function instead of a lambda function I'm keep getting
 compiler errors. Code :
 Entity MapIntegrator (ref Entity me) {
 me.Integrador3Orden (iDeltaT);
 return me;
 }
 objects = array( taskPool.map!MapIntegrator(objects) );

 With taskPool.Map I get this errors :
 simulator.d(196): Error: template instance map!(MapIntegrator) cannot
 use local 'MapIntegrator' as parameter to non-global template
 map(functions...)


Could you please move MapIntegrator() to module-level. Then it should work.

Ali
Dec 11 2012
parent reply "Zardoz" <luis.panadero gmail.com> writes:
On Tuesday, 11 December 2012 at 17:50:31 UTC, Ali Çehreli wrote:

 On 12/11/2012 08:12 AM, Zardoz wrote:


 Could you please move MapIntegrator() to module-level. Then it 
 should work.

 Ali


I try it and now even with normal Map function give me errors 
with dmd !

public Entity MapIntegrator ( Entity me) {
   me.Integrador3Orden ();
   return me;
}

void main() {
   Entity[] objects;
...
   objects = array( map!MapIntegrator(objects) );
...
}

With this error :
dmd -w -wi -version=SReduction simulator.d entity.d vector.d 
-ofreduceSim
simulator.d(194): Error: template 
std.algorithm.map!(MapIntegrator).map does not match any function 
template declaration
/usr/include/dmd/phobos/std/algorithm.d(369): Error: template 
std.algorithm.map!(MapIntegrator).map(Range) if 
(isInputRange!(Unqual!(Range))) cannot deduce template function 
from argument types !()(Entity[])
Dec 12 2012
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 12/12/2012 05:47 AM, Zardoz wrote:

 On Tuesday, 11 December 2012 at 17:50:31 UTC, Ali Çehreli wrote:

 On 12/11/2012 08:12 AM, Zardoz wrote:


 Could you please move MapIntegrator() to module-level. Then it should
 work.

 Ali


 I try it and now even with normal Map function give me errors with dmd !

 public Entity MapIntegrator ( Entity me) {
 me.Integrador3Orden ();
 return me;
 }

 void main() {
 Entity[] objects;
 ...
 objects = array( map!MapIntegrator(objects) );
 ...
 }

 With this error :
 dmd -w -wi -version=SReduction simulator.d entity.d vector.d -ofreduceSim
 simulator.d(194): Error: template std.algorithm.map!(MapIntegrator).map
 does not match any function template declaration
 /usr/include/dmd/phobos/std/algorithm.d(369): Error: template
 std.algorithm.map!(MapIntegrator).map(Range) if
 (isInputRange!(Unqual!(Range))) cannot deduce template function from
 argument types !()(Entity[])


Strange. The following program works for me with dmd 2.060. It uses both 
the regular and parallel versions of map and reduce:

import std.array;
import std.algorithm;
import std.parallelism;

struct Entity
{
     void Integrador3Orden()
     {}
}

public Entity MapIntegrator ( Entity me) {
   me.Integrador3Orden ();
   return me;
}

void main() {
   Entity[] objects;

   objects = array( map!MapIntegrator(objects) );
   objects = array(taskPool.map!MapIntegrator(objects));

   int[] acelByObjs;
   int reduced = reduce!"a + b"(0, acelByObjs);
   reduced = taskPool.reduce!"a + b"(0, acelByObjs);
}

Ali
Dec 12 2012