www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - ndslice, using a slice in place of T[] in template parameters

reply Jay Norwood <jayn prismnet.com> writes:
I cut this median template from Jack Stouffer's article and was 
attempting to use it in  a parallel function.  As shown, it 
builds and execute correctly, but it failed to compile if I 
attempting to use
medians[i] = median(vec,slb[task]);

in place of the
medians[i] = median(vec,dbuf[j .. k]);

Is there a cast needed?

================
import std.array : array;
import std.algorithm;
import std.datetime;
import std.conv : to;
import std.stdio;
import std.experimental.ndslice;

shared double[] medians;
double[] data;
shared double[] dbuf;
int numTasks;
const int smalld = 1000;
const int bigd = 10_000;
const int fulld = bigd*smalld;

/**
Params:
r = input range
buf = buffer with length no less than the number of elements in 
`r`
Returns:
median value over the range `r`
*/
T median(Range, T)(Range r, T[] buf)
{
     import std.algorithm.sorting: sort;

     size_t n;

     foreach (e; r) {
         buf[n++] = e;
     }

     buf[0 .. n].sort();
     immutable m = n >> 1;
     return n & 1 ? buf[m] : cast(T)((buf[m - 1] + buf[m]) / 2);
}


void f3() {
     import std.parallelism;
     auto sl = data.sliced(smalld,bigd);
     auto slb = dbuf.sliced(numTasks,bigd);
     foreach(i,vec; parallel(sl)){
         int task = taskPool.workerIndex;
         int j = task*bigd;
         int k = j+bigd;
         medians[i] = median(vec,dbuf[j .. k]);
     }
}


void main() {
     import std.parallelism;
     numTasks = taskPool.size+1;
     data = new double[fulld];
     dbuf = new double[bigd*numTasks];
     medians = new double[smalld];

     for(int i=0;i<fulld;i++){ data[i] = i/(fulld*1.0);}
     StopWatch sw3;

     sw3.start();
     f3() ;
     auto r3 = sw3.peek().msecs;
     writeln("medians parallel:",medians);

     writeln("parallel time medians msec:",r3);
}
Jan 10 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 10 January 2016 at 22:00:20 UTC, Jay Norwood wrote:

 I cut this median template from Jack Stouffer's article and was 
 attempting to use it in  a parallel function.  As shown, it 
 builds and execute correctly, but it failed to compile if I 
 attempting to use
 medians[i] = median(vec,slb[task]);

 [...]


Could you please provide full code and error (git gists)? -- Ilya
Jan 10 2016
parent reply Jay Norwood <jayn prismnet.com> writes:
On Sunday, 10 January 2016 at 22:23:18 UTC, Ilya Yaroshenko wrote:


 Could you please provide full code and error (git gists)? -- 
 Ilya


ok, thanks.
I'm building with DMD32 D Compiler v2.069.2 on Win32.  The 
dub.json is included.

https://gist.github.com/jnorwood/affd05b69795c20989a3
Jan 10 2016
next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 10 January 2016 at 23:24:24 UTC, Jay Norwood wrote:

 On Sunday, 10 January 2016 at 22:23:18 UTC, Ilya Yaroshenko 
 wrote:


 Could you please provide full code and error (git gists)? -- 
 Ilya


 ok, thanks.
 I'm building with DMD32 D Compiler v2.069.2 on Win32.  The 
 dub.json is included.

 https://gist.github.com/jnorwood/affd05b69795c20989a3


Just use normal arrays for buffer (median accepts array on second 
argument for optimisation reasons).

BTW, dip80-ndslice moved to http://code.dlang.org/packages/mir
-- Ilya
Jan 10 2016
parent reply Jay Norwood <jayn prismnet.com> writes:
On Sunday, 10 January 2016 at 23:31:47 UTC, Ilya Yaroshenko wrote:

 Just use normal arrays for buffer (median accepts array on 
 second argument for optimisation reasons).


ok, I think I see. I created a slice(numTasks, bigd) over an 
allocated double[] dbuf, but slb[task] will be returning some 
struct instead of the double[] that i need in this case.

If I add .array to the Slice, it does compile, and executes, but 
slower than using the buffer directly.

medians[i] = median(vec, slb[task].array);
parallel time medians msec:113

original version using the computed slice of the original 
allocated dbuf.
medians[i] = median(vec,dbuf[j .. k]);
parallel time medians msec:85

The .array appears to make a copy. Is there some other call in 
ndslice to return the double[] slice of the original array?
Jan 10 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 11 January 2016 at 00:39:04 UTC, Jay Norwood wrote:

 On Sunday, 10 January 2016 at 23:31:47 UTC, Ilya Yaroshenko 
 wrote:

 Just use normal arrays for buffer (median accepts array on 
 second argument for optimisation reasons).


 ok, I think I see. I created a slice(numTasks, bigd) over an 
 allocated double[] dbuf, but slb[task] will be returning some 
 struct instead of the double[] that i need in this case.

 If I add .array to the Slice, it does compile, and executes, 
 but slower than using the buffer directly.

 medians[i] = median(vec, slb[task].array);
 parallel time medians msec:113

 original version using the computed slice of the original 
 allocated dbuf.
 medians[i] = median(vec,dbuf[j .. k]);
 parallel time medians msec:85

 The .array appears to make a copy. Is there some other call in 
 ndslice to return the double[] slice of the original array?


I will add such function. But it is not safe to do so (Slice can 
have strides not equal to 1). So it is like a hack (&ret[0, 0, 
0])[0 .. ret.elementsCount]).

Have you made comparison between my and yours parallel versions?
https://github.com/9il/examples/blob/parallel/image_processing/median-filter/source/app.d
-- Ilya
Jan 10 2016
parent Jay Norwood <jayn prismnet.com> writes:
On Monday, 11 January 2016 at 00:50:37 UTC, Ilya Yaroshenko wrote:

 I will add such function. But it is not safe to do so (Slice 
 can have strides not equal to 1). So it is like a hack (&ret[0, 
 0, 0])[0 .. ret.elementsCount]).

 Have you made comparison between my and yours parallel versions?
 https://github.com/9il/examples/blob/parallel/image_processing/median-filter/source/app.d
 -- Ilya


Thanks.  No, I haven't studied it previously, but I see how you 
used the 'hack' in your code, and it works out to the statement 
below in my case.

medians[i] = median(vec, (&slb[task,0])[0 .. bigd]);

which compiled.  It ran in the faster time without the .array 
copying.
parallel time medians msec:87

That 'hack' seems to be related to the third from below.
https://dlang.org/spec/arrays.html
b = a;
b = a[];
b = a[0 .. a.length];
Jan 10 2016
prev sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 10 January 2016 at 23:24:24 UTC, Jay Norwood wrote:

 On Sunday, 10 January 2016 at 22:23:18 UTC, Ilya Yaroshenko 
 wrote:


 Could you please provide full code and error (git gists)? -- 
 Ilya


 ok, thanks.
 I'm building with DMD32 D Compiler v2.069.2 on Win32.  The 
 dub.json is included.

 https://gist.github.com/jnorwood/affd05b69795c20989a3


I have create parallel test to (it requires mir v0.10.0-beta )
https://github.com/9il/examples/blob/parallel/image_processing/median-filter/source/app.d
Could you please create a benchmark with default values of nc & 
nc for single thread app, your parallel version, and my.
My version has some additional overhead and I am interesting if 
it is significant.
-- Ilya
Jan 10 2016