digitalmars.D.announce - Using D and std.ndslice as a Numpy Replacement

Jack Stouffer (2/2) Jan 02 2016 http://jackstouffer.com/blog/nd_slice.html

=?UTF-8?Q?Ali_=c3=87ehreli?= (4/6) Jan 02 2016 Broken link in "For a more in depth look at ranges, see The official D

Jack Stouffer (2/5) Jan 02 2016 Fixed, thanks!

Ilya Yaroshenko (7/9) Jan 02 2016 I just wanted to write to you that dip80-ndslice was moved to mir

Ilya Yaroshenko (6/17) Jan 02 2016 EDIT:

Andrei Alexandrescu (2/21) Jan 03 2016 What is the relationship between mir and std.experimental.ndslice? -- An...

Ilya (8/35) Jan 03 2016 1. mir.ndslice is a developer version of std.experimental.ndslice

Andrei Alexandrescu (3/8) Jan 03 2016 The care you show for your users is impressive. We should take a page

Ilya Yaroshenko (7/9) Jan 02 2016 Jack, Thank you for the article!

Jack Stouffer (4/6) Jan 02 2016 I don't know what you mean here, I made sure to call

Ilya Yaroshenko (15/21) Jan 02 2016 In the article:

Jack Stouffer (6/28) Jan 03 2016 I still have to disagree with you that the example I submitted

Ilya Yaroshenko (6/14) Jan 03 2016 No, for real world math calculations, most std.ndslice will be at
David Nadlinger (6/11) Jan 03 2016 What leads you to this belief? (Beyond cache locality

Jack Stouffer (7/14) Jan 03 2016 The example in the article and the example I submitted to

Ilya (3/19) Jan 03 2016 To be clear: there is NO data in Article example. Only CPU

Jack Stouffer (8/10) Jan 03 2016 Ok, I see were I made the mistake, I apologize. I believed that

Ilya (7/18) Jan 03 2016 Please add links to

Jack Stouffer (5/11) Jan 04 2016 Updated. I also changed the example slightly from the

Walter Bright (3/5) Jan 02 2016 https://news.ycombinator.com/item?id=10828450 (but access through the fr...
jmh530 (2/4) Jan 02 2016 Great piece. And great work done by those who worked on nd_slice.
Joakim (3/5) Jan 03 2016 Nicely written, good to see you explain all the code, enjoyed

Jack Stouffer <jack jackstouffer.com> writes:

http://jackstouffer.com/blog/nd_slice.html

https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

Jan 02 2016

=?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:

On 01/02/2016 11:49 AM, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

Broken link in "For a more in depth look at ranges, see The official D 
tutorial's section on ranges".

Ali

Jan 02 2016

Jack Stouffer <jack jackstouffer.com> writes:

On Saturday, 2 January 2016 at 22:15:03 UTC, Ali Çehreli wrote:
 Broken link in "For a more in depth look at ranges, see The 
 official D tutorial's section on ranges".

 Ali

Fixed, thanks!

Jan 02 2016

Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:

On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

I just wanted to write to you that dip80-ndslice was moved to mir 
http://code.dlang.org/packages/mir

     "dependencies": {
         "dip80-ndslice": "~>0.8.7"
     },

Ilya

Jan 02 2016

Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:

On Saturday, 2 January 2016 at 23:23:38 UTC, Ilya Yaroshenko 
wrote:
 On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer 
 wrote:
 http://jackstouffer.com/blog/nd_slice.html

https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

 I just wanted to write to you that dip80-ndslice was moved to 
 mir http://code.dlang.org/packages/mir

     "dependencies": {
         "dip80-ndslice": "~>0.8.7"
     },

 Ilya

EDIT:

	"dependencies": {
		"mir": "~>0.9.0-beta"
	}

Jan 02 2016

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 1/2/16 6:24 PM, Ilya Yaroshenko wrote:
 On Saturday, 2 January 2016 at 23:23:38 UTC, Ilya Yaroshenko wrote:
 On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

 I just wanted to write to you that dip80-ndslice was moved to mir
 http://code.dlang.org/packages/mir

     "dependencies": {
         "dip80-ndslice": "~>0.8.7"
     },

 Ilya

 EDIT:

      "dependencies": {
          "mir": "~>0.9.0-beta"
      }

What is the relationship between mir and std.experimental.ndslice? -- Andrei

Jan 03 2016

Ilya <ilyayaroshenko gmail.com> writes:

On Sunday, 3 January 2016 at 23:18:16 UTC, Andrei Alexandrescu 
wrote:
 On 1/2/16 6:24 PM, Ilya Yaroshenko wrote:
 On Saturday, 2 January 2016 at 23:23:38 UTC, Ilya Yaroshenko 
 wrote:
 On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer 
 wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

 I just wanted to write to you that dip80-ndslice was moved to 
 mir
 http://code.dlang.org/packages/mir

     "dependencies": {
         "dip80-ndslice": "~>0.8.7"
     },

 Ilya

 EDIT:

      "dependencies": {
          "mir": "~>0.9.0-beta"
      }

 What is the relationship between mir and 
 std.experimental.ndslice? -- Andrei

1. mir.ndslice is a developer version of std.experimental.ndslice
2. mir can be used with DMD front end >= 2.068, so ndslice can be 
used with LDC 0.17.0-alpha. It is important for benchmarks.
3. mir is going to be a testing package for the future std.la 
(generic BLAS implementation)

-- Ilya

Jan 03 2016

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 1/3/16 6:52 PM, Ilya wrote:
 1. mir.ndslice is a developer version of std.experimental.ndslice
 2. mir can be used with DMD front end >= 2.068, so ndslice can be used
 with LDC 0.17.0-alpha. It is important for benchmarks.
 3. mir is going to be a testing package for the future std.la (generic
 BLAS implementation)

The care you show for your users is impressive. We should take a page 
from your book, hopefully with your own help. Thanks! -- Andrei

Jan 03 2016

Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:

On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

Jack, Thank you for the article!

Link to the benchmark at GitHub 
https://github.com/DlangScience/examples/blob/master/mean_of_columns.d

This benchmark is _not_ lazy, so ndslice faster than Numpy only 
3.5 times.
  -- Ilya

Jan 02 2016

Jack Stouffer <jack jackstouffer.com> writes:

On Saturday, 2 January 2016 at 23:51:09 UTC, Ilya Yaroshenko 
wrote:
 This benchmark is _not_ lazy, so ndslice faster than Numpy only 
 3.5 times.

I don't know what you mean here, I made sure to call 
std.array.array to force allocation.

Jan 02 2016

Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:

On Sunday, 3 January 2016 at 00:09:33 UTC, Jack Stouffer wrote:
 On Saturday, 2 January 2016 at 23:51:09 UTC, Ilya Yaroshenko 
 wrote:
 This benchmark is _not_ lazy, so ndslice faster than Numpy 
 only 3.5 times.

 I don't know what you mean here, I made sure to call 
 std.array.array to force allocation.

In the article:
     auto means = 100_000.iota <---- 100_000.iota is lazy range
         .sliced(100, 1000)
         .transposed
         .map!(r => sum(r) / r.length)
         .array;               <---- allocation of the result

In GitHub:
     means = data           <---- data is allocated array, it is 
fair test for real world
         .sliced(100, 1000)
         .transposed
         .map!(r => sum(r, 0L) / cast(double) r.length)
         .array;             <---- allocation of the result
  -- Ilya

Jan 02 2016

Jack Stouffer <jack jackstouffer.com> writes:

On Sunday, 3 January 2016 at 00:17:23 UTC, Ilya Yaroshenko wrote:
 On Sunday, 3 January 2016 at 00:09:33 UTC, Jack Stouffer wrote:
 On Saturday, 2 January 2016 at 23:51:09 UTC, Ilya Yaroshenko 
 wrote:
 This benchmark is _not_ lazy, so ndslice faster than Numpy 
 only 3.5 times.

 I don't know what you mean here, I made sure to call 
 std.array.array to force allocation.

 In the article:
     auto means = 100_000.iota <---- 100_000.iota is lazy range
         .sliced(100, 1000)
         .transposed
         .map!(r => sum(r) / r.length)
         .array;               <---- allocation of the result

 In GitHub:
     means = data           <---- data is allocated array, it is 
 fair test for real world
         .sliced(100, 1000)
         .transposed
         .map!(r => sum(r, 0L) / cast(double) r.length)
         .array;             <---- allocation of the result
  -- Ilya

I still have to disagree with you that the example I submitted 
was fair. Accessing global memory in D is going to be much slower 
than accessing stack memory, and sense most std.ndslice 
calculations are going to be on the stack, I believe my benchmark 
is indicative of normal use.

Jan 03 2016

Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:

On Sunday, 3 January 2016 at 18:56:07 UTC, Jack Stouffer wrote:
 On Sunday, 3 January 2016 at 00:17:23 UTC, Ilya Yaroshenko 
 wrote:
  [...]

 I still have to disagree with you that the example I submitted 
 was fair. Accessing global memory in D is going to be much 
 slower than accessing stack memory, and sense most std.ndslice 
 calculations are going to be on the stack, I believe my 
 benchmark is indicative of normal use.

No, for real world math calculations, most std.ndslice will be at 
global memory.
Examples: all SciD (if port it to ndslice), future BLAS, future 
LAPACK.
-- Ilya

Jan 03 2016

David Nadlinger <code klickverbot.at> writes:

On Sunday, 3 January 2016 at 18:56:07 UTC, Jack Stouffer wrote:
 I still have to disagree with you that the example I submitted 
 was fair. Accessing global memory in D is going to be much 
 slower than accessing stack memory, […]

What leads you to this belief? (Beyond cache locality 
considerations, which are not so important if the data is large.)

 and sense most std.ndslice calculations are going to be on the 
 stack, I believe my benchmark is indicative of normal use.

Your iota example does not read the data from memory at all 
(neither stack nor heap), instead computing it on the fly.

  — David

Jan 03 2016

Jack Stouffer <jack jackstouffer.com> writes:

On Monday, 4 January 2016 at 00:24:51 UTC, David Nadlinger wrote:
 On Sunday, 3 January 2016 at 18:56:07 UTC, Jack Stouffer wrote:
 I still have to disagree with you that the example I submitted 
 was fair. Accessing global memory in D is going to be much 
 slower than accessing stack memory, […]

 What leads you to this belief? (Beyond cache locality 
 considerations, which are not so important if the data is 
 large.)

The example in the article and the example I submitted to 
DlangScience/examples have very different speeds: 
https://github.com/DlangScience/examples/blob/master/mean_of_columns.d

Article example:                  5 µs
DlangScience example:  145 µs

Both when compiled with LDC

Jan 03 2016

Ilya <ilyayaroshenko gmail.com> writes:

On Monday, 4 January 2016 at 01:03:49 UTC, Jack Stouffer wrote:
 On Monday, 4 January 2016 at 00:24:51 UTC, David Nadlinger 
 wrote:
 On Sunday, 3 January 2016 at 18:56:07 UTC, Jack Stouffer wrote:
 I still have to disagree with you that the example I 
 submitted was fair. Accessing global memory in D is going to 
 be much slower than accessing stack memory, […]

 What leads you to this belief? (Beyond cache locality 
 considerations, which are not so important if the data is 
 large.)

 The example in the article and the example I submitted to 
 DlangScience/examples have very different speeds: 
 https://github.com/DlangScience/examples/blob/master/mean_of_columns.d

 Article example:                  5 µs
 DlangScience example:  145 µs

 Both when compiled with LDC

To be clear: there is NO data in Article example. Only CPU 
registers are used. It is not fair. -- Ilya

Jan 03 2016

Jack Stouffer <jack jackstouffer.com> writes:

On Monday, 4 January 2016 at 01:09:30 UTC, Ilya wrote:
 To be clear: there is NO data in Article example. Only CPU 
 registers are used. It is not fair. -- Ilya

Ok, I see were I made the mistake, I apologize. I believed that 
since I was only testing the np.mean line of code, that the lazy 
generation of the D in the D version would have no effects on the 
comparison. But I forgot the fact that the lazy generation would 
mean that no memory needed to be accessed in the D code and 
therefore it's not apples to apples with the Numpy code.

Thank you for clarifying this, I will update the article.

Jan 03 2016

Ilya <ilyayaroshenko gmail.com> writes:

On Monday, 4 January 2016 at 01:47:04 UTC, Jack Stouffer wrote:
 On Monday, 4 January 2016 at 01:09:30 UTC, Ilya wrote:
 To be clear: there is NO data in Article example. Only CPU 
 registers are used. It is not fair. -- Ilya

 Ok, I see were I made the mistake, I apologize. I believed that 
 since I was only testing the np.mean line of code, that the 
 lazy generation of the D in the D version would have no effects 
 on the comparison. But I forgot the fact that the lazy 
 generation would mean that no memory needed to be accessed in 
 the D code and therefore it's not apples to apples with the 
 Numpy code.

 Thank you for clarifying this, I will update the article.

Please add links to
1. Mir: http://code.dlang.org/packages/mir
2. Documentation: 
http://dlang.org/phobos-prerelease/std_experimental_ndslice.html
3. DlangScience example

Thanks! -- Ilya

Jan 03 2016

Jack Stouffer <jack jackstouffer.com> writes:

On Monday, 4 January 2016 at 02:01:05 UTC, Ilya wrote:
 Please add links to
 1. Mir: http://code.dlang.org/packages/mir
 2. Documentation: 
 http://dlang.org/phobos-prerelease/std_experimental_ndslice.html
 3. DlangScience example

 Thanks! -- Ilya

Updated. I also changed the example slightly from the 
DlangScience example by increasing the array size because some 
people on HN were complaining that the size of the data wasn't 
large enough for the benchmark to be meaningful.

Jan 04 2016

Walter Bright <newshound2 digitalmars.com> writes:

On 1/2/2016 11:49 AM, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

https://news.ycombinator.com/item?id=10828450 (but access through the front 
page, not this link)

Jan 02 2016

jmh530 <john.michael.hall gmail.com> writes:

On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

Great piece. And great work done by those who worked on nd_slice.

Jan 02 2016

Joakim <dlang joakim.fea.st> writes:

On Saturday, 2 January 2016 at 19:49:05 UTC, Jack Stouffer wrote:
 http://jackstouffer.com/blog/nd_slice.html

 https://www.reddit.com/r/programming/comments/3z6f7a/using_d_and_stdndslice_as_a_numpy_replacement/

Nicely written, good to see you explain all the code, enjoyed 
reading it.

Jan 03 2016

D Programming

C/C++ Programming

Other

digitalmars.D.announce - Using D and std.ndslice as a Numpy Replacement