www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Finally full multidimensional arrays support in D

reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
Multidimensional arrays indexing and slicing syntax is finally added [1] 
(thanks to Kenji Hara). So it was a good cause to update my 
multidimensional arrays library implementation and add support for the 
new syntax. So here we are: [2].

Also should we add it to the standard library?

[1] https://github.com/D-Programming-Language/dmd/pull/443
[2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html

-- 
Денис В. Шеломовский
Denis V. Shelomovskij
Mar 17 2014
next sibling parent "watcher" <watcher watcher.com> writes:
+100



On Monday, 17 March 2014 at 17:39:41 UTC, Denis Shelomovskij 
wrote:

 Multidimensional arrays indexing and slicing syntax is finally 
 added [1] (thanks to Kenji Hara). So it was a good cause to 
 update my multidimensional arrays library implementation and 
 add support for the new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] 
 http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html
Mar 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Mar 17, 2014 at 09:39:40PM +0400, Denis Shelomovskij wrote:

 Multidimensional arrays indexing and slicing syntax is finally added
 [1] (thanks to Kenji Hara). So it was a good cause to update my
 multidimensional arrays library implementation and add support for
 the new syntax. So here we are: [2].
 
 Also should we add it to the standard library?
 
 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html


[...]

Let's do it! Put it up for review!

I almost gave up hope that this day would come, but this morning I
looked at the dmd commit log and saw Kenji's pull was merged. You
couldn't believe how happy I was. Finally, sane multidimensional array
support!!


T

-- 
Almost all proofs have bugs, but almost all theorems are true. -- Paul Pedersen
Mar 17 2014
prev sibling next sibling parent reply "Jared Miller" <jared economicmodeling.com> writes:
Good news and kudos to Kenji.

And yes, I think that a matrix / linear algebra library, as well 
as NumPy-style ND-Arrays are great candidates for future Phobos 
modules.

We've done a good bit of work with matrices at my workplace 
(EMSI), so I bet someone here could find time to contribute a 
bit. As a start, we've got some basic BLAS and MKL bindings that 
we really should clean up and get into Deimos.

On Monday, 17 March 2014 at 17:39:41 UTC, Denis Shelomovskij 
wrote:

 Multidimensional arrays indexing and slicing syntax is finally 
 added [1] (thanks to Kenji Hara). So it was a good cause to 
 update my multidimensional arrays library implementation and 
 add support for the new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] 
 http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html
Mar 17 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Jared Miller:


 And yes, I think that a matrix / linear algebra library, as 
 well as NumPy-style ND-Arrays are great candidates for future 
 Phobos modules.


I suggest to not put such library in Phobos before few years of 
usage in the wild.

Bye,
bearophile
Mar 17 2014
parent reply "CJS" <Prometheus85 hotmail.com> writes:
On Monday, 17 March 2014 at 21:25:34 UTC, bearophile wrote:

 Jared Miller:


 And yes, I think that a matrix / linear algebra library, as 
 well as NumPy-style ND-Arrays are great candidates for future 
 Phobos modules.


 I suggest to not put such library in Phobos before few years of 
 usage in the wild.


+1

Good matrix support would be awesome. But getting it wrong would 
be a catastrophe. I don't really support ever putting it in 
phobos, but if it is, then it should only be added after lots of 
experience.
Apr 17 2014
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Apr 17, 2014 at 03:16:20PM +0000, CJS via Digitalmars-d wrote:

 On Monday, 17 March 2014 at 21:25:34 UTC, bearophile wrote:

Jared Miller:


And yes, I think that a matrix / linear algebra library, as well as
NumPy-style ND-Arrays are great candidates for future Phobos
modules.


I suggest to not put such library in Phobos before few years of usage
in the wild.


 
 +1
 
 Good matrix support would be awesome. But getting it wrong would be a
 catastrophe. I don't really support ever putting it in phobos, but if
 it is, then it should only be added after lots of experience.


I've been longing for a high-quality, flexible, generic linear algebra
library in D. I don't have the time / resources to implement it myself,
otherwise I would.

But I agree that any such candidate library needs to be put in real-life
use for a while before being considered for Phobos.

I think the first step would be to refine Denis' n-dimensional array
library until it's Phobos-quality, then linear algebra specific
adaptations can be built on top. I think the two should be separated,
even if they are still related. Conflating 2D arrays with matrices at a
fundamental level is a mistake IMO. 2D arrays are just one of the
possible representations of a matrix, and any linear algebra library
should be flexible enough to use other representations (e.g., sparse
matrices).


T

-- 
He who laughs last thinks slowest.
Apr 17 2014
prev sibling next sibling parent reply "Christof Schardt" <csnews schardt.info> writes:
I looked to the doku-page and expected to see, how some basic
matrix-operations could be performed.
Like e.g. (basic syntax) this:

 Dim  a(10,12)
 a(3,4) = 7
 a(i,j) = a(j,i) * 12

Instead I found exotic assignments and calculations.
Did I expect the wrong thing?
Mar 17 2014
parent Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
18.03.2014 1:35, Christof Schardt :

 I looked to the doku-page and expected to see, how some basic
 matrix-operations could be performed.
 Like e.g. (basic syntax) this:

   Dim  a(10,12)
   a(3,4) = 7
   a(i,j) = a(j,i) * 12

 Instead I found exotic assignments and calculations.
 Did I expect the wrong thing?


Sorry for that. Do you mead there is a lack of simple indexing examples 
or lack of some matrix functionality? The latter case is if you meant 
"transpose and multiply by 12" with `a(i,j) = a(j,i) * 12`.

-- 
 . 
Denis V. Shelomovskij
Mar 20 2014
prev sibling next sibling parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
17.03.2014 21:39, Denis Shelomovskij пишет:

 Multidimensional arrays indexing and slicing syntax is finally added [1]
 (thanks to Kenji Hara). So it was a good cause to update my
 multidimensional arrays library implementation and add support for the
 new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html


An important note:
My proposal is just a sliceable multidimensional array implementation, a 
try to provide the same effort we have for simple arrays for 
multidimensional ones. It isn't aimed to be a dedicated matrix / linear 
algebra module.


-- 
Денис В. Шеломовский
Denis V. Shelomovskij
Mar 17 2014
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Mar 18, 2014 at 01:55:31AM +0400, Denis Shelomovskij wrote:

 17.03.2014 21:39, Denis Shelomovskij пишет:

Multidimensional arrays indexing and slicing syntax is finally added [1]
(thanks to Kenji Hara). So it was a good cause to update my
multidimensional arrays library implementation and add support for the
new syntax. So here we are: [2].

Also should we add it to the standard library?

[1] https://github.com/D-Programming-Language/dmd/pull/443
[2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html


 
 An important note:
 My proposal is just a sliceable multidimensional array implementation,
 a try to provide the same effort we have for simple arrays for
 multidimensional ones. It isn't aimed to be a dedicated matrix /
 linear algebra module.


[...]

It will be a good low-level module that will allow a linear algebra
module to be built on.

I am against the idea of conflating multidimensional arrays with
matrices; the operations and interpretations are not the same, and it
leads to poor designs (I've tried it before, and the results were not so
good). Better have a general multidimensional array with per-element
operations, and then build matrix algebra as a separate module by using
multidimensional array as underlying storage implementation.


T

-- 
Answer: Because it breaks the logical sequence of discussion. / Question: Why
is top posting bad?
Mar 17 2014
parent "Mason McGill" <mmcgill caltech.edu> writes:
 It will be a good low-level module that will allow a linear 
 algebra
 module to be built on.


Agreed.  In fields like graphics and computer vision, large 
multidimensional arrays aren't always matrices/tensors in the 
linear algebra sense.  And it's always possible to specialize.  
NumPy got this right.
Mar 17 2014
prev sibling next sibling parent reply "Stefan Frijters" <sfrijters gmail.com> writes:
On Monday, 17 March 2014 at 17:39:41 UTC, Denis Shelomovskij 
wrote:

 Multidimensional arrays indexing and slicing syntax is finally 
 added [1] (thanks to Kenji Hara). So it was a good cause to 
 update my multidimensional arrays library implementation and 
 add support for the new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] 
 http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html


First of all, thank you very much for making such nice additions 
to D available for general use. I finally got around to giving 
this a spin. I'm using it for a proof-of-context HPC simuation 
code written in D (currently mostly experimenting with D's 
features), and as such I'm interfacing with the C MPI library to 
communicate between processes. The basis of the simulation is a 
3D lattice, so I was eagerly awaiting a nice solution in D. So 
far I've run into two things while using your library. The first 
is that I need to provide void pointers to the data to the MPI 
functions, so I currently hacked your code to make the _data 
storage array publicly accessible and that seems to work. To give 
an idea, I currently have code like this (just a snippet):

arr = multidimArray!T(nxH, nyH, nzH);
// [...] fill the array with data
// Prepare a buffer to receive a slice from another process.
rbuffer = multidimArray!T(haloSize, nyH, nzH);
// Prepare a buffer to send a slice to another process.
sbuffer = arr[$-2*haloSize-1..$ - haloSize-1, 0..$, 0..$].dup;
// Here I now use the pointer of the storage arrays to send the 
buffer around.
MPI_Sendrecv(sbuffer._data.ptr, nyH * nzH, MPI_INT, M.nbx[1], 0, 
rbuffer._data.ptr, nyH * nzH, mpiType, M.nbx[0], 0, M.comm, 
&mpiStatus);
// Put the buffer in the correct spot in the main array.
arr[0..haloSize, 0..$, 0..$] = rbuffer;

Am I missing a nicer way to accomplish this? I like the 
compactness of the code (compared to what I'm currently used to 
with our F90 simulation code). Secondly, the property that 
returns the dimensions of the array is called 'dimentions' (with 
a t), this should be fixed.

Regards,

Stefan
Apr 16 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Stefan Frijters:


 First of all, thank you very much for making such nice 
 additions to D available for general use. I finally got around 
 to giving this a spin.


Recently I've shown a possible usage example of the 
multidimensional arrays indexing and slicing syntax:
http://forum.dlang.org/thread/cizugfrkaunlkzyjptnp forum.dlang.org

Bye,
bearophile
Apr 16 2014
prev sibling next sibling parent reply "Stefan Frijters" <sfrijters gmail.com> writes:
I've been using the multidimensional arrays for a while now, but 
recently I've run into a problem w.r.t. optimization:

import std.stdio;
import unstd.multidimarray;

void main() {
   MultidimArray!(double, 3) arr;
   arr = multidimArray!double([1,2,42]);
   writeln(arr.lengths);
}

If I compile with 'dmd multidimtest.d unstd/multidimarray.d', I 
get [1,2,42], as expected, but when I compile in release mode 
'dmd multidimtest.d unstd/multidimarray.d -release' I get 
[0,0,0]. Any ideas what might cause this?

Also, when I compile with -w (which I normally prefer) I get a 
list of errors of the type

unstd/multidimarray.d(101): Warning: calling 
unstd.multidimarray.MultidimArray!(double, 
3LU).MultidimArray.this without side effects discards return 
value of type inout(MultidimArray!(double, 3LU)), prepend a 
cast(void) if intentional

I've acted on these errors in the version I use for my HPC test 
code, and I have not noticed any deleterious effects; would you 
accept a pull request to fix these?

Kind regards,

Stefan Frijters
Jun 10 2014
parent "Stefan Frijters" <sfrijters gmail.com> writes:
On Tuesday, 10 June 2014 at 19:04:18 UTC, Stefan Frijters wrote:

 I've been using the multidimensional arrays for a while now, 
 but recently I've run into a problem w.r.t. optimization:

 import std.stdio;
 import unstd.multidimarray;

 void main() {
   MultidimArray!(double, 3) arr;
   arr = multidimArray!double([1,2,42]);
   writeln(arr.lengths);
 }

 If I compile with 'dmd multidimtest.d unstd/multidimarray.d', I 
 get [1,2,42], as expected, but when I compile in release mode 
 'dmd multidimtest.d unstd/multidimarray.d -release' I get 
 [0,0,0]. Any ideas what might cause this?


I've looked into this and it seems there is required code inside 
a contract 'in', which is omitted with -release.

I've gone ahead and opened a pull request to fix this: 
https://bitbucket.org/denis-sh/unstandard/pull-request/2/bugfixes-for-multidimarrayd

Stefan
Jun 11 2014
prev sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 17.03.2014 18:39, schrieb Denis Shelomovskij:

 Multidimensional arrays indexing and slicing syntax is finally added [1]
 (thanks to Kenji Hara). So it was a good cause to update my
 multidimensional arrays library implementation and add support for the
 new syntax. So here we are: [2].

 Also should we add it to the standard library?

 [1] https://github.com/D-Programming-Language/dmd/pull/443
 [2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimarray.html


Yes please
Jun 10 2014