www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Matrix-type-friendly syntax and more

reply bearophile <bearophileHUGS lycos.com> writes:
Kenji Hara has just copied in GIT a large patch by Don:
https://github.com/9rnsr/dmd/branches/opDollar

It allows user defined collections usable with this syntax:

x[$-2, y[$-6, $-9], $-2]

See:
http://d.puremagic.com/issues/show_bug.cgi?id=3474


But a syntax like this too is quite important for a user-defined matrix type to
be used in numerical code:

m[i..j, k..w]

Similar slicing and dicing of arrays is used all the time in NumPy (array
library for Python) code.

This is a comment by Don about it:

 (4) I have NOT implemented $ inside opSlice(), opSliceAssign().
 It could be done, but I believe those language features need work. They don't
 permit multi-dimensional slicing. I think they should be removed, and the
 functionality folded into opIndex.
Isn't it better to improve opSlice() instead of deprecating it? I don't remember if people appreciate the idea of a stride (like the third step argument of std.range.iota), this too is used in scientific array-oriented code: m[i..j:2, k..w:3] --------------- In Python code there is often the need for a good multi-dimensional array type, even in not-scientific code. In the Python standard library there is a array module, but it's almost a joke, it's 1D, and it's not used much: http://docs.python.org/library/array.html Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library). The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible array libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ ) and numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html), tells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want. I think that's a good tradeoff between the opposed needs of: - Keeping Phobos of reasonable size (to not increase too much the burden of its management, to not slow down too much its development); - Avoiding the risk of incompatible multi-dimensional array types. Most code out there is able to build on a common foundation. This avoids duplication (like the creation of Python numarray and Numeric), allows a better focusing of efforts and speeds up the development of a language-wide standard for such arrays; - Offer a nD array type to the casual D programmer, even one that doesn't want or can't install other libraries. Even some 30-lines long D programs need multi-dimensional arrays, but they often don't need a complex scientific library too (example of a problem: in the preconditions of my functions that take an array of arrays I always have to test the input is not jagged and it is a rectangular matrix. Such test is not necessary for a multi-dimensional array that is never jagged). Putting the bare bones multi-dimensional array type in Phobos allows people to use them with zero other installs. This multi-dimensional Phobos array type doesn't even need to contain code to invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change of shape, iteration... Everything else is in modules/packages external to Phobos. I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules. Bye, bearophile
Oct 09 2011
next sibling parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
09.10.2011 21:29, bearophile ïèøåò:
 Kenji Hara has just copied in GIT a large patch by Don:
 https://github.com/9rnsr/dmd/branches/opDollar

 It allows user defined collections usable with this syntax:

 x[$-2, y[$-6, $-9], $-2]

 See:
 http://d.puremagic.com/issues/show_bug.cgi?id=3474


 But a syntax like this too is quite important for a user-defined matrix type
to be used in numerical code:

 m[i..j, k..w]

 Similar slicing and dicing of arrays is used all the time in NumPy (array
library for Python) code.

 This is a comment by Don about it:

 (4) I have NOT implemented $ inside opSlice(), opSliceAssign().
 It could be done, but I believe those language features need work. They don't
 permit multi-dimensional slicing. I think they should be removed, and the
 functionality folded into opIndex.
Isn't it better to improve opSlice() instead of deprecating it? I don't remember if people appreciate the idea of a stride (like the third step argument of std.range.iota), this too is used in scientific array-oriented code: m[i..j:2, k..w:3] --------------- In Python code there is often the need for a good multi-dimensional array type, even in not-scientific code. In the Python standard library there is a array module, but it's almost a joke, it's 1D, and it's not used much: http://docs.python.org/library/array.html Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library). The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible array libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ ) and numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html), tells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want. I think that's a good tradeoff between the opposed needs of: - Keeping Phobos of reasonable size (to not increase too much the burden of its management, to not slow down too much its development); - Avoiding the risk of incompatible multi-dimensional array types. Most code out there is able to build on a common foundation. This avoids duplication (like the creation of Python numarray and Numeric), allows a better focusing of efforts and speeds up the development of a language-wide standard for such arrays; - Offer a nD array type to the casual D programmer, even one that doesn't want or can't install other libraries. Even some 30-lines long D programs need multi-dimensional arrays, but they often don't need a complex scientific library too (example of a problem: in the preconditions of my functions that take an array of arrays I always have to test the input is not jagged and it is a rectangular matrix. Such test is not necessary for a multi-dimensional array that is never jagged). Putting the bare bones multi-dimensional array type in Phobos allows people to use them with zero other installs. This multi-dimensional Phobos array type doesn't even need to contain code to invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change of shape, iteration... Everything else is in modules/packages external to Phobos. I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules. Bye, bearophile
Suddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of adding it into Phobos, but just had no time for review of my code. Draft version is in attachment. rarray.d - something like Fortran array main.d - example of using rarray module output.txt - main.d output
Oct 09 2011
next sibling parent reply kenji hara <k.hara.pg gmail.com> writes:
I got an idea for multidimentional indexing and slicing.
http://d.puremagic.com/issues/show_bug.cgi?id=3D6798

I think opSlice!n (n is dimension integer typed size_t) is rarely
used, therefore using that name for the enhancement is almost safe for
backward compatibility.

Kenji Hara

2011/10/10 Denis Shelomovskij <verylonglogin.reg gmail.com>:
 09.10.2011 21:29, bearophile =D0=C9=DB=C5=D4:
 Kenji Hara has just copied in GIT a large patch by Don:
 https://github.com/9rnsr/dmd/branches/opDollar

 It allows user defined collections usable with this syntax:

 x[$-2, y[$-6, $-9], $-2]

 See:
 http://d.puremagic.com/issues/show_bug.cgi?id=3D3474


 But a syntax like this too is quite important for a user-defined matrix
 type to be used in numerical code:

 m[i..j, k..w]

 Similar slicing and dicing of arrays is used all the time in NumPy (arra=
y
 library for Python) code.

 This is a comment by Don about it:

 (4) I have NOT implemented $ inside opSlice(), opSliceAssign().
 It could be done, but I believe those language features need work. They
 don't
 permit multi-dimensional slicing. I think they should be removed, and t=
he
 functionality folded into opIndex.
Isn't it better to improve opSlice() instead of deprecating it? I don't remember if people appreciate the idea of a stride (like the thi=
rd
 step argument of std.range.iota), this too is used in scientific
 array-oriented code:

 m[i..j:2, k..w:3]

 ---------------

 In Python code there is often the need for a good multi-dimensional arra=
y
 type, even in not-scientific code.

 In the Python standard library there is a array module, but it's almost =
a
 joke, it's 1D, and it's not used much:
 http://docs.python.org/library/array.html

 Python programmers use the multi-dimensional array of NumPy, it's widely
 used around the world, it's used by SciPy too (a scientific programming
 library).

 The experience of Python, with its sorely felt lack of a good
 multi-dimensional array type, the failure of its array module, and the
 success of NumPy, the problems caused by two precedent incompatible arra=
y
 libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ )=
and
 numarray
 (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html=
),
 tells me that it will be good to have a bare-bones, but efficient
 multi-dimensional array type. Plus external libraries (not present in
 Phobos) that use those arrays to implement all the things they want.

 I think that's a good tradeoff between the opposed needs of:
 - Keeping Phobos of reasonable size (to not increase too much the burden
 of its management, to not slow down too much its development);
 - Avoiding the risk of incompatible multi-dimensional array types. Most
 code out there is able to build on a common foundation. This avoids
 duplication (like the creation of Python numarray and Numeric), allows a
 better focusing of efforts and speeds up the development of a language-w=
ide
 standard for such arrays;
 - Offer a nD array type to the casual D programmer, even one that doesn'=
t
 want or can't install other libraries. Even some 30-lines long D program=
s
 need multi-dimensional arrays, but they often don't need a complex
 scientific library too (example of a problem: in the preconditions of my
 functions that take an array of arrays I always have to test the input i=
s
 not jagged and it is a rectangular matrix. Such test is not necessary fo=
r a
 multi-dimensional array that is never jagged). Putting the bare bones
 multi-dimensional array type in Phobos allows people to use them with ze=
ro
 other installs.

 This multi-dimensional Phobos array type doesn't even need to contain co=
de
 to invert a matrix or compute determinant, etc. It just needs basic
 operations like allocation, indexing, multi-dimensional slicing, change =
of
 shape, iteration... Everything else is in modules/packages external to
 Phobos.

 I am not suggesting to put a sparse multi-dimensional array type in
 Phobos. This need is much less common in casual short programs. This is
 better left to external modules.

 Bye,
 bearophile
Suddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of addi=
ng
 it into Phobos, but just had no time for review of my code. Draft version=
is
 in attachment.

 rarray.d - something like Fortran array
 main.d - example of using rarray module
 output.txt - main.d output
Oct 09 2011
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
kenji hara:

 I got an idea for multidimentional indexing and slicing.
 http://d.puremagic.com/issues/show_bug.cgi?id=6798
Thank you, you are good. So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2)) Bye, bearophile
Oct 09 2011
next sibling parent reply kenji hara <k.hara.pg gmail.com> writes:
2011/10/10 bearophile <bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]

 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Oct 09 2011
parent reply Don <nospam nospam.com> writes:
On 10.10.2011 04:41, kenji hara wrote:
 2011/10/10 bearophile<bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]

 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Personally, I think that since strided operations are so inefficient, they should remain ugly.
Oct 09 2011
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Don:

 Personally, I think that since strided operations are so inefficient, 
 they should remain ugly.
In Chapel (that's a high performance language full of nice ideas) the stride is present, see page 16 "Range operators": http://chapel.cray.com/tutorials/CUG2011/CUG11-2-Basics.pdf Bye, bearophile
Oct 10 2011
prev sibling next sibling parent reply kennytm <kennytm gmail.com> writes:
Don <nospam nospam.com> wrote:
 On 10.10.2011 04:41, kenji hara wrote:
 2011/10/10 bearophile<bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]
 
 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Personally, I think that since strided operations are so inefficient, they should remain ugly.
Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].
Oct 10 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 10 Oct 2011 08:46:05 -0400, kennytm <kennytm gmail.com> wrote:
 Don <nospam nospam.com> wrote:
 On 10.10.2011 04:41, kenji hara wrote:
 2011/10/10 bearophile<bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]

 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Personally, I think that since strided operations are so inefficient, they should remain ugly.
Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].
Regarding (1), from a computer vision/graphics perspective, being able to only operate on the red channel, for example, is very nice. And I use striding all the time in my CUDA code, but that's not as applicable in D, yet. Striding tends to be most useful for working with packed data of some kind.
Oct 10 2011
parent reply kennytm <kennytm gmail.com> writes:
"Robert Jacques" <sandford jhu.edu> wrote:
 On Mon, 10 Oct 2011 08:46:05 -0400, kennytm <kennytm gmail.com> wrote:
 Don <nospam nospam.com> wrote:
 On 10.10.2011 04:41, kenji hara wrote:
 2011/10/10 bearophile<bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]
 
 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Personally, I think that since strided operations are so inefficient, they should remain ugly.
Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].
Regarding (1), from a computer vision/graphics perspective, being able to only operate on the red channel, for example, is very nice. And I use striding all the time in my CUDA code, but that's not as applicable in D, yet. Striding tends to be most useful for working with packed data of some kind.
Why not use a 'Color' struct?
Oct 10 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 11 Oct 2011 01:57:06 -0400, kennytm <kennytm gmail.com> wrote:
 "Robert Jacques" <sandford jhu.edu> wrote:
 On Mon, 10 Oct 2011 08:46:05 -0400, kennytm <kennytm gmail.com> wrote:
 Don <nospam nospam.com> wrote:
 On 10.10.2011 04:41, kenji hara wrote:
 2011/10/10 bearophile<bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]

 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Personally, I think that since strided operations are so inefficient, they should remain ugly.
Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].
Regarding (1), from a computer vision/graphics perspective, being able to only operate on the red channel, for example, is very nice. And I use striding all the time in my CUDA code, but that's not as applicable in D, yet. Striding tends to be most useful for working with packed data of some kind.
Why not use a 'Color' struct?
Not all images have the same layout, so having a single Color struct doesn't cut it. (RGB, RGBA, BGR, BGRA, ARGB, ABGR) Although, being able to say: structArr[0..$].red; would be very handy indeed. But all that is / should be is a opDispatch wrapper around a strided operation: byteArr[2..$:3]
Oct 11 2011
parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
I don't think strides are necessary. You can always opApply your image
in a certain way. Define a private struct, which has an opApply, which
iterates over every third channel of each pixel.
lazy ranges are generally better, then arrays, because with them, you
can avoid copying anything altogether.

On Tue, Oct 11, 2011 at 6:36 PM, Robert Jacques <sandford jhu.edu> wrote:
 On Tue, 11 Oct 2011 01:57:06 -0400, kennytm <kennytm gmail.com> wrote:
 "Robert Jacques" <sandford jhu.edu> wrote:
 On Mon, 10 Oct 2011 08:46:05 -0400, kennytm <kennytm gmail.com> wrote:
 Don <nospam nospam.com> wrote:
 On 10.10.2011 04:41, kenji hara wrote:
 2011/10/10 bearophile<bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]

 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Personally, I think that since strided operations are so inefficient, they should remain ugly.
Slicing an nD static array is just as in/efficient as strides. That said, I'm still -1 on having a..b:c since (1) I see no practical need for strides and (2) even if we striding it doesn't need to be in the form a..b:c, e.g. arr[stride(0..$, 2)].
Regarding (1), from a computer vision/graphics perspective, being able to only operate on the red channel, for example, is very nice. And I use striding all the time in my CUDA code, but that's not as applicable in D, yet. Striding tends to be most useful for working with packed data of some kind.
Why not use a 'Color' struct?
Not all images have the same layout, so having a single Color struct doesn't cut it. (RGB, RGBA, BGR, BGRA, ARGB, ABGR) Although, being able to say: structArr[0..$].red; would be very handy indeed. But all that is / should be is a opDispatch wrapper around a strided operation: byteArr[2..$:3]
Oct 11 2011
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Gor Gyolchanyan:

 I don't think strides are necessary. You can always opApply your image
 in a certain way. Define a private struct, which has an opApply, which
 iterates over every third channel of each pixel.
See: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=146466
 lazy ranges are generally better, then arrays, because with them, you
 can avoid copying anything altogether.
Strides is just a syntax. The user defined type is later allowed to perform a lazy striding. Bye, bearophile
Oct 11 2011
prev sibling next sibling parent Norbert Nemec <Norbert Nemec-online.de> writes:
On 11.10.2011 17:48, Gor Gyolchanyan wrote:
 I don't think strides are necessary. You can always opApply your image
 in a certain way. Define a private struct, which has an opApply, which
 iterates over every third channel of each pixel.
 lazy ranges are generally better, then arrays, because with them, you
 can avoid copying anything altogether.
Actually, a strided slice to a multidimensional array can easily be implemented as a view to the original data without any copy action, just like any builtin array slice is a view to the data. Independent of that: of course strides are not "necessary", but they are an extremely convenient syntax for expressing numerical array operations.
Oct 11 2011
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 11 Oct 2011 11:48:54 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:
 I don't think strides are necessary. You can always opApply your image
 in a certain way. Define a private struct, which has an opApply, which
 iterates over every third channel of each pixel.
 lazy ranges are generally better, then arrays, because with them, you
 can avoid copying anything altogether.
First, slicing / striding doesn't imply copying. Second, striding allows you to mutate the original data, which many lazy ranges do not. Third, there is a heavy amount of over-head to opApply, as its delegate based. Forth, declaring a custom type is way too much programmer boilerplate, even using something like Tuple!(ubyte,ubyte,ubyte,ubyte) is overly verbose. Fifth, my example was about dynamically supporting multiple data formats using the same code, something not possible with a custom struct. Sixth, we are talking about multi-dimensional arrays, not D's built-in arrays; lazy ranges don't exactly apply. Although, on the subject of Multi-dimensional ranges/iteration, has anyone else taken a look at the FLAME project? (http://z.cs.utexas.edu/wiki/flame.wiki) Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.
Oct 11 2011
parent reply bearophile <bearophileHUGS lycos.com> writes:
Robert Jacques:

[... several things]
 Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.
OK, a nice striding syntax for user defined types will be a good thing to have in D2. I have suggested the x..y:y syntax. Is this usable? Is this going to cause problems? Bye, bearophile
Oct 11 2011
parent reply travert phare.normalesup.org (Christophe) writes:
bearophile , dans le message (digitalmars.D:146534), a écrit :
 Robert Jacques:
 
[... several things]
 Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.
OK, a nice striding syntax for user defined types will be a good thing to have in D2. I have suggested the x..y:y syntax. Is this usable? Is this going to cause problems? Bye, bearophile
: is already used for labels. If slice types are allowed, this can cause trouble: auto a = 0 .. 10; // a slice!(int, int) type. a : 2; // is it a label or a stride ? Of course, the example here is obvious, but maybe in real code this can cause trouble. stride(x..y, z) works just as good, and does not let people think it is a very efficient langage builtin. The langage just have to allow to define iota-like types with .. Then a[x..y] becomes an opIndex() like Kenji suggested, and slices can be used anywhere, like in : multidimArray.columns(1..5);
Nov 03 2011
parent Norbert Nemec <Norbert Nemec-online.de> writes:
Strided ranges could be a tertiary operator

similar to
	x ? a : b
one could also parse
	a .. b : c
as one entity.

About the efficiency issue, once again: for multi-dimensional arrays, 
strides have less overhead than for one-dimensional arrays: All but one 
dimension need run-time defined strides anyway.



On 03.11.2011 13:52, Christophe wrote:
 bearophile , dans le message (digitalmars.D:146534), a écrit :
 Robert Jacques:

 [... several things]
 Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.
OK, a nice striding syntax for user defined types will be a good thing to have in D2. I have suggested the x..y:y syntax. Is this usable? Is this going to cause problems? Bye, bearophile
: is already used for labels. If slice types are allowed, this can cause trouble: auto a = 0 .. 10; // a slice!(int, int) type. a : 2; // is it a label or a stride ? Of course, the example here is obvious, but maybe in real code this can cause trouble. stride(x..y, z) works just as good, and does not let people think it is a very efficient langage builtin. The langage just have to allow to define iota-like types with .. Then a[x..y] becomes an opIndex() like Kenji suggested, and slices can be used anywhere, like in : multidimArray.columns(1..5);
Nov 05 2011
prev sibling parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
On 10.10.2011 07:34, Don wrote:
 On 10.10.2011 04:41, kenji hara wrote:
 2011/10/10 bearophile<bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]

 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Personally, I think that since strided operations are so inefficient, they should remain ugly.
Inefficient?! For multidimensional arrays, all but one dimension have non-trivial, runtime-dependent strides anyway. Anyway, the expressiveness gained by strides is significant and the effort to allow it in the language is minimal.
Oct 10 2011
parent reply Don <nospam nospam.com> writes:
On 10.10.2011 20:03, Norbert Nemec wrote:
 On 10.10.2011 07:34, Don wrote:
 On 10.10.2011 04:41, kenji hara wrote:
 2011/10/10 bearophile<bearophileHUGS lycos.com>:
 So is this:
 y[$-6, 0..$:2]

 Translated like this?
 y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
I have no thought about it. I'm not sure that the additional stepping is really useful, but I think adding it into syntax is not impossible -- it might not conflict with associative array literal. Kenji Hara
Personally, I think that since strided operations are so inefficient, they should remain ugly.
Inefficient?! For multidimensional arrays, all but one dimension have non-trivial, runtime-dependent strides anyway.
The problem is with built-in arrays.
 Anyway, the expressiveness gained by strides is significant and the
 effort to allow it in the language is minimal.
That's true, IF you are happy to have it as a feature which can only be used in user-defined types. If you want it apply to built-in arrays as well, the effort is immense.
Oct 11 2011
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Don:

 That's true, IF you are happy to have it as a feature which can only be 
 used in user-defined types. If you want it apply to built-in arrays as 
 well, the effort is immense.
This is not wonderful, but for me it's acceptable. Having that syntax support only for user defined types is quite better than having to use a syntax like stride(...) suggested by someone else. ------------- For a simple but nice usage example of strides in Chapel language see pages 14-33 here: http://chapel.cray.com/tutorials/SC10/M10-8-Samples.pdf Strided grid description var cell_centers = [1..7 by 2, 1..5 by 2]; var vertical_edges = [0..8 by 2, 1..5 by 2]; var horizontal_edges = [1..7 by 2, 0..6 by 2]; var vertices = [0..8 by 2, 0..6 by 2]; (See the slides for more info.) Bye, bearophile
Oct 11 2011
prev sibling parent Norbert Nemec <Norbert Nemec-online.de> writes:
On 11.10.2011 09:05, Don wrote:
 On 10.10.2011 20:03, Norbert Nemec wrote:
 Inefficient?! For multidimensional arrays, all but one dimension have
 non-trivial, runtime-dependent strides anyway.
The problem is with built-in arrays.
Sure. For one-dimensional arrays, the overhead would be significant and the use would be limited to exotic cases.
 Anyway, the expressiveness gained by strides is significant and the
 effort to allow it in the language is minimal.
That's true, IF you are happy to have it as a feature which can only be used in user-defined types. If you want it apply to built-in arrays as well, the effort is immense.
Indeed, I am interested in this feature purely for defining multidimensional arrays for use in numerics.
Oct 11 2011
prev sibling parent Norbert Nemec <Norbert Nemec-online.de> writes:
On 10.10.2011 03:18, bearophile wrote:
 kenji hara:
 
 I got an idea for multidimentional indexing and slicing.
 http://d.puremagic.com/issues/show_bug.cgi?id=6798
Thank you, you are good. So is this: y[$-6, 0..$:2] Translated like this? y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
Nice! This seems like the way to go.
Oct 09 2011
prev sibling parent travert phare.normalesup.org (Christophe) writes:
kenji hara , dans le message (digitalmars.D:146395), a écrit :
 I got an idea for multidimentional indexing and slicing.
 http://d.puremagic.com/issues/show_bug.cgi?id=6798
 
 I think opSlice!n (n is dimension integer typed size_t) is rarely
 used, therefore using that name for the enhancement is almost safe for
 backward compatibility.
 
Another option, that I must have posted somewhere else in the past: T..T is a type similar to a Tuple with two values, and represents a range going from the first element to the second element excluded. Operations are defined for T..T just like they are for arrays: ref T front(size_t..size_t a) { assert(a[0] < a[1]; return a[0]; } void popFront(size_t..size_t a) { assert(a[0] < a[1]_; ++a[0]; } bool empty(size_t..size_t a) { return a[0] == a[1]; } etc... Any template that takes a range can take a slice, since front, popFront and empty are defined. map!"a*2"(1..10); // instead of map!"a*2"(iota(1, 10)); foreach (a, x..y) is a normal iteration over a range. a[1..2, 3] calls A.opIndex(int..int, int). etc...
Nov 03 2011
prev sibling next sibling parent reply kenji hara <k.hara.pg gmail.com> writes:
Posted pull request.
https://github.com/D-Programming-Language/dmd/pull/443

This patch does not break existing codes (at least dmd test suite and
phobos building).

I have not added stride syntax (e.g. a[0..$:2]), but I think that
adding it is not so difficult.

Kenji Hara

2011/10/10 kenji hara <k.hara.pg gmail.com>:
 I got an idea for multidimentional indexing and slicing.
 http://d.puremagic.com/issues/show_bug.cgi?id=3D6798

 I think opSlice!n (n is dimension integer typed size_t) is rarely
 used, therefore using that name for the enhancement is almost safe for
 backward compatibility.

 Kenji Hara

 2011/10/10 Denis Shelomovskij <verylonglogin.reg gmail.com>:
 09.10.2011 21:29, bearophile =D0=C9=DB=C5=D4:
 Kenji Hara has just copied in GIT a large patch by Don:
 https://github.com/9rnsr/dmd/branches/opDollar

 It allows user defined collections usable with this syntax:

 x[$-2, y[$-6, $-9], $-2]

 See:
 http://d.puremagic.com/issues/show_bug.cgi?id=3D3474


 But a syntax like this too is quite important for a user-defined matrix
 type to be used in numerical code:

 m[i..j, k..w]

 Similar slicing and dicing of arrays is used all the time in NumPy (arr=
ay
 library for Python) code.

 This is a comment by Don about it:

 (4) I have NOT implemented $ inside opSlice(), opSliceAssign().
 It could be done, but I believe those language features need work. The=
y
 don't
 permit multi-dimensional slicing. I think they should be removed, and =
the
 functionality folded into opIndex.
Isn't it better to improve opSlice() instead of deprecating it? I don't remember if people appreciate the idea of a stride (like the th=
ird
 step argument of std.range.iota), this too is used in scientific
 array-oriented code:

 m[i..j:2, k..w:3]

 ---------------

 In Python code there is often the need for a good multi-dimensional arr=
ay
 type, even in not-scientific code.

 In the Python standard library there is a array module, but it's almost=
a
 joke, it's 1D, and it's not used much:
 http://docs.python.org/library/array.html

 Python programmers use the multi-dimensional array of NumPy, it's widel=
y
 used around the world, it's used by SciPy too (a scientific programming
 library).

 The experience of Python, with its sorely felt lack of a good
 multi-dimensional array type, the failure of its array module, and the
 success of NumPy, the problems caused by two precedent incompatible arr=
ay
 libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ =
) and
 numarray
 (http://www.stsci.edu/resources/software_hardware/numarray/numarray.htm=
l),
 tells me that it will be good to have a bare-bones, but efficient
 multi-dimensional array type. Plus external libraries (not present in
 Phobos) that use those arrays to implement all the things they want.

 I think that's a good tradeoff between the opposed needs of:
 - Keeping Phobos of reasonable size (to not increase too much the burde=
n
 of its management, to not slow down too much its development);
 - Avoiding the risk of incompatible multi-dimensional array types. Most
 code out there is able to build on a common foundation. This avoids
 duplication (like the creation of Python numarray and Numeric), allows =
a
 better focusing of efforts and speeds up the development of a language-=
wide
 standard for such arrays;
 - Offer a nD array type to the casual D programmer, even one that doesn=
't
 want or can't install other libraries. Even some 30-lines long D progra=
ms
 need multi-dimensional arrays, but they often don't need a complex
 scientific library too (example of a problem: in the preconditions of m=
y
 functions that take an array of arrays I always have to test the input =
is
 not jagged and it is a rectangular matrix. Such test is not necessary f=
or a
 multi-dimensional array that is never jagged). Putting the bare bones
 multi-dimensional array type in Phobos allows people to use them with z=
ero
 other installs.

 This multi-dimensional Phobos array type doesn't even need to contain c=
ode
 to invert a matrix or compute determinant, etc. It just needs basic
 operations like allocation, indexing, multi-dimensional slicing, change=
of
 shape, iteration... Everything else is in modules/packages external to
 Phobos.

 I am not suggesting to put a sparse multi-dimensional array type in
 Phobos. This need is much less common in casual short programs. This is
 better left to external modules.

 Bye,
 bearophile
Suddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of add=
ing
 it into Phobos, but just had no time for review of my code. Draft versio=
n is
 in attachment.

 rarray.d - something like Fortran array
 main.d - example of using rarray module
 output.txt - main.d output
Oct 10 2011
parent bearophile <bearophileHUGS lycos.com> writes:
Kenji Hara:

 Posted pull request.
 https://github.com/D-Programming-Language/dmd/pull/443
Things are getting interesting. Thank you. Such syntax is the building material for a good enough user-defined dense nD array. I presume the memory representation of such arrays (possibly in Phobos) will be tiled, to help CPU caches.
 This patch does not break existing codes (at least dmd test suite and
 phobos building).
Given how important multi-dimensional slicing it, a bit of breakage is acceptable if it allows to make the situation simpler & cleaner. People that will implement a nD array years from now will thank you :-)
 I have not added stride syntax (e.g. a[0..$:2]), but I think that
 adding it is not so difficult.
OK. Bye, bearophile
Oct 10 2011
prev sibling parent Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
 Suddenly, I had a talk with an Fortran guy at previous weekend and have
 wrote something like Fortran arrays in D last week. I had an idea of
 adding it into Phobos, but just had no time for review of my code. Draft
 version is in attachment.

 rarray.d - something like Fortran array
 main.d - example of using rarray module
 output.txt - main.d output
My usable draft example of multidimensional rectangular arrays (source looks working stable and usable but still isn't ready): http://deoma-cmd.ru/d/docs/src/my/rarray.html Posted now just for illustrating existed syntax abilities. IMHO, it is not too bad: adding `R[]` isn't a big overhead.
Oct 13 2011
prev sibling parent bearophile <bearophileHUGS lycos.com> writes:
Robert Jacques:

[... several things]
 Seventh, Matlab, NumPy, Octave, BLAS, etc. all support striding.
OK, a nice striding syntax for user defined types will be a good thing to have in D2. I have suggested the x..y:y syntax. Is this usable? Is this going to cause problems? Bye, bearophile
Oct 13 2011