www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Argumnentation against external function operator overloading is

reply HaraldZealot <harald_zealot tut.by> writes:
In current D, overloading operator like "+" with external 
function is prohibited. There is the rationale [1] (see second 
paragraph).

BUT this rationale is totally UNCONVINCING. I can say that resume 
of rationale is: "it is impossible because it brakes some C++ 
patterns and behaviour". Further I will try to demonstrate such 
resume.

Let start with an example where external operator overloading can 
be useful. Imagine,  we want to create some lazy matrix algebra, 
where all operation return new object of corresponding to 
operation subtype. E.g. `transpose` returns `TransposedMatrix`, 
where `opIndex` just call `opIndex` of internally stored matrix 
with switched order of indices, `plus` returns `MatrixSum`, where 
`opIndex` just calculated the sum of two corresponding elements 
of stored matrices and so on. Of course we want that any 
resulting subtypes can interact with each other, so we require 
common "denominator". There are two ways compile-time duck-typing 
  with template bloating, or common interface (yeah, with garbage 
collection in the simplest implementation), but the last allows 
us to have some run-time parameters. So regard common interface 
approach: our interface should be minimal as possible, for our 
purposes having `rows`, `columns` and `opIndex` seems to be 
sufficient.
OK, we have:
```d
Matrix plus(Matrix A, Matrix B)
{
     final class MatrixSum : Matrix
     {...}
     ...
     return new MatrixSum(A, B);
}
...
C=plus(A, B);
```
(Or even `C=A.plus(B)` because of UFCS). And this works perfect.
If we want some sugar like `C=A+B` we are in trouble. We can 
create function `Matrix opBinary(string op)(Matrix A, Matrix B) 
if(op == "+")`, it even can be called with full name, but not 
with "+" operator, because now compiler just doesn't look for 
external function as overload for operator. If we want have this 
as member we should ad opBinary to our interface, what leads to 
some new trouble: now each type should have opBinary, but how we 
should implement this for example for `TransposedMatrix` or even 
more interesting (because of kind of type recursion) how we 
should implement it for `MatrixSum`. OK, me and Mathias Lang have 
found workaround for my particular case we should implement (and 
probably make `final`) opBinary for `Matrix` interface, which in 
its turn calls function `plus` we already described. But its 
create lines of code for nothing, because operators are 
considered as too exceptional.

Let us return to rationale.
First point is "Operator overloading can only be done with an 
argument as an object". Why??? it seems to come from "C++ mind", 
before UFCS was implemented in language. If we have `1.to!string` 
and `A.plus(B)` working, what wrong with `A+B` or `2*B` (where A 
and B matrices). Moreover for case like `2*B` (multiplication by 
scalar) we should have a bit ugly `opBinaryRight(string 
op)(double scalar)` as member function. It isn't convinced.

Second point is "Operator overloads USUALLY need access to 
private members of a class". It has nothing common with operator, 
it is only demonstrate  our C++ behave to work with private 
members in operator. But having access to private member isn't 
necessary for operator to work. In my example `plus` function 
uses only public API of my classes, it can be done such way in 
many, many cases. And also often this external function would be 
placed in the same module, so it is already have "friend" in such 
case. If external function in external module want to have access 
to private API, it is only a sign of bad design, and should be a 
problem of particular programmer how to rework the design, not 
case for compiler to step in. It isn't convinced.

Third point is totally unnecessary because we don't need access 
to private members.

So, no one of point convinces me.

What can really can convince me that rationale like: "if we do so 
total parsing algorithm will be corrupted, and we can't use D 
any-more", or at least "with that feature compiler becomes 100 
times slower".


So if someone has real rationale not to have operator overloading 
as external function I'm curios to arguments.


[1] http://dlang.org/rationale.html
Sep 21 2016
next sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Wednesday, 21 September 2016 at 17:57:17 UTC, HaraldZealot 
wrote:

 In current D, overloading operator like "+" with external 
 function is prohibited. There is the rationale [1] (see second 
 paragraph).

 [...]


I am completely agree. We should support external operator 
overloading for ndslice extension. - Ilya
Sep 21 2016
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 21.09.2016 19:57, HaraldZealot wrote:

 So if someone has real rationale not to have operator overloading as
 external function I'm curios to arguments.


 [1] http://dlang.org/rationale.html


There is no technical reason that would make the implementation of this 
feature difficult, if that is your question.

Basically, the rationale is: external operators cannot be used in 
generic code that does not import the module defining the operators. C++ 
works around this using ADL. Walter (justifiably) does not like ADL, 
hence the limitation.

(I don't agree with that line of reasoning: obviously this is not only 
an issue for operators, but for any UFCS function; operators are mere 
syntactic sugar.)
Sep 21 2016
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 21.09.2016 21:01, Timon Gehr wrote:

 On 21.09.2016 19:57, HaraldZealot wrote:

 So if someone has real rationale not to have operator overloading as
 external function I'm curios to arguments.


 [1] http://dlang.org/rationale.html


 There is no technical reason that would make the implementation of this
 feature difficult, if that is your question.

 Basically, the rationale is: external operators cannot be used in
 generic code that does not import the module defining the operators. C++
 works around this using ADL. Walter (justifiably) does not like ADL,
 hence the limitation.

 (I don't agree with that line of reasoning: obviously this is not only
 an issue for operators, but for any UFCS function; operators are mere
 syntactic sugar.)


BTW, another argument in favour of free function operators is opOpAssign 
for classes.
Sep 21 2016
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 21 September 2016 at 19:01:40 UTC, Timon Gehr wrote:

 Basically, the rationale is: external operators cannot be used 
 in generic code that does not import the module defining the 
 operators.


So why not have the struct/class explicitly import external 
operators? You can do this currently with a separate module 
defining them and then the struct/class imports the whole thing. 
Kind of annoying, but would get the job done, no?

Alternately, you could invent some new syntax. Maybe something 
like

import this : Matrix plus(Matrix A, Matrix B);
Sep 21 2016
prev sibling next sibling parent reply HaraldZealot <harald_zealot tut.by> writes:
On Wednesday, 21 September 2016 at 19:01:40 UTC, Timon Gehr wrote:

 Basically, the rationale is: external operators cannot be used 
 in generic code that does not import the module defining the 
 operators.


Could you give some elaborate example, for now I can't imagine 
what your mean.
Sep 21 2016
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 21.09.2016 22:53, HaraldZealot wrote:

 On Wednesday, 21 September 2016 at 19:01:40 UTC, Timon Gehr wrote:

 Basically, the rationale is: external operators cannot be used in
 generic code that does not import the module defining the operators.


 Could you give some elaborate example, for now I can't imagine what your
 mean.


module a;

struct Foo{}

Foo opBinary(string op:"+")(Foo a, Foo b){ return Foo(); }

---

module b;

T add(T)(T a,T b){
     return a + b;
}

---

module c;
import a,b;

void main(){
     Foo x=add(Foo(),Foo()); // error
}
Sep 21 2016
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Sep 21, 2016 at 08:53:06PM +0000, HaraldZealot via Digitalmars-d wrote:

 On Wednesday, 21 September 2016 at 19:01:40 UTC, Timon Gehr wrote:

 
 Basically, the rationale is: external operators cannot be used in
 generic code that does not import the module defining the operators.


 
 Could you give some elaborate example, for now I can't imagine what
 your mean.


Here's a simple example:

	// usertype.d
	module usertype;
	struct UserType { ... }
	auto opBinary(string op : "+")(UserType u1, UserType u2) { ... }

	// generic_code.d
	module generic_code;
	auto algorithm(T,U)(T t, U u) {
		return t + u;
	}

	// main.d
	module main;
	import usertype;
	import generic_code;
	void main() {
		UserType u1, u2;
		auto r = u1 + u2;		// OK
		auto s = algorithm(u1, u2);	// NO GOOD
	}

The problem here is that generic_code.d doesn't (and shouldn't!) import
usertype.d, so usertype.opBinary is not visible in generic_code.d. So
when algorithm() tries to look up the '+' operator in `t + u`, it can't
find the declaration and fails.  There is no way to find the correct
opBinary() because it's not part of UserType, so algorithm() has no way
to access that symbol.

Using the operator in module main is OK, because main (rightfully)
imports usertype.d, so the operator is visible. But any generic code
that main imports will have a problem because they can't (and
shouldn't!) know ahead of time which modules contain the declaration
they need.

In C++ this problem is solved using ADL, but ADL brings with it other
problems, the root of which is that it breaks module encapsulation.

There's actually some instances of this problem (w.r.t. UFCS) in Phobos,
that currently requires ugly workarounds like importing modules that the
generic code really shouldn't be depending on.


T

-- 
Computerese Irregular Verb Conjugation: I have preferences.  You have biases. 
He/She has prejudices. -- Gene Wirchenko
Sep 21 2016
next sibling parent reply HaraldZealot <harald_zealot tut.by> writes:
On Wednesday, 21 September 2016 at 21:14:15 UTC, H. S. Teoh wrote:


Thank you both, I see now.

So it seems to be essential point. But because we already have 
the same problem with UFCS, I don't see why we should prohibit 
external overloading of operator, it is just inequality (in 
political sense) for operators :)

In any case we at least should change rationale, because current 
three points are have nothing with real problem and shouldn't be 
obstacle.

But probably the best solution is implementing external operator 
overloading, just to omit special case. And problem with generic 
code solve independently for all UFCS functions including 
operators.



 Sep 21 2016



 next sibling parent reply HaraldZealot <harald_zealot tut.by>  writes:


On Thursday, 22 September 2016 at 05:38:53 UTC, HaraldZealot 
wrote:

 And problem  with generic code solve independently for all UFCS 
 functions including operators.


Unfortunately I don't know compiler and all related stuff 
internally (time to change this ;) ), but it seems to me there is 
a way to solve problem with UFCS and generic code without 
breakage of module encapsulation. If understand correctly when we 
instantiate template with struct/class we pass to template some 
kind of "static signature of type" (if I may it call so). With 
the static signature of type I mean all member function of class 
or struct. If we instead of this before instantiate any type 
(including POD) create "dynamic signature of type" which include 
all visible at the moment of call function and templates for this 
type.



 Sep 21 2016



 next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Thu, Sep 22, 2016 at 06:09:39AM +0000, HaraldZealot via Digitalmars-d wrote:

 On Thursday, 22 September 2016 at 05:38:53 UTC, HaraldZealot wrote:

 And problem  with generic code solve independently for all UFCS
 functions including operators.


 
 Unfortunately I don't know compiler and all related stuff internally
 (time to change this ;) ), but it seems to me there is a way to solve
 problem with UFCS and generic code without breakage of module
 encapsulation. If understand correctly when we instantiate template
 with struct/class we pass to template some kind of "static signature
 of type" (if I may it call so).  With the static signature of type I
 mean all member function of class or struct. If we instead of this
 before instantiate any type (including POD) create "dynamic signature
 of type" which include all visible at the moment of call function and
 templates for this type.


It's not so simple.  The UFCS operator overload could be declared in a
different module from the type itself.  Then there is no guarantee that
it will be found.  Multiple calls to the same template function with the
same argument types may result in different semantics, depending on what
was imported.

Or there could be more than one module that declares the operator,
possibly with conflicting semantics.  Normally this isn't a problem (D's
module system will trigger an overload conflict and require explicit FQN
to unambiguously select the right overload), but FQN's are not an option
when the call is made from generic code.


T

-- 
Ignorance is bliss... until you suffer the consequences!



 Sep 22 2016



  parent reply HaraldZealot <harald_zealot tut.by>  writes:


On Thursday, 22 September 2016 at 07:14:52 UTC, H. S. Teoh wrote:

 It's not so simple.  The UFCS operator overload could be 
 declared in a different module from the type itself.  Then 
 there is no guarantee that it will be found.  Multiple calls to 
 the same template function with the same argument types may 
 result in different semantics, depending on what was imported.
 ...
 T


I mean (in terms of your example) that in `main` before 
instantiate an `algorithm` we parse all symbols visible at this 
point from `main`, select all possible symbols which can be 
called with `UserType` (including templates, meh o_O), make the 
set union for the such table for the rest templates params, and 
give this symbol table the `algorithm`.

But yes this creates problem that in different module we can have 
different instantiation :(

Probably with other radical approach (see bellow *) for generic 
we can solve this, but this just destroy true templates (from 
which we benefits now) and provide something like Java-like 
solution :(

* radical approach for generic: each generic function creates a 
pseudo parameter for function pointers on the base of signature 
of function used in the body of generic. The caller of generic 
just fill this pseudo parameters with real function visible for 
caller at call-point.



 Sep 22 2016



  parent reply HaraldZealot <harald_zealot tut.by>  writes:


On Thursday, 22 September 2016 at 08:53:26 UTC, HaraldZealot 
wrote:


OK, it seems to me it's time to investigate a community opinion.


So let's vote for the following sentence:

"It would be good to have an operator overloading even without 
support in generic function"



 Sep 22 2016



  parent  HaraldZealot <harald_zealot tut.by>  writes:


On Thursday, 22 September 2016 at 08:58:54 UTC, HaraldZealot 
wrote:

 So let's vote for the following sentence:

 "It would be good to have an operator overloading even without 
 support in generic function"


Yes



 Sep 22 2016



prev sibling  parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Thursday, September 22, 2016 00:14:52 H. S. Teoh via Digitalmars-d wrote:

 Normally this isn't a problem (D's
 module system will trigger an overload conflict and require explicit FQN
 to unambiguously select the right overload), but FQN's are not an option
 when the call is made from generic code.


And in the case of operator overloads, FQN makes no sense at all, since it
wouldn't be using the operator anymore.

- Jonathan M Davis



 Sep 22 2016



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 22.09.2016 10:44, Jonathan M Davis via Digitalmars-d wrote:

 On Thursday, September 22, 2016 00:14:52 H. S. Teoh via Digitalmars-d wrote:

 Normally this isn't a problem (D's
 module system will trigger an overload conflict and require explicit FQN
 to unambiguously select the right overload), but FQN's are not an option
 when the call is made from generic code.


 And in the case of operator overloads, FQN makes no sense at all, since it
 wouldn't be using the operator anymore.

 - Jonathan M Davis


FQN disables UFCS. Nothing specific to operators here.

There is no reason why there should be any difference between a + b and 
a.opBinary!"+"(b). In fact, 2.opBinary!"+"(3) should work too.



 Sep 23 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Friday, 23 September 2016 at 08:50:56 UTC, Timon Gehr wrote:

 FQN disables UFCS. Nothing specific to operators here.

 There is no reason why there should be any difference between a 
 + b and a.opBinary!"+"(b). In fact, 2.opBinary!"+"(3) should 
 work too.


Currently this is tricky to implement in the compiler.
And it widens the scope for name-conflicts immensely!

I do not see a case where UFCS overloaded operators are worth the 
trouble they introduce.



 Sep 23 2016



  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 23.09.2016 12:44, Stefan Koch wrote:

 On Friday, 23 September 2016 at 08:50:56 UTC, Timon Gehr wrote:

 FQN disables UFCS. Nothing specific to operators here.

 There is no reason why there should be any difference between a + b
 and a.opBinary!"+"(b). In fact, 2.opBinary!"+"(3) should work too.


 Currently this is tricky to implement in the compiler.


This can easily be implemented in the parser and in object.d without 
even changing semantic. This is not the best implementation strategy, 
but it demonstrates that implementation can be simple. I'm curious to 
know what strategy you have in mind, and why it would be tricky.

BTW: One case for why built-in types should have member call syntax for 
operators are all the places in e.g. Phobos where code special-cases 
built-in types in order to be able to use opCmp directly for three-way 
comparison on user-defined types.


 And it widens the scope for name-conflicts immensely!
 ...


Operator overloading makes sense for a small minority of types and D has 
plenty of mechanisms to deal with name conflicts: overload sets, 
template constraints, private aliases, FQNs.


 I do not see a case where UFCS overloaded operators are worth the
 trouble they introduce.


IMHO it's a trivial surface language feature allowing convenient syntax 
for a restricted set of user types. The current limitations are slightly 
confusing (because a.opBinary!"+"(b) is somehow not the same as 
a.opBinary!"+"(b)), and also annoying when you run into them. This is 
not the first time this is discussed.



 Sep 23 2016



prev sibling  parent  Joseph Rushton Wakeling <joseph.wakeling webdrake.net>  writes:


On Thursday, 22 September 2016 at 05:38:53 UTC, HaraldZealot 
wrote:

 So it seems to be essential point. But because we already have 
 the same problem with UFCS, I don't see why we should prohibit 
 external overloading of operator, it is just inequality (in 
 political sense) for operators :)


I'm not sure that it's fundamentally different from issues that 
can affect other functions.  I seem to remember running into some 
issues along these lines when I tried defining math functions 
(e.g. `cos`) to work with an `Imaginary` type I was trying to add 
to `std.complex`.

If you're writing generic math code and you want to take the 
cosine of a number whose type is not explicitly known, how do you 
know which `cos` to use?



 Sep 22 2016



prev sibling  parent  krzaq <dlangmailinglist krzaq.cc>  writes:


On Wednesday, 21 September 2016 at 21:14:15 UTC, H. S. Teoh wrote:

 The problem here is that generic_code.d doesn't (and 
 shouldn't!) import
 usertype.d, so usertype.opBinary is not visible in 
 generic_code.d. So
 when algorithm() tries to look up the '+' operator in `t + u`, 
 it can't
 find the declaration and fails.  There is no way to find the 
 correct
 opBinary() because it's not part of UserType, so algorithm() 
 has no way
 to access that symbol.

 Using the operator in module main is OK, because main 
 (rightfully) imports usertype.d, so the operator is visible. 
 But any generic code that main imports will have a problem 
 because they can't (and shouldn't!) know ahead of time which 
 modules contain the declaration they need.

 [...]

 T


Why wouldn't templates just pull the "context" into them? I'm 
sorry if this is a naïve question.



 Sep 24 2016



prev sibling  parent reply pineapple <meapineapple gmail.com>  writes:


On Wednesday, 21 September 2016 at 19:01:40 UTC, Timon Gehr wrote:

 There is no technical reason that would make the implementation 
 of this feature difficult, if that is your question.

 Basically, the rationale is: external operators cannot be used 
 in generic code that does not import the module defining the 
 operators. C++ works around this using ADL. Walter 
 (justifiably) does not like ADL, hence the limitation.

 (I don't agree with that line of reasoning: obviously this is 
 not only an issue for operators, but for any UFCS function; 
 operators are mere syntactic sugar.)



The greatest offender I've found is how in phobos, arrays do not 
behave as ranges without importing the module defining their 
range operations.

Which I think is obnoxious, and maybe there's a solution to make 
both cases less so, but I definitely don't think it's an argument 
for exclusion.



 Sep 22 2016



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/22/16 6:38 AM, pineapple wrote:

 The greatest offender I've found is how in phobos, arrays do not behave
 as ranges without importing the module defining their range operations.


Would make sense to move those few primitives to object.d. I've been 
thinking of that a long time ago but back then there was a vague stance 
that object.d shouldn't contain templates. Since then that has changed. 
-- Andrei



 Sep 22 2016



 next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Thursday, September 22, 2016 08:51:59 Andrei Alexandrescu via Digitalmars-d 
wrote:

 On 9/22/16 6:38 AM, pineapple wrote:

 The greatest offender I've found is how in phobos, arrays do not behave
 as ranges without importing the module defining their range operations.


 Would make sense to move those few primitives to object.d. I've been
 thinking of that a long time ago but back then there was a vague stance
 that object.d shouldn't contain templates. Since then that has changed.
 -- Andrei


The main problem with moving them there is auto-decoding. front and popFront
for strings require std.utf in order to do their thing. So, if we move them
to druntime, then that code would need to be duplicated (though similar code
already has to exist in druntime for foreach loops), and we'd have a problem
with invalid unicode in that it couldn't through std.utf.UTFException like
it would now (though ideally, we'd stop throwing on invalid unicode and use
the replacement character).

That being said, I agree that the range functions for arrays should go in
object.d. It's just that the way we handle narrow strings as ranges makes it
problematic.

- Jonathan M Davis



 Sep 22 2016



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/22/16 10:20 AM, Jonathan M Davis via Digitalmars-d wrote:

 The main problem with moving them there is auto-decoding. front and popFront
 for strings require std.utf in order to do their thing.


Yah, that's a little hurdle. -- Andrei



 Sep 22 2016



  parent  Martin Nowak <code dawg.eu>  writes:


On Thursday, 22 September 2016 at 14:33:36 UTC, Andrei 
Alexandrescu wrote:

 On 9/22/16 10:20 AM, Jonathan M Davis via Digitalmars-d wrote:

 The main problem with moving them there is auto-decoding. 
 front and popFront
 for strings require std.utf in order to do their thing.




Druntime has it's own utf decoding code, necessary for foreach 
auto transcoding.
Moving that to a template and syncing it with the better 
optimized Phobos implementation would be a good thing.


 Yah, that's a little hurdle. -- Andrei


At least empty should go to object, could help to solve part of 
the wrong if (arr) usage.



 Sep 24 2016



prev sibling next sibling parent  "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com>  writes:


On Thu, Sep 22, 2016 at 07:20:49AM -0700, Jonathan M Davis via Digitalmars-d
wrote:

 On Thursday, September 22, 2016 08:51:59 Andrei Alexandrescu via Digitalmars-d 
 wrote:

 On 9/22/16 6:38 AM, pineapple wrote:

 The greatest offender I've found is how in phobos, arrays do not
 behave as ranges without importing the module defining their range
 operations.


 Would make sense to move those few primitives to object.d. I've been
 thinking of that a long time ago but back then there was a vague
 stance that object.d shouldn't contain templates. Since then that
 has changed.  -- Andrei


 
 The main problem with moving them there is auto-decoding.


[...]

Yet another nail in the coffin of autodecoding.  But I digress. ;-)


--T



 Sep 22 2016



prev sibling  parent reply pineapple <meapineapple gmail.com>  writes:


On Thursday, 22 September 2016 at 12:51:59 UTC, Andrei 
Alexandrescu wrote:

 On 9/22/16 6:38 AM, pineapple wrote:

 The greatest offender I've found is how in phobos, arrays do 
 not behave
 as ranges without importing the module defining their range 
 operations.


 Would make sense to move those few primitives to object.d. I've 
 been thinking of that a long time ago but back then there was a 
 vague stance that object.d shouldn't contain templates. Since 
 then that has changed. -- Andrei


I strongly disagree - I have been working a library that does not 
treat arrays as ranges, but as a type that a range can be created 
from. This design difference has proven rather elegant and solves 
a lot of problems I've run into using phobos. I think the 
(unlikely but ideal) solution is for phobos to adopt a similar 
approach, and not to warp the core language to accommodate 
phobos' strange design decisions.

Also, auto-decoding is a plague and it need not spread any 
farther than it already has done.

https://github.com/pineapplemachine/mach.d/blob/master/readme.md#arrays-arent-ranges



 Sep 24 2016



  parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sunday, September 25, 2016 00:36:58 pineapple via Digitalmars-d wrote:

 On Thursday, 22 September 2016 at 12:51:59 UTC, Andrei

 Alexandrescu wrote:

 On 9/22/16 6:38 AM, pineapple wrote:

 The greatest offender I've found is how in phobos, arrays do
 not behave
 as ranges without importing the module defining their range
 operations.


 Would make sense to move those few primitives to object.d. I've
 been thinking of that a long time ago but back then there was a
 vague stance that object.d shouldn't contain templates. Since
 then that has changed. -- Andrei


 I strongly disagree - I have been working a library that does not
 treat arrays as ranges, but as a type that a range can be created
 from. This design difference has proven rather elegant and solves
 a lot of problems I've run into using phobos. I think the
 (unlikely but ideal) solution is for phobos to adopt a similar
 approach, and not to warp the core language to accommodate
 phobos' strange design decisions.

 Also, auto-decoding is a plague and it need not spread any
 farther than it already has done.

 https://github.com/pineapplemachine/mach.d/blob/master/readme.md#arrays-aren
 t-ranges


Considering that a random access range is essentially an abstraction for a
dynamic array and that ranges were designed with that in mind, I don't know
how you can argue that dynamic arrays shouldn't be treated as ranges.

The auto-decoding is certainly an issue, but that only affects ranges of
char and wchar. The vast majority of dynamic arrays are perfectly normal
ranges.

- Jonathan M Davis



 Sep 24 2016



  parent reply pineapple <meapineapple gmail.com>  writes:


On Sunday, 25 September 2016 at 04:06:41 UTC, Jonathan M Davis 
wrote:

 Considering that a random access range is essentially an 
 abstraction for a dynamic array and that ranges were designed 
 with that in mind, I don't know how you can argue that dynamic 
 arrays shouldn't be treated as ranges.

 The auto-decoding is certainly an issue, but that only affects 
 ranges of char and wchar. The vast majority of dynamic arrays 
 are perfectly normal ranges.

 - Jonathan M Davis


Phobos' range facilities vomit when you try to deal with static 
arrays, and they have to be coerced into dynamic arrays before 
they can be dealt with. This is silly.

It gets under my skin that length and opIndex and opSlice produce 
different results in phobos' ranges depending on one's current 
position in the range. This doesn't make sense to me, and the 
only reason I can conceive of it having become how ranges work 
throughout phobos is because that's how dynamic arrays work if 
you force them to act as though they were ranges.

In every single other language I've used, the concept of an 
Iterable and an Iterator are distinct and very separate. An 
Iterator is something that can be iterated over; an Iterable is 
something which can produce an Iterator for iterating over its 
contents. In D, arrays are Iterables, and phobos endeavors to 
force them to be Iterators as well. It defies years of basic 
design wisdom regarding how to differentiate a collection and the 
means by which one enumerates the items in that collection.

Arrays are Iterables which should be able to produce an Iterator, 
in D's case a range. They should not themselves be Iterators.

"Poisoned m&ms are certainly an issue, but that only affects 
people who unwittingly eat them. You see, the vast majority of 
m&ms are perfectly innocuous."



 Sep 25 2016



 next sibling parent  Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sunday, September 25, 2016 10:58:24 pineapple via Digitalmars-d wrote:

 On Sunday, 25 September 2016 at 04:06:41 UTC, Jonathan M Davis

 wrote:

 Considering that a random access range is essentially an
 abstraction for a dynamic array and that ranges were designed
 with that in mind, I don't know how you can argue that dynamic
 arrays shouldn't be treated as ranges.

 The auto-decoding is certainly an issue, but that only affects
 ranges of char and wchar. The vast majority of dynamic arrays
 are perfectly normal ranges.

 - Jonathan M Davis


 Phobos' range facilities vomit when you try to deal with static
 arrays, and they have to be coerced into dynamic arrays before
 they can be dealt with. This is silly.


Except that static arrays are bona fide containers, and their length can't
be mutated. They don't make any sense as ranges at all.


 It gets under my skin that length and opIndex and opSlice produce
 different results in phobos' ranges depending on one's current
 position in the range. This doesn't make sense to me, and the
 only reason I can conceive of it having become how ranges work
 throughout phobos is because that's how dynamic arrays work if
 you force them to act as though they were ranges.


It's because ranges are effectively a sliding window over whatever they're
iterating over. The only way that it would make sense for slicing or
indexing a range to refer to anything other than what the range currently
refers to is if there were some sort of container that they were a slice of,
and the vast majority of ranges aren't that way. They're not designed to
refer to anything other than what they currently refer to, and there is no
original for the indices to refer back to. A range is simply a list of
values. It may or may not be randomly accessible, and it may or may not
refer to anything else like a container. It could be purely generative (e.g.
a range over the fibonacci sequence or an infinite list of random numbers).
It really makes no sense for indexing or slicing ranges to do anything but
have the indices treat the current front of the range as index 0. I can
understand that that might be confusing at first if you're thinking of them
as specifically refering to elements of a container, but many ranges have
nothing to do with a container at all.


 In every single other language I've used, the concept of an
 Iterable and an Iterator are distinct and very separate. An
 Iterator is something that can be iterated over; an Iterable is
 something which can produce an Iterator for iterating over its
 contents. In D, arrays are Iterables, and phobos endeavors to
 force them to be Iterators as well. It defies years of basic
 design wisdom regarding how to differentiate a collection and the
 means by which one enumerates the items in that collection.

 Arrays are Iterables which should be able to produce an Iterator,
 in D's case a range. They should not themselves be Iterators.


Dynamic arrays aren't really containers in D. Unlike containers, they don't
own or manage their own memory. They're only slices of that memory. The fact
that you can append to them, and the GC will then increase the amount of
memory that the dynamic array refers to (possibly allocating a new block of
memory and changing which block of memory the dynamic array is a slice of)
certainly muddies things, but it's still not the case the dynamic array owns
or manages its memory. What owns or manages the memory depends entirely on
what allocated the memory (usually the GC, but not always). Even in the case
of concatention or appending, its the GC that manages the memory, not the
dynamic array (e.g. a dynamic array which is a slice of malloced memory is
going to result in a memory leak if you then append to it, and nothing else
was keeping track of that memory; the array itself doesn't know or care who
owns or manages its memory). And multiple dynamic arrays can refer to
exactly the same block of memory, which is a clear indicator that they don't
own or manage their own memory. And in that respect, they're definitely more
like iterators than containers.

So, yes, dynamic arrays in D are weird hybrids, but aside from operations
related to concatenation or appending, they don't act like containers at
all. They act much more like a pair of iterators being passed around
together. And both the language and Phobos are very consistent about that.

The one place with dynamic arrays and ranges that is totally wacko is narrow
strings, because they're auto-decoded, because then Phobos tries to stop
treating them like dynamic arrays. If it didn't do that, AFAIK it would be
completely consistent with how it treated dynamic arrays.

I think that you'll have an easier time understanding dynamic arrays in D
and ranges in general if you stopped trying to treat dynamic arrays as if
they were containers, since they really aren't. And the few operations that
make them sort of act like containers are not part of the range API and
therefore don't conflict with ranges at all.

- Jonathan M Davis



 Sep 25 2016



prev sibling next sibling parent reply ZombineDev <petar.p.kirov gmail.com>  writes:


On Sunday, 25 September 2016 at 10:58:24 UTC, pineapple wrote:


 It gets under my skin that length and opIndex and opSlice 
 produce different results in phobos' ranges depending on one's 
 current position in the range. This doesn't make sense to me, 
 and the only reason I can conceive of it having become how 
 ranges work throughout phobos is because that's how dynamic 
 arrays work if you force them to act as though they were ranges.


But which opIndex and which length? Those of the container, or 
those of the range? It doesn't make any sense to expect 
container[].takeExactly(7).length to be different than 7. If 
range.length returns the length of the container this would break 
every sensible algorithm out there. How would you even implement 
binary search (https://rosettacode.org/wiki/Binary_search#D)? 
It's completely wrong to expect indexing operations to work on 
the original container. What you do in situations where there is 
no container, such as generators (e.g. iota 
https://github.com/dlang/phobos/blob/v2.071.2/std/range/package.d#L4722) and
data coming from the network? And how would you get the number of remaining
elements in the range?


 Phobos' range facilities vomit when you try to deal with static 
 arrays, and they have to be coerced into dynamic arrays before 
 they can be dealt with. This is silly.


What's the problem here? Just slice them using the [] syntax 
(i.e. the analog of asrange in your library).


 In every single other language I've used, the concept of an 
 Iterable and an Iterator are distinct and very separate. An 
 Iterator is something that can be iterated over; an Iterable is 
 something which can produce an Iterator for iterating over its 
 contents. In D, arrays are Iterables, and phobos endeavors to 
 force them to be Iterators as well. It defies years of basic 
 design wisdom regarding how to differentiate a collection and 
 the means by which one enumerates the items in that collection.

 Arrays are Iterables which should be able to produce an 
 Iterator, in D's case a range. They should not themselves be 
 Iterators.


There's seems to be a misunderstanding what are D's ranges and 
how they're meant to be used. Containers such as static arrays 
(i.e T[n]) are Iterable-s whereas slices (i.e. T[]) are 
Iterator-s by your taxonomy. You get the Iterator from the 
iterable static array using the opSlice (i.e. []) method. Ranges 
(and in particular slices) are just positions in an array. The 
same analogy holds for the containers in std.container: you get a 
range using the container[], or container.opSlice() syntax. 
Regardless where the range points to and if it shrinks, the 
container.length stays the same, assuming you haven't added or 
removed any elements.

I guess your misunderstanding stems from the fact that you call 
T[] arrays. This however is wrong. T[] is just slice of an array. 
If you want to use arrays similar to those in other languages, 
use std.container.array : Array 
(http://dlang.org/phobos/std_container_array).

D's built-in dynamic arrays are hidden from you and you only get 
to interact with them by referring to their elements by using 
slices. For example, have a look at the following code:

```
import std.algorithm.comparison : equal;

int[] arr = [1, 2, 3, 4];

void append5(int[] s)
{
     s ~= 5; // 1)
}

append5(arr);

assert (arr.equal([1, 2, 3, 4]));
```

As you can see in the example above, since int[] is just a 
slice/range, appending to it does not modify the underlying 
container. Instead a new container is created (*) and `s` is made 
to point to it in 1). If you were to use a "proper" container 
such as std.container.array, the results would be different:

```
import std.algorithm.comparison : equal;
import std.container.array;

auto arr = Array!int([1, 2, 3, 4]);

void append5(Array!int s)
{
     s ~= 5;
}

append5(arr);

assert (arr[].equal([1, 2, 3, 4, 5]));
```

For more information, I strongly suggest reading Steven's article 
: http://dlang.org/d-array-article.html (*)



 Sep 25 2016



 next sibling parent reply pineapple <meapineapple gmail.com>  writes:


On Sunday, 25 September 2016 at 13:10:42 UTC, ZombineDev wrote:

 But which opIndex and which length? Those of the container, or 
 those of the range? It doesn't make any sense to expect 
 container[].takeExactly(7).length to be different than 7. If 
 range.length returns the length of the container this would 
 break every sensible algorithm out there. How would you even 
 implement binary search 
 (https://rosettacode.org/wiki/Binary_search#D)? It's completely 
 wrong to expect indexing operations to work on the original 
 container. What you do in situations where there is no 
 container, such as generators (e.g. iota 
 https://github.com/dlang/phobos/blob/v2.071.2/std/range/package.d#L4722) and
data coming from the network? And how would you get the number of remaining
elements in the range?


The argument is not that the length of a range should always 
evaluate to the length of some container it enumerates, but that 
the length of the range should not change based on your position 
in the range. In phobos, popFront effectively reduces the length 
of the range by one, and offsets all further calls to opIndex and 
opSlice. I expect length, opIndex, and opSlice to behave the same 
whether I just built the range, am halfway through the range, or 
have fully consumed the range.


 Phobos' range facilities vomit when you try to deal with 
 static arrays, and they have to be coerced into dynamic arrays 
 before they can be dealt with. This is silly.


 What's the problem here? Just slice them using the [] syntax 
 (i.e. the analog of asrange in your library).


As Jonathan pointed out there's a lot of muddying of how dynamic 
arrays are only sorta-kinda genuine arrays. But when using them 
one should be able to rely on dynamic and static arrays have the 
same interface, except that one can be resized and appended to 
and another cannot. Having to slice static arrays to pass them 
into phobos' HOFs is icky and completely unnecessary. And though 
you might say that you could just append `[]` to every array you 
pass in, but while it will work for static and dynamic arrays it 
won't work for everything that can be passed to a function 
accepting a range.

Consistency of syntax for performing the same operation upon many 
different types is not to be undervalued.



 Sep 25 2016



  parent reply pineapple <meapineapple gmail.com>  writes:


On Sunday, 25 September 2016 at 11:48:38 UTC, Jonathan M Davis 
wrote:

 It's because ranges are effectively a sliding window over 
 whatever they're iterating over.


I think this is the difference in perception - ranges do not 
_have_ to be sliding windows, they can just as well be windows 
that don't move. I find the latter approach to be more workable 
and intuitive that the former, and in almost every case the 
approach results in more concise and performant code for 
performing those operations.

When would I ever want `range[0]` to give me the same thing as 
`range.front`? I want it to give me the first item in the range's 
view at the time of its creation.

I strongly oppose any change that makes the former approach an 
inherent part of D. I honestly don't think the latter approach 
should be an inherent part of D, either, but I'd have a hell of a 
time trying to use the language if it wasn't an option.



 Sep 25 2016



  parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sunday, September 25, 2016 13:40:14 pineapple via Digitalmars-d wrote:

 On Sunday, 25 September 2016 at 11:48:38 UTC, Jonathan M Davis

 wrote:

 It's because ranges are effectively a sliding window over
 whatever they're iterating over.


 I think this is the difference in perception - ranges do not
 _have_ to be sliding windows, they can just as well be windows
 that don't move. I find the latter approach to be more workable
 and intuitive that the former, and in almost every case the
 approach results in more concise and performant code for
 performing those operations.

 When would I ever want `range[0]` to give me the same thing as
 `range.front`? I want it to give me the first item in the range's
 view at the time of its creation.


You're going to be better off if you stop trying to think like there's some
sort of original range here. What you have right now is what you have. The
first element is index 0, and whether the range was just created or whether
it's had a million elements popped off the front is irrelevant. If you want
access to elements that aren't currently in the range, then you need to save
the range and keep that around so that you can access _that_ range instead
of the one that you currently have. Remember that many ranges are generative
and aren't backed by any sort of container and that there is no original
range that's been iterated over, and index 0 couldn't have anything to refer
to except for the current front.

And if you don't like how D's ranges are designed, sorry, but it's never
been like what you're suggesting, and changing it now would break a ton of
code even if we agreed that it were a good idea, and you're not going to get
that agreement. What we have with ranges right now is by no means perfect,
but the whole design is based on the sliding window concept - just like
D's dynamic arrays are - and that has worked fantastically for us.


 I strongly oppose any change that makes the former approach an
 inherent part of D. I honestly don't think the latter approach
 should be an inherent part of D, either, but I'd have a hell of a
 time trying to use the language if it wasn't an option.


The way it works now is how it's always worked with dynamic arrays and
ranges in D. If you're trying do anything else, you're just going to run
into problems in the long run - particularly when interacting with code
written by anyone else. So, while you're obviously free to do whatever you
want with your own code, don't expect Phobos or D code in general to change
how ranges fundamentally work.

- Jonathan M Davis



 Sep 25 2016



  parent reply pineapple <meapineapple gmail.com>  writes:


On Sunday, 25 September 2016 at 13:57:04 UTC, Jonathan M Davis 
wrote:

 The way it works now is how it's always worked with dynamic 
 arrays and ranges in D. If you're trying do anything else, 
 you're just going to run into problems in the long run - 
 particularly when interacting with code written by anyone else. 
 So, while you're obviously free to do whatever you want with 
 your own code, don't expect Phobos or D code in general to 
 change how ranges fundamentally work.


That change is exactly what I'm arguing against - that the front, 
popFront, etc. functions defined for dynamic arrays in phobos 
should not be adopted by the core language.

On Thursday, 22 September 2016 at 12:51:59 UTC, Andrei 
Alexandrescu wrote:

 Would make sense to move those few primitives to object.d. I've 
 been thinking of that a long time ago but back then there was a 
 vague stance that object.d shouldn't contain templates. Since 
 then that has changed. -- Andrei


Please do not do this - there ways to handle ranges other than 
the approach phobos has taken. That's it, that's the point I'm 
trying to make.



 Sep 25 2016



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/25/16 4:05 PM, pineapple wrote:

 On Sunday, 25 September 2016 at 13:57:04 UTC, Jonathan M Davis wrote:

 The way it works now is how it's always worked with dynamic arrays and
 ranges in D. If you're trying do anything else, you're just going to
 run into problems in the long run - particularly when interacting with
 code written by anyone else. So, while you're obviously free to do
 whatever you want with your own code, don't expect Phobos or D code in
 general to change how ranges fundamentally work.


 That change is exactly what I'm arguing against - that the front,
 popFront, etc. functions defined for dynamic arrays in phobos should not
 be adopted by the core language.

 On Thursday, 22 September 2016 at 12:51:59 UTC, Andrei Alexandrescu wrote:

 Would make sense to move those few primitives to object.d. I've been
 thinking of that a long time ago but back then there was a vague
 stance that object.d shouldn't contain templates. Since then that has
 changed. -- Andrei


 Please do not do this - there ways to handle ranges other than the
 approach phobos has taken. That's it, that's the point I'm trying to make.


We learned with time that any step we're trying to take toward progress 
in a matter of design (i.e. no mechanical rules, reasonable people may 
disagree), a faction will strongly oppose it. I speculate this has to do 
with our community being self-selected as opinionated folks who don't do 
well with conventional wisdom.

At the same time, we can't let this gridlock development of D. We must 
go with what we think is good.

It seems you want to define ranges with similar syntax but subtle 
semantic differences, e.g. r.front and r[0] to mean different things. 
The entire Phobos is designed under the assumptions that ranges work a 
specific way, so in order to design a different mechanism you may want 
to use different syntactic interfaces.


Andrei



 Sep 25 2016



  parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sunday, September 25, 2016 16:50:04 Andrei Alexandrescu via Digitalmars-d 
wrote:

 It seems you want to define ranges with similar syntax but subtle
 semantic differences, e.g. r.front and r[0] to mean different things.
 The entire Phobos is designed under the assumptions that ranges work a
 specific way, so in order to design a different mechanism you may want
 to use different syntactic interfaces.


The reality of the matter is that anyone who tries to define the range
primitives to work differently than how Phobos uses them (and druntime in
various places even if it's not in object.d yet) is going to be screwed as
soon as they interact with code written by anyone else. Anyone looking to
make r[0] do something different than give you r.front might as well just
redefine popFront to mean popBack and vice versa for all that it's going to
work with other people's code.

So, if they want their code to work with anyone else's code they pretty much
need to use their own set of range primitives that do not conflict with the
standard ones rather than trying to redefine the standard ones. And if they
don't care about interacting with anyone else's code, they can always just
fork druntime and Phobos to make them do whatever they want. But trying to
redefine some of the basic primitives that D's runtime and standard library
use while still trying to interact with anyone else's code is a recipe for
disaster.

- Jonathan M Davis



 Sep 25 2016



  parent  pineapple <meapineapple gmail.com>  writes:


On Sunday, 25 September 2016 at 15:25:38 UTC, Jonathan M Davis 
wrote:

 So, if they want their code to work with anyone else's code 
 they pretty much need to use their own set of range primitives 
 that do not conflict with the standard ones rather than trying 
 to redefine the standard ones. And if they don't care about 
 interacting with anyone else's code, they can always just fork 
 druntime and Phobos to make them do whatever they want. But 
 trying to redefine some of the basic primitives that D's 
 runtime and standard library use while still trying to interact 
 with anyone else's code is a recipe for disaster.

 - Jonathan M Davis


I don't mind writing my own code rather than interacting with 
someone else's, and I severely dislike many of the design 
decisions made regarding phobos. Which is why I've been building 
my own alternative to the standard library that I can use as a 
basis for software I develop. It depends on phobos for only a 
handful of things, and I'm working toward a point where I won't 
need it for anything. I recognize that the preference is unusual, 
but I insist on my and others' ability to pursue such a 
preference.

On Sunday, 25 September 2016 at 14:50:04 UTC, Andrei Alexandrescu 
wrote:

 It seems you want to define ranges with similar syntax but 
 subtle semantic differences, e.g. r.front and r[0] to mean 
 different things. The entire Phobos is designed under the 
 assumptions that ranges work a specific way, so in order to 
 design a different mechanism you may want to use different 
 syntactic interfaces.


I have no problem with phobos being phobos, and treating ranges 
as it does. I don't want the core language to adopt the same way 
of treating ranges because while I recognize that it is far too 
late to change phobos' way of thinking about ranges - much less 
the community's - I think it was a mistake and that the quality 
of D as a language shouldn't suffer for its sake. The core 
language should define the bare minimum that it needs to for 
ranges to be a useful concept - as it does now - and should leave 
the rest up to phobos or whatever else is actually implementing 
the ranges.

On Sunday, 25 September 2016 at 14:50:04 UTC, Andrei Alexandrescu 
wrote:

 I speculate this has to do with our community being 
 self-selected as opinionated folks who don't do well with 
 conventional wisdom.


You have described me to a T.



 Sep 25 2016



prev sibling  parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sunday, September 25, 2016 13:10:42 ZombineDev via Digitalmars-d wrote:

 D's built-in dynamic arrays are hidden from you and you only get
 to interact with them by referring to their elements by using
 slices.


That's a common misconception propagated by Steven's otherwise excellent
article. As far as the official language spec goes, T[] _is_ the dynamic
array. What backs it is irrelevant as far as that goes. It's just that if
it's backed by the GC, then when you append to it, the GC might not have to
allocate a new memory buffer for the array. All of the operations on a
dynamic array work the same regardless of what backs it, and focusing on the
memory buffer being the array risks causing you problems when you have to
deal with dynamic arrays backed by something else.

But on the whole, what you said was right. It's just the terminology that's
off.

- Jonathan M Davis



 Sep 25 2016



  parent  Steven Schveighoffer <schveiguy yahoo.com>  writes:


On 9/25/16 9:48 AM, Jonathan M Davis via Digitalmars-d wrote:

 On Sunday, September 25, 2016 13:10:42 ZombineDev via Digitalmars-d wrote:

 D's built-in dynamic arrays are hidden from you and you only get
 to interact with them by referring to their elements by using
 slices.


 That's a common misconception propagated by Steven's otherwise excellent
 article. As far as the official language spec goes, T[] _is_ the dynamic
 array. What backs it is irrelevant as far as that goes. It's just that if
 it's backed by the GC, then when you append to it, the GC might not have to
 allocate a new memory buffer for the array. All of the operations on a
 dynamic array work the same regardless of what backs it, and focusing on the
 memory buffer being the array risks causing you problems when you have to
 deal with dynamic arrays backed by something else.


D can call an animal that quacks, has a flat bill, and feathers a 
sparrow all it wants, but it sure doesn't act like one ;)

-Steve



 Sep 27 2016



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/25/16 12:58 PM, pineapple wrote:

 On Sunday, 25 September 2016 at 04:06:41 UTC, Jonathan M Davis wrote:

 Considering that a random access range is essentially an abstraction
 for a dynamic array and that ranges were designed with that in mind, I
 don't know how you can argue that dynamic arrays shouldn't be treated
 as ranges.

 The auto-decoding is certainly an issue, but that only affects ranges
 of char and wchar. The vast majority of dynamic arrays are perfectly
 normal ranges.

 - Jonathan M Davis


 Phobos' range facilities vomit when you try to deal with static arrays,
 and they have to be coerced into dynamic arrays before they can be dealt
 with. This is silly.


Yah, it comes as a surprise to many that static arrays are more akin to 
structs than to arrays. They really are records that happen to have 
several elements of the same type. Providing indexing for them is a 
convenience that somewhat adds to the confusion, and converting to 
dynamic arrays is unsafe because it essentially escapes the innards of 
the struct. Statically-sized arrays are odd, that's for sure, and that's 
the case in C and C++ as well.


 It gets under my skin that length and opIndex and opSlice produce
 different results in phobos' ranges depending on one's current position
 in the range. This doesn't make sense to me, and the only reason I can
 conceive of it having become how ranges work throughout phobos is
 because that's how dynamic arrays work if you force them to act as
 though they were ranges.


Better get used to it. Ranges are a generalization of D's slices, not 
the other way around.


 In every single other language I've used, the concept of an Iterable and
 an Iterator are distinct and very separate. An Iterator is something
 that can be iterated over; an Iterable is something which can produce an
 Iterator for iterating over its contents. In D, arrays are Iterables,
 and phobos endeavors to force them to be Iterators as well. It defies
 years of basic design wisdom regarding how to differentiate a collection
 and the means by which one enumerates the items in that collection.


Ranges don't need necessarily an associated Iterable. This is the case 
in other languages, too; Lisp/Scheme/Haskell/etc lists are iterables and 
at the same time their own iterators. But indeed std.container is 
designed to have containers distinct from their associated ranges, which 
raises the interesting question what should happen if a range gets 
"orphaned", i.e. it's still active after its container ceases to exist.


 Arrays are Iterables which should be able to produce an Iterator, in D's
 case a range. They should not themselves be Iterators.


Yah, std.container.Array follows that. It's not an be-all-end-all of 
design though.


Andrei



 Sep 25 2016



 next sibling parent  pineapple <meapineapple gmail.com>  writes:


On Sunday, 25 September 2016 at 13:45:01 UTC, Andrei Alexandrescu 
wrote:

 Ranges don't need necessarily an associated Iterable. This is 
 the case in other languages, too; Lisp/Scheme/Haskell/etc lists 
 are iterables and at the same time their own iterators. But 
 indeed std.container is designed to have containers distinct 
 from their associated ranges, which raises the interesting 
 question what should happen if a range gets "orphaned", i.e. 
 it's still active after its container ceases to exist.


And an Iterator doesn't necessarily need an associated Iterable.

I fully recognize how phobos handles ranges, and that it has 
become "the right way" to do things. But I disagree, and I do not 
want the core language to force me to take phobos' approach. I 
greatly appreciate that the only properties of ranges the core 
language cares about are `empty`, `front`, `back`, `popFront`, 
`popBack` - it does not itself imply how `opIndex` and `opSlice` 
are meant to behave for ranges - and I think it should stay that 
way. I see phobos' arrays-as-ranges functions as poor design, and 
most certainly not something that should become inextricably tied 
with the core language.



 Sep 25 2016



prev sibling next sibling parent  Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Sunday, September 25, 2016 15:45:01 Andrei Alexandrescu via Digitalmars-d 
wrote:

 Ranges don't need necessarily an associated Iterable. This is the case
 in other languages, too; Lisp/Scheme/Haskell/etc lists are iterables and
 at the same time their own iterators. But indeed std.container is
 designed to have containers distinct from their associated ranges, which
 raises the interesting question what should happen if a range gets
 "orphaned", i.e. it's still active after its container ceases to exist.


Another thing to consider here is that given how most range-based functions
will create a new range from the container's range when you pass it to them,
when operating on ranges over containers, you _very_ quickly end up with
ranges that really have nothing to do with the container anymore (hence the
fun problems with removing elements from a container by passing a range to
it). And that highlights how ranges really don't act like they're backed by
containers.

But the safety issue that comes with ranges over containers where the
container goes away is definitely a thorny one - as is the fact that
removing elements from the container while you have active ranges that refer
to it can do funny things to those ranges. I tend to favor C++'s approach to
iterators and how they stay valid, but with D's focus on safety, I don't
know that that's acceptable (though enforcing safety tends to result in
additional overhead which isn't in line with the efficiency goals that go
with being a system language). So, I don't know what the best approach is.

- Jonathan M Davis



 Sep 25 2016



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 9/25/2016 6:45 AM, Andrei Alexandrescu wrote:

 Yah, it comes as a surprise to many that static arrays are more akin to structs
 than to arrays. They really are records that happen to have several elements of
 the same type. Providing indexing for them is a convenience that somewhat adds
 to the confusion, and converting to dynamic arrays is unsafe because it
 essentially escapes the innards of the struct. Statically-sized arrays are odd,
 that's for sure, and that's the case in C and C++ as well.


I'd like to emphasize that this is an important insight. A static array is 
semantically equivalent to a struct with fields of all the same type, and the 
fields can be accessed by index rather than field name. This insight has driven 
some simplifying assumptions internal to how the compiler is implemented, as 
well as the specification for how things work (like passing a static array as a 
function parameter works like passing a struct).

Another simplifying insight is that a struct is a tuple of fields. I tried to 
generalize this by having the arguments to a function be a tuple as well, but 
ran afoul of the ABI for calling conventions, argghh. It would really be
awesome 
to have tuples, structs, static arrays, and function argument lists be
literally 
the same thing, but sadly that does not seem practical at the moment.



 Sep 25 2016



prev sibling  parent reply Sai <test test.com>  writes:


 The greatest offender I've found is how in phobos, arrays do 
 not behave as ranges without importing the module defining 
 their range operations.


Could you please tell me what module is it? is it std.array?



 Sep 23 2016



  parent  Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Friday, September 23, 2016 13:47:06 Sai via Digitalmars-d wrote:

 The greatest offender I've found is how in phobos, arrays do
 not behave as ranges without importing the module defining
 their range operations.


 Could you please tell me what module is it? is it std.array?


It was, but they were moved to std.range.primitives some time in the last
year or two. They're still publicly imported in std.array though, so
importing std.array will work.

- Jonathan M Davis



 Sep 23 2016