www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Generality creep

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter and I were looking at indexOf in the standard library. It has 
several overloads, of which let's look at the first two:

https://dlang.org/library/std/string/index_of.html

Now, why couldn't they be merged into a single function with this 
constraint (their bodies are identical):

ptrdiff_t indexOf(Range) (
   Range s,
   dchar c,
   CaseSensitive cs = Yes.caseSensitive
)
if (isInputRange!Range && isSomeChar!(ElementType!Range));

It makes sense to look for a character in any range of characters.

Attempting to build that fails because of this type fails to work:

     struct TestAliasedString
     {
         string get()  safe  nogc pure nothrow { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

The intuition is that the function should be general enough to figure 
out that, hey, TestAliasedString is kinda sorta a subtype of string. So 
the call should work.

So let's see why it doesn't work - i.e. why is TestAliasedString an 
input range? The definition of isInputRange (in std.range.primitives) is:

enum bool isInputRange(R) =
     is(typeof(R.init) == R)
     && is(ReturnType!((R r) => r.empty) == bool)
     && is(typeof((return ref R r) => r.front))
     && !is(ReturnType!((R r) => r.front) == void)
     && is(typeof((R r) => r.popFront));

Turns out the second clause fails. That takes us to the definition of 
empty in the same module:

 property bool empty(T)(auto ref scope const(T) a)
if (is(typeof(a.length) : size_t))
{
     return !a.length;
}

The intent is fairly clear - if a range defines empty as a size_t 
(somewhat oddly relaxed to "convertible to size_t"), then empty can be 
nicely defined in terms of length. Cool. But empty doesn't work with 
TestAliasedString due to an overlooked matter: the "const". A mutable 
TestAliasedString converts to a string, but a const or immutable 
TestAliasedString does NOT convert to a const string! So this fixes that 
matter:

     struct TestAliasedString
     {
         string get()  safe  nogc pure nothrow { return _s; }
         const(string) get()  safe  nogc pure nothrow const { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

That makes empty() work, but also raises a nagging question: what was 
the relationship of TestAliasedString to string before this change? 
Surely that wasn't subtyping. (My response would be: "Odd.") And why was 
Phobos under the obligation to cater for such a type and its tenuous 
relationship to a range?

But wait, there's more. Things still don't work because of popFront. 
Looking at its definition:

void popFront(C)(scope ref inout(C)[] str)  trusted pure nothrow
if (isNarrowString!(C[]))
{ ... }

So, reasonably this function takes the range by reference so it can 
modify its internals. HOWEVER! The implementation of 
TestAliasedString.get() returns an rvalue, i.e. it's the equivalent of a 
conversion involving a temporary. Surely that's not to match, whether in 
the current language or the one after the rvalue DIP.

The change that does make the code work is:

     struct TestAliasedString
     {
         ref string get()  safe  nogc pure nothrow { return _s; }
         ref const(string) get()  safe  nogc pure nothrow const { return 
_s; }
         alias get this;
          disable this(this);
         string _s;
     }

This indeed does implement a subtyping relationship, and passes the 
isInputRange test.

What's the moral of the story here? Generality is good, but it seems in 
several places in phobos (of which this is just one example), a 
combination of vague specification and aiming for a nice ideal of "work 
with anything remotely reasonable" has backfired into a morass of 
inconsistently defined and supported corner cases.

For this case in particular - I don't think we should support all types 
that support some half-hearted form of subtyping, at the cost of 
reducing generality and deprecating working code.
Mar 18
next sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:
  property bool empty(T)(auto ref scope const(T) a)
 if (is(typeof(a.length) : size_t))
 {
     return !a.length;
 }

 The intent is fairly clear - if a range defines empty as a 
 size_t (somewhat oddly relaxed to "convertible to size_t"), 
 then empty can be nicely defined in terms of length.
If it was intended for arrays only, then property bool empty(T)(scope const T[] a) { return !a.length; } But if not, what is the reason to define length, but not empty? Anyway, the type may need to be mutable, e.g. it might want to allocate and store the array, so property bool empty(T)(auto ref scope T a) if (is(typeof(a.length) : size_t)) { return !a.length; }
 That makes empty() work, but also raises a nagging question: 
 what was the relationship of TestAliasedString to string before 
 this change? Surely that wasn't subtyping. (My response would 
 be: "Odd.") And why was Phobos under the obligation to cater 
 for such a type and its tenuous relationship to a range?
The design rationale is not documented :) so we can only guess. My guess is that it was done for some string container (appender?) that returns rvalue string, so the change to ref return would be incorrect.
 But wait, there's more. Things still don't work because of 
 popFront. Looking at its definition:

 void popFront(C)(scope ref inout(C)[] str)  trusted pure nothrow
 if (isNarrowString!(C[]))
 { ... }

 So, reasonably this function takes the range by reference so it 
 can modify its internals. HOWEVER! The implementation of 
 TestAliasedString.get() returns an rvalue, i.e. it's the 
 equivalent of a conversion involving a temporary. Surely that's 
 not to match, whether in the current language or the one after 
 the rvalue DIP.
That's conflation of collections and ranges. A collection would be a range factory.
 What's the moral of the story here? Generality is good, but it 
 seems in several places in phobos (of which this is just one 
 example), a combination of vague specification and aiming for a 
 nice ideal of "work with anything remotely reasonable" has 
 backfired into a morass of inconsistently defined and supported 
 corner cases.

 For this case in particular - I don't think we should support 
 all types that support some half-hearted form of subtyping, at 
 the cost of reducing generality and deprecating working code.
Isn't generality one of phobos acceptance criteria?
Mar 19
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/19/2019 2:01 AM, Kagamin wrote:
 Isn't generality one of phobos acceptance criteria?
I'm not so sure about that. Should Phobos work on abstractions that have a square wheel? Or only well-formed abstractions? In my experience, when an abstraction has a square wheel, it's genesis was a poor understanding of how to build an abstraction, not a requirement of that abstraction. Hence, rejecting it helps the programmer build better abstractions. I.e. should Phobos accept things because it can, or because it should?
Mar 19
prev sibling next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:
 And why was Phobos under the obligation to cater for such a 
 type and its tenuous relationship to a range?
There's two paths to investigate this: 1. Use git blame to find the commit when it was introduced, and follow the thread to the pull request, Bugzilla issue, etc. 2. Open a draft PR with a fix, and see what breaks (both within D repositories i.e. auto-tester, and in the D ecosystem at large i.e. the project tester). Good to see you working on D actively again BTW.
Mar 19
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/19/19 9:04 AM, Vladimir Panteleev wrote:
 On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu wrote:
 And why was Phobos under the obligation to cater for such a type and 
 its tenuous relationship to a range?
There's two paths to investigate this: 1. Use git blame to find the commit when it was introduced, and follow the thread to the pull request, Bugzilla issue, etc. 2. Open a draft PR with a fix, and see what breaks (both within D repositories i.e. auto-tester, and in the D ecosystem at large i.e. the project tester).
Thanks. I'm less preoccupied with figuring out who wrote that code, as with addressing the systemic matter. My working hypothesis is a (milder) form of "the road to hell is paved with good intentions" - some arguably undesirable generality has accumulated slowly as many contributors with varying design sensibilities operated under the vague desideratum that more generality is better, ran into a variety of complex cases, and contributed their solution to the standard library. I got word a similar phenomenon is at work in Scala - the language and its libraries are very powerful, but the boundaries of power are fuzzy and some users are in a continuous state of frustration that some constructs that ought to work do not due to byzantine limitations. At any rate, there seems to be some good opportunity to significantly simplifying the standard library by dropping some of the more obscure supported cases.
Mar 20
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:
 Walter and I were looking at indexOf in the standard library. 
 It has several overloads, of which let's look at the first two:

 [...]
+1 that these corner cases should not be supported in phobos, but at this point how could phobos drop support for these cases? In fact I'm pretty sure I've written a fair amount of code that depends on them (i.e. using alias this to subtype with an rvalue...hey, we all make mistakes). Of course I'm more than willing to fix my code but we're talking about a huge breaking change here. What's the path forward?
Mar 19
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/19/2019 6:35 AM, Jonathan Marler wrote:
 +1 that these corner cases should not be supported in phobos, but at this
point 
 how could phobos drop support for these cases? In fact I'm pretty sure I've 
 written a fair amount of code that depends on them (i.e. using alias this to 
 subtype with an rvalue...hey, we all make mistakes). Of course I'm more than 
 willing to fix my code but we're talking about a huge breaking change here. 
 What's the path forward?
I'd suggest making a separate overload for the square wheels, and deprecate them.
Mar 19
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/18/19 10:52 PM, Andrei Alexandrescu wrote:
  property bool empty(T)(auto ref scope const(T) a)
 if (is(typeof(a.length) : size_t))
 {
      return !a.length;
 }
 
 The intent is fairly clear - if a range defines empty as a size_t 
 (somewhat oddly relaxed to "convertible to size_t"), then empty can be 
 nicely defined in terms of length. Cool.
This is implemented wrong. import std.range; struct S { size_t length() const { return 0; } } void main() { S s; assert(s.length == 0); OK assert(s.empty); // Error, cannot deduce function } The problem is a simple one: typeof(a.length) is not size_t, but a function type. If you put property on the length function it would work, but I think we should not require that. -Steve
Mar 19
parent reply Kagamin <spam here.lot> writes:
On Tuesday, 19 March 2019 at 14:08:24 UTC, Steven Schveighoffer 
wrote:
 The problem is a simple one: typeof(a.length) is not size_t, 
 but a function type.

 If you put  property on the length function it would work, but 
 I think we should not require that.

 -Steve
struct S { size_t length() const { return 0; } } auto ref property(T)(auto ref T a){ return a; } void f() { S s; assert(s.length == 0); //OK static assert(is(typeof(property(s.length))==size_t)); } Can this work?
Mar 20
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/20/19 1:41 PM, Kagamin wrote:
 On Tuesday, 19 March 2019 at 14:08:24 UTC, Steven Schveighoffer wrote:
 The problem is a simple one: typeof(a.length) is not size_t, but a 
 function type.

 If you put  property on the length function it would work, but I think 
 we should not require that.

 -Steve
struct S {    size_t length() const { return 0; } } auto ref property(T)(auto ref T a){ return a; } void f() {    S s;    assert(s.length == 0); //OK    static assert(is(typeof(property(s.length))==size_t)); } Can this work?
Yep, that's a good idea actually. Make it inout, to cut down on template bloat. I actually use something less nice in iopipe[1], so I may switch to this. -Steve [1] https://github.com/schveiguy/iopipe/blob/master/source/iopipe/traits.d#L111-L119
Mar 20
next sibling parent reply Kagamin <spam here.lot> writes:
On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer 
wrote:
 Yep, that's a good idea actually. Make it inout, to cut down on 
 template bloat.
T property(T)(auto ref T a); This still compiles. Will it bloat?
Mar 21
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/21/19 12:03 PM, Kagamin wrote:
 On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer wrote:
 Yep, that's a good idea actually. Make it inout, to cut down on 
 template bloat.
T property(T)(auto ref T a); This still compiles. Will it bloat?
It's not the auto ref, it's the mutability: auto ref property1(T)(auto ref T a) { pragma(msg, T.stringof ~ " without inout"); return a; } auto ref property2(T)(auto ref inout(T) a) { pragma(msg, T.stringof ~ " with inout"); return a; } void main() { const int a = 1; immutable int b = 2; int c = 3; // 3 instantiations assert(a.property1 == a); assert(b.property1 == b); assert(c.property1 == c); // 1 instantiation assert(a.property2 == a); assert(b.property2 == b); assert(c.property2 == c); } output: const(int) without inout immutable(int) without inout int without inout int with inout -Steve
Mar 22
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer 
wrote:
 On 3/20/19 1:41 PM, Kagamin wrote:
 struct S
 {
     size_t length() const { return 0; }
 }
 
 auto ref property(T)(auto ref T a){ return a; }
 
 void f()
 {
     S s;
     assert(s.length == 0); //OK
     static assert(is(typeof(property(s.length))==size_t));
 }
 
 Can this work?
Yep, that's a good idea actually. Make it inout, to cut down on template bloat. I actually use something less nice in iopipe[1], so I may switch to this.
Rather than define a new `property` symbol, why not rewrite the template constraint for `empty` as: property bool empty (T) (auto ref scope T a) if(is(ReturnType!((T t) => t.length) : size_t)) { return !a.length; } This matches what is done in the `isInputRange` check for the existence of `empty` (I suspect it's done that way because it's quite common for `SomeRange.empty` to be an enum rather than a method or property).
Mar 21
next sibling parent reply Kagamin <spam here.lot> writes:
On Thursday, 21 March 2019 at 16:11:04 UTC, Joseph Rushton 
Wakeling wrote:
  property bool empty (T) (auto ref scope T a)
     if(is(ReturnType!((T t) => t.length) : size_t))
 {
     return !a.length;
 }
Here comes generality creep :) struct A { int a; disable this(this); bool opCast(){ return !!a; } } struct S { A a; ref A p() { return a; } } T property(T)(auto ref T a); import std.traits; void f() { S s; assert(!s.p); //negation works static assert(is(typeof(property(s.p))==A)); static assert(is(ReturnType!(()=>s.p):A)); } The ReturnType assert doesn't pass. Any idea why? Well, maybe it shouldn't.
Mar 21
parent Kagamin <spam here.lot> writes:
Apparently there are options

T property(T)(scope T a);
will check that the value can be retained from the property for 
local use

T property(T)(T a);
will check that the value can be returned from function (not 
scoped)
Mar 22
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 21, 2019 10:11:04 AM MDT Joseph Rushton Wakeling via 
Digitalmars-d wrote:
 On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer

 wrote:
 On 3/20/19 1:41 PM, Kagamin wrote:
 struct S
 {

     size_t length() const { return 0; }

 }

 auto ref property(T)(auto ref T a){ return a; }

 void f()
 {

     S s;
     assert(s.length == 0); //OK
     static assert(is(typeof(property(s.length))==size_t));

 }

 Can this work?
Yep, that's a good idea actually. Make it inout, to cut down on template bloat. I actually use something less nice in iopipe[1], so I may switch to this.
Rather than define a new `property` symbol, why not rewrite the template constraint for `empty` as: property bool empty (T) (auto ref scope T a) if(is(ReturnType!((T t) => t.length) : size_t)) { return !a.length; } This matches what is done in the `isInputRange` check for the existence of `empty` (I suspect it's done that way because it's quite common for `SomeRange.empty` to be an enum rather than a method or property).
Honestly, this is another example of over-generality causing problems. length shouldn't ever be anything other than size_t. Anything else causes problems. - Jonathan M Davis
Mar 30
prev sibling next sibling parent 9il <ilyayaroshenko gmail.com> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:
 Walter and I were looking at indexOf in the standard library.

 What's the moral of the story here? Generality is good, but it 
 seems in several places in phobos (of which this is just one 
 example), a combination of vague specification and aiming for a 
 nice ideal of "work with anything remotely reasonable" has 
 backfired into a morass of inconsistently defined and supported 
 corner cases.
Phobos looks too generalized for me and not enough standardized. For (well known) example, min/max functions for integers. Both for security and safety reasons it is better to raise a compile-time error rather than to implement clever signity-aware comparison. I did this in Mir. There a lot of such small nitpicks but their reimplementation make life either for middle size and large projects. It would be nice to have well-standardized generic building blocks like `mir`/`max` in Phobos with verifiable BetterC availability. Best, Ilya
Mar 19
prev sibling next sibling parent Rubn <where is.this> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:
 Walter and I were looking at indexOf in the standard library. 
 It has several overloads, of which let's look at the first two:

 https://dlang.org/library/std/string/index_of.html

 Now, why couldn't they be merged into a single function with 
 this constraint (their bodies are identical):

 ptrdiff_t indexOf(Range) (
   Range s,
   dchar c,
   CaseSensitive cs = Yes.caseSensitive
 )
 if (isInputRange!Range && isSomeChar!(ElementType!Range));

 It makes sense to look for a character in any range of 
 characters.

 Attempting to build that fails because of this type fails to 
 work:

     struct TestAliasedString
     {
         string get()  safe  nogc pure nothrow { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

 The intuition is that the function should be general enough to 
 figure out that, hey, TestAliasedString is kinda sorta a 
 subtype of string. So the call should work.

 So let's see why it doesn't work - i.e. why is 
 TestAliasedString an input range? The definition of 
 isInputRange (in std.range.primitives) is:

 enum bool isInputRange(R) =
     is(typeof(R.init) == R)
     && is(ReturnType!((R r) => r.empty) == bool)
     && is(typeof((return ref R r) => r.front))
     && !is(ReturnType!((R r) => r.front) == void)
     && is(typeof((R r) => r.popFront));

 Turns out the second clause fails. That takes us to the 
 definition of empty in the same module:

  property bool empty(T)(auto ref scope const(T) a)
 if (is(typeof(a.length) : size_t))
 {
     return !a.length;
 }

 The intent is fairly clear - if a range defines empty as a 
 size_t (somewhat oddly relaxed to "convertible to size_t"), 
 then empty can be nicely defined in terms of length. Cool. But 
 empty doesn't work with TestAliasedString due to an overlooked 
 matter: the "const". A mutable TestAliasedString converts to a 
 string, but a const or immutable TestAliasedString does NOT 
 convert to a const string! So this fixes that matter:

     struct TestAliasedString
     {
         string get()  safe  nogc pure nothrow { return _s; }
         const(string) get()  safe  nogc pure nothrow const { 
 return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

 That makes empty() work, but also raises a nagging question: 
 what was the relationship of TestAliasedString to string before 
 this change? Surely that wasn't subtyping. (My response would 
 be: "Odd.") And why was Phobos under the obligation to cater 
 for such a type and its tenuous relationship to a range?

 But wait, there's more. Things still don't work because of 
 popFront. Looking at its definition:

 void popFront(C)(scope ref inout(C)[] str)  trusted pure nothrow
 if (isNarrowString!(C[]))
 { ... }

 So, reasonably this function takes the range by reference so it 
 can modify its internals. HOWEVER! The implementation of 
 TestAliasedString.get() returns an rvalue, i.e. it's the 
 equivalent of a conversion involving a temporary. Surely that's 
 not to match, whether in the current language or the one after 
 the rvalue DIP.

 The change that does make the code work is:

     struct TestAliasedString
     {
         ref string get()  safe  nogc pure nothrow { return _s; }
         ref const(string) get()  safe  nogc pure nothrow const 
 { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

 This indeed does implement a subtyping relationship, and passes 
 the isInputRange test.

 What's the moral of the story here? Generality is good, but it 
 seems in several places in phobos (of which this is just one 
 example), a combination of vague specification and aiming for a 
 nice ideal of "work with anything remotely reasonable" has 
 backfired into a morass of inconsistently defined and supported 
 corner cases.

 For this case in particular - I don't think we should support 
 all types that support some half-hearted form of subtyping, at 
 the cost of reducing generality and deprecating working code.
So basically one more reason to avoid const entirely?
Mar 19
prev sibling next sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:
 The change that does make the code work is:

     struct TestAliasedString
     {
         ref string get()  safe  nogc pure nothrow { return _s; }
         ref const(string) get()  safe  nogc pure nothrow const 
 { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }
Isn't this exactly the kind of code where you're supposed to use inout?
Mar 20
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
A topical example:

https://github.com/dlang/phobos/pull/6346#issuecomment-477504005
Mar 28
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 09:52, Walter Bright wrote:
 A topical example:
 
 https://github.com/dlang/phobos/pull/6346#issuecomment-477504005
That pull request was open for a year(!) waiting for someone to make a decision. And before that I had asked about the issue on the forum [1]. If you want things to go your way, consider answering when someone asks for direction. By the way, I didn't get an email notification from GitHub for that comment. Does GitHub stop sending those when the pull request is merged? That would be worrying. Did anyone else get a notificaton? [1] https://forum.dlang.org/post/p96gs4$22fp$1 digitalmars.com
Mar 28
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 7:02 AM, ag0aep6g wrote:
 On 28.03.19 09:52, Walter Bright wrote:
 A topical example:

 https://github.com/dlang/phobos/pull/6346#issuecomment-477504005
That pull request was open for a year(!) waiting for someone to make a decision. And before that I had asked about the issue on the forum [1]. If you want things to go your way, consider answering when someone asks for direction.
That's good work actually (which underlines the irony of the situation) - once refRange got in the question of supporting it came naturally. Generality crept in, one good intention at a time. We should use D's good name lookup mechanism to create a versioning strategy a la std.v2 that reuses the good parts of std and fixes the bad parts. That way we focus on adding good things and letting bad things in maintenance mode, as opposed to changing existing things. Even C++, with its rather poor modularity mechanism, has made std::tr1 and std::tr2 work.
Mar 28
prev sibling next sibling parent ag0aep6g <anonymous example.com> writes:
On 28.03.19 12:02, ag0aep6g wrote:
 By the way, I didn't get an email notification from GitHub for that 
 comment.
Scratch that, it arrived. Just took a couple of hours.
Mar 28
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 4:02 AM, ag0aep6g wrote:
 If you want things to go your way, consider answering when someone asks for 
 direction.
I simply cannot keep a finger on everything that is happening, and have to rely on delegating. This one caught my eye because I had just finished making those ranges DIP1000 compatible, and then some large changes to them followed right afterwards, and I thought they might be related.
Mar 28
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 22:33, Walter Bright wrote:
 This one caught my eye because I had just 
 finished making those ranges DIP1000 compatible, and then some large 
 changes to them followed right afterwards, and I thought they might be 
 related.
I don't mind you being against those changes, but the changes so far are hardly "large". Ignoring the tests, both my PRs combined add meager 20 lines. And that's for seven bug fixes. By the way, you recently made changes to `save` methods in std.utf: https://github.com/dlang/phobos/pull/6900 https://github.com/dlang/phobos/pull/6903 I'm not sure why you made those, but they touch the same kind of code as my PRs, and they actually fix the issue I was fixing, too. Are you planning on applying that pattern to all of Phobos, or just std.utf for some reason? If you're going for all of Phobos, you might end up fixing issue 18657 as a side effect. But removing RefRange.opAssign would still be nice, because it's just weird.
Mar 28
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 4:05 PM, ag0aep6g wrote:
 I don't mind you being against those changes, but the changes so far are
hardly 
 "large". Ignoring the tests, both my PRs combined add meager 20 lines. And 
 that's for seven bug fixes.
Right, but those changes cut across a large number of algorithms, and will cause every algorithm implementer to do the same. That makes it a very large change. I'm glad I noticed this before a lot more work was expended in that direction.
 By the way, you recently made changes to `save` methods in std.utf:
 
 https://github.com/dlang/phobos/pull/6900
 https://github.com/dlang/phobos/pull/6903
 
 I'm not sure why you made those,
The problem I ran into was ranges that had const fields would fail, because one cannot assign to a const field. But one can construct a const field. Doing some digging, it became apparent that save() is equivalent to copy construction, and so should be implemented using copy construction. Assignment and construction are fundamentally different, and save() is clearly a construction.
 but they touch the same kind of code as my PRs, 
 and they actually fix the issue I was fixing, too. Are you planning on
applying 
 that pattern to all of Phobos, or just std.utf for some reason?
My immediate goal was simply to get phobos to compile with DIP1000. Over time, I expect a well-implemented save() function should use copy construction. I did write a PR to that effect: https://github.com/dlang/phobos/pull/6905/files Your advice on that would be appreciated.
 If you're going for all of Phobos, you might end up fixing issue 18657 as a
side 
 effect. But removing RefRange.opAssign would still be nice, because it's just 
 weird.
Fixing RefRange.save() using copy construction, and ditching opAssign, would be acceptable.
Mar 28
parent reply ag0aep6g <anonymous example.com> writes:
On 29.03.19 00:30, Walter Bright wrote:
 The problem I ran into was ranges that had const fields would fail, 
 because one cannot assign to a const field. But one can construct a 
 const field. Doing some digging, it became apparent that save() is 
 equivalent to copy construction, and so should be implemented using copy 
 construction.
Is Phobos supposed to support ranges with const fields? Won't that be generality creep? A quick test shows that all five examples from issue 18657 fail for ranges with const fields. With my changes, the first three work, and the other two don't fail in `save` anymore, but still in other places. Supporting const fields will require at least the same amount of busywork as supporting RefRange, because the underlying issue is the same: Can ranges be non-assignable and should Phobos support such ranges? [...]
 Fixing RefRange.save() using copy construction, and ditching opAssign, 
 would be acceptable.
How does RefRange.save need fixing?
Mar 28
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 5:40 PM, ag0aep6g wrote:
 On 29.03.19 00:30, Walter Bright wrote:
 The problem I ran into was ranges that had const fields would fail, because 
 one cannot assign to a const field. But one can construct a const field. Doing 
 some digging, it became apparent that save() is equivalent to copy 
 construction, and so should be implemented using copy construction.
Is Phobos supposed to support ranges with const fields? Won't that be generality creep?
Phobos already has such ranges, such as Repeat. Besides, if you accept the notion that save() performs a construction, not an assignment, and that construction and assignment are fundamentally different, then save() must be implemented using construction. I can't think of a case where this would break existing code.
 A quick test shows that all five examples from issue 18657 fail for ranges
with 
 const fields. With my changes, the first three work, and the other two don't 
 fail in `save` anymore, but still in other places.
 
 Supporting const fields will require at least the same amount of busywork as 
 supporting RefRange, because the underlying issue is the same: Can ranges be 
 non-assignable and should Phobos support such ranges?
Ranges don't need to be assignable. ForwardRanges need to supply a save(), which means copy construction, not assignment.
 Fixing RefRange.save() using copy construction, and ditching opAssign, would 
 be acceptable.
How does RefRange.save need fixing?
I thought you wrote that copy construction would make RefRange work?
Mar 28
parent reply ag0aep6g <anonymous example.com> writes:
On 29.03.19 02:55, Walter Bright wrote:
 Ranges don't need to be assignable. ForwardRanges need to supply a 
 save(), which means copy construction, not assignment.
Issue 18657 is all about RefRange not being assignable (without nasty surprises). Changing Phobos to avoid assignment of ranges will fix the issue. All the mighty talk about "generality creep" and "first principles" is besides the point. You agree with making changes all over Phobos. You just don't like the weird `move` code. That's okay. I agree that fixing the issue with constructors is nicer. [...]
 How does RefRange.save need fixing?
I thought you wrote that copy construction would make RefRange work?
Implementing the `save` methods of higher ranges that way makes them compatible with RefRange, because they no longer call RefRange.opAssign. RefRange.save is fine as it is (w.r.t issue 18657).
Mar 29
parent reply ag0aep6g <anonymous example.com> writes:
On 29.03.19 08:41, ag0aep6g wrote:
 I agree that fixing the issue with constructors is nicer.
PR to change my `move`s to constructor calls: https://github.com/dlang/phobos/pull/6939
Mar 29
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/29/2019 7:28 AM, ag0aep6g wrote:
 https://github.com/dlang/phobos/pull/6939
Thank you
Apr 02
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 4:52 AM, Walter Bright wrote:
 A topical example:
 
 https://github.com/dlang/phobos/pull/6346#issuecomment-477504005
See also the related issue: https://issues.dlang.org/show_bug.cgi?id=18657 Indeed so, thanks. The entire thing with ranges whereby save() does something else than a mere copy has been a mistake and supporting it has been an expensive distraction. The intent of save() was as a mere flag to signal that the range is a forward range and not an input range. It should have been a simpler policy - e.g. ranges that advertise being forward ranges would define this inside: enum bool isForward = true; Ranges should have one of two categories: * for input ranges, all copies share and advance the same underlying position; * for forward ranges, each copy holds its own position. Anything else is not a range. There's a lot of good stuff in Phobos, and then there's a lot of fat added to support an ill-intended notion of generality.
Mar 28
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 12:12, Andrei Alexandrescu wrote:
 See also the related issue: https://issues.dlang.org/show_bug.cgi?id=18657
 
 Indeed so, thanks. The entire thing with ranges whereby save() does 
 something else than a mere copy has been a mistake and supporting it has 
 been an expensive distraction.
`save` is a red herring with regards to issue 18657. The problem isn't RefRange.save, it's RefRange.opAssign. The issue manifests often in implementations of `save`, but that's just because they *assign* the result of `save` over an existing range. Note that there's no `save` call in issue 18657's third example (`choose`). The question is whether ranges are allowed to implement opAssign like RefRange does. If so, all of Phobos needs fixing to avoid calling opAssign. If not, RefRange.opAssign should be removed.
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 7:45 AM, ag0aep6g wrote:
 On 28.03.19 12:12, Andrei Alexandrescu wrote:
 See also the related issue: 
 https://issues.dlang.org/show_bug.cgi?id=18657

 Indeed so, thanks. The entire thing with ranges whereby save() does 
 something else than a mere copy has been a mistake and supporting it 
 has been an expensive distraction.
`save` is a red herring with regards to issue 18657. The problem isn't RefRange.save, it's RefRange.opAssign.
The problem is RefRange.
Mar 28
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 12:51, Andrei Alexandrescu wrote:
 The problem is RefRange.
too terse
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 7:59 AM, ag0aep6g wrote:
 On 28.03.19 12:51, Andrei Alexandrescu wrote:
 The problem is RefRange.
too terse
Reasoning from first principles would go as follows. Context: algorithms are expressed naturally in terms of ranges, and composite (higher-order) ranges are built and work with ranges. Event: RefRange comes along, exposes an oddity ("save(), copying, and assignment could do slightly different things"), breaks a bunch of these structures, and requires arcane changes. The conclusion is not to operate such changes everywhere (i.e. reason by analogy). The right conclusion is that save() is unnecessarily general and underspecified.
Mar 28
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 8:37 AM, Andrei Alexandrescu wrote:
 On 3/28/19 7:59 AM, ag0aep6g wrote:
 On 28.03.19 12:51, Andrei Alexandrescu wrote:
 The problem is RefRange.
too terse
Reasoning from first principles would go as follows. Context: algorithms are expressed naturally in terms of ranges, and composite (higher-order) ranges are built and work with ranges. Event: RefRange comes along, exposes an oddity ("save(), copying, and assignment could do slightly different things"), breaks a bunch of these structures, and requires arcane changes. The conclusion is not to operate such changes everywhere (i.e. reason by analogy). The right conclusion is that save() is unnecessarily general and underspecified.
Similar issues: "People may attempt to use Phobos algorithms with ranges of enums based on character types, or other subtypes of character types." Reasoning by analogy: those should work, too. Leads to generality creep. Reasoning from first principles: UTF8 is predicated on one byte type, no real gain to be made. "There are several ways of iterating strings, e.g. by code point, by code unit, or by grapheme." Reasoning by analogy: strings should be ranges, hence we need to choose one default mode of iteration. Reasoning from first principles: strings are wholes, not ranges, and iteration is chosen as needed. "There are UTF8, UTF16, and UTF32 encodings." Reasoning by analogy: they should all work the same. Reasoning from first principles: what's really needed here? Realistically nobody uses UTF32, UTF16 is an evolutionary dead-end still alive mostly as a mild annoyance on Windows systems, and then everybody and their cat is using UTF8. It follows we should consolidate on one encapsulated string type that is NOT an array (because arrays are ranges) and that offers iteration modes on demand. All that gobbledygook with e.g. file and path functions supporting lazy ranges of characters should go. It's an unnecessary and expensive to maintain distraction.
Mar 28
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 13:37, Andrei Alexandrescu wrote:
 The conclusion is not to operate such changes everywhere (i.e. reason by 
 analogy). The right conclusion is that save() is unnecessarily general 
 and underspecified.
If you want to resolve the issue by amending the definition of ranges, that's very fine with me. I'd much prefer that solution over going through all of Phobos. But if you focus on `save`, you're missing the point of issue 18657, which is opAssign. A range can have a misbehaving opAssign without being a forward range.
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 9:04 AM, ag0aep6g wrote:
 On 28.03.19 13:37, Andrei Alexandrescu wrote:
 The conclusion is not to operate such changes everywhere (i.e. reason 
 by analogy). The right conclusion is that save() is unnecessarily 
 general and underspecified.
If you want to resolve the issue by amending the definition of ranges, that's very fine with me. I'd much prefer that solution over going through all of Phobos. But if you focus on `save`, you're missing the point of issue 18657, which is opAssign. A range can have a misbehaving opAssign without being a forward range.
Then some ranges are not meant to be assignable.
Mar 28
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 14:05, Andrei Alexandrescu wrote:
 Then some ranges are not meant to be assignable.
Should Phobos be compatible with those ranges?
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 9:16 AM, ag0aep6g wrote:
 On 28.03.19 14:05, Andrei Alexandrescu wrote:
 Then some ranges are not meant to be assignable.
Should Phobos be compatible with those ranges?
A variety of algorithm and data structures in Phobos are relying on assignment. Presumably a good part of them can be converted with ease to use single assignment, but not all. The overall message is we got bogged down on the "wrong" side of generality - cross-cutting and nonscalable code additions to support unprincipled and low-impact corner cases. Part of that is we've been cagey about defining copy and assignment semantics of ranges in a simple and comprehensive manner. It seems to me going with these is the right thing: * Input ranges are copyable and assignable, and have pointer semantics (all copies refer to the same underlying position, and advancing one advances all others). * Forward ranges are copyable and assignable, but distinct copies refer to distinct positions in the range such that advancing one does not advance the others. * We don't support other semantics.
Mar 28
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 1:04 PM, Andrei Alexandrescu wrote:
 On 3/28/19 9:16 AM, ag0aep6g wrote:
 On 28.03.19 14:05, Andrei Alexandrescu wrote:
 Then some ranges are not meant to be assignable.
Should Phobos be compatible with those ranges?
A variety of algorithm and data structures in Phobos are relying on assignment. Presumably a good part of them can be converted with ease to use single assignment, but not all. The overall message is we got bogged down on the "wrong" side of generality - cross-cutting and nonscalable code additions to support unprincipled and low-impact corner cases. Part of that is we've been cagey about defining copy and assignment semantics of ranges in a simple and comprehensive manner. It seems to me going with these is the right thing: * Input ranges are copyable and assignable, and have pointer semantics (all copies refer to the same underlying position, and advancing one advances all others). * Forward ranges are copyable and assignable, but distinct copies refer to distinct positions in the range such that advancing one does not advance the others. * We don't support other semantics.
Forget to add - no more save(). We just use some sort of flag and simple introspection.
Mar 28
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 3/28/19 1:05 PM, Andrei Alexandrescu wrote:
 Part of that is we've been cagey about defining copy and assignment 
 semantics of ranges in a simple and comprehensive manner. It seems to 
 me going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics 
 (all copies refer to the same underlying position, and advancing one 
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies 
 refer to distinct positions in the range such that advancing one does 
 not advance the others.

 * We don't support other semantics.
Forget to add - no more save(). We just use some sort of flag and simple introspection.
I say "Hooray!" to that! I've long felt that 'save()' seemed a rather strange duplication of assignment/copying (or perhaps more accurately, as Walter pointed out, copy construction). That seemed ugly enough in and of itself, but to make matters much worse, this in turn left the semantics for passing and assigning ranges underdefined and, in effect, needed to treated by algorithms as undefined behavior. And THAT meant that nearly any occurrence of failing to pass ranges by ref (surprisingly common in phobos, at least at the time I noticed this) were much guaranteed to be a bug factory (In my experience, this was especially problematic when working with output ranges.)
Apr 04
prev sibling next sibling parent reply =?UTF-8?B?THXDrXM=?= Marques <luis luismarques.eu> writes:
On Thursday, 28 March 2019 at 17:04:58 UTC, Andrei Alexandrescu 
wrote:
 The overall message is we got bogged down on the "wrong" side 
 of generality - cross-cutting and nonscalable code additions to 
 support unprincipled and low-impact corner cases.
Having a document that formally specifies what a range is and what its properties are might have helped with that and a lot more. It would still help with all of the issues yet to come.
Mar 28
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 1:33 PM, Luís Marques wrote:
 On Thursday, 28 March 2019 at 17:04:58 UTC, Andrei Alexandrescu wrote:
 The overall message is we got bogged down on the "wrong" side of 
 generality - cross-cutting and nonscalable code additions to support 
 unprincipled and low-impact corner cases.
Having a document that formally specifies what a range is and what its properties are might have helped with that and a lot more. It would still help with all of the issues yet to come.
Affirmative. I'm thinking of putting together a DIP for that.
Mar 28
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 05:33:33PM +0000, Lus Marques via Digitalmars-d wrote:
 On Thursday, 28 March 2019 at 17:04:58 UTC, Andrei Alexandrescu wrote:
 The overall message is we got bogged down on the "wrong" side of
 generality - cross-cutting and nonscalable code additions to support
 unprincipled and low-impact corner cases.
Having a document that formally specifies what a range is and what its properties are might have helped with that and a lot more. It would still help with all of the issues yet to come.
Yes, the range API needs a formal, precise spec that leaves no holes. Currently, the semantics of .save is one such issue. Another issue is what we used to call "transience": is the return value of .front required to persist past the next invocation of popFront? std.stdio.File.byLine is a big example that does *not* respect this, and as a result several Phobos algorithms cannot be used with it without incurring buggy behaviour. OTOH, supporting transience means code like the following won't work: auto e = r.front; r.popFront; if (r.front == e) ... because e becomes invalid after .popFront. Last time I checked, several Phobos algorithms freely use this kind of construct (saving the value of .front and checking it later), which means it doesn't work with transient ranges. We need a clear decision as to whether transient ranges are supported. If not, std.stdio.File.byLine needs to be deprecated. Or if they should be supported, then the range spec should clearly state that it is illegal to save the value of .front and expect it to remain valid after .popFront is called. (Personally, I lean towards the latter option, but either way, this needs to be stated clearly and explicitly in the range spec.) T -- Nearly all men can stand adversity, but if you want to test a man's character, give him power. -- Abraham Lincoln
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 1:49 PM, H. S. Teoh wrote:
 On Thu, Mar 28, 2019 at 05:33:33PM +0000, Luís Marques via Digitalmars-d
wrote:
 On Thursday, 28 March 2019 at 17:04:58 UTC, Andrei Alexandrescu wrote:
 The overall message is we got bogged down on the "wrong" side of
 generality - cross-cutting and nonscalable code additions to support
 unprincipled and low-impact corner cases.
Having a document that formally specifies what a range is and what its properties are might have helped with that and a lot more. It would still help with all of the issues yet to come.
Yes, the range API needs a formal, precise spec that leaves no holes. Currently, the semantics of .save is one such issue. Another issue is what we used to call "transience": is the return value of .front required to persist past the next invocation of popFront? std.stdio.File.byLine is a big example that does *not* respect this, and as a result several Phobos algorithms cannot be used with it without incurring buggy behaviour.
Yah, for such we need I think a more primitive notion of UnbufferedRange. An unbuffered range of T has only one API: bool fetchNext(ref T); Note that the user provides all state, which is interesting. The primitives fills the object and moves to the next one. Returns true on success. Returns persistently false at end of range even if called multiple times. An optional interface (for efficiency's sake) would be: size_t fetchNextN(scope T[]); which fills a full array and returns the number of elements filled. These are trivially implementable from the outside for arrays etc. Then we'd have input ranges, which have a buffer of at least one element, i.e. today's input ranges. Input ranges that are not forward ranges are liable to reuse their buffers, so after a call to front(), a call to popFront() may overwrite the current front. This is because by construction input ranges that are not forward ranges do not iterate objects in memory, but instead they transfer data from somewhere else into memory, chunkwise. Forward ranges and above are guaranteed to walk real objects in memory and as such provide better guarantees on the persistence of their contents.
 OTOH, supporting transience means code like
 the following won't work:
 
 	auto e = r.front;
 	r.popFront;
 	if (r.front == e) ...
 
 because e becomes invalid after .popFront.  Last time I checked, several
 Phobos algorithms freely use this kind of construct (saving the value of
 .front and checking it later), which means it doesn't work with
 transient ranges.
 
 We need a clear decision as to whether transient ranges are supported.
 If not, std.stdio.File.byLine needs to be deprecated. Or if they should
 be supported, then the range spec should clearly state that it is
 illegal to save the value of .front and expect it to remain valid after
 .popFront is called. (Personally, I lean towards the latter option, but
 either way, this needs to be stated clearly and explicitly in the range
 spec.)
If we support any notion of input range, what follows is a sequence of forced moves: we must support iteration of objects that are not in memory, hence we need to support transiency of data.
Mar 28
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 3/28/19 2:10 PM, Andrei Alexandrescu wrote:
 
 Yah, for such we need I think a more primitive notion of 
 UnbufferedRange. An unbuffered range of T has only one API:
 
 bool fetchNext(ref T);
 
 Note that the user provides all state, which is interesting. The 
 primitives fills the object and moves to the next one. Returns true on 
 success. Returns persistently false at end of range even if called 
 multiple times. An optional interface (for efficiency's sake) would be:
 
 size_t fetchNextN(scope T[]);
 
 which fills a full array and returns the number of elements filled.
Something like that is definitely needed. I remember needing to augment the existing range concept with something like that when I was dealing with crypto hashes. And obviously it'd be very important for I/O steams. Having such thing standardized in phobos would be great. Steve's input on this may be very important, given his work in iopipe.
 Then we'd have input ranges, which have a buffer of at least one 
 element, i.e. today's input ranges. Input ranges that are not forward 
 ranges are liable to reuse their buffers, so after a call to front(), a 
 call to popFront() may overwrite the current front. This is because by 
 construction input ranges that are not forward ranges do not iterate 
 objects in memory, but instead they transfer data from somewhere else 
 into memory, chunkwise.
This sounds reasonable, but I would REALLY like if we could come up with some way to actually *prevent* the user from accessing a stored .front after the next call to .popFront instead of relying on "programming by convention". Because this seems to run in complete opposition to D philosophy. A static check would be fantastic if possible, or at least injecting an optional run-time check analogous to bounds checking. Not sure how feasible either would be, though.
Apr 04
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 Part of that is we've been cagey about defining copy and assignment
 semantics of ranges in a simple and comprehensive manner. It seems to
 me going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer semantics
 (all copies refer to the same underlying position, and advancing one
 advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one does
 not advance the others.
 
 * We don't support other semantics.
What about classes that wrap ranges? E.g., std.ranges.interfaces. Since classes are reference types, this would mean it's impossible to use those interfaces with forward ranges or above, which is a show-stopping limitation (e.g., if you need to return two different range types selected at runtime -- the current solution is to wrap them in the appropriate class using std.ranges.interfaces). T -- Time flies like an arrow. Fruit flies like a banana.
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 1:42 PM, H. S. Teoh wrote:
 On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]
 Part of that is we've been cagey about defining copy and assignment
 semantics of ranges in a simple and comprehensive manner. It seems to
 me going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics
 (all copies refer to the same underlying position, and advancing one
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one does
 not advance the others.

 * We don't support other semantics.
What about classes that wrap ranges?
Those must go. Supporting classes and structs within the same framework is a costly low-yield endeavor.
 E.g., std.ranges.interfaces.
One could pin-point the introduction of that as a clear point when we veered from the nice highway into the cornfields.
 Since
 classes are reference types, this would mean it's impossible to use
 those interfaces with forward ranges or above, which is a show-stopping
 limitation (e.g., if you need to return two different range types
 selected at runtime -- the current solution is to wrap them in the
 appropriate class using std.ranges.interfaces).
The obvious solution is to use structs that wrap pointers or class references. We don't preclude virtuals and dynamic behavior.
Mar 28
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 01:56:08PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 3/28/19 1:42 PM, H. S. Teoh wrote:
 On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]
 Part of that is we've been cagey about defining copy and
 assignment semantics of ranges in a simple and comprehensive
 manner. It seems to me going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer
 semantics (all copies refer to the same underlying position, and
 advancing one advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one
 does not advance the others.
 
 * We don't support other semantics.
What about classes that wrap ranges?
Those must go. Supporting classes and structs within the same framework is a costly low-yield endeavor.
So this will be a major breaking change, since there's a significant amount of user code that currently relies on this functionality. Which means we're seriously considering Phobos v2 here? I'd support that. It's time to reverse all those decisions that in hindsight were poor ones. Like autodecoding.
 E.g., std.ranges.interfaces.
[...]
 Since classes are reference types, this would mean it's impossible
 to use those interfaces with forward ranges or above, which is a
 show-stopping limitation (e.g., if you need to return two different
 range types selected at runtime -- the current solution is to wrap
 them in the appropriate class using std.ranges.interfaces).
The obvious solution is to use structs that wrap pointers or class references. We don't preclude virtuals and dynamic behavior.
IOW, all ranges will be required to be structs? And if we need dynamic behaviour it will have to be a struct that overrides opAssign to copy the state of the underlying range (for forward ranges)? I suppose that could work. There will be friction, though, if you need to access the underlying class hierarchy (e.g., the struct would have to export some sort of API for casting the wrapped class reference to some interface or base class that you might want to pass into a function expecting such). T -- Life is too short to run proprietary software. -- Bdale Garbee
Mar 28
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 3/28/19 2:19 PM, H. S. Teoh wrote:
 
 IOW, all ranges will be required to be structs?  And if we need dynamic
 behaviour it will have to be a struct that overrides opAssign to copy
 the state of the underlying range (for forward ranges)?
 
IIUC, you'd want to use a copy constructor for that, not opAssign.
Apr 04
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 18:04, Andrei Alexandrescu wrote:
 On 3/28/19 9:16 AM, ag0aep6g wrote:
 On 28.03.19 14:05, Andrei Alexandrescu wrote:
 Then some ranges are not meant to be assignable.
Should Phobos be compatible with those ranges?
[...]
 It seems to me 
 going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer semantics 
 (all copies refer to the same underlying position, and advancing one 
 advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies refer 
 to distinct positions in the range such that advancing one does not 
 advance the others.
 
 * We don't support other semantics.
Are you going to actually change the range definitions in that way? Are you saying that I should wait until then with fixing range issues in Phobos? I'd expect those new definitions to be years away from becoming reality. It doesn't seem wise to leave bugs open today just because you have vague plans for fundamental changes in the distant future.
Mar 28
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:12 PM, ag0aep6g wrote:
 On 28.03.19 18:04, Andrei Alexandrescu wrote:
 On 3/28/19 9:16 AM, ag0aep6g wrote:
 On 28.03.19 14:05, Andrei Alexandrescu wrote:
 Then some ranges are not meant to be assignable.
Should Phobos be compatible with those ranges?
[...]
 It seems to me going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics 
 (all copies refer to the same underlying position, and advancing one 
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies 
 refer to distinct positions in the range such that advancing one does 
 not advance the others.

 * We don't support other semantics.
Are you going to actually change the range definitions in that way? Are you saying that I should wait until then with fixing range issues in Phobos? I'd expect those new definitions to be years away from becoming reality. It doesn't seem wise to leave bugs open today just because you have vague plans for fundamental changes in the distant future.
We've been worrying too much about changing things. On any given issue, even those that widely agreed to be clowny, two factions emerge. One advocates change and the other advocates against code breakage. Interestingly the factions are different depending on the issue, i.e. a given person may be in favor of change in one matter and in favor of backwards compatibility in another. At any rate, the result is years-long stalemate and trench warfare on the slightest problems. We should do what C, C++, Java, C# and many other languages do - leave past mistakes in maintenance mode and move on with new stuff. It's like blockchain - once done, it can't be undone. You do a new one. C has about three ways of converting numbers to strings and back. C++ added a couple, poorly designed. Then they added a better one. And don't get me started about I/O (two ways of doing it in C, and another one in C++ that nobody in their right mind would use). Same with Java pre- and post-generics interfaces for comparing objects. It's just that if something is bad they define a better one and move on. We have a language with virtually no global scope - a versioning engineer's dream. Yet we are coy to tap into that power. Instead, we wring hands over the breakages that would happen if we change std.json. The time is ripe for std.v2, which would do the right things and support simple migration.
Mar 28
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 19:30, Andrei Alexandrescu wrote:
 We should do what C, C++, Java, C# and many other languages do - leave 
 past mistakes in maintenance mode and move on with new stuff. It's like 
 blockchain - once done, it can't be undone. You do a new one. C has 
 about three ways of converting numbers to strings and back. C++ added a 
 couple, poorly designed. Then they added a better one. And don't get me 
 started about I/O (two ways of doing it in C, and another one in C++ 
 that nobody in their right mind would use). Same with Java pre- and 
 post-generics interfaces for comparing objects. It's just that if 
 something is bad they define a better one and move on.
 
 We have a language with virtually no global scope - a versioning 
 engineer's dream. Yet we are coy to tap into that power. Instead, we 
 wring hands over the breakages that would happen if we change std.json. 
 The time is ripe for std.v2, which would do the right things and support 
 simple migration.
I'm not against that at all, but I'm not convinced that you can make it happen quickly. Remember that this sub-thread started with a mention of my pull request that was open for a year, sitting idle most of the time. So until new Phobos materializes some more, I'd appreciate if we could fix issue 18657 one way or the other in old Phobos.
Mar 28
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:48 PM, ag0aep6g wrote:
 On 28.03.19 19:30, Andrei Alexandrescu wrote:
 We should do what C, C++, Java, C# and many other languages do - leave 
 past mistakes in maintenance mode and move on with new stuff. It's 
 like blockchain - once done, it can't be undone. You do a new one. C 
 has about three ways of converting numbers to strings and back. C++ 
 added a couple, poorly designed. Then they added a better one. And 
 don't get me started about I/O (two ways of doing it in C, and another 
 one in C++ that nobody in their right mind would use). Same with Java 
 pre- and post-generics interfaces for comparing objects. It's just 
 that if something is bad they define a better one and move on.

 We have a language with virtually no global scope - a versioning 
 engineer's dream. Yet we are coy to tap into that power. Instead, we 
 wring hands over the breakages that would happen if we change 
 std.json. The time is ripe for std.v2, which would do the right things 
 and support simple migration.
I'm not against that at all, but I'm not convinced that you can make it happen quickly.
Three solid engineers would make this a quarter's worth of work. I don't think we have them. Remember that this sub-thread started with a mention of
 my pull request that was open for a year, sitting idle most of the time.
 
 So until new Phobos materializes some more, I'd appreciate if we could 
 fix issue 18657 one way or the other in old Phobos.
For that I think we should bite the bullet and let RefRange go.
Mar 28
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 19:51, Andrei Alexandrescu wrote:
 Three solid engineers would make this a quarter's worth of work. I don't 
 think we have them.
[...]
 For that I think we should bite the bullet and let RefRange go.
So new Phobos won't happen for a long while, and old Phobos gets abandoned instead of fixed. Can't say that I'm hyped.
Mar 28
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:07 PM, ag0aep6g wrote:
 On 28.03.19 19:51, Andrei Alexandrescu wrote:
 Three solid engineers would make this a quarter's worth of work. I 
 don't think we have them.
[...]
 For that I think we should bite the bullet and let RefRange go.
So new Phobos won't happen for a long while, and old Phobos gets abandoned instead of fixed. Can't say that I'm hyped.
Deprecating RefRange is not abandon.
Mar 28
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 11:48 AM, ag0aep6g wrote:
 So until new Phobos materializes some more, I'd appreciate if we could fix
issue 
 18657 one way or the other in old Phobos.
1. deprecate RefRange 2. resolve https://issues.dlang.org/show_bug.cgi?id=18657 as "wont fix" 3. revert the recent changes to support RefRange, do it now before it gets released
Mar 28
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 02:30:24PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 We've been worrying too much about changing things.
Wow. That's a pretty drastic (and refreshing!) change of stance from the past years of insisting that we should do our utmost to maintain backward compatibility. I, for one, welcome this change. [...]
 We have a language with virtually no global scope - a versioning
 engineer's dream. Yet we are coy to tap into that power. Instead, we
 wring hands over the breakages that would happen if we change
 std.json. The time is ripe for std.v2, which would do the right things
 and support simple migration.
[...] Yes, I welcome the idea of std.v2. And we should provide reasonable migration paths to std.v2, so that after X number of years, we can deprecate the current stuff, and move std.v2.* to std.*. (Or maybe leave it as std.v2.*, in case there will ever be a need for std.v3.*. I hope not, but hindsight is always 20/20 so it seems inevitable going forward.) T -- IBM = I Blame Microsoft
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:56 PM, H. S. Teoh wrote:
 On Thu, Mar 28, 2019 at 02:30:24PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]
 We've been worrying too much about changing things.
Wow. That's a pretty drastic (and refreshing!) change of stance from the past years of insisting that we should do our utmost to maintain backward compatibility. I, for one, welcome this change.
Thanks. The irony of it all is that adding new stuff and leaving the old stuff in maintenance mode is simultaneously the rock-solid compatibility stance and the ultimate road to progress. It does happen, and has happened several times in my lifetime, that an entire community is missing an important point. It seems we missed the point that change is not what we need... it's addition.
Mar 28
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 03:09:53PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 3/28/19 2:56 PM, H. S. Teoh wrote:
 On Thu, Mar 28, 2019 at 02:30:24PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]
 We've been worrying too much about changing things.
Wow. That's a pretty drastic (and refreshing!) change of stance from the past years of insisting that we should do our utmost to maintain backward compatibility. I, for one, welcome this change.
Thanks. The irony of it all is that adding new stuff and leaving the old stuff in maintenance mode is simultaneously the rock-solid compatibility stance and the ultimate road to progress.
But if that's the case, the stuff in maintenance mode should at least receive maintenance fixes of the kind being proposed for RefRange? Let's agree that RefRange was a bad idea and all that. But there are users using it that need fixes for the issues they encountered. Even if said fixes are perhaps also just as bad of an idea as RefRange itself was to begin with, for the sake of maintenence we'd still be obligated to provide at least workarounds for current bugs?
 It does happen, and has happened several times in my lifetime, that an
 entire community is missing an important point. It seems we missed the
 point that change is not what we need... it's addition.
If you're thinking about dropping druntime altogether, that's no mere addition, that's a MAJOR change. One that I'd welcome, to be frank, but it's not going to happen without major breakage of existing codebases. Are you sure you've thought this through? T -- The most powerful one-line C program: #include "/dev/tty" -- IOCCC
Mar 28
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:19 PM, H. S. Teoh wrote:
 On Thu, Mar 28, 2019 at 03:09:53PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 On 3/28/19 2:56 PM, H. S. Teoh wrote:
 On Thu, Mar 28, 2019 at 02:30:24PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]
 We've been worrying too much about changing things.
Wow. That's a pretty drastic (and refreshing!) change of stance from the past years of insisting that we should do our utmost to maintain backward compatibility. I, for one, welcome this change.
Thanks. The irony of it all is that adding new stuff and leaving the old stuff in maintenance mode is simultaneously the rock-solid compatibility stance and the ultimate road to progress.
But if that's the case, the stuff in maintenance mode should at least receive maintenance fixes of the kind being proposed for RefRange?
Changing the entire standard library to accommodate wackiness in RefRange is a poor executive decision regardless. The recent changes must be undone.
Mar 28
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 20:24, Andrei Alexandrescu wrote:
 Changing the entire standard library to accommodate wackiness in 
 RefRange is a poor executive decision regardless. The recent changes 
 must be undone.
No one is arguing in favor of doing what I've in my PRs on the issue. Not even me. I've started that endeavor, because the only tangible response I got when I asked what do do was from Jonathan M Davis who said: "it could be argued that generic, range-based functions simply shouldn't ever be assigning one range to another."[1] If you prefer and approve removing RefRange.opAssign instead, I'm all for that. Also note that Walter himself has done similar pull requests: https://github.com/dlang/phobos/pull/6900 https://github.com/dlang/phobos/pull/6903 He uses constructors instead of "wonky" `move`, but the effect with regards to issue 18657 is the same: RefRange.opAssign is not called. If that's an acceptable way to make Phobos compatible with RefRange, that's also fine with me. [1] https://forum.dlang.org/post/mailman.1557.1521939012.3374.digitalmars-d puremagic.com
Mar 28
next sibling parent ag0aep6g <anonymous example.com> writes:
On 28.03.19 20:43, ag0aep6g wrote:
 On 28.03.19 20:24, Andrei Alexandrescu wrote:
 Changing the entire standard library to accommodate wackiness in 
 RefRange is a poor executive decision regardless. The recent changes 
 must be undone.
[...]
 He uses constructors instead of "wonky" `move`
correction: "wacky", not "wonky" ;)
Mar 28
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:43 PM, ag0aep6g wrote:
 On 28.03.19 20:24, Andrei Alexandrescu wrote:
 Changing the entire standard library to accommodate wackiness in 
 RefRange is a poor executive decision regardless. The recent changes 
 must be undone.
No one is arguing in favor of doing what I've in my PRs on the issue. Not even me.
Yes, I saw your pain seeping through the lines. The valiant effort hasn't gone unnoticed and is appreciated.
 I've started that endeavor, because the only tangible response I got 
 when I asked what do do was from Jonathan M Davis who said: "it could be 
 argued that generic, range-based functions simply shouldn't ever be 
 assigning one range to another."[1]
This is a sign things have gotten off the rail. Assignment should be accessible and have obvious semantics.
 If you prefer and approve removing RefRange.opAssign instead, I'm all 
 for that.
 
 Also note that Walter himself has done similar pull requests:
 
 https://github.com/dlang/phobos/pull/6900
 https://github.com/dlang/phobos/pull/6903
 
 He uses constructors instead of "wonky" `move`, but the effect with 
 regards to issue 18657 is the same: RefRange.opAssign is not called. If 
 that's an acceptable way to make Phobos compatible with RefRange, that's 
 also fine with me.
 
 
 [1] 
 https://forum.dlang.org/post/mailman.1557.1521939012.3374.digital
ars-d puremagic.com 
RefRange is a cancer. Changing all data structures and algorithms to support it is the cancer going into metastasis. We must excise the cancerous tissue.
Mar 28
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 21:05, Andrei Alexandrescu wrote:
 This is a sign things have gotten off the rail. Assignment should be 
 accessible and have obvious semantics.
Let's get rid of RefRange.opAssign then. Then assignment of RefRange has obvious semantics. RefRange.opAssign is not needed for RefRange to be useful. [...]
 RefRange is a cancer.
 
 Changing all data structures and algorithms to support it is the cancer 
 going into metastasis.
Again, no one here is arguing for "changing all data structures and algorithms".
 We must excise the cancerous tissue.
We can cut out the bad parts, but leave the useful core of RefRange intact, can't we? The useful core is: a range that wraps a pointer to another range in order to achieve reference semantics. Often, a pointer to a range is already a range, but not always. E.g., `int[]*` is not a range.
Mar 28
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 3:13:17 PM MDT ag0aep6g via Digitalmars-d wrote:
 On 28.03.19 21:05, Andrei Alexandrescu wrote:
 We must excise the cancerous tissue.
We can cut out the bad parts, but leave the useful core of RefRange intact, can't we? The useful core is: a range that wraps a pointer to another range in order to achieve reference semantics. Often, a pointer to a range is already a range, but not always. E.g., `int[]*` is not a range.
The problem is that reference semantics and forward ranges do not go well together at all. As things stand, range-based code is supposed to work with forward ranges that are reference types, and we have save to try and deal with the fact that some ranges are value types, some are reference types, and some are pseudo-reference types, but it really doesn't work very well. And algorithms that require save fundamentally need to copy the range at some point, pretty much destroying the usefulness of something like RefRange. Basic input ranges are fundamentally reference types or pseudo-reference types, because if they could have value semantics, then they could be forward ranges. Forward ranges fundamentally have to be copyable with value semantics to even work properly. We've just got this messed up situation where we're using save to do the copy instead of postblit or copy constuctors, because we're trying to handle types that aren't value types as if they were. RefRange is trying to force reference semantics, which then fundamentally doesn't fit with forward ranges. It sort of half fits right now, because of save, but as with forward ranges that are reference types, it causes problems, and code frequently isn't going to work with it without extra work. As such, I really don't think that RefRange makes sense fundamentally. It _can_ work in corner cases, and years ago, I had a piece of code where it was really useful, which is why I wrote it. But it was a mistake. It's trying to fight against how forward ranges work, and while it might occasionally be useful, in general, it's going to cause problems. And if we actually go forward with requiring that basic input ranges have reference semantics and that forward ranges have value semantics like Andrei is proposing, then RefRange _really_ doesn't work. - Jonathan M Davis
Mar 30
parent reply ag0aep6g <anonymous example.com> writes:
On 30.03.19 19:32, Jonathan M Davis wrote:
 RefRange is trying to force reference semantics, which then fundamentally
 doesn't fit with forward ranges. It sort of half fits right now, because of
 save, but as with forward ranges that are reference types, it causes
 problems, and code frequently isn't going to work with it without extra
 work.
 
 As such, I really don't think that RefRange makes sense fundamentally.
You haven't spelled it out, but I suppose you also think that classes don't make sense as forward ranges. And Andrei has stated something to the effect that they wouldn't be supported in his redesign. That is fine for a range redesign. But with the definitions we have right now, classes can be forward ranges, and Phobos should work with them. And if Phobos works with classes, it can work with RefRange (if we cut out opAssign). If RefRange really is fundamentally at odds with current ranges, I'd be interested in an example where it causes trouble that isn't due to opAssign (or opSlice() which also shouldn't be there). Maybe I just fail to see the problem. One issue I'm aware of is that RefRange does a GC allocation on `save`. That isn't pretty, but I wouldn't call it fatal. I've said before that "we can cut out the bad parts" of RefRange, referring to opAssign, but I think I was wrong. Deprecating assignment would be so limiting that we might as well deprecate the whole thing. Then again, Walter has made (limited) efforts to support ranges that are non-assignable due to const fields. Nobody acknowledges that those exact changes also work to accommodate RefRange, but they do.
Mar 30
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/30/2019 12:38 PM, ag0aep6g wrote:
 Then again, Walter has made (limited) efforts to support ranges that are 
 non-assignable due to const fields. Nobody acknowledges that those exact
changes 
 also work to accommodate RefRange, but they do.
I'm fine with adjusting `save()` implementations to use construction rather than assignment semantics, because that's the way `save()` is defined to behave (although it wasn't clearly spelled out).
Mar 30
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/19 8:05 PM, Walter Bright wrote:
 On 3/30/2019 12:38 PM, ag0aep6g wrote:
 Then again, Walter has made (limited) efforts to support ranges that 
 are non-assignable due to const fields. Nobody acknowledges that those 
 exact changes also work to accommodate RefRange, but they do.
I'm fine with adjusting `save()` implementations to use construction rather than assignment semantics, because that's the way `save()` is defined to behave (although it wasn't clearly spelled out).
Great, so ag0aep6g you're good to go with your PR. I'd approved it yesterday.
Mar 30
parent reply ag0aep6g <anonymous example.com> writes:
On 31.03.19 01:14, Andrei Alexandrescu wrote:
 Great, so ag0aep6g you're good to go with your PR. I'd approved it 
 yesterday.
Thanks, guys. Glad we managed to work this out. Just to be clear, I can't merge PRs. It sounds like you might expect me to do that.
Mar 30
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/19 8:19 PM, ag0aep6g wrote:
 On 31.03.19 01:14, Andrei Alexandrescu wrote:
 Great, so ag0aep6g you're good to go with your PR. I'd approved it 
 yesterday.
Thanks, guys. Glad we managed to work this out. Just to be clear, I can't merge PRs. It sounds like you might expect me to do that.
Did so. Thanks for the work.
Mar 30
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, March 30, 2019 1:38:46 PM MDT ag0aep6g via Digitalmars-d wrote:
 On 30.03.19 19:32, Jonathan M Davis wrote:
 RefRange is trying to force reference semantics, which then
 fundamentally
 doesn't fit with forward ranges. It sort of half fits right now, because
 of save, but as with forward ranges that are reference types, it causes
 problems, and code frequently isn't going to work with it without extra
 work.

 As such, I really don't think that RefRange makes sense fundamentally.
You haven't spelled it out, but I suppose you also think that classes don't make sense as forward ranges. And Andrei has stated something to the effect that they wouldn't be supported in his redesign. That is fine for a range redesign. But with the definitions we have right now, classes can be forward ranges, and Phobos should work with them. And if Phobos works with classes, it can work with RefRange (if we cut out opAssign). If RefRange really is fundamentally at odds with current ranges, I'd be interested in an example where it causes trouble that isn't due to opAssign (or opSlice() which also shouldn't be there). Maybe I just fail to see the problem. One issue I'm aware of is that RefRange does a GC allocation on `save`. That isn't pretty, but I wouldn't call it fatal. I've said before that "we can cut out the bad parts" of RefRange, referring to opAssign, but I think I was wrong. Deprecating assignment would be so limiting that we might as well deprecate the whole thing. Then again, Walter has made (limited) efforts to support ranges that are non-assignable due to const fields. Nobody acknowledges that those exact changes also work to accommodate RefRange, but they do.
Fundamentally, forward ranges are essentially value types. They've just had their copy operation moved to save in order to accommodate classes. They have to be copyable to work (even if that copying is done via save), and any function that requires a forward range _will_ copy it via save, or it wouldn't require a forward range. Once the range is saved, RefRange is pretty useless, because the results of what are done to the copy don't affect the original. And to use RefRange, you're basically depending on implementation details of a particular algorithm and when it chooses to call save and what exactly it does with the original before and after calling save - something that is not part of the function's API or any guarantee that it makes. And because the actual semantics of copying a range depend on how it's implemented, algorithms can't rely on whether copying a range will result in an independent copy or not and are free to rearrange when they do or don't copy a range so long as they don't use the original range after it's been copied. They're also free to change when save is called so long as the function's result is the same. It's also assumed that the function can do whatever it pleases with the original so long as it doesn't violate the range API, since in general, you can't use a range again after it's been copied (because doing so would not work the same with all range types). RefRange on the other hand is basically trying to get access to the results of the first part of the algorithm prior to the call save (and possibly some of what's done to the original after the call to save). It's purposefully setting up the code to reuse a range in circumstances where you can't reuse a range in generic code. The code isn't necessarily generic, which does change things, but if the algorithm you're passing RefRange too is generic, then it wasn't written with RefRange in mind, and there will be no expectation that the caller will do anything with the range that was passed in. If anything, the expection is that they won't do anything with it except maybe assign the result of the function to it to reuse the variable. Basically, RefRange is trying to fight how forward ranges work, and while that works in corner cases, in general, it's a mess. And the odds of code that uses RefRange breaking due to changes to algorithms that would otherwise be fine are pretty high, because code using RefRange is depending on the implementation details of any functions that RefRange is passed to. So, while RefRange might be okay in code that's written specifically for use with RefRange, in general, it's a terrible idea. And I really regret that I got into Phobos. The fact that I did just shows how much less I understood about ranges at the time. - Jonathan M Davis
Mar 30
parent ag0aep6g <anonymous example.com> writes:
On 31.03.19 08:53, Jonathan M Davis wrote:
 Once the range is saved, RefRange is
 pretty useless, because the results of what are done to the copy don't
 affect the original.
 
 And to use RefRange, you're basically depending on implementation details of
 a particular algorithm and when it chooses to call save and what exactly it
 does with the original before and after calling save - something that is not
 part of the function's API or any guarantee that it makes. 
[...]
 RefRange on the other hand is basically trying to get access to the results
 of the first part of the algorithm prior to the call save (and possibly some
 of what's done to the original after the call to save).
All that also applies to other ranges that have reference semantics. [...]
 So, while RefRange might be okay in code that's written specifically for use
 with RefRange, in general, it's a terrible idea. And I really regret that I
 got into Phobos. The fact that I did just shows how much less I understood
 about ranges at the time.
RefRange is painted as this big scary monster that ruins everything (Andrei called it "cancer", you call it a "terrible idea"). In reality, it might not be the most useful thing, but it works just as well as other reference-type ranges. Or rather, it would, if it weren't for that pesky opAssign. RefRange.opAssign ruins everything.
Mar 31
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 2:05:41 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:
 On 3/28/19 3:43 PM, ag0aep6g wrote:
 I've started that endeavor, because the only tangible response I got
 when I asked what do do was from Jonathan M Davis who said: "it could be
 argued that generic, range-based functions simply shouldn't ever be
 assigning one range to another."[1]
This is a sign things have gotten off the rail. Assignment should be accessible and have obvious semantics.
That's really the problem. Some operations on ranges are not well defined, and so you can't rely on them in generic code. The prime example is copying (though assignment then gets tied up in that). If a range is a reference type, then iterating the copy iterates the original; if a range is a value type, then iterating the copy doesn't affect the original; and if a range is a pseudo-reference type, then exactly what happens depends on how it's implemented. The end result is that you basically can't use a range after it's been copied (which includes passing it to a function or using it with foreach), because what happens depends on how the range was implemented. I've thought for a while now that how things should really work would be to require that basic input ranges be reference types and that forward ranges be value types (ditching save, requiring copy construction instead, which then requires wrapping classes in structs to be ranges), though that potentially means that calling the same function on basic input ranges and forward ranges gets a bit sticky. The biggest problem though of course is compatibility, but if we're seriously looking at doing a v2 for major stuff in Phobos or Phobos as a whole, then we definitely should be sitting down and working through what we did wrong with ranges and fix this sort of thing. - Jonathan M Davis
Mar 30
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 12:19 PM, H. S. Teoh wrote:
 But if that's the case, the stuff in maintenance mode should at least
 receive maintenance fixes of the kind being proposed for RefRange?
Don't need those fixes because it never worked. Maintenance fixes would be when language changes break the library (such as the library work I did to get DIP1000 to work).
Mar 28
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 12:09 PM, Andrei Alexandrescu wrote:
 It does happen, and has happened several times in my lifetime, that an entire 
 community is missing an important point. It seems we missed the point that 
 change is not what we need... it's addition.
Reminds me of: https://github.com/dlang/undeaD That library turned out to be necessary because some old software of mine used them and there was no reasonable migration path to the newer modules.
Mar 28
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei Alexandrescu 
wrote:

 We've been worrying too much about changing things.
Wow! That is refreshing to read.
 We should do what C, C++, Java, C# and many other languages do 
 - leave past mistakes in maintenance mode and move on with new 
 stuff. It's like blockchain - once done, it can't be undone. 
 You do a new one. C has about three ways of converting numbers 
 to strings and back. C++ added a couple, poorly designed. Then 
 they added a better one. And don't get me started about I/O 
 (two ways of doing it in C, and another one in C++ that nobody 
 in their right mind would use). Same with Java pre- and 
 post-generics interfaces for comparing objects. It's just that 
 if something is bad they define a better one and move on.
Hear, hear. It applies to much outside of software as well. Reading this I'm getting all psyched about D again. Mike
Mar 28
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei Alexandrescu 
wrote:
 On 3/28/19 2:12 PM, ag0aep6g wrote:
 [...]
We've been worrying too much about changing things. On any given issue, even those that widely agreed to be clowny, two factions emerge. One advocates change and the other advocates against code breakage. Interestingly the factions are different depending on the issue, i.e. a given person may be in favor of change in one matter and in favor of backwards compatibility in another. At any rate, the result is years-long stalemate and trench warfare on the slightest problems. [...]
This is a complete 180 from what you were saying a year ago. What happened?
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 10:03 PM, Jonathan Marler wrote:
 On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei Alexandrescu wrote:
 On 3/28/19 2:12 PM, ag0aep6g wrote:
 [...]
We've been worrying too much about changing things. On any given issue, even those that widely agreed to be clowny, two factions emerge. One advocates change and the other advocates against code breakage. Interestingly the factions are different depending on the issue, i.e. a given person may be in favor of change in one matter and in favor of backwards compatibility in another. At any rate, the result is years-long stalemate and trench warfare on the slightest problems. [...]
This is a complete 180 from what you were saying a year ago. What happened?
"I'd rather be right than consistent." -- Winston Churchill
Mar 28
next sibling parent Jonathan Marler <johnnymarler gmail.com> writes:
On Friday, 29 March 2019 at 02:40:43 UTC, Andrei Alexandrescu 
wrote:
 On 3/28/19 10:03 PM, Jonathan Marler wrote:
 On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei 
 Alexandrescu wrote:
 On 3/28/19 2:12 PM, ag0aep6g wrote:
 [...]
We've been worrying too much about changing things. On any given issue, even those that widely agreed to be clowny, two factions emerge. One advocates change and the other advocates against code breakage. Interestingly the factions are different depending on the issue, i.e. a given person may be in favor of change in one matter and in favor of backwards compatibility in another. At any rate, the result is years-long stalemate and trench warfare on the slightest problems. [...]
This is a complete 180 from what you were saying a year ago. What happened?
"I'd rather be right than consistent." -- Winston Churchill
A good rule to live by and be aware of as many tend to stick to their guns rather than admitting they're not perfect. In any case I am very pleased to see this new attitude. I was just genuinely wondering what caused it.
Mar 28
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Friday, 29 March 2019 at 02:40:43 UTC, Andrei Alexandrescu 
wrote:
 On 3/28/19 10:03 PM, Jonathan Marler wrote:
 On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei 
 Alexandrescu wrote:
 On 3/28/19 2:12 PM, ag0aep6g wrote:
 [...]
We've been worrying too much about changing things. On any given issue, even those that widely agreed to be clowny, two factions emerge. One advocates change and the other advocates against code breakage. Interestingly the factions are different depending on the issue, i.e. a given person may be in favor of change in one matter and in favor of backwards compatibility in another. At any rate, the result is years-long stalemate and trench warfare on the slightest problems. [...]
This is a complete 180 from what you were saying a year ago. What happened?
"I'd rather be right than consistent." -- Winston Churchill
Kudos!
Mar 29
prev sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On 29/03/2019 03:40, Andrei Alexandrescu wrote:
 On 3/28/19 10:03 PM, Jonathan Marler wrote:
 On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei Alexandrescu wrote:
 We've been worrying too much about changing things.

 [...]
This is a complete 180 from what you were saying a year ago. What happened?
"I'd rather be right than consistent." -- Winston Churchill
I too applaud this insight. As many may know, I'm helping a company move away from a fossilized language and it has decided to make its business depend on D. Although reacting to deprecations won't always come convenient, distributing the cost of evolution is much more preferable than making the massive investment they are in for now. So I'm sure they appreciate seeing these signs of D staying dynamic, averting sedimentation. Bastiaan.
Mar 29
prev sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 3/28/19 2:30 PM, Andrei Alexandrescu wrote:
 
 We've been worrying too much about changing things.
Yup. *Very* glad to hear this coming from you of all people ;)
 We have a language with virtually no global scope - a versioning 
 engineer's dream. Yet we are coy to tap into that power. 
Yes, precisely! Hear hear!
Apr 04
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 07:12:36PM +0100, ag0aep6g via Digitalmars-d wrote:
 On 28.03.19 18:04, Andrei Alexandrescu wrote:
[...]
 It seems to me going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer
 semantics (all copies refer to the same underlying position, and
 advancing one advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one
 does not advance the others.
 
 * We don't support other semantics.
Are you going to actually change the range definitions in that way? Are you saying that I should wait until then with fixing range issues in Phobos? I'd expect those new definitions to be years away from becoming reality. It doesn't seem wise to leave bugs open today just because you have vague plans for fundamental changes in the distant future.
The only way I can see this happening is if we start a new iteration of Phobos, like std.v2.* where these fundamental changes can be done in a way that isn't totally disruptive to existing code. There would have to be a migration path from the current Phobos so that existing code can gradually move to the new APIs. Implementing these changes directly in the current Phobos is going to break SO much code it will make Walter & Andrei's stance on stabilizing the language completely laughable. (OTOH, I would welcome something like std.v2.*, where we can reverse those decisions that in hindsight were poorly made, like autodecoding. And getting rid of subpar modules like std.xml (and incorporating Jonathan's dxml in its place).) T -- Almost all proofs have bugs, but almost all theorems are true. -- Paul Pedersen
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:30 PM, H. S. Teoh wrote:
 The only way I can see this happening is if we start a new iteration of
 Phobos, like std.v2.*
There'd be carefully carved subsets per capability, such as std.v2.betterc.*, std.v2.nogc.*, std.v2.noexcept.*, std.v2.safe. The capability sets can be combined in alphabetic ordering. E.g. std.v2.noexcept.nogc.* has the subset that uses no gc and no exceptions, but std.v2.nogc.noexcept does not exist. Through the magic of public imports and aliasing there would be no code duplication. Introspection could help, too, a la "define public aliases for all safe definitions in that module". Some improvements to ddoc might be needed to "see" through aliases.
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:41 PM, Andrei Alexandrescu wrote:
 On 3/28/19 2:30 PM, H. S. Teoh wrote:
 The only way I can see this happening is if we start a new iteration of
 Phobos, like std.v2.*
There'd be carefully carved subsets per capability, such as std.v2.betterc.*, std.v2.nogc.*, std.v2.noexcept.*, std.v2.safe. The capability sets can be combined in alphabetic ordering. E.g. std.v2.noexcept.nogc.* has the subset that uses no gc and no exceptions, but std.v2.nogc.noexcept does not exist. Through the magic of public imports and aliasing there would be no code duplication. Introspection could help, too, a la "define public aliases for all safe definitions in that module". Some improvements to ddoc might be needed to "see" through aliases.
Oh, and druntime must go. The whole distinction between the runtime library and the standard library is clowny and has more to do with poorly ported tradition from other languages, than with anything else. We need one standard library that is entirely pay-as-you-go (e.g. empty main() means no library file is ever needed for linking) and that offers an opt-in continuum.
Mar 28
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 02:47:49PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 Oh, and druntime must go.
 
 The whole distinction between the runtime library and the standard
 library is clowny and has more to do with poorly ported tradition from
 other languages, than with anything else.
OK, *this* is going to be an even more drastic change. So what happens with the current compiler bindings to druntime?
 We need one standard library that is entirely pay-as-you-go (e.g.
 empty main() means no library file is ever needed for linking) and
 that offers an opt-in continuum.
That would be a very welcome change. But I'd also have to warn that this may not be as trivial as you appear to think. There's a lot of deep-seated stuff in the compiler that heavily ties it to the current druntime implementation. Replacing all of that is going to be a non-trivial feat. The TypeInfo stuff, for one thing, will take some work to reach your vision of a completely opt-in library. Currently, trying to link without druntime will cause a bunch of errors related to missing TypeInfo's. It *is* clowny, I'll give you that -- the compiler really should not be emitting this stuff (or should be using weak symbols for it) unless it's actually *used* by the user code. Similarly, we should get rid of the implicit Monitor field in classes -- useless deadweight unless the user code needs synchronized classes. Or the GC initialization in rt_init / rt_term. Or, for that matter, the initialization code that packages command-line arguments into a string[] before invoking Dmain. To achieve not linking any library for empty main() the compiler would have to stop emitting calls to rt_init / rt_term unless the program uses the GC. And it'd have to emit main() as `main` rather than `Dmain` if you're going to want to omit the current standard initialization code. (And don't forget the need to invoke module static ctors -- in topological order, which requires some amount of code to get right.) And so on and so forth. IOW, what we're looking at here is nothing short of what essentially boils down to D3. I applaud this direction, but let's not kid ourselves that this is going to happen overnight. T -- Music critic: "That's an imitation fugue!"
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:12 PM, H. S. Teoh wrote:
 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.
Or at all. The entire effort would require three strong like-minded engineers, and I don't know them.
Mar 28
next sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 28 March 2019 at 20:14:19 UTC, Andrei Alexandrescu 
wrote:
 On 3/28/19 3:12 PM, H. S. Teoh wrote:
 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.
Or at all. The entire effort would require three strong like-minded engineers, and I don't know them.
Can't we crowd-found this initiative? Also, if we lay out the rough ground lines and draft, the community could pitch in and it wouldn't necessarily be a one man's mission. Divide et impera!
Mar 28
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 08:25:10PM +0000, Seb via Digitalmars-d wrote:
 On Thursday, 28 March 2019 at 20:14:19 UTC, Andrei Alexandrescu wrote:
 On 3/28/19 3:12 PM, H. S. Teoh wrote:
 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.
Or at all. The entire effort would require three strong like-minded engineers, and I don't know them.
I'd like to see this effort happen, one way or another. Couldn't you or somebody you trust come up with a design of the core principles and a rough draft, then the rest of the community can flesh it out? It doesn't have to be someone working on it full-time. Also, it could be designed to be backwards-compatible with the current Phobos where this wouldn't compromise the new design, so that it could be done piecemeal instead of a one-shot effort. Doing it piecemeal makes it much more likely to succeed, IMO. For example, you could design a std.v2.range.primitives with the new range API (without .save) and have the community test-drive that. Then once that's solid, add other pieces to it like std.v2.range, std.v2.algorithm, etc.. At each stage you have a minimum viable product that can be used for some limited scope of the full anticipated functionality, rather than trying to chew off the entire thing in one go and finding yourself choking on on overly-large project scope.
 Can't we crowd-found this initiative? Also, if we lay out the rough
 ground lines and draft, the community could pitch in and it wouldn't
 necessarily be a one man's mission.  Divide et impera!
+1. T -- Notwithstanding the eloquent discontent that you have just respectfully expressed at length against my verbal capabilities, I am afraid that I must unfortunately bring it to your attention that I am, in fact, NOT verbose.
Mar 28
prev sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 20:14:19 UTC, Andrei Alexandrescu 
wrote:
 On 3/28/19 3:12 PM, H. S. Teoh wrote:
 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.
Or at all. The entire effort would require three strong like-minded engineers, and I don't know them.
Or a team of subordinate apprentice/junior engineers (who have demonstrated sufficient potential) working under the mentorship of a master who is willing to make an investment. Mike
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 9:26 PM, Mike Franklin wrote:
 On Thursday, 28 March 2019 at 20:14:19 UTC, Andrei Alexandrescu wrote:
 On 3/28/19 3:12 PM, H. S. Teoh wrote:
 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.
Or at all. The entire effort would require three strong like-minded engineers, and I don't know them.
Or a team of subordinate apprentice/junior engineers (who have demonstrated sufficient potential) working under the mentorship of a master who is willing to make an investment.
This is a lot more difficult to set up than it seems. One reason is that surprisingly few of the contributors come here to learn, to acquire knowledge. Most come to dispense. I spent long times reviewing pull requests in our community, calibrated to the reviews standards at Facebook (however controversial, they have an impressive software engineering force). There, the code review process is locked in a virtuous circle: you get good reviews that help you improve, and you'd be motivated to give good reviews to improve others'. Here, my reviews were more often than not met with hostility. As a pattern, a reviewer would be more willing to write the code once and then defend it in its current state, rather than improve it through review. In a few extreme cases, people flat out told me they'll abandon the PR if I don't take it as it is. Larger disagreements on matters of architecture and design were often taken as personal offenses. Of course, retaliation was soon to follow. My own ideas and code would be met with suspicion and I'd need to push them uphill instead of them being encouraged and improved, which provided disincentive for trying any creative work. I'd be routinely followed on github and countered at whatever I'd do or say. Of course this happened to Walter even more, and the only therapy we got was commiseration via the occasional email titled "Somebody shoot me" or similar, and containing just a link to a github comment. I'd wake up in the morning for days, then weeks and months, with one thought: "I have so many things to do today, and I like none of them." The irony! Here I was, independent, working for myself on whatever I dreamed of. Yet this was worse than the worst jobs I've ever had. How did I get into this situation? I think it has to do with a simple reality - I was trying to mentor people who didn't want to be mentored. So I think it would be difficult to establish a master/mentee dynamics.
Mar 29
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/29/19 9:57 AM, Andrei Alexandrescu wrote:
 As a pattern, a reviewer would be more willing to write the code once 
s/reviewer/author/
Mar 29
prev sibling next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 29 March 2019 at 13:57:57 UTC, Andrei Alexandrescu 
wrote:
 Here, my reviews were more often than not met with hostility.

 As a pattern, a reviewer would be more willing to write the 
 code once and then defend it in its current state, rather than 
 improve it through review. In a few extreme cases, people flat 
 out told me they'll abandon the PR if I don't take it as it is.
Where I work, we're trying our best to do design reviews before starting large implementations and to seek reviewer feedback throughout the process. We do this because it seems clear that after the code is already written and after several days' to weeks' worth of effort has gone into a feature, the cost to making fundamental design changes is as great as it's going to get, and developer attachment to the existing code as well as frustration with the reviewer is high, especially when the reviewer asks for fundamental design changes. And this, to pivot to another debate we had recently, is exactly why it's a counterproductive approach to do DIP reviews only at the end.
Mar 29
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/29/19 11:12 AM, FeepingCreature wrote:
 On Friday, 29 March 2019 at 13:57:57 UTC, Andrei Alexandrescu wrote:
 Here, my reviews were more often than not met with hostility.

 As a pattern, a reviewer would be more willing to write the code once 
 and then defend it in its current state, rather than improve it 
 through review. In a few extreme cases, people flat out told me 
 they'll abandon the PR if I don't take it as it is.
Where I work, we're trying our best to do design reviews before starting large implementations and to seek reviewer feedback throughout the process. We do this because it seems clear that after the code is already written and after several days' to weeks' worth of effort has gone into a feature, the cost to making fundamental design changes is as great as it's going to get, and developer attachment to the existing code as well as frustration with the reviewer is high, especially when the reviewer asks for fundamental design changes.
There are few more productive moments at Facebook than those when five solid engineers get in a room and pound out a design. Keywords being "five", "solid", and "room". The moral equivalent for us would be a virtual room - a forum or a discussion around a github pull request. The problem is access and numbers. Consider a Gedankenexperiment whereby Facebook opened a room to the public, capacity 12, and offered it to any passer-by willing to discuss design with a couple of Facebook engineers. Now, one of two things may happen. One is, the notion is very successful and people would clamor to get into the room. Natural dynamics would lead to some form of selection at the door that would allow only the best of the best enter. Another is, the notion is only mildly intriguing. So there's a few folks inside, and nobody waiting outside. People come and go because they don't see a huge emulation at the door creating incentive to prove their worth inside. Now there would be competition, but just against the two Facebook engineers who are perceived as the benchmark against which everyone needs to compare. (Remember, nobody is at the door to put pressure.) Anyone can enter and stay for any period of time, and their voice will be as loud as anyone else's in the room. So they'd want to outdo the Facebook engineers, and if they riposte, anyone can throw their hands in the air and leave. Look at what happens in Rust. People get over each other to add quality to the language, because if they don't, the next guy has a better proposal, idea, or code. There's a crowd at the door, putting pressure on the folks within. Folks don't go around telling Niko Matsakis he's a chowderhead who could learn a thing or two. It's most interesting what happens with Walter. He is one of the best hackers (in the Paul Graham sense) I've ever known, and that says a lot; I know quite a few, and I could also hack my way out of a paper bag. Walter is literally the guy who could write a line of code where most others would need one hundred. Yet he is hardly revered in our community, and his views on doing software are routinely debated and often derided. His inability to debate is part of it (he's a hacker, not a politician), but I think the lion's share goes to the "Facebook room" thought experiment above. Not enough people at the door.
 And this, to pivot to another debate we had recently, is exactly why 
 it's a counterproductive approach to do DIP reviews only at the end.
Yes, that would be helpful. One problem we have there is proposals that cannot be improved via review. Years ago, I had the hots for a girl so I did what many silly guys do - I friended her. She confessed she had aspirations for writing. Being likely inclined as well and wanting to please, I enthusiastically proffered interest in her endeavor. So she gave me a manuscript to read. It was... bad. Not just bad, fractally bad - from the overall arc down to grammar and punctuation. Wanting to say something nice about it, I went for advice to my uncle, a published author (his obituary is online - Nicuta Tanase). He said, kid, just mention a couple of things that are objective and would improve things. And so I did, not realizing I was creating a larger problem for myself. Because a week later, she came with another draft in hand, saying: "I fixed it! Now it's good for publishing... right?" Needless to say, that didn't go well for my dating life.
Mar 29
next sibling parent throw away <throw away.com> writes:
Now on Hacker News:

https://news.ycombinator.com/newest
Mar 29
prev sibling next sibling parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Friday, 29 March 2019 at 16:26:06 UTC, Andrei Alexandrescu 
wrote:
 Years ago, I had the hots for a girl so I did what many silly 
 guys do - I friended her. She confessed she had aspirations for 
 writing. Being likely inclined as well and wanting to please, I 
 enthusiastically proffered interest in her endeavor. So she 
 gave me a manuscript to read.
I don't think this comparison is appropriate, and honestly, I think it's a little disrespectful to contributors. Like, contributors aren't trying to get a date. They don't want to be teased, they don't want to play social games, they don't want things to be dramatic or fun or romantic. They want to submit a change, and they want it to be straightforward. Also, they're mostly volunteers, helping on their free time out of passion. Comparing them to a girl hamstringing you for writing advice seems extremely condescending.
 It was... bad. Not just bad, fractally bad - from the overall 
 arc down to grammar and punctuation. Wanting to say something 
 nice about it, I went for advice to my uncle, a published 
 author (his obituary is online - Nicuta Tanase). He said, kid, 
 just mention a couple of things that are objective and would 
 improve things.

 And so I did, not realizing I was creating a larger problem for 
 myself. Because a week later, she came with another draft in 
 hand, saying: "I fixed it! Now it's good for publishing... 
 right?"
That doesn't mean early reviews aren't valuable. Like, I get that some people get sore when you tell them that a submission they've spent a lot of effort on just isn't good enough, and sometimes they will be unreasonable no matter what you say. (and, as I've said before, the community has a tendency to jump to "Oh, W&A are being stubborn and ignoring The Will of The People again" every time you make a design decision someone doesn't like) But the response to that is *more* communication, not less. If someone's PR is too bad to be merged, you're not going to help anyone by letting the person work on code that has no chance to be accepted. Same thing for DIPs. Saying "This would be much easier if the community was bigger and people fought for my attention" is just wishful thinking. You don't solve a communication problem by becoming a cult icon, you solve it by communicating better, and earlier.
 Look at what happens in Rust. People get over each other to add 
 quality to the language, because if they don't, the next guy 
 has a better proposal, idea, or code. There's a crowd at the 
 door, putting pressure on the folks within. Folks don't go 
 around telling Niko Matsakis he's a chowderhead who could learn 
 a thing or two.
Seriously, I'm worried that the takeway you're getting from all this is "the community is unreasonable, unlike Rust's community which has a healthy respect for the maintainers". It's not just that. The D community is has a ton of people whose experience writing a PR is "I spent two weeks writing that code, got a comment from Walter six months later that asked to clarify what X did, I added some documentation, and I haven't had any news for a year". If you look at the open PRs on rust, all PRs in page 3 are less than two weeks old. By comparison, some PRs in page 3 of dmd are from May 2018, almost a year ago! I realize that the D team has way less resources than Mozilla to dedicate to following PRs, but acting like the only problem is that PR authors are capricious is just disingenuous.
Mar 29
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Friday, 29 March 2019 at 21:28:40 UTC, Olivier FAURE wrote:
 Seriously, I'm worried that the takeway you're getting from all 
 this is "the community is unreasonable, unlike Rust's community 
 which has a healthy respect for the maintainers". It's not just 
 that. The D community is has a ton of people whose experience 
 writing a PR is "I spent two weeks writing that code, got a 
 comment from Walter six months later that asked to clarify what 
 X did, I added some documentation, and I haven't had any news 
 for a year".
I don't how long you've been around, but this community seems to me as indeed unreasonable and often disrespectful (I've been doing it too at times). The sheer number of people trolling and wanting to hurt other people personally over the years has been really disconcerting. We need to attract users who _need_ D to make stuff, contribute back (in time and money), and stop asking random useless things every other day of the week. And the problem compounded because the abusive behaviour sometimes led to the most langage change! The problem with D is that it has an "unscalable" community of very demanding people that bring not enough, so like a startup without scale economics it has difficulty to scale. As a friend working on Eclipse told me, "all things being equal, having users is a liability" so you have to ask them to create more value than being removed. Something that we do that I think doesn't work is... the lack of a Code of Conduct and a rule of not banning anyone, which opens the gate of small, constant, rampant abuse that can surface at the first occasion. And something we should do more is put trust into the leadership more because, of all the mistakes they have been accused of, so many of the decisions ended up being the right ones.
Apr 03
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 3 April 2019 at 12:33:38 UTC, Guillaume Piolat 
wrote:
 The sheer number of people trolling and wanting to hurt other 
 people personally over the years has been really disconcerting.
I realize that it's difficult to give examples here without calling specific people out, but without giving examples I have no way to judge what your standard for "trolling and wanting to hurt other people personally" is. That's kind of a pretty hefty accusation to make without the ability to ethically provide evidence. I think either you need to explicitly call people out here or you need to formulate this more leniently, as it stands it's kind of like announcing "several people in this room are psychotic killers, I can't say who, but that's why we need to take urgent action." It is fundamentally epistemically improper to demand action on weighty evidence that cannot be ethically provided.
Apr 03
next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 3 April 2019 at 14:09:49 UTC, FeepingCreature wrote:
 It is fundamentally epistemically improper to demand action on 
 weighty evidence that cannot be ethically provided.
I've been perhaps overstating my point, but no I'm not going to namecall - let's talk about the anonymous accounts. One prime example is in one message juste above ours. Once Andrei started saying things remotely uncomfortable for D, one "helpful" anonymous individual posted it to HN. Surely the intent couldn't be positive for D. It gives the impression it's OK to disrespect the leadership, and that it's OK to shit in the soup. Such anonymous actions have been going on since the beginning of D times, it gives a baseline upon more respectable personas can act in ways they wouldn't have. I don't _demand_ anything, one action that would make sense if - in my opinion - to stop anonymity in these forums, and add accountability back.
Apr 03
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 3 April 2019 at 14:09:49 UTC, FeepingCreature wrote:
 On Wednesday, 3 April 2019 at 12:33:38 UTC, Guillaume Piolat 
 wrote:
 The sheer number of people trolling and wanting to hurt other 
 people personally over the years has been really disconcerting.
I realize that it's difficult to give examples here without calling specific people out, but without giving examples I have no way to judge what your standard for "trolling and wanting to hurt other people personally" is. That's kind of a pretty hefty accusation to make without the ability to ethically provide evidence. I think either you need to explicitly call people out here or you need to formulate this more leniently, as it stands it's kind of like announcing "several people in this room are psychotic killers, I can't say who, but that's why we need to take urgent action." It is fundamentally epistemically improper to demand action on weighty evidence that cannot be ethically provided.
They seems to come and go, sometimes posing as enthusiastic users who want some specific thing to happen and get all upset when people don't take too kindly to it. Sometimes they impersonate other community members and stoke the flames. Unfortunately I don't have any links offhand (I'm madly packing) and have forgotten the names they posted under. The egregious one tend to be deleted from the forum interface (although I think they are still on the mailing lis that backs it).
Apr 03
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/3/19 8:33 AM, Guillaume Piolat wrote:
 I don't how long you've been around, but this community seems to me as 
 indeed unreasonable and often disrespectful (I've been doing it too at 
 times).
I also think we have patterns of negativity. In the recent days I have received a number of private messages also mentioning that as an ongoing difficulty (you know who you are; thank you). I'm not very worried about some heat in casual discussions in the forums, though all of us could use more civility. I am, however, worried about negativity "in production", i.e. on github and in DIP-related and other consequential exchanges in the forums. My current hypothesis is it has to do with people's lack of time. Consider modeling a "specialist short on time". For a while I tried to model lack of time as a sort of decrease in IQ. For sure that's happened to me. Whenever I gave myself only a few minutes to make a decision on a pull request, that would definitely count as a net decrease in competence. However, that model is not very accurate. For example, a specialist short on time would still be able to point out a mistake that would escape a non-specialist. So a better model is, a specialist short on time in a design or code review has a more negative bent than one with time. This is because software is quintessentially constructive. In math, proving a negative is difficult because the burden is to prove something can't exist. In software, proving a positive is more difficult because it must be constructed. What is easy in code and design reviews is small "proofs" that an existing proposal has a problem. It's also time effective - a threading specialist may see a five-line race pattern immediately. A programming languages expert would see how qualifiers are bad for postblits in a minute. Most of all, pointing out a negative gives the reviewer undeniable expertise high ground and a sense of doing the right thing. I pointed out a problem, the reasoning goes, thus alerting people that a mistake could be made. The reviewer gets a good jolt of satisfaction and moves on with their day. The fallacy within is akin of the one of broken windows. The first-order reasoning goes, a dangerous mistake is being averted and correctness or at least some notion of consistency (people in software design love consistency) is being preserved. The second-order effects, however, are numerous and pernicious. The main problem is, again, the constructive nature of software: often, a pull request or some other proposal embedding a mistake originates from a genuine need. People need to use malloc and free in pure code. They need reference-counted collections that are also immutable. Then, our archetypal specialist-short-on-time reviewer points out how that is problematic and considers the matter done with. "The problem is solved," - the subtext reads almost triumphantly - "it can't be done". To a non-investor specialist interested in correctness, this is a perfectly valid outcome. To the poor sap who wanted to get work done, that's none too helpful. Contagion is another second-order effect. People in the community, would-be contributors and reviewers, see specialists who are mostly negatively biased. Clearly they know what they're talking about, so they are admired and set the standard everybody is aspiring to. Soon enough, it becomes "comme il faut" to point out weaknesses in proposed code instead of doing the much more difficult and time-consuming work of helping improving it, proposing better alternatives, and such. So all reviewers turn negative, whether they are specialists or not. As soon as I'd put a pull request that had any informational entropy to it (i.e. not 100% obvious and boring), I might as well put the ear to the ground to hear the sound of shovels digging. The Vickers getting mounted. Barb wire getting rolled. Reviewers were getting ready for trench warfare. It has sapped all of my creative streak. My students, enthusiastic but scared soldiers, hesitate when they hear my whistle. "They'll kill me if I put this pull request out for review," I heard more than once put this way or another. More specialists would help, but many specialists with a short time each would be less than helpful. We need steady energy, not just the occasional jolt of power. Electrical grid, not lightning. Great Work done by invested specialists has the true potential to turn negativity around.
Apr 04
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 29 March 2019 at 16:26:06 UTC, Andrei Alexandrescu wrote:
 It's most interesting what happens with Walter. He is one of 
 the best hackers (in the Paul Graham sense) I've ever known
Walter is a brilliant programmer whose historic experience is exceedingly valuable. You're also an excellent programmer and a good writer who brings very valuable higher-thought discipline to library design. Where you guys are weak is project management. A project manager needs to set clear expectations, facilitate communication among the team, prioritize work based on real world requirements, and motivate the team to do that work, striking the right balance between autonomy and direction to maintain morale while meeting requirements. I'm sure facebook has some good managers who keep that room of engineers focused on the big picture and willing to come back to work week after week. D needs a good manager too. They make a real difference. I betcha if we got a good manager, contributors would be a lot happier working in a shared direction and the manpower problems the PR queue sees now would evaporate. And best of all, it'd let you and Walter go back to doing the awesome work you really excel at, instead of arguing with the community so much.
Mar 29
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Mar 29, 2019 at 07:45:35PM -0400, Adam D. Ruppe via Digitalmars-d wrote:
 On Friday, 29 March 2019 at 16:26:06 UTC, Andrei Alexandrescu wrote:
 It's most interesting what happens with Walter.
[...]
 Walter is a brilliant programmer whose historic experience is
 exceedingly valuable. You're also an excellent programmer and a good
 writer who brings very valuable higher-thought discipline to library
 design.
 
 Where you guys are weak is project management. A project manager needs
 to set clear expectations, facilitate communication among the team,
 prioritize work based on real world requirements, and motivate the
 team to do that work, striking the right balance between autonomy and
 direction to maintain morale while meeting requirements.
+1. I spent the last hour writing a whole bunch of stuff to respond to this, but decided instead to distill it all to just two things: (1) It's clear that Walter & Andrei excel at technical expertise, but lack in the people skills department. This has been shown time and again in the way various interactions with different people panned out over the years. (Please don't take this as an insult or a personal attack, because I don't intend it that way, and I also recognize myself to be belong to the same category. We techie types simply aren't good at people skills, that's all there is to it.) (2) OTOH, people skills are exactly what's needed to foster a healthy, thriving community around D. It's unrealistic to expect a volunteer-run open source project to operate the same way as a company that pays its employees to do assigned work. A volunteer project *can* produce work of equivalent quality, but the lack of monetary compensation must be replaced by a different kind of "currency": an equivalent amount of inspirational leadership. I.e., inspire the people enough and they'll go out of their way to do whatever you want, pay or no pay. And a big part of this is effective communication of the high-level goals of the project and what's expected of would-be contributions. Fail to do that, and be prepared for the same reaction you might get if you gave an employee a sudden big pay cut while still expecting the same amount (and quality) of work as before. The lack of manpower, etc., are not a cause of our problems; it's a symptom of our lack of proper leadership (in the social sense -- clearly, W&A have excellent technical leadership; but that's not what's missing here). I don't blame Walter and Andrei for this -- we techie types just aren't good at this sort of stuff. We need a manager who is. tl;dr: Walter & Andrei shouldn't be expected to bear the brunt of the social aspect of community building. That's not their forte, and even if they could do it, it'd be a waste of their technical talents. We need a proper manager who can do a good job with the community-related / people-related stuff, and leave W&A to do what they're best at doing.
 I'm sure facebook has some good managers who keep that room of
 engineers focused on the big picture and willing to come back to work
 week after week. D needs a good manager too. They make a real
 difference.
 
 I betcha if we got a good manager, contributors would be a lot happier
 working in a shared direction and the manpower problems the PR queue
 sees now would evaporate. And best of all, it'd let you and Walter go
 back to doing the awesome work you really excel at, instead of arguing
 with the community so much.
Yes and yes. It's about time we acknowledge where our weaknesses lie, and take real steps at fixing the problem, instead of continuing to lie to ourselves with the fantasy that social problems can be cured by technical expertise. We techie types all wish it were so, but sadly the real world simply doesn't work that way. T -- Claiming that your operating system is the best in the world because more people use it is like saying McDonalds makes the best food in the world. -- Carl B. Constantine
Mar 29
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 30 March 2019 at 01:10:59 UTC, H. S. Teoh wrote:
 And a big part of this is effective communication of the 
 high-level goals of the project and what's expected of would-be 
 contributions.
I'm hoping we can set a lot of that high level goals at the dconf AGM (along with a lot of other problems). I'll be publishing the (draft, to be added to by others) agenda soon.
Mar 30
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/29/19 9:10 PM, H. S. Teoh wrote:
 It's clear that Walter & Andrei excel at technical expertise, but
 lack in the people skills department.
As Walter tells me on the phone when it comes about community, "the main problem with our community is it's poorly led" - a very humble, humbling, and positive take. Speaking for myself, my doing a poor job at leadership in our community is so obvious to me, it has almost become a cop-out. It's second to none in terms of failure per unit of time invested, so in a way simply admitting I'm just not good at it is the easy way to explain an overwhelmingly complex reality. This had me revisit the past for similarities and differences. And that inevitably takes to C++. In the C++ community the outcome has been quite the opposite. After Modern C++ Design (of which a friend went so far as to say "showed Bjarne what C++ really was about"), I have inadvertently enjoyed enormous impact and inspirational power, whilst being aloof to it. One forum post was enough to determine Hans Boehm and Herb Sutter to lead the standardization of threads in C++11; off-the-cuff ideas and conversations became N4259 or P0323R1; Mojo, an afternoon's worth of work, may have provided at least part of the reason for rvalue references. To this day, I am receiving credit about things I'd all but forgotten about. I could burp in a keynote and someone will create a great C++ template out of it. I knew things came full-circle when I got this question following my "The Next Big Thing" talk: "Have you heard of std::conditional?" (meant as a clever comeback to my plea for "static if"). I realized that both the person asking and myself had forgotten I'd invented the blessed thing. And this is most puzzling, because I haven't given C++ the time of the day in a long while, so in terms of at-least-somewhat-related-to-leadership-impact per unit of effort, my participation C++ has been quite productive. (And for a good part it's been unwitting and almost unwilling, a la Stepan Trofimovich Verkhovensky in Dostoyevsky's "Demons" - hopefully in a positive way though.) Not to say that arguably the very best of my work is to be found in D, many miles away from what I could ever do for C++. But even not counting that: with me being a common part of the inequation, I can be simplified away. Which leads to the most puzzling question: why have the outcomes across the C++ and D communities have been so different? The answer to this question may unlock the potential of our community.
Mar 30
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 31 March 2019 at 00:18:29 UTC, Andrei Alexandrescu 
wrote:
 Not to say that arguably the very best of my work is to be 
 found in D, many miles away from what I could ever do for C++. 
 But even not counting that: with me being a common part of the 
 inequation, I can be simplified away.
That would assume that your output (not input) to the two are equal and more importantly separable form the outputs of others. The second one is definitely not true.
 Which leads to the most puzzling question: why have the 
 outcomes across the C++ and D communities have been so 
 different?
Leadership and vision (or rather lack thereof) are two of the most critical issues holding D back. Forget quality for the moment, it can be well approximated by a simple game of numbers: D: Walter and Andrei No vision document for I've forgotten how long No regular direction steering meetings with users (mod that one ~5 months ago) One conference a year A few local regional get togethers DLF C++: Herb, Marshall, Bjarne, ... Multiple direction documents / roadmaps for language, library, ecosystem, HPC, ... C++ standards committees: 1 week every 3 months I don't even know how many conferences per year Who knows how many local gatherings C++ alliance / C++ standards body And I'll be blunt, much of your recent leadership has actually been gatekeeping, especially w.r.t refactoring. How to remedy this? Well _I'm_ starting with: the DConf AGM (draft agenda to be published soon™). DLF quarterly meetings, especially if we can get them to coincide with various regional quarterly gatherings Greater (corporate) participation in DLF processes, vision outreach. What are you going to do?
Mar 30
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/30/2019 6:51 PM, Nicholas Wilson wrote:
 And I'll be blunt, much of your recent leadership has actually been
gatekeeping, 
 especially w.r.t refactoring.
I know we're having differences about what constitutes a good refactoring, and what isn't. I hope that between us at DConf with the aid of some suds we can reach a consensus on this.
Mar 30
prev sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Sunday, 31 March 2019 at 00:18:29 UTC, Andrei Alexandrescu 
wrote:
 why have the outcomes across the C++ and D communities have 
 been so different?

 The answer to this question may unlock the potential of our 
 community.
Isn't suggesting ideas for language features the easy part? As opposed to actually making a formal proposal, teasing out the corner cases, comparing alternatives, getting reviewers to accept it, etc? If your experience in C++ has been mostly writing libraries and proposing ideas, and your experience in D has been mostly reviewing actual work and dissecting complete proposals, then of course C++ will seem easier to you. I mean, it's not like the standardization process in C++ is painless. In fact, it's renowned for being a byzantine bureaucracy.
Mar 31
prev sibling parent Rubn <where is.this> writes:
On Friday, 29 March 2019 at 16:26:06 UTC, Andrei Alexandrescu 
wrote:
 Look at what happens in Rust. People get over each other to add 
 quality to the language, because if they don't, the next guy 
 has a better proposal, idea, or code. There's a crowd at the 
 door, putting pressure on the folks within. Folks don't go 
 around telling Niko Matsakis he's a chowderhead who could learn 
 a thing or two.
There's a saying, keep your eyes on your own plate. You really want to compare to rust? How about starting with the fact D doesn't even provide a 64-bit binary on Windows for their compiler. That's the sort of trivial problems D has yet to even solve. You best look at your own plate first before even considering to compare to others.
Mar 30
prev sibling next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 29 March 2019 at 13:57:57 UTC, Andrei Alexandrescu 
wrote:

 Or a team of subordinate apprentice/junior engineers (who have 
 demonstrated sufficient potential) working under the 
 mentorship of a master who is willing to make an investment.
This is a lot more difficult to set up than it seems. One reason is that surprisingly few of the contributors come here to learn, to acquire knowledge.
[...]
 I think it has to do with a simple reality - I was trying to 
 mentor people who didn't want to be mentored.

 So I think it would be difficult to establish a master/mentee 
 dynamics.
I understand (as my head hangs in disappointment). I've observed it for myself, and may even be guilty of it in a few instances. This is why I added the word "subordinate" to my sentence before I submitted it. The apprentice would have to agree to subordination as part of the up-front agreement. It's not just knowledge the apprentice is seeking. (S)he may be also be seeking credentials and experience. The credentials include the trophy of accomplishment and the privilege to use the master's name as a reference in their future prospects. Insubordination results in being fired and losing those credentials, not to mention the time and effort spent, and the opportunity to expand their knowledge. The master would have to be selective, just like an employer, to find the right people for the apprenticeships, and there would have to be some up-front agreement with conditions and expectations. I don't have any experience with this kind of thing, so perhaps I don't know what I'm talking about. It's just a shame that people are willing to take out a loan for 10s of thousands of dollars for an education built around contrived assignments, when there's an opportunity here to do real meaningful work, led by some of the most reputable names in the field, that doesn't cost anything. What am I missing? Mike
Mar 29
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/29/2019 8:38 AM, Mike Franklin wrote:
 It's just a shame that people are willing to take out a 
 loan for 10s of thousands of dollars for an education built around contrived 
 assignments, when there's an opportunity here to do real meaningful work, led
by 
 some of the most reputable names in the field, that doesn't cost anything. 
What 
 am I missing?
One thing I really enjoy is working with people who are better at it than I am. It's the fastest, most effective way to up your game, and besides, it's fun.
Mar 29
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Friday, 29 March 2019 at 13:57:57 UTC, Andrei Alexandrescu 
wrote:
 This is a lot more difficult to set up than it seems.

 One reason is that surprisingly few of the contributors come 
 here to learn, to acquire knowledge. Most come to dispense. I 
 spent long times reviewing pull requests in our community, 
 calibrated to the reviews standards at Facebook (however 
 controversial, they have an impressive software engineering 
 force).
Therein lies your problem, you are attempting to fit a square peg in a round hole. You are not dealing with well paid highly qualified engineers whose job it is to do what needs to be done. For the most part you are dealing with an intermittent army of volunteers of a range of abilities for whom this is a hobby.
 There, the code review process is locked in a virtuous circle: 
 you get good reviews that help you improve, and you'd be 
 motivated to give good reviews to improve others'.

 Here, my reviews were more often than not met with hostility.
I can't speak for your phobos reviews, but most of the interactions I've had with you on dmd were not at all useful.
 As a pattern, [an author] would be more willing to write the 
 code once and then defend it in its current state, rather than 
 improve it through review. In a few extreme cases, people flat 
 out told me they'll abandon the PR if I don't take it as it is.
Been there https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733 done that https://github.com/dlang/dmd/pull/8557#issuecomment-416798558
 Larger disagreements on matters of architecture and design were 
 often taken as personal offenses.
With responses like that I can see why.
 I'd wake up in the morning for days, then weeks and months, 
 with one thought: "I have so many things to do today, and I 
 like none of them." The irony! Here I was, independent, working 
 for myself on whatever I dreamed of. Yet this was worse than 
 the worst jobs I've ever had. How did I get into this 
 situation? I think it has to do with a simple reality - I was 
 trying to mentor people who didn't want to be mentored.
Probably, chances are most of them wanted to get shit done...
 So I think it would be difficult to establish a master/mentee 
 dynamics.
... but there are definitely those out there looking for experience, looking to become better programmers (GSoC students are a good source).
Mar 30
prev sibling next sibling parent reply destructionator gmail.com writes:
On Thursday, 28 March 2019 at 19:12:40 UTC, H. S. Teoh wrote:
 The TypeInfo stuff, for one thing, will take some work to reach 
 your vision of a completely opt-in library.  Currently, trying 
 to link without druntime will cause a bunch of errors related 
 to missing TypeInfo's.
Not true - that was fixed like a year ago. See http://dpldocs.info/this-week-in-d/Blog.Posted_2019_02_25.html#no-runtime-d The code there compiles with no extra code; it just works. and my "last year in D" post on the official blog called this out with the release (I just don't remember the link)
 Or the GC initialization in rt_init / rt_term.  Or, for that 
 matter, the initialization code that packages command-line 
 arguments into a string[] before invoking Dmain.
The GC is initialized on the first use. Lots of movement in this direction already happened in 2018!
Mar 28
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 19:23:30 UTC, destructionator wrote:
 On Thursday, 28 March 2019 at 19:12:40 UTC, H. S. Teoh wrote:
 The TypeInfo stuff, for one thing, will take some work to 
 reach your vision of a completely opt-in library.  Currently, 
 trying to link without druntime will cause a bunch of errors 
 related to missing TypeInfo's.
Not true - that was fixed like a year ago. See http://dpldocs.info/this-week-in-d/Blog.Posted_2019_02_25.html#no-runtime-d The code there compiles with no extra code; it just works. and my "last year in D" post on the official blog called this out with the release (I just don't remember the link)
[...]
 Lots of movement in this direction already happened in 2018!
Yes, the release that enabled that was 2.079 (https://dlang.org/changelog/2.079.0.html) It started with `ModuleInfo` which set a precedent for then doing `TypeInfo` and `Throwable` https://github.com/dlang/dmd/pull/7395 https://github.com/dlang/dmd/pull/7768 https://github.com/dlang/dmd/pull/7799 https://github.com/dlang/dmd/pull/7786 The work is also ongoing with Dan Printzell recently deciding on decoupling `TypeInfo` from the runtime hooks for his GSoC proposal (https://forum.dlang.org/post/dunewuhfbkqnbtemicmk forum.dlang.org). Let's be sure to give him our support. Mike
Mar 28
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 03:23:30PM -0400, destructionator--- via Digitalmars-d
wrote:
 On Thursday, 28 March 2019 at 19:12:40 UTC, H. S. Teoh wrote:
[...]
 Or the GC initialization in rt_init / rt_term.  Or, for that 
 matter, the initialization code that packages command-line 
 arguments into a string[] before invoking Dmain.
The GC is initialized on the first use.
Yes, but not calling rt_init still causes a segfault when you try to use the GC. So, we're not quite there yet.
 Lots of movement in this direction already happened in 2018!
That's good to know. I'm still wondering, though, how we're going to drop druntime while still maintain backward compatibility. Or how we could skip the startup code that invokes module ctors and packages command-line args into string[]. It would be awesome if we could pull this off, but I don't currently see how we could do it without major breakage. T -- Why waste time learning, when ignorance is instantaneous? -- Hobbes, from Calvin & Hobbes
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:31 PM, H. S. Teoh wrote:
 On Thu, Mar 28, 2019 at 03:23:30PM -0400, destructionator--- via Digitalmars-d
wrote:
 On Thursday, 28 March 2019 at 19:12:40 UTC, H. S. Teoh wrote:
[...]
 Or the GC initialization in rt_init / rt_term.  Or, for that
 matter, the initialization code that packages command-line
 arguments into a string[] before invoking Dmain.
The GC is initialized on the first use.
Yes, but not calling rt_init still causes a segfault when you try to use the GC. So, we're not quite there yet.
That'd need to be fixed, and it should be possible without disrupting clients.
 Lots of movement in this direction already happened in 2018!
That's good to know. I'm still wondering, though, how we're going to drop druntime while still maintain backward compatibility. Or how we could skip the startup code that invokes module ctors and packages command-line args into string[]. It would be awesome if we could pull this off, but I don't currently see how we could do it without major breakage.
Compiler flag.
Mar 28
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 20:02:17 UTC, Andrei Alexandrescu 
wrote:

 I'm still wondering, though, how we're going to drop druntime 
 while
 still maintain backward compatibility.  Or how we could skip 
 the startup
 code that invokes module ctors and packages command-line args 
 into
 string[].  It would be awesome if we could pull this off, but 
 I don't
 currently see how we could do it without major breakage.
Compiler flag.
D is so powerful, this can also be done in other ways as well. As Andrei alluded to in another post, we just need to embrace addition. Keep the existing logic path in place, and create a new one that does what we want it to do. With design-by-introspection, and probably a number of other awesome D facilities, the correct logic path can be chosen at compile-time based on what the user has declared in their code. Another idea is allowing "module inheritance" for lack of a better term. Users can "inherit" from an "abstract" module but override the default startup and shutdown logic. Perhaps we refactor the current logic path to inherit from this "abstract" module but add all of the existing startup and shutdown implementation as overrides. The compiler is then modified to "inherit" from that module by default. Users that don't want the default can "inherit" from the "abstract" module and provide their own implementation. I obviously haven't thought all of this through; I'm just trying to illustrate D is so powerful that there are other ways to achieve a similar result without always resorting to compiler flags, though a compiler flag may be the best option. Mike
Mar 28
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 10:47 PM, Mike Franklin wrote:
 On Thursday, 28 March 2019 at 20:02:17 UTC, Andrei Alexandrescu wrote:
 
 I'm still wondering, though, how we're going to drop druntime while
 still maintain backward compatibility.  Or how we could skip the startup
 code that invokes module ctors and packages command-line args into
 string[].  It would be awesome if we could pull this off, but I don't
 currently see how we could do it without major breakage.
Compiler flag.
D is so powerful, this can also be done in other ways as well. As Andrei alluded to in another post, we just need to embrace addition. Keep the existing logic path in place, and create a new one that does what we want it to do.  With design-by-introspection, and probably a number of other awesome D facilities, the correct logic path can be chosen at compile-time based on what the user has declared in their code.
*nod*
 Another idea is allowing "module inheritance" for lack of a better 
 term.  Users can "inherit" from an "abstract" module but override the 
 default startup and shutdown logic.  Perhaps we refactor the current 
 logic path to inherit from this "abstract" module but add all of the 
 existing startup and shutdown implementation as overrides.  The compiler 
 is then modified to "inherit" from that module by default.  Users that 
 don't want the default can "inherit" from the "abstract" module and 
 provide their own implementation.
Yah, I actually had even code for an idea - after each module is loaded, the following code is automatically injected at the end of it: mixin(__onImportedModule(__MODULE__)); This is evaluated in the context of the importer (not imported), an essential detail. By default of course the function returns the empty string. This allows user code to add things like serialization, factories and other introspection-based utilities automatically upon import.
Mar 29
next sibling parent =?UTF-8?B?THXDrXM=?= Marques <luis luismarques.eu> writes:
On Friday, 29 March 2019 at 13:05:31 UTC, Andrei Alexandrescu 
wrote:
 Yah, I actually had even code for an idea - after each module 
 is loaded, the following code is automatically injected at the 
 end of it:

 mixin(__onImportedModule(__MODULE__));

 This is evaluated in the context of the importer (not 
 imported), an essential detail. By default of course the 
 function returns the empty string.

 This allows user code to add things like serialization, 
 factories and other introspection-based utilities automatically 
 upon import.
Jean-Louis argued for that overall feature at last year's DConf, and I also agree. Such a feature would be useful for a lot of things, although it's one of those "with great power comes great responsibility" kind of features.
Mar 29
prev sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Friday, 29 March 2019 at 13:05:31 UTC, Andrei Alexandrescu 
wrote:
 This allows user code to add things like serialization, 
 factories and other introspection-based utilities automatically 
 upon import.
You mean something like this? // someFile.proto syntax = "proto2"; message SomeType { required int32 x = 1; required int32 y = 2; optional string label = 3; } message AnotherType { required SomeType xy = 1; required int32 z = 2; } // foobar.d import "someFile.proto" : SomeType, AnotherType; Because that would be seriously awesome.
Mar 29
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 18:47:49 UTC, Andrei Alexandrescu 
wrote:

 Oh, and druntime must go.

 The whole distinction between the runtime library and the 
 standard library is clowny and has more to do with poorly 
 ported tradition from other languages, than with anything else.

 We need one standard library that is entirely pay-as-you-go 
 (e.g. empty main() means no library file is ever needed for 
 linking) and that offers an opt-in continuum.
yes, Yes, YES! Now that's a vision I can get behind. Time to start dusting off my DMD and druntime forks. I'm in. Mike
Mar 28
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 12:47:49 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:
 On 3/28/19 2:41 PM, Andrei Alexandrescu wrote:
 On 3/28/19 2:30 PM, H. S. Teoh wrote:
 The only way I can see this happening is if we start a new iteration of
 Phobos, like std.v2.*
There'd be carefully carved subsets per capability, such as std.v2.betterc.*, std.v2.nogc.*, std.v2.noexcept.*, std.v2.safe. The capability sets can be combined in alphabetic ordering. E.g. std.v2.noexcept.nogc.* has the subset that uses no gc and no exceptions, but std.v2.nogc.noexcept does not exist. Through the magic of public imports and aliasing there would be no code duplication. Introspection could help, too, a la "define public aliases for all safe definitions in that module". Some improvements to ddoc might be needed to "see" through aliases.
Oh, and druntime must go. The whole distinction between the runtime library and the standard library is clowny and has more to do with poorly ported tradition from other languages, than with anything else. We need one standard library that is entirely pay-as-you-go (e.g. empty main() means no library file is ever needed for linking) and that offers an opt-in continuum.
LOL. That's a big thing to drop in as an afterthought. There are some useful organizational aspects of having druntime, but it has become a bigger and bigger problem over time as we've tried to do stuff that's required to be in druntime but which effectively requires stuff that's in Phobos - a long standing example of that being UTF encoding and decoding. AFAIK, there are still a number of hurdles to truly making it pay-as-you-go, but if we can do it, it would be great. This would also be a great opportunity to fix some of the issues with shared in druntime (in particular with the mutex and conditional variable stuff as well as Thread). Too much of it doesn't use shared properly (if at all), and in the case of Thread, it has safe functions that end up passing thread-local objects to a new thread, which would be much easier to do with something like v2 or moving all of that to std (certainly, there are things that should be fixed there that we can't fix in-place). All in all though, I think that we're going to have to plan carefully with how we go forward with major changes like this so that we minimize the risk of splitting the community. With D's module system, we have a lot of flexibility, but when we start replacing core code and concepts with new, incompatible versions in separate modules, we do risk compatibility problems. However, full-sale replacement of some modules would definitely fix some of the issues that replacing stuff in pieces within a module has. - Jonathan M Davis
Mar 30
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/19 2:49 PM, Jonathan M Davis wrote:
 This would also be a great opportunity to fix some of the issues with shared
 in druntime
The problem here is, of course, that nobody knows exactly what shared is supposed to do. Not even me. Not even Walter. One monumental piece of work would be DIP 1021 "Define the semantics of shared". Then people who build with -dip1021 know what to do and what to expect. Now that would patch one of the bigger holes mentioned in the other post.
Mar 30
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 31 March 2019 at 00:41:15 UTC, Andrei Alexandrescu 
wrote:
 On 3/30/19 2:49 PM, Jonathan M Davis wrote:
 This would also be a great opportunity to fix some of the 
 issues with shared
 in druntime
The problem here is, of course, that nobody knows exactly what shared is supposed to do. Not even me. Not even Walter.
If that is the case then you should not be so hostile to suggestions *cough *Manu *cough* to improve that (maybe that was more Walter than you, I don't remember).
 One monumental piece of work would be DIP 1021 "Define the 
 semantics of shared". Then people who build with -dip1021 know 
 what to do and what to expect. Now that would patch one of the 
 bigger holes mentioned in the other post.
Its in the vision section for the dconf AGM as: shared ( and safe and shared)
Mar 30
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/19 10:08 PM, Nicholas Wilson wrote:
 One monumental piece of work would be DIP 1021 "Define the semantics 
 of shared". Then people who build with -dip1021 know what to do and 
 what to expect. Now that would patch one of the bigger holes mentioned 
 in the other post.
Its in the vision section for the dconf AGM as: shared ( and safe and shared)
The necessity to work on shared was also present in the January 2015 vision document (https://wiki.dlang.org/Vision/2015H1): "Nail down fuzzily-defined areas of the language (e.g. shared semantics, property)." Writing it down doesn't get it done. Defining shared properly would take a team with a programming languages expert, a threading expert, and an application expert. (Some roles may be realized within the same person.) I know people within our community with the required expertise. But I don't know any who'd also have the time to embark on this. I got burned out on writing vision documents (Walter was never a fan so I did them all) because it was difficult to figure their impact. Contributors asked for a vision document. So we started one. Then contributors continued doing what they were doing.
Mar 31
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 7:30 AM, Andrei Alexandrescu wrote:
 But I don't know any who'd also have the time to embark on this.
s/any/enough/
Mar 31
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 31 March 2019 at 11:30:08 UTC, Andrei Alexandrescu 
wrote:
 I got burned out on writing vision documents (Walter was never 
 a fan so I did them all) because it was difficult to figure 
 their impact. Contributors asked for a vision document. So we 
 started one. Then contributors continued doing what they were 
 doing.
I have to say I find the posts you have made the last few days to be very open and containing a lot of truth. It is very refreshing and inspiring to read. I hope you can find your way again and do what you do best.
Mar 31
parent reply bachmeier <no spam.net> writes:
On Sunday, 31 March 2019 at 12:03:30 UTC, Sebastiaan Koppe wrote:

 I have to say I find the posts you have made the last few days 
 to be very open and containing a lot of truth. It is very 
 refreshing and inspiring to read.
At the risk of being negative, I had the opposite reaction after reading his posts. It is clear that he has no interest in being the leader of an open source project. He doesn't want to waste his time with average programmers, he only wants to spend his time working with the best of the best (by some metric) software engineers. That's his option, of course, but it's as welcoming to new contributors as a rattlesnake pit (on par with Linus but using better language). An open source project leader needs to find ways to bring in new contributors, including some who are below average, and help them find a way to contribute. There needs to be an interest/willingness to think about and participate in all areas of the project, even non-elite things like documentation, newbie guides, tools, and marketing the language. Open source leader and working on a narrow set of technical issues (like pounding out designs of some sort with best-of-the-best software engineers) have little overlap.
 I hope you can find your way again and do what you do best.
So do I, and he should work on the things that interest him, rather than trying to run an open source project.
Mar 31
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 10:12 AM, bachmeier wrote:
 It is clear that he has no interest in being the leader of an open 
 source project. He doesn't want to waste his time with average 
 programmers, he only wants to spend his time working with the best of 
 the best (by some metric) software engineers.
Good leadership is inspiring people from all walks of life toward doing extraordinary things. Point being, not all need to be extraordinary. Groups of extraordinary people working together do happen, but are rare enough to receive their own phrase - "Gang of Four", "Paypal Mafia", "K&R". I knew this well before embarking on working on D, and I had no illusion about my own skill being remarkable to start with. So I had no unreasonable expectations going in or staying in. What was surprising, which I alluded to in another post, is that the advice for improvement I'd dispense was frequently debated and occasionally taken as an offense. For whatever reason, I failed to hold the competence high ground. It's as if there was a samizdat manual by which contributors must do two essential things - antagonize Walter and debate my reviews. Needless to say, my growing testy didn't help one bit. There's this subtle Romanian proverb translated loosely as "it's mighty difficult helping who doesn't want to be helped". (I first heard it thirty years ago in the military and thought it's kind of silly.) This has led to many thousand yard stare evenings I have yet to recover from.
Mar 31
parent reply Rubn <where is.this> writes:
On Sunday, 31 March 2019 at 15:28:55 UTC, Andrei Alexandrescu 
wrote:
 There's this subtle Romanian proverb translated loosely as 
 "it's mighty difficult helping who doesn't want to be helped". 
 (I first heard it thirty years ago in the military and thought 
 it's kind of silly.)
Look at this comment as someone pointed out before: https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733 Do you think after reading this someone is going to be more inclined to work on a pull request or less inclined? You basically said the entire pull request was useless and that any other pull requests like it should not happen. This does not promote or nurture discussion that leads to understanding, this simply terminates it. Not sure which is worse between Walter not saying anything and leaving pull requests open for years without any additional comment. Or your comments that you think are "help". I made a pull request to implement a feature I didn't think was that big of a change. Walter left a response that the change needed a DIP. Then some years later the feature gets added without a DIP. Walter never replied to my comments requesting for clarification. Scheisse like that is why people don't feel like contributing.
Mar 31
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Sunday, 31 March 2019 at 16:18:35 UTC, Rubn wrote:

 Look at this comment as someone pointed out before:

 https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733
Yes, Andrei's review was a bit too strong but he did change course later in that thread. Also recognize that the original author did not provide much motivation or justification (though, I'm guilty of that too). I encourage contributors to please provide a short essay with your PRs to help everyone see the larger picture. No one can pay attention to everything going on, and not everyone who reviews pull requests is intimately involved with the overall long-term objective that some PRs move forward. It also makes things easier on the reviewer as they don't need to do so much of their own investigation to determine context (that always irritated me a little when I was reviewing; make it easy on me), and it increases the likelihood of a PR getting a review and getting accepted. I do ask Andrei and Walter to understand that their reviews hold *much* more weight than others. One minor objection from them makes others think "Well, Walter and Andrei don't like this, therefore my opinion is irrelevant". The fundamental problem is there isn't any real due process to challenge and ask for reconsideration. (e.g. https://github.com/dlang/dmd/pull/9506#issuecomment-477936480 -- No response). Mike When I was reviewing, it was always in the back of my mind "Are Walter and Andrei going to scold me if I merge this?", especially after they did once. In a way, we do work on behalf of Andrei and Walter and we wnat Mike
Mar 31
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Monday, 1 April 2019 at 00:01:04 UTC, Mike Franklin wrote:

 When I was reviewing, it was always in the back of my mind "Are 
 Walter and Andrei going to scold me if I merge this?", 
 especially after they did once. In a way, we do work on behalf 
 of Andrei and Walter and we wnat

 Mike
Oops, meant to delete that, but now I'm obliged to finish. ...want them to approve. Mike
Mar 31
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 1 April 2019 at 00:01:04 UTC, Mike Franklin wrote:
 On Sunday, 31 March 2019 at 16:18:35 UTC, Rubn wrote:

 Look at this comment as someone pointed out before:

 https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733
Yes, Andrei's review was a bit too strong but he did change course later in that thread.
Not to any actionable degree, the request for change is _still_ there.
 ... The fundamental problem is there isn't any real due process 
 to challenge and ask for reconsideration.  (e.g. 
 https://github.com/dlang/dmd/pull/9506#issuecomment-477936480  
 -- No response).
Yup.
 Also recognize that the original author did not provide much
 motivation or justification (though, I'm guilty of that too).
Oh I know that, this was done specifically for Iain, and therefore I gave no justification because he knew exactly what it was for, W&A came and trashed the party. Hmm, I shouldn't be so harsh, Walter's was a reasonable request (to split the bug fix from the refactor). Andrei's OTOH was a drive-by putdown, and he should learn a lesson from that that there exists a line which when crossed will cause people to cut their losses. In fact I think most (perhaps this is a memory retention bias, but still) of my dealings with Andrei in DMD have been drive-by, usually with bad outcomes.
Mar 31
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 8:01 PM, Mike Franklin wrote:
 On Sunday, 31 March 2019 at 16:18:35 UTC, Rubn wrote:
 
 Look at this comment as someone pointed out before:

 https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733
Yes, Andrei's review was a bit too strong but he did change course later in that thread.  Also recognize that the original author did not provide much motivation or justification (though, I'm guilty of that too).
Those were unnecessarily strong words. Allow me to give a bit more context whilst clarifying I am not defending or condoning that attitude. That comment came after a few phone conversations whereby Walter mentioned he has a deluge of pull requests to review. He said if he's to spend due time on all of those, he'd be unable to do any work of his own. Those pull requests that have these things in common (note that some may not apply to the pragma pull request, I'd just wrongly put it in the same bin): * they are large * the improvement they bring needs arguing * their quality could be improved so they need careful review and a couple of passes of changes This is what I call Good Work. Good Work is often sizable in quantity, and is visibly the result of concerted effort by a competent person. Those unfamiliar with a codebase and with the subtleties of the task at hand cannot produce Good Work. (Contrast that with Bad Work: by definition, it is easy to inspect and reject. It does not create a major slowdown in a project.) Good Work is more often than not complex. Complexity is misrecognized as evidence of the complexity of the problem; the task was hard, the reasoning goes, because the solution is difficult in proportion. Good Work begets more Good Work. Typically Good Work produces context, opportunity, and precedent for more of the same. The same reviewer who rubber stamped a piece of Good Work will have an idea how to produce more Good Work derived from it. The kind of environment where Good Work is revered encourages its creation, in a cycle that creates the illusion of progress. Because Good Work is complex, it produces "bug ripples" whereby increasingly complex Good Work fixes one bug but is liable to introduce others. Good Work is difficult to argue against. The main argument in favor, which is very difficult to counter, is that Good Work is better than Bad Work and better than No Work. It is easy to create an opinion trend in favor of this or that Good Work. However, there are problems with Good Work. The first one is scale: an accumulation of Good Work does not add up to Great Work. More often, the increasing entropy leads to the thermal death of the project. An accumulation of Good Work is what leads to six nested calls in an advanced library to convert a string to an integer. Even if they are inlined away in execution, their smell persists. An accumulation of Good Work is what leads to a multitude of "is this kinda sorta almost like-a string hey I really mean it this time quite like a string" tests, or "we must to add trusted every five lines of code". The other problem of Good Work is that it takes the air out of the room, causing Great Work to suffocate. Good Work takes great effort to author, debate, review, debug, and maintain. All that takes away time and mental share that should go into Great Work instead. Which brings us to Great Work. Great Work solves difficult problems with a paucity of means; it is quintessentially and surprisingly /not/ proportional response. Because of that, it is often deceptively simple, but always in a way that is impossible to anticipate yet obvious in afterthought. Scala's implicits and D's static if are great work. Great Work often reformulates an entire challenge to reveal false choices or artificial constraints. Alexander the Great figured it doesn't matter how to get rid of the Gordian knot, so he cut it. While philosophers were still puzzling over The Ship of Theseus, Richard Feynman pointed out that the very notion of identity is ill-defined because a philosopher wouldn't even be able to tell which atoms belong to a chair, and which don't. Replacing complex type-based metaprogramming with trivial compile-time evaluation is great work. Great Work is often recognizable to out-of-domain people, the person on the street. In contrast, Good Work has high-brow expertise as a prerequisite. Beethoven's Fourth Symphony is a respectable repertoire piece for any Philharmonic, one that music aficionados would listen to with respect. I once saw a janitor, a little old lady with no previous exposure to classical music, crying when she heard Bach's Air for the first time. Using the same programming language on the client and server, or at compile-time and run-time, or for computation and its constraints, is great work clear as rain. You don't need to be an expert to appreciate that. Great Work is what we should all aspire to. Great Work is the cure for Good Work. This, however, brings up a question: it seems that Great Work is not really easy to do on a regular basis. What to do, therefore, on a "regular" day when inspiration doesn't strike? This brings us to Right Work. Right Work is work that is undeniably, pound-on-the-table good, however unexciting or trivial. Right Work will be silently appreciated by one's peers as the incontestably right thing to do. Paying your rent, debts, and other bills is Right Work. Right Work is not necessarily simple. Becoming a better spouse or teaching your kids that lying is not a good policy - that's also Right Work. In a software project, reducing the number of global variables is Right Work. Making public data private is Right Work. (Careful, perversions are always possible. This is not Right work: public int percent; ==> private int _percent; int percent() { return _percent; } void percent(int p) { _percent = p; } The obscure word "phronesis" - thanks Laeeth - should be a buzzword in software engineering circles.) Reducing state and mutation is Right Work. That means adding the "immutable" qualifier to variables and the "pure" attribute to functions. Making unsafe code safe is Right Work. That means inserting " safe" wherever possible, and identifying the smallest " trusted" primitives. Replacing legacy pointer-and-length ad-hoc pairs with slices is Right Work. So is replacing a complicated unstructured loop with a structured foreach loop. Most refactorings that meaningfully reduce the number of lines of code are Right Work. Of course, that could be perverted, too. I'm not talking more statements per line of code. I'm talking more /work/ per line of code. Much Right work paradoxically requires less virtuosity than complicated artifacts of Good Work. A successful software system is a construction of Great Work on a foundation of Right Work, with the inevitable Good Work here and there. That's where we want to be.
Apr 01
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 01, 2019 at 07:22:40PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 That comment came after a few phone conversations whereby Walter
 mentioned he has a deluge of pull requests to review. He said if he's
 to spend due time on all of those, he'd be unable to do any work of
 his own. Those pull requests that have these things in common (note
 that some may not apply to the pragma pull request, I'd just wrongly
 put it in the same bin):
 
 * they are large
 * the improvement they bring needs arguing
 * their quality could be improved so they need careful review and a
 couple of passes of changes
 
 This is what I call Good Work.
[...]
 Which brings us to Great Work.
[...]
 This brings us to Right Work.
[...]
 A successful software system is a construction of Great Work on a
 foundation of Right Work, with the inevitable Good Work here and
 there.
 
 That's where we want to be.
Had this been posted openly at the time, there might have been a lot less frustration. This is an area I feel desperately needs improvement. This repeated lack of communication of this sort of thought process in one form or another is what leads to repeated frustrations and conflicts, because would-be contributors feel like information is being withheld that would have enabled them to better understand what is expected of them. Even if these were just vague thoughts or a crude outline of what you have in mind, and nowhere near something formal and "presentable", providing partial information is better than no information at all. T -- In theory, there is no difference between theory and practice.
Apr 01
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/1/2019 6:11 PM, H. S. Teoh wrote:
 Had this been posted openly at the time, there might have been a lot
 less frustration.  This is an area I feel desperately needs improvement.
 This repeated lack of communication of this sort of thought process in
 one form or another is what leads to repeated frustrations and
 conflicts, because would-be contributors feel like information is being
 withheld that would have enabled them to better understand what is
 expected of them.  Even if these were just vague thoughts or a crude
 outline of what you have in mind, and nowhere near something formal and
 "presentable", providing partial information is better than no
 information at all.
Actually, Andrei and I simply assumed this was obvious. Obviously, it wasn't.
Apr 02
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 2 April 2019 at 08:26:34 UTC, Walter Bright wrote:
 [snip]

 Actually, Andrei and I simply assumed this was obvious. 
 Obviously, it wasn't.
Maybe he should adapt it into a D blog post?
Apr 02
parent reply Rose <rose sarajevo.com> writes:
On Tuesday, 2 April 2019 at 11:15:42 UTC, jmh530 wrote:
 On Tuesday, 2 April 2019 at 08:26:34 UTC, Walter Bright wrote:
 [snip]

 Actually, Andrei and I simply assumed this was obvious. 
 Obviously, it wasn't.
Maybe he should adapt it into a D blog post?
For now on reddit. https://www.reddit.com/r/programming/comments/b8igay/great_work_is_the_cure_for_good_work/
Apr 02
parent bachmeier <no spam.net> writes:
On Tuesday, 2 April 2019 at 12:39:12 UTC, Rose wrote:
 On Tuesday, 2 April 2019 at 11:15:42 UTC, jmh530 wrote:
 On Tuesday, 2 April 2019 at 08:26:34 UTC, Walter Bright wrote:
 [snip]

 Actually, Andrei and I simply assumed this was obvious. 
 Obviously, it wasn't.
Maybe he should adapt it into a D blog post?
For now on reddit. https://www.reddit.com/r/programming/comments/b8igay/great_work_is_the_cure_for_good_work/
It's not unusual for a programmer to have strong opinions about programming. While Andrei's actions will have an impact on our community (and I was critical of some of the things he's done earlier in the thread) I don't think there's anything particularly newsworthy about this thread. I've seen nothing to suggest he's unhinged or anything other than frustrated.
Apr 02
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Apr 02, 2019 at 01:26:34AM -0700, Walter Bright via Digitalmars-d wrote:
 On 4/1/2019 6:11 PM, H. S. Teoh wrote:
 Had this been posted openly at the time, there might have been a lot
 less frustration.  This is an area I feel desperately needs
 improvement.  This repeated lack of communication of this sort of
 thought process in one form or another is what leads to repeated
 frustrations and conflicts, because would-be contributors feel like
 information is being withheld that would have enabled them to better
 understand what is expected of them.  Even if these were just vague
 thoughts or a crude outline of what you have in mind, and nowhere
 near something formal and "presentable", providing partial
 information is better than no information at all.
Actually, Andrei and I simply assumed this was obvious. Obviously, it wasn't.
Therein lies the frequent cause of miscommunications: assuming that the other party knows or understands something, which may not necessarily be the case. Especially when we're talking about random strangers on the internet, who may have had a completely different experience and background as yourself, and therefore in all likelihood understands and sees things differently. Stating all assumptions up-front helps to bridge this gap. T -- Ph.D. = Permanent head Damage
Apr 02
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 1 April 2019 at 23:22:40 UTC, Andrei Alexandrescu 
wrote:
 On 3/31/19 8:01 PM, Mike Franklin wrote:
 On Sunday, 31 March 2019 at 16:18:35 UTC, Rubn wrote:
 
 Look at this comment as someone pointed out before:

 https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733
Yes, Andrei's review was a bit too strong but he did change course later in that thread.  Also recognize that the original author did not provide much motivation or justification (though, I'm guilty of that too).
Those were unnecessarily strong words. Allow me to give a bit more context whilst clarifying I am not defending or condoning that attitude. That comment came after a few phone conversations whereby Walter mentioned he has a deluge of pull requests to review. He said if he's to spend due time on all of those, he'd be unable to do any work of his own.
_I_ do most of the reviewing now, and this doesn't seem to have had any effect, see https://github.com/dlang/dmd/pull/9494 and related PRs.
 Those pull requests that have these things in common (note that 
 some may not apply to the pragma pull request, I'd just wrongly 
 put it in the same bin):

 * they are large
 * the improvement they bring needs arguing
When they are requisites for fixing bugs they don't, as was the case with most of the ones I'm pissed off about, and Jacob's recent ones are to make DMD as a library more useful...
 * their quality could be improved so they need careful review 
 and a couple of passes of changes
(Exactly which is not at all what has, and _still is_ being done.) ... In particular you are typically seeing a snapshot of a PR when you comment, and see only that change in isolation and miss the conversation in past PR discussion about where this is going. It is perfectly fine, in fact I encourage you, to inquire why this specific change is being made, we'll point you to the previous discussion. What is _not_ fine is the drive-by style of review. This includes "Please don't let bureaucracy stand in the way of progress".
 A successful software system is a construction of Great Work on 
 a foundation of Right Work, with the inevitable Good Work here 
 and there.

 That's where we want to be.
The dlang ecosystem operates as an army of volunteers, you don't _get to choose_ the standard of work that comes in. It is was it is, and it will stay that way until such a time as it is not run by an army of volunteers. Yes, you can try to improve it (and so you should) but it needs to come off that that is your intention. Far too often, it comes off that you don't understand what or why the PR is doing what it is doing.
Apr 01
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-04-02 01:22, Andrei Alexandrescu wrote:

 This brings us to Right Work.
I'm the first to say that I'm all for improving the code base. I really hate how difficult it can be to convince management that a code base need to be improved instead of working on a feature. But, to be blunt, if you get what you want and everyone is doing Right Work as you describe it, there won't be any progress. No bugs fixed or no new features. I don't know how often you expect this Great Work to occur, which might mean progress. Unfortunately the source code is not the product, it's the application, the tool, the library or whatever it might be. -- /Jacob Carlborg
Apr 02
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 31 March 2019 at 11:30:08 UTC, Andrei Alexandrescu 
wrote:
 The necessity to work on shared was also present in the January 
 2015 vision document (https://wiki.dlang.org/Vision/2015H1): 
 "Nail down fuzzily-defined areas of the language (e.g. shared 
 semantics,  property)."

 Writing it down doesn't get it done.
Indeed thats why its on the AGM.
 Defining shared properly would take a team with a programming 
 languages expert, a threading expert, and an application 
 expert. (Some roles may be realized within the same person.) I 
 know people within our community with the required expertise.
c.f. DIP1000? I don't think you or Walter are memory safety experts, and despite the issues that occurred with the implementation (and documentation(!)) process (hindsight is 20/20), what you have achieved is quite remarkable: a blacklisted (bad things are disallowed), inductive (if all the the code I call is safe and nothing I do is on the blacklist, then so am I) model of memory safety that works! What Manu has (tried to, somewhat unsuccessfully) argue for is the same approach to thread safety with shared: tie the correct use of shared to safe/ system/ trusted, enforce the correct use of user defined and verified (i.e. trusted) thread safe primitives. Then shared (as a function parameter attribute) in safe code becomes a requirement to act in a thread safe manner with that argument under the above framework. No-one is expecting to get the implementation perfect first time, we will miss important details, and thats OK. We fix them, document them (this is the single biggest problem with the way DIP1000 was developed and the reason PRs were stalled) and move on with the satisfaction that the implementation is better than it was before.
 But I don't know any who'd also have the time to embark on this.
If the foundations can be set out, pre-approved, with sufficient clarity at dconf, then I'm confidant we can get to the point where we can iteratively improve it. Who knows, Timon might implement the whole thing in the hackathon. I don't think it will be as hard as it seems to get somewhere useful. We already have safe/ system/ trusted, shared is already a thing (albeit not very useful currently), all we need to do is define the blacklist and how the induction should work. Again, they don't need to be perfect first time.
 I got burned out on writing vision documents (Walter was never 
 a fan so I did them all) because it was difficult to figure 
 their impact. Contributors asked for a vision document. So we 
 started one. Then contributors continued doing what they were 
 doing.
I understood vision documents to set out the intended direction of the DLF, irrespective of what contributors are doing, of course they're going to continue to do what they were doing, that what volunteers do. To me they were more like a politicians promise (apologies for the analogy), a statement of intention, and if it doesn't match up to reality, then its no big deal as long as progress is still being made, the next one is written to take that into account with its projections. Anyway enough about what they were. What they will become is a byproduct of the minutes of the AGM / quarterly meetings, they will practically write themselves. If we have more (corporate) participation in them (as I hope) then chances are they will reflect reality to a much high degree of fidelity.
Mar 31
prev sibling parent reply =?UTF-8?B?THXDrXM=?= Marques <luis luismarques.eu> writes:
On Sunday, 31 March 2019 at 11:30:08 UTC, Andrei Alexandrescu 
wrote:
 I got burned out on writing vision documents (Walter was never 
 a fan so I did them all) because it was difficult to figure 
 their impact. Contributors asked for a vision document. So we 
 started one. Then contributors continued doing what they were 
 doing.
I suggest you write informal status documents in the wiki instead, on a best-effort basis. Instead of writing a document that's pie in the sky "it would be great to have X in the following 6 months, let's pretend we know we can do it", write something more like "I've been working on X, and I've noticed a lot of little problems with Y. Here's what we could do about it, what I'm trying to do Z but why it might not be enough", etc. Do little updates when possible and motivated. Maybe update it sometimes with links to pull requests that you feel helped with the problems, so others can study them and learn what things are especially welcome, etc. Things like that. In summary, play to our strengths. D has a small community. Keep it close and cozy, not distant and formal.
Mar 31
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 10:47 AM, Luís Marques wrote:
 I suggest you write informal status documents in the wiki instead, on a 
 best-effort basis.
A great idea. Walter has doing this on his twitter account: https://twitter.com/WalterBright
Mar 31
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, March 30, 2019 6:41:15 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:
 On 3/30/19 2:49 PM, Jonathan M Davis wrote:
 This would also be a great opportunity to fix some of the issues with
 shared in druntime
The problem here is, of course, that nobody knows exactly what shared is supposed to do. Not even me. Not even Walter. One monumental piece of work would be DIP 1021 "Define the semantics of shared". Then people who build with -dip1021 know what to do and what to expect. Now that would patch one of the bigger holes mentioned in the other post.
I confess that I find the amount of confusion over shared to be confusing, though maybe I'm missing something. Yes, it has some details that need to be worked out, but I would have thought that basics would be well understood by now. For shared to work, it has to prevent code that isn't thread-safe - which means either making any operation which isn't guaranteed to be thread-safe illegal and/or making the compiler insert code that guarantees thread-safety. The latter is very hard if not impossible, if nothing else, because that would require that the compiler actually understand the threading and synchronization mechanisms being used in a given situation (e.g. associating a mutex with a shared variable and then somehow guaranteeing that it's always locked at the appropriate time and unlocked at the appropriate time). So, basically, that means that any operations that aren't guaranted to be thread-safe then need to be illegal for shared, and then to actually read or modify a shared object, you have to either use atomics, or you have to use whatever synchronization mechanisms that you want to be using (e.g. mutexes), cast away shared while the object is protected, operate on the object as thread-local, make sure that no thread-local references to it exist when you're done, and then release the mutex. It's exactly what you'd be doing in C++ code except that you have to worry about casting away shared while actually operating on the object, because the type system is preventing you from shooting yourself in the foot by reading or writing to shared objects, since that's not thread-safe without synchronization mechanisms that aren't part of the type system. It seems to be confusing for many people, because they expect to actually be able to directly operate on shared objects, but you can't do that and guarantee any kind of thread-safety unless you're calling stuff that takes care of the protection for you (e.g. a type could contain a mutex and lock it in its member functions, thereby encapsulating all of the casting away of shared - which is basically what synchronized classes were supposed to do, just with the outer level only, whereas you can do more than that if you're doing it manually and write the code correctly). But shared types in general aren't necessarily going to have any kind of synchronization mechanism built in (e.g. shared(int*) certainly won't). The primary problem that I see with shared as things stand is that it still allows operations which aren't guaranteed to be thread-safe (e.g. copying). It may need some tweaks beyond that, but as far as I can tell, it mostly works as-is. However, it's much harder to use than it should be because the stuff in core.sync has only been partially made to work properly with shared. I expect that when you get down to all of the nitty gritty details of making operations that aren't guaranteed to be thread-safe illegal that there could be some hairy things that need to be sorted out, but we've already made _some_ of them illegal, and the addition of copy constructors to the language actually fixes one of the big hurdles, which is making it possible to make it thread-safe to copy an object (by having its copy constructor be shared and handle the mutex or atomics or whatnot internally). So, by no means am I claiming that we have it all figured out, but I would have thought that it would primarily be an issue of figuring out how to correctly make operations that aren't guaranteed to be thread-safe illegal (just like finishing safe by making operations system when the compiler can't guarantee that they're memory safe). Some of the details could turn out to be nasty, but I would have thought that it would just be a matter of working through them and that as things stand, the basic design of shared works. Maybe I'm missing something here, but it seems to me that it _should_ be pretty straightforward to just move forward from the idea that operations on shared objects have to be thread-safe or be illegal. That may involve more casting than would be ideal, but I don't see how we can really do much with the language understanding enough to do the casting for you, and the result is basically what you get in C++ - except that the compiler is preventing you from shooting yourself in the foot outside of system code where you're handling the casting away of shared. So, it's pretty much the same situation as safe vs system in the sense that shared code disallows operations that the compiler can't guarantee the safety of, and you have to do system stuff to guarantee the safety yourself in the cases where you need to do that stuff. Regardless of the details of what we want to do with shared though, core.sync doesn't handle it properly. Mostly, it doesn't use shared at all, and when it does, it's hacked on. So, at some point here, we really need to replace what's there with a v2 of some sort. And until we do, no matter how well shared works on its own, it's going to be seriously hampered, because the constructs in core.sync are core to writing threaded code. - Jonathan M Davis
Mar 30
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 3/30/19 2:49 PM, Jonathan M Davis wrote:
 This would also be a great opportunity to fix some of the issues with shared
 in druntime
The problem here is, of course, that nobody knows exactly what shared is supposed to do. Not even me. Not even Walter.
As an immediate stop-gap, shared must have *no read or write access* to data members. Simple. Make that change. Right now. Then shared will at least be useful and not embarrassing, and you can continue to argue about what shared means while I get to work.
 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.
Sure. But in the meantime, fix the objective bug with the current semantics where you can read/write un-protected data freely. Right now. Please for the love of god.
Mar 31
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Sunday, 31 March 2019 at 22:25:30 UTC, Manu wrote:
 Sure. But in the meantime, fix the objective bug with the 
 current semantics where you can read/write un-protected data 
 freely. Right now. Please for the love of god.
Apropos "things to do right now", to derail the thread a little in my own interest, could I again ask to decouple synchronized and shared for the time being? There is literally no way to write a class that has both threadsafety and invariant-safety right now, because D has no way to pull invariants into the class body without pulling in the `shared` mess. Look, if I'm using synchronized(this), as a rule I don't *care* about shared, because I'm inherently using another *kind* of thread safety to atomic primitives. Uncoupling shared from synchronized at the class level would make synchronized classes actually usable and useful without restructuring the entire class for no gain.
Apr 01
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 1, 2019 1:08:29 AM MDT FeepingCreature via Digitalmars-d 
wrote:
 On Sunday, 31 March 2019 at 22:25:30 UTC, Manu wrote:
 Sure. But in the meantime, fix the objective bug with the
 current semantics where you can read/write un-protected data
 freely. Right now. Please for the love of god.
Apropos "things to do right now", to derail the thread a little in my own interest, could I again ask to decouple synchronized and shared for the time being? There is literally no way to write a class that has both threadsafety and invariant-safety right now, because D has no way to pull invariants into the class body without pulling in the `shared` mess. Look, if I'm using synchronized(this), as a rule I don't *care* about shared, because I'm inherently using another *kind* of thread safety to atomic primitives. Uncoupling shared from synchronized at the class level would make synchronized classes actually usable and useful without restructuring the entire class for no gain.
How is synchronized unrelated to shared? shared is part of anything and everything in D which involves sharing data across threads. The only exception is __gshared, which is only supposed to be used for C globals (which as I understand it, can't be shared, because that wouldn't work with the linker), and you have to be _really_ careful whenever __gshared is used, because the compiler assumes that it's actually thread-local, because __gshared is not actually part of the type. synchronized is just one way to provide a mutex for protecting shared data. The data itself still needs to be shared and then usually has to have shared cast away while the data is protected by the mutex that goes with synchronized (at which point, it's up to the programmer to ensure that no thread-local references to the data escape or exist once the mutex is unlocked). synchronized classes (which have never been fully implemented) are supposed to deal with automatically casting away the outer level of shared within the member functions, but it's still the same mechanism that you have to do now manually. If you're not using shared with synchronized, then you're doing something seriously wrong - probably using __gshared, which is just begging for trouble. In general, any object using synchronized methods should be shared. - Jonathan M Davis
Apr 01
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 1 April 2019 at 10:11:04 UTC, Jonathan M Davis wrote:
 On Monday, April 1, 2019 1:08:29 AM MDT FeepingCreature via 
 Digitalmars-d wrote:
 Uncoupling shared from synchronized at the class level would 
 make synchronized classes actually usable and useful without 
 restructuring the entire class for no gain.
How is synchronized unrelated to shared?
In the one way that matters - in practice. :) I'll be honest; we use synchronized a whole lot and shared in maybe ten lines total. How? Well, in the obvious way: void method() { synchronized (this) { } }. No shared, no fuss. I would be surprised if this was not the predominant way of synchronizing classes in D in production right now, precisely because you *don't* have to bother with shared. And if you do it that way, invariants are inherently useless.
 The only exception is __gshared, which is only supposed to be 
 used for C globals
I'm sorry, but there's a very great distance here, at least in our codebase, between "only supposed to" (C globals) and "what it's actually used for" (*any* threadshared globals). Nobody wants to touch shared, but we still need threading.
Apr 01
parent FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 1 April 2019 at 12:28:37 UTC, FeepingCreature wrote:
 I'm sorry, but there's a very great distance here, at least in 
 our codebase, between "only supposed to" (C globals) and "what 
 it's actually used for" (*any* threadshared globals). Nobody 
 wants to touch shared, but we still need threading.
Sorry for the double post, but I realized I'd forgotten to explain *why* we do it that way. You will note that the way I explained threading is the way that basically any C-based language that doesn't have threading correctness as an explicit selling point does it: as a thin wrapper around pthreads. It's the "natural", "normal" way to do threading, the way that programmers have been doing it for literal decades. Threads and mutexes. If D wants people to do it in a different way, that solution needs to be slick, operational and *reliable*. Any solution that starts with "well you can just cast it away cause it doesn't really work yet" doesn't have the reaction of "well we're just gonna do it that way then, because there's no other way, and hope the D devs clean up shared one of these *years*" - the reaction is going to be the entirely predictable "well, we're just gonna go back to the C way then." That's where D is right now, and that's why I look at all the discussion about shared with a kind of amused bewilderment. Thread safety is simply fundamentally not about exposing atomic access to data in our minds. Thread safety is about having classes, that own data, access to which they can lock to one thread. Synchronized(this). That's the established, commonsense, baseline idiom. It shouldn't be surprising that we prefer it to randomly casting away modifiers that the language randomly forces on us despite them being highly experimental and subject to change. Get the normal, established way to do threading working consistently and reliably *now*. Then think about shared. That's my opinion, at least.
Apr 01
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 6:25 PM, Manu wrote:
 On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 3/30/19 2:49 PM, Jonathan M Davis wrote:
 This would also be a great opportunity to fix some of the issues with shared
 in druntime
The problem here is, of course, that nobody knows exactly what shared is supposed to do. Not even me. Not even Walter.
As an immediate stop-gap, shared must have *no read or write access* to data members. Simple. Make that change. Right now. Then shared will at least be useful and not embarrassing, and you can continue to argue about what shared means while I get to work.
 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.
Sure. But in the meantime, fix the objective bug with the current semantics where you can read/write un-protected data freely. Right now. Please for the love of god.
Incidentally Walter and I discussed in the early days the idea that "shared" would offer no access whatsoever aside from a few special functions in druntime such as "atomicRead" and "atomicWrite". Of course, that raised the question how those functions could be implemented - the possible solution being some casts for typing and asm for the needed memory barriers. In the end we got scared that there was no precedent in other languages, and we could not predict whether that would have been good or bad. The result is the current semantics, which should be a felony in the 48 contiguous US states. This does not work as a two stages process, though the "stop the bleeding first then come with the new solution" metaphor seems attractive. The main issues being when we break code that people got to work, we need to offer the alternative as well. Another being that the exact kind of things we disable/enable may be dependent on the ultimate solution. This would be a large effort requiring a strong team. Walter, yourself, and I would be helpful participants but I think between the three of us we don't have the theoretical chops to pull this off. At least I know I don't. We need the likes of Timon Gehr, Johan Engelen, and David Nadlinger (whom I cc'd just in case).
Apr 03
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 4 April 2019 at 02:05:15 UTC, Andrei Alexandrescu 
wrote:
 On 3/31/19 6:25 PM, Manu wrote:
 Sure. But in the meantime, fix the objective bug with the 
 current
 semantics where you can read/write un-protected data freely. 
 Right
 now. Please for the love of god
This does not work as a two stages process, though the "stop the bleeding first then come with the new solution" metaphor seems attractive. The main issues being when we break code that people got to work, we need to offer the alternative as well. Another being that the exact kind of things we disable/enable may be dependent on the ultimate solution.
Well whatever happens I'll be gobsmacked if its not behind an opt in switch. With that in mind, if Manu gets use out of the stopgap of disabling read/write access, then I think we should implement that ASAP and then listen to whatever he complains about next ;)
 This would be a large effort requiring a strong team. Walter, 
 yourself, and I would be helpful participants but I think 
 between the three of us we don't have the theoretical chops to 
 pull this off. At least I know I don't. We need the likes of 
 Timon Gehr, Johan Engelen, and David Nadlinger (whom I cc'd 
 just in case).
I don't think we are going to be able to do this without iterating on the design and closing holes and nuisances that we discover. I'm not saying that it is a bad idea to design up front as much as we can, but we shouldn't wast time getting hung up on design when implementation can give gains to users and guidance to the design.
Apr 03
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/3/19 11:09 PM, Nicholas Wilson wrote:
 On Thursday, 4 April 2019 at 02:05:15 UTC, Andrei Alexandrescu wrote:
 On 3/31/19 6:25 PM, Manu wrote:
 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god
This does not work as a two stages process, though the "stop the bleeding first then come with the new solution" metaphor seems attractive. The main issues being when we break code that people got to work, we need to offer the alternative as well. Another being that the exact kind of things we disable/enable may be dependent on the ultimate solution.
Well whatever happens I'll be gobsmacked if its not behind an opt in switch. With that in mind, if Manu gets use out of the stopgap of disabling read/write access, then I think we should implement that ASAP and then listen to whatever he complains about next ;)
 This would be a large effort requiring a strong team. Walter, 
 yourself, and I would be helpful participants but I think between the 
 three of us we don't have the theoretical chops to pull this off. At 
 least I know I don't. We need the likes of Timon Gehr, Johan Engelen, 
 and David Nadlinger (whom I cc'd just in case).
I don't think we are going to be able to do this without iterating on the design and closing holes and nuisances that we discover. I'm not saying that it is a bad idea to design up front as much as we can, but we shouldn't wast time getting hung up on design when implementation can give gains to users and guidance to the design.
I don't think this works for programming language design. In fact I'm positive it doesn't. It's the way we've done things so far.
Apr 03
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/4/19 12:24 AM, Andrei Alexandrescu wrote:
 On 4/3/19 11:09 PM, Nicholas Wilson wrote:
 On Thursday, 4 April 2019 at 02:05:15 UTC, Andrei Alexandrescu wrote:
 On 3/31/19 6:25 PM, Manu wrote:
 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god
This does not work as a two stages process, though the "stop the bleeding first then come with the new solution" metaphor seems attractive. The main issues being when we break code that people got to work, we need to offer the alternative as well. Another being that the exact kind of things we disable/enable may be dependent on the ultimate solution.
Well whatever happens I'll be gobsmacked if its not behind an opt in switch. With that in mind, if Manu gets use out of the stopgap of disabling read/write access, then I think we should implement that ASAP and then listen to whatever he complains about next ;)
 This would be a large effort requiring a strong team. Walter, 
 yourself, and I would be helpful participants but I think between the 
 three of us we don't have the theoretical chops to pull this off. At 
 least I know I don't. We need the likes of Timon Gehr, Johan Engelen, 
 and David Nadlinger (whom I cc'd just in case).
I don't think we are going to be able to do this without iterating on the design and closing holes and nuisances that we discover. I'm not saying that it is a bad idea to design up front as much as we can, but we shouldn't wast time getting hung up on design when implementation can give gains to users and guidance to the design.
I don't think this works for programming language design. In fact I'm positive it doesn't. It's the way we've done things so far.
Well I'm exaggerating. I mean to say every time we did it that way, the result hasn't been good.
Apr 03
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 4 April 2019 at 04:25:59 UTC, Andrei Alexandrescu 
wrote:
 On 4/4/19 12:24 AM, Andrei Alexandrescu wrote:
 On 4/3/19 11:09 PM, Nicholas Wilson wrote:
 I don't think we are going to be able to do this without 
 iterating on the design and closing holes and nuisances that 
 we discover. I'm not saying that it is a bad idea to design 
 up front as much as we can, but we shouldn't wast time 
 getting hung up on design when implementation can give gains 
 to users and guidance to the design.
I don't think this works for programming language design. In fact I'm positive it doesn't. It's the way we've done things so far.
Well I'm exaggerating. I mean to say every time we did it that way,
Examples please?
 the result hasn't been good.
e.g. DIP1000 was bad, not because it was iterated upon to fix the holes in it, but because the changes were not communicated properly and not documented. I suggest we don't make those same mistakes again.
Apr 03
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/4/19 1:50 AM, Nicholas Wilson wrote:
 On Thursday, 4 April 2019 at 04:25:59 UTC, Andrei Alexandrescu wrote:
 On 4/4/19 12:24 AM, Andrei Alexandrescu wrote:
 On 4/3/19 11:09 PM, Nicholas Wilson wrote:
 I don't think we are going to be able to do this without iterating 
 on the design and closing holes and nuisances that we discover. I'm 
 not saying that it is a bad idea to design up front as much as we 
 can, but we shouldn't wast time getting hung up on design when 
 implementation can give gains to users and guidance to the design.
I don't think this works for programming language design. In fact I'm positive it doesn't. It's the way we've done things so far.
Well I'm exaggerating. I mean to say every time we did it that way,
Examples please?
Shared itself, the postblit, lazy, properties, alias this - are all "first-order thinking" ideas that are not bad, but fail to take into consideration second-order interactions and their consequences. (A good read: https://fs.blog/2016/04/second-order-thinking/) Language design is all about second-order thinking.
 the result hasn't been good.
e.g. DIP1000 was bad, not because it was iterated upon to fix the holes in it, but because the changes were not communicated properly and not documented. I suggest we don't make those same mistakes again.
DIP1000 is actually an example of second-order thinking. Walter pored over it for months before writing and implementing it. So are the recently-introduced copy constructors. We had what we thought was a workable design at probably one dozen times during the process. All had large flaws. Incrementalism is an anti-pattern in language design.
Apr 04
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 4 April 2019 at 11:10:09 UTC, Andrei Alexandrescu 
wrote:
 On 4/4/19 1:50 AM, Nicholas Wilson wrote:
 e.g. DIP1000 was bad, not because it was iterated upon to fix 
 the holes in it, but because the changes were not communicated 
 properly and not documented. I suggest we don't make those 
 same mistakes again.
DIP1000 is actually an example of second-order thinking. Walter pored over it for months before writing and implementing it.
That is was a good idea does not excuse how sloppily the procedure of implementation was handled. I note also that it underwent significant changes post implementation. That is the iteration I'm taking about (just handled better, i.e. docs & community engagement).
 Incrementalism is an anti-pattern in language design.
I'm talking about design iteration, not language by incremental feature addition. Incremental feature addition is difficult, if not impossible, to undo if it turns out it was a bad idea. Iterative design, by definition, does not suffer from that problem.
Apr 04
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/4/2019 4:10 AM, Andrei Alexandrescu wrote:
 DIP1000 is actually an example of second-order thinking. Walter pored over it 
 for months before writing and implementing it.
So far, the only real shortcoming in the initial design was revealed by the put() semantics, and was fixed with that PR that transmitted scope-ness through the first argument. On the other hand, while it would be great if we could anticipate all second order effects, it's unrealistic to always get that right, and no language does.
Apr 04
prev sibling parent Manu <turkeyman gmail.com> writes:
On Wed, Apr 3, 2019 at 9:25 PM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 4/3/19 11:09 PM, Nicholas Wilson wrote:
 On Thursday, 4 April 2019 at 02:05:15 UTC, Andrei Alexandrescu wrote:
 On 3/31/19 6:25 PM, Manu wrote:
 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god
This does not work as a two stages process, though the "stop the bleeding first then come with the new solution" metaphor seems attractive. The main issues being when we break code that people got to work, we need to offer the alternative as well. Another being that the exact kind of things we disable/enable may be dependent on the ultimate solution.
Well whatever happens I'll be gobsmacked if its not behind an opt in switch. With that in mind, if Manu gets use out of the stopgap of disabling read/write access, then I think we should implement that ASAP and then listen to whatever he complains about next ;)
 This would be a large effort requiring a strong team. Walter,
 yourself, and I would be helpful participants but I think between the
 three of us we don't have the theoretical chops to pull this off. At
 least I know I don't. We need the likes of Timon Gehr, Johan Engelen,
 and David Nadlinger (whom I cc'd just in case).
I don't think we are going to be able to do this without iterating on the design and closing holes and nuisances that we discover. I'm not saying that it is a bad idea to design up front as much as we can, but we shouldn't wast time getting hung up on design when implementation can give gains to users and guidance to the design.
I don't think this works for programming language design. In fact I'm positive it doesn't. It's the way we've done things so far.
You say your original design worked how I suggest (I'm not surprised, it's the only thing that makes sense), so... close the circuit! Maybe it was a success, and nobody ever had the chance to demonstrate it. We've waited so long to try it, so let us try it out!
Apr 03
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, April 3, 2019 8:05:15 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:
 On 3/31/19 6:25 PM, Manu wrote:
 On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:
 On 3/30/19 2:49 PM, Jonathan M Davis wrote:
 This would also be a great opportunity to fix some of the issues with
 shared in druntime
The problem here is, of course, that nobody knows exactly what shared is supposed to do. Not even me. Not even Walter.
As an immediate stop-gap, shared must have *no read or write access* to data members. Simple. Make that change. Right now. Then shared will at least be useful and not embarrassing, and you can continue to argue about what shared means while I get to work.
 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what
 to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.
Sure. But in the meantime, fix the objective bug with the current semantics where you can read/write un-protected data freely. Right now. Please for the love of god.
Incidentally Walter and I discussed in the early days the idea that "shared" would offer no access whatsoever aside from a few special functions in druntime such as "atomicRead" and "atomicWrite". Of course, that raised the question how those functions could be implemented - the possible solution being some casts for typing and asm for the needed memory barriers. In the end we got scared that there was no precedent in other languages, and we could not predict whether that would have been good or bad. The result is the current semantics, which should be a felony in the 48 contiguous US states. This does not work as a two stages process, though the "stop the bleeding first then come with the new solution" metaphor seems attractive. The main issues being when we break code that people got to work, we need to offer the alternative as well. Another being that the exact kind of things we disable/enable may be dependent on the ultimate solution. This would be a large effort requiring a strong team. Walter, yourself, and I would be helpful participants but I think between the three of us we don't have the theoretical chops to pull this off. At least I know I don't. We need the likes of Timon Gehr, Johan Engelen, and David Nadlinger (whom I cc'd just in case).
shared already tries to prevent various read/write operations that aren't guaranteed to be thread-safe. So, it's already gone partially in that direction, and if it's supposed to guarantee any kind of thread-safety, that's the only way that it can go unless it's going to try to insert code that ensures thread-safety - which would mean doing stuff like having mutexes built in to what it's doing so that it knows what to lock and when, and that's not something that's going to work as a low level construct. Even the concept of synchronized classes as proposed in TDPL couldn't go far enough with it to be very useful, because it could only safely cast away the outer layer of shared. I thought that it was pretty clear that the way we were going - and have to go - is to make it so that you simply can't read or write to shared data without either using operations that are guaranteed to be thread-safe or by casting away shared (at which point, the code is system, and it's up to the programmer to get it right, and that code is properly segregated thanks to how safe and trusted work). I grant you that we really do need the best people we have to look at this at the low level and make sure that it works properly, but I don't see where else we can go with this if we want any kind of guarantees from shared. If we decide that we don't want shared to provide guarantees and that we want to allow shared data to be read and written to regardless of whether the compiler can guarantee thread-safety, then we can do that, but then things that are currently illegal should be made legal (e.g. IIRC, incrementing shared integers is illegal), and at that point, shared is just indicating which variables are shared across threads without actually protecting you at all. That would put it more in line with languages like C++, but it also would mean that it would be a lot less useful for catching bugs. Either way, what we have right now is halfway in between. It makes some stuff that isn't thread-safe illegal but doesn't make it all illegal, making it a pain to use if you don't cast away shared like you would need to if all operations which weren't guaranteed to be thread-safe were illegal but simultaneously failing to actually protect you and guarantee that the code with operations that are potentially not thread-safe is system. It looks to me like it has to go one way or the other. Either make it all illegal, thus requiring code that deals directly with synchronization mechanisms to use casts and be trusted - or make all of the read/write operations on shared objects legal and just restrict it in that it can't be converted to or from thread-local without a cast. - Jonathan M Davis
Apr 03
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Wed, Apr 3, 2019 at 7:10 PM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 3/31/19 6:25 PM, Manu wrote:
 On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 3/30/19 2:49 PM, Jonathan M Davis wrote:
 This would also be a great opportunity to fix some of the issues with shared
 in druntime
The problem here is, of course, that nobody knows exactly what shared is supposed to do. Not even me. Not even Walter.
As an immediate stop-gap, shared must have *no read or write access* to data members. Simple. Make that change. Right now. Then shared will at least be useful and not embarrassing, and you can continue to argue about what shared means while I get to work.
 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.
Sure. But in the meantime, fix the objective bug with the current semantics where you can read/write un-protected data freely. Right now. Please for the love of god.
Incidentally Walter and I discussed in the early days the idea that "shared" would offer no access whatsoever aside from a few special functions in druntime such as "atomicRead" and "atomicWrite". Of course, that raised the question how those functions could be implemented - the possible solution being some casts for typing and asm for the needed memory barriers.
Casting away shared seems to be the only reasonable option with respect to this general design for shared. This applies just fine to atomics; casting away shared is effectively asserting that you have created a thread-local context for some window of time that you can perform a thread-safe interaction. Atomic operations have an effective zero-length execution window, so from that perspective, it's correct that you are able to cast-away shared in order to perform a single atomic operation on an int. The assertion that you have a thread-local lease on the int for the duration of an atomic operation naturally holds by definition.
 In the end we got scared that there was no precedent in other languages,
 and we could not predict whether that would have been good or bad. The
 result is the current semantics, which should be a felony in the 48
 contiguous US states.
Okay... so you're effectively saying you had a possibly-good idea, but instead of trying it, instead did something else that's just straight-up broken? So, like, how about we actually try the thing before we decide it didn't work? We have nothing to lose!
 This does not work as a two stages process, though the "stop the
 bleeding first then come with the new solution" metaphor seems
 attractive.
You're saying a design was proposed, and it's *almost* implemented in the language. It's been there for longer than I've been around, but we're still yet to actually try out the design as is almost implemented so long ago. It's really weird that we're declaring the design failed before trying it out! We've had, like... 12 years to try it out? Why haven't we tried it yet? We're not bleeding from a failed design. If there is a wound in place, it's the fact that the intended design was only half-implemented, and then left that way for over a decade. Making this change would give something to work with. If it turns out it's not a workable solution after all, then it would be good to know before we eject it into space! Better design may emerge from understanding how this design failed. But it hasn't failed yet, because we haven't actually tried it yet... it's just been sitting in limbo waiting while people scratch their head and try to understand what happened here. That process is surely more useful than the current situation, which is that `shared` means nothing, and nobody quite knows what it's for other than a sort of self-documentation. I understand you may prefer a strong design proposal, but nobody has moved the bar in the decade I've been waiting. We're clearly not making ground in the way you prefer, so let's just actually implement the design almost-implemented 12(+?) years ago, and see how terrible it actually is? I have a fairly strong sense of what will emerge, I suspect it'll be workable and useful.
 The main issues being when we break code that people got to
 work, we need to offer the alternative as well. Another being that the
 exact kind of things we disable/enable may be dependent on the ultimate
 solution.
I'm fairly sure we're not 'breaking' anything. Any code that breaks with this change is almost certainly already broken. The remedy would be to add a cast to their code, and it will be exactly as it is now; probably still broken. But it's really not very disruptive.
 This would be a large effort requiring a strong team. Walter, yourself,
 and I would be helpful participants but I think between the three of us
 we don't have the theoretical chops to pull this off. At least I know I
 don't. We need the likes of Timon Gehr, Johan Engelen, and David
 Nadlinger (whom I cc'd just in case).
Go for it... but like, maybe first, how about we actually try out the design you came up with 12 years ago before we declare it a failure and spend another few years trying to do something else? We have no evidence it's a failure, only that the half-implementation of the original design is a failure, and mostly for the reason that the half-semantics just don't mean anything, not that the original idea is broken.
Apr 03
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On the plus side of shared as it is at the moment, and it's a pretty big plus, 
is it identifies what data is shared, and how it's integrated into the type 
system is good.

With C and C++, when you've got a shared memory bug, you have no idea where to 
start. In D, you look at the shared stuff. You're not going to have 
inadvertently shared variables.
Apr 04
next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
On Thursday, 4 April 2019 at 21:05:51 UTC, Walter Bright wrote:
 On the plus side of shared as it is at the moment, and it's a 
 pretty big plus, is it identifies what data is shared, and how 
 it's integrated into the type system is good.

 With C and C++, when you've got a shared memory bug, you have 
 no idea where to start. In D, you look at the shared stuff. 
 You're not going to have inadvertently shared variables.
I mean, given that synchronized classes don't strip out the first layer of shared, as is necessary to make them, basically, *at all* usable, we're currently forced to use synchronized(this) ... which does not force shared at all. So this seems optimistic. Thinking more about it, I feel shared is a fundamentally wrong direction. Even stripping out the first layer, I'll not be able to naturally read or modify anything with references, say an int array, because the language can't know that nothing else has a view on the array. I'd be limited to arrays of immutable data, with all the "can never overwrite a field, even though I own the array" problems that brings. Istm that the more natural way to use shared in the context of classes would be to talk not in terms of whether data is shared or not, but whether data is *owned* by the class. If I transitively owned the array in a synchronized class, I could overwrite what I want at will, confident that any access would be happening behind a synchronized block. ie. struct S { int[] data; // reference } synchronized class S { private owned S[] array; // valid despite shared - array is not shared, because we own it. void test1(size_t index) { array[index] = owned(S)(null); } // not valid - `data` references caller data; we can't take ownership of it. void test2(int[] data) { array[index] = owned(S)(data); } // valid - we have exclusive ownership on data.dup void test3(int[] data) { array[index] = owned(S)(data.dup); } // invalid - can't escape owned data. owned(S) test4() { return array.front; } // valid S test5() { return S(array.front.data.dup); } } As far as I can see, in production use synchronization is about atomic access only in the very rarest of cases. I can't imagine why a domain class would ever return anything shared (who would trust the outside world to know how to correctly access it?), and inside class methods, presuming we own our data, as classes *should anyways*, shared has no benefit because there's only one reader/writer anyways. In comparison, imo, owned would formalize the way that threadsafe classes should be written *anyways*.
Apr 05
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/4/19 5:05 PM, Walter Bright wrote:
 On the plus side of shared as it is at the moment, and it's a pretty big 
 plus, is it identifies what data is shared, and how it's integrated into 
 the type system is good.
 
 With C and C++, when you've got a shared memory bug, you have no idea 
 where to start. In D, you look at the shared stuff. You're not going to 
 have inadvertently shared variables.
Such an outlook means stagnation and smothers progress. Shared is nowhere near where it should be. Internalizing that is key to getting it right. I've said this many many times privately and publicly: If we never make the argument "yeah but C++ is worse" again, that would be too soon.
Apr 05
prev sibling parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Thursday, 4 April 2019 at 21:05:51 UTC, Walter Bright wrote:
 On the plus side of shared as it is at the moment, and it's a 
 pretty big plus, is it identifies what data is shared, and how 
 it's integrated into the type system is good.

 With C and C++, when you've got a shared memory bug, you have 
 no idea where to start. In D, you look at the shared stuff. 
 You're not going to have inadvertently shared variables.
Sorry, Walter. But it is indeed easy to track down shared memory bugs than introducing design level bugs with shared.
Apr 05
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, March 31, 2019 4:25:30 PM MDT Manu via Digitalmars-d wrote:
 On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:
 On 3/30/19 2:49 PM, Jonathan M Davis wrote:
 This would also be a great opportunity to fix some of the issues with
 shared in druntime
The problem here is, of course, that nobody knows exactly what shared is supposed to do. Not even me. Not even Walter.
As an immediate stop-gap, shared must have *no read or write access* to data members. Simple. Make that change. Right now. Then shared will at least be useful and not embarrassing, and you can continue to argue about what shared means while I get to work.
 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.
Sure. But in the meantime, fix the objective bug with the current semantics where you can read/write un-protected data freely. Right now. Please for the love of god.
Honestly, that's really what shared is missing. _Some_ operations which aren't guaranteed to be thread-safe are illegal, but many aren't. With that fixed, I don't know what else shared would actually need. The stuff in core.sync really needs to be fixed up to use shared correctly (which is critical for being able to use shared cleanly), but at that point, shared itself would be doing what it needs to be doing. We could then potentially add stuff like synchronized classes on top of that, but that's just making shared easier to use, not actually required for shared to work. - Jonathan M Davis
Apr 01
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 11:04:58 AM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:
 On 3/28/19 9:16 AM, ag0aep6g wrote:
 On 28.03.19 14:05, Andrei Alexandrescu wrote:
 Then some ranges are not meant to be assignable.
Should Phobos be compatible with those ranges?
A variety of algorithm and data structures in Phobos are relying on assignment. Presumably a good part of them can be converted with ease to use single assignment, but not all. The overall message is we got bogged down on the "wrong" side of generality - cross-cutting and nonscalable code additions to support unprincipled and low-impact corner cases. Part of that is we've been cagey about defining copy and assignment semantics of ranges in a simple and comprehensive manner. It seems to me going with these is the right thing: * Input ranges are copyable and assignable, and have pointer semantics (all copies refer to the same underlying position, and advancing one advances all others). * Forward ranges are copyable and assignable, but distinct copies refer to distinct positions in the range such that advancing one does not advance the others. * We don't support other semantics.
I've thought for a while now that this is what we should do if we could start over. The problem has been that it breaks a ton of code, and a migration path is difficult. Doing a v2 version of Phobos would really help with making fixes like that - though then we have to deal with stuff like whether a library is written to work with v1 or v2, and with stuff like ranges effectively working via duck typing, that could get a bit interesting. If we can do it without splitting the community though, it would allow us to fix some of the big mistakes we made with our core stuff (auto-decoding being another). - Jonathan M Davis
Mar 30
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 11:42:50 AM MDT H. S. Teoh via Digitalmars-d 
wrote:
 On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via
 Digitalmars-d wrote: [...]

 Part of that is we've been cagey about defining copy and assignment
 semantics of ranges in a simple and comprehensive manner. It seems to
 me going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics
 (all copies refer to the same underlying position, and advancing one
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one does
 not advance the others.

 * We don't support other semantics.
What about classes that wrap ranges? E.g., std.ranges.interfaces. Since classes are reference types, this would mean it's impossible to use those interfaces with forward ranges or above, which is a show-stopping limitation (e.g., if you need to return two different range types selected at runtime -- the current solution is to wrap them in the appropriate class using std.ranges.interfaces).
It's not a problem for basic input ranges, because they'd then have reference semantics. The problem would just be forward ranges, and the solution to that is to use wrapper ranges which are structs. Then they can define a copy constructor which gives value semantics in place of save. The reality of the matter though is that code should be _really_ wary of using classes for forward ranges, because it requires calling new a lot, and it's really inefficient. So, while we probably should support it for those use cases that really need it, if code doesn't really need it, it shouldn't do it. For instance, I tested dxml with a whole range of range types (including ranges that were classes) when benchmarking, and performance tanked when using ranges that were classes because of all of the memory allocation that was happening. - Jonathan M Davis
Mar 30
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 5:37 AM, Andrei Alexandrescu wrote:
 The conclusion is not to operate such changes everywhere (i.e. reason by 
 analogy). The right conclusion is that save() is unnecessarily general and 
 underspecified.
I've opined that `save()` should be implemented using copy construction, not assignment.
Mar 28
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 19.03.19 03:52, Andrei Alexandrescu wrote:
 
 Turns out the second clause fails. That takes us to the definition of 
 empty in the same module:
 
  property bool empty(T)(auto ref scope const(T) a)
 if (is(typeof(a.length) : size_t))
 {
      return !a.length;
 }
 ...
Why should length be required to be `const` when almost everything that's `const` is not even a range?
 The intent is fairly clear - if a range defines empty as a size_t 
 (somewhat oddly relaxed to "convertible to size_t"), then empty can be 
 nicely defined in terms of length. Cool. But empty doesn't work with 
 TestAliasedString due to an overlooked matter: the "const". A mutable 
 TestAliasedString converts to a string, but a const or immutable 
 TestAliasedString does NOT convert to a const string! So this fixes that 
 matter:
 
      struct TestAliasedString
      {
          string get()  safe  nogc pure nothrow { return _s; }
          const(string) get()  safe  nogc pure nothrow const { return
_s; }
          alias get this;
           disable this(this);
          string _s;
      }
 
Reasoning from first principles, the right fix is actually to allow non-const length.
 That makes empty() work, but also raises a nagging question: what was 
 the relationship of TestAliasedString to string before this change? 
 Surely that wasn't subtyping. (My response would be: "Odd.") And why was 
 Phobos under the obligation to cater for such a type and its tenuous 
 relationship to a range?
The problem isn't `alias this`. It's `const`. The following type does not pass isInputRange either: struct ApparentlyNotARange{ size_t k=10; int front(){ return 0; } void popFront(){ --k; } property size_t length(){ return k; } } While this does: struct ApparentlyNotARange{ size_t k=10; int front(){ return 0; } void popFront(){ --k; } property size_t length()const{ return k; } } The reason why Phobos ought to be under the obligation to cater for such a type is because failure to do so is an artificial limitation. It works by default, but someone went out of their way to make sure it does not work if `length` is not annotated `const` (or ` property`). This is particularly strange because there is no such thing as a const range.
Apr 01
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/19 5:23 PM, Timon Gehr wrote:
 Reasoning from first principles, the right fix is actually to allow 
 non-const length.
D'accordo. Can you please put forth a PR to that effect?
Apr 01
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/19 8:04 PM, Andrei Alexandrescu wrote:
 On 4/1/19 5:23 PM, Timon Gehr wrote:
 Reasoning from first principles, the right fix is actually to allow 
 non-const length.
D'accordo. Can you please put forth a PR to that effect?
My attempts to delegate go a lot like my wife's attempts to get me to help around the house :o). https://github.com/dlang/phobos/pull/6944
Apr 05