www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Generality creep

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter and I were looking at indexOf in the standard library. It has 
several overloads, of which let's look at the first two:

https://dlang.org/library/std/string/index_of.html

Now, why couldn't they be merged into a single function with this 
constraint (their bodies are identical):

ptrdiff_t indexOf(Range) (
   Range s,
   dchar c,
   CaseSensitive cs = Yes.caseSensitive
)
if (isInputRange!Range && isSomeChar!(ElementType!Range));

It makes sense to look for a character in any range of characters.

Attempting to build that fails because of this type fails to work:

     struct TestAliasedString
     {
         string get()  safe  nogc pure nothrow { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

The intuition is that the function should be general enough to figure 
out that, hey, TestAliasedString is kinda sorta a subtype of string. So 
the call should work.

So let's see why it doesn't work - i.e. why is TestAliasedString an 
input range? The definition of isInputRange (in std.range.primitives) is:

enum bool isInputRange(R) =
     is(typeof(R.init) == R)
     && is(ReturnType!((R r) => r.empty) == bool)
     && is(typeof((return ref R r) => r.front))
     && !is(ReturnType!((R r) => r.front) == void)
     && is(typeof((R r) => r.popFront));

Turns out the second clause fails. That takes us to the definition of 
empty in the same module:

 property bool empty(T)(auto ref scope const(T) a)
if (is(typeof(a.length) : size_t))
{
     return !a.length;
}

The intent is fairly clear - if a range defines empty as a size_t 
(somewhat oddly relaxed to "convertible to size_t"), then empty can be 
nicely defined in terms of length. Cool. But empty doesn't work with 
TestAliasedString due to an overlooked matter: the "const". A mutable 
TestAliasedString converts to a string, but a const or immutable 
TestAliasedString does NOT convert to a const string! So this fixes that 
matter:

     struct TestAliasedString
     {
         string get()  safe  nogc pure nothrow { return _s; }
         const(string) get()  safe  nogc pure nothrow const { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

That makes empty() work, but also raises a nagging question: what was 
the relationship of TestAliasedString to string before this change? 
Surely that wasn't subtyping. (My response would be: "Odd.") And why was 
Phobos under the obligation to cater for such a type and its tenuous 
relationship to a range?

But wait, there's more. Things still don't work because of popFront. 
Looking at its definition:

void popFront(C)(scope ref inout(C)[] str)  trusted pure nothrow
if (isNarrowString!(C[]))
{ ... }

So, reasonably this function takes the range by reference so it can 
modify its internals. HOWEVER! The implementation of 
TestAliasedString.get() returns an rvalue, i.e. it's the equivalent of a 
conversion involving a temporary. Surely that's not to match, whether in 
the current language or the one after the rvalue DIP.

The change that does make the code work is:

     struct TestAliasedString
     {
         ref string get()  safe  nogc pure nothrow { return _s; }
         ref const(string) get()  safe  nogc pure nothrow const { return 
_s; }
         alias get this;
          disable this(this);
         string _s;
     }

This indeed does implement a subtyping relationship, and passes the 
isInputRange test.

What's the moral of the story here? Generality is good, but it seems in 
several places in phobos (of which this is just one example), a 
combination of vague specification and aiming for a nice ideal of "work 
with anything remotely reasonable" has backfired into a morass of 
inconsistently defined and supported corner cases.

For this case in particular - I don't think we should support all types 
that support some half-hearted form of subtyping, at the cost of 
reducing generality and deprecating working code.
Mar 18 2019
next sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:

  property bool empty(T)(auto ref scope const(T) a)
 if (is(typeof(a.length) : size_t))
 {
     return !a.length;
 }

 The intent is fairly clear - if a range defines empty as a 
 size_t (somewhat oddly relaxed to "convertible to size_t"), 
 then empty can be nicely defined in terms of length.


If it was intended for arrays only, then
 property bool empty(T)(scope const T[] a)
{
     return !a.length;
}

But if not, what is the reason to define length, but not empty? 
Anyway, the type may need to be mutable, e.g. it might want to 
allocate and store the array, so
 property bool empty(T)(auto ref scope T a)
if (is(typeof(a.length) : size_t))
{
     return !a.length;
}


 That makes empty() work, but also raises a nagging question: 
 what was the relationship of TestAliasedString to string before 
 this change? Surely that wasn't subtyping. (My response would 
 be: "Odd.") And why was Phobos under the obligation to cater 
 for such a type and its tenuous relationship to a range?


The design rationale is not documented :) so we can only guess.
My guess is that it was done for some string container 
(appender?) that returns rvalue string, so the change to ref 
return would be incorrect.


 But wait, there's more. Things still don't work because of 
 popFront. Looking at its definition:

 void popFront(C)(scope ref inout(C)[] str)  trusted pure nothrow
 if (isNarrowString!(C[]))
 { ... }

 So, reasonably this function takes the range by reference so it 
 can modify its internals. HOWEVER! The implementation of 
 TestAliasedString.get() returns an rvalue, i.e. it's the 
 equivalent of a conversion involving a temporary. Surely that's 
 not to match, whether in the current language or the one after 
 the rvalue DIP.


That's conflation of collections and ranges. A collection would 
be a range factory.


 What's the moral of the story here? Generality is good, but it 
 seems in several places in phobos (of which this is just one 
 example), a combination of vague specification and aiming for a 
 nice ideal of "work with anything remotely reasonable" has 
 backfired into a morass of inconsistently defined and supported 
 corner cases.

 For this case in particular - I don't think we should support 
 all types that support some half-hearted form of subtyping, at 
 the cost of reducing generality and deprecating working code.


Isn't generality one of phobos acceptance criteria?
Mar 19 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/19/2019 2:01 AM, Kagamin wrote:

 Isn't generality one of phobos acceptance criteria?


I'm not so sure about that. Should Phobos work on abstractions that have a 
square wheel? Or only well-formed abstractions? In my experience, when an 
abstraction has a square wheel, it's genesis was a poor understanding of how to 
build an abstraction, not a requirement of that abstraction. Hence, rejecting
it 
helps the programmer build better abstractions.

I.e. should Phobos accept things because it can, or because it should?
Mar 19 2019
prev sibling next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:

 And why was Phobos under the obligation to cater for such a 
 type and its tenuous relationship to a range?


There's two paths to investigate this:

1. Use git blame to find the commit when it was introduced, and 
follow the thread to the pull request, Bugzilla issue, etc.

2. Open a draft PR with a fix, and see what breaks (both within D 
repositories i.e. auto-tester, and in the D ecosystem at large 
i.e. the project tester).

Good to see you working on D actively again BTW.
Mar 19 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/19/19 9:04 AM, Vladimir Panteleev wrote:

 On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu wrote:

 And why was Phobos under the obligation to cater for such a type and 
 its tenuous relationship to a range?


 
 There's two paths to investigate this:
 
 1. Use git blame to find the commit when it was introduced, and follow 
 the thread to the pull request, Bugzilla issue, etc.
 
 2. Open a draft PR with a fix, and see what breaks (both within D 
 repositories i.e. auto-tester, and in the D ecosystem at large i.e. the 
 project tester).


Thanks. I'm less preoccupied with figuring out who wrote that code, as 
with addressing the systemic matter. My working hypothesis is a (milder) 
form of "the road to hell is paved with good intentions" - some arguably 
undesirable generality has accumulated slowly as many contributors with 
varying design sensibilities operated under the vague desideratum that 
more generality is better, ran into a variety of complex cases, and 
contributed their solution to the standard library.

I got word a similar phenomenon is at work in Scala - the language and 
its libraries are very powerful, but the boundaries of power are fuzzy 
and some users are in a continuous state of frustration that some 
constructs that ought to work do not due to byzantine limitations.

At any rate, there seems to be some good opportunity to significantly 
simplifying the standard library by dropping some of the more obscure 
supported cases.
Mar 20 2019
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:

 Walter and I were looking at indexOf in the standard library. 
 It has several overloads, of which let's look at the first two:

 [...]


+1 that these corner cases should not be supported in phobos, but 
at this point how could phobos drop support for these cases? In 
fact I'm pretty sure I've written a fair amount of code that 
depends on them (i.e. using alias this to subtype with an 
rvalue...hey, we all make mistakes). Of course I'm more than 
willing to fix my code but we're talking about a huge breaking 
change here. What's the path forward?
Mar 19 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/19/2019 6:35 AM, Jonathan Marler wrote:

 +1 that these corner cases should not be supported in phobos, but at this
point 
 how could phobos drop support for these cases? In fact I'm pretty sure I've 
 written a fair amount of code that depends on them (i.e. using alias this to 
 subtype with an rvalue...hey, we all make mistakes). Of course I'm more than 
 willing to fix my code but we're talking about a huge breaking change here. 
 What's the path forward?


I'd suggest making a separate overload for the square wheels, and deprecate
them.
Mar 19 2019
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/18/19 10:52 PM, Andrei Alexandrescu wrote:

  property bool empty(T)(auto ref scope const(T) a)
 if (is(typeof(a.length) : size_t))
 {
      return !a.length;
 }
 
 The intent is fairly clear - if a range defines empty as a size_t 
 (somewhat oddly relaxed to "convertible to size_t"), then empty can be 
 nicely defined in terms of length. Cool.


This is implemented wrong.

import std.range;

struct S
{
    size_t length() const { return 0; }
}

void main()
{
    S s;
    assert(s.length == 0); OK
    assert(s.empty); // Error, cannot deduce function
}

The problem is a simple one: typeof(a.length) is not size_t, but a 
function type.

If you put  property on the length function it would work, but I think 
we should not require that.

-Steve
Mar 19 2019
parent reply Kagamin <spam here.lot> writes:
On Tuesday, 19 March 2019 at 14:08:24 UTC, Steven Schveighoffer 
wrote:

 The problem is a simple one: typeof(a.length) is not size_t, 
 but a function type.

 If you put  property on the length function it would work, but 
 I think we should not require that.

 -Steve


struct S
{
    size_t length() const { return 0; }
}

auto ref property(T)(auto ref T a){ return a; }

void f()
{
    S s;
    assert(s.length == 0); //OK
    static assert(is(typeof(property(s.length))==size_t));
}

Can this work?
Mar 20 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/20/19 1:41 PM, Kagamin wrote:

 On Tuesday, 19 March 2019 at 14:08:24 UTC, Steven Schveighoffer wrote:

 The problem is a simple one: typeof(a.length) is not size_t, but a 
 function type.

 If you put  property on the length function it would work, but I think 
 we should not require that.

 -Steve


 
 struct S
 {
     size_t length() const { return 0; }
 }
 
 auto ref property(T)(auto ref T a){ return a; }
 
 void f()
 {
     S s;
     assert(s.length == 0); //OK
     static assert(is(typeof(property(s.length))==size_t));
 }
 
 Can this work?


Yep, that's a good idea actually. Make it inout, to cut down on template 
bloat.

I actually use something less nice in iopipe[1], so I may switch to this.

-Steve

[1] 
https://github.com/schveiguy/iopipe/blob/master/source/iopipe/traits.d#L111-L119
Mar 20 2019
next sibling parent reply Kagamin <spam here.lot> writes:
On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer 
wrote:

 Yep, that's a good idea actually. Make it inout, to cut down on 
 template bloat.


T property(T)(auto ref T a);

This still compiles. Will it bloat?
Mar 21 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/21/19 12:03 PM, Kagamin wrote:

 On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer wrote:

 Yep, that's a good idea actually. Make it inout, to cut down on 
 template bloat.


 
 T property(T)(auto ref T a);
 
 This still compiles. Will it bloat?


It's not the auto ref, it's the mutability:

auto ref property1(T)(auto ref T a)
{
   pragma(msg, T.stringof ~ " without inout");
   return a;
}

auto ref property2(T)(auto ref inout(T) a)
{
    pragma(msg, T.stringof ~ " with inout");
    return a;
}

void main()
{
    const int a = 1;
    immutable int b = 2;
    int c = 3;

    // 3 instantiations
    assert(a.property1 == a);
    assert(b.property1 == b);
    assert(c.property1 == c);

    // 1 instantiation
    assert(a.property2 == a);
    assert(b.property2 == b);
    assert(c.property2 == c);
}

output:

const(int) without inout
immutable(int) without inout
int without inout
int with inout

-Steve
Mar 22 2019
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer 
wrote:

 On 3/20/19 1:41 PM, Kagamin wrote:

 struct S
 {
     size_t length() const { return 0; }
 }
 
 auto ref property(T)(auto ref T a){ return a; }
 
 void f()
 {
     S s;
     assert(s.length == 0); //OK
     static assert(is(typeof(property(s.length))==size_t));
 }
 
 Can this work?


 Yep, that's a good idea actually. Make it inout, to cut down on 
 template bloat.

 I actually use something less nice in iopipe[1], so I may 
 switch to this.


Rather than define a new `property` symbol, why not rewrite the 
template constraint for `empty` as:


 property bool empty (T) (auto ref scope T a)
     if(is(ReturnType!((T t) => t.length) : size_t))
{
     return !a.length;
}

This matches what is done in the `isInputRange` check for the 
existence of `empty` (I suspect it's done that way because it's 
quite common for `SomeRange.empty` to be an enum rather than a 
method or property).
Mar 21 2019
next sibling parent reply Kagamin <spam here.lot> writes:
On Thursday, 21 March 2019 at 16:11:04 UTC, Joseph Rushton 
Wakeling wrote:

  property bool empty (T) (auto ref scope T a)
     if(is(ReturnType!((T t) => t.length) : size_t))
 {
     return !a.length;
 }


Here comes generality creep :)

struct A
{
     int a;
      disable this(this);
     bool opCast(){ return !!a; }
}
struct S
{
     A a;
     ref A p() { return a; }
}

T property(T)(auto ref T a);
import std.traits;

void f()
{
     S s;
     assert(!s.p); //negation works
     static assert(is(typeof(property(s.p))==A));
     static assert(is(ReturnType!(()=>s.p):A));
}

The ReturnType assert doesn't pass. Any idea why? Well, maybe it 
shouldn't.
Mar 21 2019
parent Kagamin <spam here.lot> writes:
Apparently there are options

T property(T)(scope T a);
will check that the value can be retained from the property for 
local use

T property(T)(T a);
will check that the value can be returned from function (not 
scoped)
Mar 22 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 21, 2019 10:11:04 AM MDT Joseph Rushton Wakeling via 
Digitalmars-d wrote:

 On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer

 wrote:

 On 3/20/19 1:41 PM, Kagamin wrote:

 struct S
 {

     size_t length() const { return 0; }

 }

 auto ref property(T)(auto ref T a){ return a; }

 void f()
 {

     S s;
     assert(s.length == 0); //OK
     static assert(is(typeof(property(s.length))==size_t));

 }

 Can this work?


 Yep, that's a good idea actually. Make it inout, to cut down on
 template bloat.

 I actually use something less nice in iopipe[1], so I may
 switch to this.


 Rather than define a new `property` symbol, why not rewrite the
 template constraint for `empty` as:


  property bool empty (T) (auto ref scope T a)
      if(is(ReturnType!((T t) => t.length) : size_t))
 {
      return !a.length;
 }

 This matches what is done in the `isInputRange` check for the
 existence of `empty` (I suspect it's done that way because it's
 quite common for `SomeRange.empty` to be an enum rather than a
 method or property).


Honestly, this is another example of over-generality causing problems.
length shouldn't ever be anything other than size_t. Anything else causes
problems.

- Jonathan M Davis
Mar 30 2019
prev sibling next sibling parent 9il <ilyayaroshenko gmail.com> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:

 Walter and I were looking at indexOf in the standard library.

 What's the moral of the story here? Generality is good, but it 
 seems in several places in phobos (of which this is just one 
 example), a combination of vague specification and aiming for a 
 nice ideal of "work with anything remotely reasonable" has 
 backfired into a morass of inconsistently defined and supported 
 corner cases.


Phobos looks too generalized for me and not enough standardized. 
For (well known) example, min/max functions for integers. Both 
for security and safety reasons it is better to raise a 
compile-time error rather than to implement clever signity-aware 
comparison. I did this in Mir. There a lot of such small nitpicks 
but their reimplementation make life either for middle size and 
large projects. It would be nice to have well-standardized 
generic building blocks like `mir`/`max` in Phobos with 
verifiable BetterC availability.

Best,
Ilya
Mar 19 2019
prev sibling next sibling parent Rubn <where is.this> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:

 Walter and I were looking at indexOf in the standard library. 
 It has several overloads, of which let's look at the first two:

 https://dlang.org/library/std/string/index_of.html

 Now, why couldn't they be merged into a single function with 
 this constraint (their bodies are identical):

 ptrdiff_t indexOf(Range) (
   Range s,
   dchar c,
   CaseSensitive cs = Yes.caseSensitive
 )
 if (isInputRange!Range && isSomeChar!(ElementType!Range));

 It makes sense to look for a character in any range of 
 characters.

 Attempting to build that fails because of this type fails to 
 work:

     struct TestAliasedString
     {
         string get()  safe  nogc pure nothrow { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

 The intuition is that the function should be general enough to 
 figure out that, hey, TestAliasedString is kinda sorta a 
 subtype of string. So the call should work.

 So let's see why it doesn't work - i.e. why is 
 TestAliasedString an input range? The definition of 
 isInputRange (in std.range.primitives) is:

 enum bool isInputRange(R) =
     is(typeof(R.init) == R)
     && is(ReturnType!((R r) => r.empty) == bool)
     && is(typeof((return ref R r) => r.front))
     && !is(ReturnType!((R r) => r.front) == void)
     && is(typeof((R r) => r.popFront));

 Turns out the second clause fails. That takes us to the 
 definition of empty in the same module:

  property bool empty(T)(auto ref scope const(T) a)
 if (is(typeof(a.length) : size_t))
 {
     return !a.length;
 }

 The intent is fairly clear - if a range defines empty as a 
 size_t (somewhat oddly relaxed to "convertible to size_t"), 
 then empty can be nicely defined in terms of length. Cool. But 
 empty doesn't work with TestAliasedString due to an overlooked 
 matter: the "const". A mutable TestAliasedString converts to a 
 string, but a const or immutable TestAliasedString does NOT 
 convert to a const string! So this fixes that matter:

     struct TestAliasedString
     {
         string get()  safe  nogc pure nothrow { return _s; }
         const(string) get()  safe  nogc pure nothrow const { 
 return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

 That makes empty() work, but also raises a nagging question: 
 what was the relationship of TestAliasedString to string before 
 this change? Surely that wasn't subtyping. (My response would 
 be: "Odd.") And why was Phobos under the obligation to cater 
 for such a type and its tenuous relationship to a range?

 But wait, there's more. Things still don't work because of 
 popFront. Looking at its definition:

 void popFront(C)(scope ref inout(C)[] str)  trusted pure nothrow
 if (isNarrowString!(C[]))
 { ... }

 So, reasonably this function takes the range by reference so it 
 can modify its internals. HOWEVER! The implementation of 
 TestAliasedString.get() returns an rvalue, i.e. it's the 
 equivalent of a conversion involving a temporary. Surely that's 
 not to match, whether in the current language or the one after 
 the rvalue DIP.

 The change that does make the code work is:

     struct TestAliasedString
     {
         ref string get()  safe  nogc pure nothrow { return _s; }
         ref const(string) get()  safe  nogc pure nothrow const 
 { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }

 This indeed does implement a subtyping relationship, and passes 
 the isInputRange test.

 What's the moral of the story here? Generality is good, but it 
 seems in several places in phobos (of which this is just one 
 example), a combination of vague specification and aiming for a 
 nice ideal of "work with anything remotely reasonable" has 
 backfired into a morass of inconsistently defined and supported 
 corner cases.

 For this case in particular - I don't think we should support 
 all types that support some half-hearted form of subtyping, at 
 the cost of reducing generality and deprecating working code.


So basically one more reason to avoid const entirely?
Mar 19 2019
prev sibling next sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 19 March 2019 at 02:52:34 UTC, Andrei Alexandrescu 
wrote:

 The change that does make the code work is:

     struct TestAliasedString
     {
         ref string get()  safe  nogc pure nothrow { return _s; }
         ref const(string) get()  safe  nogc pure nothrow const 
 { return _s; }
         alias get this;
          disable this(this);
         string _s;
     }


Isn't this exactly the kind of code where you're supposed to use 
inout?
Mar 20 2019
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
A topical example:

https://github.com/dlang/phobos/pull/6346#issuecomment-477504005
Mar 28 2019
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 09:52, Walter Bright wrote:

 A topical example:
 
 https://github.com/dlang/phobos/pull/6346#issuecomment-477504005


That pull request was open for a year(!) waiting for someone to make a 
decision. And before that I had asked about the issue on the forum [1].

If you want things to go your way, consider answering when someone asks 
for direction.

By the way, I didn't get an email notification from GitHub for that 
comment. Does GitHub stop sending those when the pull request is merged? 
That would be worrying. Did anyone else get a notificaton?


[1] https://forum.dlang.org/post/p96gs4$22fp$1 digitalmars.com
Mar 28 2019
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 7:02 AM, ag0aep6g wrote:

 On 28.03.19 09:52, Walter Bright wrote:

 A topical example:

 https://github.com/dlang/phobos/pull/6346#issuecomment-477504005


 
 That pull request was open for a year(!) waiting for someone to make a 
 decision. And before that I had asked about the issue on the forum [1].
 
 If you want things to go your way, consider answering when someone asks 
 for direction.


That's good work actually (which underlines the irony of the situation) 
- once refRange got in the question of supporting it came naturally. 
Generality crept in, one good intention at a time.

We should use D's good name lookup mechanism to create a versioning 
strategy a la std.v2 that reuses the good parts of std and fixes the bad 
parts. That way we focus on adding good things and letting bad things in 
maintenance mode, as opposed to changing existing things. Even C++, with 
its rather poor modularity mechanism, has made std::tr1 and std::tr2 work.
Mar 28 2019
prev sibling next sibling parent ag0aep6g <anonymous example.com> writes:
On 28.03.19 12:02, ag0aep6g wrote:

 By the way, I didn't get an email notification from GitHub for that 
 comment.


Scratch that, it arrived. Just took a couple of hours.
Mar 28 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 4:02 AM, ag0aep6g wrote:

 If you want things to go your way, consider answering when someone asks for 
 direction.


I simply cannot keep a finger on everything that is happening, and have to rely 
on delegating. This one caught my eye because I had just finished making those 
ranges DIP1000 compatible, and then some large changes to them followed right 
afterwards, and I thought they might be related.
Mar 28 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 22:33, Walter Bright wrote:

 This one caught my eye because I had just 
 finished making those ranges DIP1000 compatible, and then some large 
 changes to them followed right afterwards, and I thought they might be 
 related.


I don't mind you being against those changes, but the changes so far are 
hardly "large". Ignoring the tests, both my PRs combined add meager 20 
lines. And that's for seven bug fixes.

By the way, you recently made changes to `save` methods in std.utf:

https://github.com/dlang/phobos/pull/6900
https://github.com/dlang/phobos/pull/6903

I'm not sure why you made those, but they touch the same kind of code as 
my PRs, and they actually fix the issue I was fixing, too. Are you 
planning on applying that pattern to all of Phobos, or just std.utf for 
some reason?

If you're going for all of Phobos, you might end up fixing issue 18657 
as a side effect. But removing RefRange.opAssign would still be nice, 
because it's just weird.
Mar 28 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 4:05 PM, ag0aep6g wrote:

 I don't mind you being against those changes, but the changes so far are
hardly 
 "large". Ignoring the tests, both my PRs combined add meager 20 lines. And 
 that's for seven bug fixes.


Right, but those changes cut across a large number of algorithms, and will
cause 
every algorithm implementer to do the same. That makes it a very large change. 
I'm glad I noticed this before a lot more work was expended in that direction.



 By the way, you recently made changes to `save` methods in std.utf:
 
 https://github.com/dlang/phobos/pull/6900
 https://github.com/dlang/phobos/pull/6903
 
 I'm not sure why you made those,


The problem I ran into was ranges that had const fields would fail, because one 
cannot assign to a const field. But one can construct a const field. Doing some 
digging, it became apparent that save() is equivalent to copy construction, and 
so should be implemented using copy construction.

Assignment and construction are fundamentally different, and save() is clearly
a 
construction.



 but they touch the same kind of code as my PRs, 
 and they actually fix the issue I was fixing, too. Are you planning on
applying 
 that pattern to all of Phobos, or just std.utf for some reason?


My immediate goal was simply to get phobos to compile with DIP1000. Over time,
I 
expect a well-implemented save() function should use copy construction. I did 
write a PR to that effect:

https://github.com/dlang/phobos/pull/6905/files

Your advice on that would be appreciated.



 If you're going for all of Phobos, you might end up fixing issue 18657 as a
side 
 effect. But removing RefRange.opAssign would still be nice, because it's just 
 weird.


Fixing RefRange.save() using copy construction, and ditching opAssign, would be 
acceptable.
Mar 28 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 29.03.19 00:30, Walter Bright wrote:

 The problem I ran into was ranges that had const fields would fail, 
 because one cannot assign to a const field. But one can construct a 
 const field. Doing some digging, it became apparent that save() is 
 equivalent to copy construction, and so should be implemented using copy 
 construction.


Is Phobos supposed to support ranges with const fields? Won't that be 
generality creep?

A quick test shows that all five examples from issue 18657 fail for 
ranges with const fields. With my changes, the first three work, and the 
other two don't fail in `save` anymore, but still in other places.

Supporting const fields will require at least the same amount of 
busywork as supporting RefRange, because the underlying issue is the 
same: Can ranges be non-assignable and should Phobos support such ranges?

[...]

 Fixing RefRange.save() using copy construction, and ditching opAssign, 
 would be acceptable.


How does RefRange.save need fixing?
Mar 28 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 5:40 PM, ag0aep6g wrote:

 On 29.03.19 00:30, Walter Bright wrote:

 The problem I ran into was ranges that had const fields would fail, because 
 one cannot assign to a const field. But one can construct a const field. Doing 
 some digging, it became apparent that save() is equivalent to copy 
 construction, and so should be implemented using copy construction.


 
 Is Phobos supposed to support ranges with const fields? Won't that be
generality 
 creep?


Phobos already has such ranges, such as Repeat.

Besides, if you accept the notion that save() performs a construction, not an 
assignment, and that construction and assignment are fundamentally different, 
then save() must be implemented using construction.

I can't think of a case where this would break existing code.



 A quick test shows that all five examples from issue 18657 fail for ranges
with 
 const fields. With my changes, the first three work, and the other two don't 
 fail in `save` anymore, but still in other places.
 
 Supporting const fields will require at least the same amount of busywork as 
 supporting RefRange, because the underlying issue is the same: Can ranges be 
 non-assignable and should Phobos support such ranges?


Ranges don't need to be assignable. ForwardRanges need to supply a save(),
which 
means copy construction, not assignment.



 Fixing RefRange.save() using copy construction, and ditching opAssign, would 
 be acceptable.


 
 How does RefRange.save need fixing?


I thought you wrote that copy construction would make RefRange work?
Mar 28 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 29.03.19 02:55, Walter Bright wrote:

 Ranges don't need to be assignable. ForwardRanges need to supply a 
 save(), which means copy construction, not assignment.


Issue 18657 is all about RefRange not being assignable (without nasty 
surprises). Changing Phobos to avoid assignment of ranges will fix the 
issue.

All the mighty talk about "generality creep" and "first principles" is 
besides the point. You agree with making changes all over Phobos. You 
just don't like the weird `move` code. That's okay. I agree that fixing 
the issue with constructors is nicer.

[...]

 How does RefRange.save need fixing?


 
 I thought you wrote that copy construction would make RefRange work?


Implementing the `save` methods of higher ranges that way makes them 
compatible with RefRange, because they no longer call RefRange.opAssign.

RefRange.save is fine as it is (w.r.t issue 18657).
Mar 29 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 29.03.19 08:41, ag0aep6g wrote:

 I agree that fixing the issue with constructors is nicer.


PR to change my `move`s to constructor calls:

https://github.com/dlang/phobos/pull/6939
Mar 29 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/29/2019 7:28 AM, ag0aep6g wrote:

 https://github.com/dlang/phobos/pull/6939


Thank you
Apr 02 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 4:52 AM, Walter Bright wrote:

 A topical example:
 
 https://github.com/dlang/phobos/pull/6346#issuecomment-477504005


See also the related issue: https://issues.dlang.org/show_bug.cgi?id=18657

Indeed so, thanks. The entire thing with ranges whereby save() does 
something else than a mere copy has been a mistake and supporting it has 
been an expensive distraction.

The intent of save() was as a mere flag to signal that the range is a 
forward range and not an input range. It should have been a simpler 
policy - e.g. ranges that advertise being forward ranges would define 
this inside:

enum bool isForward = true;

Ranges should have one of two categories:

* for input ranges, all copies share and advance the same underlying 
position;

* for forward ranges, each copy holds its own position.

Anything else is not a range.

There's a lot of good stuff in Phobos, and then there's a lot of fat 
added to support an ill-intended notion of generality.
Mar 28 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 12:12, Andrei Alexandrescu wrote:

 See also the related issue: https://issues.dlang.org/show_bug.cgi?id=18657
 
 Indeed so, thanks. The entire thing with ranges whereby save() does 
 something else than a mere copy has been a mistake and supporting it has 
 been an expensive distraction.


`save` is a red herring with regards to issue 18657.

The problem isn't RefRange.save, it's RefRange.opAssign. The issue 
manifests often in implementations of `save`, but that's just because 
they *assign* the result of `save` over an existing range.

Note that there's no `save` call in issue 18657's third example (`choose`).

The question is whether ranges are allowed to implement opAssign like 
RefRange does. If so, all of Phobos needs fixing to avoid calling 
opAssign. If not, RefRange.opAssign should be removed.
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 7:45 AM, ag0aep6g wrote:

 On 28.03.19 12:12, Andrei Alexandrescu wrote:

 See also the related issue: 
 https://issues.dlang.org/show_bug.cgi?id=18657

 Indeed so, thanks. The entire thing with ranges whereby save() does 
 something else than a mere copy has been a mistake and supporting it 
 has been an expensive distraction.


 
 `save` is a red herring with regards to issue 18657.
 
 The problem isn't RefRange.save, it's RefRange.opAssign.


The problem is RefRange.
Mar 28 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 12:51, Andrei Alexandrescu wrote:

 The problem is RefRange.


too terse
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 7:59 AM, ag0aep6g wrote:

 On 28.03.19 12:51, Andrei Alexandrescu wrote:

 The problem is RefRange.


 
 too terse


Reasoning from first principles would go as follows. Context: algorithms 
are expressed naturally in terms of ranges, and composite (higher-order) 
ranges are built and work with ranges. Event: RefRange comes along, 
exposes an oddity ("save(), copying, and assignment could do slightly 
different things"), breaks a bunch of these structures, and requires 
arcane changes.

The conclusion is not to operate such changes everywhere (i.e. reason by 
analogy). The right conclusion is that save() is unnecessarily general 
and underspecified.
Mar 28 2019
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 8:37 AM, Andrei Alexandrescu wrote:

 On 3/28/19 7:59 AM, ag0aep6g wrote:

 On 28.03.19 12:51, Andrei Alexandrescu wrote:

 The problem is RefRange.


 too terse


 
 Reasoning from first principles would go as follows. Context: algorithms 
 are expressed naturally in terms of ranges, and composite (higher-order) 
 ranges are built and work with ranges. Event: RefRange comes along, 
 exposes an oddity ("save(), copying, and assignment could do slightly 
 different things"), breaks a bunch of these structures, and requires 
 arcane changes.
 
 The conclusion is not to operate such changes everywhere (i.e. reason by 
 analogy). The right conclusion is that save() is unnecessarily general 
 and underspecified.


Similar issues:

"People may attempt to use Phobos algorithms with ranges of enums based 
on character types, or other subtypes of character types." Reasoning by 
analogy: those should work, too. Leads to generality creep. Reasoning 
from first principles: UTF8 is predicated on one byte type, no real gain 
to be made.

"There are several ways of iterating strings, e.g. by code point, by 
code unit, or by grapheme." Reasoning by analogy: strings should be 
ranges, hence we need to choose one default mode of iteration. Reasoning 
from first principles: strings are wholes, not ranges, and iteration is 
chosen as needed.

"There are UTF8, UTF16, and UTF32 encodings." Reasoning by analogy: they 
should all work the same. Reasoning from first principles: what's really 
needed here? Realistically nobody uses UTF32, UTF16 is an evolutionary 
dead-end still alive mostly as a mild annoyance on Windows systems, and 
then everybody and their cat is using UTF8. It follows we should 
consolidate on one encapsulated string type that is NOT an array 
(because arrays are ranges) and that offers iteration modes on demand. 
All that gobbledygook with e.g. file and path functions supporting lazy 
ranges of characters should go. It's an unnecessary and expensive to 
maintain distraction.
Mar 28 2019
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 13:37, Andrei Alexandrescu wrote:

 The conclusion is not to operate such changes everywhere (i.e. reason by 
 analogy). The right conclusion is that save() is unnecessarily general 
 and underspecified.


If you want to resolve the issue by amending the definition of ranges, 
that's very fine with me. I'd much prefer that solution over going 
through all of Phobos.

But if you focus on `save`, you're missing the point of issue 18657, 
which is opAssign. A range can have a misbehaving opAssign without being 
a forward range.
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 9:04 AM, ag0aep6g wrote:

 On 28.03.19 13:37, Andrei Alexandrescu wrote:

 The conclusion is not to operate such changes everywhere (i.e. reason 
 by analogy). The right conclusion is that save() is unnecessarily 
 general and underspecified.


 
 If you want to resolve the issue by amending the definition of ranges, 
 that's very fine with me. I'd much prefer that solution over going 
 through all of Phobos.
 
 But if you focus on `save`, you're missing the point of issue 18657, 
 which is opAssign. A range can have a misbehaving opAssign without being 
 a forward range.


Then some ranges are not meant to be assignable.
Mar 28 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 14:05, Andrei Alexandrescu wrote:

 Then some ranges are not meant to be assignable.


Should Phobos be compatible with those ranges?
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 9:16 AM, ag0aep6g wrote:

 On 28.03.19 14:05, Andrei Alexandrescu wrote:

 Then some ranges are not meant to be assignable.


 
 Should Phobos be compatible with those ranges?


A variety of algorithm and data structures in Phobos are relying on 
assignment. Presumably a good part of them can be converted with ease to 
use single assignment, but not all.

The overall message is we got bogged down on the "wrong" side of 
generality - cross-cutting and nonscalable code additions to support 
unprincipled and low-impact corner cases.

Part of that is we've been cagey about defining copy and assignment 
semantics of ranges in a simple and comprehensive manner. It seems to me 
going with these is the right thing:

* Input ranges are copyable and assignable, and have pointer semantics 
(all copies refer to the same underlying position, and advancing one 
advances all others).

* Forward ranges are copyable and assignable, but distinct copies refer 
to distinct positions in the range such that advancing one does not 
advance the others.

* We don't support other semantics.
Mar 28 2019
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 1:04 PM, Andrei Alexandrescu wrote:

 On 3/28/19 9:16 AM, ag0aep6g wrote:

 On 28.03.19 14:05, Andrei Alexandrescu wrote:

 Then some ranges are not meant to be assignable.


 Should Phobos be compatible with those ranges?


 
 A variety of algorithm and data structures in Phobos are relying on 
 assignment. Presumably a good part of them can be converted with ease to 
 use single assignment, but not all.
 
 The overall message is we got bogged down on the "wrong" side of 
 generality - cross-cutting and nonscalable code additions to support 
 unprincipled and low-impact corner cases.
 
 Part of that is we've been cagey about defining copy and assignment 
 semantics of ranges in a simple and comprehensive manner. It seems to me 
 going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer semantics 
 (all copies refer to the same underlying position, and advancing one 
 advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies refer 
 to distinct positions in the range such that advancing one does not 
 advance the others.
 
 * We don't support other semantics.


Forget to add - no more save(). We just use some sort of flag and simple 
introspection.
Mar 28 2019
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 3/28/19 1:05 PM, Andrei Alexandrescu wrote:

 Part of that is we've been cagey about defining copy and assignment 
 semantics of ranges in a simple and comprehensive manner. It seems to 
 me going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics 
 (all copies refer to the same underlying position, and advancing one 
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies 
 refer to distinct positions in the range such that advancing one does 
 not advance the others.

 * We don't support other semantics.


 
 Forget to add - no more save(). We just use some sort of flag and simple 
 introspection.


I say "Hooray!" to that!

I've long felt that 'save()' seemed a rather strange duplication of 
assignment/copying (or perhaps more accurately, as Walter pointed out, 
copy construction).

That seemed ugly enough in and of itself, but to make matters much 
worse, this in turn left the semantics for passing and assigning ranges 
underdefined and, in effect, needed to treated by algorithms as 
undefined behavior. And THAT meant that nearly any occurrence of failing 
to pass ranges by ref (surprisingly common in phobos, at least at the 
time I noticed this) were much guaranteed to be a bug factory (In my 
experience, this was especially problematic when working with output 
ranges.)
Apr 04 2019
prev sibling next sibling parent reply =?UTF-8?B?THXDrXM=?= Marques <luis luismarques.eu> writes:
On Thursday, 28 March 2019 at 17:04:58 UTC, Andrei Alexandrescu 
wrote:

 The overall message is we got bogged down on the "wrong" side 
 of generality - cross-cutting and nonscalable code additions to 
 support unprincipled and low-impact corner cases.


Having a document that formally specifies what a range is and 
what its properties are might have helped with that and a lot 
more. It would still help with all of the issues yet to come.
Mar 28 2019
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 1:33 PM, Luís Marques wrote:

 On Thursday, 28 March 2019 at 17:04:58 UTC, Andrei Alexandrescu wrote:

 The overall message is we got bogged down on the "wrong" side of 
 generality - cross-cutting and nonscalable code additions to support 
 unprincipled and low-impact corner cases.


 
 Having a document that formally specifies what a range is and what its 
 properties are might have helped with that and a lot more. It would 
 still help with all of the issues yet to come.


Affirmative. I'm thinking of putting together a DIP for that.
Mar 28 2019
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 05:33:33PM +0000, Luís Marques via Digitalmars-d wrote:

 On Thursday, 28 March 2019 at 17:04:58 UTC, Andrei Alexandrescu wrote:

 The overall message is we got bogged down on the "wrong" side of
 generality - cross-cutting and nonscalable code additions to support
 unprincipled and low-impact corner cases.


 
 Having a document that formally specifies what a range is and what its
 properties are might have helped with that and a lot more. It would
 still help with all of the issues yet to come.


Yes, the range API needs a formal, precise spec that leaves no holes.
Currently, the semantics of .save is one such issue.

Another issue is what we used to call "transience": is the return value
of .front required to persist past the next invocation of popFront?
std.stdio.File.byLine is a big example that does *not* respect this, and
as a result several Phobos algorithms cannot be used with it without
incurring buggy behaviour.  OTOH, supporting transience means code like
the following won't work:

	auto e = r.front;
	r.popFront;
	if (r.front == e) ...

because e becomes invalid after .popFront.  Last time I checked, several
Phobos algorithms freely use this kind of construct (saving the value of
.front and checking it later), which means it doesn't work with
transient ranges.

We need a clear decision as to whether transient ranges are supported.
If not, std.stdio.File.byLine needs to be deprecated. Or if they should
be supported, then the range spec should clearly state that it is
illegal to save the value of .front and expect it to remain valid after
.popFront is called. (Personally, I lean towards the latter option, but
either way, this needs to be stated clearly and explicitly in the range
spec.)


T

-- 
Nearly all men can stand adversity, but if you want to test a man's character,
give him power. -- Abraham Lincoln
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 1:49 PM, H. S. Teoh wrote:

 On Thu, Mar 28, 2019 at 05:33:33PM +0000, Luís Marques via Digitalmars-d
wrote:

 On Thursday, 28 March 2019 at 17:04:58 UTC, Andrei Alexandrescu wrote:

 The overall message is we got bogged down on the "wrong" side of
 generality - cross-cutting and nonscalable code additions to support
 unprincipled and low-impact corner cases.


 Having a document that formally specifies what a range is and what its
 properties are might have helped with that and a lot more. It would
 still help with all of the issues yet to come.


 
 Yes, the range API needs a formal, precise spec that leaves no holes.
 Currently, the semantics of .save is one such issue.
 
 Another issue is what we used to call "transience": is the return value
 of .front required to persist past the next invocation of popFront?
 std.stdio.File.byLine is a big example that does *not* respect this, and
 as a result several Phobos algorithms cannot be used with it without
 incurring buggy behaviour.


Yah, for such we need I think a more primitive notion of 
UnbufferedRange. An unbuffered range of T has only one API:

bool fetchNext(ref T);

Note that the user provides all state, which is interesting. The 
primitives fills the object and moves to the next one. Returns true on 
success. Returns persistently false at end of range even if called 
multiple times. An optional interface (for efficiency's sake) would be:

size_t fetchNextN(scope T[]);

which fills a full array and returns the number of elements filled.

These are trivially implementable from the outside for arrays etc.

Then we'd have input ranges, which have a buffer of at least one 
element, i.e. today's input ranges. Input ranges that are not forward 
ranges are liable to reuse their buffers, so after a call to front(), a 
call to popFront() may overwrite the current front. This is because by 
construction input ranges that are not forward ranges do not iterate 
objects in memory, but instead they transfer data from somewhere else 
into memory, chunkwise.

Forward ranges and above are guaranteed to walk real objects in memory 
and as such provide better guarantees on the persistence of their contents.


 OTOH, supporting transience means code like
 the following won't work:
 
 	auto e = r.front;
 	r.popFront;
 	if (r.front == e) ...
 
 because e becomes invalid after .popFront.  Last time I checked, several
 Phobos algorithms freely use this kind of construct (saving the value of
 .front and checking it later), which means it doesn't work with
 transient ranges.
 
 We need a clear decision as to whether transient ranges are supported.
 If not, std.stdio.File.byLine needs to be deprecated. Or if they should
 be supported, then the range spec should clearly state that it is
 illegal to save the value of .front and expect it to remain valid after
 .popFront is called. (Personally, I lean towards the latter option, but
 either way, this needs to be stated clearly and explicitly in the range
 spec.)


If we support any notion of input range, what follows is a sequence of 
forced moves: we must support iteration of objects that are not in 
memory, hence we need to support transiency of data.
Mar 28 2019
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 3/28/19 2:10 PM, Andrei Alexandrescu wrote:

 
 Yah, for such we need I think a more primitive notion of 
 UnbufferedRange. An unbuffered range of T has only one API:
 
 bool fetchNext(ref T);
 
 Note that the user provides all state, which is interesting. The 
 primitives fills the object and moves to the next one. Returns true on 
 success. Returns persistently false at end of range even if called 
 multiple times. An optional interface (for efficiency's sake) would be:
 
 size_t fetchNextN(scope T[]);
 
 which fills a full array and returns the number of elements filled.


Something like that is definitely needed. I remember needing to augment 
the existing range concept with something like that when I was dealing 
with crypto hashes. And obviously it'd be very important for I/O steams. 
Having such thing standardized in phobos would be great.

Steve's input on this may be very important, given his work in iopipe.


 Then we'd have input ranges, which have a buffer of at least one 
 element, i.e. today's input ranges. Input ranges that are not forward 
 ranges are liable to reuse their buffers, so after a call to front(), a 
 call to popFront() may overwrite the current front. This is because by 
 construction input ranges that are not forward ranges do not iterate 
 objects in memory, but instead they transfer data from somewhere else 
 into memory, chunkwise.


This sounds reasonable, but I would REALLY like if we could come up with 
some way to actually *prevent* the user from accessing a stored .front 
after the next call to .popFront instead of relying on "programming by 
convention". Because this seems to run in complete opposition to D 
philosophy. A static check would be fantastic if possible, or at least 
injecting an optional run-time check analogous to bounds checking. Not 
sure how feasible either would be, though.
Apr 04 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 Part of that is we've been cagey about defining copy and assignment
 semantics of ranges in a simple and comprehensive manner. It seems to
 me going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer semantics
 (all copies refer to the same underlying position, and advancing one
 advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one does
 not advance the others.
 
 * We don't support other semantics.


What about classes that wrap ranges? E.g., std.ranges.interfaces.  Since
classes are reference types, this would mean it's impossible to use
those interfaces with forward ranges or above, which is a show-stopping
limitation (e.g., if you need to return two different range types
selected at runtime -- the current solution is to wrap them in the
appropriate class using std.ranges.interfaces).


T

-- 
Time flies like an arrow. Fruit flies like a banana.
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 1:42 PM, H. S. Teoh wrote:

 On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 Part of that is we've been cagey about defining copy and assignment
 semantics of ranges in a simple and comprehensive manner. It seems to
 me going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics
 (all copies refer to the same underlying position, and advancing one
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one does
 not advance the others.

 * We don't support other semantics.


 
 What about classes that wrap ranges?


Those must go. Supporting classes and structs within the same framework 
is a costly low-yield endeavor.


 E.g., std.ranges.interfaces.


One could pin-point the introduction of that as a clear point when we 
veered from the nice highway into the cornfields.


 Since
 classes are reference types, this would mean it's impossible to use
 those interfaces with forward ranges or above, which is a show-stopping
 limitation (e.g., if you need to return two different range types
 selected at runtime -- the current solution is to wrap them in the
 appropriate class using std.ranges.interfaces).


The obvious solution is to use structs that wrap pointers or class 
references. We don't preclude virtuals and dynamic behavior.
Mar 28 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 01:56:08PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 3/28/19 1:42 PM, H. S. Teoh wrote:

 On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 Part of that is we've been cagey about defining copy and
 assignment semantics of ranges in a simple and comprehensive
 manner. It seems to me going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer
 semantics (all copies refer to the same underlying position, and
 advancing one advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one
 does not advance the others.
 
 * We don't support other semantics.


 
 What about classes that wrap ranges?


 
 Those must go. Supporting classes and structs within the same
 framework is a costly low-yield endeavor.


So this will be a major breaking change, since there's a significant
amount of user code that currently relies on this functionality.  Which
means we're seriously considering Phobos v2 here?  I'd support that.
It's time to reverse all those decisions that in hindsight were poor
ones.  Like autodecoding.



 E.g., std.ranges.interfaces.




[...]

 Since classes are reference types, this would mean it's impossible
 to use those interfaces with forward ranges or above, which is a
 show-stopping limitation (e.g., if you need to return two different
 range types selected at runtime -- the current solution is to wrap
 them in the appropriate class using std.ranges.interfaces).


 
 The obvious solution is to use structs that wrap pointers or class
 references. We don't preclude virtuals and dynamic behavior.


IOW, all ranges will be required to be structs?  And if we need dynamic
behaviour it will have to be a struct that overrides opAssign to copy
the state of the underlying range (for forward ranges)?

I suppose that could work.  There will be friction, though, if you need
to access the underlying class hierarchy (e.g., the struct would have to
export some sort of API for casting the wrapped class reference to some
interface or base class that you might want to pass into a function
expecting such).


T

-- 
Life is too short to run proprietary software. -- Bdale Garbee
Mar 28 2019
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 3/28/19 2:19 PM, H. S. Teoh wrote:

 
 IOW, all ranges will be required to be structs?  And if we need dynamic
 behaviour it will have to be a struct that overrides opAssign to copy
 the state of the underlying range (for forward ranges)?
 


IIUC, you'd want to use a copy constructor for that, not opAssign.
Apr 04 2019
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 18:04, Andrei Alexandrescu wrote:

 On 3/28/19 9:16 AM, ag0aep6g wrote:

 On 28.03.19 14:05, Andrei Alexandrescu wrote:

 Then some ranges are not meant to be assignable.


 Should Phobos be compatible with those ranges?




[...]

 It seems to me 
 going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer semantics 
 (all copies refer to the same underlying position, and advancing one 
 advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies refer 
 to distinct positions in the range such that advancing one does not 
 advance the others.
 
 * We don't support other semantics.


Are you going to actually change the range definitions in that way? Are 
you saying that I should wait until then with fixing range issues in Phobos?

I'd expect those new definitions to be years away from becoming reality. 
It doesn't seem wise to leave bugs open today just because you have 
vague plans for fundamental changes in the distant future.
Mar 28 2019
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:12 PM, ag0aep6g wrote:

 On 28.03.19 18:04, Andrei Alexandrescu wrote:

 On 3/28/19 9:16 AM, ag0aep6g wrote:

 On 28.03.19 14:05, Andrei Alexandrescu wrote:

 Then some ranges are not meant to be assignable.


 Should Phobos be compatible with those ranges?




 [...]

 It seems to me going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics 
 (all copies refer to the same underlying position, and advancing one 
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies 
 refer to distinct positions in the range such that advancing one does 
 not advance the others.

 * We don't support other semantics.


 
 Are you going to actually change the range definitions in that way? Are 
 you saying that I should wait until then with fixing range issues in 
 Phobos?
 
 I'd expect those new definitions to be years away from becoming reality. 
 It doesn't seem wise to leave bugs open today just because you have 
 vague plans for fundamental changes in the distant future.


We've been worrying too much about changing things.

On any given issue, even those that widely agreed to be clowny, two 
factions emerge. One advocates change and the other advocates against 
code breakage. Interestingly the factions are different depending on the 
issue, i.e. a given person may be in favor of change in one matter and 
in favor of backwards compatibility in another. At any rate, the result 
is years-long stalemate and trench warfare on the slightest problems.


past mistakes in maintenance mode and move on with new stuff. It's like 
blockchain - once done, it can't be undone. You do a new one. C has 
about three ways of converting numbers to strings and back. C++ added a 
couple, poorly designed. Then they added a better one. And don't get me 
started about I/O (two ways of doing it in C, and another one in C++ 
that nobody in their right mind would use). Same with Java pre- and 
post-generics interfaces for comparing objects. It's just that if 
something is bad they define a better one and move on.

We have a language with virtually no global scope - a versioning 
engineer's dream. Yet we are coy to tap into that power. Instead, we 
wring hands over the breakages that would happen if we change std.json. 
The time is ripe for std.v2, which would do the right things and support 
simple migration.
Mar 28 2019
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 19:30, Andrei Alexandrescu wrote:


 past mistakes in maintenance mode and move on with new stuff. It's like 
 blockchain - once done, it can't be undone. You do a new one. C has 
 about three ways of converting numbers to strings and back. C++ added a 
 couple, poorly designed. Then they added a better one. And don't get me 
 started about I/O (two ways of doing it in C, and another one in C++ 
 that nobody in their right mind would use). Same with Java pre- and 
 post-generics interfaces for comparing objects. It's just that if 
 something is bad they define a better one and move on.
 
 We have a language with virtually no global scope - a versioning 
 engineer's dream. Yet we are coy to tap into that power. Instead, we 
 wring hands over the breakages that would happen if we change std.json. 
 The time is ripe for std.v2, which would do the right things and support 
 simple migration.


I'm not against that at all, but I'm not convinced that you can make it 
happen quickly. Remember that this sub-thread started with a mention of 
my pull request that was open for a year, sitting idle most of the time.

So until new Phobos materializes some more, I'd appreciate if we could 
fix issue 18657 one way or the other in old Phobos.
Mar 28 2019
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:48 PM, ag0aep6g wrote:

 On 28.03.19 19:30, Andrei Alexandrescu wrote:


 past mistakes in maintenance mode and move on with new stuff. It's 
 like blockchain - once done, it can't be undone. You do a new one. C 
 has about three ways of converting numbers to strings and back. C++ 
 added a couple, poorly designed. Then they added a better one. And 
 don't get me started about I/O (two ways of doing it in C, and another 
 one in C++ that nobody in their right mind would use). Same with Java 
 pre- and post-generics interfaces for comparing objects. It's just 
 that if something is bad they define a better one and move on.

 We have a language with virtually no global scope - a versioning 
 engineer's dream. Yet we are coy to tap into that power. Instead, we 
 wring hands over the breakages that would happen if we change 
 std.json. The time is ripe for std.v2, which would do the right things 
 and support simple migration.


 
 I'm not against that at all, but I'm not convinced that you can make it 
 happen quickly.


Three solid engineers would make this a quarter's worth of work. I don't 
think we have them.

  Remember that this sub-thread started with a mention of

 my pull request that was open for a year, sitting idle most of the time.
 
 So until new Phobos materializes some more, I'd appreciate if we could 
 fix issue 18657 one way or the other in old Phobos.


For that I think we should bite the bullet and let RefRange go.
Mar 28 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 19:51, Andrei Alexandrescu wrote:

 Three solid engineers would make this a quarter's worth of work. I don't 
 think we have them.


[...]

 For that I think we should bite the bullet and let RefRange go.


So new Phobos won't happen for a long while, and old Phobos gets 
abandoned instead of fixed. Can't say that I'm hyped.
Mar 28 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:07 PM, ag0aep6g wrote:

 On 28.03.19 19:51, Andrei Alexandrescu wrote:

 Three solid engineers would make this a quarter's worth of work. I 
 don't think we have them.


 [...]

 For that I think we should bite the bullet and let RefRange go.


 
 So new Phobos won't happen for a long while, and old Phobos gets 
 abandoned instead of fixed. Can't say that I'm hyped.


Deprecating RefRange is not abandon.
Mar 28 2019
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 11:48 AM, ag0aep6g wrote:

 So until new Phobos materializes some more, I'd appreciate if we could fix
issue 
 18657 one way or the other in old Phobos.


1. deprecate RefRange

2. resolve https://issues.dlang.org/show_bug.cgi?id=18657 as "wont fix"

3. revert the recent changes to support RefRange, do it now before it gets
released
Mar 28 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 02:30:24PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 We've been worrying too much about changing things.


Wow.  That's a pretty drastic (and refreshing!) change of stance from
the past years of insisting that we should do our utmost to maintain
backward compatibility.  I, for one, welcome this change.


[...]

 We have a language with virtually no global scope - a versioning
 engineer's dream. Yet we are coy to tap into that power. Instead, we
 wring hands over the breakages that would happen if we change
 std.json. The time is ripe for std.v2, which would do the right things
 and support simple migration.


[...]

Yes, I welcome the idea of std.v2.  And we should provide reasonable
migration paths to std.v2, so that after X number of years, we can
deprecate the current stuff, and move std.v2.* to std.*.  (Or maybe
leave it as std.v2.*, in case there will ever be a need for std.v3.*. I
hope not, but hindsight is always 20/20 so it seems inevitable going
forward.)


T

-- 
IBM = I Blame Microsoft
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:56 PM, H. S. Teoh wrote:

 On Thu, Mar 28, 2019 at 02:30:24PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 We've been worrying too much about changing things.


 
 Wow.  That's a pretty drastic (and refreshing!) change of stance from
 the past years of insisting that we should do our utmost to maintain
 backward compatibility.  I, for one, welcome this change.


Thanks. The irony of it all is that adding new stuff and leaving the old 
stuff in maintenance mode is simultaneously the rock-solid compatibility 
stance and the ultimate road to progress.

It does happen, and has happened several times in my lifetime, that an 
entire community is missing an important point. It seems we missed the 
point that change is not what we need... it's addition.
Mar 28 2019
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 03:09:53PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 3/28/19 2:56 PM, H. S. Teoh wrote:

 On Thu, Mar 28, 2019 at 02:30:24PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 We've been worrying too much about changing things.


 
 Wow.  That's a pretty drastic (and refreshing!) change of stance
 from the past years of insisting that we should do our utmost to
 maintain backward compatibility.  I, for one, welcome this change.


 
 Thanks. The irony of it all is that adding new stuff and leaving the
 old stuff in maintenance mode is simultaneously the rock-solid
 compatibility stance and the ultimate road to progress.


But if that's the case, the stuff in maintenance mode should at least
receive maintenance fixes of the kind being proposed for RefRange?
Let's agree that RefRange was a bad idea and all that.  But there are
users using it that need fixes for the issues they encountered.  Even if
said fixes are perhaps also just as bad of an idea as RefRange itself
was to begin with, for the sake of maintenence we'd still be obligated
to provide at least workarounds for current bugs?



 It does happen, and has happened several times in my lifetime, that an
 entire community is missing an important point. It seems we missed the
 point that change is not what we need... it's addition.


If you're thinking about dropping druntime altogether, that's no mere
addition, that's a MAJOR change.  One that I'd welcome, to be frank, but
it's not going to happen without major breakage of existing codebases.
Are you sure you've thought this through?


T

-- 
The most powerful one-line C program: #include "/dev/tty" -- IOCCC
Mar 28 2019
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:19 PM, H. S. Teoh wrote:

 On Thu, Mar 28, 2019 at 03:09:53PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 3/28/19 2:56 PM, H. S. Teoh wrote:

 On Thu, Mar 28, 2019 at 02:30:24PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 We've been worrying too much about changing things.


 Wow.  That's a pretty drastic (and refreshing!) change of stance
 from the past years of insisting that we should do our utmost to
 maintain backward compatibility.  I, for one, welcome this change.


 Thanks. The irony of it all is that adding new stuff and leaving the
 old stuff in maintenance mode is simultaneously the rock-solid
 compatibility stance and the ultimate road to progress.


 
 But if that's the case, the stuff in maintenance mode should at least
 receive maintenance fixes of the kind being proposed for RefRange?


Changing the entire standard library to accommodate wackiness in 
RefRange is a poor executive decision regardless. The recent changes 
must be undone.
Mar 28 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 20:24, Andrei Alexandrescu wrote:

 Changing the entire standard library to accommodate wackiness in 
 RefRange is a poor executive decision regardless. The recent changes 
 must be undone.


No one is arguing in favor of doing what I've in my PRs on the issue. 
Not even me.

I've started that endeavor, because the only tangible response I got 
when I asked what do do was from Jonathan M Davis who said: "it could be 
argued that generic, range-based functions simply shouldn't ever be 
assigning one range to another."[1]

If you prefer and approve removing RefRange.opAssign instead, I'm all 
for that.

Also note that Walter himself has done similar pull requests:

https://github.com/dlang/phobos/pull/6900
https://github.com/dlang/phobos/pull/6903

He uses constructors instead of "wonky" `move`, but the effect with 
regards to issue 18657 is the same: RefRange.opAssign is not called. If 
that's an acceptable way to make Phobos compatible with RefRange, that's 
also fine with me.


[1] 
https://forum.dlang.org/post/mailman.1557.1521939012.3374.digitalmars-d puremagic.com
Mar 28 2019
next sibling parent ag0aep6g <anonymous example.com> writes:
On 28.03.19 20:43, ag0aep6g wrote:

 On 28.03.19 20:24, Andrei Alexandrescu wrote:

 Changing the entire standard library to accommodate wackiness in 
 RefRange is a poor executive decision regardless. The recent changes 
 must be undone.




[...]

 He uses constructors instead of "wonky" `move`


correction: "wacky", not "wonky" ;)
Mar 28 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:43 PM, ag0aep6g wrote:

 On 28.03.19 20:24, Andrei Alexandrescu wrote:

 Changing the entire standard library to accommodate wackiness in 
 RefRange is a poor executive decision regardless. The recent changes 
 must be undone.


 
 No one is arguing in favor of doing what I've in my PRs on the issue. 
 Not even me.


Yes, I saw your pain seeping through the lines. The valiant effort 
hasn't gone unnoticed and is appreciated.


 I've started that endeavor, because the only tangible response I got 
 when I asked what do do was from Jonathan M Davis who said: "it could be 
 argued that generic, range-based functions simply shouldn't ever be 
 assigning one range to another."[1]


This is a sign things have gotten off the rail. Assignment should be 
accessible and have obvious semantics.


 If you prefer and approve removing RefRange.opAssign instead, I'm all 
 for that.
 
 Also note that Walter himself has done similar pull requests:
 
 https://github.com/dlang/phobos/pull/6900
 https://github.com/dlang/phobos/pull/6903
 
 He uses constructors instead of "wonky" `move`, but the effect with 
 regards to issue 18657 is the same: RefRange.opAssign is not called. If 
 that's an acceptable way to make Phobos compatible with RefRange, that's 
 also fine with me.
 
 
 [1] 
 https://forum.dlang.org/post/mailman.1557.1521939012.3374.digital
ars-d puremagic.com 


RefRange is a cancer.

Changing all data structures and algorithms to support it is the cancer 
going into metastasis.

We must excise the cancerous tissue.
Mar 28 2019
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 28.03.19 21:05, Andrei Alexandrescu wrote:

 This is a sign things have gotten off the rail. Assignment should be 
 accessible and have obvious semantics.


Let's get rid of RefRange.opAssign then. Then assignment of RefRange has 
obvious semantics. RefRange.opAssign is not needed for RefRange to be 
useful.

[...]

 RefRange is a cancer.
 
 Changing all data structures and algorithms to support it is the cancer 
 going into metastasis.


Again, no one here is arguing for "changing all data structures and 
algorithms".


 We must excise the cancerous tissue.


We can cut out the bad parts, but leave the useful core of RefRange 
intact, can't we?

The useful core is: a range that wraps a pointer to another range in 
order to achieve reference semantics. Often, a pointer to a range is 
already a range, but not always. E.g., `int[]*` is not a range.
Mar 28 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 3:13:17 PM MDT ag0aep6g via Digitalmars-d wrote:

 On 28.03.19 21:05, Andrei Alexandrescu wrote:

 We must excise the cancerous tissue.


 We can cut out the bad parts, but leave the useful core of RefRange
 intact, can't we?

 The useful core is: a range that wraps a pointer to another range in
 order to achieve reference semantics. Often, a pointer to a range is
 already a range, but not always. E.g., `int[]*` is not a range.


The problem is that reference semantics and forward ranges do not go well
together at all. As things stand, range-based code is supposed to work with
forward ranges that are reference types, and we have save to try and deal
with the fact that some ranges are value types, some are reference types,
and some are pseudo-reference types, but it really doesn't work very well.
And algorithms that require save fundamentally need to copy the range at
some point, pretty much destroying the usefulness of something like
RefRange.

Basic input ranges are fundamentally reference types or pseudo-reference
types, because if they could have value semantics, then they could be
forward ranges. Forward ranges fundamentally have to be copyable with value
semantics to even work properly. We've just got this messed up situation
where we're using save to do the copy instead of postblit or copy
constuctors, because we're trying to handle types that aren't value types as
if they were.

RefRange is trying to force reference semantics, which then fundamentally
doesn't fit with forward ranges. It sort of half fits right now, because of
save, but as with forward ranges that are reference types, it causes
problems, and code frequently isn't going to work with it without extra
work.

As such, I really don't think that RefRange makes sense fundamentally. It
_can_ work in corner cases, and years ago, I had a piece of code where it
was really useful, which is why I wrote it. But it was a mistake. It's
trying to fight against how forward ranges work, and while it might
occasionally be useful, in general, it's going to cause problems.

And if we actually go forward with requiring that basic input ranges have
reference semantics and that forward ranges have value semantics like Andrei
is proposing, then RefRange _really_ doesn't work.

- Jonathan M Davis
Mar 30 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 30.03.19 19:32, Jonathan M Davis wrote:

 RefRange is trying to force reference semantics, which then fundamentally
 doesn't fit with forward ranges. It sort of half fits right now, because of
 save, but as with forward ranges that are reference types, it causes
 problems, and code frequently isn't going to work with it without extra
 work.
 
 As such, I really don't think that RefRange makes sense fundamentally.


You haven't spelled it out, but I suppose you also think that classes 
don't make sense as forward ranges. And Andrei has stated something to 
the effect that they wouldn't be supported in his redesign.

That is fine for a range redesign. But with the definitions we have 
right now, classes can be forward ranges, and Phobos should work with 
them. And if Phobos works with classes, it can work with RefRange (if we 
cut out opAssign).

If RefRange really is fundamentally at odds with current ranges, I'd be 
interested in an example where it causes trouble that isn't due to 
opAssign (or opSlice() which also shouldn't be there). Maybe I just fail 
to see the problem.

One issue I'm aware of is that RefRange does a GC allocation on `save`. 
That isn't pretty, but I wouldn't call it fatal.

I've said before that "we can cut out the bad parts" of RefRange, 
referring to opAssign, but I think I was wrong. Deprecating assignment 
would be so limiting that we might as well deprecate the whole thing.

Then again, Walter has made (limited) efforts to support ranges that are 
non-assignable due to const fields. Nobody acknowledges that those exact 
changes also work to accommodate RefRange, but they do.
Mar 30 2019
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/30/2019 12:38 PM, ag0aep6g wrote:

 Then again, Walter has made (limited) efforts to support ranges that are 
 non-assignable due to const fields. Nobody acknowledges that those exact
changes 
 also work to accommodate RefRange, but they do.


I'm fine with adjusting `save()` implementations to use construction rather
than 
assignment semantics, because that's the way `save()` is defined to behave 
(although it wasn't clearly spelled out).
Mar 30 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/19 8:05 PM, Walter Bright wrote:

 On 3/30/2019 12:38 PM, ag0aep6g wrote:

 Then again, Walter has made (limited) efforts to support ranges that 
 are non-assignable due to const fields. Nobody acknowledges that those 
 exact changes also work to accommodate RefRange, but they do.


 
 I'm fine with adjusting `save()` implementations to use construction 
 rather than assignment semantics, because that's the way `save()` is 
 defined to behave (although it wasn't clearly spelled out).


Great, so ag0aep6g you're good to go with your PR. I'd approved it 
yesterday.
Mar 30 2019
parent reply ag0aep6g <anonymous example.com> writes:
On 31.03.19 01:14, Andrei Alexandrescu wrote:

 Great, so ag0aep6g you're good to go with your PR. I'd approved it 
 yesterday.


Thanks, guys. Glad we managed to work this out.

Just to be clear, I can't merge PRs. It sounds like you might expect me 
to do that.
Mar 30 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/19 8:19 PM, ag0aep6g wrote:

 On 31.03.19 01:14, Andrei Alexandrescu wrote:

 Great, so ag0aep6g you're good to go with your PR. I'd approved it 
 yesterday.


 
 Thanks, guys. Glad we managed to work this out.
 
 Just to be clear, I can't merge PRs. It sounds like you might expect me 
 to do that.


Did so. Thanks for the work.
Mar 30 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, March 30, 2019 1:38:46 PM MDT ag0aep6g via Digitalmars-d wrote:

 On 30.03.19 19:32, Jonathan M Davis wrote:

 RefRange is trying to force reference semantics, which then
 fundamentally
 doesn't fit with forward ranges. It sort of half fits right now, because
 of save, but as with forward ranges that are reference types, it causes
 problems, and code frequently isn't going to work with it without extra
 work.

 As such, I really don't think that RefRange makes sense fundamentally.


 You haven't spelled it out, but I suppose you also think that classes
 don't make sense as forward ranges. And Andrei has stated something to
 the effect that they wouldn't be supported in his redesign.

 That is fine for a range redesign. But with the definitions we have
 right now, classes can be forward ranges, and Phobos should work with
 them. And if Phobos works with classes, it can work with RefRange (if we
 cut out opAssign).

 If RefRange really is fundamentally at odds with current ranges, I'd be
 interested in an example where it causes trouble that isn't due to
 opAssign (or opSlice() which also shouldn't be there). Maybe I just fail
 to see the problem.

 One issue I'm aware of is that RefRange does a GC allocation on `save`.
 That isn't pretty, but I wouldn't call it fatal.

 I've said before that "we can cut out the bad parts" of RefRange,
 referring to opAssign, but I think I was wrong. Deprecating assignment
 would be so limiting that we might as well deprecate the whole thing.

 Then again, Walter has made (limited) efforts to support ranges that are
 non-assignable due to const fields. Nobody acknowledges that those exact
 changes also work to accommodate RefRange, but they do.


Fundamentally, forward ranges are essentially value types. They've just had
their copy operation moved to save in order to accommodate classes. They
have to be copyable to work (even if that copying is done via save), and any
function that requires a forward range _will_ copy it via save, or it
wouldn't require a forward range. Once the range is saved, RefRange is
pretty useless, because the results of what are done to the copy don't
affect the original.

And to use RefRange, you're basically depending on implementation details of
a particular algorithm and when it chooses to call save and what exactly it
does with the original before and after calling save - something that is not
part of the function's API or any guarantee that it makes. And because the
actual semantics of copying a range depend on how it's implemented,
algorithms can't rely on whether copying a range will result in an
independent copy or not and are free to rearrange when they do or don't copy
a range so long as they don't use the original range after it's been copied.
They're also free to change when save is called so long as the function's
result is the same. It's also assumed that the function can do whatever it
pleases with the original so long as it doesn't violate the range API, since
in general, you can't use a range again after it's been copied (because
doing so would not work the same with all range types).

RefRange on the other hand is basically trying to get access to the results
of the first part of the algorithm prior to the call save (and possibly some
of what's done to the original after the call to save). It's purposefully
setting up the code to reuse a range in circumstances where you can't reuse
a range in generic code. The code isn't necessarily generic, which does
change things, but if the algorithm you're passing RefRange too is generic,
then it wasn't written with RefRange in mind, and there will be no
expectation that the caller will do anything with the range that was passed
in. If anything, the expection is that they won't do anything with it except
maybe assign the result of the function to it to reuse the variable.

Basically, RefRange is trying to fight how forward ranges work, and while
that works in corner cases, in general, it's a mess. And the odds of code
that uses RefRange breaking due to changes to algorithms that would
otherwise be fine are pretty high, because code using RefRange is depending
on the implementation details of any functions that RefRange is passed to.

So, while RefRange might be okay in code that's written specifically for use
with RefRange, in general, it's a terrible idea. And I really regret that I
got into Phobos. The fact that I did just shows how much less I understood
about ranges at the time.

- Jonathan M Davis
Mar 30 2019
parent ag0aep6g <anonymous example.com> writes:
On 31.03.19 08:53, Jonathan M Davis wrote:

 Once the range is saved, RefRange is
 pretty useless, because the results of what are done to the copy don't
 affect the original.
 
 And to use RefRange, you're basically depending on implementation details of
 a particular algorithm and when it chooses to call save and what exactly it
 does with the original before and after calling save - something that is not
 part of the function's API or any guarantee that it makes. 


[...]

 RefRange on the other hand is basically trying to get access to the results
 of the first part of the algorithm prior to the call save (and possibly some
 of what's done to the original after the call to save).


All that also applies to other ranges that have reference semantics.

[...]

 So, while RefRange might be okay in code that's written specifically for use
 with RefRange, in general, it's a terrible idea. And I really regret that I
 got into Phobos. The fact that I did just shows how much less I understood
 about ranges at the time.


RefRange is painted as this big scary monster that ruins everything 
(Andrei called it "cancer", you call it a "terrible idea"). In reality, 
it might not be the most useful thing, but it works just as well as 
other reference-type ranges. Or rather, it would, if it weren't for that 
pesky opAssign. RefRange.opAssign ruins everything.
Mar 31 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 2:05:41 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:

 On 3/28/19 3:43 PM, ag0aep6g wrote:

 I've started that endeavor, because the only tangible response I got
 when I asked what do do was from Jonathan M Davis who said: "it could be
 argued that generic, range-based functions simply shouldn't ever be
 assigning one range to another."[1]


 This is a sign things have gotten off the rail. Assignment should be
 accessible and have obvious semantics.


That's really the problem. Some operations on ranges are not well defined,
and so you can't rely on them in generic code. The prime example is copying
(though assignment then gets tied up in that). If a range is a reference
type, then iterating the copy iterates the original; if a range is a value
type, then iterating the copy doesn't affect the original; and if a range is
a pseudo-reference type, then exactly what happens depends on how it's
implemented. The end result is that you basically can't use a range after
it's been copied (which includes passing it to a function or using it with
foreach), because what happens depends on how the range was implemented.

I've thought for a while now that how things should really work would be to
require that basic input ranges be reference types and that forward ranges
be value types (ditching save, requiring copy construction instead, which
then requires wrapping classes in structs to be ranges), though that
potentially means that calling the same function on basic input ranges and
forward ranges gets a bit sticky. The biggest problem though of course is
compatibility, but if we're seriously looking at doing a v2 for major stuff
in Phobos or Phobos as a whole, then we definitely should be sitting down
and working through what we did wrong with ranges and fix this sort of
thing.

- Jonathan M Davis
Mar 30 2019
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 12:19 PM, H. S. Teoh wrote:

 But if that's the case, the stuff in maintenance mode should at least
 receive maintenance fixes of the kind being proposed for RefRange?


Don't need those fixes because it never worked. Maintenance fixes would be when 
language changes break the library (such as the library work I did to get 
DIP1000 to work).
Mar 28 2019
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 12:09 PM, Andrei Alexandrescu wrote:

 It does happen, and has happened several times in my lifetime, that an entire 
 community is missing an important point. It seems we missed the point that 
 change is not what we need... it's addition.


Reminds me of:

https://github.com/dlang/undeaD

That library turned out to be necessary because some old software of mine used 
them and there was no reasonable migration path to the newer modules.
Mar 28 2019
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei Alexandrescu 
wrote:


 We've been worrying too much about changing things.


Wow! That is refreshing to read.



 - leave past mistakes in maintenance mode and move on with new 
 stuff. It's like blockchain - once done, it can't be undone. 
 You do a new one. C has about three ways of converting numbers 
 to strings and back. C++ added a couple, poorly designed. Then 
 they added a better one. And don't get me started about I/O 
 (two ways of doing it in C, and another one in C++ that nobody 
 in their right mind would use). Same with Java pre- and 
 post-generics interfaces for comparing objects. It's just that 
 if something is bad they define a better one and move on.


Hear, hear.  It applies to much outside of software as well.

Reading this I'm getting all psyched about D again.

Mike
Mar 28 2019
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei Alexandrescu 
wrote:

 On 3/28/19 2:12 PM, ag0aep6g wrote:

 [...]


 We've been worrying too much about changing things.

 On any given issue, even those that widely agreed to be clowny, 
 two factions emerge. One advocates change and the other 
 advocates against code breakage. Interestingly the factions are 
 different depending on the issue, i.e. a given person may be in 
 favor of change in one matter and in favor of backwards 
 compatibility in another. At any rate, the result is years-long 
 stalemate and trench warfare on the slightest problems.

 [...]


This is a complete 180 from what you were saying a year ago. What 
happened?
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 10:03 PM, Jonathan Marler wrote:

 On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei Alexandrescu wrote:

 On 3/28/19 2:12 PM, ag0aep6g wrote:

 [...]


 We've been worrying too much about changing things.

 On any given issue, even those that widely agreed to be clowny, two 
 factions emerge. One advocates change and the other advocates against 
 code breakage. Interestingly the factions are different depending on 
 the issue, i.e. a given person may be in favor of change in one matter 
 and in favor of backwards compatibility in another. At any rate, the 
 result is years-long stalemate and trench warfare on the slightest 
 problems.

 [...]


 
 This is a complete 180 from what you were saying a year ago. What happened?


"I'd rather be right than consistent." -- Winston Churchill
Mar 28 2019
next sibling parent Jonathan Marler <johnnymarler gmail.com> writes:
On Friday, 29 March 2019 at 02:40:43 UTC, Andrei Alexandrescu 
wrote:

 On 3/28/19 10:03 PM, Jonathan Marler wrote:

 On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei 
 Alexandrescu wrote:

 On 3/28/19 2:12 PM, ag0aep6g wrote:

 [...]


 We've been worrying too much about changing things.

 On any given issue, even those that widely agreed to be 
 clowny, two factions emerge. One advocates change and the 
 other advocates against code breakage. Interestingly the 
 factions are different depending on the issue, i.e. a given 
 person may be in favor of change in one matter and in favor 
 of backwards compatibility in another. At any rate, the 
 result is years-long stalemate and trench warfare on the 
 slightest problems.

 [...]


 
 This is a complete 180 from what you were saying a year ago. 
 What happened?


 "I'd rather be right than consistent." -- Winston Churchill


A good rule to live by and be aware of as many tend to stick to 
their guns rather than admitting they're not perfect.

In any case I am very pleased to see this new attitude. I was 
just genuinely wondering what caused it.
Mar 28 2019
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Friday, 29 March 2019 at 02:40:43 UTC, Andrei Alexandrescu 
wrote:

 On 3/28/19 10:03 PM, Jonathan Marler wrote:

 On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei 
 Alexandrescu wrote:

 On 3/28/19 2:12 PM, ag0aep6g wrote:

 [...]


 We've been worrying too much about changing things.

 On any given issue, even those that widely agreed to be 
 clowny, two factions emerge. One advocates change and the 
 other advocates against code breakage. Interestingly the 
 factions are different depending on the issue, i.e. a given 
 person may be in favor of change in one matter and in favor 
 of backwards compatibility in another. At any rate, the 
 result is years-long stalemate and trench warfare on the 
 slightest problems.

 [...]


 
 This is a complete 180 from what you were saying a year ago. 
 What happened?


 "I'd rather be right than consistent." -- Winston Churchill


Kudos!
Mar 29 2019
prev sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On 29/03/2019 03:40, Andrei Alexandrescu wrote:

 On 3/28/19 10:03 PM, Jonathan Marler wrote:

 On Thursday, 28 March 2019 at 18:30:24 UTC, Andrei Alexandrescu wrote:

 We've been worrying too much about changing things.

 [...]


 This is a complete 180 from what you were saying a year ago. What 
 happened?


 
 "I'd rather be right than consistent." -- Winston Churchill


I too applaud this insight. As many may know, I'm helping a company move 
away from a fossilized language and it has decided to make its business 
depend on D. Although reacting to deprecations won't always come 
convenient, distributing the cost of evolution is much more preferable 
than making the massive investment they are in for now. So I'm sure they 
appreciate seeing these signs of D staying dynamic, averting sedimentation.

Bastiaan.
Mar 29 2019
prev sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 3/28/19 2:30 PM, Andrei Alexandrescu wrote:

 
 We've been worrying too much about changing things.


Yup. *Very* glad to hear this coming from you of all people ;)



 We have a language with virtually no global scope - a versioning 
 engineer's dream. Yet we are coy to tap into that power. 


Yes, precisely! Hear hear!
Apr 04 2019
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 07:12:36PM +0100, ag0aep6g via Digitalmars-d wrote:

 On 28.03.19 18:04, Andrei Alexandrescu wrote:


[...]

 It seems to me going with these is the right thing:
 
 * Input ranges are copyable and assignable, and have pointer
 semantics (all copies refer to the same underlying position, and
 advancing one advances all others).
 
 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one
 does not advance the others.
 
 * We don't support other semantics.


 
 Are you going to actually change the range definitions in that way?
 Are you saying that I should wait until then with fixing range issues
 in Phobos?
 
 I'd expect those new definitions to be years away from becoming
 reality. It doesn't seem wise to leave bugs open today just because
 you have vague plans for fundamental changes in the distant future.


The only way I can see this happening is if we start a new iteration of
Phobos, like std.v2.* where these fundamental changes can be done in a
way that isn't totally disruptive to existing code.  There would have to
be a migration path from the current Phobos so that existing code can
gradually move to the new APIs. 

Implementing these changes directly in the current Phobos is going to
break SO much code it will make Walter & Andrei's stance on stabilizing
the language completely laughable.

(OTOH, I would welcome something like std.v2.*, where we can reverse
those decisions that in hindsight were poorly made, like autodecoding.
And getting rid of subpar modules like std.xml (and incorporating
Jonathan's dxml in its place).)


T

-- 
Almost all proofs have bugs, but almost all theorems are true. -- Paul Pedersen
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:30 PM, H. S. Teoh wrote:

 The only way I can see this happening is if we start a new iteration of
 Phobos, like std.v2.*


There'd be carefully carved subsets per capability, such as 
std.v2.betterc.*, std.v2.nogc.*, std.v2.noexcept.*, std.v2.safe.

The capability sets can be combined in alphabetic ordering. E.g. 
std.v2.noexcept.nogc.* has the subset that uses no gc and no exceptions, 
but std.v2.nogc.noexcept does not exist.

Through the magic of public imports and aliasing there would be no code 
duplication. Introspection could help, too, a la "define public aliases 
for all  safe definitions in that module". Some improvements to ddoc 
might be needed to "see" through aliases.
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 2:41 PM, Andrei Alexandrescu wrote:

 On 3/28/19 2:30 PM, H. S. Teoh wrote:

 The only way I can see this happening is if we start a new iteration of
 Phobos, like std.v2.*


 
 There'd be carefully carved subsets per capability, such as 
 std.v2.betterc.*, std.v2.nogc.*, std.v2.noexcept.*, std.v2.safe.
 
 The capability sets can be combined in alphabetic ordering. E.g. 
 std.v2.noexcept.nogc.* has the subset that uses no gc and no exceptions, 
 but std.v2.nogc.noexcept does not exist.
 
 Through the magic of public imports and aliasing there would be no code 
 duplication. Introspection could help, too, a la "define public aliases 
 for all  safe definitions in that module". Some improvements to ddoc 
 might be needed to "see" through aliases.


Oh, and druntime must go.

The whole distinction between the runtime library and the standard 
library is clowny and has more to do with poorly ported tradition from 
other languages, than with anything else.

We need one standard library that is entirely pay-as-you-go (e.g. empty 
main() means no library file is ever needed for linking) and that offers 
an opt-in continuum.
Mar 28 2019
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 02:47:49PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 Oh, and druntime must go.
 
 The whole distinction between the runtime library and the standard
 library is clowny and has more to do with poorly ported tradition from
 other languages, than with anything else.


OK, *this* is going to be an even more drastic change.  So what happens
with the current compiler bindings to druntime?



 We need one standard library that is entirely pay-as-you-go (e.g.
 empty main() means no library file is ever needed for linking) and
 that offers an opt-in continuum.


That would be a very welcome change.

But I'd also have to warn that this may not be as trivial as you appear
to think.  There's a lot of deep-seated stuff in the compiler that
heavily ties it to the current druntime implementation.  Replacing all
of that is going to be a non-trivial feat.

The TypeInfo stuff, for one thing, will take some work to reach your
vision of a completely opt-in library.  Currently, trying to link
without druntime will cause a bunch of errors related to missing
TypeInfo's.  It *is* clowny, I'll give you that -- the compiler really
should not be emitting this stuff (or should be using weak symbols for
it) unless it's actually *used* by the user code.

Similarly, we should get rid of the implicit Monitor field in classes --
useless deadweight unless the user code needs synchronized classes.

Or the GC initialization in rt_init / rt_term.  Or, for that matter, the
initialization code that packages command-line arguments into a string[]
before invoking Dmain.  To achieve not linking any library for empty
main() the compiler would have to stop emitting calls to rt_init /
rt_term unless the program uses the GC.  And it'd have to emit main() as
`main` rather than `Dmain` if you're going to want to omit the current
standard initialization code.  (And don't forget the need to invoke
module static ctors -- in topological order, which requires some amount
of code to get right.)

And so on and so forth.

IOW, what we're looking at here is nothing short of what essentially
boils down to D3.  I applaud this direction, but let's not kid ourselves
that this is going to happen overnight.


T

-- 
Music critic: "That's an imitation fugue!"
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:12 PM, H. S. Teoh wrote:

 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.


Or at all. The entire effort would require three strong like-minded 
engineers, and I don't know them.
Mar 28 2019
next sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 28 March 2019 at 20:14:19 UTC, Andrei Alexandrescu 
wrote:

 On 3/28/19 3:12 PM, H. S. Teoh wrote:

 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.


 Or at all. The entire effort would require three strong 
 like-minded engineers, and I don't know them.


Can't we crowd-found this initiative? Also, if we lay out the 
rough ground lines and draft, the community could pitch in and it 
wouldn't necessarily be a one man's mission.
Divide et impera!
Mar 28 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 08:25:10PM +0000, Seb via Digitalmars-d wrote:

 On Thursday, 28 March 2019 at 20:14:19 UTC, Andrei Alexandrescu wrote:

 On 3/28/19 3:12 PM, H. S. Teoh wrote:

 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.


 
 Or at all. The entire effort would require three strong like-minded
 engineers, and I don't know them.




I'd like to see this effort happen, one way or another.  Couldn't you or
somebody you trust come up with a design of the core principles and a
rough draft, then the rest of the community can flesh it out?  It
doesn't have to be someone working on it full-time.

Also, it could be designed to be backwards-compatible with the current
Phobos where this wouldn't compromise the new design, so that it could
be done piecemeal instead of a one-shot effort.  Doing it piecemeal
makes it much more likely to succeed, IMO.  For example, you could
design a std.v2.range.primitives with the new range API (without .save)
and have the community test-drive that. Then once that's solid, add
other pieces to it like std.v2.range, std.v2.algorithm, etc..  At each
stage you have a minimum viable product that can be used for some
limited scope of the full anticipated functionality, rather than trying
to chew off the entire thing in one go and finding yourself choking on
on overly-large project scope.



 Can't we crowd-found this initiative? Also, if we lay out the rough
 ground lines and draft, the community could pitch in and it wouldn't
 necessarily be a one man's mission.  Divide et impera!


+1.


T

-- 
Notwithstanding the eloquent discontent that you have just respectfully
expressed at length against my verbal capabilities, I am afraid that I must
unfortunately bring it to your attention that I am, in fact, NOT verbose.
Mar 28 2019
prev sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 20:14:19 UTC, Andrei Alexandrescu 
wrote:

 On 3/28/19 3:12 PM, H. S. Teoh wrote:

 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.


 Or at all. The entire effort would require three strong 
 like-minded engineers, and I don't know them.


Or a team of subordinate apprentice/junior engineers (who have 
demonstrated sufficient potential) working under the mentorship 
of a master who is willing to make an investment.

Mike
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 9:26 PM, Mike Franklin wrote:

 On Thursday, 28 March 2019 at 20:14:19 UTC, Andrei Alexandrescu wrote:

 On 3/28/19 3:12 PM, H. S. Teoh wrote:

 I applaud this direction, but let's not kid ourselves
 that this is going to happen overnight.


 Or at all. The entire effort would require three strong like-minded 
 engineers, and I don't know them.


 
 Or a team of subordinate apprentice/junior engineers (who have 
 demonstrated sufficient potential) working under the mentorship of a 
 master who is willing to make an investment.


This is a lot more difficult to set up than it seems.

One reason is that surprisingly few of the contributors come here to 
learn, to acquire knowledge. Most come to dispense. I spent long times 
reviewing pull requests in our community, calibrated to the reviews 
standards at Facebook (however controversial, they have an impressive 
software engineering force). There, the code review process is locked in 
a virtuous circle: you get good reviews that help you improve, and you'd 
be motivated to give good reviews to improve others'.

Here, my reviews were more often than not met with hostility.

As a pattern, a reviewer would be more willing to write the code once 
and then defend it in its current state, rather than improve it through 
review. In a few extreme cases, people flat out told me they'll abandon 
the PR if I don't take it as it is. Larger disagreements on matters of 
architecture and design were often taken as personal offenses. Of 
course, retaliation was soon to follow. My own ideas and code would be 
met with suspicion and I'd need to push them uphill instead of them 
being encouraged and improved, which provided disincentive for trying 
any creative work. I'd be routinely followed on github and countered at 
whatever I'd do or say. Of course this happened to Walter even more, and 
the only therapy we got was commiseration via the occasional email 
titled "Somebody shoot me" or similar, and containing just a link to a 
github comment.

I'd wake up in the morning for days, then weeks and months, with one 
thought: "I have so many things to do today, and I like none of them." 
The irony! Here I was, independent, working for myself on whatever I 
dreamed of. Yet this was worse than the worst jobs I've ever had. How 
did I get into this situation? I think it has to do with a simple 
reality - I was trying to mentor people who didn't want to be mentored.

So I think it would be difficult to establish a master/mentee dynamics.
Mar 29 2019
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/29/19 9:57 AM, Andrei Alexandrescu wrote:

 As a pattern, a reviewer would be more willing to write the code once 


s/reviewer/author/
Mar 29 2019
prev sibling next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 29 March 2019 at 13:57:57 UTC, Andrei Alexandrescu 
wrote:

 Here, my reviews were more often than not met with hostility.

 As a pattern, a reviewer would be more willing to write the 
 code once and then defend it in its current state, rather than 
 improve it through review. In a few extreme cases, people flat 
 out told me they'll abandon the PR if I don't take it as it is.


Where I work, we're trying our best to do design reviews before 
starting large implementations and to seek reviewer feedback 
throughout the process. We do this because it seems clear that 
after the code is already written and after several days' to 
weeks' worth of effort has gone into a feature, the cost to 
making fundamental design changes is as great as it's going to 
get, and developer attachment to the existing code as well as 
frustration with the reviewer is high, especially when the 
reviewer asks for fundamental design changes.

And this, to pivot to another debate we had recently, is exactly 
why it's a counterproductive approach to do DIP reviews only at 
the end.
Mar 29 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/29/19 11:12 AM, FeepingCreature wrote:

 On Friday, 29 March 2019 at 13:57:57 UTC, Andrei Alexandrescu wrote:

 Here, my reviews were more often than not met with hostility.

 As a pattern, a reviewer would be more willing to write the code once 
 and then defend it in its current state, rather than improve it 
 through review. In a few extreme cases, people flat out told me 
 they'll abandon the PR if I don't take it as it is.


 
 Where I work, we're trying our best to do design reviews before starting 
 large implementations and to seek reviewer feedback throughout the 
 process. We do this because it seems clear that after the code is 
 already written and after several days' to weeks' worth of effort has 
 gone into a feature, the cost to making fundamental design changes is as 
 great as it's going to get, and developer attachment to the existing 
 code as well as frustration with the reviewer is high, especially when 
 the reviewer asks for fundamental design changes.


There are few more productive moments at Facebook than those when five 
solid engineers get in a room and pound out a design.

Keywords being "five", "solid", and "room". The moral equivalent for us 
would be a virtual room - a forum or a discussion around a github pull 
request. The problem is access and numbers.

Consider a Gedankenexperiment whereby Facebook opened a room to the 
public, capacity 12, and offered it to any passer-by willing to discuss 
design with a couple of Facebook engineers.

Now, one of two things may happen. One is, the notion is very successful 
and people would clamor to get into the room. Natural dynamics would 
lead to some form of selection at the door that would allow only the 
best of the best enter.

Another is, the notion is only mildly intriguing. So there's a few folks 
inside, and nobody waiting outside. People come and go because they 
don't see a huge emulation at the door creating incentive to prove their 
worth inside. Now there would be competition, but just against the two 
Facebook engineers who are perceived as the benchmark against which 
everyone needs to compare. (Remember, nobody is at the door to put 
pressure.) Anyone can enter and stay for any period of time, and their 
voice will be as loud as anyone else's in the room.  So they'd want to 
outdo the Facebook engineers, and if they riposte, anyone can throw 
their hands in the air and leave.

Look at what happens in Rust. People get over each other to add quality 
to the language, because if they don't, the next guy has a better 
proposal, idea, or code. There's a crowd at the door, putting pressure 
on the folks within. Folks don't go around telling Niko Matsakis he's a 
chowderhead who could learn a thing or two.

It's most interesting what happens with Walter. He is one of the best 
hackers (in the Paul Graham sense) I've ever known, and that says a lot; 
I know quite a few, and I could also hack my way out of a paper bag. 
Walter is literally the guy who could write a line of code where most 
others would need one hundred. Yet he is hardly revered in our 
community, and his views on doing software are routinely debated and 
often derided. His inability to debate is part of it (he's a hacker, not 
a politician), but I think the lion's share goes to the "Facebook room" 
thought experiment above. Not enough people at the door.


 And this, to pivot to another debate we had recently, is exactly why 
 it's a counterproductive approach to do DIP reviews only at the end.


Yes, that would be helpful. One problem we have there is proposals that 
cannot be improved via review.

Years ago, I had the hots for a girl so I did what many silly guys do - 
I friended her. She confessed she had aspirations for writing. Being 
likely inclined as well and wanting to please, I enthusiastically 
proffered interest in her endeavor. So she gave me a manuscript to read.

It was... bad. Not just bad, fractally bad - from the overall arc down 
to grammar and punctuation. Wanting to say something nice about it, I 
went for advice to my uncle, a published author (his obituary is online 
- Nicuta Tanase). He said, kid, just mention a couple of things that are 
objective and would improve things.

And so I did, not realizing I was creating a larger problem for myself. 
Because a week later, she came with another draft in hand, saying: "I 
fixed it! Now it's good for publishing... right?"

Needless to say, that didn't go well for my dating life.
Mar 29 2019
next sibling parent throw away <throw away.com> writes:
Now on Hacker News:

https://news.ycombinator.com/newest
Mar 29 2019
prev sibling next sibling parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Friday, 29 March 2019 at 16:26:06 UTC, Andrei Alexandrescu 
wrote:

 Years ago, I had the hots for a girl so I did what many silly 
 guys do - I friended her. She confessed she had aspirations for 
 writing. Being likely inclined as well and wanting to please, I 
 enthusiastically proffered interest in her endeavor. So she 
 gave me a manuscript to read.


I don't think this comparison is appropriate, and honestly, I 
think it's a little disrespectful to contributors.

Like, contributors aren't trying to get a date. They don't want 
to be teased, they don't want to play social games, they don't 
want things to be dramatic or fun or romantic. They want to 
submit a change, and they want it to be straightforward.

Also, they're mostly volunteers, helping on their free time out 
of passion. Comparing them to a girl hamstringing you for writing 
advice seems extremely condescending.


 It was... bad. Not just bad, fractally bad - from the overall 
 arc down to grammar and punctuation. Wanting to say something 
 nice about it, I went for advice to my uncle, a published 
 author (his obituary is online - Nicuta Tanase). He said, kid, 
 just mention a couple of things that are objective and would 
 improve things.

 And so I did, not realizing I was creating a larger problem for 
 myself. Because a week later, she came with another draft in 
 hand, saying: "I fixed it! Now it's good for publishing... 
 right?"


That doesn't mean early reviews aren't valuable.

Like, I get that some people get sore when you tell them that a 
submission they've spent a lot of effort on just isn't good 
enough, and sometimes they will be unreasonable no matter what 
you say.

(and, as I've said before, the community has a tendency to jump 
to "Oh, W&A are being stubborn and ignoring The Will of The 
People again" every time you make a design decision someone 
doesn't like)

But the response to that is *more* communication, not less. If 
someone's PR is too bad to be merged, you're not going to help 
anyone by letting the person work on code that has no chance to 
be accepted. Same thing for DIPs.

Saying "This would be much easier if the community was bigger and 
people fought for my attention" is just wishful thinking. You 
don't solve a communication problem by becoming a cult icon, you 
solve it by communicating better, and earlier.


 Look at what happens in Rust. People get over each other to add 
 quality to the language, because if they don't, the next guy 
 has a better proposal, idea, or code. There's a crowd at the 
 door, putting pressure on the folks within. Folks don't go 
 around telling Niko Matsakis he's a chowderhead who could learn 
 a thing or two.


Seriously, I'm worried that the takeway you're getting from all 
this is "the community is unreasonable, unlike Rust's community 
which has a healthy respect for the maintainers". It's not just 
that. The D community is has a ton of people whose experience 
writing a PR is "I spent two weeks writing that code, got a 
comment from Walter six months later that asked to clarify what X 
did, I added some documentation, and I haven't had any news for a 
year".

If you look at the open PRs on rust, all PRs in page 3 are less 
than two weeks old. By comparison, some PRs in page 3 of dmd are 
from May 2018, almost a year ago!

I realize that the D team has way less resources than Mozilla to 
dedicate to following PRs, but acting like the only problem is 
that PR authors are capricious is just disingenuous.
Mar 29 2019
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Friday, 29 March 2019 at 21:28:40 UTC, Olivier FAURE wrote:

 Seriously, I'm worried that the takeway you're getting from all 
 this is "the community is unreasonable, unlike Rust's community 
 which has a healthy respect for the maintainers". It's not just 
 that. The D community is has a ton of people whose experience 
 writing a PR is "I spent two weeks writing that code, got a 
 comment from Walter six months later that asked to clarify what 
 X did, I added some documentation, and I haven't had any news 
 for a year".


I don't how long you've been around, but this community seems to 
me as indeed unreasonable and often disrespectful (I've been 
doing it too at times).

The sheer number of people trolling and wanting to hurt other 
people personally over the years has been really disconcerting.

We need to attract users who _need_ D to make stuff, contribute 
back (in time and money), and stop asking random useless things 
every other day of the week. And the  problem compounded because 
the abusive behaviour sometimes led to the most langage change!

The problem with D is that it has an "unscalable" community of 
very demanding people that bring not enough, so like a startup 
without scale economics it has difficulty to scale.

As a friend working on Eclipse told me, "all things being equal, 
having users is a liability" so you have to ask them to create 
more value than being removed.

Something that we do that I think doesn't work is... the lack of 
a Code of Conduct and a rule of not banning anyone, which opens 
the gate of small, constant, rampant abuse that can surface at 
the first occasion.

And something we should do more is put trust into the leadership 
more because, of all the mistakes they have been accused of, so 
many of the decisions ended up being the right ones.
Apr 03 2019
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 3 April 2019 at 12:33:38 UTC, Guillaume Piolat 
wrote:

 The sheer number of people trolling and wanting to hurt other 
 people personally over the years has been really disconcerting.


I realize that it's difficult to give examples here without 
calling specific people out, but without giving examples I have 
no way to judge what your standard for "trolling and wanting to 
hurt other people personally" is. That's kind of a pretty hefty 
accusation to make without the ability to ethically provide 
evidence. I think either you need to explicitly call people out 
here or you need to formulate this more leniently, as it stands 
it's kind of like announcing "several people in this room are 
psychotic killers, I can't say who, but that's why we need to 
take urgent action."

It is fundamentally epistemically improper to demand action on 
weighty evidence that cannot be ethically provided.
Apr 03 2019
next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 3 April 2019 at 14:09:49 UTC, FeepingCreature wrote:

 It is fundamentally epistemically improper to demand action on 
 weighty evidence that cannot be ethically provided.


I've been perhaps overstating my point, but no I'm not going to 
namecall - let's talk about the anonymous accounts.

One prime example is in one message juste above ours.

Once Andrei started saying things remotely uncomfortable for D, 
one "helpful" anonymous individual posted it to HN. Surely the 
intent couldn't be positive for D.

It gives the impression it's OK to disrespect the leadership, and 
that it's OK to shit in the soup.

Such anonymous actions have been going on since the beginning of 
D times, it gives a baseline upon more respectable personas can 
act in ways they wouldn't have.

I don't _demand_ anything, one action that would make sense if - 
in my opinion - to stop anonymity in these forums, and add 
accountability back.
Apr 03 2019
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 3 April 2019 at 14:09:49 UTC, FeepingCreature wrote:

 On Wednesday, 3 April 2019 at 12:33:38 UTC, Guillaume Piolat 
 wrote:

 The sheer number of people trolling and wanting to hurt other 
 people personally over the years has been really disconcerting.


 I realize that it's difficult to give examples here without 
 calling specific people out, but without giving examples I have 
 no way to judge what your standard for "trolling and wanting to 
 hurt other people personally" is. That's kind of a pretty hefty 
 accusation to make without the ability to ethically provide 
 evidence. I think either you need to explicitly call people out 
 here or you need to formulate this more leniently, as it stands 
 it's kind of like announcing "several people in this room are 
 psychotic killers, I can't say who, but that's why we need to 
 take urgent action."

 It is fundamentally epistemically improper to demand action on 
 weighty evidence that cannot be ethically provided.


They seems to come and go, sometimes posing as enthusiastic users 
who want some specific thing to happen and get all upset when 
people don't take too kindly to it. Sometimes they impersonate  
other community members and stoke the flames.

Unfortunately I don't have any links offhand (I'm madly packing) 
and have forgotten the names they posted under. The egregious one 
tend to be deleted from the forum interface (although I think 
they are still on the mailing lis that backs it).
Apr 03 2019
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/3/19 8:33 AM, Guillaume Piolat wrote:

 I don't how long you've been around, but this community seems to me as 
 indeed unreasonable and often disrespectful (I've been doing it too at 
 times).


I also think we have patterns of negativity. In the recent days I have 
received a number of private messages also mentioning that as an ongoing 
difficulty (you know who you are; thank you). I'm not very worried about 
some heat in casual discussions in the forums, though all of us could 
use more civility. I am, however, worried about negativity "in 
production", i.e. on github and in DIP-related and other consequential 
exchanges in the forums.

My current hypothesis is it has to do with people's lack of time.

Consider modeling a "specialist short on time". For a while I tried to 
model lack of time as a sort of decrease in IQ. For sure that's happened 
to me. Whenever I gave myself only a few minutes to make a decision on a 
pull request, that would definitely count as a net decrease in competence.

However, that model is not very accurate. For example, a specialist 
short on time would still be able to point out a mistake that would 
escape a non-specialist.

So a better model is, a specialist short on time in a design or code 
review has a more negative bent than one with time. This is because 
software is quintessentially constructive. In math, proving a negative 
is difficult because the burden is to prove something can't exist. In 
software, proving a positive is more difficult because it must be 
constructed. What is easy in code and design reviews is small "proofs" 
that an existing proposal has a problem. It's also time effective - a 
threading specialist may see a five-line race pattern immediately. A 
programming languages expert would see how qualifiers are bad for 
postblits in a minute.

Most of all, pointing out a negative gives the reviewer undeniable 
expertise high ground and a sense of doing the right thing. I pointed 
out a problem, the reasoning goes, thus alerting people that a mistake 
could be made. The reviewer gets a good jolt of satisfaction and moves 
on with their day.

The fallacy within is akin of the one of broken windows. The first-order 
reasoning goes, a dangerous mistake is being averted and correctness or 
at least some notion of consistency (people in software design love 
consistency) is being preserved.

The second-order effects, however, are numerous and pernicious. The main 
problem is, again, the constructive nature of software: often, a pull 
request or some other proposal embedding a mistake originates from a 
genuine need. People need to use malloc and free in pure code. They need 
reference-counted collections that are also immutable. Then, our 
archetypal specialist-short-on-time reviewer points out how that is 
problematic and considers the matter done with. "The problem is solved," 
- the subtext reads almost triumphantly - "it can't be done". To a 
non-investor specialist interested in correctness, this is a perfectly 
valid outcome. To the poor sap who wanted to get work done, that's none 
too helpful.

Contagion is another second-order effect. People in the community, 
would-be contributors and reviewers, see specialists who are mostly 
negatively biased. Clearly they know what they're talking about, so they 
are admired and set the standard everybody is aspiring to. Soon enough, 
it becomes "comme il faut" to point out weaknesses in proposed code 
instead of doing the much more difficult and time-consuming work of 
helping improving it, proposing better alternatives, and such. So all 
reviewers turn negative, whether they are specialists or not.

As soon as I'd put a pull request that had any informational entropy to 
it (i.e. not 100% obvious and boring), I might as well put the ear to 
the ground to hear the sound of shovels digging. The Vickers getting 
mounted. Barb wire getting rolled. Reviewers were getting ready for 
trench warfare. It has sapped all of my creative streak. My students, 
enthusiastic but scared soldiers, hesitate when they hear my whistle. 
"They'll kill me if I put this pull request out for review," I heard 
more than once put this way or another.

More specialists would help, but many specialists with a short time each 
would be less than helpful. We need steady energy, not just the 
occasional jolt of power. Electrical grid, not lightning. Great Work 
done by invested specialists has the true potential to turn negativity 
around.
Apr 04 2019
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 29 March 2019 at 16:26:06 UTC, Andrei Alexandrescu wrote:

 It's most interesting what happens with Walter. He is one of 
 the best hackers (in the Paul Graham sense) I've ever known


Walter is a brilliant programmer whose historic experience is exceedingly
valuable. You're also an excellent programmer and a good writer who brings very
valuable higher-thought discipline to library design.

Where you guys are weak is project management. A project manager needs to set
clear expectations, facilitate communication among the team, prioritize work
based on real world requirements, and motivate the team to do that work,
striking the right balance between autonomy and direction to maintain morale
while meeting requirements.

I'm sure facebook has some good managers who keep that room of engineers
focused on the big picture and willing to come back to work week after week. D
needs a good manager too. They make a real difference.

I betcha if we got a good manager, contributors would be a lot happier working
in a shared direction and the manpower problems the PR queue sees now would
evaporate. And best of all, it'd let you and Walter go back to doing the
awesome work you really excel at, instead of arguing with the community so much.
Mar 29 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Mar 29, 2019 at 07:45:35PM -0400, Adam D. Ruppe via Digitalmars-d wrote:

 On Friday, 29 March 2019 at 16:26:06 UTC, Andrei Alexandrescu wrote:

 It's most interesting what happens with Walter.




[...]

 Walter is a brilliant programmer whose historic experience is
 exceedingly valuable. You're also an excellent programmer and a good
 writer who brings very valuable higher-thought discipline to library
 design.
 
 Where you guys are weak is project management. A project manager needs
 to set clear expectations, facilitate communication among the team,
 prioritize work based on real world requirements, and motivate the
 team to do that work, striking the right balance between autonomy and
 direction to maintain morale while meeting requirements.


+1.

I spent the last hour writing a whole bunch of stuff to respond to this,
but decided instead to distill it all to just two things:

(1) It's clear that Walter & Andrei excel at technical expertise, but
lack in the people skills department.  This has been shown time and
again in the way various interactions with different people panned out
over the years.  (Please don't take this as an insult or a personal
attack, because I don't intend it that way, and I also recognize myself
to be belong to the same category. We techie types simply aren't good at
people skills, that's all there is to it.)

(2) OTOH, people skills are exactly what's needed to foster a healthy,
thriving community around D.  It's unrealistic to expect a volunteer-run
open source project to operate the same way as a company that pays its
employees to do assigned work.  A volunteer project *can* produce work
of equivalent quality, but the lack of monetary compensation must be
replaced by a different kind of "currency": an equivalent amount of
inspirational leadership. I.e., inspire the people enough and they'll go
out of their way to do whatever you want, pay or no pay. And a big part
of this is effective communication of the high-level goals of the
project and what's expected of would-be contributions. Fail to do that,
and be prepared for the same reaction you might get if you gave an
employee a sudden big pay cut while still expecting the same amount (and
quality) of work as before.

The lack of manpower, etc., are not a cause of our problems; it's a
symptom of our lack of proper leadership (in the social sense --
clearly, W&A have excellent technical leadership; but that's not what's
missing here).  I don't blame Walter and Andrei for this -- we techie
types just aren't good at this sort of stuff.  We need a manager who is.

tl;dr: Walter & Andrei shouldn't be expected to bear the brunt of the
social aspect of community building. That's not their forte, and even if
they could do it, it'd be a waste of their technical talents.  We need a
proper manager who can do a good job with the community-related /
people-related stuff, and leave W&A to do what they're best at doing.



 I'm sure facebook has some good managers who keep that room of
 engineers focused on the big picture and willing to come back to work
 week after week. D needs a good manager too. They make a real
 difference.
 
 I betcha if we got a good manager, contributors would be a lot happier
 working in a shared direction and the manpower problems the PR queue
 sees now would evaporate. And best of all, it'd let you and Walter go
 back to doing the awesome work you really excel at, instead of arguing
 with the community so much.


Yes and yes.  It's about time we acknowledge where our weaknesses lie,
and take real steps at fixing the problem, instead of continuing to lie
to ourselves with the fantasy that social problems can be cured by
technical expertise.  We techie types all wish it were so, but sadly the
real world simply doesn't work that way.


T

-- 
Claiming that your operating system is the best in the world because more
people use it is like saying McDonalds makes the best food in the world. --
Carl B. Constantine
Mar 29 2019
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 30 March 2019 at 01:10:59 UTC, H. S. Teoh wrote:

 And a big part of this is effective communication of the 
 high-level goals of the project and what's expected of would-be 
 contributions.


I'm hoping we can set a lot of that high level goals at the dconf 
AGM (along with a lot of other problems). I'll be publishing the 
(draft, to be added to by others) agenda soon.
Mar 30 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/29/19 9:10 PM, H. S. Teoh wrote:

 It's clear that Walter & Andrei excel at technical expertise, but
 lack in the people skills department.


As Walter tells me on the phone when it comes about community, "the main 
problem with our community is it's poorly led" - a very humble, 
humbling, and positive take.

Speaking for myself, my doing a poor job at leadership in our community 
is so obvious to me, it has almost become a cop-out. It's second to none 
in terms of failure per unit of time invested, so in a way simply 
admitting I'm just not good at it is the easy way to explain an 
overwhelmingly complex reality.

This had me revisit the past for similarities and differences. And that 
inevitably takes to C++.

In the C++ community the outcome has been quite the opposite. After 
Modern C++ Design (of which a friend went so far as to say "showed 
Bjarne what C++ really was about"), I have inadvertently enjoyed 
enormous impact and inspirational power, whilst being aloof to it. One 
forum post was enough to determine Hans Boehm and Herb Sutter to lead 
the standardization of threads in C++11; off-the-cuff ideas and 
conversations became N4259 or P0323R1; Mojo, an afternoon's worth of 
work, may have provided at least part of the reason for rvalue 
references. To this day, I am receiving credit about things I'd all but 
forgotten about. I could burp in a keynote and someone will create a 
great C++ template out of it. I knew things came full-circle when I got 
this question following my "The Next Big Thing" talk: "Have you heard of 
std::conditional?" (meant as a clever comeback to my plea for "static 
if"). I realized that both the person asking and myself had forgotten 
I'd invented the blessed thing.

And this is most puzzling, because I haven't given C++ the time of the 
day in a long while, so in terms of 
at-least-somewhat-related-to-leadership-impact per unit of effort, my 
participation C++ has been quite productive. (And for a good part it's 
been unwitting and almost unwilling, a la Stepan Trofimovich 
Verkhovensky in Dostoyevsky's "Demons" - hopefully in a positive way 
though.)

Not to say that arguably the very best of my work is to be found in D, 
many miles away from what I could ever do for C++. But even not counting 
that: with me being a common part of the inequation, I can be simplified 
away. Which leads to the most puzzling question: why have the outcomes 
across the C++ and D communities have been so different?

The answer to this question may unlock the potential of our community.
Mar 30 2019
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 31 March 2019 at 00:18:29 UTC, Andrei Alexandrescu 
wrote:

 Not to say that arguably the very best of my work is to be 
 found in D, many miles away from what I could ever do for C++. 
 But even not counting that: with me being a common part of the 
 inequation, I can be simplified away.


That would assume that your output (not input) to the two are 
equal and more importantly separable form the outputs of others. 
The second one is definitely not true.


 Which leads to the most puzzling question: why have the 
 outcomes across the C++ and D communities have been so 
 different?


Leadership and vision (or rather lack thereof) are two of the 
most critical issues holding D back. Forget quality for the 
moment, it can be well approximated by a simple game of numbers:

D:
     Walter and Andrei
     No vision document for I've forgotten how long
     No regular direction steering meetings with users (mod that 
one ~5 months ago)
     One conference a year
     A few local regional get togethers
     DLF

C++:
     Herb, Marshall, Bjarne, ...
     Multiple direction documents / roadmaps for language, 
library, ecosystem, HPC, ...
     C++ standards committees: 1 week every 3 months
     I don't even know how many conferences per year
     Who knows how many local gatherings
     C++ alliance / C++ standards body

And I'll be blunt, much of your recent leadership has actually 
been gatekeeping, especially w.r.t refactoring.

How to remedy this?

Well _I'm_ starting with:
the DConf AGM (draft agenda to be published soonâ„¢).
DLF quarterly meetings, especially if we can get them to coincide 
with various regional quarterly gatherings
Greater (corporate) participation in DLF processes, vision 
outreach.

What are you going to do?
Mar 30 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/30/2019 6:51 PM, Nicholas Wilson wrote:

 And I'll be blunt, much of your recent leadership has actually been
gatekeeping, 
 especially w.r.t refactoring.


I know we're having differences about what constitutes a good refactoring, and 
what isn't. I hope that between us at DConf with the aid of some suds we can 
reach a consensus on this.
Mar 30 2019
prev sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Sunday, 31 March 2019 at 00:18:29 UTC, Andrei Alexandrescu 
wrote:

 why have the outcomes across the C++ and D communities have 
 been so different?

 The answer to this question may unlock the potential of our 
 community.


Isn't suggesting ideas for language features the easy part? As 
opposed to actually making a formal proposal, teasing out the 
corner cases, comparing alternatives, getting reviewers to accept 
it, etc?

If your experience in C++ has been mostly writing libraries and 
proposing ideas, and your experience in D has been mostly 
reviewing actual work and dissecting complete proposals, then of 
course C++ will seem easier to you.

I mean, it's not like the standardization process in C++ is 
painless. In fact, it's renowned for being a byzantine 
bureaucracy.
Mar 31 2019
prev sibling parent Rubn <where is.this> writes:
On Friday, 29 March 2019 at 16:26:06 UTC, Andrei Alexandrescu 
wrote:

 Look at what happens in Rust. People get over each other to add 
 quality to the language, because if they don't, the next guy 
 has a better proposal, idea, or code. There's a crowd at the 
 door, putting pressure on the folks within. Folks don't go 
 around telling Niko Matsakis he's a chowderhead who could learn 
 a thing or two.


There's a saying, keep your eyes on your own plate. You really 
want to compare to rust? How about starting with the fact D 
doesn't even provide a 64-bit binary on Windows for their 
compiler. That's the sort of trivial problems D has yet to even 
solve. You best look at your own plate first before even 
considering to compare to others.
Mar 30 2019
prev sibling next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 29 March 2019 at 13:57:57 UTC, Andrei Alexandrescu 
wrote:


 Or a team of subordinate apprentice/junior engineers (who have 
 demonstrated sufficient potential) working under the 
 mentorship of a master who is willing to make an investment.


 This is a lot more difficult to set up than it seems.

 One reason is that surprisingly few of the contributors come 
 here to learn, to acquire knowledge.


[...]

 I think it has to do with a simple reality - I was trying to 
 mentor people who didn't want to be mentored.

 So I think it would be difficult to establish a master/mentee 
 dynamics.


I understand (as my head hangs in disappointment).  I've observed 
it for myself, and may even be guilty of it in a few instances.

This is why I added the word "subordinate" to my sentence before 
I submitted it.  The apprentice would have to agree to 
subordination as part of the up-front agreement.

It's not just knowledge the apprentice is seeking.  (S)he may be 
also be seeking credentials and experience.  The credentials 
include the trophy of accomplishment and the privilege to use the 
master's name as a reference in their future prospects.  
Insubordination results in being fired and losing those 
credentials, not to mention the time and effort spent, and the 
opportunity to expand their knowledge.

The master would have to be selective, just like an employer, to 
find the right people for the apprenticeships, and there would 
have to be some up-front agreement with conditions and 
expectations.

I don't have any experience with this kind of thing, so perhaps I 
don't know what I'm talking about.  It's just a shame that people 
are willing to take out a loan for 10s of thousands of dollars 
for an education built around contrived assignments, when there's 
an opportunity here to do real meaningful work, led by some of 
the most reputable names in the field, that doesn't cost 
anything.  What am I missing?

Mike
Mar 29 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/29/2019 8:38 AM, Mike Franklin wrote:

 It's just a shame that people are willing to take out a 
 loan for 10s of thousands of dollars for an education built around contrived 
 assignments, when there's an opportunity here to do real meaningful work, led
by 
 some of the most reputable names in the field, that doesn't cost anything. 
What 
 am I missing?


One thing I really enjoy is working with people who are better at it than I am. 
It's the fastest, most effective way to up your game, and besides, it's fun.
Mar 29 2019
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Friday, 29 March 2019 at 13:57:57 UTC, Andrei Alexandrescu 
wrote:

 This is a lot more difficult to set up than it seems.

 One reason is that surprisingly few of the contributors come 
 here to learn, to acquire knowledge. Most come to dispense. I 
 spent long times reviewing pull requests in our community, 
 calibrated to the reviews standards at Facebook (however 
 controversial, they have an impressive software engineering 
 force).


Therein lies your problem, you are attempting to fit a square peg 
in a round hole.
You are not dealing with well paid highly qualified engineers 
whose job it is to do what needs to be done.
For the most part you are dealing with an intermittent army of 
volunteers of a range of abilities for whom this is a hobby.


 There, the code review process is locked in a virtuous circle: 
 you get good reviews that help you improve, and you'd be 
 motivated to give good reviews to improve others'.

 Here, my reviews were more often than not met with hostility.


I can't speak for your phobos reviews, but most of the 
interactions I've had with you on dmd were not at all useful.


 As a pattern, [an author] would be more willing to write the 
 code once and then defend it in its current state, rather than 
 improve it through review. In a few extreme cases, people flat 
 out told me they'll abandon the PR if I don't take it as it is.


Been there
https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733
done that
https://github.com/dlang/dmd/pull/8557#issuecomment-416798558


 Larger disagreements on matters of architecture and design were 
 often taken as personal offenses.


With responses like that I can see why.


 I'd wake up in the morning for days, then weeks and months, 
 with one thought: "I have so many things to do today, and I 
 like none of them." The irony! Here I was, independent, working 
 for myself on whatever I dreamed of. Yet this was worse than 
 the worst jobs I've ever had. How did I get into this 
 situation? I think it has to do with a simple reality - I was 
 trying to mentor people who didn't want to be mentored.


Probably, chances are most of them wanted to get shit done...


 So I think it would be difficult to establish a master/mentee 
 dynamics.


... but there are definitely those out there looking for 
experience, looking to become better programmers (GSoC students 
are a good source).
Mar 30 2019
prev sibling next sibling parent reply destructionator gmail.com writes:
On Thursday, 28 March 2019 at 19:12:40 UTC, H. S. Teoh wrote:

 The TypeInfo stuff, for one thing, will take some work to reach 
 your vision of a completely opt-in library.  Currently, trying 
 to link without druntime will cause a bunch of errors related 
 to missing TypeInfo's.


Not true - that was fixed like a year ago.

See http://dpldocs.info/this-week-in-d/Blog.Posted_2019_02_25.html#no-runtime-d

The code there compiles with no extra code; it just works.

and my "last year in D" post on the official blog called this out with the
release (I just don't remember the link)



 Or the GC initialization in rt_init / rt_term.  Or, for that 
 matter, the initialization code that packages command-line 
 arguments into a string[] before invoking Dmain.


The GC is initialized on the first use.

Lots of movement in this direction already happened in 2018!
Mar 28 2019
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 19:23:30 UTC, destructionator wrote:

 On Thursday, 28 March 2019 at 19:12:40 UTC, H. S. Teoh wrote:

 The TypeInfo stuff, for one thing, will take some work to 
 reach your vision of a completely opt-in library.  Currently, 
 trying to link without druntime will cause a bunch of errors 
 related to missing TypeInfo's.


 Not true - that was fixed like a year ago.

 See 
 http://dpldocs.info/this-week-in-d/Blog.Posted_2019_02_25.html#no-runtime-d

 The code there compiles with no extra code; it just works.

 and my "last year in D" post on the official blog called this 
 out with the release (I just don't remember the link)


[...]


 Lots of movement in this direction already happened in 2018!


Yes, the release that enabled that was 2.079 
(https://dlang.org/changelog/2.079.0.html)

It started with `ModuleInfo` which set a precedent for then doing 
`TypeInfo` and `Throwable`

https://github.com/dlang/dmd/pull/7395
https://github.com/dlang/dmd/pull/7768
https://github.com/dlang/dmd/pull/7799
https://github.com/dlang/dmd/pull/7786

The work is also ongoing with Dan Printzell recently deciding on 
decoupling `TypeInfo` from the runtime hooks for his GSoC 
proposal 
(https://forum.dlang.org/post/dunewuhfbkqnbtemicmk forum.dlang.org).  Let's be
sure to give him our support.

Mike
Mar 28 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 28, 2019 at 03:23:30PM -0400, destructionator--- via Digitalmars-d
wrote:

 On Thursday, 28 March 2019 at 19:12:40 UTC, H. S. Teoh wrote:


[...]

 Or the GC initialization in rt_init / rt_term.  Or, for that 
 matter, the initialization code that packages command-line 
 arguments into a string[] before invoking Dmain.


 
 The GC is initialized on the first use.


Yes, but not calling rt_init still causes a segfault when you try to use
the GC.  So, we're not quite there yet.



 Lots of movement in this direction already happened in 2018!


That's good to know.

I'm still wondering, though, how we're going to drop druntime while
still maintain backward compatibility.  Or how we could skip the startup
code that invokes module ctors and packages command-line args into
string[].  It would be awesome if we could pull this off, but I don't
currently see how we could do it without major breakage.


T

-- 
Why waste time learning, when ignorance is instantaneous? -- Hobbes, from
Calvin & Hobbes
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 3:31 PM, H. S. Teoh wrote:

 On Thu, Mar 28, 2019 at 03:23:30PM -0400, destructionator--- via Digitalmars-d
wrote:

 On Thursday, 28 March 2019 at 19:12:40 UTC, H. S. Teoh wrote:


 [...]

 Or the GC initialization in rt_init / rt_term.  Or, for that
 matter, the initialization code that packages command-line
 arguments into a string[] before invoking Dmain.


 The GC is initialized on the first use.


 
 Yes, but not calling rt_init still causes a segfault when you try to use
 the GC.  So, we're not quite there yet.


That'd need to be fixed, and it should be possible without disrupting 
clients.


 Lots of movement in this direction already happened in 2018!


 
 That's good to know.
 
 I'm still wondering, though, how we're going to drop druntime while
 still maintain backward compatibility.  Or how we could skip the startup
 code that invokes module ctors and packages command-line args into
 string[].  It would be awesome if we could pull this off, but I don't
 currently see how we could do it without major breakage.


Compiler flag.
Mar 28 2019
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 20:02:17 UTC, Andrei Alexandrescu 
wrote:


 I'm still wondering, though, how we're going to drop druntime 
 while
 still maintain backward compatibility.  Or how we could skip 
 the startup
 code that invokes module ctors and packages command-line args 
 into
 string[].  It would be awesome if we could pull this off, but 
 I don't
 currently see how we could do it without major breakage.


 Compiler flag.


D is so powerful, this can also be done in other ways as well.

As Andrei alluded to in another post, we just need to embrace 
addition.  Keep the existing logic path in place, and create a 
new one that does what we want it to do.  With 
design-by-introspection, and probably a number of other awesome D 
facilities, the correct logic path can be chosen at compile-time 
based on what the user has declared in their code.

Another idea is allowing "module inheritance" for lack of a 
better term.  Users can "inherit" from an "abstract" module but 
override the default startup and shutdown logic.  Perhaps we 
refactor the current logic path to inherit from this "abstract" 
module but add all of the existing startup and shutdown 
implementation as overrides.  The compiler is then modified to 
"inherit" from that module by default.  Users that don't want the 
default can "inherit" from the "abstract" module and provide 
their own implementation.

I obviously haven't thought all of this through; I'm just trying 
to illustrate D is so powerful that there are other ways to 
achieve a similar result without always resorting to compiler 
flags, though a compiler flag may be the best option.

Mike
Mar 28 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/28/19 10:47 PM, Mike Franklin wrote:

 On Thursday, 28 March 2019 at 20:02:17 UTC, Andrei Alexandrescu wrote:
 

 I'm still wondering, though, how we're going to drop druntime while
 still maintain backward compatibility.  Or how we could skip the startup
 code that invokes module ctors and packages command-line args into
 string[].  It would be awesome if we could pull this off, but I don't
 currently see how we could do it without major breakage.


 Compiler flag.


 
 D is so powerful, this can also be done in other ways as well.
 
 As Andrei alluded to in another post, we just need to embrace addition.  
 Keep the existing logic path in place, and create a new one that does 
 what we want it to do.  With design-by-introspection, and probably a 
 number of other awesome D facilities, the correct logic path can be 
 chosen at compile-time based on what the user has declared in their code.


*nod*


 Another idea is allowing "module inheritance" for lack of a better 
 term.  Users can "inherit" from an "abstract" module but override the 
 default startup and shutdown logic.  Perhaps we refactor the current 
 logic path to inherit from this "abstract" module but add all of the 
 existing startup and shutdown implementation as overrides.  The compiler 
 is then modified to "inherit" from that module by default.  Users that 
 don't want the default can "inherit" from the "abstract" module and 
 provide their own implementation.


Yah, I actually had even code for an idea - after each module is loaded, 
the following code is automatically injected at the end of it:

mixin(__onImportedModule(__MODULE__));

This is evaluated in the context of the importer (not imported), an 
essential detail. By default of course the function returns the empty 
string.

This allows user code to add things like serialization, factories and 
other introspection-based utilities automatically upon import.
Mar 29 2019
next sibling parent =?UTF-8?B?THXDrXM=?= Marques <luis luismarques.eu> writes:
On Friday, 29 March 2019 at 13:05:31 UTC, Andrei Alexandrescu 
wrote:

 Yah, I actually had even code for an idea - after each module 
 is loaded, the following code is automatically injected at the 
 end of it:

 mixin(__onImportedModule(__MODULE__));

 This is evaluated in the context of the importer (not 
 imported), an essential detail. By default of course the 
 function returns the empty string.

 This allows user code to add things like serialization, 
 factories and other introspection-based utilities automatically 
 upon import.


Jean-Louis argued for that overall feature at last year's DConf, 
and I also agree. Such a feature would be useful for a lot of 
things, although it's one of those "with great power comes great 
responsibility" kind of features.
Mar 29 2019
prev sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Friday, 29 March 2019 at 13:05:31 UTC, Andrei Alexandrescu 
wrote:

 This allows user code to add things like serialization, 
 factories and other introspection-based utilities automatically 
 upon import.


You mean something like this?

     // someFile.proto
     syntax = "proto2";

     message SomeType {
       required int32 x = 1;
       required int32 y = 2;
       optional string label = 3;
     }

     message AnotherType {
       required SomeType xy = 1;
       required int32 z = 2;
     }


     // foobar.d
     import "someFile.proto" : SomeType, AnotherType;

Because that would be seriously awesome.
Mar 29 2019
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 18:47:49 UTC, Andrei Alexandrescu 
wrote:


 Oh, and druntime must go.

 The whole distinction between the runtime library and the 
 standard library is clowny and has more to do with poorly 
 ported tradition from other languages, than with anything else.

 We need one standard library that is entirely pay-as-you-go 
 (e.g. empty main() means no library file is ever needed for 
 linking) and that offers an opt-in continuum.


yes, Yes, YES!  Now that's a vision I can get behind.  Time to 
start dusting off my DMD and druntime forks.  I'm in.

Mike
Mar 28 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 12:47:49 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:

 On 3/28/19 2:41 PM, Andrei Alexandrescu wrote:

 On 3/28/19 2:30 PM, H. S. Teoh wrote:

 The only way I can see this happening is if we start a new iteration of
 Phobos, like std.v2.*


 There'd be carefully carved subsets per capability, such as
 std.v2.betterc.*, std.v2.nogc.*, std.v2.noexcept.*, std.v2.safe.

 The capability sets can be combined in alphabetic ordering. E.g.
 std.v2.noexcept.nogc.* has the subset that uses no gc and no exceptions,
 but std.v2.nogc.noexcept does not exist.

 Through the magic of public imports and aliasing there would be no code
 duplication. Introspection could help, too, a la "define public aliases
 for all  safe definitions in that module". Some improvements to ddoc
 might be needed to "see" through aliases.


 Oh, and druntime must go.

 The whole distinction between the runtime library and the standard
 library is clowny and has more to do with poorly ported tradition from
 other languages, than with anything else.

 We need one standard library that is entirely pay-as-you-go (e.g. empty
 main() means no library file is ever needed for linking) and that offers
 an opt-in continuum.


LOL. That's a big thing to drop in as an afterthought. There are some useful
organizational aspects of having druntime, but it has become a bigger and
bigger problem over time as we've tried to do stuff that's required to be in
druntime but which effectively requires stuff that's in Phobos - a long
standing example of that being UTF encoding and decoding. AFAIK, there are
still a number of hurdles to truly making it pay-as-you-go, but if we can do
it, it would be great.

This would also be a great opportunity to fix some of the issues with shared
in druntime (in particular with the mutex and conditional variable stuff as
well as Thread). Too much of it doesn't use shared properly (if at all), and
in the case of Thread, it has  safe functions that end up passing
thread-local objects to a new thread, which would be much easier to do with
something like v2 or moving all of that to std (certainly, there are things
that should be fixed there that we can't fix in-place).

All in all though, I think that we're going to have to plan carefully with
how we go forward with major changes like this so that we minimize the risk
of splitting the community. With D's module system, we have a lot of
flexibility, but when we start replacing core code and concepts with new,
incompatible versions in separate modules, we do risk compatibility
problems. However, full-sale replacement of some modules would definitely
fix some of the issues that replacing stuff in pieces within a module has.

- Jonathan M Davis
Mar 30 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/19 2:49 PM, Jonathan M Davis wrote:

 This would also be a great opportunity to fix some of the issues with shared
 in druntime


The problem here is, of course, that nobody knows exactly what shared is 
supposed to do. Not even me. Not even Walter.

One monumental piece of work would be DIP 1021 "Define the semantics of 
shared". Then people who build with -dip1021 know what to do and what to 
expect. Now that would patch one of the bigger holes mentioned in the 
other post.
Mar 30 2019
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 31 March 2019 at 00:41:15 UTC, Andrei Alexandrescu 
wrote:

 On 3/30/19 2:49 PM, Jonathan M Davis wrote:

 This would also be a great opportunity to fix some of the 
 issues with shared
 in druntime


 The problem here is, of course, that nobody knows exactly what 
 shared is supposed to do. Not even me. Not even Walter.


If that is the case then you should not be so hostile to 
suggestions *cough *Manu *cough* to improve that (maybe that was 
more Walter than you, I don't remember).


 One monumental piece of work would be DIP 1021 "Define the 
 semantics of shared". Then people who build with -dip1021 know 
 what to do and what to expect. Now that would patch one of the 
 bigger holes mentioned in the other post.


Its in the vision section for the dconf AGM as:
shared ( and  safe and shared)
Mar 30 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/19 10:08 PM, Nicholas Wilson wrote:

 One monumental piece of work would be DIP 1021 "Define the semantics 
 of shared". Then people who build with -dip1021 know what to do and 
 what to expect. Now that would patch one of the bigger holes mentioned 
 in the other post.


 
 Its in the vision section for the dconf AGM as:
 shared ( and  safe and shared)


The necessity to work on shared was also present in the January 2015 
vision document (https://wiki.dlang.org/Vision/2015H1): "Nail down 
fuzzily-defined areas of the language (e.g. shared semantics,  property)."

Writing it down doesn't get it done.

Defining shared properly would take a team with a programming languages 
expert, a threading expert, and an application expert. (Some roles may 
be realized within the same person.) I know people within our community 
with the required expertise. But I don't know any who'd also have the 
time to embark on this.

I got burned out on writing vision documents (Walter was never a fan so 
I did them all) because it was difficult to figure their impact. 
Contributors asked for a vision document. So we started one. Then 
contributors continued doing what they were doing.
Mar 31 2019
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 7:30 AM, Andrei Alexandrescu wrote:

 But I don't know any who'd also have the time to embark on this.


s/any/enough/
Mar 31 2019
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 31 March 2019 at 11:30:08 UTC, Andrei Alexandrescu 
wrote:

 I got burned out on writing vision documents (Walter was never 
 a fan so I did them all) because it was difficult to figure 
 their impact. Contributors asked for a vision document. So we 
 started one. Then contributors continued doing what they were 
 doing.


I have to say I find the posts you have made the last few days to 
be very open and containing a lot of truth. It is very refreshing 
and inspiring to read. I hope you can find your way again and do 
what you do best.
Mar 31 2019
parent reply bachmeier <no spam.net> writes:
On Sunday, 31 March 2019 at 12:03:30 UTC, Sebastiaan Koppe wrote:


 I have to say I find the posts you have made the last few days 
 to be very open and containing a lot of truth. It is very 
 refreshing and inspiring to read.


At the risk of being negative, I had the opposite reaction after 
reading his posts. It is clear that he has no interest in being 
the leader of an open source project. He doesn't want to waste 
his time with average programmers, he only wants to spend his 
time working with the best of the best (by some metric) software 
engineers.

That's his option, of course, but it's as welcoming to new 
contributors as a rattlesnake pit (on par with Linus but using 
better language). An open source project leader needs to find 
ways to bring in new contributors, including some who are below 
average, and help them find a way to contribute. There needs to 
be an interest/willingness to think about and participate in all 
areas of the project, even non-elite things like documentation, 
newbie guides, tools, and marketing the language.

Open source leader and working on a narrow set of technical 
issues (like pounding out designs of some sort with 
best-of-the-best software engineers) have little overlap.


 I hope you can find your way again and do what you do best.


So do I, and he should work on the things that interest him, 
rather than trying to run an open source project.
Mar 31 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 10:12 AM, bachmeier wrote:

 It is clear that he has no interest in being the leader of an open 
 source project. He doesn't want to waste his time with average 
 programmers, he only wants to spend his time working with the best of 
 the best (by some metric) software engineers.


Good leadership is inspiring people from all walks of life toward doing 
extraordinary things. Point being, not all need to be extraordinary. 
Groups of extraordinary people working together do happen, but are rare 
enough to receive their own phrase - "Gang of Four", "Paypal Mafia", "K&R".

I knew this well before embarking on working on D, and I had no illusion 
about my own skill being remarkable to start with. So I had no 
unreasonable expectations going in or staying in.

What was surprising, which I alluded to in another post, is that the 
advice for improvement I'd dispense was frequently debated and 
occasionally taken as an offense. For whatever reason, I failed to hold 
the competence high ground. It's as if there was a samizdat manual by 
which contributors must do two essential things - antagonize Walter and 
debate my reviews. Needless to say, my growing testy didn't help one 
bit. There's this subtle Romanian proverb translated loosely as "it's 
mighty difficult helping who doesn't want to be helped". (I first heard 
it thirty years ago in the military and thought it's kind of silly.)

This has led to many thousand yard stare evenings I have yet to recover 
from.
Mar 31 2019
parent reply Rubn <where is.this> writes:
On Sunday, 31 March 2019 at 15:28:55 UTC, Andrei Alexandrescu 
wrote:

 There's this subtle Romanian proverb translated loosely as 
 "it's mighty difficult helping who doesn't want to be helped". 
 (I first heard it thirty years ago in the military and thought 
 it's kind of silly.)


Look at this comment as someone pointed out before:

https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733

Do you think after reading this someone is going to be more 
inclined to work on a pull request or less inclined? You 
basically said the entire pull request was useless and that any 
other pull requests like it should not happen. This does not 
promote or nurture discussion that leads to understanding, this 
simply terminates it.

Not sure which is worse between Walter not saying anything and 
leaving pull requests open for years without any additional 
comment. Or your comments that you think are "help".

I made a pull request to implement a feature I didn't think was 
that big of a change. Walter left a response that the change 
needed a DIP. Then some years later the feature gets added 
without a DIP. Walter never replied to my comments requesting for 
clarification. Scheisse like that is why people don't feel like 
contributing.
Mar 31 2019
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Sunday, 31 March 2019 at 16:18:35 UTC, Rubn wrote:


 Look at this comment as someone pointed out before:

 https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733


Yes, Andrei's review was a bit too strong but he did change 
course later in that thread.  Also recognize that the original 
author did not provide much motivation or justification (though, 
I'm guilty of that too).

I encourage contributors to please provide a short essay with 
your PRs to help everyone see the larger picture.  No one can pay 
attention to everything going on, and not everyone who reviews 
pull requests is intimately involved with the overall long-term 
objective that some PRs move forward.

It also makes things easier on the reviewer as they don't need to 
do so much of their own investigation to determine context (that 
always irritated me a little when I was reviewing; make it easy 
on me), and it increases the likelihood of a PR getting a review 
and getting accepted.

I do ask Andrei and Walter to understand that their reviews hold 
*much* more weight than others.  One minor objection from them 
makes others think "Well, Walter and Andrei don't like this, 
therefore my opinion is irrelevant".  The fundamental problem is 
there isn't any real due process to challenge and ask for 
reconsideration.  (e.g. 
https://github.com/dlang/dmd/pull/9506#issuecomment-477936480  -- 
No response).

Mike

When I was reviewing, it was always in the back of my mind "Are 
Walter and Andrei going to scold me if I merge this?", especially 
after they did once. In a way, we do work on behalf of Andrei and 
Walter and we wnat

Mike
Mar 31 2019
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Monday, 1 April 2019 at 00:01:04 UTC, Mike Franklin wrote:


 When I was reviewing, it was always in the back of my mind "Are 
 Walter and Andrei going to scold me if I merge this?", 
 especially after they did once. In a way, we do work on behalf 
 of Andrei and Walter and we wnat

 Mike


Oops, meant to delete that, but now I'm obliged to finish.

...want them to approve.

Mike
Mar 31 2019
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 1 April 2019 at 00:01:04 UTC, Mike Franklin wrote:

 On Sunday, 31 March 2019 at 16:18:35 UTC, Rubn wrote:


 Look at this comment as someone pointed out before:

 https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733


 Yes, Andrei's review was a bit too strong but he did change 
 course later in that thread.


Not to any actionable degree, the request for change is _still_ 
there.


 ... The fundamental problem is there isn't any real due process 
 to challenge and ask for reconsideration.  (e.g. 
 https://github.com/dlang/dmd/pull/9506#issuecomment-477936480  
 -- No response).


Yup.


 Also recognize that the original author did not provide much
 motivation or justification (though, I'm guilty of that too).


Oh I know that, this was done specifically for Iain, and 
therefore I gave no justification because he knew exactly what it 
was for, W&A came and trashed the party. Hmm, I shouldn't be so 
harsh, Walter's was a reasonable request (to split the bug fix 
from the refactor).

Andrei's OTOH was a drive-by putdown, and he should learn a 
lesson from that that there exists a line which when crossed will 
cause people to cut their losses. In fact I think most (perhaps 
this is a memory retention bias, but still) of my dealings with 
Andrei in DMD have been drive-by, usually with bad outcomes.
Mar 31 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 8:01 PM, Mike Franklin wrote:

 On Sunday, 31 March 2019 at 16:18:35 UTC, Rubn wrote:
 

 Look at this comment as someone pointed out before:

 https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733


 
 Yes, Andrei's review was a bit too strong but he did change course later 
 in that thread.  Also recognize that the original author did not provide 
 much motivation or justification (though, I'm guilty of that too).


Those were unnecessarily strong words.

Allow me to give a bit more context whilst clarifying I am not defending 
or condoning that attitude.

That comment came after a few phone conversations whereby Walter 
mentioned he has a deluge of pull requests to review. He said if he's to 
spend due time on all of those, he'd be unable to do any work of his 
own. Those pull requests that have these things in common (note that 
some may not apply to the pragma pull request, I'd just wrongly put it 
in the same bin):

* they are large
* the improvement they bring needs arguing
* their quality could be improved so they need careful review and a 
couple of passes of changes

This is what I call Good Work.

Good Work is often sizable in quantity, and is visibly the result of 
concerted effort by a competent person. Those unfamiliar with a codebase 
and with the subtleties of the task at hand cannot produce Good Work.

(Contrast that with Bad Work: by definition, it is easy to inspect and 
reject. It does not create a major slowdown in a project.)

Good Work is more often than not complex. Complexity is misrecognized as 
evidence of the complexity of the problem; the task was hard, the 
reasoning goes, because the solution is difficult in proportion.

Good Work begets more Good Work. Typically Good Work produces context, 
opportunity, and precedent for more of the same. The same reviewer who 
rubber stamped a piece of Good Work will have an idea how to produce 
more Good Work derived from it. The kind of environment where Good Work 
is revered encourages its creation, in a cycle that creates the illusion 
of progress. Because Good Work is complex, it produces "bug ripples" 
whereby increasingly complex Good Work fixes one bug but is liable to 
introduce others.

Good Work is difficult to argue against. The main argument in favor, 
which is very difficult to counter, is that Good Work is better than Bad 
Work and better than No Work. It is easy to create an opinion trend in 
favor of this or that Good Work.

However, there are problems with Good Work. The first one is scale: an 
accumulation of Good Work does not add up to Great Work. More often, the 
increasing entropy leads to the thermal death of the project.

An accumulation of Good Work is what leads to six nested calls in an 
advanced library to convert a string to an integer. Even if they are 
inlined away in execution, their smell persists.

An accumulation of Good Work is what leads to a multitude of "is this 
kinda sorta almost like-a string hey I really mean it this time quite 
like a string" tests, or "we must to add  trusted every five lines of code".

The other problem of Good Work is that it takes the air out of the room, 
causing Great Work to suffocate. Good Work takes great effort to author, 
debate, review, debug, and maintain. All that takes away time and mental 
share that should go into Great Work instead.

Which brings us to Great Work.

Great Work solves difficult problems with a paucity of means; it is 
quintessentially and surprisingly /not/ proportional response. Because 
of that, it is often deceptively simple, but always in a way that is 
impossible to anticipate yet obvious in afterthought. Scala's implicits 
and D's static if are great work.

Great Work often reformulates an entire challenge to reveal false 
choices or artificial constraints. Alexander the Great figured it 
doesn't matter how to get rid of the Gordian knot, so he cut it. While 
philosophers were still puzzling over The Ship of Theseus, Richard 
Feynman pointed out that the very notion of identity is ill-defined 
because a philosopher wouldn't even be able to tell which atoms belong 
to a chair, and which don't. Replacing complex type-based 
metaprogramming with trivial compile-time evaluation is great work.

Great Work is often recognizable to out-of-domain people, the person on 
the street. In contrast, Good Work has high-brow expertise as a 
prerequisite. Beethoven's Fourth Symphony is a respectable repertoire 
piece for any Philharmonic, one that music aficionados would listen to 
with respect. I once saw a janitor, a little old lady with no previous 
exposure to classical music, crying when she heard Bach's Air for the 
first time. Using the same programming language on the client and 
server, or at compile-time and run-time, or for computation and its 
constraints, is great work clear as rain. You don't need to be an expert 
to appreciate that.

Great Work is what we should all aspire to. Great Work is the cure for 
Good Work.

This, however, brings up a question: it seems that Great Work is not 
really easy to do on a regular basis. What to do, therefore, on a 
"regular" day when inspiration doesn't strike?

This brings us to Right Work.

Right Work is work that is undeniably, pound-on-the-table good, however 
unexciting or trivial.

Right Work will be silently appreciated by one's peers as the 
incontestably right thing to do. Paying your rent, debts, and other 
bills is Right Work.

Right Work is not necessarily simple. Becoming a better spouse or 
teaching your kids that lying is not a good policy - that's also Right Work.

In a software project, reducing the number of global variables is Right 
Work.

Making public data private is Right Work. (Careful, perversions are 
always possible. This is not Right work:

public int percent;

==>

private int _percent;
int percent() { return _percent; }
void percent(int p) { _percent = p; }

The obscure word "phronesis" - thanks Laeeth - should be a buzzword in 
software engineering circles.)

Reducing state and mutation is Right Work. That means adding the 
"immutable" qualifier to variables and the "pure" attribute to functions.

Making unsafe code safe is Right Work. That means inserting " safe" 
wherever possible, and identifying the smallest " trusted" primitives.

Replacing legacy pointer-and-length ad-hoc pairs with slices is Right 
Work. So is replacing a complicated unstructured loop with a structured 
foreach loop.

Most refactorings that meaningfully reduce the number of lines of code 
are Right Work. Of course, that could be perverted, too. I'm not talking 
more statements per line of code. I'm talking more /work/ per line of code.

Much Right work paradoxically requires less virtuosity than complicated 
artifacts of Good Work.

A successful software system is a construction of Great Work on a 
foundation of Right Work, with the inevitable Good Work here and there.

That's where we want to be.
Apr 01 2019
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 01, 2019 at 07:22:40PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 That comment came after a few phone conversations whereby Walter
 mentioned he has a deluge of pull requests to review. He said if he's
 to spend due time on all of those, he'd be unable to do any work of
 his own. Those pull requests that have these things in common (note
 that some may not apply to the pragma pull request, I'd just wrongly
 put it in the same bin):
 
 * they are large
 * the improvement they bring needs arguing
 * their quality could be improved so they need careful review and a
 couple of passes of changes
 
 This is what I call Good Work.


[...]

 Which brings us to Great Work.


[...]

 This brings us to Right Work.


[...]

 A successful software system is a construction of Great Work on a
 foundation of Right Work, with the inevitable Good Work here and
 there.
 
 That's where we want to be.


Had this been posted openly at the time, there might have been a lot
less frustration.  This is an area I feel desperately needs improvement.
This repeated lack of communication of this sort of thought process in
one form or another is what leads to repeated frustrations and
conflicts, because would-be contributors feel like information is being
withheld that would have enabled them to better understand what is
expected of them.  Even if these were just vague thoughts or a crude
outline of what you have in mind, and nowhere near something formal and
"presentable", providing partial information is better than no
information at all.


T

-- 
In theory, there is no difference between theory and practice.
Apr 01 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/1/2019 6:11 PM, H. S. Teoh wrote:

 Had this been posted openly at the time, there might have been a lot
 less frustration.  This is an area I feel desperately needs improvement.
 This repeated lack of communication of this sort of thought process in
 one form or another is what leads to repeated frustrations and
 conflicts, because would-be contributors feel like information is being
 withheld that would have enabled them to better understand what is
 expected of them.  Even if these were just vague thoughts or a crude
 outline of what you have in mind, and nowhere near something formal and
 "presentable", providing partial information is better than no
 information at all.


Actually, Andrei and I simply assumed this was obvious. Obviously, it wasn't.
Apr 02 2019
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 2 April 2019 at 08:26:34 UTC, Walter Bright wrote:

 [snip]

 Actually, Andrei and I simply assumed this was obvious. 
 Obviously, it wasn't.


Maybe he should adapt it into a D blog post?
Apr 02 2019
parent reply Rose <rose sarajevo.com> writes:
On Tuesday, 2 April 2019 at 11:15:42 UTC, jmh530 wrote:

 On Tuesday, 2 April 2019 at 08:26:34 UTC, Walter Bright wrote:

 [snip]

 Actually, Andrei and I simply assumed this was obvious. 
 Obviously, it wasn't.


 Maybe he should adapt it into a D blog post?


For now on reddit. 
https://www.reddit.com/r/programming/comments/b8igay/great_work_is_the_cure_for_good_work/
Apr 02 2019
parent bachmeier <no spam.net> writes:
On Tuesday, 2 April 2019 at 12:39:12 UTC, Rose wrote:

 On Tuesday, 2 April 2019 at 11:15:42 UTC, jmh530 wrote:

 On Tuesday, 2 April 2019 at 08:26:34 UTC, Walter Bright wrote:

 [snip]

 Actually, Andrei and I simply assumed this was obvious. 
 Obviously, it wasn't.


 Maybe he should adapt it into a D blog post?


 For now on reddit. 
 https://www.reddit.com/r/programming/comments/b8igay/great_work_is_the_cure_for_good_work/


It's not unusual for a programmer to have strong opinions about 
programming. While Andrei's actions will have an impact on our 
community (and I was critical of some of the things he's done 
earlier in the thread) I don't think there's anything 
particularly newsworthy about this thread. I've seen nothing to 
suggest he's unhinged or anything other than frustrated.
Apr 02 2019
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Apr 02, 2019 at 01:26:34AM -0700, Walter Bright via Digitalmars-d wrote:

 On 4/1/2019 6:11 PM, H. S. Teoh wrote:

 Had this been posted openly at the time, there might have been a lot
 less frustration.  This is an area I feel desperately needs
 improvement.  This repeated lack of communication of this sort of
 thought process in one form or another is what leads to repeated
 frustrations and conflicts, because would-be contributors feel like
 information is being withheld that would have enabled them to better
 understand what is expected of them.  Even if these were just vague
 thoughts or a crude outline of what you have in mind, and nowhere
 near something formal and "presentable", providing partial
 information is better than no information at all.


 
 Actually, Andrei and I simply assumed this was obvious. Obviously, it
 wasn't.


Therein lies the frequent cause of miscommunications: assuming that the
other party knows or understands something, which may not necessarily be
the case.  Especially when we're talking about random strangers on the
internet, who may have had a completely different experience and
background as yourself, and therefore in all likelihood understands and
sees things differently.  Stating all assumptions up-front helps to
bridge this gap.


T

-- 
Ph.D. = Permanent head Damage
Apr 02 2019
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 1 April 2019 at 23:22:40 UTC, Andrei Alexandrescu 
wrote:

 On 3/31/19 8:01 PM, Mike Franklin wrote:

 On Sunday, 31 March 2019 at 16:18:35 UTC, Rubn wrote:
 

 Look at this comment as someone pointed out before:

 https://github.com/dlang/dmd/pull/8557#pullrequestreview-149952733


 
 Yes, Andrei's review was a bit too strong but he did change 
 course later in that thread.  Also recognize that the original 
 author did not provide much motivation or justification 
 (though, I'm guilty of that too).


 Those were unnecessarily strong words.

 Allow me to give a bit more context whilst clarifying I am not 
 defending or condoning that attitude.

 That comment came after a few phone conversations whereby 
 Walter mentioned he has a deluge of pull requests to review. He 
 said if he's to spend due time on all of those, he'd be unable 
 to do any work of his own.


_I_ do most of the reviewing now, and this doesn't seem to have 
had any effect, see https://github.com/dlang/dmd/pull/9494 and 
related PRs.


 Those pull requests that have these things in common (note that 
 some may not apply to the pragma pull request, I'd just wrongly 
 put it in the same bin):

 * they are large
 * the improvement they bring needs arguing


When they are requisites for fixing bugs they don't, as was the 
case with most of the ones I'm pissed off about, and Jacob's 
recent ones are to make DMD as a library more useful...


 * their quality could be improved so they need careful review 
 and a couple of passes of changes


(Exactly which is not at all what has, and _still is_ being done.)
... In particular you are typically seeing a snapshot of a PR 
when you comment, and see only that change in isolation and miss 
the conversation in past PR discussion about where this is going. 
It is perfectly fine, in fact I encourage you, to inquire why 
this specific change is being made, we'll point you to the 
previous discussion.

What is _not_ fine is the drive-by style of review. This includes 
"Please don't let bureaucracy stand in the way of progress".


 A successful software system is a construction of Great Work on 
 a foundation of Right Work, with the inevitable Good Work here 
 and there.

 That's where we want to be.


The dlang ecosystem operates as an army of volunteers, you don't 
_get to choose_ the standard of work that comes in. It is was it 
is, and it will stay that way until such a time as it is not run 
by an army of volunteers. Yes, you can try to improve it (and so 
you should) but it needs to come off that that is your intention. 
Far too often, it comes off that you don't understand what or why 
the PR is doing what it is doing.
Apr 01 2019
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-04-02 01:22, Andrei Alexandrescu wrote:


 This brings us to Right Work.


I'm the first to say that I'm all for improving the code base. I really 
hate how difficult it can be to convince management that a code base 
need to be improved instead of working on a feature. But, to be blunt, 
if you get what you want and everyone is doing Right Work as you 
describe it, there won't be any progress. No bugs fixed or no new 
features. I don't know how often you expect this Great Work to occur, 
which might mean progress. Unfortunately the source code is not the 
product, it's the application, the tool, the library or whatever it 
might be.

-- 
/Jacob Carlborg
Apr 02 2019
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 31 March 2019 at 11:30:08 UTC, Andrei Alexandrescu 
wrote:

 The necessity to work on shared was also present in the January 
 2015 vision document (https://wiki.dlang.org/Vision/2015H1): 
 "Nail down fuzzily-defined areas of the language (e.g. shared 
 semantics,  property)."

 Writing it down doesn't get it done.


Indeed thats why its on the AGM.


 Defining shared properly would take a team with a programming 
 languages expert, a threading expert, and an application 
 expert. (Some roles may be realized within the same person.) I 
 know people within our community with the required expertise.


c.f. DIP1000? I don't think you or Walter are memory safety 
experts, and despite the issues that occurred with the 
implementation (and documentation(!)) process (hindsight is 
20/20), what you have achieved is quite remarkable: a blacklisted 
(bad things are disallowed), inductive (if all the the code I 
call is  safe and nothing I do is on the blacklist, then so am I) 
model of memory safety that works!

What Manu has (tried to, somewhat unsuccessfully) argue for is 
the same approach to thread safety with shared: tie the correct 
use of shared to  safe/ system/ trusted, enforce the correct use 
of user defined and verified (i.e.  trusted) thread safe 
primitives. Then shared (as a function parameter attribute) in 
 safe code becomes a requirement to act in a thread safe manner 
with that argument under the above framework.

No-one is expecting to get the implementation perfect first time, 
we will miss important details, and thats OK. We fix them, 
document them (this is the single biggest problem with the way 
DIP1000 was developed and the reason PRs were stalled) and move 
on with the satisfaction that the implementation is better than 
it was before.


 But I don't know any who'd also have the time to embark on this.


If the foundations can be set out, pre-approved, with sufficient 
clarity at dconf, then I'm confidant we can get to the point 
where we can iteratively improve it. Who knows, Timon might 
implement the whole thing in the hackathon.

I don't think it will be as hard as it seems to get somewhere 
useful. We already have  safe/ system/ trusted, shared is already 
a thing (albeit not very useful currently), all we need to do is 
define the blacklist and how the induction should work. Again, 
they don't need to be perfect first time.


 I got burned out on writing vision documents (Walter was never 
 a fan so I did them all) because it was difficult to figure 
 their impact. Contributors asked for a vision document. So we 
 started one. Then contributors continued doing what they were 
 doing.


I understood vision documents to set out the intended direction 
of the DLF, irrespective of what contributors are doing, of 
course they're going to continue to do what they were doing, that 
what volunteers do. To me they were more like a politicians 
promise (apologies for the analogy), a statement of intention, 
and if it doesn't match up to reality, then its no big deal as 
long as progress is still being made, the next one is written to 
take that into account with its projections.

Anyway enough about what they were. What they will become is a 
byproduct of the minutes of the AGM / quarterly meetings, they 
will practically write themselves.
If we have more (corporate) participation in them (as I hope) 
then chances are they will reflect reality to a much high degree 
of fidelity.
Mar 31 2019
prev sibling parent reply =?UTF-8?B?THXDrXM=?= Marques <luis luismarques.eu> writes:
On Sunday, 31 March 2019 at 11:30:08 UTC, Andrei Alexandrescu 
wrote:

 I got burned out on writing vision documents (Walter was never 
 a fan so I did them all) because it was difficult to figure 
 their impact. Contributors asked for a vision document. So we 
 started one. Then contributors continued doing what they were 
 doing.


I suggest you write informal status documents in the wiki 
instead, on a best-effort basis. Instead of writing a document 
that's pie in the sky "it would be great to have X in the 
following 6 months, let's pretend we know we can do it", write 
something more like "I've been working on X, and I've noticed a 
lot of little problems with Y. Here's what we could do about it, 
what I'm trying to do Z but why it might not be enough", etc. Do 
little updates when possible and motivated. Maybe update it 
sometimes with links to pull requests that you feel helped with 
the problems, so others can study them and learn what things are 
especially welcome, etc. Things like that.

In summary, play to our strengths. D has a small community. Keep 
it close and cozy, not distant and formal.
Mar 31 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 10:47 AM, Luís Marques wrote:

 I suggest you write informal status documents in the wiki instead, on a 
 best-effort basis.


A great idea. Walter has doing this on his twitter account:

https://twitter.com/WalterBright
Mar 31 2019
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, March 30, 2019 6:41:15 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:

 On 3/30/19 2:49 PM, Jonathan M Davis wrote:

 This would also be a great opportunity to fix some of the issues with
 shared in druntime


 The problem here is, of course, that nobody knows exactly what shared is
 supposed to do. Not even me. Not even Walter.

 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.


I confess that I find the amount of confusion over shared to be confusing,
though maybe I'm missing something. Yes, it has some details that need to be
worked out, but I would have thought that basics would be well understood by
now. For shared to work, it has to prevent code that isn't thread-safe -
which means either making any operation which isn't guaranteed to be
thread-safe illegal and/or making the compiler insert code that guarantees
thread-safety. The latter is very hard if not impossible, if nothing else,
because that would require that the compiler actually understand the
threading and synchronization mechanisms being used in a given situation
(e.g. associating a mutex with a shared variable and then somehow
guaranteeing that it's always locked at the appropriate time and unlocked at
the appropriate time). So, basically, that means that any operations that
aren't guaranted to be thread-safe then need to be illegal for shared, and
then to actually read or modify a shared object, you have to either use
atomics, or you have to use whatever synchronization mechanisms that you
want to be using (e.g. mutexes), cast away shared while the object is
protected, operate on the object as thread-local, make sure that no
thread-local references to it exist when you're done, and then release the
mutex. It's exactly what you'd be doing in C++ code except that you have to
worry about casting away shared while actually operating on the object,
because the type system is preventing you from shooting yourself in the foot
by reading or writing to shared objects, since that's not thread-safe
without synchronization mechanisms that aren't part of the type system.

It seems to be confusing for many people, because they expect to actually be
able to directly operate on shared objects, but you can't do that and
guarantee any kind of thread-safety unless you're calling stuff that takes
care of the protection for you (e.g. a type could contain a mutex and lock
it in its member functions, thereby encapsulating all of the casting away of
shared - which is basically what synchronized classes were supposed to do,
just with the outer level only, whereas you can do more than that if you're
doing it manually and write the code correctly). But shared types in general
aren't necessarily going to have any kind of synchronization mechanism built
in (e.g. shared(int*) certainly won't).

The primary problem that I see with shared as things stand is that it still
allows operations which aren't guaranteed to be thread-safe (e.g. copying).
It may need some tweaks beyond that, but as far as I can tell, it mostly
works as-is. However, it's much harder to use than it should be because the
stuff in core.sync has only been partially made to work properly with
shared.

I expect that when you get down to all of the nitty gritty details of making
operations that aren't guaranteed to be thread-safe illegal that there could
be some hairy things that need to be sorted out, but we've already made
_some_ of them illegal, and the addition of copy constructors to the
language actually fixes one of the big hurdles, which is making it possible
to make it thread-safe to copy an object (by having its copy constructor be
shared and handle the mutex or atomics or whatnot internally). So, by no
means am I claiming that we have it all figured out, but I would have
thought that it would primarily be an issue of figuring out how to correctly
make operations that aren't guaranteed to be thread-safe illegal (just like
finishing  safe by making operations  system when the compiler can't
guarantee that they're memory safe). Some of the details could turn out to
be nasty, but I would have thought that it would just be a matter of working
through them and that as things stand, the basic design of shared works.

Maybe I'm missing something here, but it seems to me that it _should_ be
pretty straightforward to just move forward from the idea that operations on
shared objects have to be thread-safe or be illegal. That may involve more
casting than would be ideal, but I don't see how we can really do much with
the language understanding enough to do the casting for you, and the result
is basically what you get in C++ - except that the compiler is preventing
you from shooting yourself in the foot outside of  system code where you're
handling the casting away of shared. So, it's pretty much the same situation
as  safe vs  system in the sense that shared code disallows operations that
the compiler can't guarantee the safety of, and you have to do  system stuff
to guarantee the safety yourself in the cases where you need to do that
stuff.

Regardless of the details of what we want to do with shared though,
core.sync doesn't handle it properly. Mostly, it doesn't use shared at all,
and when it does, it's hacked on. So, at some point here, we really need to
replace what's there with a v2 of some sort. And until we do, no matter how
well shared works on its own, it's going to be seriously hampered, because
the constructs in core.sync are core to writing threaded code.

- Jonathan M Davis
Mar 30 2019
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 3/30/19 2:49 PM, Jonathan M Davis wrote:

 This would also be a great opportunity to fix some of the issues with shared
 in druntime


 The problem here is, of course, that nobody knows exactly what shared is
 supposed to do. Not even me. Not even Walter.


As an immediate stop-gap, shared must have *no read or write access*
to data members. Simple. Make that change. Right now.
Then shared will at least be useful and not embarrassing, and you can
continue to argue about what shared means while I get to work.


 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.


Sure. But in the meantime, fix the objective bug with the current
semantics where you can read/write un-protected data freely. Right
now. Please for the love of god.
Mar 31 2019
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Sunday, 31 March 2019 at 22:25:30 UTC, Manu wrote:

 Sure. But in the meantime, fix the objective bug with the 
 current semantics where you can read/write un-protected data 
 freely. Right now. Please for the love of god.


Apropos "things to do right now", to derail the thread a little 
in my own interest, could I  again ask to decouple synchronized 
and shared for the time being? There is literally no way to write 
a class that has both threadsafety and invariant-safety right 
now, because D has no way to pull invariants into the class body 
without pulling in the `shared` mess. Look, if I'm using 
synchronized(this), as a rule I don't *care* about shared, 
because I'm inherently using another *kind* of thread safety to 
atomic primitives.

Uncoupling shared from synchronized at the class level would make 
synchronized classes actually usable and useful without 
restructuring the entire class for no gain.
Apr 01 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 1, 2019 1:08:29 AM MDT FeepingCreature via Digitalmars-d 
wrote:

 On Sunday, 31 March 2019 at 22:25:30 UTC, Manu wrote:

 Sure. But in the meantime, fix the objective bug with the
 current semantics where you can read/write un-protected data
 freely. Right now. Please for the love of god.


 Apropos "things to do right now", to derail the thread a little
 in my own interest, could I  again ask to decouple synchronized
 and shared for the time being? There is literally no way to write
 a class that has both threadsafety and invariant-safety right
 now, because D has no way to pull invariants into the class body
 without pulling in the `shared` mess. Look, if I'm using
 synchronized(this), as a rule I don't *care* about shared,
 because I'm inherently using another *kind* of thread safety to
 atomic primitives.

 Uncoupling shared from synchronized at the class level would make
 synchronized classes actually usable and useful without
 restructuring the entire class for no gain.


How is synchronized unrelated to shared? shared is part of anything and
everything in D which involves sharing data across threads. The only
exception is __gshared, which is only supposed to be used for C globals
(which as I understand it, can't be shared, because that wouldn't work with
the linker), and you have to be _really_ careful whenever __gshared is used,
because the compiler assumes that it's actually thread-local, because
__gshared is not actually part of the type.

synchronized is just one way to provide a mutex for protecting shared data.
The data itself still needs to be shared and then usually has to have shared
cast away while the data is protected by the mutex that goes with
synchronized (at which point, it's up to the programmer to ensure that no
thread-local references to the data escape or exist once the mutex is
unlocked). synchronized classes (which have never been fully implemented)
are supposed to deal with automatically casting away the outer level of
shared within the member functions, but it's still the same mechanism that
you have to do now manually. If you're not using shared with synchronized,
then you're doing something seriously wrong - probably using __gshared,
which is just begging for trouble. In general, any object using synchronized
methods should be shared.

- Jonathan M Davis
Apr 01 2019
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 1 April 2019 at 10:11:04 UTC, Jonathan M Davis wrote:

 On Monday, April 1, 2019 1:08:29 AM MDT FeepingCreature via 
 Digitalmars-d wrote:

 Uncoupling shared from synchronized at the class level would 
 make synchronized classes actually usable and useful without 
 restructuring the entire class for no gain.


 How is synchronized unrelated to shared?


In the one way that matters - in practice. :)

I'll be honest; we use synchronized a whole lot and shared in 
maybe ten lines total. How? Well, in the obvious way: void 
method() { synchronized (this) { } }. No shared, no fuss.

I would be surprised if this was not the predominant way of 
synchronizing classes in D in production right now, precisely 
because you *don't* have to bother with shared.

And if you do it that way, invariants are inherently useless.


 The only exception is __gshared, which is only supposed to be 
 used for C globals


I'm sorry, but there's a very great distance here, at least in 
our codebase, between "only supposed to" (C globals) and "what 
it's actually used for" (*any* threadshared globals). Nobody 
wants to touch shared, but we still need threading.
Apr 01 2019
parent FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 1 April 2019 at 12:28:37 UTC, FeepingCreature wrote:

 I'm sorry, but there's a very great distance here, at least in 
 our codebase, between "only supposed to" (C globals) and "what 
 it's actually used for" (*any* threadshared globals). Nobody 
 wants to touch shared, but we still need threading.


Sorry for the double post, but I realized I'd forgotten to 
explain *why* we do it that way.

You will note that the way I explained threading is the way that 
basically any C-based language that doesn't have threading 
correctness as an explicit selling point does it: as a thin 
wrapper around pthreads. It's the "natural", "normal" way to do 
threading, the way that programmers have been doing it for 
literal decades. Threads and mutexes.

If D wants people to do it in a different way, that solution 
needs to be slick, operational and *reliable*. Any solution that 
starts with "well you can just cast it away cause it doesn't 
really work yet" doesn't have the reaction of "well we're just 
gonna do it that way then, because there's no other way, and hope 
the D devs clean up shared one of these *years*" - the reaction 
is going to be the entirely predictable "well, we're just gonna 
go back to the C way then." That's where D is right now, and 
that's why I look at all the discussion about shared with a kind 
of amused bewilderment. Thread safety is simply fundamentally not 
about exposing atomic access to data in our minds. Thread safety 
is about having classes, that own data, access to which they can 
lock to one thread. Synchronized(this). That's the established, 
commonsense, baseline idiom. It shouldn't be surprising that we 
prefer it to randomly casting away modifiers that the language 
randomly forces on us despite them being highly experimental and 
subject to change.

Get the normal, established way to do threading working 
consistently and reliably *now*. Then think about shared.

That's my opinion, at least.
Apr 01 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/19 6:25 PM, Manu wrote:

 On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 3/30/19 2:49 PM, Jonathan M Davis wrote:

 This would also be a great opportunity to fix some of the issues with shared
 in druntime


 The problem here is, of course, that nobody knows exactly what shared is
 supposed to do. Not even me. Not even Walter.


 
 As an immediate stop-gap, shared must have *no read or write access*
 to data members. Simple. Make that change. Right now.
 Then shared will at least be useful and not embarrassing, and you can
 continue to argue about what shared means while I get to work.
 

 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.


 
 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god.


Incidentally Walter and I discussed in the early days the idea that 
"shared" would offer no access whatsoever aside from a few special 
functions in druntime such as "atomicRead" and "atomicWrite". Of course, 
that raised the question how those functions could be implemented - the 
possible solution being some casts for typing and asm for the needed 
memory barriers.

In the end we got scared that there was no precedent in other languages, 
and we could not predict whether that would have been good or bad. The 
result is the current semantics, which should be a felony in the 48 
contiguous US states.

This does not work as a two stages process, though the "stop the 
bleeding first then come with the new solution" metaphor seems 
attractive. The main issues being when we break code that people got to 
work, we need to offer the alternative as well. Another being that the 
exact kind of things we disable/enable may be dependent on the ultimate 
solution.

This would be a large effort requiring a strong team. Walter, yourself, 
and I would be helpful participants but I think between the three of us 
we don't have the theoretical chops to pull this off. At least I know I 
don't. We need the likes of Timon Gehr, Johan Engelen, and David 
Nadlinger (whom I cc'd just in case).
Apr 03 2019
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 4 April 2019 at 02:05:15 UTC, Andrei Alexandrescu 
wrote:

 On 3/31/19 6:25 PM, Manu wrote:

 Sure. But in the meantime, fix the objective bug with the 
 current
 semantics where you can read/write un-protected data freely. 
 Right
 now. Please for the love of god


 This does not work as a two stages process, though the "stop 
 the bleeding first then come with the new solution" metaphor 
 seems attractive. The main issues being when we break code that 
 people got to work, we need to offer the alternative as well. 
 Another being that the exact kind of things we disable/enable 
 may be dependent on the ultimate solution.


Well whatever happens I'll be gobsmacked if its not behind an opt 
in switch.
With that in mind, if Manu gets use out of the stopgap of 
disabling read/write access, then I think we should implement 
that ASAP and then listen to whatever he complains about next ;)


 This would be a large effort requiring a strong team. Walter, 
 yourself, and I would be helpful participants but I think 
 between the three of us we don't have the theoretical chops to 
 pull this off. At least I know I don't. We need the likes of 
 Timon Gehr, Johan Engelen, and David Nadlinger (whom I cc'd 
 just in case).


I don't think we are going to be able to do this without 
iterating on the design and closing holes and nuisances that we 
discover. I'm not saying that it is a bad idea to design up front 
as much as we can, but we shouldn't wast time getting hung up on 
design when implementation can give gains to users and guidance 
to the design.
Apr 03 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/3/19 11:09 PM, Nicholas Wilson wrote:

 On Thursday, 4 April 2019 at 02:05:15 UTC, Andrei Alexandrescu wrote:

 On 3/31/19 6:25 PM, Manu wrote:

 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god


 This does not work as a two stages process, though the "stop the 
 bleeding first then come with the new solution" metaphor seems 
 attractive. The main issues being when we break code that people got 
 to work, we need to offer the alternative as well. Another being that 
 the exact kind of things we disable/enable may be dependent on the 
 ultimate solution.


 
 Well whatever happens I'll be gobsmacked if its not behind an opt in 
 switch.
 With that in mind, if Manu gets use out of the stopgap of disabling 
 read/write access, then I think we should implement that ASAP and then 
 listen to whatever he complains about next ;)
 

 This would be a large effort requiring a strong team. Walter, 
 yourself, and I would be helpful participants but I think between the 
 three of us we don't have the theoretical chops to pull this off. At 
 least I know I don't. We need the likes of Timon Gehr, Johan Engelen, 
 and David Nadlinger (whom I cc'd just in case).


 
 I don't think we are going to be able to do this without iterating on 
 the design and closing holes and nuisances that we discover. I'm not 
 saying that it is a bad idea to design up front as much as we can, but 
 we shouldn't wast time getting hung up on design when implementation can 
 give gains to users and guidance to the design.


I don't think this works for programming language design. In fact I'm 
positive it doesn't. It's the way we've done things so far.
Apr 03 2019
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/4/19 12:24 AM, Andrei Alexandrescu wrote:

 On 4/3/19 11:09 PM, Nicholas Wilson wrote:

 On Thursday, 4 April 2019 at 02:05:15 UTC, Andrei Alexandrescu wrote:

 On 3/31/19 6:25 PM, Manu wrote:

 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god


 This does not work as a two stages process, though the "stop the 
 bleeding first then come with the new solution" metaphor seems 
 attractive. The main issues being when we break code that people got 
 to work, we need to offer the alternative as well. Another being that 
 the exact kind of things we disable/enable may be dependent on the 
 ultimate solution.


 Well whatever happens I'll be gobsmacked if its not behind an opt in 
 switch.
 With that in mind, if Manu gets use out of the stopgap of disabling 
 read/write access, then I think we should implement that ASAP and then 
 listen to whatever he complains about next ;)


 This would be a large effort requiring a strong team. Walter, 
 yourself, and I would be helpful participants but I think between the 
 three of us we don't have the theoretical chops to pull this off. At 
 least I know I don't. We need the likes of Timon Gehr, Johan Engelen, 
 and David Nadlinger (whom I cc'd just in case).


 I don't think we are going to be able to do this without iterating on 
 the design and closing holes and nuisances that we discover. I'm not 
 saying that it is a bad idea to design up front as much as we can, but 
 we shouldn't wast time getting hung up on design when implementation 
 can give gains to users and guidance to the design.


 
 I don't think this works for programming language design. In fact I'm 
 positive it doesn't. It's the way we've done things so far.


Well I'm exaggerating. I mean to say every time we did it that way, the 
result hasn't been good.
Apr 03 2019
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 4 April 2019 at 04:25:59 UTC, Andrei Alexandrescu 
wrote:

 On 4/4/19 12:24 AM, Andrei Alexandrescu wrote:

 On 4/3/19 11:09 PM, Nicholas Wilson wrote:

 I don't think we are going to be able to do this without 
 iterating on the design and closing holes and nuisances that 
 we discover. I'm not saying that it is a bad idea to design 
 up front as much as we can, but we shouldn't wast time 
 getting hung up on design when implementation can give gains 
 to users and guidance to the design.


 
 I don't think this works for programming language design. In 
 fact I'm positive it doesn't. It's the way we've done things 
 so far.


 Well I'm exaggerating. I mean to say every time we did it that 
 way,


Examples please?


 the result hasn't been good.


e.g. DIP1000 was bad, not because it was iterated upon to fix the 
holes in it, but because the changes were not communicated 
properly and not documented. I suggest we don't make those same 
mistakes again.
Apr 03 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/4/19 1:50 AM, Nicholas Wilson wrote:

 On Thursday, 4 April 2019 at 04:25:59 UTC, Andrei Alexandrescu wrote:

 On 4/4/19 12:24 AM, Andrei Alexandrescu wrote:

 On 4/3/19 11:09 PM, Nicholas Wilson wrote:

 I don't think we are going to be able to do this without iterating 
 on the design and closing holes and nuisances that we discover. I'm 
 not saying that it is a bad idea to design up front as much as we 
 can, but we shouldn't wast time getting hung up on design when 
 implementation can give gains to users and guidance to the design.


 I don't think this works for programming language design. In fact I'm 
 positive it doesn't. It's the way we've done things so far.


 Well I'm exaggerating. I mean to say every time we did it that way,


 
 Examples please?


Shared itself, the postblit, lazy, properties, alias this - are all 
"first-order thinking" ideas that are not bad, but fail to take into 
consideration second-order interactions and their consequences.

(A good read: https://fs.blog/2016/04/second-order-thinking/)

Language design is all about second-order thinking.


 the result hasn't been good.


 
 e.g. DIP1000 was bad, not because it was iterated upon to fix the holes 
 in it, but because the changes were not communicated properly and not 
 documented. I suggest we don't make those same mistakes again.


DIP1000 is actually an example of second-order thinking. Walter pored 
over it for months before writing and implementing it.

So are the recently-introduced copy constructors. We had what we thought 
was a workable design at probably one dozen times during the process. 
All had large flaws.

Incrementalism is an anti-pattern in language design.
Apr 04 2019
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 4 April 2019 at 11:10:09 UTC, Andrei Alexandrescu 
wrote:

 On 4/4/19 1:50 AM, Nicholas Wilson wrote:

 e.g. DIP1000 was bad, not because it was iterated upon to fix 
 the holes in it, but because the changes were not communicated 
 properly and not documented. I suggest we don't make those 
 same mistakes again.


 DIP1000 is actually an example of second-order thinking. Walter 
 pored over it for months before writing and implementing it.


That is was a good idea does not excuse how sloppily the 
procedure of implementation was handled. I note also that it 
underwent significant changes post implementation. That is the 
iteration I'm taking about (just handled better, i.e. docs & 
community engagement).


 Incrementalism is an anti-pattern in language design.


I'm talking about design iteration, not language by incremental 
feature addition.
Incremental feature addition is difficult, if not impossible, to 
undo if it turns out it was a bad idea. Iterative design, by 
definition, does not suffer from that problem.
Apr 04 2019
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/4/2019 4:10 AM, Andrei Alexandrescu wrote:

 DIP1000 is actually an example of second-order thinking. Walter pored over it 
 for months before writing and implementing it.


So far, the only real shortcoming in the initial design was revealed by the 
put() semantics, and was fixed with that PR that transmitted scope-ness through 
the first argument.

On the other hand, while it would be great if we could anticipate all second 
order effects, it's unrealistic to always get that right, and no language does.
Apr 04 2019
prev sibling parent Manu <turkeyman gmail.com> writes:
On Wed, Apr 3, 2019 at 9:25 PM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 4/3/19 11:09 PM, Nicholas Wilson wrote:

 On Thursday, 4 April 2019 at 02:05:15 UTC, Andrei Alexandrescu wrote:

 On 3/31/19 6:25 PM, Manu wrote:

 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god


 This does not work as a two stages process, though the "stop the
 bleeding first then come with the new solution" metaphor seems
 attractive. The main issues being when we break code that people got
 to work, we need to offer the alternative as well. Another being that
 the exact kind of things we disable/enable may be dependent on the
 ultimate solution.


 Well whatever happens I'll be gobsmacked if its not behind an opt in
 switch.
 With that in mind, if Manu gets use out of the stopgap of disabling
 read/write access, then I think we should implement that ASAP and then
 listen to whatever he complains about next ;)


 This would be a large effort requiring a strong team. Walter,
 yourself, and I would be helpful participants but I think between the
 three of us we don't have the theoretical chops to pull this off. At
 least I know I don't. We need the likes of Timon Gehr, Johan Engelen,
 and David Nadlinger (whom I cc'd just in case).


 I don't think we are going to be able to do this without iterating on
 the design and closing holes and nuisances that we discover. I'm not
 saying that it is a bad idea to design up front as much as we can, but
 we shouldn't wast time getting hung up on design when implementation can
 give gains to users and guidance to the design.


 I don't think this works for programming language design. In fact I'm
 positive it doesn't. It's the way we've done things so far.


You say your original design worked how I suggest (I'm not surprised,
it's the only thing that makes sense), so... close the circuit!
Maybe it was a success, and nobody ever had the chance to demonstrate
it. We've waited so long to try it, so let us try it out!
Apr 03 2019
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, April 3, 2019 8:05:15 PM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:

 On 3/31/19 6:25 PM, Manu wrote:

 On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:

 On 3/30/19 2:49 PM, Jonathan M Davis wrote:

 This would also be a great opportunity to fix some of the issues with
 shared in druntime


 The problem here is, of course, that nobody knows exactly what shared
 is
 supposed to do. Not even me. Not even Walter.


 As an immediate stop-gap, shared must have *no read or write access*
 to data members. Simple. Make that change. Right now.
 Then shared will at least be useful and not embarrassing, and you can
 continue to argue about what shared means while I get to work.


 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what
 to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.


 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god.


 Incidentally Walter and I discussed in the early days the idea that
 "shared" would offer no access whatsoever aside from a few special
 functions in druntime such as "atomicRead" and "atomicWrite". Of course,
 that raised the question how those functions could be implemented - the
 possible solution being some casts for typing and asm for the needed
 memory barriers.

 In the end we got scared that there was no precedent in other languages,
 and we could not predict whether that would have been good or bad. The
 result is the current semantics, which should be a felony in the 48
 contiguous US states.

 This does not work as a two stages process, though the "stop the
 bleeding first then come with the new solution" metaphor seems
 attractive. The main issues being when we break code that people got to
 work, we need to offer the alternative as well. Another being that the
 exact kind of things we disable/enable may be dependent on the ultimate
 solution.

 This would be a large effort requiring a strong team. Walter, yourself,
 and I would be helpful participants but I think between the three of us
 we don't have the theoretical chops to pull this off. At least I know I
 don't. We need the likes of Timon Gehr, Johan Engelen, and David
 Nadlinger (whom I cc'd just in case).


shared already tries to prevent various read/write operations that aren't
guaranteed to be thread-safe. So, it's already gone partially in that
direction, and if it's supposed to guarantee any kind of thread-safety,
that's the only way that it can go unless it's going to try to insert code
that ensures thread-safety - which would mean doing stuff like having
mutexes built in to what it's doing so that it knows what to lock and when,
and that's not something that's going to work as a low level construct. Even
the concept of synchronized classes as proposed in TDPL couldn't go far
enough with it to be very useful, because it could only safely cast away the
outer layer of shared. I thought that it was pretty clear that the way we
were going - and have to go - is to make it so that you simply can't read or
write to shared data without either using operations that are guaranteed to
be thread-safe or by casting away shared (at which point, the code is
 system, and it's up to the programmer to get it right, and that code is
properly segregated thanks to how  safe and  trusted work).

I grant you that we really do need the best people we have to look at this
at the low level and make sure that it works properly, but I don't see where
else we can go with this if we want any kind of guarantees from shared. If
we decide that we don't want shared to provide guarantees and that we want
to allow shared data to be read and written to regardless of whether the
compiler can guarantee thread-safety, then we can do that, but then things
that are currently illegal should be made legal (e.g. IIRC, incrementing
shared integers is illegal), and at that point, shared is just indicating
which variables are shared across threads without actually protecting you at
all. That would put it more in line with languages like C++, but it also
would mean that it would be a lot less useful for catching bugs.

Either way, what we have right now is halfway in between. It makes some
stuff that isn't thread-safe illegal but doesn't make it all illegal, making
it a pain to use if you don't cast away shared like you would need to if all
operations which weren't guaranteed to be thread-safe were illegal but
simultaneously failing to actually protect you and guarantee that the code
with operations that are potentially not thread-safe is  system. It looks to
me like it has to go one way or the other. Either make it all illegal, thus
requiring code that deals directly with synchronization mechanisms to use
casts and be  trusted - or make all of the read/write operations on shared
objects legal and just restrict it in that it can't be converted to or from
thread-local without a cast.

- Jonathan M Davis
Apr 03 2019
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Wed, Apr 3, 2019 at 7:10 PM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 3/31/19 6:25 PM, Manu wrote:

 On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On 3/30/19 2:49 PM, Jonathan M Davis wrote:

 This would also be a great opportunity to fix some of the issues with shared
 in druntime


 The problem here is, of course, that nobody knows exactly what shared is
 supposed to do. Not even me. Not even Walter.


 As an immediate stop-gap, shared must have *no read or write access*
 to data members. Simple. Make that change. Right now.
 Then shared will at least be useful and not embarrassing, and you can
 continue to argue about what shared means while I get to work.


 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.


 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god.


 Incidentally Walter and I discussed in the early days the idea that
 "shared" would offer no access whatsoever aside from a few special
 functions in druntime such as "atomicRead" and "atomicWrite". Of course,
 that raised the question how those functions could be implemented - the
 possible solution being some casts for typing and asm for the needed
 memory barriers.


Casting away shared seems to be the only reasonable option with
respect to this general design for shared.

This applies just fine to atomics; casting away shared is effectively
asserting that you have created a thread-local context for some window
of time that you can perform a thread-safe interaction.
Atomic operations have an effective zero-length execution window, so
from that perspective, it's correct that you are able to cast-away
shared in order to perform a single atomic operation on an int. The
assertion that you have a thread-local lease on the int for the
duration of an atomic operation naturally holds by definition.


 In the end we got scared that there was no precedent in other languages,
 and we could not predict whether that would have been good or bad. The
 result is the current semantics, which should be a felony in the 48
 contiguous US states.


Okay... so you're effectively saying you had a possibly-good idea, but
instead of trying it, instead did something else that's just
straight-up broken?
So, like, how about we actually try the thing before we decide it
didn't work? We have nothing to lose!


 This does not work as a two stages process, though the "stop the
 bleeding first then come with the new solution" metaphor seems
 attractive.


You're saying a design was proposed, and it's *almost* implemented in
the language. It's been there for longer than I've been around, but
we're still yet to actually try out the design as is almost
implemented so long ago. It's really weird that we're declaring the
design failed before trying it out! We've had, like... 12 years to try
it out? Why haven't we tried it yet?

We're not bleeding from a failed design. If there is a wound in place,
it's the fact that the intended design was only half-implemented, and
then left that way for over a decade.

Making this change would give something to work with. If it turns out
it's not a workable solution after all, then it would be good to know
before we eject it into space!
Better design may emerge from understanding how this design failed.
But it hasn't failed yet, because we haven't actually tried it yet...
it's just been sitting in limbo waiting while people scratch their
head and try to understand what happened here.

That process is surely more useful than the current situation, which
is that `shared` means nothing, and nobody quite knows what it's for
other than a sort of self-documentation.

I understand you may prefer a strong design proposal, but nobody has
moved the bar in the decade I've been waiting.
We're clearly not making ground in the way you prefer, so let's just
actually implement the design almost-implemented 12(+?) years ago, and
see how terrible it actually is?

I have a fairly strong sense of what will emerge, I suspect it'll be
workable and useful.


 The main issues being when we break code that people got to
 work, we need to offer the alternative as well. Another being that the
 exact kind of things we disable/enable may be dependent on the ultimate
 solution.


I'm fairly sure we're not 'breaking' anything. Any code that breaks
with this change is almost certainly already broken.
The remedy would be to add a cast to their code, and it will be
exactly as it is now; probably still broken. But it's really not very
disruptive.


 This would be a large effort requiring a strong team. Walter, yourself,
 and I would be helpful participants but I think between the three of us
 we don't have the theoretical chops to pull this off. At least I know I
 don't. We need the likes of Timon Gehr, Johan Engelen, and David
 Nadlinger (whom I cc'd just in case).


Go for it... but like, maybe first, how about we actually try out the
design you came up with 12 years ago before we declare it a failure
and spend another few years trying to do something else?
We have no evidence it's a failure, only that the half-implementation
of the original design is a failure, and mostly for the reason that
the half-semantics just don't mean anything, not that the original
idea is broken.
Apr 03 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On the plus side of shared as it is at the moment, and it's a pretty big plus, 
is it identifies what data is shared, and how it's integrated into the type 
system is good.

With C and C++, when you've got a shared memory bug, you have no idea where to 
start. In D, you look at the shared stuff. You're not going to have 
inadvertently shared variables.
Apr 04 2019
next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
On Thursday, 4 April 2019 at 21:05:51 UTC, Walter Bright wrote:

 On the plus side of shared as it is at the moment, and it's a 
 pretty big plus, is it identifies what data is shared, and how 
 it's integrated into the type system is good.

 With C and C++, when you've got a shared memory bug, you have 
 no idea where to start. In D, you look at the shared stuff. 
 You're not going to have inadvertently shared variables.


I mean, given that synchronized classes don't strip out the first 
layer of shared, as is necessary to make them, basically, *at 
all* usable, we're currently forced to use synchronized(this) ... 
which does not force shared at all. So this seems optimistic.

Thinking more about it, I feel shared is a fundamentally wrong 
direction. Even stripping out the first layer, I'll not be able 
to naturally read or modify anything with references, say an int 
array, because the language can't know that nothing else has a 
view on the array. I'd be limited to arrays of immutable data, 
with all the "can never overwrite a field, even though I own the 
array" problems that brings.

Istm that the more natural way to use shared in the context of 
classes would be to talk not in terms of whether data is shared 
or not, but whether data is *owned* by the class. If I 
transitively owned the array in a synchronized class, I could 
overwrite what I want at will, confident that any access would be 
happening behind a synchronized block.

ie.

struct S
{
   int[] data; // reference
}

synchronized class S
{
   private owned S[] array;

   // valid despite shared - array is not shared, because we own 
it.
   void test1(size_t index) { array[index] = owned(S)(null); }

   // not valid - `data` references caller data; we can't take 
ownership of it.
   void test2(int[] data) { array[index] = owned(S)(data); }

   // valid - we have exclusive ownership on data.dup
   void test3(int[] data) { array[index] = owned(S)(data.dup); }

   // invalid - can't escape owned data.
   owned(S) test4() { return array.front; }

   // valid
   S test5() { return S(array.front.data.dup); }
}

As far as I can see, in production use synchronization is about 
atomic access only in the very rarest of cases. I can't imagine 
why a domain class would ever return anything shared (who would 
trust the outside world to know how to correctly access it?), and 
inside class methods, presuming we own our data, as classes 
*should anyways*, shared has no benefit because there's only one 
reader/writer anyways. In comparison, imo, owned would formalize 
the way that threadsafe classes should be written *anyways*.
Apr 05 2019
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/4/19 5:05 PM, Walter Bright wrote:

 On the plus side of shared as it is at the moment, and it's a pretty big 
 plus, is it identifies what data is shared, and how it's integrated into 
 the type system is good.
 
 With C and C++, when you've got a shared memory bug, you have no idea 
 where to start. In D, you look at the shared stuff. You're not going to 
 have inadvertently shared variables.


Such an outlook means stagnation and smothers progress. Shared is 
nowhere near where it should be. Internalizing that is key to getting it 
right.

I've said this many many times privately and publicly: If we never make 
the argument "yeah but C++ is worse" again, that would be too soon.
Apr 05 2019
prev sibling parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Thursday, 4 April 2019 at 21:05:51 UTC, Walter Bright wrote:

 On the plus side of shared as it is at the moment, and it's a 
 pretty big plus, is it identifies what data is shared, and how 
 it's integrated into the type system is good.

 With C and C++, when you've got a shared memory bug, you have 
 no idea where to start. In D, you look at the shared stuff. 
 You're not going to have inadvertently shared variables.


Sorry, Walter. But it is indeed easy to track down shared memory 
bugs than introducing design level bugs with shared.
Apr 05 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, March 31, 2019 4:25:30 PM MDT Manu via Digitalmars-d wrote:

 On Sat, Mar 30, 2019 at 5:45 PM Andrei Alexandrescu via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:

 On 3/30/19 2:49 PM, Jonathan M Davis wrote:

 This would also be a great opportunity to fix some of the issues with
 shared in druntime


 The problem here is, of course, that nobody knows exactly what shared is
 supposed to do. Not even me. Not even Walter.


 As an immediate stop-gap, shared must have *no read or write access*
 to data members. Simple. Make that change. Right now.
 Then shared will at least be useful and not embarrassing, and you can
 continue to argue about what shared means while I get to work.


 One monumental piece of work would be DIP 1021 "Define the semantics of
 shared". Then people who build with -dip1021 know what to do and what to
 expect. Now that would patch one of the bigger holes mentioned in the
 other post.


 Sure. But in the meantime, fix the objective bug with the current
 semantics where you can read/write un-protected data freely. Right
 now. Please for the love of god.


Honestly, that's really what shared is missing. _Some_ operations which
aren't guaranteed to be thread-safe are illegal, but many aren't. With that
fixed, I don't know what else shared would actually need. The stuff in
core.sync really needs to be fixed up to use shared correctly (which is
critical for being able to use shared cleanly), but at that point, shared
itself would be doing what it needs to be doing. We could then potentially
add stuff like synchronized classes on top of that, but that's just making
shared easier to use, not actually required for shared to work.

- Jonathan M Davis
Apr 01 2019
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 11:04:58 AM MDT Andrei Alexandrescu via 
Digitalmars-d wrote:

 On 3/28/19 9:16 AM, ag0aep6g wrote:

 On 28.03.19 14:05, Andrei Alexandrescu wrote:

 Then some ranges are not meant to be assignable.


 Should Phobos be compatible with those ranges?


 A variety of algorithm and data structures in Phobos are relying on
 assignment. Presumably a good part of them can be converted with ease to
 use single assignment, but not all.

 The overall message is we got bogged down on the "wrong" side of
 generality - cross-cutting and nonscalable code additions to support
 unprincipled and low-impact corner cases.

 Part of that is we've been cagey about defining copy and assignment
 semantics of ranges in a simple and comprehensive manner. It seems to me
 going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics
 (all copies refer to the same underlying position, and advancing one
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies refer
 to distinct positions in the range such that advancing one does not
 advance the others.

 * We don't support other semantics.


I've thought for a while now that this is what we should do if we could
start over. The problem has been that it breaks a ton of code, and a
migration path is difficult. Doing a v2 version of Phobos would really help
with making fixes like that - though then we have to deal with stuff like
whether a library is written to work with v1 or v2, and with stuff like
ranges effectively working via duck typing, that could get a bit
interesting. If we can do it without splitting the community though, it
would allow us to fix some of the big mistakes we made with our core stuff
(auto-decoding being another).

- Jonathan M Davis
Mar 30 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 28, 2019 11:42:50 AM MDT H. S. Teoh via Digitalmars-d 
wrote:

 On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via
 Digitalmars-d wrote: [...]


 Part of that is we've been cagey about defining copy and assignment
 semantics of ranges in a simple and comprehensive manner. It seems to
 me going with these is the right thing:

 * Input ranges are copyable and assignable, and have pointer semantics
 (all copies refer to the same underlying position, and advancing one
 advances all others).

 * Forward ranges are copyable and assignable, but distinct copies
 refer to distinct positions in the range such that advancing one does
 not advance the others.

 * We don't support other semantics.


 What about classes that wrap ranges? E.g., std.ranges.interfaces.  Since
 classes are reference types, this would mean it's impossible to use
 those interfaces with forward ranges or above, which is a show-stopping
 limitation (e.g., if you need to return two different range types
 selected at runtime -- the current solution is to wrap them in the
 appropriate class using std.ranges.interfaces).


It's not a problem for basic input ranges, because they'd then have
reference semantics. The problem would just be forward ranges, and the
solution to that is to use wrapper ranges which are structs. Then they can
define a copy constructor which gives value semantics in place of save.

The reality of the matter though is that code should be _really_ wary of
using classes for forward ranges, because it requires calling new a lot, and
it's really inefficient. So, while we probably should support it for those
use cases that really need it, if code doesn't really need it, it shouldn't
do it. For instance, I tested dxml with a whole range of range types
(including ranges that were classes) when benchmarking, and performance
tanked when using ranges that were classes because of all of the memory
allocation that was happening.

- Jonathan M Davis
Mar 30 2019
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/28/2019 5:37 AM, Andrei Alexandrescu wrote:

 The conclusion is not to operate such changes everywhere (i.e. reason by 
 analogy). The right conclusion is that save() is unnecessarily general and 
 underspecified.


I've opined that `save()` should be implemented using copy construction, not 
assignment.
Mar 28 2019
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 19.03.19 03:52, Andrei Alexandrescu wrote:

 
 Turns out the second clause fails. That takes us to the definition of 
 empty in the same module:
 
  property bool empty(T)(auto ref scope const(T) a)
 if (is(typeof(a.length) : size_t))
 {
      return !a.length;
 }
 ...


Why should length be required to be `const` when almost everything 
that's `const` is not even a range?


 The intent is fairly clear - if a range defines empty as a size_t 
 (somewhat oddly relaxed to "convertible to size_t"), then empty can be 
 nicely defined in terms of length. Cool. But empty doesn't work with 
 TestAliasedString due to an overlooked matter: the "const". A mutable 
 TestAliasedString converts to a string, but a const or immutable 
 TestAliasedString does NOT convert to a const string! So this fixes that 
 matter:
 
      struct TestAliasedString
      {
          string get()  safe  nogc pure nothrow { return _s; }
          const(string) get()  safe  nogc pure nothrow const { return
_s; }
          alias get this;
           disable this(this);
          string _s;
      }
 


Reasoning from first principles, the right fix is actually to allow 
non-const length.


 That makes empty() work, but also raises a nagging question: what was 
 the relationship of TestAliasedString to string before this change? 
 Surely that wasn't subtyping. (My response would be: "Odd.") And why was 
 Phobos under the obligation to cater for such a type and its tenuous 
 relationship to a range?


The problem isn't `alias this`. It's `const`. The following type does 
not pass isInputRange either:

struct ApparentlyNotARange{
     size_t k=10;
     int front(){ return 0; }
     void popFront(){ --k; }
      property size_t length(){ return k; }
}

While this does:

struct ApparentlyNotARange{
     size_t k=10;
     int front(){ return 0; }
     void popFront(){ --k; }
      property size_t length()const{ return k; }
}

The reason why Phobos ought to be under the obligation to cater for such 
a type is because failure to do so is an artificial limitation. It works 
by default, but someone went out of their way to make sure it does not 
work if `length` is not annotated `const` (or ` property`). This is 
particularly strange because there is no such thing as a const range.
Apr 01 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/19 5:23 PM, Timon Gehr wrote:

 Reasoning from first principles, the right fix is actually to allow 
 non-const length.


D'accordo. Can you please put forth a PR to that effect?
Apr 01 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/19 8:04 PM, Andrei Alexandrescu wrote:

 On 4/1/19 5:23 PM, Timon Gehr wrote:

 Reasoning from first principles, the right fix is actually to allow 
 non-const length.


 
 D'accordo. Can you please put forth a PR to that effect?


My attempts to delegate go a lot like my wife's attempts to get me to 
help around the house :o).

https://github.com/dlang/phobos/pull/6944
Apr 05 2019