www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Struggling to implement parallel foreach...

reply Manu <turkeyman gmail.com> writes:
So, I want to get to here:

foreach(x; data.parallel(bucketSize))
{
  // run on worker threads...
}

Phobos alleges to have code for this, but it's not threadsafe, and
missing all the pieces that I'm having trouble with.

The TL;DR boils down to this; opApply receives a delegate that is NOT shared.

Problem:

struct S
{
  int opApply(scope int delegate(Iter) shared loopBody) { ... }
}

S s;
foreach(i; s) { ... }

Error: function `S.opApply(scope int delegate(int) shared loopBody)`
is not callable using argument types `(int delegate(int i) pure
nothrow  nogc  safe)`
cannot pass argument `__foreachbody2` of type `int delegate(int i)
pure nothrow  nogc  safe` to parameter `scope int delegate(int) shared
loopBody`

Obviously the loop body is not a shared function... but that's what is required.

So, it seems the arguments to opApply are considered when infering the
types for the foreach loop counters, so features of the delegate are
being inferred already, but I need the attributes of the delegate to
be inferred too, specifically, the `scope` attribute on the delegate
needs to be applied to the loop function.

This has cascading issues; the closure contains references to outer
objects, but if the delegate is `shared`, then all those references
transitively inherit the shared attribute too. It should work exactly
like I if it was writing a `shared` method to some struct, and all the
members are shared... so the body of the foreach needs to typecheck
assuming the transitive application of the inferred attributes from
the opApply signature.

As an experiment, I tried this:

void test()
{
        int x;
        void fun() shared
        {
            x++; // <-- error here
        }
        fun();
}

And I got this error:

Error: `shared` function `test.fun` cannot access non-shared data `x`

That looks like a bug. `fun` is a delegate, and it's shared, which
means its `this` pointer is shared, which means `this.x` should be
shared... but that seems not to be the case.

I think this is a bug. The compile error should have been that I was
unable to call fun() with a non-shared delegate (since the calling
scope is not shared).

I think fixing that issue, combined with inferring the opApply
argument attributes onto the foreach body lambda give everything I
need to implement a threadsafe foreach.

That was super hard to follow, sorry in advance!
Jun 13
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.06.19 03:40, Manu wrote:
 void test()
 {
          int x;
          void fun() shared
          {
              x++; // <-- error here
          }
          fun();
 }
 
 And I got this error:
 
 Error: `shared` function `test.fun` cannot access non-shared data `x`
 
 That looks like a bug.
It's by design. It's not pretty that the qualifiers have a different meaning for member functions and nested functions (because the nested function meaning could be useful for member functions too), but it's what we have.
 `fun` is a delegate, and it's shared, which
 means its `this` pointer is shared, which means `this.x` should be
 shared... but that seems not to be the case.
 ...
There is no `this` pointer. For local functions, `shared` means that every variable accessed in the context is shared.
 I think this is a bug. The compile error should have been that I was
 unable to call fun() with a non-shared delegate (since the calling
 scope is not shared).
There is no way to make the "calling scope shared".
Jun 13
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Thu, Jun 13, 2019 at 7:00 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 14.06.19 03:40, Manu wrote:
 void test()
 {
          int x;
          void fun() shared
          {
              x++; // <-- error here
          }
          fun();
 }

 And I got this error:

 Error: `shared` function `test.fun` cannot access non-shared data `x`

 That looks like a bug.
It's by design. It's not pretty that the qualifiers have a different meaning for member functions and nested functions (because the nested function meaning could be useful for member functions too), but it's what we have.
Okay. Well, that seems like bad design. Why is it that way? How do we fix this? I've been working towards parallel foreach for like 18 months here, I finally have all my ducks in a line, try and type the thing, and it doesn't work...
 `fun` is a delegate, and it's shared, which
 means its `this` pointer is shared, which means `this.x` should be
 shared... but that seems not to be the case.
 ...
There is no `this` pointer.
What do you mean? Local functions receive a context just like any method... if it's attributed, then it should be attributed.
 For local functions, `shared` means that
 every variable accessed in the context is shared.
Not clear what you mean. It sounds like you suggest what I expect, but that's not what I observe. I expect that if the context is shared, then the transitive rule applies normal. Everything accessed should be shared...?
 I think this is a bug. The compile error should have been that I was
 unable to call fun() with a non-shared delegate (since the calling
 scope is not shared).
There is no way to make the "calling scope shared".
Yes. So I expect the compile error I suggested?
Jun 13
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 14.06.19 07:51, Manu wrote:
 On Thu, Jun 13, 2019 at 7:00 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 14.06.19 03:40, Manu wrote:
 void test()
 {
           int x;
           void fun() shared
           {
               x++; // <-- error here
           }
           fun();
 }

 And I got this error:

 Error: `shared` function `test.fun` cannot access non-shared data `x`

 That looks like a bug.
It's by design. It's not pretty that the qualifiers have a different meaning for member functions and nested functions (because the nested function meaning could be useful for member functions too), but it's what we have.
Okay. Well, that seems like bad design. Why is it that way? How do we fix this? I've been working towards parallel foreach for like 18 months here, I finally have all my ducks in a line, try and type the thing, and it doesn't work... ...
The simplified example above should definitely not compile, and the error message is fine. Your parallel `foreach` should certainly work, and the compiler needs to be fixed to support it. I don't see however what one has to do with the other.
 `fun` is a delegate, and it's shared, which
 means its `this` pointer is shared, which means `this.x` should be
 shared... but that seems not to be the case.
 ...
There is no `this` pointer.
What do you mean? Local functions receive a context just like any method... if it's attributed, then it should be attributed. ...
There is no such thing. This makes no sense. You can't "attribute a context".
 For local functions, `shared` means that
 every variable accessed in the context is shared.
Not clear what you mean. It sounds like you suggest what I expect, but that's not what I observe. I expect that if the context is shared, then the transitive rule applies normal. Everything accessed should be shared...? ...
That doesn't mean it is implicitly made shared, it has to be shared to begin with.
 I think this is a bug. The compile error should have been that I was
 unable to call fun() with a non-shared delegate (since the calling
 scope is not shared).
There is no way to make the "calling scope shared".
Yes. So I expect the compile error I suggested?
That error makes no sense. Your problem is that local functions don't infer context qualifiers. E.g.: void main(){ pragma(msg, typeof(delegate(){})); // should be: void delegate() immutable pure nothrow nogc safe } Currently, `immutable` is not included. This should certainly be fixed. Even if there is not full inference, this should certainly work: void foo(void delegate()shared dg){} void main(){ foo((){}); }
Jun 14
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/13/2019 6:55 PM, Timon Gehr wrote:
 It's not pretty that the qualifiers have a different meaning for 
 member functions and nested functions (because the nested function meaning
could 
 be useful for member functions too), but it's what we have.
Keep in mind that the address of a nested function and the address of a member function have the same ABI. In fact, closures are implemented as member functions under the hood.
Jun 18
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 11:10, Walter Bright wrote:
 On 6/13/2019 6:55 PM, Timon Gehr wrote:
 It's not pretty that the qualifiers have a different meaning for 
 member functions and nested functions (because the nested function 
 meaning could be useful for member functions too), but it's what we have.
Keep in mind that the address of a nested function and the address of a member function have the same ABI. In fact, closures are implemented as member functions under the hood.
I know. This is how it should be, and it is not a problem. Basically, I was complaining about the aesthetics of the fact that there is no trivial way to write a member function for only e.g. the immutable part of a struct instance. As I said, I don't think this part needs fixing.
Jun 18
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On Tue, Jun 18, 2019 at 7:15 PM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 6/13/2019 6:55 PM, Timon Gehr wrote:
 It's not pretty that the qualifiers have a different meaning for
 member functions and nested functions (because the nested function meaning
could
 be useful for member functions too), but it's what we have.
Keep in mind that the address of a nested function and the address of a member function have the same ABI. In fact, closures are implemented as member functions under the hood.
Right, as it should. But it genuinely is weird that the object is not qualified by the qualifier. If that were the case, it would all wring out in the wash rather than all these bugs and oddities.
Jun 18
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 14:09, Manu wrote:
 On Tue, Jun 18, 2019 at 7:15 PM Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 6/13/2019 6:55 PM, Timon Gehr wrote:
 It's not pretty that the qualifiers have a different meaning for
 member functions and nested functions (because the nested function meaning
could
 be useful for member functions too), but it's what we have.
Keep in mind that the address of a nested function and the address of a member function have the same ABI. In fact, closures are implemented as member functions under the hood.
Right, as it should. But it genuinely is weird that the object is not qualified by the qualifier.
It is, the compiler just saves an allocation. I think this optimization would be valid in general.
 If that were the case, it would all wring
 out in the wash rather than all these bugs and oddities.
 
Again, this is untrue. Most bugs around function qualifiers are not restricted to nested functions. There might have been a couple fewer bugs if closures were type checked after lowering, but this has other issues (error message quality).
Jun 18
prev sibling parent reply Kagamin <spam here.lot> writes:
On Friday, 14 June 2019 at 01:40:46 UTC, Manu wrote:
 This has cascading issues; the closure contains references to 
 outer objects, but if the delegate is `shared`, then all those 
 references transitively inherit the shared attribute too. It 
 should work exactly like I if it was writing a `shared` method 
 to some struct, and all the members are shared... so the body 
 of the foreach needs to typecheck assuming the transitive 
 application of the inferred attributes from the opApply 
 signature.
Data attributes on delegates are not well supported because of https://issues.dlang.org/show_bug.cgi?id=1983 You have to construct the delegate and call the function in the usual way.
 As an experiment, I tried this:

 void test()
 {
         int x;
         void fun() shared
         {
             x++; // <-- error here
         }
         fun();
 }

 And I got this error:

 Error: `shared` function `test.fun` cannot access non-shared 
 data `x`

 That looks like a bug. `fun` is a delegate, and it's shared, 
 which means its `this` pointer is shared, which means `this.x` 
 should be shared... but that seems not to be the case.

 I think this is a bug. The compile error should have been that 
 I was
 unable to call fun() with a non-shared delegate (since the 
 calling
 scope is not shared).
Unqualified local variable is thread local and shouldn't be accessed by shared closure, because then it would be implicitly shared between threads.
Jun 14
parent reply Manu <turkeyman gmail.com> writes:
On Fri, Jun 14, 2019 at 1:30 AM Kagamin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Friday, 14 June 2019 at 01:40:46 UTC, Manu wrote:
 This has cascading issues; the closure contains references to
 outer objects, but if the delegate is `shared`, then all those
 references transitively inherit the shared attribute too. It
 should work exactly like I if it was writing a `shared` method
 to some struct, and all the members are shared... so the body
 of the foreach needs to typecheck assuming the transitive
 application of the inferred attributes from the opApply
 signature.
Data attributes on delegates are not well supported because of https://issues.dlang.org/show_bug.cgi?id=1983 You have to construct the delegate and call the function in the usual way.
 As an experiment, I tried this:

 void test()
 {
         int x;
         void fun() shared
         {
             x++; // <-- error here
         }
         fun();
 }

 And I got this error:

 Error: `shared` function `test.fun` cannot access non-shared
 data `x`

 That looks like a bug. `fun` is a delegate, and it's shared,
 which means its `this` pointer is shared, which means `this.x`
 should be shared... but that seems not to be the case.

 I think this is a bug. The compile error should have been that
 I was
 unable to call fun() with a non-shared delegate (since the
 calling
 scope is not shared).
Unqualified local variable is thread local and shouldn't be accessed by shared closure, because then it would be implicitly shared between threads.
Right, exactly... the compile error should be at the assignment of the not-shared closure to the shared delegate. The error would be something like "can't assign `Context*` to `shared(Context)*`". So, the function can certainly exist, but for shared, you should get an error when you attempt to call it passing the closure to the function. const however would work as expected. So why do I care about a shared local function? Because I can create supporting machinery to assert thread-safety and call it correctly, just like you do to use shared in any way at all. This is an absolutely necessary step that leads to parallel-for loops... one step at a time.
Jun 14
parent reply Kagamin <spam here.lot> writes:
On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
 Right, exactly... the compile error should be at the assignment 
 of the
 not-shared closure to the shared delegate. The error would be
 something like "can't assign `Context*` to `shared(Context)*`".
 So, the function can certainly exist, but for shared, you 
 should get
 an error when you attempt to call it passing the closure to the
 function.
The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
Jun 14
parent reply Manu <turkeyman gmail.com> writes:
On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
 Right, exactly... the compile error should be at the assignment
 of the
 not-shared closure to the shared delegate. The error would be
 something like "can't assign `Context*` to `shared(Context)*`".
 So, the function can certainly exist, but for shared, you
 should get
 an error when you attempt to call it passing the closure to the
 function.
The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting. Try it with const, it's simpler to understand. You can incorrectly mutate x from a const context. A local function is essentially: void localFun(Context*, Args...); A const local function should be: void localFun(const(Context)*, Args...) const // <- const applies to the context pointer, as usual. That does not appear to be the case however. For example: void test() { int x; void fun() const { pragma(msg, typeof(x)); // should print `const(int)` because typeof(const(Context*).x) == const(int), but it incorrectly prints `int` ++x; // <- should be an error, but the context pointer is not const, so this compiles! } } The context pointer is missing the qualifier.
Jun 14
next sibling parent reply Exil <Exil gmall.com> writes:
Looks like it was an overlooked detail? There are a few messy 
implementation details with function/delegate. If you take the 
address of a member function, without a variable. You get a 
function, from the looks of it that function can be const, which 
doesn't make sense. But then the fact it is a function and not a 
delegate also doesn't make sense.

struct S {
    void foo() const {}
}

pragma(msg, typeof(&S.foo)); // void function() const

Anyways even if you do shared like you would in a struct. You 
can't implicitly convert to shared in the way you can const.

struct D {
     int a;
     void foo() shared {
         pragma(msg, typeof(a)); // int
     }

     void bar() const {
         pragma(msg, typeof(a)); // const int
     }
}

So it is a bug for const at the very least, but shared is ya...
Jun 14
parent Manu <turkeyman gmail.com> writes:
On Fri, Jun 14, 2019 at 1:55 PM Exil via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Looks like it was an overlooked detail? There are a few messy
 implementation details with function/delegate. If you take the
 address of a member function, without a variable. You get a
 function, from the looks of it that function can be const, which
 doesn't make sense. But then the fact it is a function and not a
 delegate also doesn't make sense.

 struct S {
     void foo() const {}
 }

 pragma(msg, typeof(&S.foo)); // void function() const

 Anyways even if you do shared like you would in a struct. You
 can't implicitly convert to shared in the way you can const.
Correct, you can not implicitly convert to shared, the error message should be "Can not convert `Context*` to `shared(Context)*`", or something to that effect. That is the error I expect. But *within* the function, the context is `shared(Context)*`, and that is transitive as usual. The error should be when making the CALL to the function failing to perform the implicit conversion of the context. Where this is interesting for shared, is that I will write the appropriate machinery to assure a safe calling environment, and then cast the shared explicitly. That is how you do any interactions with shared, and it's no different here; you write unsafe code to confirm the proper environment, and then cast shared. The library is written to implement and confirm appropriate context.
 struct D {
      int a;
      void foo() shared {
          pragma(msg, typeof(a)); // int   *** no, it is `shared(int)`
      }

      void bar() const {
          pragma(msg, typeof(a)); // const int
      }
 }

 So it is a bug for const at the very least, but shared is ya...
No, in your code above, the function receives a `shared(Context)*`, and D qualifiers are transitive, so therefore the pragma should print `shared(int)`, there is absolutely no magic here. Any attempt at magic here is a terrible bug. If you try and call foo(), you should get a compile error at the callsite that it can't pass `Context*` to `shared(Context)*` which the function receives. So you can't call a shared local function... but that doesn't mean it's illegal to define one. Like I say above, the value here is that you can implement machinery to assert a valid calling environment and force the cast to make the call.
Jun 14
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.06.19 20:51, Manu wrote:
 On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
 Right, exactly... the compile error should be at the assignment
 of the
 not-shared closure to the shared delegate. The error would be
 something like "can't assign `Context*` to `shared(Context)*`".
 So, the function can certainly exist, but for shared, you
 should get
 an error when you attempt to call it passing the closure to the
 function.
The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting.
There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on? This is not the first time you do this. Sometimes, you spend some time on your own to form some partially-correct opinion and then you stick to it against all evidence to the contrary.
 Try it with const, it's simpler to understand.
That's because it's a completely different case.
 You can incorrectly
 mutate x from a const context.
 
 A local function is essentially:
 void localFun(Context*, Args...);
 
 A const local function should be:
 void localFun(const(Context)*, Args...) const  // <- const applies to
 the context pointer, as usual.
 ...
Yes, but this is not "as usual". In particular, this is not how it works for `immutable` and `shared`, because it makes no sense. With your suggestion it would be impossible to ever call an `immutable` or `shared` local function unless all data that is transitively reachable from the current stack frame is `immutable` or `shared`, respectively. That's clearly useless.
 That does not appear to be the case however.
 For example:
 
 void test()
 {
    int x;
    void fun() const
    {
      pragma(msg, typeof(x)); // should print `const(int)` because
 typeof(const(Context*).x) == const(int), but it incorrectly prints
 `int`
      ++x; // <- should be an error, but the context pointer is not
 const, so this compiles!
    }
 }
 
 The context pointer is missing the qualifier.
 
Yes, in this instance, but your actual blocker is that there is no `shared` inference for local functions.
Jun 14
next sibling parent Manu <turkeyman gmail.com> writes:
On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 14.06.19 20:51, Manu wrote:
 On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
 Right, exactly... the compile error should be at the assignment
 of the
 not-shared closure to the shared delegate. The error would be
 something like "can't assign `Context*` to `shared(Context)*`".
 So, the function can certainly exist, but for shared, you
 should get
 an error when you attempt to call it passing the closure to the
 function.
The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting.
There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on? This is not the first time you do this. Sometimes, you spend some time on your own to form some partially-correct opinion and then you stick to it against all evidence to the contrary.
Ok... Why would a function that receives a context object, which also explicitly qualifies the context, not apply the qualifier to the context?
 Try it with const, it's simpler to understand.
That's because it's a completely different case.
Not even remotely. Context* const(Context)* shared(Context)* Qualifiers hare a regular and consistent semantic when applied to things. There are no 'cases' here that I can see... what is the 'case' that I'm missing?
 You can incorrectly
 mutate x from a const context.

 A local function is essentially:
 void localFun(Context*, Args...);

 A const local function should be:
 void localFun(const(Context)*, Args...) const  // <- const applies to
 the context pointer, as usual.
 ...
Yes, but this is not "as usual". In particular, this is not how it works for `immutable` and `shared`, because it makes no sense.
What do you mean? That's *exactly* how it works with immutable and shared, and any qualifier applied to any context pointer. A qualified method accepts a qualified context pointer, and expects the calling code to supply a valid context argument. There's no magic here, and there shouldn't be.
 With your
 suggestion it would be impossible to ever call an `immutable` or
 `shared` local function unless all data that is transitively reachable
 from the current stack frame is `immutable` or `shared`, respectively.
Yes, that is **EXACTLY** what my suggestion expects. The calling code supplies a `Context*`, and all qualifiers other than `const` would emit the predictable compile error because there is no implicit conversion from `Context*` to `immutable(Context)*`, etc. The function itself must still respect the fact that it received a qualified context, or it's just lying.
 That's clearly useless.
It's not useless at all, because the compiler will still emit the function, and I can call it by force. What's _actually_ useless, is having the function accept qualifiers on the definition that don't do anything. The user shouldn't qualify the function if they don't want the context to be qualified as they specify.
 That does not appear to be the case however.
 For example:

 void test()
 {
    int x;
    void fun() const
    {
      pragma(msg, typeof(x)); // should print `const(int)` because
 typeof(const(Context*).x) == const(int), but it incorrectly prints
 `int`
      ++x; // <- should be an error, but the context pointer is not
 const, so this compiles!
    }
 }

 The context pointer is missing the qualifier.
Yes, in this instance, but your actual blocker is that there is no `shared` inference for local functions.
It's not 'inference', placing a qualifier after the function parameter list specifies an absolute and explicit qualification of the context. const and shared aren't different 'cases', qualifiers apply to the context, and if that appears to be useless or invalid, or an obvious compile error because implicit conversion is impossible, then that's the expected behaviour. In the case of shared; I can produce the appropriate calling context and cast it. But I can't write a valid function right now, because the context qualifier is disregarded.
Jun 14
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 14.06.19 20:51, Manu wrote:
 On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
 Right, exactly... the compile error should be at the assignment
 of the
 not-shared closure to the shared delegate. The error would be
 something like "can't assign `Context*` to `shared(Context)*`".
 So, the function can certainly exist, but for shared, you
 should get
 an error when you attempt to call it passing the closure to the
 function.
The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting.
There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on?
So, what is actually going on? (I believe I'm aware what is going on, because behaviour is self-evident). And more importantly, how is it useful? Why is it so useful that it should it violate default behaviour, and expectation? I know how it's not useful.
Jun 14
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.06.19 04:40, Manu wrote:
 On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 14.06.19 20:51, Manu wrote:
 On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
 Right, exactly... the compile error should be at the assignment
 of the
 not-shared closure to the shared delegate. The error would be
 something like "can't assign `Context*` to `shared(Context)*`".
 So, the function can certainly exist, but for shared, you
 should get
 an error when you attempt to call it passing the closure to the
 function.
The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting.
There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on?
...
(I'm going to ignore your other post, as it is just more rambling.) Note that this part of the language is extremely poorly implemented in DMD right now. There are quite a few independent bugs plaguing local function and delegate qualifiers. But the fact that you can't access an unshared variable from a `shared` local function is by design; it is not one of those bugs. The fact that `const` on local functions does not work properly has no bearing on `shared` on local functions. DMD implements those things independently.
 So, what is actually going on?
(NOTE: I am first explaining the actual design here, not the buggy implementation. This would make your parallel `foreach` code compile. You can't argue against what is below on the basis that it is obviously nonsense because it prevents your parallel `foreach` from compiling.) For member functions, the qualifiers affect the this pointer. I.e., you need to construct a differently-qualified receiver to call a `shared` or `immutable` member function. This is not the only behavior that would make sense. Qualifiers on member functions could also just restrict how that member function accesses other members. For local functions, the qualifiers specify how the context is _accessed_, not how the entire thing is qualified. If your local function is `shared` that means the function may only access `shared` data. void main(){ int x; // not shared, yet part of stack frame shared(int) y; int foo()shared{ // x=2; // error, x is not shared return y; // ok } // ... } Similarly, if your local function is `immutable`, it may only access `immutable` data. If your local function is `const`, it may only access `const` data, but as everything implicitly converts to `const`, it can actually access everything, it may just not modify it, accessed variables are `const`-qualified. (It's a bug in DMD that they are not.) (Const is a special case because this is the only case where you can interpret what is going on as slapping the `const` qualifier on the entire context. It is not necessarily the best way to think about what is going on.) This is the way this was intended to work, but apparently it was never fully implemented, leaving behind quite a few type system holes, for example: void main(){ int x; void bar(){ x=2; } void foo()shared{ bar(); // this shouldn't compile, but it does } } Local functions should _infer_ those attributes. So if your function (like the implicitly-generated lambda representing your `foreach` body), only accesses `shared` variables, it should be automatically `shared`-qualified.
 (I believe I'm aware what is going on, because behaviour is self-evident).
(You have demonstrated that this is not the case.)
 And more importantly, how is it useful? 
You have argued that there shouldn't be any way to call `shared`-qualified local functions. I don't understand how you can possibly think that behavior is useful. It would preclude your parallel `foreach` code from working! The intended design is useful in the sense that it would make your parallel `foreach` code work in a compiler-checked thread safe way, because the compiler would simply automatically check that the loop body only accesses shared variables (or variables local to the loop body, of course).
 Why is it so useful that it should it violate default behaviour, and
expectation?
It does not.
 I know how it's not useful.
 
I am not defending the fact that your parallel `foreach` code does not compile! This has to be fixed. But if it is fixed, and if D passes a shared delegate to your parallel `foreach`, where `shared` means it is safe to call that delegate from other threads, how would that not be useful?
Jun 14
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Fri, Jun 14, 2019 at 9:10 PM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 15.06.19 04:40, Manu wrote:
 On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 14.06.19 20:51, Manu wrote:
 On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
 Right, exactly... the compile error should be at the assignment
 of the
 not-shared closure to the shared delegate. The error would be
 something like "can't assign `Context*` to `shared(Context)*`".
 So, the function can certainly exist, but for shared, you
 should get
 an error when you attempt to call it passing the closure to the
 function.
The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting.
There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on?
...
(I'm going to ignore your other post, as it is just more rambling.) Note that this part of the language is extremely poorly implemented in DMD right now. There are quite a few independent bugs plaguing local function and delegate qualifiers. But the fact that you can't access an unshared variable from a `shared` local function is by design; it is not one of those bugs. The fact that `const` on local functions does not work properly has no bearing on `shared` on local functions. DMD implements those things independently.
 So, what is actually going on?
(NOTE: I am first explaining the actual design here, not the buggy implementation. This would make your parallel `foreach` code compile. You can't argue against what is below on the basis that it is obviously nonsense because it prevents your parallel `foreach` from compiling.) For member functions, the qualifiers affect the this pointer. I.e., you need to construct a differently-qualified receiver to call a `shared` or `immutable` member function. This is not the only behavior that would make sense. Qualifiers on member functions could also just restrict how that member function accesses other members. For local functions, the qualifiers specify how the context is _accessed_, not how the entire thing is qualified. If your local function is `shared` that means the function may only access `shared` data. void main(){ int x; // not shared, yet part of stack frame shared(int) y; int foo()shared{ // x=2; // error, x is not shared return y; // ok } // ... } Similarly, if your local function is `immutable`, it may only access `immutable` data. If your local function is `const`, it may only access `const` data, but as everything implicitly converts to `const`, it can actually access everything, it may just not modify it, accessed variables are `const`-qualified. (It's a bug in DMD that they are not.) (Const is a special case because this is the only case where you can interpret what is going on as slapping the `const` qualifier on the entire context. It is not necessarily the best way to think about what is going on.) This is the way this was intended to work, but apparently it was never fully implemented, leaving behind quite a few type system holes, for example: void main(){ int x; void bar(){ x=2; } void foo()shared{ bar(); // this shouldn't compile, but it does } } Local functions should _infer_ those attributes. So if your function (like the implicitly-generated lambda representing your `foreach` body), only accesses `shared` variables, it should be automatically `shared`-qualified.
 (I believe I'm aware what is going on, because behaviour is self-evident).
(You have demonstrated that this is not the case.)
No, I genuinely understand. You've demonstrated that I know exactly what was intended, and it's terrible. With this semantic, it's true I can call the function, but the function fundamentally doesn't work. There's no point making the function callable if the function can't do anything. I need a useful function to call before worrying about how to call it. We can get there in better ways without a complex web of special-case rules. This existing semantic expects that the context has shared objects floating around to interact with. When have you ever declared a shared object anywhere?
 And more importantly, how is it useful?
You have argued that there shouldn't be any way to call `shared`-qualified local functions. I don't understand how you can possibly think that behavior is useful. It would preclude your parallel `foreach` code from working!
I can't move on that front without fixing this first. I have a clear plan to get there, but this design choice is blocking progress. It seems pointless, can you show me any cases demonstrating where this case-specific complexity is useful?
 The intended design is useful in the sense that it would make your
 parallel `foreach` code work in a compiler-checked thread safe way,
 because the compiler would simply automatically check that the loop body
 only accesses shared variables (or variables local to the loop body, of
 course).

 Why is it so useful that it should it violate default behaviour, and
expectation?
It does not.
It does though, there is a completely different set of semantics applied to a local function than to normal methods. That's extremely surprising. The weird thing is, both methods AND local functions can be assigned to delegates... delegates do not distinguish between these 2 distinct sets of semantics, and I believe that's why these pipe-through-a-delegate-to-do-invalid-behaviour bugs exist. That would require even more special-case code to correct. A better design would be that no special case exists, and then no special case corrections to special cases should exist either. This is a bad design. Local functions context pointers should behave identical to methods, and then delegates, and code that calls delegates know what semantics they're dealing with. A 'this' pointer is a 'this' pointer, don't make it more complicated than that.
 I know how it's not useful.
I am not defending the fact that your parallel `foreach` code does not compile! This has to be fixed. But if it is fixed, and if D passes a shared delegate to your parallel `foreach`, where `shared` means it is safe to call that delegate from other threads,
I don't believe that guarantee is possible. `shared` is fundamentally unsafe, we've argued this for a long time. It is possible to build safe tools with shared, but at the low level, it is not.
 how would that not be useful?
Because the function can't access anything. This is the essential case: struct S { void method() shared; } void test() { S s; void localFun() shared { s.method(); // <- s must transitively receive const from the context pointer, otherwise this doesn't work } } Current semantics would reject this, because s is not shared, and therefore not accessible by localFun. The unnecessary complexity inhibits the only useful thing that a shared function can do.
Jun 14
parent reply Kagamin <spam here.lot> writes:
On Saturday, 15 June 2019 at 06:13:09 UTC, Manu wrote:
 struct S
 {
   void method() shared;
 }

 void test()
 {
   S s;
   void localFun() shared
   {
     s.method(); // <- s must transitively receive const from the
 context pointer, otherwise this doesn't work
   }
 }

 Current semantics would reject this, because s is not shared, 
 and
 therefore not accessible by localFun.
 The unnecessary complexity inhibits the only useful thing that a
 shared function can do.
Declare `s` as shared and it will work. It would receive qualifier from context, but unshared variables can't be captured by shared closure to begin with because unshared data can't part of shared context, because that would break the guarantee that unshared data shouldn't be shared. Currently compiler implements check which variables can be captured and which can't.
Jun 16
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Sun, Jun 16, 2019 at 6:00 PM Kagamin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 15 June 2019 at 06:13:09 UTC, Manu wrote:
 struct S
 {
   void method() shared;
 }

 void test()
 {
   S s;
   void localFun() shared
   {
     s.method(); // <- s must transitively receive const from the
 context pointer, otherwise this doesn't work
   }
 }

 Current semantics would reject this, because s is not shared,
 and
 therefore not accessible by localFun.
 The unnecessary complexity inhibits the only useful thing that a
 shared function can do.
Declare `s` as shared and it will work. It would receive qualifier from context, but unshared variables can't be captured by shared closure to begin with because unshared data can't part of shared context, because that would break the guarantee that unshared data shouldn't be shared. Currently compiler implements check which variables can be captured and which can't.
I don't think the capture should be shared, only the function argument has the shared attribute applied, ie: Capture cap; void localFun(shared(Capture)* ctx); localFun(cap); // <- error: can't pass `Context*` to `shared(Context)*`... form there, we can begin to do something interesting. In you consider the const case, this will work perfectly fine, and everything will be as expected with absolutely no magic of any kind. immutable local functions can not be called under any circumstances; this seems like common sense to me. inout is fine too, just like const. shared is the interesting one; we shouldn't be able to pass the capture to the function because Context* -> shared(Context)*, but we can start to talk about ways this can work. One way is that the promotion is actally perfectly valid! Because you are calling a local function from the local thread; so the shared function will have a shared reference to the local context only for the life of the function call, and when it returns, the shared reference must end with the function. We can do this by declaring the local function: `void localFun(scope shared(Context)* ctx);` A shared local function is valid so long as that function does NOT escape any of those shared references. It might want to attribute `return` too.
From there, parallel foreach can be implemented as a  trusted function.
Jun 16
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 17.06.19 03:17, Manu wrote:
 On Sun, Jun 16, 2019 at 6:00 PM Kagamin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Saturday, 15 June 2019 at 06:13:09 UTC, Manu wrote:
 struct S
 {
    void method() shared;
 }

 void test()
 {
    S s;
    void localFun() shared
    {
      s.method(); // <- s must transitively receive const from the
 context pointer, otherwise this doesn't work
    }
 }

 Current semantics would reject this, because s is not shared,
 and
 therefore not accessible by localFun.
 The unnecessary complexity inhibits the only useful thing that a
 shared function can do.
Declare `s` as shared and it will work. It would receive qualifier from context, but unshared variables can't be captured by shared closure to begin with because unshared data can't part of shared context, because that would break the guarantee that unshared data shouldn't be shared. Currently compiler implements check which variables can be captured and which can't.
I don't think the capture should be shared, only the function argument has the shared attribute applied, ie: Capture cap; void localFun(shared(Capture)* ctx); localFun(cap); // <- error: can't pass `Context*` to `shared(Context)*`... form there, we can begin to do something interesting. In you consider the const case, this will work perfectly fine, and everything will be as expected with absolutely no magic of any kind. immutable local functions can not be called under any circumstances; this seems like common sense to me.
It's not common sense, it's nonsense. It should be common sense that it is nonsense. I don't understand how you can possibly reach that conclusion. Closures with an immutable-qualified context can only capture immutable variables. What is so surprising about this?
 inout is fine too, just like const.
Absolutely not. You can't implicitly promote stuff to `inout`.
 shared is the interesting one; we shouldn't be able to pass the
 capture to the function because Context* -> shared(Context)*, but we
 can start to talk about ways this can work.
 One way is that the promotion is actally perfectly valid! Because you
 are calling a local function from the local thread; so the shared
 function will have a shared reference to the local context only for
 the life of the function call, and when it returns, the shared
 reference must end with the function. We can do this by declaring the
 local function: `void localFun(scope shared(Context)* ctx);`
 A shared local function is valid so long as that function does NOT
 escape any of those shared references. It might want to attribute
 `return` too.
This idea is not at all contingent on the nonsensical parts of your vision, but I don't know if there is a simple way to make this work. It will require more care.
From there, parallel foreach can be implemented as a  trusted function.
So your evil master plan is: 1. Make qualified local functions useless. 2. Make qualified local functions more useful. 3. Profit. Just skip phase 1, please.
Jun 16
parent reply Manu <turkeyman gmail.com> writes:
On Mon., 17 Jun. 2019, 11:40 am Timon Gehr via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:

 On 17.06.19 03:17, Manu wrote:
 On Sun, Jun 16, 2019 at 6:00 PM Kagamin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Saturday, 15 June 2019 at 06:13:09 UTC, Manu wrote:
 struct S
 {
    void method() shared;
 }

 void test()
 {
    S s;
    void localFun() shared
    {
      s.method(); // <- s must transitively receive const from the
 context pointer, otherwise this doesn't work
    }
 }

 Current semantics would reject this, because s is not shared,
 and
 therefore not accessible by localFun.
 The unnecessary complexity inhibits the only useful thing that a
 shared function can do.
Declare `s` as shared and it will work. It would receive qualifier from context, but unshared variables can't be captured by shared closure to begin with because unshared data can't part of shared context, because that would break the guarantee that unshared data shouldn't be shared. Currently compiler implements check which variables can be captured and which can't.
I don't think the capture should be shared, only the function argument has the shared attribute applied, ie: Capture cap; void localFun(shared(Capture)* ctx); localFun(cap); // <- error: can't pass `Context*` to `shared(Context)*`... form there, we can begin to do something interesting. In you consider the const case, this will work perfectly fine, and everything will be as expected with absolutely no magic of any kind. immutable local functions can not be called under any circumstances; this seems like common sense to me.
It's not common sense, it's nonsense. It should be common sense that it is nonsense. I don't understand how you can possibly reach that conclusion. Closures with an immutable-qualified context can only capture immutable variables. What is so surprising about this?
What's the use of that? Can you point me at any code like that? Make a const local function if you want to stop it mutating stuff... what's interesting about an immutable local function?
 inout is fine too, just like const.

 Absolutely not. You can't implicitly promote stuff to `inout`.
What are you taking about? inout accepts mutable, const, or immutable. I don't know what you could mean? Context objects are mutable, unless called via another local function where they would have gained the respective qualifier, and that would interact with inout correctly, eliminating some existing false-mutability bugs.
 shared is the interesting one; we shouldn't be able to pass the
 capture to the function because Context* -> shared(Context)*, but we
 can start to talk about ways this can work.
 One way is that the promotion is actally perfectly valid! Because you
 are calling a local function from the local thread; so the shared
 function will have a shared reference to the local context only for
 the life of the function call, and when it returns, the shared
 reference must end with the function. We can do this by declaring the
 local function: `void localFun(scope shared(Context)* ctx);`
 A shared local function is valid so long as that function does NOT
 escape any of those shared references. It might want to attribute
 `return` too.
This idea is not at all contingent on the nonsensical
How is uniformity nonsense? Are you saying that qualified methods are nonsense? How could applying proven, uniform, and predictable rules be considered nonsense? Can you show me how these special-case rules are useful? Qualified local functions are extremely rare; I suspect someone just dun some design that sounded cool and felt chuffed, but probably never used it. parts of your
 vision, but I don't know if there is a simple way to make this work. It
 will require more care.
I agree there's certainly no simple way to do it assuming the current design; I did try and imagine a solution extensively under the existing implementation for quite some time, but I don't believe it's possible without effectively transforming it towards the default semantics. ...which isn't actually surprising! You shouldn't be surprised that uniform semantics interact properly with all the other existing language. If the design just worked like any normal qualified this pointer, there'd be nothing to think about other than how to call the shared function, which I think is a problem we can defeat. Everything else would work elegantly with no special code in the compiler.
From there, parallel foreach can be implemented as a  trusted function.
So your evil master plan is: 1. Make qualified local functions useless.
Not at all, const and inout work as expected (surely the 99% case), and reflective meta will actually work too. Existing bugs are all eliminated instantly. `immutable` has no useful meaning anyway (just use const? The thing about local functions is that they're local. It's not API material), shared is the only interesting case that's affected, and I can't imagine another way to make it useful. 2. Make qualified local functions more useful.

If you can do this without 1, then great. I couldn't imagine a solution.
The problem is specifically the non-uniformity and special case rules.

3. Profit.
 Just skip phase 1, please.
I've asked a couple of times for a demonstration of how existing semantics are useful? It's certainly complex, non-uniform, and surprising. The closure has un-qualified objects in it despite the function qualification, which means typeof() doesn't work right, and derivative meta that depends on that is broken by consequence. Explain to me how qualifying the context makes it useless? How does it make anything that works right now stop working? I think that makes it uniform, only meaningfully affects shared (in a positive way), and removes a mountain of complex special case code. But you know what, do what you want, I'll shut up... If it just gets fixed, I won't complain.

Jun 16
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 17.06.19 08:15, Manu wrote:
     Closures with an immutable-qualified context can only
     capture immutable variables. What is so surprising about this?
 
 
 What's the use of that? Can you point me at any code like that?
No, I don't have in my head an index of all existing D code keyed on which features it is using (not even my own), but I know how to design a programming language correctly. One reason why I don't have a lot of examples demonstrating usage of qualifiers is that I avoid them as well as I can -- their implementation is broken, and currently I have no interest in dealing with all of those bugs. If I spent an unreasonable amount of time I would probably be able to produce plausible examples, but this would be a waste of my time, because it is not what I am good at or enjoy doing. It would be a lot more productive if you produced the examples and trusted me more when I explain the language design considerations. I also want the language to improve, this is why I push back against bad aspects of otherwise good ideas. It does not mean I oppose the general direction things are moving.
 Make a const local function if you want to stop it mutating stuff...
That doesn't stop others from mutating. The _only way_ to get a strongly pure delegate literal is to qualify its context immutable. I hope this is reason enough without a concrete usage example.
 what's interesting about an immutable local function?
 ...
You can call an immutable-qualified delegate only if the context pointer is immutable-qualified. If it is const-qualified, you can't do that without breaking the type system. It would be so weird if the only way to make a strongly pure delegate would be to create a local `static struct`, explicitly capture the `immutable` variables in it's members, and take the address of an `immutable` member function of the resulting immutable `struct` instance. Maybe this way of thinking about it helps: You don't capture the entire stack frame and the context does not point to the entire stack frame. It only points to the variables that were actually captured.
      > inout is fine too, just like const.
 
     Absolutely not. You can't implicitly promote stuff to `inout`.
 
 
 What are you taking about? inout accepts mutable, const, or immutable. I 
 don't know what you could mean?
What I meant is `inout` does not work "just like const". Maybe this is not what you meant to imply. Conceptually, `inout` is a polymorphic parameter. It represents an actual qualifier at each specific point in the program, you just don't know what it is. You can't implicitly convert a mutable variable into an `inout` variable, because it is possible that `inout` means `immutable` during the actual execution of the program.
 Context objects are mutable, unless called via another local function 
 where they would have gained the respective qualifier, and that would 
 interact with inout correctly, eliminating some existing 
 false-mutability bugs.
 ...
I hate how complicated it is to explain why this is nonsense, especially because it is only a tangent in your post, but I'll bite. `inout` strikes again. (BTW: How much time do you spend thinking about your posts in this thread? I am wasting hours upon hours. I can't keep this up much longer, I have other things I need to do... Please start figuring out the holes in your suggestions on your own. Please be more careful with the assertions that you make.) The issue with `inout` is that there is scope confusion: there can be multiple enclosing function scopes, all of which have an `inout` qualifier, and all of those different polymorphic parameters have the same name: "inout". Right now the language deals with this (somewhat unsuccessfully implemented in DMD!), by effectively disallowing nested `inout` functions (their `inout` is treated as being the same as that of the enclosing function). Therefore, right now there is no way to "interact with inout correctly" in the way you suggest, the `inout` qualifier on a nested function cannot be instantiated with a concrete value, because this happens only once the _outermost_ function gets called. Of course, you will now just say that this is all bollocks and we should change all of that too. So let's summarize. What you are saying is: 1. Any local function can capture any local variable from an enclosing scope. 2. The type of those variables pick up the context qualifier of the local functions trough which they are captured. 3. `inout` local functions can be called from other local functions and their context `inout` is instantiated with the qualifier of the other local function. But you can't actually qualify a `foo` context with `bar`s `inout`, because you would immediately get `inout` confusion. Of course, the compiler would still happily do it. So your proposal would lead to at least one `inout` bug: int* foo(inout(int)* x) safe{ inout(int)* bar()inout{ return x; // ok, type of x is inout(inout(int)*) } int* baz(){ return bar(); } return baz(); } void main() safe{ immutable(int)* x=...; int* y=foo(x); // x and y alias, UB assert(x is y); } (The function `baz` is just to make it very explicit that this is what you propose, I think you agree that `foo` could return bar(dummy) directly with the same result.) The right solution would actually be to have proper scoping for `inout`, where additionally you can have multiple names for `inout`, e.g. `inout!"foo"` and `inout!"bar"`. This would fix every `inout` false-mutability bug, with your weird context interpretation or without. But then the correct design is that a function with `inout` context would only be able to capture variables with the right type of `inout`, the one as which its context is qualified. Otherwise you couldn't call `inout` local functions.
      > shared is the interesting one; we shouldn't be able to pass the
      > capture to the function because Context* -> shared(Context)*, but we
      > can start to talk about ways this can work.
      > One way is that the promotion is actally perfectly valid! Because you
      > are calling a local function from the local thread; so the shared
      > function will have a shared reference to the local context only for
      > the life of the function call, and when it returns, the shared
      > reference must end with the function. We can do this by declaring the
      > local function: `void localFun(scope shared(Context)* ctx);`
      > A shared local function is valid so long as that function does NOT
      > escape any of those shared references. It might want to attribute
      > `return` too.
 
     This idea is not at all contingent on the nonsensical
 
 
 How is uniformity nonsense?
Probably I should have been more careful in my first post. I did explicitly say that the local function meaning could be useful for member functions and not the other way around, but I guess you ignored that and just concluded "non-uniform". It's not really all that non-uniform, I can implement `immutable` capturing manually like this: // version with closures: int delegate()immutable bar(){ immutable(int) x = 3; int foo()immutable{ return x; } return &foo; } // version without closures: int delegate()immutable bar(){ struct StackFrame{ immutable(int) x = 3; } auto theFrame=new StackFrame(); struct Closure{ int* xp; int foo()immutable{ return *xp; } } return &new immutable(Closure)(&theFrame.x).foo; } Of course, the compiler will do with one heap allocation to get an equivalent result, which is currently impossible to do with member functions.
 Are you saying that qualified methods are nonsense?
Qualifying the `this` pointer is not nonsense, because you can actually get a value of that type. Qualifying the full stack frame is nonsense, because there is no way to create an instance of a qualified stack frame, except for qualifiers where you can implicitly promote an unqualified stack frame, and there is a very useful way to interpret qualified capturing, which is easy to understand (see above). However, I do indeed dislike that there is no way to say a member function will only access `immutable` members: struct S{ immutable x; int y; int foo(){ // only accesses immutable data return x; } } void main(){ // could work, but typesystem can't show it: int delegate()immutable dg=&new S(1,2).foo; } There is a workaround: struct T{ int x; int foo()immutable{ return x; } } struct S{ immutable(T) t; int y; this(int x,int y){ t=T(x); y=y; } } void main(){ int delegate()immutable dg=&new S(1,2).t.foo; } This workaround would also work in a version of the language that implements your set of rules (with `inout` bugs and all): But I'd much rather write this than the above: void main(){ immutable(int) x=1; int y=2; int delegate()immutable dg=()=>x; writeln(dg()); } With your suggested changes, this would not compile, for no benefit at all. It would stop qualifier inference for nested functions in its tracks.
 How could applying proven, uniform, and predictable rules be 
 considered nonsense?
The nonsense is not in how you apply the rules (that part is fine, you correctly derive the consequences), it is in how you select the rules and how you justify selecting those rules.
 Can you show me how these special-case rules are useful?
I have argued above that there is no special case.
 Qualified local functions are extremely rare;
Because they don't work properly, which you have already noticed. E.g., above I should be allowed to return `int delegate()` instead of `int delegate()immutable`, but the compiler won't let me. It's full of bugs. I would use qualifiers if they worked, and I would use them more if I had to write multithreaded code.
 I suspect someone just dun some design 
 that sounded cool and felt chuffed, but probably never used it.
 
     parts of your
     vision, but I don't know if there is a simple way to make this work. It
     will require more care.
 
 
 I agree there's certainly no simple way to do it assuming the current 
 design; 
I have shown you how to manually construct `immutable` closures with nothing but structs and member functions. I hope that clarifies that this assertions is plain nonsense.
 I did try and imagine a solution extensively under the existing 
 implementation for quite some time, but I don't believe it's possible 
 without effectively transforming it towards the default semantics.
 ...which isn't actually surprising! You shouldn't be surprised that 
 uniform semantics interact properly with all the other existing 
 language. If the design just worked like any normal qualified this 
 pointer, there'd be nothing to think about other than how to call the 
 shared function, which I think is a problem we can defeat. Everything 
 else would work elegantly with no special code in the compiler.
 ...
There would be the same amount of special code in the compiler, because stack frames are not structs.
      >>From there, parallel foreach can be implemented as a  trusted
     function.
      >
 
     So your evil master plan is:
 
     1. Make qualified local functions useless.
 
 
 Not at all, const and inout work as expected
I agree: `const` works fine, `inout` continues to be buggy, as expected.
 (surely the 99% case),
I hate 99% features.
 and reflective meta will actually work too.
When does it not work? (I am not doubting that it does not work, but I very much doubt it is because qualified closures are designed usefully.)
 Existing bugs are all eliminated instantly.
That's plain snake oil and completely untrue. Have you ever seen the DMD source code?
 `immutable` has no useful meaning anyway
Wtf.
 (just use const? The 
 thing about local functions is that they're local. It's not API 
 material),
Yes, it is. Qualified delegate types can appear in APIs. Your very first example in this thread is an example of how it is API material.
 shared is the only interesting case that's affected, and I 
 can't imagine another way to make it useful.
 ...
I hope after reading this and the previous post you will notice that there is actually no blocker.
     2. Make qualified local functions more useful.
 
 
 If you can do this without 1, then great.
I can.
 I couldn't imagine a solution. 
 The problem is specifically the non-uniformity and special case rules.
 ...
Do you still feel that way now?
     3. Profit.
 
     Just skip phase 1, please.
 
 
 I've asked a couple of times for a demonstration of how existing 
 semantics are useful? It's certainly complex, non-uniform, and surprising.
 The closure has un-qualified objects in it despite the function 
 qualification, which means typeof() doesn't work right, and derivative 
 meta that depends on that is broken by consequence.
 ...
The closure does not have unqualified objects in it. Why is it a problem that it is not necessarily contiguous in memory because the compiler optimizes away allocations?
 Explain to me how qualifying the context makes it useless? How does it 
 make anything that works right now stop working?
I have shown a few examples in this post and argued why they are useful. (E.g., strongly pure delegates.)
 I think that makes it uniform, only meaningfully affects shared (in a 
 positive way), and removes a mountain of complex special case code.
 ...
Step 1 only removes functionality, it does not eliminate any blockers for anything, including your proposed redefinition of `shared` to mean `threadsafe`.
 But you know what, do what you want, I'll shut up... If it just gets 
 fixed, I won't complain.
Having wasted a lot of time on this thread, it would still be great if you made an effort to understand my points.
Jun 17
parent reply Manu <turkeyman gmail.com> writes:
On Tue, Jun 18, 2019 at 8:10 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 17.06.19 08:15, Manu wrote:
     Closures with an immutable-qualified context can only
     capture immutable variables. What is so surprising about this?


 What's the use of that? Can you point me at any code like that?
No, I don't have in my head an index of all existing D code keyed on which features it is using (not even my own), but I know how to design a programming language correctly. One reason why I don't have a lot of examples demonstrating usage of qualifiers is that I avoid them as well as I can -- their implementation is broken, and currently I have no interest in dealing with all of those bugs. If I spent an unreasonable amount of time I would probably be able to produce plausible examples, but this would be a waste of my time, because it is not what I am good at or enjoy doing. It would be a lot more productive if you produced the examples and trusted me more when I explain the language design considerations. I also want the language to improve, this is why I push back against bad aspects of otherwise good ideas. It does not mean I oppose the general direction things are moving.
 Make a const local function if you want to stop it mutating stuff...
That doesn't stop others from mutating. The _only way_ to get a strongly pure delegate literal is to qualify its context immutable. I hope this is reason enough without a concrete usage example.
 what's interesting about an immutable local function?
 ...
You can call an immutable-qualified delegate only if the context pointer is immutable-qualified. If it is const-qualified, you can't do that without breaking the type system. It would be so weird if the only way to make a strongly pure delegate would be to create a local `static struct`, explicitly capture the `immutable` variables in it's members, and take the address of an `immutable` member function of the resulting immutable `struct` instance. Maybe this way of thinking about it helps: You don't capture the entire stack frame and the context does not point to the entire stack frame. It only points to the variables that were actually captured.
      > inout is fine too, just like const.

     Absolutely not. You can't implicitly promote stuff to `inout`.


 What are you taking about? inout accepts mutable, const, or immutable. I
 don't know what you could mean?
What I meant is `inout` does not work "just like const". Maybe this is not what you meant to imply. Conceptually, `inout` is a polymorphic parameter. It represents an actual qualifier at each specific point in the program, you just don't know what it is. You can't implicitly convert a mutable variable into an `inout` variable, because it is possible that `inout` means `immutable` during the actual execution of the program.
 Context objects are mutable, unless called via another local function
 where they would have gained the respective qualifier, and that would
 interact with inout correctly, eliminating some existing
 false-mutability bugs.
 ...
I hate how complicated it is to explain why this is nonsense, especially because it is only a tangent in your post, but I'll bite. `inout` strikes again. (BTW: How much time do you spend thinking about your posts in this thread? I am wasting hours upon hours. I can't keep this up much longer, I have other things I need to do... Please start figuring out the holes in your suggestions on your own. Please be more careful with the assertions that you make.) The issue with `inout` is that there is scope confusion: there can be multiple enclosing function scopes, all of which have an `inout` qualifier, and all of those different polymorphic parameters have the same name: "inout". Right now the language deals with this (somewhat unsuccessfully implemented in DMD!), by effectively disallowing nested `inout` functions (their `inout` is treated as being the same as that of the enclosing function). Therefore, right now there is no way to "interact with inout correctly" in the way you suggest, the `inout` qualifier on a nested function cannot be instantiated with a concrete value, because this happens only once the _outermost_ function gets called. Of course, you will now just say that this is all bollocks and we should change all of that too. So let's summarize. What you are saying is: 1. Any local function can capture any local variable from an enclosing scope. 2. The type of those variables pick up the context qualifier of the local functions trough which they are captured. 3. `inout` local functions can be called from other local functions and their context `inout` is instantiated with the qualifier of the other local function. But you can't actually qualify a `foo` context with `bar`s `inout`, because you would immediately get `inout` confusion. Of course, the compiler would still happily do it. So your proposal would lead to at least one `inout` bug: int* foo(inout(int)* x) safe{ inout(int)* bar()inout{ return x; // ok, type of x is inout(inout(int)*) } int* baz(){ return bar(); } return baz(); } void main() safe{ immutable(int)* x=...; int* y=foo(x); // x and y alias, UB assert(x is y); } (The function `baz` is just to make it very explicit that this is what you propose, I think you agree that `foo` could return bar(dummy) directly with the same result.) The right solution would actually be to have proper scoping for `inout`, where additionally you can have multiple names for `inout`, e.g. `inout!"foo"` and `inout!"bar"`. This would fix every `inout` false-mutability bug, with your weird context interpretation or without. But then the correct design is that a function with `inout` context would only be able to capture variables with the right type of `inout`, the one as which its context is qualified. Otherwise you couldn't call `inout` local functions.
      > shared is the interesting one; we shouldn't be able to pass the
      > capture to the function because Context* -> shared(Context)*, but we
      > can start to talk about ways this can work.
      > One way is that the promotion is actally perfectly valid! Because you
      > are calling a local function from the local thread; so the shared
      > function will have a shared reference to the local context only for
      > the life of the function call, and when it returns, the shared
      > reference must end with the function. We can do this by declaring the
      > local function: `void localFun(scope shared(Context)* ctx);`
      > A shared local function is valid so long as that function does NOT
      > escape any of those shared references. It might want to attribute
      > `return` too.

     This idea is not at all contingent on the nonsensical


 How is uniformity nonsense?
Probably I should have been more careful in my first post. I did explicitly say that the local function meaning could be useful for member functions and not the other way around, but I guess you ignored that and just concluded "non-uniform". It's not really all that non-uniform, I can implement `immutable` capturing manually like this: // version with closures: int delegate()immutable bar(){ immutable(int) x = 3; int foo()immutable{ return x; } return &foo; } // version without closures: int delegate()immutable bar(){ struct StackFrame{ immutable(int) x = 3; } auto theFrame=new StackFrame(); struct Closure{ int* xp; int foo()immutable{ return *xp; } } return &new immutable(Closure)(&theFrame.x).foo; } Of course, the compiler will do with one heap allocation to get an equivalent result, which is currently impossible to do with member functions.
 Are you saying that qualified methods are nonsense?
Qualifying the `this` pointer is not nonsense, because you can actually get a value of that type. Qualifying the full stack frame is nonsense, because there is no way to create an instance of a qualified stack frame, except for qualifiers where you can implicitly promote an unqualified stack frame, and there is a very useful way to interpret qualified capturing, which is easy to understand (see above).
Right... this is my entire point, and why I say that immutable local methods make no sense. Local methods are not yet delegates; they may be captured into a delegate, but that's an interesting operation, and that's also the moment where `immutable` becomes interesting. At the time you want to capture a delegate from a local function, you can copy If we create a closure by copying referenced data into an immutable object, then it doesn't depend on referenced elements being immutable, only copied to immutable. Such a closure could be allocated on the stack for scope delegates.
 However, I do indeed dislike that there is no way to say a member
 function will only access `immutable` members:

 struct S{
      immutable x;
      int y;
      int foo(){ // only accesses immutable data
          return x;
      }
 }

 void main(){
       // could work, but typesystem can't show it:
      int delegate()immutable dg=&new S(1,2).foo;
 }

 There is a workaround:

 struct T{
      int x;
      int foo()immutable{
          return x;
      }
 }
 struct S{
      immutable(T) t;
      int y;
      this(int x,int y){ t=T(x); y=y; }
 }

 void main(){
      int delegate()immutable dg=&new S(1,2).t.foo;
 }

 This workaround would also work in a version of the language that
 implements your set of rules (with `inout` bugs and all):

 But I'd much rather write this than the above:

 void main(){
      immutable(int) x=1;
      int y=2;
      int delegate()immutable dg=()=>x;
      writeln(dg());
 }

 With your suggested changes, this would not compile, for no benefit at
 all. It would stop qualifier inference for nested functions in its tracks.

 How could applying proven, uniform, and predictable rules be
 considered nonsense?
The nonsense is not in how you apply the rules (that part is fine, you correctly derive the consequences), it is in how you select the rules and how you justify selecting those rules.
 Can you show me how these special-case rules are useful?
I have argued above that there is no special case.
 Qualified local functions are extremely rare;
Because they don't work properly, which you have already noticed. E.g., above I should be allowed to return `int delegate()` instead of `int delegate()immutable`, but the compiler won't let me. It's full of bugs. I would use qualifiers if they worked, and I would use them more if I had to write multithreaded code.
 I suspect someone just dun some design
 that sounded cool and felt chuffed, but probably never used it.

     parts of your
     vision, but I don't know if there is a simple way to make this work. It
     will require more care.


 I agree there's certainly no simple way to do it assuming the current
 design;
I have shown you how to manually construct `immutable` closures with nothing but structs and member functions. I hope that clarifies that this assertions is plain nonsense.
 I did try and imagine a solution extensively under the existing
 implementation for quite some time, but I don't believe it's possible
 without effectively transforming it towards the default semantics.
 ...which isn't actually surprising! You shouldn't be surprised that
 uniform semantics interact properly with all the other existing
 language. If the design just worked like any normal qualified this
 pointer, there'd be nothing to think about other than how to call the
 shared function, which I think is a problem we can defeat. Everything
 else would work elegantly with no special code in the compiler.
 ...
There would be the same amount of special code in the compiler, because stack frames are not structs.
      >>From there, parallel foreach can be implemented as a  trusted
     function.
      >

     So your evil master plan is:

     1. Make qualified local functions useless.


 Not at all, const and inout work as expected
I agree: `const` works fine, `inout` continues to be buggy, as expected.
 (surely the 99% case),
I hate 99% features.
 and reflective meta will actually work too.
When does it not work? (I am not doubting that it does not work, but I very much doubt it is because qualified closures are designed usefully.)
 Existing bugs are all eliminated instantly.
That's plain snake oil and completely untrue. Have you ever seen the DMD source code?
 `immutable` has no useful meaning anyway
Wtf.
 (just use const? The
 thing about local functions is that they're local. It's not API
 material),
Yes, it is. Qualified delegate types can appear in APIs. Your very first example in this thread is an example of how it is API material.
 shared is the only interesting case that's affected, and I
 can't imagine another way to make it useful.
 ...
I hope after reading this and the previous post you will notice that there is actually no blocker.
     2. Make qualified local functions more useful.


 If you can do this without 1, then great.
I can.
 I couldn't imagine a solution.
 The problem is specifically the non-uniformity and special case rules.
 ...
Do you still feel that way now?
     3. Profit.

     Just skip phase 1, please.


 I've asked a couple of times for a demonstration of how existing
 semantics are useful? It's certainly complex, non-uniform, and surprising.
 The closure has un-qualified objects in it despite the function
 qualification, which means typeof() doesn't work right, and derivative
 meta that depends on that is broken by consequence.
 ...
The closure does not have unqualified objects in it. Why is it a problem that it is not necessarily contiguous in memory because the compiler optimizes away allocations?
 Explain to me how qualifying the context makes it useless? How does it
 make anything that works right now stop working?
I have shown a few examples in this post and argued why they are useful. (E.g., strongly pure delegates.)
 I think that makes it uniform, only meaningfully affects shared (in a
 positive way), and removes a mountain of complex special case code.
 ...
Step 1 only removes functionality, it does not eliminate any blockers for anything, including your proposed redefinition of `shared` to mean `threadsafe`.
 But you know what, do what you want, I'll shut up... If it just gets
 fixed, I won't complain.
Having wasted a lot of time on this thread, it would still be great if you made an effort to understand my points.
I started replying here; but in a large part, you're telling me things I am absolutely aware of as if they're details I missed... and where I disagree, you'll just hate anyway. I genuinely don't intend to waste your time. I believe 80% of the issue you have with what I'm saying is because you either misunderstand what I say, I poorly communicate what I mean, or because I skip the broader details of my suggestions because it's too long. I'm starting to see that it *might* even be possible that our visions are actually very closely aligned, but presented slightly differently. I can't work via email, it kills me, and it wastes our time. Are you in the US? Where do you live? I will fly to you, we can fix it with a whiteboard because forums are the most bullshit form of communication ever divised by man. This is what dconf should be is for, but instead we spend the whole time watching lectures. I wish I had the time + energy to fork the language and just prove shit, rather than waste time talking about it.
Jun 17
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 01:34, Manu wrote:
 ...
 
 ...
 I'm starting to see that it*might*  even be possible that our visions
 are actually very closely aligned, but presented slightly differently.
 ...
I think this is the case. In this thread I was mostly fighting to keep qualified closures working. Qualified closures other than `threadsafe` are perhaps not key to your use cases.
 I can't work via email, it kills me, and it wastes our time. Are you
 in the US? Where do you live?
Switzerland. I'm a PhD student at ETH Zürich.
 I will fly to you, we can fix it with a whiteboard
 because forums are the most bullshit form of communication
 ever divised by man.
 ...
I fully agree with that. There is too much time between misunderstanding and correction.
 This is what dconf should be is for, but instead we spend the whole
 time watching lectures.
 ...
Unfortunately I wasn't able to make it to DConf because of paper deadlines and other obligations and commitments...
 I wish I had the time + energy to fork the language and just prove
 shit, rather than waste time talking about it.
So do I.
Jun 17
prev sibling next sibling parent SashaGreat <sasha hotmail.com> writes:
On Monday, 17 June 2019 at 23:34:17 UTC, Manu wrote:
 I can't work via email, it kills me, and it wastes our time. 
 Are you in the US? Where do you live? I will fly to you, we can 
 fix it with a whiteboard because forums are the most bullshit 
 form of communication ever divised by man.
Lots of projects are handled remotely these days and while forum can be a cumbersome, why you two don't go to a Web Cam session to solve this? If you need to sit together in person to solve the problems like these days... well sorry to say but you are completely out of scope.
Jun 17
prev sibling parent M.M. <matus email.cz> writes:
On Monday, 17 June 2019 at 23:34:17 UTC, Manu wrote:
 This is what dconf should be is for, but instead we spend the 
 whole time watching lectures.
I largely agree. dconf should be about personal meetings and exchanges (that's a huge difference to what Joakim once suggested), and I agree that the lectures at dconf should be much shorter, with an exception of very few in-depth talks (that's a huge agreement to what Joakim once suggested).
Jun 18
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On Sun, Jun 16, 2019 at 6:00 PM Kagamin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 15 June 2019 at 06:13:09 UTC, Manu wrote:
 struct S
 {
   void method() shared;
 }

 void test()
 {
   S s;
   void localFun() shared
   {
     s.method(); // <- s must transitively receive const from the
 context pointer, otherwise this doesn't work
   }
 }

 Current semantics would reject this, because s is not shared,
 and
 therefore not accessible by localFun.
 The unnecessary complexity inhibits the only useful thing that a
 shared function can do.
Declare `s` as shared and it will work.
`s` is not declared as shared... nothing is ever declared as shared, that's why the current design is completely useless.
Jun 16
parent reply Kagamin <spam here.lot> writes:
On Monday, 17 June 2019 at 01:26:53 UTC, Manu wrote:
 Declare `s` as shared and it will work.
`s` is not declared as shared... nothing is ever declared as shared, that's why the current design is completely useless.
Rewamping the language is too disrupting and I don't think it can be good, you better fork it.
Jun 17
parent reply Manu <turkeyman gmail.com> writes:
On Mon., 17 Jun. 2019, 7:25 pm Kagamin via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:

 On Monday, 17 June 2019 at 01:26:53 UTC, Manu wrote:
 Declare `s` as shared and it will work.
`s` is not declared as shared... nothing is ever declared as shared, that's why the current design is completely useless.
Rewamping the language is too disrupting and I don't think it can be good, you better fork it.
I mean, if it comes to that... I really hope not. It would make it 100x harder to make the case to my company if I start telling them that I couldn't produce a working demo without forking the language :/

Jun 17
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 17 June 2019 at 09:47:30 UTC, Manu wrote:
 On Mon., 17 Jun. 2019, 7:25 pm Kagamin via Digitalmars-d, < 
 digitalmars-d puremagic.com> wrote:

 On Monday, 17 June 2019 at 01:26:53 UTC, Manu wrote:
 Declare `s` as shared and it will work.
`s` is not declared as shared... nothing is ever declared as shared, that's why the current design is completely useless.
Rewamping the language is too disrupting and I don't think it can be good, you better fork it.
I mean, if it comes to that... I really hope not. It would make it 100x harder to make the case to my company if I start telling them that I couldn't produce a working demo without forking the language :/
That is all moot if mutable promotes to shared under "New Shared"™, I don't think anybody disagrees that qualified local functions should work. It would be useful to decouple this from the the current discussion to avoid derailing.
Jun 17
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 17.06.19 12:18, Nicholas Wilson wrote:
 
 That is all moot if mutable promotes to shared under "New Shared"™,
I was discussing the current language. And even then, with old shared as well as new shared, my interpretation is the correct one. What "New Shared" it is actually trying to do is to introduce a new `threadsafe` qualifier. This doesn't conflict with `shared`. Roughly speaking, `threadsafe` is to unshared and `shared` what `const` is to mutable and `immutable`. threadsafe(int) <- inaccessible threadsafe(shared(int)) <- same as shared(int), accessible The "New Shared" vision is to remove type qualifier support for `shared` and to instead move `shared` into druntime in a restricted form. (Last time I discussed this with Manu, I believe he was adamant that the language shouldn't distinguish between shared and unshared data at all, so we had a long unproductive debate.) Then `shared` is repurposed to mean something completely different.
 I don't think anybody disagrees that qualified local functions should 
 work.
Yes, they do. Manu said you shouldn't be able to call an `immutable`-qualified local function! This is extremely weird, because it is always valid to drop a context qualifier!
 It would be useful to decouple this from the the current 
 discussion to avoid derailing.
I think it's related to why Manu is so confused. Basically, there are two cases for local context qualifiers: 1. can implicitly promote mutable variable (`const`, `threadsafe`/"New Shared") 2. cannot implicitly promote mutable variable (`immutable`, (old) `shared`). So far, Manu only understands case 1, because there is a crutch: it can be interpreted by creating a local context struct containing all local variables in the stack frame, of which the local function is a method. But case 2 exists even if we move from `shared` to `threadsafe`. Because Manu only understands case 1, he considers the fact that `shared` -> `threadsafe` /"New Shared" is a move from case 2 to case 1 as evidence that old shared is broken while `threadsafe`/"New Shared" is not. This is the argument that is complete nonsense that I am so annoyed about, because it leads to `immutable` capturing not working for no reason at all! Skip step 1, where you break local function qualifiers.
Jun 17
parent reply Manu <turkeyman gmail.com> writes:
On Tue, Jun 18, 2019 at 5:40 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 17.06.19 12:18, Nicholas Wilson wrote:
 That is all moot if mutable promotes to shared under "New Shared"™,
I was discussing the current language. And even then, with old shared as well as new shared, my interpretation is the correct one. What "New Shared" it is actually trying to do is to introduce a new `threadsafe` qualifier. This doesn't conflict with `shared`. Roughly speaking, `threadsafe` is to unshared and `shared` what `const` is to mutable and `immutable`. threadsafe(int) <- inaccessible threadsafe(shared(int)) <- same as shared(int), accessible
If this threadsafe concept is something that exists, I am not a party to any such conversation. There was some discussion about promotion possibility (but that never talked about 'threadsafe'), and that was universally rejected. Making shared inhibit read/write needs to happen regardless of anything else to work according to current rules.
 The "New Shared" vision is to remove type qualifier support for `shared`
 and to instead move `shared` into druntime in a restricted form. (Last
 time I discussed this with Manu, I believe he was adamant that the
 language shouldn't distinguish between shared and unshared data at all,
 so we had a long unproductive debate.)
 Then `shared` is repurposed to mean something completely different.
I don't know what you're talking about. Shared needs to have read/write access removed... I feel like that was universally agreed.
From there, I can get to work.
 I don't think anybody disagrees that qualified local functions should
 work.
Yes, they do. Manu said you shouldn't be able to call an `immutable`-qualified local function! This is extremely weird, because it is always valid to drop a context qualifier!
 It would be useful to decouple this from the the current
 discussion to avoid derailing.
I think it's related to why Manu is so confused.
I don't think I'm as confused as you'd like to think. I understand I'm proposing to obliterate a thing. It's a complex and surprising thing, and as far as I can tell, it's useless, and only a point of friction and bugs. Can you show how it's useful?
 Basically, there are two cases for local context qualifiers:

 1. can implicitly promote mutable variable (`const`, `threadsafe`/"New
 Shared")
 2. cannot implicitly promote mutable variable (`immutable`, (old) `shared`).

 So far, Manu only understands case 1, because there is a crutch: it can
 be interpreted by creating a local context struct containing all local
 variables in the stack frame, of which the local function is a method.
 But case 2 exists even if we move from `shared` to `threadsafe`. Because
 Manu only understands case 1, he considers the fact that `shared` ->
 `threadsafe` /"New Shared" is a move from case 2 to case 1 as evidence
 that old shared is broken while `threadsafe`/"New Shared" is not.

 This is the argument that is complete nonsense that I am so annoyed
 about, because it leads to `immutable` capturing not working for no
 reason at all! Skip step 1, where you break local function qualifiers.
Show me one example of an immutable local function in the wild; explain how it's useful? It's a meaningless concept; how can a functions capture be immutable? We have no language to qualify the capture, and certainly not immutable; the callstack is mutable. But I said before, solve however you like.
Jun 17
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 00:42, Manu wrote:
 On Tue, Jun 18, 2019 at 5:40 AM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 17.06.19 12:18, Nicholas Wilson wrote:
 That is all moot if mutable promotes to shared under "New Shared"™,
I was discussing the current language. And even then, with old shared as well as new shared, my interpretation is the correct one. What "New Shared" it is actually trying to do is to introduce a new `threadsafe` qualifier. This doesn't conflict with `shared`. Roughly speaking, `threadsafe` is to unshared and `shared` what `const` is to mutable and `immutable`. threadsafe(int) <- inaccessible threadsafe(shared(int)) <- same as shared(int), accessible
If this threadsafe concept is something that exists, I am not a party to any such conversation. ...
I think this is what you wanted the `shared`-qualifier to mean. I think `threadsafe` is the canonical descriptor for the concept.
 There was some discussion about promotion possibility (but that never
 talked about 'threadsafe'), and that was universally rejected.
 ...
A large part of the rejection was probably because `shared` is an existing qualifier and you argued as if it didn't already have a meaning. It's much easier to argue that a new primitive is useful if you don't need to argue at the same time that an existing primitive should be removed (people feel loss more strongly than gain ;)). I think it is more productive to keep the qualifiers separate for the time being and then at some point in the discussion decide that perhaps yes, we can drop `shared` as a qualifier and move a restricted version of it into druntime.
 Making shared inhibit read/write needs to happen regardless of
 anything else to work according to current rules.
 
 The "New Shared" vision is to remove type qualifier support for `shared`
 and to instead move `shared` into druntime in a restricted form. (Last
 time I discussed this with Manu, I believe he was adamant that the
 language shouldn't distinguish between shared and unshared data at all,
 so we had a long unproductive debate.)
 Then `shared` is repurposed to mean something completely different.
I don't know what you're talking about. Shared needs to have read/write access removed... I feel like that was universally agreed.
threadsafe(unshared) needs to have read/write access removed (completely, UB if you access with casts). shared needs to have unsynchronized read/write access removed. There are multiple ways to go about this: 1. shared data cannot be accessed directly, all accesses need to go through special druntime functions. 2. direct accesses to shared data are atomic/sequentially consistent, other kinds of consistency guarantees are supported with special druntime functions. 3. direct accesses to shared data have low consistency guarantees (some sort of lowest-common-denominator of what's provided by weak memory models), if you want more consistency guarantees, you need to use special druntime functions. Why is 1 better than 2 or 3? (I think Walter is actually leaning towards 2 at the moment.)
From there, I can get to work.
 I don't think anybody disagrees that qualified local functions should
 work.
Yes, they do. Manu said you shouldn't be able to call an `immutable`-qualified local function! This is extremely weird, because it is always valid to drop a context qualifier!
 It would be useful to decouple this from the the current
 discussion to avoid derailing.
I think it's related to why Manu is so confused.
I don't think I'm as confused as you'd like to think.
I'd have liked to think you are exactly as confused as someone who argues that useful closure capturing behavior is a blocker for a parallel `foreach` implementation, but you have been slowly backing down on that claim, so it is indeed likely that you are less confused now than you seemed to be when I wrote that post. :)
 I understand I'm proposing to obliterate a thing. It's a complex and
surprising thing,
 and as far as I can tell, it's useless, and only a point of friction
 and bugs. Can you show how it's useful?
 ...
The language needs a way to tell what kind of semantics are expected for variable accesses. The usefulness is that you (and the compiler implementer) know which code is guaranteed to keep working across compiler versions and which code is not guaranteed to work and you just got lucky and it might break on the next compiler update, leaving you with some weird concurrency bug that only happens on Tuesdays in release builds. You absolutely need some notion of shared variable. Unshared is optional, but why pessimize thread-local code, after having gone through the effort to make globals thread-local by default? It is possible that a shared _qualifier_ is ultimately too expensive and it should be replaced by some _special_ druntime constructs (which also has non-trivial implications, and needs to be designed properly). The language will however in each case need know about shared and unshared memory locations.
 Basically, there are two cases for local context qualifiers:

 1. can implicitly promote mutable variable (`const`, `threadsafe`/"New
 Shared")
 2. cannot implicitly promote mutable variable (`immutable`, (old) `shared`).

 So far, Manu only understands case 1, because there is a crutch: it can
 be interpreted by creating a local context struct containing all local
 variables in the stack frame, of which the local function is a method.
 But case 2 exists even if we move from `shared` to `threadsafe`. Because
 Manu only understands case 1, he considers the fact that `shared` ->
 `threadsafe` /"New Shared" is a move from case 2 to case 1 as evidence
 that old shared is broken while `threadsafe`/"New Shared" is not.

 This is the argument that is complete nonsense that I am so annoyed
 about, because it leads to `immutable` capturing not working for no
 reason at all! Skip step 1, where you break local function qualifiers.
Show me one example of an immutable local function in the wild; explain how it's useful? It's a meaningless concept; how can a functions capture be immutable? We have no language to qualify the capture, and certainly not immutable; the callstack is mutable.
Answered in previous post.
 But I said before, solve however you like.
 
I hope so.
Jun 17
parent reply Manu <turkeyman gmail.com> writes:
On Tue, Jun 18, 2019 at 10:00 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 18.06.19 00:42, Manu wrote:
 On Tue, Jun 18, 2019 at 5:40 AM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 17.06.19 12:18, Nicholas Wilson wrote:
 That is all moot if mutable promotes to shared under "New Shared"™,
I was discussing the current language. And even then, with old shared as well as new shared, my interpretation is the correct one. What "New Shared" it is actually trying to do is to introduce a new `threadsafe` qualifier. This doesn't conflict with `shared`. Roughly speaking, `threadsafe` is to unshared and `shared` what `const` is to mutable and `immutable`. threadsafe(int) <- inaccessible threadsafe(shared(int)) <- same as shared(int), accessible
If this threadsafe concept is something that exists, I am not a party to any such conversation. ...
I think this is what you wanted the `shared`-qualifier to mean. I think `threadsafe` is the canonical descriptor for the concept.
Okay. To be clear, I'm not talking about something else, I'm talking about shared exclusively here.
 There was some discussion about promotion possibility (but that never
 talked about 'threadsafe'), and that was universally rejected.
 ...
A large part of the rejection was probably because `shared` is an existing qualifier and you argued as if it didn't already have a meaning. It's much easier to argue that a new primitive is useful if you don't need to argue at the same time that an existing primitive should be removed (people feel loss more strongly than gain ;)). I think it is more productive to keep the qualifiers separate for the time being and then at some point in the discussion decide that perhaps yes, we can drop `shared` as a qualifier and move a restricted version of it into druntime.
 Making shared inhibit read/write needs to happen regardless of
 anything else to work according to current rules.

 The "New Shared" vision is to remove type qualifier support for `shared`
 and to instead move `shared` into druntime in a restricted form. (Last
 time I discussed this with Manu, I believe he was adamant that the
 language shouldn't distinguish between shared and unshared data at all,
 so we had a long unproductive debate.)
 Then `shared` is repurposed to mean something completely different.
I don't know what you're talking about. Shared needs to have read/write access removed... I feel like that was universally agreed.
threadsafe(unshared) needs to have read/write access removed (completely, UB if you access with casts). shared needs to have unsynchronized read/write access removed. There are multiple ways to go about this: 1. shared data cannot be accessed directly, all accesses need to go through special druntime functions. 2. direct accesses to shared data are atomic/sequentially consistent, other kinds of consistency guarantees are supported with special druntime functions. 3. direct accesses to shared data have low consistency guarantees (some sort of lowest-common-denominator of what's provided by weak memory models), if you want more consistency guarantees, you need to use special druntime functions.
1: Simplest and least opinionated approach. 2: Is only possible for exactly one type; `int`, and breaks down if there are 2 int's. Anything else is #1 anyway. Just let the user call atomicInc() for clarity rather than pretending `++` is safe. 3: What does that mean in practise? Any implementation where you can access data members directly is no better than our naming conventions in C++.
 Why is 1 better than 2 or 3? (I think Walter is actually leaning towards
 2 at the moment.)
Start with 1, reach for 2 as an expansion. 1 is subset of 2.
From there, I can get to work.
 I don't think anybody disagrees that qualified local functions should
 work.
Yes, they do. Manu said you shouldn't be able to call an `immutable`-qualified local function! This is extremely weird, because it is always valid to drop a context qualifier!
 It would be useful to decouple this from the the current
 discussion to avoid derailing.
I think it's related to why Manu is so confused.
I don't think I'm as confused as you'd like to think.
I'd have liked to think you are exactly as confused as someone who argues that useful closure capturing behavior is a blocker for a parallel `foreach` implementation, but you have been slowly backing down on that claim, so it is indeed likely that you are less confused now than you seemed to be when I wrote that post. :)
I haven't backed down. Large elements of my suggestion have been misunderstood (or were omitted from my suggestion). I know I can't convince you, and I don't want to try. You have your own ideas, I'll see where it goes.
 I understand I'm proposing to obliterate a thing. It's a complex and
surprising thing,
 and as far as I can tell, it's useless, and only a point of friction
 and bugs. Can you show how it's useful?
 ...
The language needs a way to tell what kind of semantics are expected for variable accesses. The usefulness is that you (and the compiler implementer) know which code is guaranteed to keep working across compiler versions and which code is not guaranteed to work and you just got lucky and it might break on the next compiler update, leaving you with some weird concurrency bug that only happens on Tuesdays in release builds.
I don't know what you're saying here.
 You absolutely need some notion of shared variable. Unshared is
 optional, but why pessimize thread-local code, after having gone through
 the effort to make globals thread-local by default? It is possible that
 a shared _qualifier_ is ultimately too expensive and it should be
 replaced by some _special_ druntime constructs (which also has
 non-trivial implications, and needs to be designed properly). The
 language will however in each case need know about shared and unshared
 memory locations.
Again, I have no idea what you're talking about here.
 It took a mammoth effort to arrive at agreement that shared can't
 read/write. That's uncontroversial, trivial to implement, and won't
 break anything that's not already critically broken.
 Can we start there?
 ...
See my last post. I don't think we are there yet. Walter is actually leaning towards allowing default access on `shared` variables with sequential consistency guarantees.
No such thing exists, unless there is exactly one integer.
 I don't think he is on board with the redefinition of `shared` to
 `threadsafe`. For `threadsafe`, default access on unshared data must
 obviously be disallowed. For `shared` this is less obvious if you don't
 already assume it has the new meaning.
We can drop this 'threadsafe' thing you have going on. I yielded on that discussion months ago, we need shared to not be broken first.
Jun 17
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 02:30, Manu wrote:
 On Tue, Jun 18, 2019 at 10:00 AM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 ...

 threadsafe(unshared) needs to have read/write access removed
 (completely, UB if you access with casts).

 shared needs to have unsynchronized read/write access removed. There are
 multiple ways to go about this:

 1. shared data cannot be accessed directly, all accesses need to go
 through special druntime functions.

 2. direct accesses to shared data are atomic/sequentially consistent,
 other kinds of consistency guarantees are supported with special
 druntime functions.

 3. direct accesses to shared data have low consistency guarantees (some
 sort of lowest-common-denominator of what's provided by weak memory
 models), if you want more consistency guarantees, you need to use
 special druntime functions.
1: Simplest and least opinionated approach. 2: Is only possible for exactly one type; `int`, and breaks down if there are 2 int's. Anything else is #1 anyway. Just let the user call atomicInc() for clarity rather than pretending `++` is safe. 3: What does that mean in practise? Any implementation where you can access data members directly is no better than our naming conventions in C++.
 Why is 1 better than 2 or 3? (I think Walter is actually leaning towards
 2 at the moment.)
Start with 1, reach for 2 as an expansion. 1 is subset of 2. ...
This makes sense. 1. is clearly the way to go. I think your "3:" is actually the strongest point. If some atomic accesses are transparent, it is way too easy to accidentally treat a `shared` variable as if it was unshared.
 ...
 I think it's related to why Manu is so confused.
I don't think I'm as confused as you'd like to think.
I'd have liked to think you are exactly as confused as someone who argues that useful closure capturing behavior is a blocker for a parallel `foreach` implementation, but you have been slowly backing down on that claim, so it is indeed likely that you are less confused now than you seemed to be when I wrote that post. :)
I haven't backed down.
You said something to the effect of "fix it like you want". If you maintain that qualified capturing is a blocker for parallel foreach (with opApply taking a delegate with a `shared` context). I'll maintain that you are confused. Qualified capturing is not a blocker for such a parallel foreach in any way, shape or form.
 Large elements of my suggestion have been
 misunderstood (or were omitted from my suggestion).
 I know I can't convince you, and I don't want to try. You have your
 own ideas, I'll see where it goes.
 ...
Maybe you can show a full code example where you are capturing an unqualified `int` (then typed as `shared(int)`, because the qualifier is applied to the entire stack frame) in the body of an foreach with an opApply that takes a `shared` delegate, that would actually compile and run with the semantics you have in mind. If you do that, one of the following things will happen: 1. I extend your example such that it leads to shared/unshared aliasing. 2. I accept that your example can be made to compile, but I will explain that it can be made to compile just as easily with qualified capturing. 3. I concede the point and apologize.
 I understand I'm proposing to obliterate a thing. It's a complex and
surprising thing,
 and as far as I can tell, it's useless, and only a point of friction
 and bugs. Can you show how it's useful?
 ...
...
I don't know what you're saying here. ...
My second guess for the "obliterated thing" you were referring to is qualified capturing (the first guess was the existing `shared` qualifier). If I guessed correctly this time, you were not making yourself very clear. This is not complex or surprising at all!
 I don't think he is on board with the redefinition of `shared` to
 `threadsafe`. For `threadsafe`, default access on unshared data must
 obviously be disallowed. For `shared` this is less obvious if you don't
 already assume it has the new meaning.
We can drop this 'threadsafe' thing you have going on. I yielded on that discussion months ago, we need shared to not be broken first.
That's good to know. I considered it as an alternative because Nicholas brought it up and you referred to shared being "like const".
Jun 17
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 17 June 2019 at 09:47:30 UTC, Manu wrote:
 I mean, if it comes to that... I really hope not.
 It would make it 100x harder to make the case to my company if 
 I start
 telling them that I couldn't produce a working demo without 
 forking the
 language :/
Is parallell for mandatory though? In C++ it is an add-on, isn't it? Anyway, I am back to toying with creating a unicode-syntax. Stuff like inheriting structs could easily be handled on the syntactical level, so it does not require a fork, but things like parallell for has to be done at the semantic level.
Jun 17
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 17 June 2019 at 10:20:29 UTC, Ola Fosheim Grøstad 
wrote:
 Is parallell for mandatory though?
For what Manu wants to do yes.
 In C++ it is an add-on, isn't it?
It's a standard library function. The problem is that parallel foreach uses opApply, and the lambda that it takes is not qualified as shared, which is a problem if you want to try and use shared with it. This is compounded by the fact that qualifiers don't work at all (i.e. completely broken) on local functions, which the foreach body passed to opApply is. There is also the issue that access through the (once its fixed) qualified captured context, then cannot implicitly convert to shared (as oppose to const since everything will implicitly convert to const). This will be alleviated under the new shared.
Jun 17
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 17 June 2019 at 11:24:59 UTC, Nicholas Wilson wrote:
 On Monday, 17 June 2019 at 10:20:29 UTC, Ola Fosheim Grøstad
 In C++ it is an add-on, isn't it?
It's a standard library function.
Ah yes, in C++17 it is. I was thinking of OpenMP, but I guess the OMP pragma can be wrapped up in the std lib API now.
 There is also the issue that access through the (once its 
 fixed) qualified captured context, then cannot implicitly 
 convert to shared (as oppose to const since everything will 
 implicitly convert to const). This will be alleviated under the 
 new shared.
Thanks for the explanation. So my understanding is that this will all be fixed when the semantics of "shared" has been finally decided on?
Jun 17
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 17 June 2019 at 11:38:59 UTC, Ola Fosheim Grøstad 
wrote:
 There is also the issue that access through the (once its 
 fixed) qualified captured context, then cannot implicitly 
 convert to shared (as oppose to const since everything will 
 implicitly convert to const). This will be alleviated under 
 the new shared.
Thanks for the explanation. So my understanding is that this will all be fixed when the semantics of "shared" has been finally decided on?
The qualified context thing should be fixed ASAP (unfortunately the compiler rabbit hole is rather deep...), since it is objectively broken, not just with shared, but const and immutable as well. The nicely capturing unshared locals as shared within the local function will indeed have to wait until shared is respecified. The way I think it will go is to add a `-preview=shared` and put the changes behind that. Probably starting with disabling read and write access to shared variables without casting as this was pretty much universally agreed upon at dconf as a necessary, if not sufficient, condition for thread safety. Manu (and maybe others) will then evaluate the model and see what breaks or is missing and then we repeat the design cycle. There is also a (eventually series of) dlang.org PRs to define a memory model which will define more of the semantics of shared and multithreading.
Jun 17
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 17 June 2019 at 11:55:07 UTC, Nicholas Wilson wrote:
 starting with disabling read and write access to shared 
 variables without casting as this was pretty much universally 
 agreed upon at dconf as a necessary, if not sufficient, 
 condition for thread safety.
+1 This is the way to go! Please link me to your work in progress I'll try to help.
Jun 17
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On Mon, Jun 17, 2019 at 8:25 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 17 June 2019 at 09:47:30 UTC, Manu wrote:
 I mean, if it comes to that... I really hope not.
 It would make it 100x harder to make the case to my company if
 I start
 telling them that I couldn't produce a working demo without
 forking the
 language :/
Is parallell for mandatory though? In C++ it is an add-on, isn't it?
Define 'mandatory'? ...because we currently use C++. But it would be a powerful enabler, and one of the very specific challenges we have with C++ implementations. I mean, our project needs parallel-for without any question, and the smooth implementation that D teases us with is very attractive! If I hack some crappy implementation using meta and lambdas, then I have written EXACTLY what you write in C++, and there is no case to be made. I need to show advantages of D, but parity. I have an unusual opportunity to make a very strong case for D, but I can only do that with a great showcase. I spent hours in a meeting the other day arguing various terribly ways to implement it in C++, and the take-away from the meeting was basically "well I guess we have to choose this crappy compromise", which mostly revolve around naming conventions. Everything I need to make a strong case revolves around shared; and since nobody on this forum seems to know what to do with shared (we've been arguing about it and doing exactly nothing for the 10 years I've been watching), I think I'm uniquely positioned to move the bar forward here and make something that's actually useful, and demonstrated what shared is actually for. Reality is, most people don't write highly multi-threaded code, maybe they spin a worker thread or something, but that's not what the future looks like.
 Anyway, I am back to toying with creating a unicode-syntax.
 Stuff like inheriting structs could easily be handled on the
 syntactical level, so it does not require a fork, but things like
 parallell for has to be done at the semantic level.
Inheriting structs doesn't block anything... I just started that thread to re-iterate how unbelievably tired I am of not having that tool. I hate the struct inheritance workaround so much.
Jun 17
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 17 June 2019 at 20:11:32 UTC, Manu wrote:
 I think I'm uniquely positioned to move the bar
 forward here and make something that's actually useful, and
 demonstrated what shared is actually for.
 Reality is, most people don't write highly multi-threaded code, 
 maybe
 they spin a worker thread or something, but that's not what the 
 future
 looks like.
Yes, I don't think many people write highly multi-threaded code. I think the model that is easiest to understand is that you have a queue of work to be done and then just let worker threads grab work from that queue. More complicated structures require a lot of planning and it is difficult to be 100% certain that you never can get a deadlock as the program design evolves over time. I think it is difficult to say what they future looks like, it involves many cores, I think, but perhaps also more local memory per core. So… maybe the actor model will take off at some point, but I don't know. I think the actor model is easier to grasp, so it might win out even if it adds overhead. Right now there seems to be many different hardware mechanisms that provide parallel computation that are rather specialized. SIMD, FPGA, GPU, neural network oriented cores, etc.
Jun 17
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 17.06.19 22:11, Manu wrote:
 
 Everything I need to make a strong case revolves around shared; and
 since nobody on this forum seems to know what to do with shared (we've
 been arguing about it and doing exactly nothing for the 10 years I've
 been watching),
Lack of activity doesn't imply lack of knowledge, it can just be lack of management. There is too little activity for most issues with the language that require: a) breaking a ton of code b) adding a new data qualifier It is worse if a) and b) are combined with a high specification and implementation effort and nobody who can spend enough time on the compiler has an in-depth understanding of what needs to be done in every corner of the language. I know multiple options for what to do, but they involve either a) or b). So do your suggestions. This is why there hasn't been enough activity, it is not a lack of theoretical understanding from every single forum poster.
 I think I'm uniquely positioned to move the bar
 forward here and make something that's actually useful, and
 demonstrated what shared is actually for.
I think you have useful perspectives on the application side and the prioritization of issues to help adoption in your industry, but this does not always translate to useful insights on language design. Our conversations should involve more of you talking about important applications and me talking about language design. Of course, this is all useless unless someone writes DIPs and someone implements them.
Jun 17
parent reply Manu <turkeyman gmail.com> writes:
On Tue, Jun 18, 2019 at 8:35 AM Timon Gehr via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 17.06.19 22:11, Manu wrote:
 Everything I need to make a strong case revolves around shared; and
 since nobody on this forum seems to know what to do with shared (we've
 been arguing about it and doing exactly nothing for the 10 years I've
 been watching),
Lack of activity doesn't imply lack of knowledge, it can just be lack of management. There is too little activity for most issues with the language that require: a) breaking a ton of code b) adding a new data qualifier It is worse if a) and b) are combined with a high specification and implementation effort and nobody who can spend enough time on the compiler has an in-depth understanding of what needs to be done in every corner of the language. I know multiple options for what to do, but they involve either a) or b). So do your suggestions. This is why there hasn't been enough activity, it is not a lack of theoretical understanding from every single forum poster.
It took a mammoth effort to arrive at agreement that shared can't read/write. That's uncontroversial, trivial to implement, and won't break anything that's not already critically broken. Can we start there?
 I think I'm uniquely positioned to move the bar
 forward here and make something that's actually useful, and
 demonstrated what shared is actually for.
I think you have useful perspectives on the application side and the prioritization of issues to help adoption in your industry, but this does not always translate to useful insights on language design. Our conversations should involve more of you talking about important applications and me talking about language design. Of course, this is all useless unless someone writes DIPs and someone implements them.
Just write a patch. I will use a forked language and prove that it works or not. I would gladly never post in this forum again in my life; I hate this process more more than I can describe.
Jun 17
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 01:15, Manu wrote:
 On Tue, Jun 18, 2019 at 8:35 AM Timon Gehr via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 17.06.19 22:11, Manu wrote:
 Everything I need to make a strong case revolves around shared; and
 since nobody on this forum seems to know what to do with shared (we've
 been arguing about it and doing exactly nothing for the 10 years I've
 been watching),
Lack of activity doesn't imply lack of knowledge, it can just be lack of management. There is too little activity for most issues with the language that require: a) breaking a ton of code b) adding a new data qualifier It is worse if a) and b) are combined with a high specification and implementation effort and nobody who can spend enough time on the compiler has an in-depth understanding of what needs to be done in every corner of the language. I know multiple options for what to do, but they involve either a) or b). So do your suggestions. This is why there hasn't been enough activity, it is not a lack of theoretical understanding from every single forum poster.
It took a mammoth effort to arrive at agreement that shared can't read/write. That's uncontroversial, trivial to implement, and won't break anything that's not already critically broken. Can we start there? ...
See my last post. I don't think we are there yet. Walter is actually leaning towards allowing default access on `shared` variables with sequential consistency guarantees. I don't think he is on board with the redefinition of `shared` to `threadsafe`. For `threadsafe`, default access on unshared data must obviously be disallowed. For `shared` this is less obvious if you don't already assume it has the new meaning.
 I think I'm uniquely positioned to move the bar
 forward here and make something that's actually useful, and
 demonstrated what shared is actually for.
I think you have useful perspectives on the application side and the prioritization of issues to help adoption in your industry, but this does not always translate to useful insights on language design. Our conversations should involve more of you talking about important applications and me talking about language design. Of course, this is all useless unless someone writes DIPs and someone implements them.
Just write a patch. I will use a forked language and prove that it works or not. I would gladly never post in this forum again in my life; I hate this process more more than I can describe.
Unfortunately I have another paper deadline in a few weeks, but maybe I can set aside a few weekends this summer to try to fix function qualifiers.
Jun 17
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 18 June 2019 at 00:17:06 UTC, Timon Gehr wrote:
 Unfortunately I have another paper deadline in a few weeks, but 
 maybe I can set aside a few weekends this summer to try to fix 
 function qualifiers.
I intend to beat you to it :) Suleyman Sahmi has located where things go right for the struct case[1], so it shouldn't be too hard to manufacture a fix for the closure case. [1]: https://github.com/dlang/dmd/blob/f455995f234a2091c4482bff222a2977dbfa186b/src/dmd/expressionsem.d#L885-L886
Jun 18
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 14:00, Nicholas Wilson wrote:
 On Tuesday, 18 June 2019 at 00:17:06 UTC, Timon Gehr wrote:
 Unfortunately I have another paper deadline in a few weeks, but maybe 
 I can set aside a few weekends this summer to try to fix function 
 qualifiers.
I intend to beat you to it :) ...
Great. :)
 Suleyman Sahmi has located where things go right for the struct case[1], 
 so it shouldn't be too hard to manufacture a fix for the closure case.
 
 [1]: 
 https://github.com/dlang/dmd/blob/f455995f234a2091c4482bff222a2977dbfa186b/src/dmd/expre
sionsem.d#L885-L886 
 
I believe it might be easier to locate where things go right for `shared` and `immutable` capturing and fix the logic there. I provide tests below. Note that a few missing checks and const-promotions when capturing are not the only issue with nested function qualifiers. If the following isses are fixed, we should be in pretty good shape (but probably I forgot about something): 0. Qualified capturing is not implemented fully correctly, it is particularly bad for `const`. Full test case: void fun(inout(int)*){ int* x; const(int*) cx; immutable(int*) ix; shared(int*) sx; shared(const(int*)) scx; inout(int*) wx; shared(inout(int*)) swx; const(inout(int*)) cwx; shared(const(inout(int*))) scwx; void foo(){ int* x=x; const(int)* cx=cx; // ok immutable(int)* ix=ix; // ok shared(int)* sx=sx; // ok shared(const(int*)) scx=scx; // ok inout(int)* wx=wx; // ok shared(inout(int))* swx=swx; // ok const(inout(int))* cwx=cwx; // ok shared(const(inout(int)))* scwx=scwx; // ok } void fooc()const{ int* x=x; // currently ok, shouldn't compile const(int)* x2=x; // ok const(int)* cx=cx; // ok immutable(int)* ix=ix; // ok shared(int)* sx=sx; // currently ok, shouldn't compile const(shared(int))* sx2=sx; // ok shared(const(int*)) scx=scx; // ok inout(int)* wx=wx; // currently ok, shouldn't compile const(inout(int))* wx2=wx; // ok shared(inout(int))* swx=swx; // currently ok, shouldn't compile shared(const(inout(int)))* swx2=swx; // ok const(inout(int))* cwx=cwx; // ok shared(const(inout(int)))* scwx=scwx; // ok } void fooi()immutable{ //int* x=x; // error, correct //const(int)* cx=cx; // error, correct immutable(int)* ix=ix; // ok //shared(int)* sx=sx; // error, correct //shared(const(int*)) scx=scx; // error, correct //inout(int)* wx=wx; // error, correct //shared(inout(int))* swx=swx; // error, correct //const(inout(int))* cwx=cwx; // error, correct //shared(const(inout(int)))* scwx=scwx; // error, correct } void foos()shared{ //int* x=x; // error, correct //const(int)* cx=cx; // error, correct immutable(int)* ix=ix; // ok shared(int)* sx=sx; // ok shared(const(int*)) scx=scx; // ok //inout(int)* wx=wx; // error, correct //shared(inout(int))* swx=swx; // currently error, should work //const(inout(int))* cwx=cwx; // error, correct //shared(const(inout(int)))* scwx=scwx; // currently error, should work } void foosc()shared const{ //int* x=x; // error, correct //const(int)* cx=cx; // error, correct immutable(int)* ix=ix; // ok //shared(int)* sx=sx; // error, correct //const(shared(int))* sx2=sx; // currently error, should work shared(const(int*)) scx=scx; // ok //inout(int)* wx=wx; // error, correct //const(inout(int))* wx2=wx; // currently error, should work //shared(inout(int))* swx=swx; // error, correct //const(shared(inout(int)))* swx2=swx; // currently error, should work //const(inout(int))* cwx=cwx; // error, correct //shared(const(inout(int)))* scwx=scwx; // currently error, should work } void foow()inout{ int* x=x; // currently ok, shouldn't compile immutable(int)* ix=ix; // ok shared(int)* sx=sx; // currently ok, shouldn't compile inout(int)* wx=wx; // ok shared(inout(int))* swx=swx; // ok const(inout(int))* cwx=cwx; // ok shared(const(inout(int)))* scwx=scwx; // ok } void foosw()shared inout{ //int* x=x; // error, correct immutable(int)* ix=ix; // ok //shared(int)* sx=sx; // error, correct //inout(int)* wx=wx; // error, correct shared(inout(int))* swx=swx; // ok //const(inout(int))* cwx=cwx; // error, correct shared(const(inout(int)))* scwx=scwx; // ok } void fooscw()shared const inout{ //int* x=x; // error, correct immutable(int)* ix=ix; // ok //shared(int)* sx=sx; // error, correct //inout(int)* wx=wx; // error, correct //shared(inout(int))* swx=swx; // error, correct //const(shared(inout(int)))* swx2=swx; // currently error, should compile //const(inout(int))* cwx=cwx; // error, correct shared(const(inout(int)))* scwx=scwx; // ok } } 1. I can break the type system like this: void main(){ int* x=new int; struct S{ int* delegate()pure dg1; int* dg2()pure immutable{ return dg1(); // this shouldn't compile } } // you may think the next line is the problem, but this is actually ok: auto s=immutable(S)(()=>x); immutable(int*) y=s.dg2(); assert(x is y); // mutable/immutable aliasing } The problem is that it is possible to call a immutable(T delegate(Args)). This is wrong. It should only be possible to call a qualified delegate if the delegate context has the respective (or a stronger) qualifier. So const(int* delegate()) dg = ...; dg(); // should be error const(int* delegate()immutable) dg = ...; dg(); // this is fine Note that the obvious idea of saying that the qualifier of the delegate transitively applies to the opaque delegate context does not work, because e.g. the delegate context cannot implicitly pick up a `const`, as the function pointer will access the context in a way that is typed mutable. 2. nested functions of `pure` functions are forced to be `pure`, but this is nonsense, they should just infer purity, like it is done for safe: int foo(){ return 2; } void main()pure{ enum x=(()=>foo())(); // Error: `pure` delegate `tt.main.__lambda1` cannot call impure function `tt.foo` } (I picked this example because here it is obvious that the error is nonsense, but this is a more general problem and not restricted to CTFE.) In contrast, it works perfectly fine for safe: int foo(){ return 2; } void main() safe{ enum x=(()=>foo())(); // ok! } I think it is terrible that the implementations of ` safe` inference and `pure` inference were allowed to diverge at all. 3. The same problem exists for nogc, but the fix is easy. Here it is: https://github.com/dlang/dmd/pull/9922#issuecomment-499544100 (I will create a pull request eventually, unless someone beats me to it. I would still need to write some tests.) 4. nested functions should infer `const`, `immutable`, `shared` and `inout` qualifiers based on the qualifiers of variables that they capture. 5. it should be possible to drop delegate context qualifiers: int delegate()const dgc; int delegate() dgc2=dgc; // this is correctly accepted int delegate()immutable dgi; int delegate() dgi2=dgi; // this is rejected incorrectly int delegate() dgi3=()=>dgi(); // ugly workaround int delegate()shared dgs; int delegate() dgs2=dgs; // this is rejected incorrectly int delegate() dgs3=()=>dgs(); // ugly workaround 6. `inout` should be implemented in a way that prevents scope confusion by design.
Jun 18
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 15:24, Timon Gehr wrote:
 On 18.06.19 14:00, Nicholas Wilson wrote:
 On Tuesday, 18 June 2019 at 00:17:06 UTC, Timon Gehr wrote:
 Unfortunately I have another paper deadline in a few weeks, but maybe 
 I can set aside a few weekends this summer to try to fix function 
 qualifiers.
I intend to beat you to it :) ...
Great. :)
 Suleyman Sahmi has located where things go right for the struct 
 case[1], so it shouldn't be too hard to manufacture a fix for the 
 closure case.

 [1]: 
 https://github.com/dlang/dmd/blob/f455995f234a2091c4482bff222a2977dbfa186b/src/dmd/expre
sionsem.d#L885-L886 
I believe it might be easier to locate where things go right for `shared` and `immutable` capturing and fix the logic there. I provide tests below. Note that a few missing checks and const-promotions when capturing are not the only issue with nested function qualifiers. If the following isses are fixed, we should be in pretty good shape (but probably I forgot about something): ...
In fact I did. 7. Local function calls need to be checked as if they were delegate captures. The general rule is that the code void fun(){ void bar()qualifiers1{ ... } void foo()qualifiers2{ bar(); } } should compile exactly if the following code compiles: void fun(){ void bar()qualifiers1{ ... } qualifiers2 dg=&bar; // conversion always ok void foo()qualifiers2{ dg(); // capturing of variable 'dg' always ok, but call may not be } } This yields a simple way to implement the check. Tests: void fun(inout(int)*){ void bar(){} void barc()const{} void bari()immutable{} void bars()shared{} void barsc()shared const{} void barw()inout{} void barsw()shared inout{} void barcw()const inout{} void barscw()shared const inout{} void foo(){ bar(); // ok barc(); // ok bari(); // ok bars(); // ok barsc(); // ok barsw(); // ok barcw(); // ok barscw(); // ok } void fooc()const{ bar(); // currently ok, shouldn't compile barc(); // ok bari(); // ok bars(); // currently ok, shouldn't compile barsc(); // ok barsw(); // currently ok, shouldn't compile barcw(); // ok barscw(); // ok } void fooi()immutable{ bar(); // currently ok, shouldn't compile barc(); // currently ok, shouldn't compile bari(); // ok bars(); // currently ok, shouldn't compile barsc(); // currently ok, shouldn't compile barsw(); // currently ok, shouldn't compile barcw(); // currently ok, shouldn't compile barscw(); // currently ok, shouldn't compile } void foos()shared{ bar(); // currently ok, shouldn't compile barc(); // currently ok, shouldn't compile bari(); // ok bars(); // ok barsc(); // ok barsw(); // ok barcw(); // currently ok, shouldn't compile barscw(); // ok } void foosc()shared const{ bar(); // currently ok, shouldn't compile barc(); // currently ok, shouldn't compile bari(); // ok bars(); // currently ok, shouldn't compile barsc(); // ok barsw(); // currently ok, shouldn't compile barcw(); // currently ok, shouldn't compile barscw(); // ok } void foow()inout{ bar(); // currently ok, shouldn't compile barc(); // currently ok, shouldn't compile bari(); // ok bars(); // currently ok, shouldn't compile barsc(); // currently ok, shouldn't compile barsw(); // ok barcw(); // ok barscw(); // ok } void foosw()shared inout{ bar(); // currently ok, shouldn't compile barc(); // currently ok, shouldn't compile bari(); // ok bars(); // currently ok, shouldn't compile barsc(); // currently ok, shouldn't compile barsw(); // ok barcw(); // currently ok, shouldn't compile barscw(); // ok } void fooscw()shared const inout{ bar(); // currently ok, shouldn't compile barc(); // currently ok, shouldn't compile bari(); // ok bars(); // currently ok, shouldn't compile barsc(); // currently ok, shouldn't compile barsw(); // currently ok, shouldn't compile barcw(); // currently ok, shouldn't compile barscw(); // ok } }
Jun 18
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 15:24, Timon Gehr wrote:
 On 18.06.19 14:00, Nicholas Wilson wrote:
 On Tuesday, 18 June 2019 at 00:17:06 UTC, Timon Gehr wrote:
 Unfortunately I have another paper deadline in a few weeks, but maybe 
 I can set aside a few weekends this summer to try to fix function 
 qualifiers.
I intend to beat you to it :) ...
Great. :)
 Suleyman Sahmi has located where things go right for the struct 
 case[1], so it shouldn't be too hard to manufacture a fix for the 
 closure case.

 [1]: 
 https://github.com/dlang/dmd/blob/f455995f234a2091c4482bff222a2977dbfa186b/src/dmd/expre
sionsem.d#L885-L886 
I believe it might be easier to locate where things go right for `shared` and `immutable` capturing and fix the logic there. I provide tests below. Note that a few missing checks and const-promotions when capturing are not the only issue with nested function qualifiers. If the following isses are fixed, we should be in pretty good shape (but probably I forgot about something): ...
Jun 18
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 18.06.19 15:37, Timon Gehr wrote:
 ...
(Sorry, thunderbird hiccup.)
Jun 20
prev sibling parent M.M. <matus email.cz> writes:
On Monday, 17 June 2019 at 23:15:39 UTC, Manu wrote:
 Just write a patch. I will use a forked language and prove that 
 it works or not. I would gladly never post in this forum again 
 in my life; I hate this process more more than I can describe.
Reading your in-depth discussion with Timon is such a great learning process for me. I'd guess that it's also important for other people who care about Dlang to follow your discussion. But I'd understand if you fly to Zurich to meet Timon, instead of posting here on the forum.
Jun 18
prev sibling parent reply Kagamin <spam here.lot> writes:
On Monday, 17 June 2019 at 20:11:32 UTC, Manu wrote:
 Everything I need to make a strong case revolves around shared; 
 and
 since nobody on this forum seems to know what to do with shared 
 (we've
 been arguing about it and doing exactly nothing for the 10 
 years I've
 been watching)
shared works fine, your irrational unwillingness to work with it is not a valid reason to introduce an abysmal change to the language.
Jun 18
parent Manu <turkeyman gmail.com> writes:
On Tue, Jun 18, 2019 at 7:40 PM Kagamin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 17 June 2019 at 20:11:32 UTC, Manu wrote:
 Everything I need to make a strong case revolves around shared;
 and
 since nobody on this forum seems to know what to do with shared
 (we've
 been arguing about it and doing exactly nothing for the 10
 years I've
 been watching)
shared works fine, your irrational unwillingness to work with it is not a valid reason to introduce an abysmal change to the language.
shared int x; x = 10; // <- not an error; Don't tell me shared works fine.
Jun 18
prev sibling parent Manu <turkeyman gmail.com> writes:
On Fri, Jun 14, 2019 at 11:13 PM Manu <turkeyman gmail.com> wrote:
 struct S
 {
   void method() shared;
 }

 void test()
 {
   S s;
   void localFun() shared
   {
     s.method(); // <- s must transitively receive const from the
 context pointer, otherwise this doesn't work
   }
 }
** s must transitively receive *shared* from the context
Jun 15
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Friday, 14 June 2019 at 18:51:38 UTC, Manu wrote:
 void test()
 {
   int x;
   void fun() const
   {
     pragma(msg, typeof(x)); // should print `const(int)` because
 typeof(const(Context*).x) == const(int), but it incorrectly 
 prints
 `int`
     ++x; // <- should be an error, but the context pointer is 
 not
 const, so this compiles!
   }
 }

 The context pointer is missing the qualifier.
It works for immutable qualifier, maybe type check there is too simple and fails to address const.
Jun 15
prev sibling parent reply Kagamin <spam here.lot> writes:
On Friday, 14 June 2019 at 18:51:38 UTC, Manu wrote:
 The context pointer is missing the qualifier.
I think it can be attributed to bug 1983, which Walter didn't want to fix, so anything that relies on it won't work properly any time soon.
Jun 15
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.06.19 19:00, Kagamin wrote:
 On Friday, 14 June 2019 at 18:51:38 UTC, Manu wrote:
 The context pointer is missing the qualifier.
I think it can be attributed to bug 1983,
I don't think this is the case, there are just a lot of different bugs around the same general theme because it was never properly implemented.
 which Walter didn't want to fix,
Do you have a link?
 so anything that relies on it won't work properly any time soon.
I wasn't able to find a pull request with the correct fix.
Jun 15
parent Kagamin <spam here.lot> writes:
On Sunday, 16 June 2019 at 01:16:24 UTC, Timon Gehr wrote:
 which Walter didn't want to fix,
Do you have a link?
No that's from memory.
 so anything that relies on it won't work properly any time 
 soon.
I wasn't able to find a pull request with the correct fix.
https://github.com/dlang/dmd/pull/2130 probably this
Jun 16