www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Attribute transference from callbacks?

reply Manu <turkeyman gmail.com> writes:
I've been trying out an API strategy to use sink functions a lot more to
hoist the item allocation/management logic to the user and out from the
work routines. This idea has received a lot of attention, and I heard a lot
of people saying this was a planned direction to move in phobos and friends.

void parseThings(string s, void delegate(Thing) sink)
{
  //...
  sink(Thing());
  //...
}

We have a serious problem though; a function that receives a sink function
must transfer the attributes from the sink function to itself, and we have
no language to do that...
What are the leading strategies that have been discussed to date? Is there
a general 'plan'?
Restriction attributes like pure/nothrow/ nogc/safe, etc need some
expression to operate in a similar way to; where the attribute-ness of
function matches the attributes of its delegate argument(/s).

Here's the stupidest idea ever: expand inout to take an argument...

void parseThings(string s, void delegate(Thing)  inout(nothrow)
inout( nogc) sink) inout(nothrow) inout( nogc)
{
  //...
  sink(Thing());
  //...
}

Surely people have had better ideas? But you get the point, and this is
basically essential to make the 'sink' pattern work at all in D.

No, no templates; it's not right to generate multiple copies of identical
functions just because something it CALLS would transfer an attribute. The
function itself is identical no matter the state of any attribute
transference, and so this isn't a template problem; it's a pattern matching
problem.
Dec 07
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
My solution I have proposed plenty, is attribute invalidation.

So the caller sees a more restricted set, depending upon the callback 
passed in.

However Timon didn't like that, thought it wasn't good enough.
Dec 07
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12/8/24 06:57, Richard (Rikki) Andrew Cattermole wrote:

 My solution I have proposed plenty, is attribute invalidation.
 
 So the caller sees a more restricted set, depending upon the callback 
 passed in.
 
 However Timon didn't like that, thought it wasn't good enough.
 


It is not.

Completeness: This proposal cannot express even one of the simplest 
forms of composition, function composition. It also will not compose in 
practice.

Soundness: This proposal changes the meaning of qualified delegates 
everywhere in the language and invalidates a lot of existing type system 
invariants. It will be very tricky to maintain both 
backwards-compatibility with plain attributes and soundness.

In practice, it will be complicated, very incomplete, and unsound, like 
`inout`.
Dec 08
prev sibling parent Manu <turkeyman gmail.com> writes:
On Sun, 8 Dec 2024 at 16:01, Richard (Rikki) Andrew Cattermole via
Digitalmars-d <digitalmars-d puremagic.com> wrote:


 My solution I have proposed plenty, is attribute invalidation.

 So the caller sees a more restricted set, depending upon the callback
 passed in.

 However Timon didn't like that, thought it wasn't good enough.


I agree with Timon, that sounds terrible.
Dec 08
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/8/24 06:54, Manu wrote:

 Here's the stupidest idea ever: expand inout to take an argument...
 
 void parseThings(string s, void delegate(Thing) inout(nothrow) 
 inout( nogc) sink) inout(nothrow) inout( nogc)
 {
    //...
    sink(Thing());
    //...
 }
 ...


Issue with this is you will still not know which `inout`s match up. This 
is already an issue with the existing `inout`, which just arbitrarily 
selects a convention. In particular, it does not really work with 
higher-order functions the way you'd need it to here.


 Surely people have had better ideas? But you get the point, and this is 
 basically essential to make the 'sink' pattern work at all in D.
 
 No, no templates; it's not right to generate multiple copies of identical
functions just because something it CALLS would transfer an attribute. The
function itself is identical no matter the state of any attribute transference,
and so this isn't a template problem; it's a pattern matching problem.


Well, it can be seen as a homogeneous vs heterogeneous compilation 
problem. The attributes can still act a bit like template parameters, 
but only one instance must be created that works for all of them. It's 
sometimes called parametric polymorphism.
Dec 08
parent reply Manu <turkeyman gmail.com> writes:
On Mon, 9 Dec 2024 at 01:51, Timon Gehr via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 12/8/24 06:54, Manu wrote:

 Here's the stupidest idea ever: expand inout to take an argument...

 void parseThings(string s, void delegate(Thing) inout(nothrow)
 inout( nogc) sink) inout(nothrow) inout( nogc)
 {
    //...
    sink(Thing());
    //...
 }
 ...


 Issue with this is you will still not know which `inout`s match up. This
 is already an issue with the existing `inout`, which just arbitrarily
 selects a convention. In particular, it does not really work with
 higher-order functions the way you'd need it to here.


Yes, this is obviously an issue, and we have it in D comprehensively; it's
the main issue with our 'safe' stuff too; where rust has explicit lifetime
attribution... it's essentially the same problem all round; it needs tags
that specify things which are associated.
In lieu of that though, I would say, if there are multiple things marked
inout(...) in the way I proposed, you would assume the most pessimistic
from the set of possibilities is supplied.

void inheritAttribs(string s, void delegate() inout(nothrow) inout( nogc)
fun_1, void delegate() inout(nothrow) inout( nogc) fun_2) inout(nothrow)
inout( nogc)
{
  //...
  fun_1();
  fun_2();
  //...
}

In this case, `inheritAttribs` would only be nothrow in the event BOTH
fun_1 and fun_2 are nothrow, likewise for nogc...



 Surely people have had better ideas? But you get the point, and this is

 basically essential to make the 'sink' pattern work at all in D.

 No, no templates; it's not right to generate multiple copies of


 identical functions just because something it CALLS would transfer an
 attribute. The function itself is identical no matter the state of any
 attribute transference, and so this isn't a template problem; it's a
 pattern matching problem.

 Well, it can be seen as a homogeneous vs heterogeneous compilation
 problem. The attributes can still act a bit like template parameters,
 but only one instance must be created that works for all of them. It's
 sometimes called parametric polymorphism.


Right. Do you have examples of this from other languages and how it's
expressed? Lifetime's in Rust are the obvious benchmark, but I don't see
any evidence that D has a taste for this sort of thing.

I do think that inout could work for all attributes the same as inout does
for const (with the same limitations). You could see `inout` as shorthand
for `inout(const)` under my suggestion.
Obviously it must enforce the most restrictive implementation inside the
code; my function `inheritAttribs` above must be nothrow and nogc
internally, but externally it would allow a mapping from one of the 'input'
attributes to the 'output' attribute in a non-templated way.
Dec 08
next sibling parent reply Araq <rumpf_a web.de> writes:
On Monday, 9 December 2024 at 06:47:26 UTC, Manu wrote:

 Right. Do you have examples of this from other languages and 
 how it's expressed? Lifetime's in Rust are the obvious 
 benchmark, but I don't see any evidence that D has a taste for 
 this sort of thing.



Nim uses an `effectsOf` annotation, see 
https://nim-lang.org/docs/manual.html#effect-system-effectsof-annotation

That said, IME effect systems are not worth it and the complexity 
budget is better spent elsewhere.
Dec 08
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 09/12/2024 8:00 PM, Araq wrote:

 On Monday, 9 December 2024 at 06:47:26 UTC, Manu wrote:

 Right. Do you have examples of this from other languages and how it's 
 expressed? Lifetime's in Rust are the obvious benchmark, but I don't 
 see any evidence that D has a taste for this sort of thing.


 
 
 Nim uses an `effectsOf` annotation, see https://nim-lang.org/docs/ 
 manual.html#effect-system-effectsof-annotation
 
 That said, IME effect systems are not worth it and the complexity budget 
 is better spent elsewhere.


I've been thinking of an effect system for D, specifically for 
improvements to shared.

However, I'm not seeing the benefits for this.

For  safe  nogc these are boolean, whereas nothrow is a set, pure is an 
effect. Then there is type qualifiers const, immutable and shared.

Its not one problem, its more like three that we'd like to solve here.
Dec 08
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12/9/24 08:00, Araq wrote:

 On Monday, 9 December 2024 at 06:47:26 UTC, Manu wrote:

 Right. Do you have examples of this from other languages and how it's 
 expressed? Lifetime's in Rust are the obvious benchmark, but I don't 
 see any evidence that D has a taste for this sort of thing.


 
 
 Nim uses an `effectsOf` annotation, see https://nim-lang.org/docs/ 
 manual.html#effect-system-effectsof-annotation
 ...


Does this only work if the callback is directly a parameter?


 That said, IME effect systems are not worth it and the complexity budget 
 is better spent elsewhere.


Well, I think given that we do any kind of parametric polymorphism, 
effect polymorphism is not a big leap anymore.

But sure, another way to get out of this conundrum would be to remove 
the function effect annotations completely, though I think some people 
are getting some mileage out of them and would like to be able to use 
them properly with callbacks instead.
Dec 10
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/9/24 07:47, Manu wrote:

 On Mon, 9 Dec 2024 at 01:51, Timon Gehr via Digitalmars-d <digitalmars- 
 d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:
 
     On 12/8/24 06:54, Manu wrote:
      > Here's the stupidest idea ever: expand inout to take an argument...
      >
      > void parseThings(string s, void delegate(Thing) inout(nothrow)
      > inout( nogc) sink) inout(nothrow) inout( nogc)
      > {
      >    //...
      >    sink(Thing());
      >    //...
      > }
      > ...
 
     Issue with this is you will still not know which `inout`s match up.
     This
     is already an issue with the existing `inout`, which just arbitrarily
     selects a convention. In particular, it does not really work with
     higher-order functions the way you'd need it to here.
 
 
 Yes, this is obviously an issue, and we have it in D comprehensively; 
 it's the main issue with our 'safe' stuff too; where rust has explicit 
 lifetime attribution... it's essentially the same problem all round; it 
 needs tags that specify things which are associated.
 In lieu of that though, I would say, if there are multiple things marked 
 inout(...) in the way I proposed, you would assume the most pessimistic 
 from the set of possibilities is supplied.
 
 void inheritAttribs(string s, void delegate() inout(nothrow) 
 inout( nogc) fun_1, void delegate() inout(nothrow) inout( nogc) fun_2) 
 inout(nothrow) inout( nogc)
 {
    //...
    fun_1();
    fun_2();
    //...
 }
 
 In this case, `inheritAttribs` would only be nothrow in the event BOTH 
 fun_1 and fun_2 are nothrow, likewise for nogc...
 ...


Well, this is one limitation, but the `inout` on the delegate does not 
even match up with the `inout` on `inheritAttribs` those are already 
different.

This does not compile;

```d
inout(int)* foo(inout(int)* delegate()inout dg,inout(int)* x){
     return x;
}

void main(){
     const(int)* delegate()const dg;
     const(int)* x;
     foo(dg,x); // error
}
```

It seems at least this would need to work.

`inout` is very confusing because different `inout` annotations can get 
conflated or not conflated. It's also the main reason why the 
implementation is unsound, it's not done consistently during type checking.


 
      > Surely people have had better ideas? But you get the point, and
     this is
      > basically essential to make the 'sink' pattern work at all in D.
      >
      > No, no templates; it's not right to generate multiple copies of
     identical functions just because something it CALLS would transfer
     an attribute. The function itself is identical no matter the state
     of any attribute transference, and so this isn't a template problem;
     it's a pattern matching problem.
 
     Well, it can be seen as a homogeneous vs heterogeneous compilation
     problem. The attributes can still act a bit like template parameters,
     but only one instance must be created that works for all of them. It's
     sometimes called parametric polymorphism.
 
 
 Right. Do you have examples of this from other languages and how it's 
 expressed? Lifetime's in Rust are the obvious benchmark, but I don't see 
 any evidence that D has a taste for this sort of thing.
 ...




In Haskell everything is polymorphic by default with implicit universal 
quantification over lower-case type parameters.


 I do think that inout could work for all attributes the same as inout 
 does for const (with the same limitations).


Well, as I showed above, those limitations are kind of fatal for your 
use case.


 You could see `inout` as 
 shorthand for `inout(const)` under my suggestion.
 Obviously it must enforce the most restrictive implementation inside the 
 code; my function `inheritAttribs` above must be nothrow and nogc 
 internally, but externally it would allow a mapping from one of the 
 'input' attributes to the 'output' attribute in a non-templated way.


`inout` also interacts with `immutable`, not only `const`.
Dec 10
parent Manu <turkeyman gmail.com> writes:
On Wed, 11 Dec 2024 at 08:25, Timon Gehr via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 12/9/24 07:47, Manu wrote:

 On Mon, 9 Dec 2024 at 01:51, Timon Gehr via Digitalmars-d <digitalmars-
 d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On 12/8/24 06:54, Manu wrote:
      > Here's the stupidest idea ever: expand inout to take an


 argument...

      >
      > void parseThings(string s, void delegate(Thing) inout(nothrow)
      > inout( nogc) sink) inout(nothrow) inout( nogc)
      > {
      >    //...
      >    sink(Thing());
      >    //...
      > }
      > ...

     Issue with this is you will still not know which `inout`s match up.
     This
     is already an issue with the existing `inout`, which just arbitrarily
     selects a convention. In particular, it does not really work with
     higher-order functions the way you'd need it to here.


 Yes, this is obviously an issue, and we have it in D comprehensively;
 it's the main issue with our 'safe' stuff too; where rust has explicit
 lifetime attribution... it's essentially the same problem all round; it
 needs tags that specify things which are associated.
 In lieu of that though, I would say, if there are multiple things marked
 inout(...) in the way I proposed, you would assume the most pessimistic
 from the set of possibilities is supplied.

 void inheritAttribs(string s, void delegate() inout(nothrow)
 inout( nogc) fun_1, void delegate() inout(nothrow) inout( nogc) fun_2)
 inout(nothrow) inout( nogc)
 {
    //...
    fun_1();
    fun_2();
    //...
 }

 In this case, `inheritAttribs` would only be nothrow in the event BOTH
 fun_1 and fun_2 are nothrow, likewise for nogc...
 ...


 Well, this is one limitation, but the `inout` on the delegate does not
 even match up with the `inout` on `inheritAttribs` those are already
 different.

 This does not compile;

 ```d
 inout(int)* foo(inout(int)* delegate()inout dg,inout(int)* x){
      return x;
 }

 void main(){
      const(int)* delegate()const dg;
      const(int)* x;
      foo(dg,x); // error
 }
 ```

 It seems at least this would need to work.

 `inout` is very confusing because different `inout` annotations can get
 conflated or not conflated. It's also the main reason why the
 implementation is unsound, it's not done consistently during type checking.


I agree that's a problem. It's classic D though where things are
non-uniform that way.
The resolution would probably be to always conflate all inouts; since we
can't tag them individually, we have to assume they're all uniformly
tagged; and they all take on the most restrictive state from the call.


      > Surely people have had better ideas? But you get the point, and
     this is
      > basically essential to make the 'sink' pattern work at all in D.
      >
      > No, no templates; it's not right to generate multiple copies of
     identical functions just because something it CALLS would transfer
     an attribute. The function itself is identical no matter the state
     of any attribute transference, and so this isn't a template problem;
     it's a pattern matching problem.

     Well, it can be seen as a homogeneous vs heterogeneous compilation
     problem. The attributes can still act a bit like template parameters,
     but only one instance must be created that works for all of them.


 It's

     sometimes called parametric polymorphism.


 Right. Do you have examples of this from other languages and how it's
 expressed? Lifetime's in Rust are the obvious benchmark, but I don't see
 any evidence that D has a taste for this sort of thing.
 ...




 In Haskell everything is polymorphic by default with implicit universal
 quantification over lower-case type parameters.


 I do think that inout could work for all attributes the same as inout
 does for const (with the same limitations).


 Well, as I showed above, those limitations are kind of fatal for your
 use case.


Sorry, I didn't follow what makes them fatal?


 You could see `inout` as

 shorthand for `inout(const)` under my suggestion.
 Obviously it must enforce the most restrictive implementation inside the
 code; my function `inheritAttribs` above must be nothrow and nogc
 internally, but externally it would allow a mapping from one of the
 'input' attributes to the 'output' attribute in a non-templated way.


 `inout` also interacts with `immutable`, not only `const`.


The way I saw it was that at compile time, `inout` is presumed to be
equivalent to const. You can't pass an inout(int)* to an immutable(int)*,
but you can pass to a const(int)*... but yeah, in terms of conceptual
precision, I see your point.
Dec 11
prev sibling next sibling parent reply Monkyyy <crazymonkyyy gmail.com> writes:
On Sunday, 8 December 2024 at 05:54:40 UTC, Manu wrote:

 Surely people have had better ideas? No, no templates;


I'm pretty sure this only could work with templates and with 
templates it would just happen.

Or better yet, be inlined by the optimizer using function 
litteralls

"How do I get rid of this bruise on my head, no without stopping 
hitting my head with the hammer, surely someone with medical 
skills could actually help me"
Dec 08
parent Manu <turkeyman gmail.com> writes:
On Mon, 9 Dec 2024 at 03:51, Monkyyy via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Sunday, 8 December 2024 at 05:54:40 UTC, Manu wrote:

 Surely people have had better ideas? No, no templates;


 I'm pretty sure this only could work with templates and with
 templates it would just happen.


This doesn't *require* templates. Obviously there is a template solution,
but I'm very sensitive to code repetition and template bloat.
I'm pretty bored of D's "everything is templates" mentality; it's one of
D's biggest attractions, and also D's single biggest trap.
We need to work towards making more things sane without resorting to the
sledgehammer.

Or better yet, be inlined by the optimizer using function

 litteralls


This is not about inlining; these functions are not to be inlined.



 "How do I get rid of this bruise on my head, no without stopping
 hitting my head with the hammer, surely someone with medical
 skills could actually help me"


=F0=9F=A4=A8
Dec 08
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Sunday, 8 December 2024 at 05:54:40 UTC, Manu wrote:

 I've been trying out an API strategy to use sink functions a 
 lot more to hoist the item allocation/management logic to the 
 user and out from the work routines. This idea has received a 
 lot of attention, and I heard a lot of people saying this was a 
 planned direction to move in phobos and friends.

 ```d
 void parseThings(string s, void delegate(Thing) sink)
 {
   //...
   sink(Thing());
   //...
 }
 ```

 We have a serious problem though; a function that receives a 
 sink function
 must transfer the attributes from the sink function to itself, 
 and we have
 no language to do that...
 What are the leading strategies that have been discussed to 
 date? Is there
 a general 'plan'?


I've proposed something like the following in the past, but I 
don't know how much appetite there is for it:

```d
// assuming Thing() is  nogc nothrow  safe, but not pure
void parseThings(string s,  called void delegate(Thing) sink) 
 nogc nothrow  safe
{
    sink(Thing());
}
```

Essentially, how this would work is that the compiler would 
consider all code inside the function except the call of the 
delegate to be attributed with the provided attributes. However, 
the delegate call will *always work*. All that will happen is 
that the call to `parseThings` will have any restrictions removed 
that aren't on the delegate.

So e.g. a delegate of ` nogc pure` but not `nothrow` or ` safe` 
will mean `parseThings(dg)` will be treated like it's ` nogc`.

Because the list of attributes ` nogc nothrow  safe` is 
logic-anded with ` nogc pure`.

No recompile is necessary.

If you want to insist that some attribute must be on the 
delegate, you can add the attribute to the delegate parameter. So 
if I wanted to insist that you must pass in at least a ` safe` 
delegate:

```d
void parseThings(string s,  called void delegate(Thing)  safe 
sink) { ... }
```

As far as I know, there is no plan to make something like this 
happen.

-Steve
Dec 08
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Mon, 9 Dec 2024 at 05:21, Steven Schveighoffer via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Sunday, 8 December 2024 at 05:54:40 UTC, Manu wrote:

 I've been trying out an API strategy to use sink functions a
 lot more to hoist the item allocation/management logic to the
 user and out from the work routines. This idea has received a
 lot of attention, and I heard a lot of people saying this was a
 planned direction to move in phobos and friends.

 ```d
 void parseThings(string s, void delegate(Thing) sink)
 {
   //...
   sink(Thing());
   //...
 }
 ```

 We have a serious problem though; a function that receives a
 sink function
 must transfer the attributes from the sink function to itself,
 and we have
 no language to do that...
 What are the leading strategies that have been discussed to
 date? Is there
 a general 'plan'?


 I've proposed something like the following in the past, but I
 don't know how much appetite there is for it:

 ```d
 // assuming Thing() is  nogc nothrow  safe, but not pure
 void parseThings(string s,  called void delegate(Thing) sink)
  nogc nothrow  safe
 {
     sink(Thing());
 }
 ```

 Essentially, how this would work is that the compiler would
 consider all code inside the function except the call of the
 delegate to be attributed with the provided attributes. However,
 the delegate call will *always work*. All that will happen is
 that the call to `parseThings` will have any restrictions removed
 that aren't on the delegate.

 So e.g. a delegate of ` nogc pure` but not `nothrow` or ` safe`
 will mean `parseThings(dg)` will be treated like it's ` nogc`.

 Because the list of attributes ` nogc nothrow  safe` is
 logic-anded with ` nogc pure`.

 No recompile is necessary.

 If you want to insist that some attribute must be on the
 delegate, you can add the attribute to the delegate parameter. So
 if I wanted to insist that you must pass in at least a ` safe`
 delegate:

 ```d
 void parseThings(string s,  called void delegate(Thing)  safe
 sink) { ... }
 ```

 As far as I know, there is no plan to make something like this
 happen.

 -Steve


Interesting idea. It seems like a novel solution. Approaching it from this
perspective feels hard to reason about though... it feels like it would add
serious friction to introspection activities?
Dec 08
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Monday, 9 December 2024 at 06:57:31 UTC, Manu wrote:

 Interesting idea. It seems like a novel solution. Approaching 
 it from this perspective feels hard to reason about though... 
 it feels like it would add serious friction to introspection 
 activities?


How so? Nothing is different in terms of introspection. We 
already have friction on how to tell if a call is viable. Other 
than that, I don't see a difference here.

-Steve
Dec 09
parent Manu <turkeyman gmail.com> writes:
On Tue, 10 Dec 2024 at 12:26, Steven Schveighoffer via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Monday, 9 December 2024 at 06:57:31 UTC, Manu wrote:

 Interesting idea. It seems like a novel solution. Approaching
 it from this perspective feels hard to reason about though...
 it feels like it would add serious friction to introspection
 activities?


 How so? Nothing is different in terms of introspection. We
 already have friction on how to tell if a call is viable. Other
 than that, I don't see a difference here.

 -Steve


I think the essence of the issues I imagine stem from the fact that
`parseThings` takes on different characteristics (argument attributes
change) when/*where* it's CALLED. It means you can't ask hard facts about
the type; instead you'd be forced to use more __traits(compiles)-like
evaluations to test for compatibility; and I think those are a really bad
pattern in general.
The difference to `inout`; is that it's reliable and detectible at compile
time; it assumes the most restrictive traits, because it *could* be called
that way... your suggestion seems to be the opposite of that; permissive at
compile time, and I can imagine (by guy feeling) that leading to a world of
trouble.
What I suggest with `inout(attrib)` would do the same thing as inout(const)
does today; assume the most restrictive thing at compile time so that it's
a valid call in all situations.
Dec 11
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/8/24 20:19, Steven Schveighoffer wrote:

 On Sunday, 8 December 2024 at 05:54:40 UTC, Manu wrote:

 I've been trying out an API strategy to use sink functions a lot more 
 to hoist the item allocation/management logic to the user and out from 
 the work routines. This idea has received a lot of attention, and I 
 heard a lot of people saying this was a planned direction to move in 
 phobos and friends.

 ```d
 void parseThings(string s, void delegate(Thing) sink)
 {
   //...
   sink(Thing());
   //...
 }
 ```

 We have a serious problem though; a function that receives a sink 
 function
 must transfer the attributes from the sink function to itself, and we 
 have
 no language to do that...
 What are the leading strategies that have been discussed to date? Is 
 there
 a general 'plan'?


 
 I've proposed something like the following in the past, but I don't know 
 how much appetite there is for it:
 
 ```d
 // assuming Thing() is  nogc nothrow  safe, but not pure
 void parseThings(string s,  called void delegate(Thing) sink)  nogc 
 nothrow  safe
 {
     sink(Thing());
 }
 ```
 ...


Well, this is more sane than the "contract invalidation" proposal in 
that it is explicit, does not change the meaning of existing code, and 
allows somewhat more composition patterns. It's also forward-compatible 
with a proper solution, while most likely remaining more convenient for 
the cases where it is sufficient.

So maybe it's worth pursuing to stop some of the bleeding, but this will 
run into its own limitations very quickly.

In terms of implementation, it's probably a bit tricky to catch every 
place where the compiler assumes that e.g. ` safe` actually means 
` safe`. And there are also traits like `isSafe` that would most likely 
subtly change meaning.
Dec 10
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 11/12/2024 11:45 AM, Timon Gehr wrote:

 Well, this is more sane than the "contract invalidation" proposal in 
 that it is explicit, does not change the meaning of existing code, and 
 allows somewhat more composition patterns. It's also forward-compatible 
 with a proper solution, while most likely remaining more convenient for 
 the cases where it is sufficient.
 
 So maybe it's worth pursuing to stop some of the bleeding, but this will 
 run into its own limitations very quickly.


Contract invalidation can be explicit with the help of an attribute and 
I'm fine with any argument supporting its need as I agree its probably 
the better way to go.

But, it is a quick and dirty solution that doesn't solve the underlying 
problem of type qualifiers and attributes invalidation based upon the call.

What it does do is solve ``opApply``.

This proposal of Steven's is pretty much identical in nature.

Unfortunately only you seem to have any understanding of what a solution 
could look like and I've been failing to make any head way on it.
Dec 10
prev sibling next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Sunday, 8 December 2024 at 05:54:40 UTC, Manu wrote:

 I've been trying out an API strategy to use sink functions a 
 lot more to hoist the item allocation/management logic to the 
 user and out from the work routines. This idea has received a 
 lot of attention, and I heard a lot of people saying this was a 
 planned direction to move in phobos and friends.

 [...]


I remember Mathias had a plan for this. I think.
Dec 10
prev sibling parent reply Zach Tollen <notmyrealemail whatever.eee> writes:
On Sunday, 8 December 2024 at 05:54:40 UTC, Manu wrote:

 We have a serious problem though; a function that receives a 
 sink function
 must transfer the attributes from the sink function to itself, 
 and we have
 no language to do that...
 What are the leading strategies that have been discussed to 
 date? Is there
 a general 'plan'?


This was brought up in a [thread][thread1] in DIP Ideas. I 
contributed my idiosyncratic thoughts here:

https://forum.dlang.org/post/rtvjsdyuqwmzwiggsolw forum.dlang.org


short, I proposed that the language default to inheriting 
attributes for the enclosing function at the call site, and then 
suggested a (callable function/delegate) parameter keyword 
` noimply` for those rare cases when you *don't* want the 
function to inherit the characteristics of the delegate that you 
pass. I tried to explain it in the post.

All my post did was suggest some hopefully elegant syntax for a 
number of attribute-related issues, of which this was one.

[thread1]: 
https://forum.dlang.org/thread/pfawiqhppkearetcrkno forum.dlang.org
Dec 10
parent reply Manu <turkeyman gmail.com> writes:
On Wed, 11 Dec 2024 at 08:41, Zach Tollen via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Sunday, 8 December 2024 at 05:54:40 UTC, Manu wrote:

 We have a serious problem though; a function that receives a
 sink function
 must transfer the attributes from the sink function to itself,
 and we have
 no language to do that...
 What are the leading strategies that have been discussed to
 date? Is there
 a general 'plan'?


 This was brought up in a [thread][thread1] in DIP Ideas. I
 contributed my idiosyncratic thoughts here:

 https://forum.dlang.org/post/rtvjsdyuqwmzwiggsolw forum.dlang.org


 short, I proposed that the language default to inheriting
 attributes for the enclosing function at the call site, and then
 suggested a (callable function/delegate) parameter keyword
 ` noimply` for those rare cases when you *don't* want the
 function to inherit the characteristics of the delegate that you
 pass. I tried to explain it in the post.

 All my post did was suggest some hopefully elegant syntax for a
 number of attribute-related issues, of which this was one.

 [thread1]:
 https://forum.dlang.org/thread/pfawiqhppkearetcrkno forum.dlang.org


This appears to suffer from the same category of problem as Schveighoffer's
suggestion; it's that you can't reason about the function statically
anymore. Again, the reason I approach it from the perspective of inout(...)
is that you can always statically reason about it, and its guarantees are
appropriately applied at compile time.
Dec 11
parent reply Zach Tollen <notmyrealemail whatever.eee> writes:
On Wednesday, 11 December 2024 at 22:36:27 UTC, Manu wrote:

 On Wed, 11 Dec 2024 at 08:41, Zach Tollen via Digitalmars-d <

 This was brought up in a [thread][thread1] in DIP Ideas. I 
 contributed my idiosyncratic thoughts here:

 https://forum.dlang.org/post/rtvjsdyuqwmzwiggsolw forum.dlang.org


 short, I proposed that the language default to inheriting 
 attributes for the enclosing function at the call site, and 
 then suggested a (callable function/delegate) parameter 
 keyword ` noimply` for those rare cases when you *don't* want 
 the function to inherit the characteristics of the delegate 
 that you pass. I tried to explain it in the post.


 This appears to suffer from the same category of problem as 
 Schveighoffer's suggestion; it's that you can't reason about 
 the function statically anymore.


I'm pretty sure you can. I think there's no avoiding the fact 
that the the call site is only place where the proper behavior 
for these calls can be determined. The best the function 
definition can do is to let the caller know (via e.g., 
` noimply`) whether the function is able to *negate* any 
` system`/`throw`/ gc/impure effects (the latter two should be 
understood with their obvious meaning).

For example, a function could negate these effects by not 
actually *calling* the delegate. Or by other means, which vary 
depending on the attribute. But since negation is the rare case, 
by default the language should *assume*, I believe, that any 
delegate/function that is passed in as an argument *will* be 
called. Which is to say, all the inout(...) spam in your 
suggestion would work — but it shouldn't need to: it should work 
*implicitly*. It makes sense: If you pass in a delegate which is 
` system`/`throw`/ gc/impure, the call should be treated as a 
` system`/`throw`/ gc/impure action, unless specifically negated.


 Again, the reason I approach it from the perspective of 
 inout(...) is that you can always statically reason about it, 
 and its guarantees are appropriately applied at compile time.


My suggestion applies all the same guarantees at compile time. 
The negating effects of ` noimply` can be statically checked. For 
example:

```d
void f(void delegate() sink) {
     sink();
}

void g() nothrow {
     f(() { throw new Exception(""); } ); // Error: function f() 
called with delegate which has been inferred `throw`, in 
`nothrow` function g()
}
```
Let's tell it not to imply that the sink throws/doesn't throw:
```d
void h(void delegate()  noimply(throw) sink) {
     // The compiler tests the delegate by assuming sink() is 
 system/throw/ gc/impure
     sink(); // Error: delegate sink() is  noimply(throw), but may 
throw
}
```
The error above is the result of checking the delegate's 
attributes separately from the attributes for `h()`. This would 
happen for every use of a parameter delegate tagged with 
` noimply`. (For parameters *not* tagged with ` noimply`, no 
checking is necessary — the call site propagates all the 
` system`/`throw`/ gc/impure characteristics of the delegate that 
has been passed.)

In order to get it working, you have to make sure that the throw 
can't escape:
```d
void j(void delegate()  noimply(throw) sink) {
     // passes, statically checks that the call to sink() takes 
place in a context that can't throw
     try { sink(); }
     catch(Exception) {}
}

void k() nothrow {
     // passes(!) because the parameter in j() has been statically 
confirmed to not propagate the throw
     j(() { throw new Exception(""); } );
}
```
I believe this system has all the benefits of the other 
suggestions, but with almost no syntactic noise.
Dec 14
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 15/12/2024 7:47 PM, Zach Tollen wrote:

 In order to get it working, you have to make sure that the throw can't 
 escape:
 
 |void j(void delegate()  noimply(throw) sink) { // passes, statically 
 checks that the call to sink() takes place in a context that can't throw 
 try { sink(); } catch(Exception) {} } void k() nothrow { // passes(!) 
 because the parameter in j() has been statically confirmed to not 
 propagate the throw j(() { throw new Exception(""); } ); } |
 
 I believe this system has all the benefits of the other suggestions, but 
 with almost no syntactic noise.


``nothrow`` is a very bad example to give here.

In fact, it needs an entirely separate mechanism from the others.

It isn't boolean, and has codegen changes associated with it.

Also in your example, the delegate could be marked as throwing and it 
would work today.

``nothrow`` is already scope aware.
Dec 14
parent reply Zach Tollen <notmyrealemail whatever.eee> writes:
On Sunday, 15 December 2024 at 07:04:54 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 ``nothrow`` is a very bad example to give here.

 In fact, it needs an entirely separate mechanism from the 
 others.

 It isn't boolean, and has codegen changes associated with it.


That's fair. I don't know if delegates work with exception 
handlers in a way that would permit this mechanism. Like, if a 
function is otherwise `nothrow`, but is called with a delegate 
parameter which throws, can the function be compiled to allow the 
throwing delegate to bypass the function it is called within and 
have the delegate throw straight to the calling function instead? 
Probably not, but I don't know enough about how modern exception 
handlers are programmed to know for sure.


 Also in your example, the delegate could be marked as throwing 
 and it would work today.

 ``nothrow`` is already scope aware.


Under the current system, the following is an error:
```d
// Error: function `g` may throw but is marked as `nothrow`
void g(void delegate() sink) nothrow {
     sink();
}
```
Under the new system (apart from the mechanism and codegen issues 
you raised), the above would be permitted... only if you called 
it with a throwing delegate would it be classified as throwing, 
and at the call site, but not in the function itself. This is 
much cleaner. I believe that it is primarily within this context 
that the new attribute ` noimply` would be occasionally useful, 
as a way to say that even if you call the `nothrow` function with 
a *throwable* delegate, the call can *still* be treated as 
`nothrow`, because ` noimply(throw)` statically guarantees that 
any incoming throw will be neutralized.

I'm pretty sure the purpose of ` noimply` would only become 
apparent if the new default were already adopted.
Dec 15
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 15/12/2024 10:08 PM, Zach Tollen wrote:

 On Sunday, 15 December 2024 at 07:04:54 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 ``nothrow`` is a very bad example to give here.

 In fact, it needs an entirely separate mechanism from the others.

 It isn't boolean, and has codegen changes associated with it.


 
 That's fair. I don't know if delegates work with exception handlers in a 
 way that would permit this mechanism. Like, if a function is otherwise 
 `nothrow`, but is called with a delegate parameter which throws, can the 
 function be compiled to allow the throwing delegate to bypass the 
 function it is called within and have the delegate throw straight to the 
 calling function instead? Probably not, but I don't know enough about 
 how modern exception handlers are programmed to know for sure.


Its not just how exception handlers work, but I also have to consider 
how a sum type exception handling mechanism would work. There is plenty 
of desire for it, and I've got a proposal in ideas for one.

Either way, it has codegen implications.


 Also in your example, the delegate could be marked as throwing and it 
 would work today.

 ``nothrow`` is already scope aware.


 
 Under the current system, the following is an error:
 ```d
 // Error: function `g` may throw but is marked as `nothrow`
 void g(void delegate() sink) nothrow {
      sink();
 }
 ```
 Under the new system (apart from the mechanism and codegen issues you 
 raised), the above would be permitted... only if you called it with a 
 throwing delegate would it be classified as throwing, and at the call 
 site, but not in the function itself. This is much cleaner. I believe 
 that it is primarily within this context that the new attribute 
 ` noimply` would be occasionally useful, as a way to say that even if 
 you call the `nothrow` function with a *throwable* delegate, the call 
 can *still* be treated as `nothrow`, because ` noimply(throw)` 
 statically guarantees that any incoming throw will be neutralized.
 
 I'm pretty sure the purpose of ` noimply` would only become apparent if 
 the new default were already adopted.


In your previous example you had the sink call wrapped in a try catch 
statement. That'll work.

```d
void f(void delegate() del) nothrow {
	try {
		del();
	} catch (Exception) {
	}
}
```
Dec 15
parent Zach Tollen <notmyrealemail whatever.eee> writes:
On Sunday, 15 December 2024 at 09:15:07 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 In your previous example you had the sink call wrapped in a try 
 catch statement. That'll work.

 ```d
 void f(void delegate() del) nothrow {
 	try {
 		del();
 	} catch (Exception) {
 	}
 }
 ```


What would also work would be, only in the case of `throw`, to 
statically disallow passing a throwing delegate to a `nothrow` 
function. So:
```d
// still allowed under the new system
void g(void delegate() sink) nothrow {
     sink();
}

void h() {
     g(() { throw new Exception(""); } ); // Error: `nothrow` 
function g() called with delegate which has been inferred `throw`
}
```
Dec 15