www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Perfect forwarding

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
This topic came about during beerconf (it was fun!): Write an idiomatic 
template `forward` that takes an alias `fun` and defines (generates) one 
overload for each overload of `fun`. Example:

template forward(alias fun)
{
     ...
}

Now for this function:

void myfun(int, ref double, out string);
int myfun(in string, inout double);

the instantiation would be (stylized):

template forward!myfun
{
     void myfun(int a, ref double b, out string c)
     {
         return myfun(a, b, c);
     }
     int myfun(in string a, inout double b)
     {
         return myfun(a, b);
     }
}

So the challenge is implementing forward() to do this.
Jul 25 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 26 July 2020 at 01:29:21 UTC, Andrei Alexandrescu 
wrote:

 So the challenge is implementing forward() to do this.


I know the trick to do this, but I won't post the answer yet 
because I don't want to ruin it for others who want to try.

However I'm going to point out that my trick doesn't work if you 
are trying to forward a template, since template params cannot be 
introspected. As far as I know, there is no solution to that.

My impl also does a bonus: forwarding UDAs as well. That requires 
very new dmd though because until very recently that would hit a 
compiler bug! But it is possible.


Note: part of the challenge is making sure it works with 
user-defined types too. Your first thought of an implementation 
might not, so test that with a struct from an import when there's 
a local struct with the same name....
Jul 25 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/25/20 9:44 PM, Adam D. Ruppe wrote:

 However I'm going to point out that my trick doesn't work if you are 
 trying to forward a template, since template params cannot be 
 introspected. As far as I know, there is no solution to that.


That needs fixed. What primitives would you need for that?
Jul 25 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 26 July 2020 at 02:10:19 UTC, Andrei Alexandrescu 
wrote:

 That needs fixed. What primitives would you need for that?


Hmm, hard to say, and I'm also hesitant because it makes template 
optimization even harder - the more of the innards we expose, the 
less the compiler is able to discard, even in theory. (We 
probably will need some kind of specifically restricted template 
at some point as a practical optimization matter. Stefan's not 
wrong about the complications here.)

But anyway, I think we'd have to see at a minimum a list of 
template parameters. Constraint might be useful too, but that can 
be at least tested via __traits(compiles) once you have the 
template parameter list.

The tricky thing is figuring out how to represent them in the 
language. Regular function parameters are not representable 
either... you probably know how awkward it is to get function 
default arguments. (Oh, btw, those of you doing the OP challenge, 
remember default arguments should be forwarded too!) You have to 
make a helper function slicing the params and return the value 
since D doesn't really represent this data, it is more like an 
opaque object you can just use in certain contexts.

But at least with ordinary parameters, you can drill down with 
typeof(param). That would work for template value parameters, but 
what is T? Well, is(T) might help there.

static if(is(Param)) { /* template type param */ }
else static if(is(typeof(Param) Type)) { /* template value param 
*/ }

The big remaining question is an alias param. If we had some kind 
__traits(isAlias) that could be used here as well as in other 
places of reflection. Good idea regardless imo. Then you need to 
be able ot get the alias name itself, not just the identifier of 
what it is an alias of. I think __traits(identifier) and/or 
.stringof can do this now but unable to check as I type this 
email. But should make that formally possible - remember a 
template alias param will not actually be an alias of anything 
yet! Ditto on T, it is a speculative type, not an actual type. So 
regular reflection might not quite work, just it is as close as 
we can get.

And then T... params. That probably needs a specific detection 
too, could be like __traits(isAliasSeq) or something.


So to sum up before my laptop dies:

* a way to get the list of template parameters

* a way to identify the different kinds of template params. Some 
overlap with function parameters is possible but it will be 
tricky because template(T) is not actually a type. But if the 
compiler made it a placeholder type for this purpose then we can 
reuse some of the machinery.

* alias and tuple params could use specific attention, which can 
be written in a general-purpose manner.


Then I think we can make this happen. One of the guys at work has 
hit this trouble before too, perhaps on Monday we can loop him in 
and get a second opinion to see if I missed anything.
Jul 25 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Oops, I forgot about default arguments and specializations.

Default arguments are relatively straightforward, can do it at 
least as hacky as the runtime function variety.

But specializations are tricky to represent. The closest thing we 
have in the normal language is the `is` expression... but it 
isn't exactly the same... still if it could be like an is lambda 
it might work...

To be honest I suspect the best way to do this at this point 
would be to make it part of the opaque slice type. So if you 
define a new template with it, it inherits that but otherwise you 
can't really look in. I'm not satisfied with that but it might be 
the most practical thing to do right now and can be revisited in 
the future once the basic functionality actually works.
Jul 25 2020
parent reply Mathias LANG <geod24 gmail.com> writes:
On Sunday, 26 July 2020 at 03:59:00 UTC, Adam D. Ruppe wrote:

 Oops, I forgot about default arguments and specializations.

 Default arguments are relatively straightforward, can do it at 
 least as hacky as the runtime function variety.


How do you forward this:
```
class Foo
{
     void set (time_t value = time()) {}
}
```
Jul 26 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 26 July 2020 at 10:51:44 UTC, Mathias LANG wrote:

 How do you forward this:
 ```
 class Foo
 {
     void set (time_t value = time()) {}
 }
 ```


The same as the others wrt the default argument. The fact it is 
in a class complicates things a little since `this` is not 
propagated through... but if I remove it from there it still 
works.

Here's my solution as a link so people can look at it (and prolly 
destroy holes i missed lol) if you want but not click if you 
still wanna play with the challenge.

http://arsdnet.net/dcode/forward.d

The 8 lines at top are the forward, then the rest are the test 
conditions.

BTW I'd note that just doing a speculative forward is a bit 
simpler than this - you can just do (Args...)(auto ref Args args) 
- but the linked technique does it ahead-of-time, as if you wrote 
it by hand, meaning reflection still works on the forwarded 
definitions too.
Jul 26 2020
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-07-26 04:10, Andrei Alexandrescu wrote:


 That needs fixed. What primitives would you need for that?


It has already been implemented but rejected [1]. Andrei, you were part 
of the discussions and approved the changes.

[1] https://github.com/dlang/dmd/pull/5201

-- 
/Jacob Carlborg
Jul 26 2020
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Sunday, 26 July 2020 at 11:56:16 UTC, Jacob Carlborg wrote:

 On 2020-07-26 04:10, Andrei Alexandrescu wrote:


 That needs fixed. What primitives would you need for that?


 It has already been implemented but rejected [1]. Andrei, you 
 were part of the discussions and approved the changes.

 [1] https://github.com/dlang/dmd/pull/5201


Ah, this was in the back of my head when someone mentioned 
template parameters.

I haven't looked at that in a long time.
Jul 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 7/26/20 8:19 AM, Andrej Mitrovic wrote:

 On Sunday, 26 July 2020 at 11:56:16 UTC, Jacob Carlborg wrote:

 On 2020-07-26 04:10, Andrei Alexandrescu wrote:


 That needs fixed. What primitives would you need for that?


 It has already been implemented but rejected [1]. Andrei, you were 
 part of the discussions and approved the changes.

 [1] https://github.com/dlang/dmd/pull/5201


 
 Ah, this was in the back of my head when someone mentioned template 
 parameters.
 
 I haven't looked at that in a long time.


Well... would you like to revive it?
Jul 27 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 27 July 2020 at 14:08:30 UTC, Andrei Alexandrescu 
wrote:

 On 7/26/20 8:19 AM, Andrej Mitrovic wrote:

 On Sunday, 26 July 2020 at 11:56:16 UTC, Jacob Carlborg wrote:

 On 2020-07-26 04:10, Andrei Alexandrescu wrote:


 That needs fixed. What primitives would you need for that?


 It has already been implemented but rejected [1]. Andrei, you 
 were part of the discussions and approved the changes.

 [1] https://github.com/dlang/dmd/pull/5201


 
 Ah, this was in the back of my head when someone mentioned 
 template parameters.
 
 I haven't looked at that in a long time.


 Well... would you like to revive it?


One question that was brought up recently by an unrelated DMD PR 
[1] was whether new traits require a DIP. None of the traits we 
added in the past few years have gone through the DIP process as 
far as I remember. Of course, going through the DIP process can 
only improve the quality, but on the other hand, one could argue 
that it would make things unnecessary hard for easy additions 
like isDeprecated [2] and isDisabled [3].

How high should the bar be? Should we decide on a case-by-case 
basis?

[1]: 
https://github.com/dlang/dmd/pull/11442#issuecomment-661996482
[2]: https://github.com/dlang/dmd/pull/7178
[3]: https://github.com/dlang/dmd/pull/7569
Jul 27 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 7/27/20 11:57 AM, Petar Kirov [ZombineDev] wrote:

 On Monday, 27 July 2020 at 14:08:30 UTC, Andrei Alexandrescu wrote:

 On 7/26/20 8:19 AM, Andrej Mitrovic wrote:

 On Sunday, 26 July 2020 at 11:56:16 UTC, Jacob Carlborg wrote:

 On 2020-07-26 04:10, Andrei Alexandrescu wrote:


 That needs fixed. What primitives would you need for that?


 It has already been implemented but rejected [1]. Andrei, you were 
 part of the discussions and approved the changes.

 [1] https://github.com/dlang/dmd/pull/5201


 Ah, this was in the back of my head when someone mentioned template 
 parameters.

 I haven't looked at that in a long time.


 Well... would you like to revive it?


 
 One question that was brought up recently by an unrelated DMD PR [1] was 
 whether new traits require a DIP. None of the traits we added in the 
 past few years have gone through the DIP process as far as I remember. 
 Of course, going through the DIP process can only improve the quality, 
 but on the other hand, one could argue that it would make things 
 unnecessary hard for easy additions like isDeprecated [2] and isDisabled 
 [3].
 
 How high should the bar be? Should we decide on a case-by-case basis?
 
 [1]: https://github.com/dlang/dmd/pull/11442#issuecomment-661996482
 [2]: https://github.com/dlang/dmd/pull/7178
 [3]: https://github.com/dlang/dmd/pull/7569


Thanks for asking. I'm no longer in the decision loop but here's what I 
think. There'd be somewhat obvious traits to have ("is this symbol 
deprecated?") and traits that work together toward a larger goal, such 
as perfect forwarding or general introspection itself. In the former 
case acceptance could and should be somewhat quick, whereas in the 
latter case we risk having a hodge-podge of traits that don't combine 
properly to attain the initial goal. In a way we're there right now - we 
added traits on a need basis and we're unclear on what the limits of 
what we can and what we can't do, or how to do it.
Jul 27 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 27 July 2020 at 21:08:52 UTC, Andrei Alexandrescu 
wrote:

 [..]

 Thanks for asking. I'm no longer in the decision loop but 
 here's what I think. There'd be somewhat obvious traits to have 
 ("is this symbol deprecated?") and traits that work together 
 toward a larger goal, such as perfect forwarding or general 
 introspection itself. In the former case acceptance could and 
 should be somewhat quick, whereas in the latter case we risk 
 having a hodge-podge of traits that don't combine properly to 
 attain the initial goal. In a way we're there right now - we 
 added traits on a need basis and we're unclear on what the 
 limits of what we can and what we can't do, or how to do it.


So the question is what do we do about the new traits proposed by 
[1]? How formal does the process needs to be about them? I see 3 
options:
1. Traits or otherwise any language features part of a larger 
story (e.g. template parameter introspection, perfect forwarding) 
need to go through a 2 step DIP process:
   1) A strategic/vision DIP that states the high-level problem 
that we want to solve. The goal of this DIP would be form 
community and leadership consensus on whether something is a 
strategic goal for D and whether we need to solve it.
   2) A tactical DIP that proposes concrete language features and 
defines their semantics. In this case these would be the traits 
proposed by [1].

2. The language maintainers (Atila and Walter) decide on each DMD 
PR on a case-by-case basis.

3. New traits that haven't gone through a DIP process are 
considered experimental and can be added behind a compiler switch 
(which either enables all experimental traits, or each one 
individually). Such traits continue to have experimental status 
for several releases until we gather real-world experience and we 
decide to either remove, change, or to stabilize them (make them 
generally part of the language and available without any switch).
Unlike 2. any dmd contributor (or perhaps a group of minimum of 
2/3 people?) with write access to the dmd repo can decide to 
merge such experimental traits, after some minimal requirements 
are met, such as a changelog entry and sufficiently thorough 
test(s) that demonstrate the usefulness of the feature and cover 
all knownedge edge cases (e.g. each added error message is 
tested).

 Andrei  Atila  Walter what do you guys think?

[1]: https://github.com/dlang/dmd/pull/5201
Jul 27 2020
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Tuesday, 28 July 2020 at 05:34:59 UTC, Petar Kirov 
[ZombineDev] wrote:

  Andrei  Atila  Walter what do you guys think?

 [1]: https://github.com/dlang/dmd/pull/5201


The community (and maintainers) need to settle on the desired set 
of traits, and then we can think about how to best implement it.

I'm sure the PR is very outdated by now, there's probably better 
ways of implementing it.
Jul 28 2020
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 28 July 2020 at 05:34:59 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Monday, 27 July 2020 at 21:08:52 UTC, Andrei Alexandrescu 
 wrote:

 [...]


 So the question is what do we do about the new traits proposed 
 by [1]? How formal does the process needs to be about them? I 
 see 3 options:
 1. Traits or otherwise any language features part of a larger 
 story (e.g. template parameter introspection, perfect 
 forwarding) need to go through a 2 step DIP process:
   1) A strategic/vision DIP that states the high-level problem 
 that we want to solve. The goal of this DIP would be form 
 community and leadership consensus on whether something is a 
 strategic goal for D and whether we need to solve it.
   2) A tactical DIP that proposes concrete language features 
 and defines their semantics. In this case these would be the 
 traits proposed by [1].

 [...]


I'd say new traits would be case-by-case. My gut feeling is that 
they're too small for a DIP, but I also understand why you'd ask 
if there would have to be one first.
Aug 24 2020
prev sibling parent Jean-Louis Leroy <jl leroy.nyc> writes:
On Sunday, 26 July 2020 at 02:10:19 UTC, Andrei Alexandrescu 
wrote:

 On 7/25/20 9:44 PM, Adam D. Ruppe wrote:

 However I'm going to point out that my trick doesn't work if 
 you are trying to forward a template, since template params 
 cannot be introspected. As far as I know, there is no solution 
 to that.


 That needs fixed. What primitives would you need for that?


I need it to support open method templates.
Jul 26 2020
prev sibling next sibling parent Jean-Louis Leroy <jl leroy.nyc> writes:
On Sunday, 26 July 2020 at 01:29:21 UTC, Andrei Alexandrescu 
wrote:

 This topic came about during beerconf (it was fun!): Write an 
 idiomatic template `forward` that takes an alias `fun` and 
 defines (generates) one overload for each overload of `fun`. 
 Example:

 template forward(alias fun)
 {
     ...
 }

 Now for this function:

 void myfun(int, ref double, out string);
 int myfun(in string, inout double);

 the instantiation would be (stylized):

 template forward!myfun
 {
     void myfun(int a, ref double b, out string c)
     {
         return myfun(a, b, c);
     }
     int myfun(in string a, inout double b)
     {
         return myfun(a, b);
     }
 }

 So the challenge is implementing forward() to do this.


Using bolts' experimental refraction module:

module challenge;

import std.format;
import bolts.experimental.refraction;

template forward(alias fun)
{
     enum name = __traits(identifier, fun);
     static foreach (ovl; __traits(getOverloads, __traits(parent, 
fun), name)) {
         mixin(
             {
                 enum original = refract!(ovl, "ovl");
                 return original.withBody(q{{
                             return %s(%s);
                         }}.format(original.name, 
original.argumentMixture))
                     .mixture;
             }());
     }
}

void myfun(int, ref double, out string);

int myfun(in string, inout double);

pragma(msg, typeof(__traits(getOverloads, forward!myfun, 
"myfun")[0]));
pragma(msg, typeof(__traits(getOverloads, forward!myfun, 
"myfun")[1]));

Output:

 system void(int _0, ref double _1, out string _2)
 system int(const(string) _0, inout(double) _1)

Okay the "in" is missing in front of the first argument of the 
second overload, but it doesn't seem to be there in the first 
place:

pragma(msg, typeof(__traits(getOverloads, challenge, 
"myfun")[1]));

Output:
int(const(string), inout(double))

Also the body of the forwarders should probably use 
`std.functional.forward`.
Jul 26 2020
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On Sun, Jul 26, 2020 at 11:30 AM Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 This topic came about during beerconf (it was fun!): Write an idiomatic
 template `forward` that takes an alias `fun` and defines (generates) one
 overload for each overload of `fun`. Example:

 template forward(alias fun)
 {
      ...
 }

 Now for this function:

 void myfun(int, ref double, out string);
 int myfun(in string, inout double);

 the instantiation would be (stylized):

 template forward!myfun
 {
      void myfun(int a, ref double b, out string c)
      {
          return myfun(a, b, c);
      }
      int myfun(in string a, inout double b)
      {
          return myfun(a, b);
      }
 }

 So the challenge is implementing forward() to do this.


As someone who's been doing exactly this repeatedly basically-forever, I
can say the exercise is absolutely no fun at all.
It is my opinion that if the solution involves a text-mixin, the author
gets an instant FAIL.
The major hangup in this exercise is dealing with 'storage class', which is
impossible because it's not part of the language, and instantly forces
synthesising strings.
Forwarding the default args is tricky; and I've never managed to produce a
uniform solution that doesn't suffer some edge cases, but I've always made
it work for the specific set of things I'm wrapping.

Template arg forwarding is another level above, but I think that should be
taken as a secondary matter. Solve for normal functions first, and then
maybe there's a hope.
Jul 28 2020
parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Tuesday, 28 July 2020 at 22:48:16 UTC, Manu wrote:

 It is my opinion that if the solution involves a text-mixin, 
 the author gets an instant FAIL.


But:


 The major hangup in this exercise is dealing with 'storage 
 class', which is
 impossible because it's not part of the language, and instantly 
 forces synthesising strings.


Guaranteed failure then? ;-)
Jul 28 2020
parent reply Manu <turkeyman gmail.com> writes:
On Wed, Jul 29, 2020 at 9:40 AM Jean-Louis Leroy via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Tuesday, 28 July 2020 at 22:48:16 UTC, Manu wrote:

 It is my opinion that if the solution involves a text-mixin,
 the author gets an instant FAIL.


 But:


 The major hangup in this exercise is dealing with 'storage
 class', which is
 impossible because it's not part of the language, and instantly
 forces synthesising strings.


 Guaranteed failure then? ;-)


Yes.
Jul 29 2020
next sibling parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Wednesday, 29 July 2020 at 10:38:29 UTC, Manu wrote:

 On Wed, Jul 29, 2020 at 9:40 AM Jean-Louis Leroy via 
 Digitalmars-d < digitalmars-d puremagic.com> wrote:


 On Tuesday, 28 July 2020 at 22:48:16 UTC, Manu wrote:

 It is my opinion that if the solution involves a text-mixin, 
 the author gets an instant FAIL.


 But:


 The major hangup in this exercise is dealing with 'storage
 class', which is
 impossible because it's not part of the language, and 
 instantly
 forces synthesising strings.


 Guaranteed failure then? ;-)


 Yes.


This is a problem I spent a lot of time on while trying to 
support all the variations of functions in openmethods (and the 
problem is even harder because I need to alter the storage 
classes of a subset of the parameters).

Doesn't this cut it?

module challenge;

template forward(alias fun)
{
     import std.format;
     import std.traits;
     enum name = __traits(identifier, fun);
     static foreach (ovl; __traits(getOverloads, __traits(parent, 
fun), name)) {
         mixin(q{
                 auto ref %s(Parameters!ovl args) {
                     return __traits(parent, fun).%s(args);
                 }
             }.format(name, name));
     }
}

void myfun(int, ref double, out string);

int myfun(in string, inout double);

pragma(msg, typeof(__traits(getOverloads, forward!myfun, 
"myfun")[0]));
pragma(msg, typeof(__traits(getOverloads, forward!myfun, 
"myfun")[1]));

Output:
void(int, ref double, out string)
int(const(string), inout(double))

This deals with storage classes all right.

As for the string mixin, it would be nice to be able to do with 
it, or at least to narrow it to the function name, but in this 
respect D is more C than Lisp.
Jul 29 2020
parent reply Manu <turkeyman gmail.com> writes:
On Wed, Jul 29, 2020 at 10:35 PM Jean-Louis Leroy via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Wednesday, 29 July 2020 at 10:38:29 UTC, Manu wrote:

 On Wed, Jul 29, 2020 at 9:40 AM Jean-Louis Leroy via
 Digitalmars-d < digitalmars-d puremagic.com> wrote:


 On Tuesday, 28 July 2020 at 22:48:16 UTC, Manu wrote:

 It is my opinion that if the solution involves a text-mixin,
 the author gets an instant FAIL.


 But:


 The major hangup in this exercise is dealing with 'storage
 class', which is
 impossible because it's not part of the language, and
 instantly
 forces synthesising strings.


 Guaranteed failure then? ;-)


 Yes.


 This is a problem I spent a lot of time on while trying to
 support all the variations of functions in openmethods (and the
 problem is even harder because I need to alter the storage
 classes of a subset of the parameters).

 Doesn't this cut it?

 module challenge;

 template forward(alias fun)
 {
      import std.format;
      import std.traits;
      enum name = __traits(identifier, fun);
      static foreach (ovl; __traits(getOverloads, __traits(parent,
 fun), name)) {
          mixin(q{
                  auto ref %s(Parameters!ovl args) {
                      return __traits(parent, fun).%s(args);
                  }
              }.format(name, name));
      }
 }

 void myfun(int, ref double, out string);

 int myfun(in string, inout double);

 pragma(msg, typeof(__traits(getOverloads, forward!myfun,
 "myfun")[0]));
 pragma(msg, typeof(__traits(getOverloads, forward!myfun,
 "myfun")[1]));

 Output:
 void(int, ref double, out string)
 int(const(string), inout(double))

 This deals with storage classes all right.

 As for the string mixin, it would be nice to be able to do with
 it, or at least to narrow it to the function name, but in this
 respect D is more C than Lisp.


I mean... std.format in a forward macro? I find that embarrassing. Compile
times are important.  Of course, the usual casualties occur; debugging
steps through string mixins, etc.
Also, your solution doesn't perform any interesting transformation on
args... forwarding functions usually transform some argument(/s) in some
way, or inject additional arguments.
Hard and really lame to write any of the actual meat of the exercise inside
a string.

I'm not saying your solution is bad, just that the problem is bad, and it
really sucks that this thread exists. I've been complaining about this
since day-one. 'Storage class' is the problem as usual.
Jul 29 2020
parent Jean-Louis Leroy <jl leroy.nyc> writes:
On Thursday, 30 July 2020 at 01:31:59 UTC, Manu wrote:


 Also, your solution doesn't perform any interesting 
 transformation on
 args... forwarding functions usually transform some 
 argument(/s) in some
 way, or inject additional arguments.


I struggled with that in openmethods where transformations do 
occur, and that's why I created this thingy: 
https://github.com/aliak00/bolts/blob/master/source/bolts/experi
ental/refraction.d, which allows you to create a function from a function,
while manipulating its "aspects" in any way you want.

But yeah it's just a saner way of building a string mixin, and 
annoyingly, they are almost inevitable...just creating a function 
with anything but a fixed name requires a string mixin afaik..

Andrei's challenge and Adam's variation are among the simplest 
cases because the storage classes and function attributes can be 
inferred from the wrapped function. std.typecons.wrap, on the 
other hand, has to mess with function attributes.
Jul 29 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 29 July 2020 at 10:38:29 UTC, Manu wrote:

 On Wed, Jul 29, 2020 at 9:40 AM Jean-Louis Leroy via

 Guaranteed failure then? ;-)


 Yes.


What's wrong with my solution earlier in the thread?
Jul 29 2020
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Wednesday, 29 July 2020 at 12:54:36 UTC, Adam D. Ruppe wrote:

 On Wednesday, 29 July 2020 at 10:38:29 UTC, Manu wrote:

 On Wed, Jul 29, 2020 at 9:40 AM Jean-Louis Leroy via

 Guaranteed failure then? ;-)


 Yes.


 What's wrong with my solution earlier in the thread?


That it uses a string mixin :P

What Manu is arguing is that if parameter storage classes were 
instead proper type qualifiers, then one could trivially 
manipulate them with std.meta. And then supposedly there would be 
no need to string mixins at all.
I haven't spent the time to actually verify if that's the case 
here (perhaps the only need for using a string mixin is to 
generate variable names like Param1, Param2, ... ParamN) as is(X 
== __parameters) solves the issue with forwarding the whole 
parameter list (including storage classes), but in general, this 
doesn't scale if you need to do type manipulation with std.meta, 
as `alias RefInt = ref int` drops the `ref` storage class and 
this breaks everything like staticMap, Filter, etc.
Jul 29 2020
next sibling parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Wednesday, 29 July 2020 at 16:11:02 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Wednesday, 29 July 2020 at 12:54:36 UTC, Adam D. Ruppe wrote:

 On Wednesday, 29 July 2020 at 10:38:29 UTC, Manu wrote:

 On Wed, Jul 29, 2020 at 9:40 AM Jean-Louis Leroy via

 Guaranteed failure then? ;-)


 Yes.


 What's wrong with my solution earlier in the thread?


 That it uses a string mixin :P

 What Manu is arguing is that if parameter storage classes were 
 instead proper type qualifiers, then one could trivially 
 manipulate them with std.meta. And then supposedly there would 
 be no need to string mixins at all.


If we go back to the original problem, you still need a string 
mixin to inject the function name in two places.

As for Adam's solution, it solves a slightly different problem 
but I don't see why he uses a string mixin:

template forward(alias fun)
{
     import std.traits;
      (__traits(getAttributes, fun)) auto ref 
forward(Parameters!fun args) {
         return fun(args);
     }
}

 (42) void myfun(int, ref double x, out string s);

pragma(msg, typeof(forward!myfun));
// void function(int _param_0, ref double _param_1, out string 
_param_2)  system
pragma(msg, __traits(getAttributes, forward!myfun));
// tuple(42)

Anyway, storage classes are not a difficulty, as long as you use 
the whole __parameters, or slice it (__parameters[0..1]), and 
refrain from indexing it (__parameters[0] loses storage classes). 
Strange beast...
Jul 29 2020
next sibling parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Wednesday, 29 July 2020 at 16:31:08 UTC, Jean-Louis Leroy 
wrote:

 As for Adam's solution, it solves a slightly different problem 
 but I don't see why he uses a string mixin:


OK I see why Adam uses a string mixin: to generate identifiers to 
alias __parameters to. But that is not necessary. His complete 
example sans string mixins here: 
https://gist.github.com/jll63/d6575d2ec5318c355f164e529453db73
Jul 29 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 29 July 2020 at 16:52:32 UTC, Jean-Louis Leroy 
wrote:

 OK I see why Adam uses a string mixin: to generate identifiers 
 to alias __parameters to. But that is not necessary.


Yeah, that works too, but it introduces a template which I also 
try to avoid due to compiler memory management issues (the 
trivial mixin eats less dmd ram).
Jul 29 2020
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Wednesday, 29 July 2020 at 16:31:08 UTC, Jean-Louis Leroy 
wrote:

 On Wednesday, 29 July 2020 at 16:11:02 UTC, Petar Kirov 
 [ZombineDev] wrote:

 On Wednesday, 29 July 2020 at 12:54:36 UTC, Adam D. Ruppe 
 wrote:

 On Wednesday, 29 July 2020 at 10:38:29 UTC, Manu wrote:

 On Wed, Jul 29, 2020 at 9:40 AM Jean-Louis Leroy via

 Guaranteed failure then? ;-)


 Yes.


 What's wrong with my solution earlier in the thread?


 That it uses a string mixin :P

 What Manu is arguing is that if parameter storage classes were 
 instead proper type qualifiers, then one could trivially 
 manipulate them with std.meta. And then supposedly there would 
 be no need to string mixins at all.


 If we go back to the original problem, you still need a string 
 mixin to inject the function name in two places.

 As for Adam's solution, it solves a slightly different problem 
 but I don't see why he uses a string mixin:

 template forward(alias fun)
 {
     import std.traits;
      (__traits(getAttributes, fun)) auto ref 
 forward(Parameters!fun args) {
         return fun(args);
     }
 }

  (42) void myfun(int, ref double x, out string s);

 pragma(msg, typeof(forward!myfun));
 // void function(int _param_0, ref double _param_1, out string 
 _param_2)  system
 pragma(msg, __traits(getAttributes, forward!myfun));
 // tuple(42)

 Anyway, storage classes are not a difficulty, as long as you 
 use the whole __parameters, or slice it (__parameters[0..1]), 
 and refrain from indexing it (__parameters[0] loses storage 
 classes). Strange beast...


My point, that you can't do much useful processing with 
`__parameters` (except pass all of subset of it), still stands.

Say you have a function with N parameters. Some of those 
parameters are integers and may have a UDA attached to them that 
specifies the minimum and maximum value they may receive. For 
example:

R fun(
   scope P1 arg1,
   return int* arg2,
   lazy  intetval(-16, 320) long arg3,
    interval(0, 127) uint arg4,
   ref  interval(0, 8192) size_t arg5
);

The task is create a template, which given a function type like 
typeof(&fun) returns a new function type with all integer 
parameters replaced by the smallest type that is big enough to 
hold the interval specified by the UDA and preserves all storage 
classes.


R hun(
   scope P1 arg1,
   return int* arg2,
   lazy  intetval(-16, 320) short arg3,
    interval(0, 127) ubyte arg4,
   ref  interval(0, 8192) ushort arg5
);

The way I'd like to go about solving this is like this:

template TightenIntegerParams(Fun)
{
   alias TightenIntegerParams =
     ReturnType!Fun function(
       staticMap!(
         TightenParam,
         Parameters!Fun
       )
     );
}

However the moment staticMap internally does `Args[i]` in order 
to pass the i-th element to the mapping function `F` we lose all 
information about the storage classes and UDAs.
Jul 29 2020
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 30 July 2020 at 04:56:55 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Wednesday, 29 July 2020 at 16:31:08 UTC, Jean-Louis Leroy 
 wrote:

 [...]


 My point, that you can't do much useful processing with 
 `__parameters` (except pass all of subset of it), still stands.

 Say you have a function with N parameters. Some of those 
 parameters are integers and may have a UDA attached to them 
 that specifies the minimum and maximum value they may receive. 
 For example:

 R fun(
   scope P1 arg1,
   return int* arg2,
   lazy  intetval(-16, 320) long arg3,
    interval(0, 127) uint arg4,
   ref  interval(0, 8192) size_t arg5
 );

 The task is create a template, which given a function type like 
 typeof(&fun) returns a new function type with all integer 
 parameters replaced by the smallest type that is big enough to 
 hold the interval specified by the UDA and preserves all 
 storage classes.


 R hun(
   scope P1 arg1,
   return int* arg2,
   lazy  intetval(-16, 320) short arg3,
    interval(0, 127) ubyte arg4,
   ref  interval(0, 8192) ushort arg5
 );

 The way I'd like to go about solving this is like this:

 template TightenIntegerParams(Fun)
 {
   alias TightenIntegerParams =
     ReturnType!Fun function(
       staticMap!(
         TightenParam,
         Parameters!Fun
       )
     );
 }

 However the moment staticMap internally does `Args[i]` in order 
 to pass the i-th element to the mapping function `F` we lose 
 all information about the storage classes and UDAs.


I now saw that Manu clarified what he meant.

Also I do realize that what we actually need is for 
`Parameters!fun[0]` to return a parameter object that includes 
not just the type but also the parameter name, UDAs attached and 
storage classes. If for example `ref` was a type qualifier, or 
just otherwise preserved by `alias`-ing it would help, but not 
solve the whole issue.
Jul 29 2020
parent reply Manu <turkeyman gmail.com> writes:
On Thu, Jul 30, 2020 at 3:45 PM Petar via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Thursday, 30 July 2020 at 04:56:55 UTC, Petar Kirov
 [ZombineDev] wrote:

 On Wednesday, 29 July 2020 at 16:31:08 UTC, Jean-Louis Leroy
 wrote:

 [...]


 My point, that you can't do much useful processing with
 `__parameters` (except pass all of subset of it), still stands.

 Say you have a function with N parameters. Some of those
 parameters are integers and may have a UDA attached to them
 that specifies the minimum and maximum value they may receive.
 For example:

 R fun(
   scope P1 arg1,
   return int* arg2,
   lazy  intetval(-16, 320) long arg3,
    interval(0, 127) uint arg4,
   ref  interval(0, 8192) size_t arg5
 );

 The task is create a template, which given a function type like
 typeof(&fun) returns a new function type with all integer
 parameters replaced by the smallest type that is big enough to
 hold the interval specified by the UDA and preserves all
 storage classes.


 R hun(
   scope P1 arg1,
   return int* arg2,
   lazy  intetval(-16, 320) short arg3,
    interval(0, 127) ubyte arg4,
   ref  interval(0, 8192) ushort arg5
 );

 The way I'd like to go about solving this is like this:

 template TightenIntegerParams(Fun)
 {
   alias TightenIntegerParams =
     ReturnType!Fun function(
       staticMap!(
         TightenParam,
         Parameters!Fun
       )
     );
 }

 However the moment staticMap internally does `Args[i]` in order
 to pass the i-th element to the mapping function `F` we lose
 all information about the storage classes and UDAs.


 I now saw that Manu clarified what he meant.

 Also I do realize that what we actually need is for
 `Parameters!fun[0]` to return a parameter object that includes
 not just the type but also the parameter name, UDAs attached and
 storage classes. If for example `ref` was a type qualifier, or
 just otherwise preserved by `alias`-ing it would help, but not
 solve the whole issue.


UDA's and default args are in-language concepts, alias/tuples can handle
them, and don't require any particularly special meta to handle.
Jul 30 2020
parent Boris Carvajal <boris2.9 gmail.com> writes:
On Thursday, 30 July 2020 at 08:36:26 UTC, Manu wrote:

 I now saw that Manu clarified what he meant.

 Also I do realize that what we actually need is for 
 `Parameters!fun[0]` to return a parameter object that includes 
 not just the type but also the parameter name, UDAs attached 
 and storage classes. If for example `ref` was a type 
 qualifier, or just otherwise preserved by `alias`-ing it would 
 help, but not solve the whole issue.


 UDA's and default args are in-language concepts, alias/tuples 
 can handle them, and don't require any particularly special 
 meta to handle.


You can't alias default args like: fun(time_t a = time(null))
Jul 31 2020
prev sibling next sibling parent Jean-Louis Leroy <jl leroy.nyc> writes:
On Thursday, 30 July 2020 at 04:56:55 UTC, Petar Kirov 
[ZombineDev] wrote:


 For example:

 R fun(
   scope P1 arg1,
   return int* arg2,
   lazy  intetval(-16, 320) long arg3,
    interval(0, 127) uint arg4,
   ref  interval(0, 8192) size_t arg5
 );

 The task is create a template, which given a function type like 
 typeof(&fun) returns a new function type with all integer 
 parameters replaced by the smallest type that is big enough to 
 hold the interval specified by the UDA and preserves all 
 storage classes.


 R hun(
   scope P1 arg1,
   return int* arg2,
   lazy  intetval(-16, 320) short arg3,
    interval(0, 127) ubyte arg4,
   ref  interval(0, 8192) ushort arg5
 );

 The way I'd like to go about solving this is like this:

 template TightenIntegerParams(Fun)
 {
   alias TightenIntegerParams =
     ReturnType!Fun function(
       staticMap!(
         TightenParam,
         Parameters!Fun
       )
     );
 }

 However the moment staticMap internally does `Args[i]` in order 
 to pass the i-th element to the mapping function `F` we lose 
 all information about the storage classes and UDAs.


Yes I know. My original problem was:

// from
 attrs ref int foo( otherattrs virtual!Foo obj1, lazy int var, 
ref virtual!Bar obj2);

// make:
 attrs ref int foo( otherattrs Foo obj1, lazy int var, ref Bar 
obj2) {
     return dispatch(obj1, obj2)(obj1, var, obj2);
}

Back to what you described, is it a real use case? I am thinking 
of writing an article on function generation for the D blog. I 
was wondering how often this sort of transformation is needed.
Jul 30 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 30 July 2020 at 04:56:55 UTC, Petar Kirov 
[ZombineDev] wrote:

 My point, that you can't do much useful processing with 
 `__parameters` (except pass all of subset of it), still stands.


Yeah, I agree it is hard to do a substitution there. The foo... 
static map proposal a little while ago Manu wrote about is 
potentially exciting because it enables a lot of new cases, and I 
think this is another example.
Jul 30 2020
parent Manu <turkeyman gmail.com> writes:
On Thu, Jul 30, 2020 at 10:25 PM Adam D. Ruppe via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Thursday, 30 July 2020 at 04:56:55 UTC, Petar Kirov
 [ZombineDev] wrote:

 My point, that you can't do much useful processing with
 `__parameters` (except pass all of subset of it), still stands.


 Yeah, I agree it is hard to do a substitution there. The foo...
 static map proposal a little while ago Manu wrote about is
 potentially exciting because it enables a lot of new cases, and I
 think this is another example.


This is actually one of many such known motivators for the DIP.
Jul 30 2020
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 29 July 2020 at 16:11:02 UTC, Petar Kirov 
[ZombineDev] wrote:

 That it uses a string mixin :P


Not really - that string mixin is only there to work around a 
compiler bug leaking a name out of the scope

https://issues.dlang.org/show_bug.cgi?id=21078

Notice that the only thing actually mixed in is a random number 
to give a temporary variable a unique name.

It has nothing to do with storage classes or types or anything 
else.
Jul 29 2020
prev sibling parent Manu <turkeyman gmail.com> writes:
On Wed, Jul 29, 2020 at 10:55 PM Adam D. Ruppe via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Wednesday, 29 July 2020 at 10:38:29 UTC, Manu wrote:

 On Wed, Jul 29, 2020 at 9:40 AM Jean-Louis Leroy via

 Guaranteed failure then? ;-)


 Yes.


 What's wrong with my solution earlier in the thread?


It's pretty good, except my experience is that __parameters works only in
the narrow case of a verbatim forward. I rarely find I need to implement a
verbatim forward; I almost always seem to encounter a forwarding pattern
when it is necessary to perform a parameter transformation of some kind, or
a parameter injection. You need to be able to handle the arguments and
iterate/manipulate them.
I find I'm typically synthesising wrappers or shim's, not verbatim
forwarding... which isn't strictly the topic of this thread, but it's a
100% related problem, and it seems to be what I actually need to do 95% of
the time rather than the challenge in the OP.
Jul 29 2020
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 26 July 2020 at 01:29:21 UTC, Andrei Alexandrescu 
wrote:

 This topic came about during beerconf (it was fun!): Write an 
 idiomatic template `forward` that takes an alias `fun` and 
 defines (generates) one overload for each overload of `fun`. 
 Example:

 template forward(alias fun)
 {
     ...
 }

 Now for this function:

 void myfun(int, ref double, out string);
 int myfun(in string, inout double);

 the instantiation would be (stylized):

 template forward!myfun
 {
     void myfun(int a, ref double b, out string c)
     {
         return myfun(a, b, c);
     }
     int myfun(in string a, inout double b)
     {
         return myfun(a, b);
     }
 }

 So the challenge is implementing forward() to do this.


I would be interested an actual use-cases for this?

Where would you use, forward? Where would it be the most pleasant 
solution, and others would be vastly inferior?
Jul 29 2020
parent reply Chad Joan <chadjoan gmail.com> writes:
On Wednesday, 29 July 2020 at 20:07:35 UTC, Stefan Koch wrote:

 I would be interested an actual use-cases for this?

 Where would you use, forward? Where would it be the most 
 pleasant solution, and others would be vastly inferior?


I just ran into a use-case tonight! Maybe. The caveat is that my 
use case might be pretty easy compared to what Andrei is asking 
for. My use case requires modifying the function signature a lot, 
but that can actually be exploited to make my solution easier. I 
probably only need to forward storage class. It's still kinda 
leading to code that feels wrong or isn't obvious, so maybe it 
will help your discussion.

Now for the use-case itself:

I am considering putting a "stringize" method in my types, with a 
signature (by convention) like so:
 nogc nothrow void stringize(return scope StringVacuum writer) { 
... }

(I also plan to support a UFCS-style variant, but let's not get 
distracted.)

This method allows a type to be converted to a string, much like 
with "toString", but without a direct implication of heap-based 
(e.g. GC) memory allocation. If the type's string representation 
is being consumed by a non-memory source, such as stdout or a 
file stream, then it should be *possible* to avoid memory 
allocations entirely, at least in most cases, and with the very 
technical exception of small stack-allocated buffers for handling 
things like integer/float stringization (not shown here).

Gee, it would really suck if this were some kind of departure 
from the norm that caused incompatibilities with existing code 
and infrastructure. Whoops.

But it's OK, because "stringize" is toString-but-better, so it 
should be possible to just mechanically derive a "toString" 
function from every "stringize" function. Whenever this is used, 
it does abandon the original zero-allocation-conversion-to-string 
feature, but "toString" never had that to begin with (at least 
not in the general case; just for static strings and such). Code 
that acknowledges "stringize" can level-up. Code that depends on 
"toString" still works. Cool.

Now it would be really nice if the "toString" method generated 
from a "stringize" method would inherit its compile-time 
guarantees wherever possible. And sometimes it isn't possible. If 
"stringize" is 'nothrow', then "toString" based on "stringize" 
should also be 'nothrow'. However, if "stringize" is 'nogc', this 
does not automatically provide an  nogc "toString", because 
"toString" is required to concatenate a bunch of stringize's text 
fragments, and that concatenation requires GC allocations 
regardless of how  nogc "stringize" may be.

So a minor step in this process is to forward a permutation of 
"stringize" to "toString".

Here is more detailed code:
https://pastebin.com/JGfkkTxz
I am going to be in drafting-mode for a while and I haven't tried 
to compile any of that code. Please expect mistakes; I'm 
providing this to illustrate intent. I hope it helps.
Jul 31 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 31 July 2020 at 09:10:44 UTC, Chad Joan wrote:

 Whenever this is used, it does abandon the original 
 zero-allocation-conversion-to-string feature, but "toString" 
 never had that to begin with (at least not in the general case; 
 just for static strings and such).


kinda a site node but be aware there are toString overloads that 
do offer this.

see this for example: 
https://github.com/dlang/druntime/blob/master/src/object.d#L1995
Jul 31 2020
parent Chad Joan <chadjoan gmail.com> writes:
On Friday, 31 July 2020 at 13:01:37 UTC, Adam D. Ruppe wrote:

 On Friday, 31 July 2020 at 09:10:44 UTC, Chad Joan wrote:

 Whenever this is used, it does abandon the original 
 zero-allocation-conversion-to-string feature, but "toString" 
 never had that to begin with (at least not in the general 
 case; just for static strings and such).


 kinda a site node but be aware there are toString overloads 
 that do offer this.

 see this for example: 
 https://github.com/dlang/druntime/blob/master/src/object.d#L1995


That's good to know. Thanks for mentioning that!
Jul 31 2020
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 31 July 2020 at 09:10:44 UTC, Chad Joan wrote:

 I am considering putting a "stringize" method in my types, with 
 a signature (by convention) like so:
  nogc nothrow void stringize(return scope StringVacuum writer) 
 { ... }


Are you aware that toString already supports this kind of 
overload?

The documentation for it is under `std.format.formatValue`, so it 
is a bit easy to miss:

https://dlang.org/phobos/std_format.html#.formatValue
Jul 31 2020
parent Chad Joan <chadjoan gmail.com> writes:
On Friday, 31 July 2020 at 13:48:37 UTC, Paul Backus wrote:

 On Friday, 31 July 2020 at 09:10:44 UTC, Chad Joan wrote:

 I am considering putting a "stringize" method in my types, 
 with a signature (by convention) like so:
  nogc nothrow void stringize(return scope StringVacuum writer) 
 { ... }


 Are you aware that toString already supports this kind of 
 overload?

 The documentation for it is under `std.format.formatValue`, so 
 it is a bit easy to miss:

 https://dlang.org/phobos/std_format.html#.formatValue


I was not. That's good to know. This one even accepts an 
OutputRange. Cool.

Thanks!
Jul 31 2020