www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [bolts] bolts.functioneditor - module for building function mixins?

reply Jean-Louis Leroy <jl leroy.nyc> writes:
Hi Ali (and anyone interested).

While implementing support for function attributes and parameter 
storage classes in 'openmethods', then working on my new 
'ducktyping' library (golang-style interfaces), I came up with a 
module that helps with creating function mixins derived from an 
existing function. Since it is used in two distinct projects, I 
want to put it in a library of its own.

It seems to me that this is the sort of things that bolts is all 
about. What do you think of taking it in?

You can see some code here:
https://github.com/jll63/openmethods.d/blob/using-function-editor/source/openmethods_functioneditor.d
I suggest you take a look at the unit test at the bottom. Real 
usage here: 
https://github.com/jll63/openmethods.d/blob/using-function-editor/s
urce/openmethods.d, search for occurrences of Editor.

It's still a work in progress but I am opening the discussion now 
because I will need permission from my employer. Which I will 
certainly get (they are very open-source friendly), but I will 
have to go through a bit of red tape and I will need to specify, 
in the ticket, whether I will be contributing to an existing 
open-source project (in case the module becomes part of bolts), 
or I will be leading a new project.
Apr 06 2020
parent reply aliak <something something.com> writes:
On Monday, 6 April 2020 at 20:52:56 UTC, Jean-Louis Leroy wrote:

 Hi Ali (and anyone interested).

 While implementing support for function attributes and 
 parameter storage classes in 'openmethods', then working on my 
 new 'ducktyping' library (golang-style interfaces), I came up 
 with a module that helps with creating function mixins derived 
 from an existing function. Since it is used in two distinct 
 projects, I want to put it in a library of its own.

 It seems to me that this is the sort of things that bolts is 
 all about. What do you think of taking it in?

 You can see some code here:
 https://github.com/jll63/openmethods.d/blob/using-function-editor/source/openmethods_functioneditor.d
 I suggest you take a look at the unit test at the bottom. Real 
 usage here: 
 https://github.com/jll63/openmethods.d/blob/using-function-editor/s
urce/openmethods.d, search for occurrences of Editor.


Hey! At first look it does look like something bolts-y so I'm 
very open to discussing the usecases/problems to figure it out. 
And it initially looks like very cool stuff!

Am I correct in assuming that FunctionEditor is a way to 
"stringify" various attributes of a function, in order to create 
string versions of new functions, in order to mixin new functions?

Where being able to mock interfaces is one usecase?

What other usecases would there be?

I looked at the real usage you have, but I don't think I 
understood the dispatcher and discriminator

And, one last question:

If string interpolation was around would the interface you've 
designed be the same?

I opened an issue in github as well: 
https://github.com/aliak00/bolts/issues/9 just incase it makes 
more sense to discuss it there.

Cheers!
Apr 07 2020
parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Tuesday, 7 April 2020 at 11:50:10 UTC, aliak wrote:

 Am I correct in assuming that FunctionEditor is a way to 
 "stringify" various attributes of a function, in order to 
 create string versions of new functions, in order to mixin new 
 functions?


Yes. In a first step the function is "meta-objectified", 
capturing all the function attributes (well not UDAs yet), 
parameter storage classes, and whether it is a member or a static 
function, in a template instantiation which has:
- accessors for querying all the attributes from a centralized 
place
- "mutators" that create new function meta-objects with altered 
attributes
- accessors that return strings suitable to be mixed in to 
produce function declarations or definitions (and I am going to 
add more for making a function pointer, a delegate, and maybe for 
adding template parameters)


 Where being able to mock interfaces is one usecase?


Yes.


 What other usecases would there be?


Any situation that involves wrapping a function from a template, 
across module boundaries - keeping in mind what Adam Ruppe 
explains here: 
https://stackoverflow.com/questions/32615733/struct-composition-with-mixin-and-templat


For example, my new ducktyping library works with two templates: 
InterfaceVariable(Interface), a fat pointer that contains a void* 
to an object and a Vtbl* to a table of forwarder functions; and 
as(T, IV), that returns an InterfaceVariable and builds the 
forwarders and Vtbl for type T.

Thus:

   interface Geometry
   {
      nogc nothrow void move(real dx, real dy);
   }

   alias IGeometry = InterfaceValue!Geometry;

   class Rectangle
   {
      nogc nothrow void move(real dx, real dy) { ... }
   }

   IGeometry geo = new Rectangle(...).as!IGeometry;

   geo.move(dx, dy);

...translates (more or less) to:

   struct InterfaceVariable!Geometry
   {
     struct Vtbl
     {
        nogc nothrow void function(void* obj, real a0, real a1) 
move0;
     }
     void* obj;
     Vtbl* vtbl;
      nogc nothrow void move(real dx, real dy) { vtbl.move0(obj, 
a0, a1); }
   }

   InterfaceVariable!Geometry as!(Rectangle, 
InterfaceVariable!Geometry)(Rectangle r)
   {
     static  nogc nothrow void move(void* obj, real a0, real a1)
     {
       (cast(Rectangle) obj).move(a0, a1);
     }
     static InterfaceVariable(Geometry).Vtbl vtbl = { &move0 };
     return InterfaceVariable(Geometry)(cast(void*) r, &vtbl);
   }

Note that it is essential to preserve ` nogc nothrow` (and any 
parameter storage classes even if the example doesn't show this).


 I looked at the real usage you have, but I don't think I 
 understood the dispatcher and discriminator


Given e.g.:

   module matrix;
   Matrix times(double a, virtual!Matrix b);
    method DiagonalMatrix _times(double a, DiagonalMatrix b) { ... 
}

The library generates the following code:

   // in module matrix
   alias times = Method!(matrix, "times", 0).dispatcher;
   alias times = Method!(matrix, "times", 0).discriminator;

   // in Method!(matrix, "times", 0):
   double dispatcher(double a0, Matrix a1) { return 
resolve(a1)(a0, a1); }
   Method!(matrix, "times", 0) discriminator(MethodTag, double a0, 
Matrix a1);

   // also a pointer type:
   double function(double, Matrix) Spec;

   // and a code string to build a wrapper for a method 
specialization
   // it will be mixed in a context where 'Spec' is an alias to a 
specialization
   enum wrapper(Spec) = q{
     Matrix wrapper(double a0, Matrix a1) { return Spec(a0, 
cast(DiagonalMatrix) a1); }

Of course template Method is not allowed to use names like Matrix 
or DiagonalMatrix. Instead everything is channeled via what I 
call the "anchor", in this case essentially 
`__traits(getOverloads, Module, Name)[Index]`.


 If string interpolation was around would the interface you've 
 designed be the same?


Yes. The whole point is to avoid manipulating string until the 
very end - when specifying the body. And indeed, at that point, 
interpolation will help the *client* code a lot.


 I opened an issue in github as well: 
 https://github.com/aliak00/bolts/issues/9 just incase it makes 
 more sense to discuss it there.


Let's discuss here for the time being, as others may jump in with 
useful comments and suggestions.

 From the GH "issue":


 For structs and classes I may have had a prototype that went 
 something like:


I haven't given a lot of thought to using this approach beyond 
functions. Probably manipulating enums and classes are less of a 
pain (no parameter storage classes), and may need much fewer 
string mixins.

However, I do like the idea of composing e.g. enums using a 
meta-objects, like:

   
mixin(EnumEditor!"Fruit".addEnumerator!"Grapefruit".etc.etc.asString);

On a side note, I would like to have a very simple construct that 
turns an identifier into a string (see 
https://forum.dlang.org/post/ayzrsajzcelavxdbwbji forum.dlang.org).

Finally I am not set on the template and module names. 
`FunctionEditor` is a bit errr flat. Transmogrify sounds much 
better ;-)
Apr 07 2020
parent reply aliak <something something.com> writes:
On Tuesday, 7 April 2020 at 16:58:12 UTC, Jean-Louis Leroy wrote:
[...]

 I haven't given a lot of thought to using this approach beyond 
 functions. Probably manipulating enums and classes are less of 
 a pain (no parameter storage classes), and may need much fewer 
 string mixins.


Yeah, I actually forgot the use case I was trying to make it for. 
So I agree at this time.


 However, I do like the idea of composing e.g. enums using a 
 meta-objects, like:

   
 mixin(EnumEditor!"Fruit".addEnumerator!"Grapefruit".etc.etc.asString);

 On a side note, I would like to have a very simple construct 
 that turns an identifier into a string (see 
 https://forum.dlang.org/post/ayzrsajzcelavxdbwbji forum.dlang.org).


Turn an identifier into a string?

int myint;
assert(__traits(identifier, myint) == "myint");

Like that?

The forum post made me think of something more like:

assert(isThisAValidIdentifier!("some string"))

?


 Finally I am not set on the template and module names. 
 `FunctionEditor` is a bit errr flat. Transmogrify sounds much 
 better ;-)


Hehe yeah, the name came from the Calvin and Hobbes comic books 
(https://calvinandhobbes.fandom.com/wiki/Transmogrifier)

But aaaaanyway, ok so thanks for the explanations :)

This is very cool stuff. I'm not exactly sure how the entire 
system can be seen yet - lots of questions , but if could build a 
static reflection system that is easy to use and allows for the 
mutation of compile time entities in order for derived entities 
to be mixed in, that "sounds" very cool.

I'm thinking use-cases can slowly be built up one by one, but the 
initial use-case you have is a great starting point. So, yeah, 
I'm all for it.

I opened a branch: 
https://github.com/aliak00/bolts/tree/reflection

And I put up an initial commit for variable reflection: 
https://github.com/aliak00/bolts/commit/5da534466285ae7357b56ccf9d60470a63d2ac61

At this time I have no idea how well that will work in the 
general sense but time will tell. It would be great if you could 
push small pieces of the function editor in to there (make a new 
file under the reflection module and just start with that?).

The entity traits should be integrated with the existing traits 
as well so that the source of truth remains the same - I find 
that's the hardest part about this library - there're so many 
ways to do things and it's already hard to keep track of which 
method is used where (even with a library as small as it is right 
now)
Apr 08 2020
parent Jean-Louis Leroy <jl leroy.nyc> writes:
On Wednesday, 8 April 2020 at 12:03:23 UTC, aliak wrote:


 Turn an identifier into a string?

 int myint;
 assert(__traits(identifier, myint) == "myint");

 Like that?


No, the other way around. I don't like to use the string literal 
syntax for template arguments when those strings are meant to be 
identifiers. I.e. instead of:

   mixin makeFunction!("foo", ...);
   mixin makeFunction!("0ops", ...); // weird error deep down

I'd rather write:

   mixin makeFunction!(q.foo, ...);
   mixin makeFunction!(q.0ops, ...); // error reported here

Sorta Lisp-ish. It's such a tiny thing that it doesn't deserve 
its own dub package, except on Fool's Day maybe. But maybe in a 
`bolts.quote`?


 I opened a branch: 
 https://github.com/aliak00/bolts/tree/reflection
 [...]
 At this time I have no idea how well that will work in the 
 general sense but time will tell. It would be great if you 
 could push small pieces of the function editor in to there 
 (make a new file under the reflection module and just start 
 with that?).


So right now I cannot send PRs - not until I get permission form 
my employer. I created a ticket for that but it will take some 
time.

But as part of `openmethods` it's OK, so what about I create a 
`bolts/reflection` subdir there and put stuff in it. You can 
watch it, we can discuss it, and you can send PRs against 
openmethods. It looks like dub doesn't object to having multiple 
`bolts` directories, from different packages. As soon as I get 
the green light, I will PR the entire `bolts/reflection` tree. In 
the meantime, someone who wants to play with this 
work-in-progress can dub depend on openmethods.

Since yesterday, I thought a lot about this. It looks like there 
is a path to making meta-programming a *lot* easier. I now think 
that replacing the __traits and std.traits caboodle with 
meta-objects is more important than creating string mixins, 
although of course we badly want to hide them as much as 
possible. So I am not going to focus only one code generation.

In last night's experiments I pushed the cumbersome nested 
`static foreach` loops on members and overloads to a mixin 
template in a new Aggregate meta-object. And it works 
beautifully.Here is the Mock(Interface) example again (from the 
unit test of bolts.reflection.metaaggregate):

   interface TheFullMonty
   {
     pure int foo() immutable;
      nogc  trusted nothrow ref int foo(out real, return ref int, 
lazy int) const;
      safe shared scope void foo(scope Object);
   }

   template Mocker(alias Function, int Index)
   {
     static if (is(Function.returnType.type == void)) {
       enum Mocker = Function.withBody!"";
     } else static if (Function.isRef) {
       enum Mocker = Function.withBody!q{
         static %s rv;
         return rv;
       }.format(Function.returnType.asString);
     } else {
       enum Mocker = Function.withBody!q{
         return %s.init;
       }.format(Function.returnType.asString);
     }
   }

   class Mock(Interface) : Interface
   {
     mixin Aggregate!Interface.forEachFunctionMixin!(Mocker);
   }

[Later] I renamed the branch to bolts-reflection, here: 
https://github.com/jll63/openmethods.d/tree/bolts-reflection/source/bolts/reflection
Apr 09 2020