www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Generically call a function on Variant's payload?

reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
Suppose I've wrapped a Variant in a struct/class which ensures the 
Variant *only* ever contains types which satisfy a particular constraint 
(for example: isInputRange, or hasLength, etc...).

Is there a way to call a function (ex: popFront) on the Variant, 
*without* knowing ahead of time all the possible static types it might 
might contain?
Aug 19 2018
next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 08/19/2018 08:27 PM, Nick Sabalausky (Abscissa) wrote:

 Suppose I've wrapped a Variant in a struct/class which ensures the 
 Variant *only* ever contains types which satisfy a particular constraint 
 (for example: isInputRange, or hasLength, etc...).
 
 Is there a way to call a function (ex: popFront) on the Variant, 
 *without* knowing ahead of time all the possible static types it might 
 might contain?


Maybe something involving using Variant.coerce to convert the payload to 
a single common type? Not sure how I would do that though.
Aug 19 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, August 19, 2018 6:33:06 PM MDT Nick Sabalausky (Abscissa) via 
Digitalmars-d-learn wrote:

 On 08/19/2018 08:27 PM, Nick Sabalausky (Abscissa) wrote:

 Suppose I've wrapped a Variant in a struct/class which ensures the
 Variant *only* ever contains types which satisfy a particular constraint
 (for example: isInputRange, or hasLength, etc...).

 Is there a way to call a function (ex: popFront) on the Variant,
 *without* knowing ahead of time all the possible static types it might
 might contain?


 Maybe something involving using Variant.coerce to convert the payload to
 a single common type? Not sure how I would do that though.


You could always create a wrapper type. Whatever code you're dealing with is
going to need to statically know what type it's using even that's a base
type in a class hierarchy. So, you need to present a type which actually has
the appropriate API even if internally, it can dynamically handle several
types which have that API, and it's not known ahead of time which type that
is.

- Jonathan M Davis
Aug 19 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 08/19/2018 10:23 PM, Jonathan M Davis wrote:

 On Sunday, August 19, 2018 6:33:06 PM MDT Nick Sabalausky (Abscissa) via
 Digitalmars-d-learn wrote:

 Maybe something involving using Variant.coerce to convert the payload to
 a single common type? Not sure how I would do that though.


 
 You could always create a wrapper type. Whatever code you're dealing with is
 going to need to statically know what type it's using even that's a base
 type in a class hierarchy. So, you need to present a type which actually has
 the appropriate API even if internally, it can dynamically handle several
 types which have that API, and it's not known ahead of time which type that
 is.
 


I guess the parts I'm unclear on are:

1. What mechanism does Variant.coerce!SomeType use to attempt conversion 
to SomeType?

2. How (if possible) can I provide a way to convert something to type 
SomeType which Variant.coerce!SomeType will then recognize and use?
Aug 19 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, August 19, 2018 9:08:39 PM MDT Nick Sabalausky (Abscissa) via 
Digitalmars-d-learn wrote:

 On 08/19/2018 10:23 PM, Jonathan M Davis wrote:

 On Sunday, August 19, 2018 6:33:06 PM MDT Nick Sabalausky (Abscissa) via

 Digitalmars-d-learn wrote:

 Maybe something involving using Variant.coerce to convert the payload
 to
 a single common type? Not sure how I would do that though.


 You could always create a wrapper type. Whatever code you're dealing
 with is going to need to statically know what type it's using even
 that's a base type in a class hierarchy. So, you need to present a type
 which actually has the appropriate API even if internally, it can
 dynamically handle several types which have that API, and it's not
 known ahead of time which type that is.


 I guess the parts I'm unclear on are:

 1. What mechanism does Variant.coerce!SomeType use to attempt conversion
 to SomeType?

 2. How (if possible) can I provide a way to convert something to type
 SomeType which Variant.coerce!SomeType will then recognize and use?


Glancing at coerce's implementation, it just uses std.conv.to, but it's
fairly restricted on which types it will convert to. The target type has to
be numeric, bool, convertible to Object, or an array of characters. It looks
like it will convert the object to string to do the conversion, so it's
doing slightly more than just using std.conv.to and restricting the
conversions, but really, it's basically just calling std.conv.to in a
restricted manner.

So, looking at coerce, I don't really understand why it exists. It just
seems like it would make more sense to tell folks to get the object out with
get and then convert it themselves using std.conv.to or whatever makes the
most sense for their use case. Presenting a function that uses std.conv.to
but restricts which conversions work really doesn't make sense to me.

Either way, if you're doing something like using a Variant to hold multiple
range types, I very much doubt that coerce is going to do you much good.

- Jonathan M Davis
Aug 19 2018
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 20 August 2018 at 00:27:04 UTC, Nick Sabalausky 
(Abscissa) wrote:

 Suppose I've wrapped a Variant in a struct/class which ensures 
 the Variant *only* ever contains types which satisfy a 
 particular constraint (for example: isInputRange, or hasLength, 
 etc...).

 Is there a way to call a function (ex: popFront) on the 
 Variant, *without* knowing ahead of time all the possible 
 static types it might might contain?


You are basically reinventing OOP here.

Instead of Variants, use objects that implement an interface 
(e.g., `std.range.interfaces.InputRange`). Then you can call 
methods that belong to that interface and rely on virtual method 
dispatch to choose the correct implementation at runtime.
Aug 19 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 08/19/2018 11:31 PM, Paul Backus wrote:

 
 You are basically reinventing OOP here.
 


Yes, I am. Deliberately, in fact. Class inheritance is my backup plan, 
though.

My use-case is actually very, very similar to std.digest:

There are various benefits to using a template-and-constraint based 
interface. But one notable downside is the inability to handle the 
situation where the actual type needed is only known at runtime. In 
std.digest, this dilemma is dealt with by duplicating the entire 
interface in BOTH template/constraint AND class-inheritance varieties.

Additionally, an important thing to note in the case of std.digest is 
that `isDigest!T` returns FALSE for types using the OOP version of the 
interface. I think there is (arguably) a certain merit in that:

The OOP interface obviously uses classes, whereas (in order to obtain 
the full potential benefits of a template-and-constraint approach) the 
template interface uses (and even requires) structs. Because of this, 
instances should be constructed differently ("T instance;" vs "T 
instance = new T();"). Thus, any generic code that wants to handle ANY 
digest type must be very careful to mind this distinction whenever 
creating new instances. Presumably, this may be why std.digest chose to 
NOT let OO digests satisfy the template-oriented isDigest.

And then std.digest also needs WrapperDigest, needed to convert a 
template-style digest to OO-style.

So...while std.digest succeeds at offering the best-of-both-worlds 
between template and OO approaches, it also creates a whole new mess via 
completely duplicated interfaces that aren't 100% compatible.

I want to see if I can do better.

So back to the root of the problem:

We have a bunch of types, including user-defined types we don't have 
advance knowledge of. And we need a type which can hold any of them at 
runtime.

AFAIK, there are two classic ways to do that: One is OOP, the other is 
an unbounded discriminated union (a variant). Unlike class-based 
inheritance, a variant CAN be implemented as a struct. So, at least in 
theory, a variant-based type could potentially be made which implements 
the *true* template-based interface, eliminating the need for duplicate 
APIs and the mess that entails.

There are a bunch of discriminated union types available for D, but the 
only one I'm aware of that *doesn't* require a finite-sized list of 
types known ahead-of-time is Phobos's Variant. The only problem is 
calling a function on a type not already known ahead-of-time. Maybe 
that's unsolvable? If so, then I'll fallback to the std.digest approach. 
But I'd prefer to avoid that, if possible.
Aug 20 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, August 20, 2018 1:38:13 PM MDT Nick Sabalausky (Abscissa) via 
Digitalmars-d-learn wrote:

 There are a bunch of discriminated union types available for D, but the
 only one I'm aware of that *doesn't* require a finite-sized list of
 types known ahead-of-time is Phobos's Variant. The only problem is
 calling a function on a type not already known ahead-of-time. Maybe
 that's unsolvable? If so, then I'll fallback to the std.digest approach.
 But I'd prefer to avoid that, if possible.


It can be done if you know the list of types ahead of time, but you
basically have to check which type it is at runtime and then pick which code
to run based on that type, which means that you're essentially duplicating a
portion of the code for every type - though depending, it would be possible
to do something like templatize the code so that you don't have to
explicitly dulpicate it. But in the end, aside from using classes, if you're
using a variant type, I think that you're stuck doing something like

// exact API made up here rather than bothering to take the type to look up
// Variant's API.
void popFront()
{
    foreach(T; TypesThatVariantHolds)
    {
        if(_variant.contains!T)
        {
            auto v = _variant.get!T;
            v.popFront();
            return;
        }
    }
}

On the other hand, if you don't know the exact list of types ahead of time
or require that the code calling this code tell you which type to use, then
you're pretty much stuck using something like classes or delegates. You
can't just call functions on completely unknown types, because the compiler
wouldn't know what code to generate or what to link against.

- Jonathan M Davis
Aug 20 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 08/20/2018 04:34 PM, Jonathan M Davis wrote:

 
      foreach(T; TypesThatVariantHolds)


Yea, that's what I would've just done, but I wanted to support 
user-created types not already known to my library.


 You
 can't just call functions on completely unknown types, because the compiler
 wouldn't know what code to generate or what to link against.
 


Right. D's templates have really spoiled me. Because of them, I've grown 
accustomed to invoking functions on unknown types ;)

But you're right, that's not going to work for runtime dispatch without 
enumerating each possible type. Even if I try (like I was thinking) to 
provide a templated function to convert any of the concrete types to an 
internal-only supertype, I'd still need something to runtime dispatch to 
the right template instance. Darn.

A lot of times, I really wish I could introspect across *all* linked 
modules, not just specific ones. Ex: "Hey compiler, give me a 
compile-time list of ALL types in this program with UDA  xyz". Then 
things like this could be built out of that. No need to worry about 
unknown types/symbols because they would all be known and a finite list 
could always be constructed.

Although, what would REALLY be nice is if, for every type, the compiler 
built a list of every member and how to access it (which...really it has 
to do anyway), but then uses that info to build a master runtime 
dispatch system. Types such as Variant already have runtime knowledge of 
what type is currently being held, so that info could be passed to the 
compiler-generated "master runtime dispatch" system along with what 
member to invoke/access.

Come to think of it...aren't I just describing standard run-of-the-mill 
runtime reflection? I mean, hasn't Java already been able to do that for 
ages? But then, in Java everything is already part of the class 
hierarchy anyway so maybe that's why it's possible there?
Aug 20 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, August 20, 2018 8:29:30 PM MDT Nick Sabalausky (Abscissa) via 
Digitalmars-d-learn wrote:

 On 08/20/2018 04:34 PM, Jonathan M Davis wrote:

      foreach(T; TypesThatVariantHolds)


 Yea, that's what I would've just done, but I wanted to support
 user-created types not already known to my library.


 You
 can't just call functions on completely unknown types, because the
 compiler wouldn't know what code to generate or what to link against.


 Right. D's templates have really spoiled me. Because of them, I've grown
 accustomed to invoking functions on unknown types ;)

 But you're right, that's not going to work for runtime dispatch without
 enumerating each possible type. Even if I try (like I was thinking) to
 provide a templated function to convert any of the concrete types to an
 internal-only supertype, I'd still need something to runtime dispatch to
 the right template instance. Darn.

 A lot of times, I really wish I could introspect across *all* linked
 modules, not just specific ones. Ex: "Hey compiler, give me a
 compile-time list of ALL types in this program with UDA  xyz". Then
 things like this could be built out of that. No need to worry about
 unknown types/symbols because they would all be known and a finite list
 could always be constructed.

 Although, what would REALLY be nice is if, for every type, the compiler
 built a list of every member and how to access it (which...really it has
 to do anyway), but then uses that info to build a master runtime
 dispatch system. Types such as Variant already have runtime knowledge of
 what type is currently being held, so that info could be passed to the
 compiler-generated "master runtime dispatch" system along with what
 member to invoke/access.

 Come to think of it...aren't I just describing standard run-of-the-mill
 runtime reflection? I mean, hasn't Java already been able to do that for
 ages? But then, in Java everything is already part of the class
 hierarchy anyway so maybe that's why it's possible there?


Runtime reflection is theoretically possible in D, but it requires
generating the appropriate stuff for every type involved so that the runtime
stuff has something to work with. Java built all of that into the language
and has the JVM to boot, which fundamentally changes some of what can be
done. With D, we're basically in exactly the same boat as C or C++ except
that we have better compile-time type introspection. In principle, a runtime
reflection facility could be built using that, but even if it were part of
Phobos, it would still have to be opt-in. So, I don't know how useful such a
solution would ever be outside of very specific use cases. Regardless, given
that D's object files, linking, etc. are using the C tools, it was never
going to be the case that Java-style runtime reflection would be built in to
D like it is with Java.

- Jonathan M Davis
Aug 20 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 08/20/2018 10:57 PM, Jonathan M Davis wrote:

 
 Runtime reflection is theoretically possible in D, but it requires
 generating the appropriate stuff for every type involved so that the runtime
 stuff has something to work with. Java built all of that into the language
 and has the JVM to boot, which fundamentally changes some of what can be
 done. With D, we're basically in exactly the same boat as C or C++ except
 that we have better compile-time type introspection. In principle, a runtime
 reflection facility could be built using that, but even if it were part of
 Phobos, it would still have to be opt-in. So, I don't know how useful such a
 solution would ever be outside of very specific use cases. Regardless, given
 that D's object files, linking, etc. are using the C tools, it was never
 going to be the case that Java-style runtime reflection would be built in to
 D like it is with Java.


Yea. Not to disagree at all with those reasons, but that is unfortunate. 
I've long been interested in how various features of D (not exclusively 
D, though) combine in ways that, in effect, obsolete much of traditional 
OOP inheritence-based polymorphism - offering the same abilities of OOP 
but without many of the notable downsides. Java-style runtime reflection 
would take us that much further in this regard. (Plus, it would make D 
that much more of a universal-toolbox of a language.) Oh well, dreams vs 
reality ;)
Aug 20 2018
prev sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 20 August 2018 at 00:27:04 UTC, Nick Sabalausky 
(Abscissa) wrote:

 Suppose I've wrapped a Variant in a struct/class which ensures 
 the Variant *only* ever contains types which satisfy a 
 particular constraint (for example: isInputRange, or hasLength, 
 etc...).

 Is there a way to call a function (ex: popFront) on the 
 Variant, *without* knowing ahead of time all the possible 
 static types it might might contain?


I assume you mean at compile time AoT of development of your 
library?

Yes, don't use Variant; use 
https://github.com/s-ludwig/taggedalgebraic
Aug 21 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 08/21/2018 03:03 AM, Nicholas Wilson wrote:

 On Monday, 20 August 2018 at 00:27:04 UTC, Nick Sabalausky (Abscissa) 
 wrote:

 Suppose I've wrapped a Variant in a struct/class which ensures the 
 Variant *only* ever contains types which satisfy a particular 
 constraint (for example: isInputRange, or hasLength, etc...).

 Is there a way to call a function (ex: popFront) on the Variant, 
 *without* knowing ahead of time all the possible static types it might 
 might contain?


 
 I assume you mean at compile time AoT of development of your library?
 
 Yes, don't use Variant; use https://github.com/s-ludwig/taggedalgebraic
 


TaggedAlgebraic requires a finite list of every type it can contain. 
That's the deal-breaker for what I had in mind. I need something less 
like Algebraic and more like Variant, which *doesn't* require a finite, 
canonical list of every type it can contain.

The more I think about it though, the more I'm not so sure that the 
extensibility is important enough to warrant all of this, or...more 
importantly...that all of this is even necessary anyway for users to 
accomplish the use-cases I had in mind. I think I may go with an 
entirely different approach.
Aug 21 2018
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/21/18 3:11 PM, Nick Sabalausky (Abscissa) wrote:

 On 08/21/2018 03:03 AM, Nicholas Wilson wrote:

 On Monday, 20 August 2018 at 00:27:04 UTC, Nick Sabalausky (Abscissa) 
 wrote:

 Suppose I've wrapped a Variant in a struct/class which ensures the 
 Variant *only* ever contains types which satisfy a particular 
 constraint (for example: isInputRange, or hasLength, etc...).

 Is there a way to call a function (ex: popFront) on the Variant, 
 *without* knowing ahead of time all the possible static types it 
 might might contain?


 I assume you mean at compile time AoT of development of your library?

 Yes, don't use Variant; use https://github.com/s-ludwig/taggedalgebraic


 
 TaggedAlgebraic requires a finite list of every type it can contain. 
 That's the deal-breaker for what I had in mind. I need something less 
 like Algebraic and more like Variant, which *doesn't* require a finite, 
 canonical list of every type it can contain.
 
 The more I think about it though, the more I'm not so sure that the 
 extensibility is important enough to warrant all of this, or...more 
 importantly...that all of this is even necessary anyway for users to 
 accomplish the use-cases I had in mind. I think I may go with an 
 entirely different approach.


If you examine how the code for variant works, it's quite clever. It 
establishes a "handler" that is generated with full compile-time type 
info available when the value is *assigned*, but then cast to a function 
taking a void pointer and an operation. This way, the user of the 
variant doesn't have to know what is inside, just the handler does. The 
code that sets the payload is the one that establishes how to use it.

The same type of pattern could be used to, for instance, provide all the 
functions that a range uses.

But it's not something that can be tacked on to Variant. You'd have to 
write your own type, because you'd have to handle the different 
functions that you know are common between the types in question (e.g. 
the operations supported on a variant are here: 
https://github.com/dlang/phobos/blob/master/std/variant.d#L163)

-Steve
Aug 21 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 08/21/2018 03:27 PM, Steven Schveighoffer wrote:

 
 If you examine how the code for variant works, it's quite clever. It 
 establishes a "handler" that is generated with full compile-time type 
 info available when the value is *assigned*, but then cast to a function 
 taking a void pointer and an operation. This way, the user of the 
 variant doesn't have to know what is inside, just the handler does. The 
 code that sets the payload is the one that establishes how to use it.
 
 The same type of pattern could be used to, for instance, provide all the 
 functions that a range uses.
 
 But it's not something that can be tacked on to Variant. You'd have to 
 write your own type, because you'd have to handle the different 
 functions that you know are common between the types in question (e.g. 
 the operations supported on a variant are here: 
 https://github.com/dlang/phobos/blob/master/std/variant.d#L163)


Oooh, that's really cool! I might try my hand at something like that.

I bet it actually could be generalized to a certain extent. Variant 
would just have to be templated on the expected interface. Then it could 
compile-time loop over the list of functions expected, and introspect 
each of them enough to build the handler and the forwarders. Doesn't 
exactly sound fun to implement, but I'd bet it could be done.
Aug 21 2018