www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Reimplementing the bulk of std.meta iteratively

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I'll start with the punchline: at the link below you'll find an 
alpha-quality reimplementation of std.meta in an iterative manner:

https://gist.github.com/andralex/6212ebf2b38c59c96cf66d0009336318

It's the bulk of std.meta - I stopped when it became obvious doing more 
is just more of the same. Compiles and runs with dmd 2.092 and probably 
others.

The implementation uses throughout a reification/dereification approach. 
To reify a type or alias means to convert it into a value. To dereify a 
value means to turn it back into the type or alias it originated from.

The algorithms follow the following pattern consistently:

1. Reify the compile-time parameters into values
2. Carry processing on these values with the usual algorithms
3. If needed, dereify back the results into compile-time parameters

Not all artifacts need step 3. For example, allSatisfy returns a bool, 
not a type.

For the most part implementation has been a healthy gallop that took a 
couple of hours yesterday and a couple of hours today. Its weakest point 
is it uses .stringof for types, which has issues with local types. 
Hopefully https://github.com/dlang/dmd/pull/11797 could help with that.

Most implementations are a few lines long because they get to leverage 
algorithms as implemented in std. Here's a typical one:

alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
}());

To sort an AliasSeq with most derived elements first, create a lambda to 
reify the arguments into values, sort the values, and return the sorted 
result. Then call the lambda and use dereify against its result. And 
that is IT.

Commments and ideas for improvements are welcome.
Sep 26 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Sep 26, 2020 at 12:18:27PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 The implementation uses throughout a reification/dereification
 approach. To reify a type or alias means to convert it into a value.
 To dereify a value means to turn it back into the type or alias it
 originated from.


[...]

 Its weakest point is it uses .stringof for types, which has issues
 with local types. Hopefully https://github.com/dlang/dmd/pull/11797
 could help with that.


[...]


our current state.  It closes the reification/dereification loop, and
allows CTFE manipulation of types as strings, which sidesteps the issues
of how to manipulate types in imperative code without running into all
sorts of problems like the ones that Stefan brought up.

In the future, I can imagine CTFE *generation* of these strings, which
opens up brand new ways of manipulating types.  There could be a
standard set of mangle-string manipulation utilities in druntime or
phobos, say, that let you perform operations like adding/removing
qualifiers, manipulating type lists (in string form!), so you could
implement things like Unqual as a CTFE string manipulation function
instead of a bunch of templates. Of course, Unqual is simple enough that
there's no reason to do this, but the point is, we can do much more
complicated manipulations of types without needing to resort to
recursive templates, and it leverages existing machinery instead of
inventing a slew of new ones.


T

-- 
LINUX = Lousy Interface for Nefarious Unix Xenophobes.
Sep 26 2020
parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Saturday, 26 September 2020 at 16:40:47 UTC, H. S. Teoh wrote:

 On Sat, Sep 26, 2020 at 12:18:27PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]

 The implementation uses throughout a reification/dereification 
 approach. To reify a type or alias means to convert it into a 
 value. To dereify a value means to turn it back into the type 
 or alias it originated from.


 [...]

 Its weakest point is it uses .stringof for types, which has 
 issues with local types. Hopefully 
 https://github.com/dlang/dmd/pull/11797 could help with that.


 [...]


 forward from our current state.  It closes the 
 reification/dereification loop, and allows CTFE manipulation of 
 types as strings, which sidesteps the issues of how to 
 manipulate types in imperative code without running into all 
 sorts of problems like the ones that Stefan brought up.



With all the constraints and problems.
Iff you can use a string mixin you can use __stringToType.
If you can't use a string mixin in the context you can't use 
stringToType.
Sep 26 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/26/20 1:32 PM, Stefan Koch wrote:


 With all the constraints and problems.
 Iff you can use a string mixin you can use __stringToType.
 If you can't use a string mixin in the context you can't use stringToType.


mixin("+1000");

-Steve
Sep 26 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:

 1. Reify the compile-time parameters into values
 2. Carry processing on these values with the usual algorithms
 3. If needed, dereify back the results into compile-time 
 parameters


That's the right thought, and that's what type functions were 
designed to do.

How do they compare to the vision you have?


 Commments and ideas for improvements are welcome.


The PR referenced does not give, actually give you the ability to 
dereify types within CTFE.
The PR forces a template and template has to have template 
parameters to take the mangles.
I would think that is against the actual goal.
Sep 26 2020
parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 26 September 2020 at 17:30:44 UTC, Stefan Koch wrote:

 The PR referenced does not give, actually give you the ability 
 to dereify types within CTFE.
 The PR forces a template and template has to have template 
 parameters to take the mangles.
 I would think that is against the actual goal.


It's against the goal if your goal is for meta-functions to be 
100% monomorphic. But it does accomplish the goal of turning 
template recursion (O(n) instantiations) into flat iteration 
(O(1) instantiations).

Personally, I don't think 100% monomorphism is necessary or 
particularly desirable, so I'm ok with a design that eschews it.
Sep 26 2020
prev sibling next sibling parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:

 For the most part implementation has been a healthy gallop that 
 took a couple of hours yesterday and a couple of hours today. 
 Its weakest point is it uses .stringof for types, which has 
 issues with local types. Hopefully 
 https://github.com/dlang/dmd/pull/11797 could help with that.


It's amazing to me how obvious this idea seems now that I've seen 
it. D's had type -> string serialization for ages; of course 
there should also be a way to do the reverse! And yet somehow, it 
never even crossed my mind until I read Walter's PR.

I think this is a real winner.
Sep 26 2020
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 26.09.20 18:18, Andrei Alexandrescu wrote:

 
 Most implementations are a few lines long because they get to leverage 
 algorithms as implemented in std. Here's a typical one:
 
 alias MostDerived(Args...) = dereify!({
      auto ids = reify!Args;
      sort!((a, b) => is(dereify!a : dereify!b))(ids);
      return ids;
 }());


I am pretty sure this one does not work though.
Sep 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 5:16 PM, Timon Gehr wrote:

 On 26.09.20 18:18, Andrei Alexandrescu wrote:

 Most implementations are a few lines long because they get to leverage 
 algorithms as implemented in std. Here's a typical one:

 alias MostDerived(Args...) = dereify!({
      auto ids = reify!Args;
      sort!((a, b) => is(dereify!a : dereify!b))(ids);
      return ids;
 }());


 
 I am pretty sure this one does not work though.


Oops, Steve just told me it doesn't have any unittests. So it may as 
well not be working. Research goes on.
Sep 26 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/26/20 5:42 PM, Andrei Alexandrescu wrote:

 On 9/26/20 5:16 PM, Timon Gehr wrote:

 On 26.09.20 18:18, Andrei Alexandrescu wrote:

 Most implementations are a few lines long because they get to 
 leverage algorithms as implemented in std. Here's a typical one:

 alias MostDerived(Args...) = dereify!({
      auto ids = reify!Args;
      sort!((a, b) => is(dereify!a : dereify!b))(ids);
      return ids;
 }());


 I am pretty sure this one does not work though.


 
 Oops, Steve just told me it doesn't have any unittests. So it may as 
 well not be working. Research goes on.


So just to follow up on this, the reason it doesn't work is because the 
relation between a and b must determined BEFORE the call to sort.

In other words, in order for this to work, you have to establish result 
of the lambda for all a and b in the array.

Or optionally, you can save enough information into the "Id" type that 
can be able to enable the comparison at runtime.

The general concept works, but only if you can create a relation 
function that doesn't rely on the types themselves, but on the runtime 
representation of the types.

In other words, such a function would need 3 steps:

1. transform the types into runtime data that is comparable given the types.
2. Run the runtime function that does the transformation
3. Convert the result back to the types.

Such things are not *impossible*, but something like derivative is a 
great example that is really difficult to generate a runtime function 
that can do it.

-Steve
Sep 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 6:00 PM, Steven Schveighoffer wrote:

 On 9/26/20 5:42 PM, Andrei Alexandrescu wrote:

 On 9/26/20 5:16 PM, Timon Gehr wrote:

 On 26.09.20 18:18, Andrei Alexandrescu wrote:




 Such things are not *impossible*, but something like derivative is a 
 great example that is really difficult to generate a runtime function 
 that can do it.


Yah, you'd need i.e. a member function in Id called bool 
implicitlyConvertible(Id another). The functionality needed is similar 
to that in Variant, which created the need for implicitConversionTargets 
(https://dlang.org/library/std/traits/implicit_conversion_targets.html). 
You'd need to have access to those in the Id space.
Sep 26 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/26/20 6:03 PM, Andrei Alexandrescu wrote:

 On 9/26/20 6:00 PM, Steven Schveighoffer wrote:

 On 9/26/20 5:42 PM, Andrei Alexandrescu wrote:

 On 9/26/20 5:16 PM, Timon Gehr wrote:

 On 26.09.20 18:18, Andrei Alexandrescu wrote:




 Such things are not *impossible*, but something like derivative is a 
 great example that is really difficult to generate a runtime function 
 that can do it.


 
 Yah, you'd need i.e. a member function in Id called bool 
 implicitlyConvertible(Id another). The functionality needed is similar 
 to that in Variant, which created the need for implicitConversionTargets 
 (https://dlang.org/library/std/traits/implicit_conversion_targets.html). 
 You'd need to have access to those in the Id space.


Variant works because it can access the parameter type. If you need to 
access only one type, and the other information is runtime-accessible, 
then you can do it. If both need to be runtime-accessible, you are in 
trouble.

Like try comparing 2 Variants. If Variant doesn't have any idea about 
how the types might compare ahead of time, it gives up.

-Steve
Sep 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 6:15 PM, Steven Schveighoffer wrote:

 On 9/26/20 6:03 PM, Andrei Alexandrescu wrote:

 On 9/26/20 6:00 PM, Steven Schveighoffer wrote:

 On 9/26/20 5:42 PM, Andrei Alexandrescu wrote:

 On 9/26/20 5:16 PM, Timon Gehr wrote:

 On 26.09.20 18:18, Andrei Alexandrescu wrote:




 Such things are not *impossible*, but something like derivative is a 
 great example that is really difficult to generate a runtime function 
 that can do it.


 Yah, you'd need i.e. a member function in Id called bool 
 implicitlyConvertible(Id another). The functionality needed is similar 
 to that in Variant, which created the need for 
 implicitConversionTargets 
 (https://dlang.org/library/std/traits/implicit_conversion_targets.html). 
 You'd need to have access to those in the Id space.


 
 Variant works because it can access the parameter type. If you need to 
 access only one type, and the other information is runtime-accessible, 
 then you can do it. If both need to be runtime-accessible, you are in 
 trouble.


A possibly angle here is to save that information in runtime format, 
i.e. reify the inheritance/convertibility relationship. I'm unclear on 
how to do that efficiently. (The inefficient solution would save the 
result of ImplicitConversionTargets in a member of Id that has type Id[] 
and then do searches in it.)

In an ideal world it would be nice if id1 <= id2 iff is(reify!id1 : 
reify!id2).
Sep 26 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/26/20 6:48 PM, Andrei Alexandrescu wrote:

 A possibly angle here is to save that information in runtime format, 
 i.e. reify the inheritance/convertibility relationship.


thinking it through...

All basic types have well-defined relationships that can be figured out 
using a big statically generated switch thing.

Enums can be converted to their bases, and whatever they can convert to.

Structs can be converted to their alias-this parameter.

Classes can also have an alias this, AND a set of base types.

Arrays and pointers can convert in certain cases to base types.

And, multiply all this by mutability changes (convert to const for 
things with references, and convert to anything for value types).

So for each type, you could generate a list of types that it can convert 
to. Then on each comparison, search for the target type in the list, and 
if you find it, return true.

I can't imagine this is going to be efficient. We might be able to 
factor out the const changes. But it still leaves a big list of things. 
Honestly, I don't know if the compiler does it in a more efficient way 
than searching through a list.

So it's definitely doable, I just don't know if we would want to do it, 
or if the complication of the resulting code is worth it. is(a : b) is 
just soooo much easier! And I think this is one aspect that type 
functions allow.

If there were a way to go back to type-land in the middle of 
runtime-at-ctfe-land, even if you had to define it in a restrictive way, 
then it might be much more palatable.

-Steve
Sep 26 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 10:30 PM, Steven Schveighoffer wrote:

 On 9/26/20 6:48 PM, Andrei Alexandrescu wrote:

 A possibly angle here is to save that information in runtime format, 
 i.e. reify the inheritance/convertibility relationship.


 
 thinking it through...
 
 All basic types have well-defined relationships that can be figured out 
 using a big statically generated switch thing.
 
 Enums can be converted to their bases, and whatever they can convert to.


*nod*


 Structs can be converted to their alias-this parameter.


*nod* but how is that detectable for a given type T?


 Classes can also have an alias this, AND a set of base types.


yes... https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213 
has the base classes part as a proof of concept (no interfaces and no 
alias this).


 Arrays and pointers can convert in certain cases to base types.


*nod* these would be detectable with relative eash


 And, multiply all this by mutability changes (convert to const for 
 things with references, and convert to anything for value types).
 
 So for each type, you could generate a list of types that it can convert 
 to. Then on each comparison, search for the target type in the list, and 
 if you find it, return true.
 
 I can't imagine this is going to be efficient.


In the words of a guy I heard at a panel: "I'm not sure I understood 
what the problem is, but it seems the solution is a hashtable". :o)


 We might be able to 
 factor out the const changes. But it still leaves a big list of things. 
 Honestly, I don't know if the compiler does it in a more efficient way 
 than searching through a list.
 
 So it's definitely doable, I just don't know if we would want to do it, 
 or if the complication of the resulting code is worth it. is(a : b) is 
 just soooo much easier! And I think this is one aspect that type 
 functions allow.


It's part of the introspection game: gather the information typically 
present in the innards of the compiler, in programmatically-available 
form. I mean you would expect to grab access to implicit conversion 
targets in a language that has introspection.


 If there were a way to go back to type-land in the middle of 
 runtime-at-ctfe-land, even if you had to define it in a restrictive way, 
 then it might be much more palatable.


I'd be very interested too! (My experience with reification is now 
measured in hours.) This is the kind of stuff Timon would know.
Sep 26 2020
prev sibling parent reply Stefan Koch <uplink.coder gmail.com> writes:
On Saturday, 26 September 2020 at 21:42:19 UTC, Andrei 
Alexandrescu wrote:

 On 9/26/20 5:16 PM, Timon Gehr wrote:

 On 26.09.20 18:18, Andrei Alexandrescu wrote:

 Most implementations are a few lines long because they get to 
 leverage algorithms as implemented in std. Here's a typical 
 one:

 alias MostDerived(Args...) = dereify!({
      auto ids = reify!Args;
      sort!((a, b) => is(dereify!a : dereify!b))(ids);
      return ids;
 }());


 
 I am pretty sure this one does not work though.


 Oops, Steve just told me it doesn't have any unittests. So it 
 may as well not be working. Research goes on.


This example on the other hand works:

---
/+
import std.array;
import std.range;
import std.algorithm;
doesn't work yet but only because the template connstraints need 
to be adjusted.
+/

alias alias_array = typeof(((alias T) => __traits(getAttributes, 
T))(Byte));

auto sort3(alias less, R)(ref /*alias_array*/ R r)
{
     alias tmp;

     if (r.length == 1)
         return r;
     if (r.length == 2)
     {
         if (less(r[1], r[0]))
         {
             tmp = r[0];
             r[0] = r[1];
             r[1] = tmp;
         }
     }
     if (r.length == 3)
     {
         if (less(r[2], r[1]))
         {
             tmp = r[1];
             r[1] = r[2];
             r[2] = tmp;
         }
         if (less(r[1], r[0]))
         {
             tmp = r[0];
             r[0] = r[1];
             r[1] = tmp;
         }
     }
     return r;
}

auto sortBySize(alias_array types ...)
{
     types.sort3!((alias a, alias b) => a.sizeof < b.sizeof);

     return types;
};

alias Int = int;
alias Ushort = ushort; // we need these aliases because the 
parser won't allow us to call a function with a reserved type 
identifier.
alias Byte = byte;
pragma(msg, sortBySize(Ushort, Int, Byte)); //Output:  [(byte), 
(ushort), (int)]

---
Sep 26 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 27 September 2020 at 00:10:33 UTC, Stefan Koch wrote:

 This example on the other hand works:


Here's another potential implementation:

---
template sortBySize(T...) {
         string code() {
                 // step one: gather the data in the template shim
                 static struct Item {
                         size_t idx;
                         size_t size;
                 }
                 Item[] items;
                 foreach(idx, t; T)
                         items ~= Item(idx, t.sizeof);

                 // step two: process the data with normal CTFE
                 import std.algorithm;
                 items.sort!((a, b) => a.size < b.size);

                 // step three: build the code for the template to 
bring it back
                 // this would be a fine candidate for a helper 
function
                 string code;
                 foreach(item; items) {
                         if(code.length) code ~= ", ";
                         code ~= "T[" ~ cast(char)(item.idx + '0') 
~ "]";
                 }

                 return "AliasSeq!(" ~ code ~ ")";
         }

         import std.meta;
         alias sortBySize = mixin(code());
}

pragma(msg, sortBySize!(ushort, int, byte));
---

(byte, ushort, int)



There's several optimizations here, preallocating, factoring out 
of the template, not using std.algorithm, you know those tricks 
already.

I'm not in love with it but there is some potential in the 
concept in today's D.
Sep 26 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 8:27 PM, Adam D. Ruppe wrote:

 On Sunday, 27 September 2020 at 00:10:33 UTC, Stefan Koch wrote:

 This example on the other hand works:


 
 Here's another potential implementation:
 
 ---
 template sortBySize(T...) {


[snip]

Sorting by inheritance relationship is more difficult. I got this going, 
alpha quality:

https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213

Now indeed calling std.sort against (reified) types just works.

The point is to reify the bases of the type as well so they can be 
manipulated as objects. Then defining the comparison relation is easy 
(didn't bother to make it efficient here - just a linear search).

More introspection-related matters would be reified the same way (store 
during construction in the reified object, potentially via pointers if 
they don't apply to all objects). Instance size would be an obvious 
candidate, prolly I'll put it in there as a demo. More interestingly 
we'd have things like parameters/return for functions and data members 
for classes and structs. Long way ahead.

This is really exciting stuff.
Sep 26 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei Alexandrescu 
wrote:

 This is really exciting stuff.


Which I came up with.
Sep 26 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/26/20 10:48 PM, Stefan Koch wrote:

 On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei Alexandrescu wrote:

 This is really exciting stuff.


 
 Which I came up with.


Congratulations!
Sep 26 2020
prev sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei Alexandrescu 
wrote:

 On 9/26/20 8:27 PM, Adam D. Ruppe wrote:

 On Sunday, 27 September 2020 at 00:10:33 UTC, Stefan Koch 
 wrote:

 This example on the other hand works:


 
 Here's another potential implementation:
 
 ---
 template sortBySize(T...) {


 [snip]

 Sorting by inheritance relationship is more difficult. I got 
 this going, alpha quality:

 https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213

 Now indeed calling std.sort against (reified) types just works.

 The point is to reify the bases of the type as well so they can 
 be manipulated as objects. Then defining the comparison 
 relation is easy (didn't bother to make it efficient here - 
 just a linear search).

 More introspection-related matters would be reified the same 
 way (store during construction in the reified object, 
 potentially via pointers if they don't apply to all objects). 
 Instance size would be an obvious candidate, prolly I'll put it 
 in there as a demo. More interestingly we'd have things like 
 parameters/return for functions and data members for classes 
 and structs. Long way ahead.

 This is really exciting stuff.


We're in an early stage of exploration of discovered capability, 
so I understand the excitement.  The C++ guys must have been way 
excited about their meta programming discoveries as well.

Just a respectful reminder here: a discovered way to do something 
is not a designed way to do something.  Cleverness can take you a 
very very long way (Modern C++ anyone? :-) ) but it's unlikely 
that it will take you to appropriate simplicity.

Whatever decisions are made down the road, I'm happy that we're 
actively looking at extending D's safely accessible meta 
programming capabilites now.  Sounds like a great topic for 
beerconf-day-2.
Sep 26 2020
parent reply claptrap <clap trap.com> writes:
On Sunday, 27 September 2020 at 03:46:52 UTC, Bruce Carneal wrote:

 On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei 
 Alexandrescu wrote:


 We're in an early stage of exploration of discovered 
 capability, so I understand the excitement.  The C++ guys must 
 have been way excited about their meta programming discoveries 
 as well.

 Just a respectful reminder here: a discovered way to do 
 something is not a designed way to do something.  Cleverness 
 can take you a very very long way (Modern C++ anyone? :-) ) but 
 it's unlikely that it will take you to appropriate simplicity.


D: "Look ma I can peel an apple with a tin opener!"

Mom(C++): "That's my boy... the band-aids are in the usual place."



 Sep 27 2020



  parent reply Bruce Carneal <bcarneal gmail.com>  writes:


On Sunday, 27 September 2020 at 16:32:18 UTC, claptrap wrote:

 On Sunday, 27 September 2020 at 03:46:52 UTC, Bruce Carneal 
 wrote:

 On Sunday, 27 September 2020 at 02:31:36 UTC, Andrei 
 Alexandrescu wrote:


 We're in an early stage of exploration of discovered 
 capability, so I understand the excitement.  The C++ guys must 
 have been way excited about their meta programming discoveries 
 as well.

 Just a respectful reminder here: a discovered way to do 
 something is not a designed way to do something.  Cleverness 
 can take you a very very long way (Modern C++ anyone? :-) ) 
 but it's unlikely that it will take you to appropriate 
 simplicity.


 D: "Look ma I can peel an apple with a tin opener!"

 Mom(C++): "That's my boy... the band-aids are in the usual 
 place."


Andrei and others advanced us well beyond the mandated cleverness 
of C++ meta programming.  As Atila has put it, they helped make 
things "boring" through very good design.  I deeply admire that 
work.

I hope and believe that there is another such advance available 
to us in the form of type functions.  The recently discovered 
alternative, reify/dereify, appears to be equivalent in power but 
is, comparatively, baroque.

If the two approaches are actually equivalent in power, and in 
their ability to address the current template issues, then the 
baroque should be preferred if the language is closed to 
syntactic additions.

If the language is not closed, then choosing to displace the 
already prototyped type function capability with reify/dereify 
would represent a lost opportunity to administer another 
complexity smackdown.

Disclaimer: Andrei has stated, effectively, that I have little 
standing to opine on these issues.  It is true that while I use 
templates extensively I confine myself to very simple 
combinations thereof.  I am not experiencing the template pain 
reported by others.

Best wishes to those who continue to work in this area.  Here's 
hoping you lead us to a future that is extremely boring.



 Sep 27 2020



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/27/20 10:58 PM, Bruce Carneal wrote:

 Disclaimer: Andrei has stated, effectively, that I have little standing 
 to opine on these issues.


Sorry! Didn't mean to sound dismissive at all. Meant to just say, don't 
shun templates, embrace them and thrive.



 Sep 27 2020



  parent reply Stefan Koch <uplink.coder gmail.com>  writes:


On Monday, 28 September 2020 at 03:03:01 UTC, Andrei Alexandrescu 
wrote:

 On 9/27/20 10:58 PM, Bruce Carneal wrote:

 Disclaimer: Andrei has stated, effectively, that I have little 
 standing to opine on these issues.


 Sorry! Didn't mean to sound dismissive at all. Meant to just 
 say, don't shun templates, embrace them and thrive.


In practice template can quite literally explode though.
Which could be a valid reason to shun them.
You can embrace them, of course, and thrive.
The question is for how long you will thrive.



 Sep 28 2020



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com>  writes:


On 9/28/20 8:46 AM, Stefan Koch wrote:

 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei Alexandrescu wrote:

 On 9/27/20 10:58 PM, Bruce Carneal wrote:

 Disclaimer: Andrei has stated, effectively, that I have little 
 standing to opine on these issues.


 Sorry! Didn't mean to sound dismissive at all. Meant to just say, 
 don't shun templates, embrace them and thrive.


 
 In practice template can quite literally explode though.
 Which could be a valid reason to shun them.
 You can embrace them, of course, and thrive.
 The question is for how long you will thrive.


For a very long time judging by the success C++ is enjoying with them.



 Sep 28 2020



  parent reply Bruce Carneal <bcarneal gmail.com>  writes:


On Monday, 28 September 2020 at 14:16:20 UTC, Andrei Alexandrescu 
wrote:

 On 9/28/20 8:46 AM, Stefan Koch wrote:

 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei 
 Alexandrescu wrote:

 On 9/27/20 10:58 PM, Bruce Carneal wrote:

 Disclaimer: Andrei has stated, effectively, that I have 
 little standing to opine on these issues.


 Sorry! Didn't mean to sound dismissive at all. Meant to just 
 say, don't shun templates, embrace them and thrive.


 
 In practice template can quite literally explode though.
 Which could be a valid reason to shun them.
 You can embrace them, of course, and thrive.
 The question is for how long you will thrive.


 For a very long time judging by the success C++ is enjoying 
 with them.


As you note, by employing a variety of "best practices", of 
extra-language conventions, of one-level-of-indirection wrappers, 
of "human must certify this correct" mechanisms, the C++ 
community has indeed "thrived".

We've not settled for that meta programming drudgery, that 
friction, in the past.  You know better this better than anyone 
else on the planet.  I hope we don't "settle" going forward.



 Sep 28 2020



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com>  writes:


On 9/28/20 11:55 AM, Bruce Carneal wrote:

 On Monday, 28 September 2020 at 14:16:20 UTC, Andrei Alexandrescu wrote:

 On 9/28/20 8:46 AM, Stefan Koch wrote:

 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei Alexandrescu wrote:

 On 9/27/20 10:58 PM, Bruce Carneal wrote:

 Disclaimer: Andrei has stated, effectively, that I have little 
 standing to opine on these issues.


 Sorry! Didn't mean to sound dismissive at all. Meant to just say, 
 don't shun templates, embrace them and thrive.


 In practice template can quite literally explode though.
 Which could be a valid reason to shun them.
 You can embrace them, of course, and thrive.
 The question is for how long you will thrive.


 For a very long time judging by the success C++ is enjoying with them.


 
 As you note, by employing a variety of "best practices", of 
 extra-language conventions, of one-level-of-indirection wrappers, of 
 "human must certify this correct" mechanisms, the C++ community has 
 indeed "thrived".
 
 We've not settled for that meta programming drudgery, that friction, in 
 the past.  You know better this better than anyone else on the planet.  
 I hope we don't "settle" going forward.


(Not getting some of the uses of quotation marks.)

That's a bit backhanded because it implies I promote settling for meta 
programming drudgery. Did you mean to say that?

On the contrary, I find type reification interesting exactly because it 
takes you from said drudgery to familiar land - first-class values that 
can be manipulated in traditional manner. Should you need to get back to 
type-land, dereification helps with that. We can now draw from a large 
body of existing theory and practice.

https://en.wikipedia.org/wiki/Reification_(computer_science)



 Sep 28 2020



 next sibling parent reply claptrap <clap trap.com>  writes:


On Monday, 28 September 2020 at 16:57:19 UTC, Andrei Alexandrescu 
wrote:

 On 9/28/20 11:55 AM, Bruce Carneal wrote:

 On Monday, 28 September 2020 at 14:16:20 UTC, Andrei 
 Alexandrescu wrote:

 On 9/28/20 8:46 AM, Stefan Koch wrote:

 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei 
 Alexandrescu wrote:

 On 9/27/20 10:58 PM, Bruce Carneal wrote:

 Disclaimer: Andrei has stated, effectively, that I have 
 little standing to opine on these issues.


 Sorry! Didn't mean to sound dismissive at all. Meant to 
 just say, don't shun templates, embrace them and thrive.


 In practice template can quite literally explode though.
 Which could be a valid reason to shun them.
 You can embrace them, of course, and thrive.
 The question is for how long you will thrive.


 For a very long time judging by the success C++ is enjoying 
 with them.


 
 As you note, by employing a variety of "best practices", of 
 extra-language conventions, of one-level-of-indirection 
 wrappers, of "human must certify this correct" mechanisms, the 
 C++ community has indeed "thrived".
 
 We've not settled for that meta programming drudgery, that 
 friction, in the past.  You know better this better than 
 anyone else on the planet.  I hope we don't "settle" going 
 forward.


 (Not getting some of the uses of quotation marks.)

 That's a bit backhanded because it implies I promote settling 
 for meta programming drudgery. Did you mean to say that?

 On the contrary, I find type reification interesting exactly 
 because it takes you from said drudgery to familiar land - 
 first-class values that can be manipulated in traditional 
 manner. Should you need to get back to type-land, dereification 
 helps with that. We can now draw from a large body of existing 
 theory and practice.

 https://en.wikipedia.org/wiki/Reification_(computer_science)


Who's actually heard of reification outside of compiler writers? 
Ive been reading a ton of compiler related stuff lately, but not 
heard of it before now. It's seems like an implementation detail 
leaking out to me. Instead of first class types lets have 
whatever this stuff is cause we can get it on the cheap and its 
intellectually cool.

I guess my point is if you want meta-programming that just works 
as if it was regular programming, the reify / dereify just 
emphasises that it's two different domains because you have to 
move from one to the other. It's not unification, it's a bridge 
yes, but lets be honest the user shouldn't have to think about, 
it should just work. (Even if it makes the compiler writers job 
harder)



 Sep 28 2020



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 28.09.20 23:08, claptrap wrote:

 Instead of first class types


(Stefan's type functions do not give you first-class types. A 
first-class type you could put in a runtime variable.)



 Sep 28 2020



 next sibling parent  "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Mon, Sep 28, 2020 at 11:27:56PM +0200, Timon Gehr via Digitalmars-d wrote:

 On 28.09.20 23:08, claptrap wrote:

 Instead of first class types


 
 (Stefan's type functions do not give you first-class types. A
 first-class type you could put in a runtime variable.)


(typeid(T) is a first-class type (of sorts), which we already have
today.  All we really need today is a way to retrieve T from typeid(T)
at compile-time.)


T

-- 
By understanding a machine-oriented language, the programmer will tend to use a
much more efficient method; it is much closer to reality. -- D. Knuth



 Sep 28 2020



prev sibling next sibling parent  claptrap <clap trap.com>  writes:


On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:

 On 28.09.20 23:08, claptrap wrote:

 Instead of first class types


 (Stefan's type functions do not give you first-class types. A 
 first-class type you could put in a runtime variable.)


First class in the context of compile time meta programming i 
guess is more apt. I mean what people want is to manipulate these 
things at compile time so you want them to be first class in that 
context. Could just be called reified types to keep Andrei happy 
;)



 Sep 28 2020



prev sibling next sibling parent  Bruce Carneal <bcarneal gmail.com>  writes:


On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:

 On 28.09.20 23:08, claptrap wrote:

 Instead of first class types


 (Stefan's type functions do not give you first-class types. A 
 first-class type you could put in a runtime variable.)


As I understand it, they allow types to be put in to compile time 
variables, but not run time variables.  It's more than we had 
before type functions, but less than run-time first class.  What 
name do the type theory folk give to such a capability?



 Sep 28 2020



prev sibling  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:

 On 28.09.20 23:08, claptrap wrote:

 Instead of first class types


 (Stefan's type functions do not give you first-class types. A 
 first-class type you could put in a runtime variable.)


That's true. At least as it stands right now.
I _could_ give  the alias a physical form at runtime.
(By doing the same thing that the reify template does (or rather 
will eventually do), that is exporting _all_ the properties of 
the type into an object)

However I am not actually sure what the use of this would be.
All usage that I have seen, is in generating code, which is 
kindof useless if it does not happen at compile time.

Perhaps there are others?



 Sep 28 2020



 next sibling parent reply rikki cattermole <rikki cattermole.co.nz>  writes:


On 29/09/2020 12:37 PM, Stefan Koch wrote:

 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or rather will 
 eventually do), that is exporting _all_ the properties of the type into 
 an object)
 
 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is kindof 
 useless if it does not happen at compile time.
 
 Perhaps there are others?


Serialization!



 Sep 28 2020



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 29 September 2020 at 00:03:14 UTC, rikki cattermole 
wrote:

 On 29/09/2020 12:37 PM, Stefan Koch wrote:

 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or 
 rather will eventually do), that is exporting _all_ the 
 properties of the type into an object)
 
 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is 
 kindof useless if it does not happen at compile time.
 
 Perhaps there are others?


 Serialization!


Well that one you would kindof have to do in a template anyway, 
no?



 Sep 28 2020



  parent  rikki cattermole <rikki cattermole.co.nz>  writes:


On 29/09/2020 1:22 PM, Stefan Koch wrote:

 On Tuesday, 29 September 2020 at 00:03:14 UTC, rikki cattermole wrote:

 On 29/09/2020 12:37 PM, Stefan Koch wrote:

 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or rather will 
 eventually do), that is exporting _all_ the properties of the type 
 into an object)

 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is kindof 
 useless if it does not happen at compile time.

 Perhaps there are others?


 Serialization!


 
 Well that one you would kindof have to do in a template anyway, no?


No.

Pointer arithmetic alone could do most of the work.

But if the end abstraction produced had UDA's and methods available (so 
full blown reflection), you could do pretty amazing things.

This sort of stuff really doesn't need template instances to perform.

Same goes for web routers. You really don't need much info beyond some 
UDA's and to confirm the parameters/return type are correct. The rest is 
all better off being done at runtime.



 Sep 28 2020



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 29.09.20 01:37, Stefan Koch wrote:

 On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:

 On 28.09.20 23:08, claptrap wrote:

 Instead of first class types


 (Stefan's type functions do not give you first-class types. A 
 first-class type you could put in a runtime variable.)


 
 That's true. At least as it stands right now.
 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or rather will 
 eventually do), that is exporting _all_ the properties of the type into 
 an object)
 
 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is kindof 
 useless if it does not happen at compile time.
 
 Perhaps there are others?


It's not a first-class type if you can't declare a variable of that 
type. If this does not work, it's not first-class, it's syntax sugar for 
reification:

type t = int;
auto f = (t x)=>x;



 Sep 30 2020



 next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Thu, Oct 01, 2020 at 01:17:27AM +0200, Timon Gehr via Digitalmars-d wrote:

 On 29.09.20 01:37, Stefan Koch wrote:


[...]

 That's true. At least as it stands right now.
 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or rather
 will eventually do), that is exporting _all_ the properties of the
 type into an object)
 
 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is kindof
 useless if it does not happen at compile time.
 
 Perhaps there are others?


 
 It's not a first-class type if you can't declare a variable of that
 type. If this does not work, it's not first-class, it's syntax sugar
 for reification:
 
 type t = int;
 auto f = (t x)=>x;


I think there's a lot of value to be had in making typeid(T) as the
reification of T.  At compile-time, we treat it specially by augmenting
it with the ability to do things that can only be done at compile-time,
such as recover T given typeid(T) (pass it into a template parameter,
construct new types out of it, etc.). At runtime, it reverts to the
current behaviour of typeid(T).

Only trouble is, according to Andrei, typeid has been written over so
many times that even Walter doesn't understand how it works anymore.
Meaning there are probably weird corner cases and quirky behaviours that
we may not want to duplicate at compile-time. So I dunno, this seems
like a roadblock to further progress.  (It really makes one wish for a
gradual transition to D3 where we can correct such deeply-rooted flaws
that may otherwise be uncorrectible.)


T

-- 
To provoke is to call someone stupid; to argue is to call each other stupid.



 Sep 30 2020



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/30/20 7:36 PM, H. S. Teoh wrote:

 I think there's a lot of value to be had in making typeid(T) as the
 reification of T.  At compile-time, we treat it specially by augmenting
 it with the ability to do things that can only be done at compile-time,
 such as recover T given typeid(T) (pass it into a template parameter,
 construct new types out of it, etc.). At runtime, it reverts to the
 current behaviour of typeid(T).
 
 Only trouble is, according to Andrei, typeid has been written over so
 many times that even Walter doesn't understand how it works anymore.
 Meaning there are probably weird corner cases and quirky behaviours that
 we may not want to duplicate at compile-time. So I dunno, this seems
 like a roadblock to further progress.


Yah, that would be my favorite by far. It's a one-liner change in the 
definition of the language. "Since version x.xx, the result of typeid() 
is usable during compilation." It's also the best /kind/ of change to 
the language, i.e. lifts a gratuitous restriction that prevents 
composition of distinct features (here, ctfe and typeid).

A promising path according to Walter is to develop a parallel enhanced 
feature - newtypeid! :o) - and then deprecate typeid.



 Sep 30 2020



  parent  Bruce Carneal <bcarneal gmail.com>  writes:


On Thursday, 1 October 2020 at 01:49:02 UTC, Andrei Alexandrescu 
wrote:

 On 9/30/20 7:36 PM, H. S. Teoh wrote:

 I think there's a lot of value to be had in making typeid(T) 
 as the
 reification of T.  At compile-time, we treat it specially by 
 augmenting
 it with the ability to do things that can only be done at 
 compile-time,
 such as recover T given typeid(T) (pass it into a template 
 parameter,
 construct new types out of it, etc.). At runtime, it reverts 
 to the
 current behaviour of typeid(T).
 
 Only trouble is, according to Andrei, typeid has been written 
 over so
 many times that even Walter doesn't understand how it works 
 anymore.
 Meaning there are probably weird corner cases and quirky 
 behaviours that
 we may not want to duplicate at compile-time. So I dunno, this 
 seems
 like a roadblock to further progress.


 Yah, that would be my favorite by far. It's a one-liner change 
 in the definition of the language. "Since version x.xx, the 
 result of typeid() is usable during compilation." It's also the 
 best /kind/ of change to the language, i.e. lifts a gratuitous 
 restriction that prevents composition of distinct features 
 (here, ctfe and typeid).

 A promising path according to Walter is to develop a parallel 
 enhanced feature - newtypeid! :o) - and then deprecate typeid.


Glad to see that you're open to changing the language in order to 
achieve better power/complexity ratio.  Right there with you on 
that sentiment!

Two questions:

1) How many additional additions do you estimate you'll need for 
this? and

2) How do the type function addition(s) compare?  Are they also 
of the "best kind of change" variety?



 Sep 30 2020



prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr wrote:

 On 29.09.20 01:37, Stefan Koch wrote:

 On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:

 On 28.09.20 23:08, claptrap wrote:

 Instead of first class types


 (Stefan's type functions do not give you first-class types. A 
 first-class type you could put in a runtime variable.)


 
 That's true. At least as it stands right now.
 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or 
 rather will eventually do), that is exporting _all_ the 
 properties of the type into an object)
 
 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is 
 kindof useless if it does not happen at compile time.
 
 Perhaps there are others?


 It's not a first-class type if you can't declare a variable of 
 that type. If this does not work, it's not first-class, it's 
 syntax sugar for reification:

 type t = int;
 auto f = (t x)=>x;


Yes I do understand that.
I was wondering about practical usecases.
As far as I an aware, if I made the leap to first class types, 
that would make all usage of them into static polymorphism. 
(equivalent to templates)
And with that all the issues come back.



 Sep 30 2020



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 01.10.20 04:49, Stefan Koch wrote:

 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr wrote:

 On 29.09.20 01:37, Stefan Koch wrote:

 On Monday, 28 September 2020 at 21:27:56 UTC, Timon Gehr wrote:

 On 28.09.20 23:08, claptrap wrote:

 Instead of first class types


 (Stefan's type functions do not give you first-class types. A 
 first-class type you could put in a runtime variable.)


 That's true. At least as it stands right now.
 I _could_ give  the alias a physical form at runtime.
 (By doing the same thing that the reify template does (or rather will 
 eventually do), that is exporting _all_ the properties of the type 
 into an object)

 However I am not actually sure what the use of this would be.
 All usage that I have seen, is in generating code, which is kindof 
 useless if it does not happen at compile time.

 Perhaps there are others?


 It's not a first-class type if you can't declare a variable of that 
 type. If this does not work, it's not first-class, it's syntax sugar 
 for reification:

 type t = int;
 auto f = (t x)=>x;


 
 Yes I do understand that.
 I was wondering about practical usecases.
 As far as I an aware, if I made the leap to first class types, that 
 would make all usage of them into static polymorphism. (equivalent to 
 templates)
 And with that all the issues come back.


Well, no, the restriction to compile-time-known values makes types not 
first-class. E.g., this is something you could do with first-class types 
but not with templates: (type T)=>(T x)=>x;

(In terms of compiler implementation, the runtime representation of `T` 
would contain all information that is necessary to figure out calling 
conventions of functions that take a T, it would contain sizeof, 
pointers to destructor/postblit, etc, so basically it's typeid.)

Templates are an untyped macro language on top of a simple type system 
without any form of polymorphism. Types are first-class values only 
inside that macro language, but the macro language is evaluated only at 
compile time. I.e., templates are a tool for code generation that is 
integrated with the type system, but they are not really a part of the 
type system.



 Oct 01 2020



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 10/1/2020 12:48 AM, Timon Gehr wrote:

 (In terms of compiler implementation, the runtime representation of `T` would 
 contain all information that is necessary to figure out calling conventions of 
 functions that take a T, it would contain sizeof, pointers to 
 destructor/postblit, etc, so basically it's typeid.)


I had looked into this. Unfortunately, many aspects of types are simply too 
complex to try and represent at run time. For example, is type T1 implicitly 
convertible to T2? This seemingly simple question is very complex. Yes, `alias 
this` makes it much worse :-/

I initially had high hopes for typeid back when it was originally designed 20 
years ago. It ably fulfills its original purpose of enabling the GC and 
associative arrays in a language with no templates.

I don't see much future for it other than being kept around for legacy 
compatibility.



 Oct 01 2020



 next sibling parent  Stefan Koch <uplink.coder gmail.com>  writes:


On Thursday, 1 October 2020 at 08:56:58 UTC, Walter Bright wrote:

 I had looked into this. Unfortunately, many aspects of types 
 are simply too complex to try and represent at run time. For 
 example, is type T1 implicitly convertible to T2? This 
 seemingly simple question is very complex. Yes, `alias this` 
 makes it much worse :-/



The my latest post please!
The example on type functions with the conversion matrix.

This is all you need:

     override void visit(IsExp e)
     {
         auto targe = e.targ.isTypeExpression();
         if(!targe)
         {
             e.error("is expressions within type functions may 
only use type expressions");
             result = ErrorExp.get();
             return ;
         }
         auto targ = ctfeInterpret(targe.exp);
         auto te = targ.isTypeExp();

         result = IntegerExp.createBool(te && te.type && 
te.type.ty != Terror);

         auto tspece = e.tspec ? e.tspec.isTypeExpression() : null;
         auto tspec = tspece ? ctfeInterpret(tspece.exp) : null;
         auto ts = tspec ? tspec.isTypeExp() : null;

         // handling of == and &&
         // See IsExp handling in expressionsem.d
         if (e.tspec && !e.id && !(e.parameters && 
e.parameters.dim))
         {
             if (e.tok == TOK.colon)
             {
                 result = 
IntegerExp.createBool(te.type.implicitConvTo(ts.type) != 
MATCH.nomatch);
             }
             else if(e.tok == TOK.equal)
             {
                 result = 
IntegerExp.createBool(te.type.equals(ts.type));
             }
         }
         else if (e.tok2 != TOK.reserved)
         {
             e.error("Complex pattern matching forms of is 
expressions are not supported in TypeFunctions yet");
             result = IntegerExp.createBool(false);
         }
     }



 Oct 01 2020



prev sibling  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 01.10.20 10:56, Walter Bright wrote:

 On 10/1/2020 12:48 AM, Timon Gehr wrote:

 (In terms of compiler implementation, the runtime representation of 
 `T` would contain all information that is necessary to figure out 
 calling conventions of functions that take a T, it would contain 
 sizeof, pointers to destructor/postblit, etc, so basically it's typeid.)


 
 I had looked into this. Unfortunately, many aspects of types are simply 
 too complex to try and represent at run time. For example, is type T1 
 implicitly convertible to T2? This seemingly simple question is very 
 complex. Yes, `alias this` makes it much worse :-/
 ...


The compiler can figure it out, so you *could* just put that logic into 
druntime. However, for first-class types, you don't need to check 
implicit conversions at run time, therefore this would not be necessary.


 I initially had high hopes for typeid back when it was originally 
 designed 20 years ago. It ably fulfills its original purpose of enabling 
 the GC and associative arrays in a language with no templates.
 ...


And a monomorphic type system.


 I don't see much future for it other than being kept around for legacy 
 compatibility.


I was explaining conditions for types to be "first-class". So unless 
that is a goal, typeid might not see a revival.



 Oct 01 2020



prev sibling  parent reply Atila Neves <atila.neves gmail.com>  writes:


On Thursday, 1 October 2020 at 02:49:43 UTC, Stefan Koch wrote:

 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr 
 wrote:

 On 29.09.20 01:37, Stefan Koch wrote:

 [...]


 It's not a first-class type if you can't declare a variable of 
 that type. If this does not work, it's not first-class, it's 
 syntax sugar for reification:

 type t = int;
 auto f = (t x)=>x;


 Yes I do understand that.
 I was wondering about practical usecases.


import std.algorithm: filter, equal;

type[] types = [int, uint, long, ulong];
auto size4 = types.filter!(a => a.sizeof == 4);
assert(equal(size4, [int, uint]));

No more std.meta.Filter.


 As far as I an aware, if I made the leap to first class types, 
 that would make all usage of them into static polymorphism. 
 (equivalent to templates)
 And with that all the issues come back.


I can't see how that's the case.



 Oct 08 2020



 next sibling parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 October 2020 at 09:44:33 UTC, Atila Neves wrote:

 On Thursday, 1 October 2020 at 02:49:43 UTC, Stefan Koch wrote:

 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr 
 wrote:

 On 29.09.20 01:37, Stefan Koch wrote:

 [...]


 It's not a first-class type if you can't declare a variable 
 of that type. If this does not work, it's not first-class, 
 it's syntax sugar for reification:

 type t = int;
 auto f = (t x)=>x;


 Yes I do understand that.
 I was wondering about practical usecases.


 import std.algorithm: filter, equal;

 type[] types = [int, uint, long, ulong];
 auto size4 = types.filter!(a => a.sizeof == 4);
 assert(equal(size4, [int, uint]));

 No more std.meta.Filter.


 As far as I an aware, if I made the leap to first class types, 
 that would make all usage of them into static polymorphism. 
 (equivalent to templates)
 And with that all the issues come back.


 I can't see how that's the case.


What you just described is what type functions do.
The point that with first class type you could do also this:

// cat type_list.txt: 'int, uint, long, ulong';
auto type_txt = readText("type_list.txt");
type[] types = type_txt.parseTypes()
auto size4 = types.filter!(a => a.sizeof == 4);
assert(equal(size4, [int, uint]));
type[0] myVar1;
type[1] myVar2;



 Oct 08 2020



  parent reply Atila Neves <atila.neves gmail.com>  writes:


On Thursday, 8 October 2020 at 09:51:28 UTC, Stefan Koch wrote:

 On Thursday, 8 October 2020 at 09:44:33 UTC, Atila Neves wrote:

 On Thursday, 1 October 2020 at 02:49:43 UTC, Stefan Koch wrote:

 [...]


 import std.algorithm: filter, equal;

 type[] types = [int, uint, long, ulong];
 auto size4 = types.filter!(a => a.sizeof == 4);
 assert(equal(size4, [int, uint]));

 No more std.meta.Filter.


 [...]


 I can't see how that's the case.


 What you just described is what type functions do.
 The point that with first class type you could do also this:

 // cat type_list.txt: 'int, uint, long, ulong';
 auto type_txt = readText("type_list.txt");
 type[] types = type_txt.parseTypes()
 auto size4 = types.filter!(a => a.sizeof == 4);
 assert(equal(size4, [int, uint]));
 type[0] myVar1;
 type[1] myVar2;


There's no IO at compile-time.



 Oct 08 2020



  parent  Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 October 2020 at 10:05:10 UTC, Atila Neves wrote:

 On Thursday, 8 October 2020 at 09:51:28 UTC, Stefan Koch wrote:

 On Thursday, 8 October 2020 at 09:44:33 UTC, Atila Neves wrote:

 [...]


 What you just described is what type functions do.
 The point that with first class type you could do also this:

 // cat type_list.txt: 'int, uint, long, ulong';
 auto type_txt = readText("type_list.txt");
 type[] types = type_txt.parseTypes()
 auto size4 = types.filter!(a => a.sizeof == 4);
 assert(equal(size4, [int, uint]));
 type[0] myVar1;
 type[1] myVar2;


 There's no IO at compile-time.


Exactly!
first class type can be determined at runtime!



 Oct 08 2020



prev sibling  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 October 2020 at 09:44:33 UTC, Atila Neves wrote:

 On Thursday, 1 October 2020 at 02:49:43 UTC, Stefan Koch wrote:

 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr 
 wrote:

 On 29.09.20 01:37, Stefan Koch wrote:

 [...]


 It's not a first-class type if you can't declare a variable 
 of that type. If this does not work, it's not first-class, 
 it's syntax sugar for reification:

 type t = int;
 auto f = (t x)=>x;


 Yes I do understand that.
 I was wondering about practical usecases.


 import std.algorithm: filter, equal;

 type[] types = [int, uint, long, ulong];
 auto size4 = types.filter!(a => a.sizeof == 4);
 assert(equal(size4, [int, uint]));

 No more std.meta.Filter.


 As far as I an aware, if I made the leap to first class types, 
 that would make all usage of them into static polymorphism. 
 (equivalent to templates)
 And with that all the issues come back.


 I can't see how that's the case.


This is the correct type function way of doing it:
enum type[] types = makeTypeArray(int, uint, long, ulong);
enum type[] size4 = types.filter!((type a) => a.sizeof == 4);
static assert(equal!((type a, type b) => a is b) (size4, 
makeTypeArray(int, uint)));

Currently type[] fails is isInputRange though and therefore the 
phobos filter won't instantiate.
I am debugging this.



 Oct 08 2020



  parent  Sebastiaan Koppe <mail skoppe.eu>  writes:


On Thursday, 8 October 2020 at 10:16:10 UTC, Stefan Koch wrote:

 This is the correct type function way of doing it:
 enum type[] types = makeTypeArray(int, uint, long, ulong);
 enum type[] size4 = types.filter!((type a) => a.sizeof == 4);
 static assert(equal!((type a, type b) => a is b) (size4, 
 makeTypeArray(int, uint)));

 Currently type[] fails is isInputRange though and therefore the 
 phobos filter won't instantiate.
 I am debugging this.


This is plain awesome.

The fact that it reuses the filter algorithm is a sign that both 
filter and type functions are correctly designed.

Who could ever imagined that a template would be instantiated 
with a lambda that takes a type-value as an argument. WAT!? That 
is some high-order stuff, and perfectly readable.

Andrei's de/reify is awesome as well, it has the advantage of 
working with existing language constructs, but it isn't as simple.



 Oct 08 2020



prev sibling  parent reply Bruce Carneal <bcarneal gmail.com>  writes:


On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr wrote:

 On 29.09.20 01:37, Stefan Koch wrote:

 ...


 It's not a first-class type if you can't declare a variable of 
 that type. If this does not work, it's not first-class, it's 
 syntax sugar for reification:

 type t = int;
 auto f = (t x)=>x;


Can anything computable using just the source as input be 
considered a first class type?

If so, what do we call type variables that can not be?



 Sep 30 2020



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 01.10.20 07:37, Bruce Carneal wrote:

 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr wrote:

 On 29.09.20 01:37, Stefan Koch wrote:

 ...


 It's not a first-class type if you can't declare a variable of that 
 type. If this does not work, it's not first-class, it's syntax sugar 
 for reification:

 type t = int;
 auto f = (t x)=>x;


 
 Can anything computable using just the source as input be considered a 
 first class type?
 
 If so, what do we call type variables that can not be?
 


Unfortunately I am not sure understand the question. What is an example 
of a "type variable that cannot be computed using just the source"? 
(Which source?)

Maybe this is related to what you mean?

type t = readln().strip()=="int"?int:double;
auto f = (t x)=>x;



 Oct 01 2020



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 10/1/2020 12:51 AM, Timon Gehr wrote:

 Maybe this is related to what you mean?
 
 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;



As D is a statically typed language, I don't see how that can ever work.



 Oct 01 2020



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 01.10.20 10:59, Walter Bright wrote:

 On 10/1/2020 12:51 AM, Timon Gehr wrote:

 Maybe this is related to what you mean?

 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;


 
 
 As D is a statically typed language, I don't see how that can ever work.


The example is easy to type check statically. In terms of ABI, you just 
have to make sure that `f` can find the context pointer without already 
knowing the value of `t`, then it can reconstruct everything it needs to 
know.



 Oct 01 2020



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 10/1/2020 5:41 AM, Timon Gehr wrote:

 On 01.10.20 10:59, Walter Bright wrote:

 On 10/1/2020 12:51 AM, Timon Gehr wrote:

 Maybe this is related to what you mean?

 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;



 As D is a statically typed language, I don't see how that can ever work.


 
 The example is easy to type check statically. In terms of ABI, you just have
to 
 make sure that `f` can find the context pointer without already knowing the 
 value of `t`, then it can reconstruct everything it needs to know.


I think you described D's "class" type. To make it work with int and double, 
you'd need to "box" them with a class. Like Java.



 Oct 01 2020



  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 02.10.20 06:04, Walter Bright wrote:

 On 10/1/2020 5:41 AM, Timon Gehr wrote:

 On 01.10.20 10:59, Walter Bright wrote:

 On 10/1/2020 12:51 AM, Timon Gehr wrote:

 Maybe this is related to what you mean?

 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;



 As D is a statically typed language, I don't see how that can ever work.


 The example is easy to type check statically. In terms of ABI, you 
 just have to make sure that `f` can find the context pointer without 
 already knowing the value of `t`, then it can reconstruct everything 
 it needs to know.


 
 I think you described D's "class" type. To make it work with int and 
 double, you'd need to "box" them with a class. Like Java.


No, you'd still pass them by value in appropriate registers or on the 
stack, but you only figure out at run time where the argument is 
actually located.



 Oct 02 2020



prev sibling  parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com>  writes:


On Thursday, 1 October 2020 at 08:59:04 UTC, Walter Bright wrote:

 On 10/1/2020 12:51 AM, Timon Gehr wrote:

 Maybe this is related to what you mean?
 
 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;



 As D is a statically typed language, I don't see how that can 
 ever work.


Flow analysis. You can deduce that t can only be int|double at 
the second line and then expand the code into a switch over the 
type (encoded as enum values).

It cannot work with separate compilation though.



 Oct 01 2020



prev sibling  parent  Bruce Carneal <bcarneal gmail.com>  writes:


On Thursday, 1 October 2020 at 07:51:28 UTC, Timon Gehr wrote:

 On 01.10.20 07:37, Bruce Carneal wrote:

 On Wednesday, 30 September 2020 at 23:17:27 UTC, Timon Gehr 
 wrote:

 On 29.09.20 01:37, Stefan Koch wrote:

 ...


 It's not a first-class type if you can't declare a variable 
 of that type. If this does not work, it's not first-class, 
 it's syntax sugar for reification:

 type t = int;
 auto f = (t x)=>x;


 
 Can anything computable using just the source as input be 
 considered a first class type?
 
 If so, what do we call type variables that can not be?
 


 Unfortunately I am not sure understand the question. What is an 
 example of a "type variable that cannot be computed using just 
 the source"? (Which source?)

 Maybe this is related to what you mean?

 type t = readln().strip()=="int"?int:double;
 auto f = (t x)=>x;


Yes.

The common understanding of a "statically typed language" is 
quite a bit more restrictive than "anything which can be 
determined from a fixed 'source' input" yet that is the bounding 
condition, it seems, for "static" compilation.

I lack the terminology to discuss this succinctly so I asked.  
Note, my lack of understanding is sufficient that I may not have 
posed a well formed question.  Illumination is requested.



 Oct 01 2020



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com>  writes:


On 9/28/20 5:08 PM, claptrap wrote:

 
 Who's actually heard of reification outside of compiler writers? Ive 
 been reading a ton of compiler related stuff lately, but not heard of it 
 before now. It's seems like an implementation detail leaking out to me. 
 Instead of first class types lets have whatever this stuff is cause we 
 can get it on the cheap and its intellectually cool.


Reification is first class types.



 Sep 28 2020



  parent reply claptrap <clap trap.com>  writes:


On Monday, 28 September 2020 at 21:55:45 UTC, Andrei Alexandrescu 
wrote:

 On 9/28/20 5:08 PM, claptrap wrote:

 
 Who's actually heard of reification outside of compiler 
 writers? Ive been reading a ton of compiler related stuff 
 lately, but not heard of it before now. It's seems like an 
 implementation detail leaking out to me. Instead of first 
 class types lets have whatever this stuff is cause we can get 
 it on the cheap and its intellectually cool.


 Reification is first class types.


If you have to litter your code with refiy / dereify then it's 
not first class. It's not a language feature but a library one. 
Hence not first class.



 Sep 28 2020



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/28/20 6:13 PM, claptrap wrote:

 On Monday, 28 September 2020 at 21:55:45 UTC, Andrei Alexandrescu wrote:

 On 9/28/20 5:08 PM, claptrap wrote:

 Who's actually heard of reification outside of compiler writers? Ive 
 been reading a ton of compiler related stuff lately, but not heard of 
 it before now. It's seems like an implementation detail leaking out 
 to me. Instead of first class types lets have whatever this stuff is 
 cause we can get it on the cheap and its intellectually cool.


 Reification is first class types.


 
 If you have to litter your code with refiy / dereify then it's not first 
 class. It's not a language feature but a library one. Hence not first 
 class.


I agree. Reification is a transformation that takes a non-first-class 
type and produces a first-class value.



 Sep 28 2020



prev sibling  parent reply Bruce Carneal <bcarneal gmail.com>  writes:


On Monday, 28 September 2020 at 16:57:19 UTC, Andrei Alexandrescu 
wrote:

 On 9/28/20 11:55 AM, Bruce Carneal wrote:

 On Monday, 28 September 2020 at 14:16:20 UTC, Andrei 
 Alexandrescu wrote:

 On 9/28/20 8:46 AM, Stefan Koch wrote:

 On Monday, 28 September 2020 at 03:03:01 UTC, Andrei 
 Alexandrescu wrote:

 On 9/27/20 10:58 PM, Bruce Carneal wrote:

 Disclaimer: Andrei has stated, effectively, that I have 
 little standing to opine on these issues.


 Sorry! Didn't mean to sound dismissive at all. Meant to 
 just say, don't shun templates, embrace them and thrive.


 In practice template can quite literally explode though.
 Which could be a valid reason to shun them.
 You can embrace them, of course, and thrive.
 The question is for how long you will thrive.


 For a very long time judging by the success C++ is enjoying 
 with them.


 
 As you note, by employing a variety of "best practices", of 
 extra-language conventions, of one-level-of-indirection 
 wrappers, of "human must certify this correct" mechanisms, the 
 C++ community has indeed "thrived".
 
 We've not settled for that meta programming drudgery, that 
 friction, in the past.  You know better this better than 
 anyone else on the planet.  I hope we don't "settle" going 
 forward.


 (Not getting some of the uses of quotation marks.)

 That's a bit backhanded because it implies I promote settling 
 for meta programming drudgery. Did you mean to say that?


My gosh, no!  My apologies for leaving you with that impression.  
You were and are one of the major players in the C++ 
friction/drudgery smackdown that is the D language.  I was 
attempting to say that there's an opportunity for another 
language level complexity smackdown that I believe covers, en 
passant, your reify/dereify use case.


 On the contrary, I find type reification interesting exactly 
 because it takes you from said drudgery to familiar land - 
 first-class values that can be manipulated in traditional 
 manner. Should you need to get back to type-land, dereification 
 helps with that. We can now draw from a large body of existing 
 theory and practice.


OK.  I'll really try to do better here.  Zero disrespect intended.

When you leave the type system behind, when you reify, you must 
assume responsibility for constraints that were previously, and 
seamlessly, taken care of by the type system.  The drudgery, the 
friction, follows directly from the decision to escape from the 
type system (reify) rather than remain within it (type functions).

I think of it as being similar to CT functions vs templates.  
Within CT functions you've got the type system on your side.  
Everybody loves CT functions because everything "just works" as 
you'd expect.  Near zero additional semantic load.
Wonderfully boring.

Within templates, on the other hand, you'd better get your 
big-boy britches on because it's pretty much all up to you 
pardner!  (manually inserted constraints, serious tension between 
generality and debugability, composition difficulties, 
lazy/latent bugs in the general forms, localization difficulties, 
...)

If language additions like type functions are off the table, then 
we're left with LDMs and you've produced what looks like a dandy 
in reify/dereify.  If we have to step outside the language, if 
language additions are just not in the cards any more, then 
something like reify/dereify will be the way to go.

I hope that we've not hit that wall just yet but even if we have 
D will remain, in my opinion, head and shoulders above anything 
else out there.  It is a truly wonderful language.  I am very 
grateful for the work you, Walter, and many many others have put 
in to make it so.  (most recent standout, Mathias!  I'm a 
-preview=in fan)

Finally, I'd love to hear your comments on type functions vs 
reify/dereify.  It's certainly possible that I've missed 
something.  Maybe it's a type functions+ solution we should be 
seeking (type functions for everything they can do, and some LDM 
for anything beyond their capability).



 Sep 28 2020



 next sibling parent  Bruce Carneal <bcarneal gmail.com>  writes:


On Monday, 28 September 2020 at 22:46:55 UTC, Bruce Carneal wrote:

 On Monday, 28 September 2020 at 16:57:19 UTC, Andrei 
 Alexandrescu wrote:

 On 9/28/20 11:55 AM, Bruce Carneal wrote:




...

 If language additions like type functions are off the table, 
 then we're left with LDMs


Language Deficiency Mitigation


 and you've produced what looks like a dandy in reify/dereify.  
 If we have to step outside the language, if language additions 
 are just not in the cards any more, then something like 
 reify/dereify will be the way to go.





 Sep 28 2020



prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/28/20 6:46 PM, Bruce Carneal wrote:

 
 When you leave the type system behind, when you reify, you must assume 
 responsibility for constraints that were previously, and seamlessly, 
 taken care of by the type system.  The drudgery, the friction, follows 
 directly from the decision to escape from the type system (reify) rather 
 than remain within it (type functions).
 
 I think of it as being similar to CT functions vs templates. Within CT 
 functions you've got the type system on your side. Everybody loves CT 
 functions because everything "just works" as you'd expect.  Near zero 
 additional semantic load.
 Wonderfully boring.
 
 Within templates, on the other hand, you'd better get your big-boy 
 britches on because it's pretty much all up to you pardner!  (manually 
 inserted constraints, serious tension between generality and 
 debugability, composition difficulties, lazy/latent bugs in the general 
 forms, localization difficulties, ...)
 
 If language additions like type functions are off the table, then we're 
 left with LDMs and you've produced what looks like a dandy in 
 reify/dereify.  If we have to step outside the language, if language 
 additions are just not in the cards any more, then something like 
 reify/dereify will be the way to go.
 
 I hope that we've not hit that wall just yet but even if we have D will 
 remain, in my opinion, head and shoulders above anything else out 
 there.  It is a truly wonderful language.  I am very grateful for the 
 work you, Walter, and many many others have put in to make it so.  (most 
 recent standout, Mathias!  I'm a -preview=in fan)
 
 Finally, I'd love to hear your comments on type functions vs 
 reify/dereify.  It's certainly possible that I've missed something.  
 Maybe it's a type functions+ solution we should be seeking (type 
 functions for everything they can do, and some LDM for anything beyond 
 their capability).


Years ago, I was in a panel with Simon Peyton-Jones, the creator of 
Haskell. Nicest guy around, not to mention an amazing researcher. This 
question was asked: "What is the most important principle of programming 
language design?"

His answer was so powerful, it made me literally forget everybody else's 
answer, including my own. (And there were no slouches in that panel, 
save for me: Martin Odersky, Guido van Rossum, Bertrand Meyer.) He said 
the following:

"The most important principle in a language design is to define a small 
core of essential primitives. All necessary syntactic sugar lowers to 
core constructs. Do everything else in libraries."

(He didn't use the actual term "lowering", which is familiar to our 
community, but rather something equivalent such as "reduces".)

That kind of killed the panel, in a good way. Because most other 
questions on programming language design and implementation simply made 
his point shine quietly. Oh yes, if you had a small core and build on it 
you could do this easily. And that. And the other. In a wonderfully meta 
way, all questions got instantly lowered to simpler versions of 
themselves. I will never forget that experience.

D breaks that principle in several places. It has had a cavalier 
attitude to using magic tricks in the compiler to get things going, at 
the expense of fuzzy corners and odd limit cases. Look at hashtables. 
Nobody can create an equivalent user-defined type, and worse, nobody 
knows exactly why. (I recall vaguely it has something to do, among many 
other things, with qualified keys that are statically-sized arrays. 
Hacks in the compiler make those work, but D's own type system rejects 
the equivalent code. So quite literally D's type system cannot verify 
its own capabilities.)

Or take implicit conversions. They aren't fully documented, and the only 
way to figure things out is to read most of the 7193 lines of 
https://github.com/dlang/dmd/blob/master/src/dmd/mtype.d. That's not a 
small core with a little sugar on top.

Or take the foreach statement. Through painful trial and error, somebody 
figured out all possible shapes of foreach, and defined `each` to 
support most:

https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L904

What should have been a simple forwarding problem took 190 lines that 
could be characterized as very involved. And mind you, it doesn't 
capture all cases because per 
https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L1073:

// opApply with >2 parameters. count the delegate args.
// only works if it is not templated (otherwise we cannot count the args)

I know that stuff (which will probably end on my forehead) because I 
went and attempted to improve things a bit in 
https://github.com/dlang/phobos/pull/7638/files#diff-0d6463fc6f41c5fb7
4300832e3135f5R805, 
which attempts to simplify matters by reducing the foreach cases to 
seven shapes.

To paraphrase Alan Perlis: "If your foreach has seven possible shapes, 
you probably missed some."

Over time, things did get better. We became adept of things such as 
lowering, and we require precision in DIPs.

I very strongly believe that this complexity, if unchecked, will kill 
the D language. It will sink under its own weight. We need a precise, 
clear definition of the language, we need a principled approach to 
defining and extending features. The only right way to extend D at this 
point is to remove the odd failure modes created by said cavalier 
approach to doing things. Why can't I access typeid during compilation, 
when I can access other pointers? Turns out typeid is a palimpsest that 
has been written over so many times, even Walter cannot understand it. 
He told me plainly to forget about trying to use typeid and write 
equivalent code from scratch instead. That code has achieved lifetime 
employment.

And no compiler tricks. I am very, very opposed to Walter's penchant to 
reach into his bag of tricks whenever a difficult problem comes about. 
Stop messing with the language! My discussions with him on such matters 
are like trying to talk a gambler out of the casino.

That is a long way to say I am overwhelmingly in favor of in-language 
solutions as opposed to yet another addition to the language. To me, if 
I have an in-language solution, it's game, set, and match. No need for 
language changes, thank you very much. An in-language solution doesn't 
only mean no addition is needed, but more importantly it means that the 
language has sufficient power to offer D coders means to solve that and 
many other problems.

I almost skipped a heartbeat when Adam mentioned - mid-sentence! - just 
yet another little language addition to go with that DIP for 
interpolated strings. It would make things nicer. You know what? I think 
I'd rather live without that DIP.

So I'm interested in exploring reification mainly - overwhelmingly - 
because it already works in the existing language. To the extent it has 
difficulties, it's because of undue limitations in CTFE and 
introspection. And these are problems we must fix anyway. So it's all a 
virtuous circle. Bringing myself to write `reify` and `dereify` around 
my types is a very small price to pay for all that.



 Sep 28 2020



 next sibling parent reply Paul Backus <snarwin gmail.com>  writes:


On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:

 D breaks that principle in several places. It has had a 
 cavalier attitude to using magic tricks in the compiler to get 
 things going, at the expense of fuzzy corners and odd limit 
 cases. Look at hashtables. Nobody can create an equivalent 
 user-defined type, and worse, nobody knows exactly why. (I 
 recall vaguely it has something to do, among many other things, 
 with qualified keys that are statically-sized arrays. Hacks in 
 the compiler make those work, but D's own type system rejects 
 the equivalent code. So quite literally D's type system cannot 
 verify its own capabilities.)


[...]

 I very strongly believe that this complexity, if unchecked, 
 will kill the D language. It will sink under its own weight. We 
 need a precise, clear definition of the language, we need a 
 principled approach to defining and extending features. The 
 only right way to extend D at this point is to remove the odd 
 failure modes created by said cavalier approach to doing 
 things. Why can't I access typeid during compilation, when I 
 can access other pointers? Turns out typeid is a palimpsest 
 that has been written over so many times, even Walter cannot 
 understand it. He told me plainly to forget about trying to use 
 typeid and write equivalent code from scratch instead. That 
 code has achieved lifetime employment.


I think D is already too deep in the tar pit to ever fully 
extricate itself. All of those special cases and "magic tricks" 
are there because, as you say, code depends on them to work. 
Getting read of them would break the world a hundred times over. 
If you're going to do that, you might as well start a new 
language from scratch, no?

The best we can hope for is to nail down a precise description of 
all the fiddly corner cases, so that we know what D's semantics 
*actually* are, rather than what the current spec pretends they 
are. But even that is likely to be infeasible in practice--who 
wants to volunteer to be the one to go over, say, 
funcDeclarationSemantic with a fine-toothed comb, especially 
after what happened to the last guy? [1]

The fact is, there's no incentive for anyone to do this kind of 
work other than masochism and sheer stubbornness. The amount of 
effort it would take, and the amount of time that would pass 
before you saw any benefit, are simply too high. We can talk 
about principles and good engineering all we want, but we can't 
expect people to voluntarily act against their own interest.

Maybe we can make some improvements at the margins, but my 
prediction is that the overwhelming majority of D's complexity is 
here to stay, whether we like it or not. The best we can do is 
try to offer enough positive value to D programmers to offset its 
negative effects. And if it doesn't work out, well, there's 
always Rust. ;)

[1] https://github.com/dlang/dmd/pull/8668



 Sep 28 2020



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/29/20 12:56 AM, Paul Backus wrote:

 The best we can hope for is to nail down a precise description of all 
 the fiddly corner cases, so that we know what D's semantics *actually* 
 are, rather than what the current spec pretends they are.


Definitely. I'd go even further and call that "survival". C++ has had 
tremendous success with meticulously documenting all of the odd things 
they baked in pre-standardization. And then it was a matter of updating 
things, and it all got better. Most importantly, there was and is a 
group of people who understood the vital importance of that.


 But even that 
 is likely to be infeasible in practice--who wants to volunteer to be the 
 one to go over, say, funcDeclarationSemantic with a fine-toothed comb, 
 especially after what happened to the last guy? [1]


Actually Walter made a lot of efforts to do that, too, as he very well 
understands the importance of correct specification. But nobody cared 
about his related PRs, which probably are flapping in the wind to this 
day. He has some work on defining the object model, too, which has also 
been ignored. A kernel of 3-5 strong souls to commit to this would be 
very necessary. Walter can't do all of this alone. Now he can't even do 
it formally because of the "no single person" rule.

I should add I've been unfairly harsh about Walter's inclination to do 
tricks in the compiler. That applies a lot more to the early days than 
now. As of now our views are aligned to a large proportion on core 
language vs. library features.


 The fact is, there's no incentive for anyone to do this kind of work 
 other than masochism and sheer stubbornness. The amount of effort it 
 would take, and the amount of time that would pass before you saw any 
 benefit, are simply too high. We can talk about principles and good 
 engineering all we want, but we can't expect people to voluntarily act 
 against their own interest.
 
 Maybe we can make some improvements at the margins, but my prediction is 
 that the overwhelming majority of D's complexity is here to stay, 
 whether we like it or not. The best we can do is try to offer enough 
 positive value to D programmers to offset its negative effects. And if 
 it doesn't work out, well, there's always Rust. ;)


I think things can be managed, but documentation and control are a must. 
FWIW I don't think Rust fares that well in that area - it's a very 
complex language, with the complexity mostly in the wrong places leaving 
mediocre handling of all that's interesting, and its rigorous 
documentation is greatly outmatched by its marketing brochures.



 Sep 29 2020



prev sibling next sibling parent reply claptrap <clap trap.com>  writes:


On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:

 On 9/28/20 6:46 PM, Bruce Carneal wrote:



 Years ago, I was in a panel with Simon Peyton-Jones, the 
 creator of Haskell. Nicest guy around, not to mention an 
 amazing researcher. This question was asked: "What is the most 
 important principle of programming language design?"

 His answer was so powerful, it made me literally forget 
 everybody else's answer, including my own. (And there were no 
 slouches in that panel, save for me: Martin Odersky, Guido van 
 Rossum, Bertrand Meyer.) He said the following:

 "The most important principle in a language design is to define 
 a small core of essential primitives. All necessary syntactic 
 sugar lowers to core constructs. Do everything else in 
 libraries."


I agree with pretty much all of what you've just said FWIW.

However...

How do you decide what is an essential language primitive? Or 
what is necessary syntactic sugar? I mean syntactic sugar almost 
by definition isnt necessary, it's just window dressing.

It could be argued that compile time first class types is an 
essential primitive for meta-programming?



 Sep 29 2020



  parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com>  writes:


On Tuesday, 29 September 2020 at 09:07:39 UTC, claptrap wrote:

 How do you decide what is an essential language primitive? Or 
 what is necessary syntactic sugar? I mean syntactic sugar 
 almost by definition isnt necessary, it's just window dressing.


The for loop and the while loop is syntactic sugar in C-like 
languages.
So clearly syntactic sugar is more than window dressing.

The usual way to define a minimal imperative language is to have 
concepts like "block" and "restart block" (covers  both looping 
and continue) and conditionals.

You can essentially do away with most concepts until you have 
just have a closure with coroutine capabilities + one conditional 
instruction. Two such tiny imperative languages are Self and 
Beta. So you basically merge the concept of a block and an object 
as well.. A constructor/destructor pair is just a single 
coroutine that has suspended.

However the minimal core language tend to be an on-paper 
construct for non-research languages. Theoretical. The compiler 
will most certainly have many more concepts than the minimal 
language to get better compilation times.



 Sep 29 2020



 next sibling parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com>  writes:


On Tuesday, 29 September 2020 at 10:03:12 UTC, Ola Fosheim 
Grøstad wrote:

 conditional instruction. Two such tiny imperative languages are 
 Self and Beta. So you basically merge the concept of a block 
 and an object as well.. A constructor/destructor pair is just a 
 single coroutine that has suspended.


That last sentence is hypothetical. You could construct a 
language that only consists of concurrent closures. Would it be 
slow? Only if the optimizer isn't advanced enough, but in reality 
compiler authors depend on heuristics (common programming 
patterns) so it is quite difficult to get such languages to 
perform well in practice.

Compilers are sadly not very good at high level optimization... 
Which is why languages ship with pre-canned constructs, libraries 
and runtimes...



 Sep 29 2020



prev sibling  parent reply claptrap <clap trap.com>  writes:


On Tuesday, 29 September 2020 at 10:03:12 UTC, Ola Fosheim 
Grøstad wrote:

 On Tuesday, 29 September 2020 at 09:07:39 UTC, claptrap wrote:

 How do you decide what is an essential language primitive? Or 
 what is necessary syntactic sugar? I mean syntactic sugar 
 almost by definition isnt necessary, it's just window dressing.


 The for loop and the while loop is syntactic sugar in C-like 
 languages.
 So clearly syntactic sugar is more than window dressing.


OK yeah bad analogy. My point is that if you say "only what 
syntactic sugar is necessary" it doesn't really help because what 
does "necessary" mean? If you take it literally you dont need for 
and while loops because you have if and goto. But pretty much 
every language has them, so obviously there's more too it.

its a great quote and you can see the wisdom in it but there's a 
lot lurking under the surface of the words "essential" and 
"necessary"



 Sep 29 2020



  parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com>  writes:


On Tuesday, 29 September 2020 at 13:07:35 UTC, claptrap wrote:

 its a great quote and you can see the wisdom in it but there's 
 a lot lurking under the surface of the words "essential" and 
 "necessary"


Indeed. If you only have goto and if, would programmers then 
write code where common patterns occur or would the compiler 
writer just optimize for structures that have the properties of 
noise as programmers might end up writing very different code 
structures?

I strongly suspect that one advantage of having a standard 
library and high level features is that programmers then tend to 
write code in similar patterns that compiler writers can take 
advantage of those patterns.



 Sep 29 2020



prev sibling next sibling parent  Jackel <jackel894_394 gmail.com>  writes:


On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:

 Or take the foreach statement. Through painful trial and error, 
 somebody figured out all possible shapes of foreach, and 
 defined `each` to support most:

 https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L904

 What should have been a simple forwarding problem took 190 
 lines that could be characterized as very involved. And mind 
 you, it doesn't capture all cases because per 
 https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L1073:

 // opApply with >2 parameters. count the delegate args.
 // only works if it is not templated (otherwise we cannot count 
 the args)

 I know that stuff (which will probably end on my forehead) 
 because I went and attempted to improve things a bit in 
 https://github.com/dlang/phobos/pull/7638/files#diff-0d6463fc6f41c5fb7
4300832e3135f5R805, which attempts to simplify matters by reducing the foreach
cases to seven shapes.

 To paraphrase Alan Perlis: "If your foreach has seven possible 
 shapes, you probably missed some."

 Over time, things did get better. We became adept of things 
 such as lowering, and we require precision in DIPs.


I think this is a mischaracterization of a problem. I'm not 
really a fan of how ranges are implemented, especially how UFCS 
is used to chain functions together. When they start to become 
10+ lines of a chain it is almost impossible to profile such 
code; with tools. Let alone try to understand what is happening, 
it would require to know how each function is implemented and how 
it expands and ends up being executed. Some things are lazy, 
somethings aren't; eg requiring `.array` after a `.map`. This is 
where I feel there shouldn't even be a `each` to begin with, just 
use a for loop.

Now that's my opinion, I deal with low level code that needs to 
be optimized and those UFSC chains are likened to scripting 
languages that usually don't need to worry about optimization. 
For short chains I don't think it's that big of a deal, it's 
usually easy enough to figure out what is going on and not enough 
is happening for too much to go wrong.


 And mind you, it doesn't capture all cases because per 
 https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L1073:

 // opApply with >2 parameters. count the delegate args.
 // only works if it is not templated (otherwise we cannot count 
 the args)


If you use a template of `opApply()` you lose type inference (as 
you do with regular templates). It's a limitation of the feature. 
You'd have the same problem if you have an overload of 
`opApply()` with delegates that have the same number of 
arguments; it won't know which one to pick. If you want to 
support both you'd just need 1 other case, but honestly if you 
have a template `opApply`, that's just bad practice (due to 
losing type inference). It doesn't need to be supported. The only 
reason I've ever wanted a templated opApply() is for auto 
attribute inference, but that's not worth losing the auto type 
inference you get with foreach. There was a suggestion for adding 
a new feature, or rather altering the behavior to allow for both, 
but I don't think it was well received.



 I know that stuff (which will probably end on my forehead) 
 because I went and attempted to improve things a bit in 
 https://github.com/dlang/phobos/pull/7638/files#diff-0d6463fc6f41c5fb7
4300832e3135f5R805, which attempts to simplify matters by reducing the foreach
cases to seven shapes.


You are going the wrong way about simplifying the code. Why not 
add an opApply() property to the internal container types (array, 
assoc array, etc...)? Shouldn't be that difficult. Cases 4, 5, 6, 
and 7 would collapse into one then.

Cases 1, 2, and 3 aren't really different ways of using foreach. 
They are just conveniences for ranges. That tuple expansion is 
odd though, doesn't seem to be documented and it's a feature that 
foreach doesn't have (unlike the second parameter for the index). 
Doesn't need to be there, but someone probably uses it somewhere 
now.

So really it's just 2 cases with an additional case to support an 
index counter. That fits the bill, case 1: foreach, case 2: 
phobos ranges.

It's not as bad as you make it out to be, for this case.



 Sep 29 2020



prev sibling next sibling parent reply Patrick Schluter <Patrick.Schluter bbox.fr>  writes:


On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:

 On 9/28/20 6:46 PM, Bruce Carneal wrote:


[..]

 Years ago, I was in a panel with Simon Peyton-Jones, the 
 creator of Haskell. Nicest guy around, not to mention an 
 amazing researcher. This question was asked: "What is the most 
 important principle of programming language design?"

 His answer was so powerful, it made me literally forget 
 everybody else's answer, including my own. (And there were no 
 slouches in that panel, save for me: Martin Odersky, Guido van 
 Rossum, Bertrand Meyer.) He said the following:

 "The most important principle in a language design is to define 
 a small core of essential primitives. All necessary syntactic 
 sugar lowers to core constructs. Do everything else in 
 libraries."


Sorry to not be completely convinced on the pertinence of that 
statement. For the language designer it might well be true, for 
the users of the language I'm not completely sold. There are two 
languages that imho fit that description very well, they are both 
very powerfull and extremely elegant, but boy do they suck when 
you just want to get shit done: Lisp and Forth. And (I know, one 
should not start a sentence with and) it had been mentioned often 
in this forum, it is this shear flexibility that are in the way 
of wider adoption as there are so many ways of doing things, but 
none of them completely right.
Your AA example is a good one. If it wasn't in the language, it 
would be the same issue with AA as in any other language that 
defines it in libraries: several different incompatible 
restricted implementations.

Just my uninformed 2 cents.



 Sep 29 2020



 next sibling parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com>  writes:


On Tuesday, 29 September 2020 at 12:02:21 UTC, Patrick Schluter 
wrote:

 Sorry to not be completely convinced on the pertinence of that 
 statement. For the language designer it might well be true, for 
 the users of the language I'm not completely sold. There are 
 two languages that imho fit that description very well, they 
 are both very powerfull and extremely elegant, but boy do they 
 suck when you just want to get shit done: Lisp and Forth.


Most research languages try to fit the description, but they 
rarely reach production. But in essence you are right. The key 
problem isn't functional, but syntactical. When I wrote in Beta I 
could do everything as easily as in most imperative languages, 
but everything looked the same. So you end up looking a landscape 
of pebbles in various colours, but that makes the landscape less 
visible. Compare that to landscape with roads, lakes and 
mountains.

Template programming suffers from the same problem. You can 
create fancy semantics, but the syntax becomes very hard on the 
user. It leads to less legible code as code that represent 
different concepts don't stand out.

You can say the same about some C code bases.

So template programming is fun, but it will suck until there is a 
language that finds a good way to provide syntax extensions.


 Your AA example is a good one. If it wasn't in the language, it 
 would be the same issue with AA as in any other language that 
 defines it in libraries: several different incompatible 
 restricted implementations.


There are at least two ways to generalize AA, you can either have 
a language mechanism for serialization or mandate that all types 
provide a hash function and identity comparison (equality). I 
don't see how having a compiler built in is qualitatively 
different from having a standard library implementation in the 
general case.



 Sep 29 2020



prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com>  writes:


On 9/29/20 8:02 AM, Patrick Schluter wrote:

 Your AA example is a good one. If it wasn't in the language, it would be 
 the same issue with AA as in any other language that defines it in 
 libraries: several different incompatible restricted implementations.


AA is not that far from being a library type.

If you discount the magic casting that AAs can do (which isn't all that 
useful), there is one (yes ONE) feature that AAs do that a library type 
cannot. And that is an implicit adding of an element in certain cases. 
The reason AAs can do it is because the compiler calls a different 
opIndex (it's not called opIndex, but some extern(C) hook) when it's in 
this mode, but only for AAs.

In fact, AAs are mostly already a library type, thanks to a lot of 
incremental effort over the years to lower the implementation to 
templates. Except for this one thing.

-Steve



 Sep 29 2020



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/29/20 8:55 AM, Steven Schveighoffer wrote:

 On 9/29/20 8:02 AM, Patrick Schluter wrote:

 Your AA example is a good one. If it wasn't in the language, it would 
 be the same issue with AA as in any other language that defines it in 
 libraries: several different incompatible restricted implementations.


 
 AA is not that far from being a library type.
 
 If you discount the magic casting that AAs can do (which isn't all that 
 useful), there is one (yes ONE) feature that AAs do that a library type 
 cannot. And that is an implicit adding of an element in certain cases. 
 The reason AAs can do it is because the compiler calls a different 
 opIndex (it's not called opIndex, but some extern(C) hook) when it's in 
 this mode, but only for AAs.
 
 In fact, AAs are mostly already a library type, thanks to a lot of 
 incremental effort over the years to lower the implementation to 
 templates. Except for this one thing.


Thanks, good to know. I now recall I helped a little with that. (Was it 
the length function? empty? some range-related addition?) This supports 
the original point - it took much effort over years to disentangle and 
undo all this magic.



 Sep 29 2020



prev sibling next sibling parent reply bachmeier <no spam.net>  writes:


On Tuesday, 29 September 2020 at 12:02:21 UTC, Patrick Schluter 
wrote:


 Your AA example is a good one. If it wasn't in the language, it 
 would be the same issue with AA as in any other language that 
 defines it in libraries: several different incompatible 
 restricted implementations.


That's a good example. IMO there's a difference between a 
language and bits and pieces that can be put together to call 
random libraries.



 Sep 29 2020



 next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com>  writes:


On Tuesday, 29 September 2020 at 13:38:25 UTC, bachmeier wrote:

 That's a good example. IMO there's a difference between a 
 language and bits and pieces that can be put together to call 
 random libraries.


Heh, not really. It is not unusual for languages to have a 
minimal core and default include the most commonly used parts of 
their standard library. The programmer can often not tell the 
difference.



 Sep 29 2020



  parent reply bachmeier <no spam.net>  writes:


On Tuesday, 29 September 2020 at 13:44:55 UTC, Ola Fosheim 
Grøstad wrote:

 On Tuesday, 29 September 2020 at 13:38:25 UTC, bachmeier wrote:

 That's a good example. IMO there's a difference between a 
 language and bits and pieces that can be put together to call 
 random libraries.


 Heh, not really. It is not unusual for languages to have a 
 minimal core and default include the most commonly used parts 
 of their standard library. The programmer can often not tell 
 the difference.


The statement was this:


 "The most important principle in a language design is to define 
 a small core of essential primitives. All necessary syntactic 
 sugar lowers to core constructs. Do everything else in 
 libraries."


It's possible I'm wrong, but it says to me that the language 
developers should design the language worrying only about that 
'small core of essential primitives'. Maybe 'libraries' refers to 
the standard library only, but that's not clear to me.



 Sep 29 2020



  parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com>  writes:


On Tuesday, 29 September 2020 at 18:53:26 UTC, bachmeier wrote:

 On Tuesday, 29 September 2020 at 13:44:55 UTC, Ola Fosheim 
 Grøstad wrote:
 It's possible I'm wrong, but it says to me that the language 
 developers should design the language worrying only about that 
 'small core of essential primitives'. Maybe 'libraries' refers 
 to the standard library only, but that's not clear to me.


Yes, it is not terribly clear what people are talking about in 
this thread because many different layers of a language is being 
lumped together. So let me try to paint up my view of these 
layers:

1. On the one hand you have the type system (I guess you can call 
that the constraints you can put on constructs) where you 
probably want a small set of "axioms" from which you cannot 
deduce contradictions (soundness). A small core set makes it 
possible to prove that it is consistent.

2. On the other hand you have the conceptual "quasi-theoretical" 
building blocks from which you can describe everything that is 
expressible in the language. Keeping this small is also valuable 
for the same reasons. It is easier to reason about and ensure 
correctness/soundness and build a language that is easy to grasp. 
Basically, this often embeds a "modelling idea". (everything 
should be expressible as an object, a list, a logical proposition 
or a mix of two core concepts etc). If you look at the Eiffel 
page they claim that it is a methodology supported by a language. 
The same was Simula and Beta in a sense, languages supporting 
modelling with objects. So the minimal core language is often 
tied to a modelling-idea.

3. Then you have the construct that is actually implemented in 
the compiler, which reflects the "quasi-theoretical" language, 
but might be done differently for implementation reasons. Though 
it should obey the "laws" of the quasi-theoretical language.

4. Then you have the syntax (with syntactical sugar) that allows 
user to express that. Since it is a massive undertaking to let 
libraries define new syntax it often is a lot of "syntactical 
sugar", basically a few words generating many constructs in the 
underlying "theoretical" language.

5. Then you have the language that programmers use, which 
basically is what programmers often call "idiomatic" styles of 
programming for that particular language and that includes 
frequently used libraries. When newbies write bad code and 
complain loudly the old timers will point out that they are in 
essence "abusing the language construct" and "should learn to 
speak the language before them correctly".

So, basically, a programming language design can let the 
programmer believe that the compiler provides a string concept, 
although the underlying "theoretical language" only has lists and 
enumerations. So it strings might look different from lists in 
the source, but on a "theoretical level" it isn't.

What makes this more complicated is that the "true 
quasi-theoretical" language may emerge over time. As an example: 
I think the underlying emerging core concept in C++ is vastly 
different from C, that is, the most important concept in C++ has 
turned out to be RAII in combination with exceptions. Objects 
with deterministic constructors and destructors. If one were to 
do C++ v2, then one probably should start with that and see how 
small the core language could be made.

In practice compilers tend to bolt on poorly thought out "built 
ins" that do not share the properties of the "theoretical 
language" and then they stand out as sore thumbs that exhibit 
behaviour that is perceived as bug-like by programmers.

Until recently the "aesthetics" of programming language design 
has typically been to find the "one true language" that never 
will have to change. However, languages like typescript pretty 
much show that is better if programmers can configure the 
semantics to their specific use case in a config file. Because 
the context keeps changing.

Maybe one can modify a language like D to do the same, so that 
you could get AA or even data-science libraries imported by 
default so it feels more like a scripting language with that 
config. Clearly, embedded programming and data-science 
programming have different basic needs. Seems reasonable that one 
could have seemingly builtin AA in one config geared towards 
scripting-like programming.



 Sep 29 2020



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/29/20 9:38 AM, bachmeier wrote:

 On Tuesday, 29 September 2020 at 12:02:21 UTC, Patrick Schluter wrote:
 

 Your AA example is a good one. If it wasn't in the language, it would 
 be the same issue with AA as in any other language that defines it in 
 libraries: several different incompatible restricted implementations.


 
 That's a good example. IMO there's a difference between a language and 
 bits and pieces that can be put together to call random libraries.


There is no doubt to Walter or myself that the way built-in AAs were 
done in D is a resounding failure.

The way they should have been done was a templated library type 
AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That 
would have allowed the implementation to improve along with the rest of 
the language and its library.

(Of course there's historical context to this, i.e. AAs predate 
templates and then they just continued being there, increasingly 
standing out like the proverbial sore thumb as everything else evolved 
around them.)

As things are now, AAs don't quite behave like any other things. They 
are a poor cousin to ranges. No container infrastructure grew around 
them. They are dramatically less efficient than library hashtables.

Worst of all, these problems have been well-known for years and years 
and the progress on fixing them has moved at a glacial pace.



 Sep 29 2020



 next sibling parent reply bachmeier <no spam.net>  writes:


On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei 
Alexandrescu wrote:

 On 9/29/20 9:38 AM, bachmeier wrote:

 On Tuesday, 29 September 2020 at 12:02:21 UTC, Patrick 
 Schluter wrote:
 

 Your AA example is a good one. If it wasn't in the language, 
 it would be the same issue with AA as in any other language 
 that defines it in libraries: several different incompatible 
 restricted implementations.


 
 That's a good example. IMO there's a difference between a 
 language and bits and pieces that can be put together to call 
 random libraries.


 There is no doubt to Walter or myself that the way built-in AAs 
 were done in D is a resounding failure.


If so, it's a pretty good resounding failure, because I like them 
and they would most likely not be available to me now if they 
hadn't been built into the language. Long ago when I used Common 
Lisp, there was the incredible FSet library that provided 
functional collections kind of like Clojure. Unfortunately few 
people knew about them or used them, and it doesn't look like 
anything has been done with them in a long time: 
https://common-lisp.net/project/fset/ Sometimes a library 
solution works, but it's in general a risky proposition to rely 
on libraries for such a fundamental feature.

If AAs had been introduced as a library in 2012, the repo would 
probably have a couple hundred open issues and the last 
development would have been in 2017.


 The way they should have been done was a templated library type 
 AssociativeArray!(K, V) with the V[K] type syntax sugar on top. 
 That would have allowed the implementation to improve along 
 with the rest of the language and its library.


Does anything prevent the introduction of a new library solution? 
If it's good enough, it can eventually be added to the language.

I'm not going to pretend to be a language designer. My experience 
as a language user is that a big set of core functionality works 
better than a small set plus libraries. It's possible I don't 
understand your proposal.



 Sep 29 2020



 next sibling parent  "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Tue, Sep 29, 2020 at 06:43:23PM +0000, bachmeier via Digitalmars-d wrote:

 On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:


[...]

 There is no doubt to Walter or myself that the way built-in AAs were
 done in D is a resounding failure.


 
 If so, it's a pretty good resounding failure, because I like them and
 they would most likely not be available to me now if they hadn't been
 built into the language. [...]


Same here.  Built-in AA's were one of the big reasons that drew me to D.
Now granted, the way they are implemented perhaps leaves a lot to be
desired, but having built-in support for AA's IMO is a fundamental
language feature.  Whether it's implemented in the compiler or the
standard library is secondary, but it's not something that should be
left to an external 3rd party library.

It would definitely be a detriment if D had followed the same path as
C++ in having hash tables in the standard library (extremely late in the
history of C++, I might add -- something I will probably hold forever
against C++ :-P), but requiring a lot of manual effort to use: no
default hash function for built-in types like structs or ints, need to
manually declare all sorts of things just to instantiate a lousy
hashtable, having inordinately long derived type names just to name
something that in D is as simple as `MyStruct[string]`, and just general
needless gratuitous barriers to usage.

Some level of language support is necessary to grease all these
usability gears: at the very least the language must support nice, easy
syntax to declare AA's -- even if the actual implementation is delegated
to a library type in the standard library.


[...]

 The way they should have been done was a templated library type
 AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That
 would have allowed the implementation to improve along with the rest
 of the language and its library.


 
 Does anything prevent the introduction of a new library solution? If
 it's good enough, it can eventually be added to the language.


[...]

A lot of effort has been put in to move the AA implementation into
object.d, and the current implementation has also undergone several
rounds of improvement, code quality wise and performance wise.  But
there remain some magic bits that currently cannot be duplicated in
"user space", so to speak, (as opposed to compiler space).

One is the handling of const/immutable on keys, which, in spite of being
magically handled by the compiler, nevertheless suffers from flaws: to
prevent key aliasing problems that cause inconsistencies in the AA, as
of several releases ago AA keys are now implicitly `const`. However,
this does not actually solve the problem -- because mutable can
implicitly convert to const, so the aliasing problem still exists. It
would be nice to make it immutable, but that breaks existing code in
some places, and in any case switching from const to immutable isn't
something that can be done as a gradual deprecation -- it's all or
nothing.  But regardless of the correctness issue, the handling of
const/immutable qualifiers on AA keys depends on compiler magic, and
currently either isn't expressible in library code, or else is so
cumbersome it's simply impractical.

Another is the magic handling of:

	Data[string][string] aa;
	aa["abc"]["def"] = Data(...);

For this, we need opIndexCreate, which we currently don't have:

	https://issues.dlang.org/show_bug.cgi?id=7753

There may be one or two other issues that I can't recall off the top of
my head, but basically, we've come a long way since the original AA
implementation, and now only need to remove a few more magical barriers
before a fully-library AA solution is possible in druntime.

//

Of course, all of this confirms what Andrei is saying about too much
magic being boiled into the compiler: it can do stuff library code
cannot, and because of that, fixing problems in AAs requires digging
deep into compiler innards.  Worse yet, some of the rules governing AA
magic in the compiler isn't fully compatible with the rules library code
is subjected to; and trying to replicate this behaviour in library code
is very painful, or outright impossible.


T

-- 
Doubt is a self-fulfilling prophecy.



 Sep 29 2020



prev sibling  parent reply Steven Schveighoffer <schveiguy gmail.com>  writes:


On 9/29/20 2:43 PM, bachmeier wrote:

 On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:

 There is no doubt to Walter or myself that the way built-in AAs were 
 done in D is a resounding failure.


 
 If so, it's a pretty good resounding failure, because I like them and 
 they would most likely not be available to me now if they hadn't been 
 built into the language. Long ago when I used Common Lisp, there was the 
 incredible FSet library that provided functional collections kind of 
 like Clojure. Unfortunately few people knew about them or used them, and 
 it doesn't look like anything has been done with them in a long time: 
 https://common-lisp.net/project/fset/ Sometimes a library solution 
 works, but it's in general a risky proposition to rely on libraries for 
 such a fundamental feature.
 
 If AAs had been introduced as a library in 2012, the repo would probably 
 have a couple hundred open issues and the last development would have 
 been in 2017.
 

 The way they should have been done was a templated library type 
 AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That 
 would have allowed the implementation to improve along with the rest 
 of the language and its library.


 
 Does anything prevent the introduction of a new library solution? If 
 it's good enough, it can eventually be added to the language.
 
 I'm not going to pretend to be a language designer. My experience as a 
 language user is that a big set of core functionality works better than 
 a small set plus libraries. It's possible I don't understand your proposal.


I think there's some disconnect here -- AAs as a language feature are a 
resounding success. They way they are implemented in the language is 
not. I wouldn't call it a failure, but it definitely is not as useful as 
a fully supported library type could be.

-Steve



 Sep 29 2020



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 9/29/2020 12:30 PM, Steven Schveighoffer wrote:

 I think there's some disconnect here -- AAs as a language feature are a 
 resounding success. They way they are implemented in the language is not. I 
 wouldn't call it a failure, but it definitely is not as useful as a fully 
 supported library type could be.


Yes, that is a better statement of my opinion on it.

The implementation's problems are, of course, because D's AAs were designed 
before there were templates and before there was const and immutable.



 Sep 30 2020



prev sibling  parent reply Adam D. Ruppe <destructionator gmail.com>  writes:


On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei 
Alexandrescu wrote:

 As things are now, AAs don't quite behave like any other things.


The biggest difference I see is how null works, the keyword.

void foo(string[string] aa) {}

foo(null); // works.

But that's outright impossible with a library struct.

yes, I'm bringing up my wish for implicit construction again :P



 Sep 29 2020



  parent  Steven Schveighoffer <schveiguy gmail.com>  writes:


On 9/29/20 3:32 PM, Adam D. Ruppe wrote:

 On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:

 As things are now, AAs don't quite behave like any other things.


 
 The biggest difference I see is how null works, the keyword.
 
 void foo(string[string] aa) {}
 
 foo(null); // works.
 
 But that's outright impossible with a library struct.
 
 yes, I'm bringing up my wish for implicit construction again :P


This is certainly an odd duck. I don't really want to have the language 
provide all-out implicit construction, but possibly a mechanism to say 
"I can be implicitly constructed from null" would be useful.

null is in its own category of literals, it can change into just about 
anything (class, pointer, array associative array). Another nice place 
this would be useful is std.typecons.Nullable.

-Steve



 Sep 29 2020



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/29/20 8:02 AM, Patrick Schluter wrote:

 On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei Alexandrescu wrote:

 On 9/28/20 6:46 PM, Bruce Carneal wrote:


 
 [..]

 Years ago, I was in a panel with Simon Peyton-Jones, the creator of 
 Haskell. Nicest guy around, not to mention an amazing researcher. This 
 question was asked: "What is the most important principle of 
 programming language design?"

 His answer was so powerful, it made me literally forget everybody 
 else's answer, including my own. (And there were no slouches in that 
 panel, save for me: Martin Odersky, Guido van Rossum, Bertrand Meyer.) 
 He said the following:

 "The most important principle in a language design is to define a 
 small core of essential primitives. All necessary syntactic sugar 
 lowers to core constructs. Do everything else in libraries."


 Sorry to not be completely convinced on the pertinence of that 
 statement. For the language designer it might well be true, for the 
 users of the language I'm not completely sold. There are two languages 
 that imho fit that description very well, they are both very powerfull 
 and extremely elegant, but boy do they suck when you just want to get 
 shit done: Lisp and Forth. And (I know, one should not start a sentence 
 with and) it had been mentioned often in this forum, it is this shear 
 flexibility that are in the way of wider adoption as there are so many 
 ways of doing things, but none of them completely right.
 Your AA example is a good one. If it wasn't in the language, it would be 
 the same issue with AA as in any other language that defines it in 
 libraries: several different incompatible restricted implementations.
 
 Just my uninformed 2 cents.


Completely agree with the sentiment! Also, being informed in language 
internals paraphernalia is not at stake here.

I've studied classical guitar for many years. Got a couple of prizes, 
too, but never broke out of the amateur ranks. Nevertheless, I got to 
meet a few professional players and also played to a variety of audiences.

One thing that's obvious to a music performer that is not at all clear 
to listeners is that deficiencies in performance show up as second-order 
effects.

What do I mean by that? I mean that the audience doesn't sit there with 
the sheet music in hand following the notes and the directions written. 
They won't put the finger on paper and say, hey, hey, what happened to 
the mezzoforte in here? No, they just sit back and listen to the music. 
If they like it, they don't know why. If they don't, they don't know 
why. Music critics write all about some performer's technique, touché, 
style, rubato - but listeners are like, yeah, that's beautiful or isn't. 
An exegete knows that that came out poorly because the performer moved 
their wrist too much.

No bug report will come as "D uses too many tricks in the compiler that 
are inaccessible to programs, and that's very problematic." No, they'll 
all come as problems with surprising limitations and inconsistencies, 
frustration in use, platform dependencies, deficiencies in 
implementation, lack of orthogonality, very concrete and very many bugs, 
bugs, and bugs.

Deficiencies in programming language design show up as second-order effects.

There's also something about limits that needs clarification. Small 
language kernel being good does not mean the smallest language kernel is 
the best. (In guitar technique economy of motion is a key ideal, i.e. 
one should minimize movement of both left and right hand. That of course 
doesn't mean zero movement is best, because that would produce no 
sound.) So keeping the core small and reducing sugar to it is an ideal 
to live into, not a rigid requirement.



 Sep 29 2020



prev sibling next sibling parent reply Stefan Koch <uplink.coder gmail.com>  writes:


On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:
I'd rather live without that DIP.

 So I'm interested in exploring reification mainly - 
 overwhelmingly - because it already works in the existing 
 language. To the extent it has difficulties, it's because of 
 undue limitations in CTFE and introspection. And these are 
 problems we must fix anyway. So it's all a virtuous circle. 
 Bringing myself to write `reify` and `dereify` around my types 
 is a very small price to pay for all that.


Alright mate,
Let's assume we had the minimal langauge.
Very small core.
All features are library.

You end up with LISP.
What's with optimization?
Optimization, just in case you don't know,
is nothing but a semantic invariant transform, which is based on 
proving that observed semantics are actually invariant.
Let's take:
int mul(
   int x,
   int y
) {
   int result = 0;
   for(int i = 0; i < x; i++)
   {
     result += y;
   }
   return result;
}

A sufficiently advanced optimizer should be able to most every 
calls to `mul` into an imul instruction.

If that is the case there is no need to burden the language with 
a * operator and you could make the parser much simpler.

and Indeed ldc with -O3 will generate the following sequence for 
the function:

	imull	%esi, %edi // y *= x
	xorl	%eax, %eax // result = 0
	testl	%esi, %esi // if (y)
	cmovgl	%edi, %eax // if (x > 0) result = x;
         retq               // return result;

There are a few superfluous checks, for example the loop entrance 
condition (x > 0)
So how does this compare if we write a function that only does * ?

int mul(
   int x,
   int y
) {
   return x*y;
}

The code we get, from the same compiler and the same compiler 
flags (-O3) is this:

	movl	%edi, %eax // result = x;
	imull	%esi, %eax // result *= y;
	retq               // return result;

Which removes one test, and one xor, furthermore one move becomes 
unconditional.
(That reduces code size and the load on the branch predictor.)



 Sep 29 2020



  parent reply Stefan Koch <uplink.coder gmail.com>  writes:


On Tuesday, 29 September 2020 at 12:38:42 UTC, Stefan Koch wrote:

 [continuing]


So we had this:


 int mul(
   int x,
   int y
 ) {
   int result = 0;
   for(int i = 0; i < x; i++)
   {
     result += y;
   }
   return result;
 }



And the code generated was

	imull	%esi, %edi // y *= x
	xorl	%eax, %eax // result = 0
	testl	%esi, %esi // if (y)
	cmovgl	%edi, %eax // if (x > 0) result = x;
         retq               // return result;

On first glance these instructions seem superfluous
But we have to understand that the compiler had no way of knowing 
that we actually wanted to generate a mere multilply.

For example there is the loop entrance check.
Why is it there?

We can proof it's necessity quite easily.
- let's assume we did not have it.
In that case we are allowd to enter loop body even if (x == 0)
so we would execute result += y; once.
Which would cause mul(0, 32) to result in 32. And that would be a 
violation of the semantics.

So there's no way to get rid of the check.
And this code will always perform worse than an intrinsic 'mul' 
which the compiler knows about.



 Sep 29 2020



  parent reply Dukc <ajieskola gmail.com>  writes:


On Tuesday, 29 September 2020 at 12:54:53 UTC, Stefan Koch wrote:

 So we had this:


 int mul(
   int x,
   int y
 ) {
   int result = 0;
   for(int i = 0; i < x; i++)
   {
     result += y;
   }
   return result;
 }



 And the code generated was

 	imull	%esi, %edi // y *= x
 	xorl	%eax, %eax // result = 0
 	testl	%esi, %esi // if (y)
 	cmovgl	%edi, %eax // if (x > 0) result = x;
         retq               // return result;

 On first glance these instructions seem superfluous
 But we have to understand that the compiler had no way of 
 knowing that we actually wanted to generate a mere multilply.

 For example there is the loop entrance check.
 Why is it there?

 We can proof it's necessity quite easily.
 [snip]


I don't know assembly, but I can see an even simpler reason why 
the compiler can't generate a multiply instruction. It's `mul(-5, 
5)*`.



 Sep 29 2020



  parent  Stefan Koch <uplink.coder gmail.com>  writes:


On Tuesday, 29 September 2020 at 14:42:42 UTC, Dukc wrote:

 On Tuesday, 29 September 2020 at 12:54:53 UTC, Stefan Koch 
 wrote:

 So we had this:


 [...]




 I don't know assembly, but I can see an even simpler reason why 
 the compiler can't generate a multiply instruction. It's 
 `mul(-5, 5)*`.


Which is another reason why it's harder to state the problem in a 
lower form.
I wrote the example wanting to show two semantically equivalent 
things and I did not.



 Sep 29 2020



prev sibling  parent  Bruce Carneal <bcarneal gmail.com>  writes:


On Tuesday, 29 September 2020 at 03:14:34 UTC, Andrei 
Alexandrescu wrote:

 On 9/28/20 6:46 PM, Bruce Carneal wrote:

 [...]


 Years ago, I was in a panel with Simon Peyton-Jones, the 
 creator of Haskell. Nicest guy around, not to mention an 
 amazing researcher. This question was asked: "What is the most 
 important principle of programming language design?"

 His answer was so powerful, it made me literally forget 
 everybody else's answer, including my own. (And there were no 
 slouches in that panel, save for me: Martin Odersky, Guido van 
 Rossum, Bertrand Meyer.) He said the following:

 "The most important principle in a language design is to define 
 a small core of essential primitives. All necessary syntactic 
 sugar lowers to core constructs. Do everything else in 
 libraries."

 (He didn't use the actual term "lowering", which is familiar to 
 our community, but rather something equivalent such as 
 "reduces".)

 That kind of killed the panel, in a good way. Because most 
 other questions on programming language design and 
 implementation simply made his point shine quietly. Oh yes, if 
 you had a small core and build on it you could do this easily. 
 And that. And the other. In a wonderfully meta way, all 
 questions got instantly lowered to simpler versions of 
 themselves. I will never forget that experience.

 D breaks that principle in several places. It has had a 
 cavalier attitude to using magic tricks in the compiler to get 
 things going, at the expense of fuzzy corners and odd limit 
 cases. Look at hashtables. Nobody can create an equivalent 
 user-defined type, and worse, nobody knows exactly why. (I 
 recall vaguely it has something to do, among many other things, 
 with qualified keys that are statically-sized arrays. Hacks in 
 the compiler make those work, but D's own type system rejects 
 the equivalent code. So quite literally D's type system cannot 
 verify its own capabilities.)

 Or take implicit conversions. They aren't fully documented, and 
 the only way to figure things out is to read most of the 7193 
 lines of 
 https://github.com/dlang/dmd/blob/master/src/dmd/mtype.d. 
 That's not a small core with a little sugar on top.

 Or take the foreach statement. Through painful trial and error, 
 somebody figured out all possible shapes of foreach, and 
 defined `each` to support most:

 https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L904

 What should have been a simple forwarding problem took 190 
 lines that could be characterized as very involved. And mind 
 you, it doesn't capture all cases because per 
 https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d#L1073:

 // opApply with >2 parameters. count the delegate args.
 // only works if it is not templated (otherwise we cannot count 
 the args)

 I know that stuff (which will probably end on my forehead) 
 because I went and attempted to improve things a bit in 
 https://github.com/dlang/phobos/pull/7638/files#diff-0d6463fc6f41c5fb7
4300832e3135f5R805, which attempts to simplify matters by reducing the foreach
cases to seven shapes.

 To paraphrase Alan Perlis: "If your foreach has seven possible 
 shapes, you probably missed some."

 Over time, things did get better. We became adept of things 
 such as lowering, and we require precision in DIPs.

 I very strongly believe that this complexity, if unchecked, 
 will kill the D language. It will sink under its own weight. We 
 need a precise, clear definition of the language, we need a 
 principled approach to defining and extending features. The 
 only right way to extend D at this point is to remove the odd 
 failure modes created by said cavalier approach to doing 
 things. Why can't I access typeid during compilation, when I 
 can access other pointers? Turns out typeid is a palimpsest 
 that has been written over so many times, even Walter cannot 
 understand it. He told me plainly to forget about trying to use 
 typeid and write equivalent code from scratch instead. That 
 code has achieved lifetime employment.

 And no compiler tricks. I am very, very opposed to Walter's 
 penchant to reach into his bag of tricks whenever a difficult 
 problem comes about. Stop messing with the language! My 
 discussions with him on such matters are like trying to talk a 
 gambler out of the casino.

 That is a long way to say I am overwhelmingly in favor of 
 in-language solutions as opposed to yet another addition to the 
 language. To me, if I have an in-language solution, it's game, 
 set, and match. No need for language changes, thank you very 
 much. An in-language solution doesn't only mean no addition is 
 needed, but more importantly it means that the language has 
 sufficient power to offer D coders means to solve that and many 
 other problems.

 I almost skipped a heartbeat when Adam mentioned - 
 mid-sentence! - just yet another little language addition to go 
 with that DIP for interpolated strings. It would make things 
 nicer. You know what? I think I'd rather live without that DIP.

 So I'm interested in exploring reification mainly - 
 overwhelmingly - because it already works in the existing 
 language. To the extent it has difficulties, it's because of 
 undue limitations in CTFE and introspection. And these are 
 problems we must fix anyway. So it's all a virtuous circle. 
 Bringing myself to write `reify` and `dereify` around my types 
 is a very small price to pay for all that.


My thanks for the effort apparent in your composition.  The main 
lesson I take from it is that language primitives should be 
chosen using as much light as we can bring to bear.

I believe that language additions can be used to rebase the 
language on a better set of primitives.  I believe that there is 
more to learn about meta programming and the language features 
that support it and that the D platform may be the best place to 
learn.  Finally, I believe that a postulated ideal of "a small 
set of primitives" will always be better than any actual set 
derived from experience on the frontier.

Perhaps because of your strong stand against additions generally, 
you've not taken a serious look at type functions in particular.  
I say that you've not taken a serious look because I don't have 
another way to understand your writings on the reify/dereify 
design.

If you elect to investigate type functions, I would very much 
like to hear your opinion.  I see type functions as an "it just 
works", "rebasing" addition that lets us displace inappropriate 
use of templates now, and that may illuminate additional 
simplification opportunities with use.

If you do not elect to investigate type functions, I expect we'll 
next interact on a different topic.

Again, my thanks for the thoughtful composition.



 Sep 29 2020



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 9/28/2020 3:46 PM, Bruce Carneal wrote:

 Finally, I'd love to hear your comments on type functions vs reify/dereify.  
 It's certainly possible that I've missed something.  Maybe it's a type 
 functions+ solution we should be seeking (type functions for everything they
can 
 do, and some LDM for anything beyond their capability).


Andrei and I have been mulling over what to do for a while, and have gone 
through several ideas.

What we're looking for is not a top level solution. We want a fundamental 
building block, upon which the rest of D's features can exploit naturally and 
productively. Something that can be explained in 30 seconds.

An example of this is the recent PR for __totype(string), which is the 
complement to .mangleof. It's main shortcoming is it only works with types. I'm 
currently working on an idea to make that work for any alias. It's so stupidly 
simple I'm afraid I'm missing something terribly obvious, so I beg your 
indulgence while exploring it.

Stay tuned!



 Sep 30 2020



 next sibling parent  Bruce Carneal <bcarneal gmail.com>  writes:


On Thursday, 1 October 2020 at 04:08:26 UTC, Walter Bright wrote:

 On 9/28/2020 3:46 PM, Bruce Carneal wrote:

 [...]


 Andrei and I have been mulling over what to do for a while, and 
 have gone through several ideas.

 What we're looking for is not a top level solution. We want a 
 fundamental building block, upon which the rest of D's features 
 can exploit naturally and productively. Something that can be 
 explained in 30 seconds.

 An example of this is the recent PR for __totype(string), which 
 is the complement to .mangleof. It's main shortcoming is it 
 only works with types. I'm currently working on an idea to make 
 that work for any alias. It's so stupidly simple I'm afraid I'm 
 missing something terribly obvious, so I beg your indulgence 
 while exploring it.

 Stay tuned!


Sounds exciting!  Good luck!



 Sep 30 2020



prev sibling  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 1 October 2020 at 04:08:26 UTC, Walter Bright wrote:

 On 9/28/2020 3:46 PM, Bruce Carneal wrote:

 Finally, I'd love to hear your comments on type functions vs 
 reify/dereify.  It's certainly possible that I've missed 
 something.  Maybe it's a type functions+ solution we should be 
 seeking (type functions for everything they can do, and some 
 LDM for anything beyond their capability).


 Andrei and I have been mulling over what to do for a while, and 
 have gone through several ideas.

 What we're looking for is not a top level solution. We want a 
 fundamental building block, upon which the rest of D's features 
 can exploit naturally and productively. Something that can be 
 explained in 30 seconds.


Type functions take less than 30 seconds.
They take zero seconds.

Do you remember Andrei's non working example?


 alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());


Here is the correct type function which actually works!

alias[] MostDerived(alias[] types ...)
{
     sort!((alias a, alias b) => is(a : b))(types);
     return types;
}

It is Andrei thought would work.
With type functions it just works.



 Sep 30 2020



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com>  writes:


On 10/1/20 2:51 AM, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 04:08:26 UTC, Walter Bright wrote:

 On 9/28/2020 3:46 PM, Bruce Carneal wrote:

 Finally, I'd love to hear your comments on type functions vs 
 reify/dereify.  It's certainly possible that I've missed something.  
 Maybe it's a type functions+ solution we should be seeking (type 
 functions for everything they can do, and some LDM for anything 
 beyond their capability).


 Andrei and I have been mulling over what to do for a while, and have 
 gone through several ideas.

 What we're looking for is not a top level solution. We want a 
 fundamental building block, upon which the rest of D's features can 
 exploit naturally and productively. Something that can be explained in 
 30 seconds.


 
 Type functions take less than 30 seconds.
 They take zero seconds.
 
 Do you remember Andrei's non working example?
 

 alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());


 
 Here is the correct type function which actually works!
 
 alias[] MostDerived(alias[] types ...)
 {
      sort!((alias a, alias b) => is(a : b))(types);
      return types;
 }
 
 It is Andrei thought would work.
 With type functions it just works.


 From https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213:

alias DerivedToFront(Args...) = dereify!({
     auto ids = reifyArray!Args;
     ids.sort;
     return ids;
}());

That code defines opCmp such that a < b iff a is a subtype of b, a nice 
touch reminiscent of the <: operator used in type theory. To implement 
that, the reified type stores the bases (conversion targets) of the type 
at construction time. This is easy to generalize to numeric and array 
types, and some more work to get going for things like covariant functions.

One litmus test is to redo Variant "the right way" (btw it should be 
called Any and moved to druntime). Currently Variant reifies the type 
into a pointer to function, and uses that pointer to function to 
dispatch type-dependent work. It is quite messy and incorrect in places. 
"The right way" would be for the reified type to have enough information 
to allow things like testing for subtyping/convertibility. Currently 
Variant is essentially incomplete and incorrect (see 
https://github.com/dlang/phobos/blob/master/std/variant.d#L285) because 
it builds on the shaky ground of 
https://github.com/dlang/phobos/blob/master/std/traits.d#L5027, a sort 
of a best-effort approach to figuring out the definition of implicit 
conversions scattered across the language definition (if documented in 
entirety at all). Having a clear definition of what implicitly converts 
to what would be a nice perk.

One interesting thing about Variant's primitives is that information is 
half-and-half: one half is present at compile-time, i.e. "Can I read a 
value of type T?" and the other half is present at runtime, in the 
reified format stored in the Variant. Given that the latter is dynamic, 
there's no way around reifying T as well and then using the reified 
types for testing.



 Oct 01 2020



 next sibling parent reply Bruce Carneal <bcarneal gmail.com>  writes:


On Thursday, 1 October 2020 at 15:21:40 UTC, Andrei Alexandrescu 
wrote:
...

 One litmus test is to redo Variant "the right way" (btw it 
 should be called Any and moved to druntime). Currently Variant 
 reifies the type into a pointer to function, and uses that 
 pointer to function to dispatch type-dependent work. It is 
 quite messy and incorrect in places. "The right way" would be 
 for the reified type to have enough information to allow things 
 like testing for subtyping/convertibility. Currently Variant is 
 essentially incomplete and incorrect ...


Litmus test for what?  Utility at run-time or as a simple base 
language addition for use at compile time?

My concern here is not with our ability to make something like 
what you're proposing "work".  After all, C++ "works".  My 
concern is that the proposal, by my lights anyway, is already a 
good deal past "simple" and it's not converging.

On a related note, if the reify/dereify code expands as much as I 
believe it will in order to cover all cases, expands in to a 
"shadow" front end, you'll want to look at forcing the actual 
front end to use it so that you can eliminate divergence issues. 
That could also set us up for an embedded-compiler/jit future 
which would be another justification for the work.

If that's where you're headed, if you want to destroy the 
compile-time/run-time separation, I'd be interested in hearing 
more.  I'm not a fan of embedding the compiler or exposing the 
type system for that matter, but it would be worth a listen, no 
doubt.



 Oct 01 2020



  parent  Mark <smarksc gmail.com>  writes:


On Thursday, 1 October 2020 at 18:04:09 UTC, Bruce Carneal wrote:

 If that's where you're headed, if you want to destroy the 
 compile-time/run-time separation, I'd be interested in hearing 
 more.  I'm not a fan of embedding the compiler or exposing the 
 type system for that matter, but it would be worth a listen, no 
 doubt.




worth looking into.



 Oct 09 2020



prev sibling  parent  claptrap <clap trap.com>  writes:


On Thursday, 1 October 2020 at 15:21:40 UTC, Andrei Alexandrescu 
wrote:

 On 10/1/20 2:51 AM, Stefan Koch wrote:

 On Thursday, 1 October 2020 at 04:08:26 UTC, Walter Bright 
 wrote:


 Here is the correct type function which actually works!
 
 alias[] MostDerived(alias[] types ...)
 {
      sort!((alias a, alias b) => is(a : b))(types);
      return types;
 }
 
 It is Andrei thought would work.
 With type functions it just works.


 From 
 https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213:

 alias DerivedToFront(Args...) = dereify!({
     auto ids = reifyArray!Args;
     ids.sort;
     return ids;
 }());


Try explaining that to a newbie who's never used D meta 
programming before.

It's a function that sorts a list of types from most to least 
derived.
You do it with an alias template. Here we've use the shorthand 
for an eponymous alias template. So it sort of looks like a 
regular function call but you need an equals sign after the 
template parentheses.
And the right had side needs to be an expression because its an 
alias declaration not a function declaration.
Oh and you need to wrap the right had side in a call to 
dereify(), you dont need to know why for now, you just do.
And you need to wrap the code you want to execute in an anonymous 
lamba that is called immediately.
And before you can do anything with the args passed to the 
template you need to call reify() on them.

I mean seriously? 30 seconds with a straight face?

(Im not even sure ive described it correctly)

vs a type function...

It's a function that sorts a list of types from most to least 
derived.
You write it just like a regular function but with types.



 Oct 01 2020



prev sibling next sibling parent  claptrap <clap trap.com>  writes:


On Monday, 28 September 2020 at 02:58:13 UTC, Bruce Carneal wrote:

 On Sunday, 27 September 2020 at 16:32:18 UTC, claptrap wrote:


 I hope and believe that there is another such advance available 
 to us in the form of type functions.  The recently discovered 
 alternative, reify/dereify, appears to be equivalent in power 
 but is, comparatively, baroque.

 If the two approaches are actually equivalent in power, and in 
 their ability to address the current template issues, then the 
 baroque should be preferred if the language is closed to 
 syntactic additions.

 If the language is not closed, then choosing to displace the 
 already prototyped type function capability with reify/dereify 
 would represent a lost opportunity to administer another 
 complexity smackdown.


Yeah I think A&W etc.. are more concerned about keeping compiler 
complexity down than they are about a clean intuitive language 
interface.



 Sep 28 2020



prev sibling  parent  Joseph Rushton Wakeling <joseph.wakeling webdrake.net>  writes:


On Monday, 28 September 2020 at 02:58:13 UTC, Bruce Carneal wrote:

 I hope and believe that there is another such advance available 
 to us in the form of type functions.  The recently discovered 
 alternative, reify/dereify, appears to be equivalent in power 
 but is, comparatively, baroque.


Having been following Stefan's work on type functions for some 
time now, and really liking the friendly syntax that it allows, I 
would really appreciate a more detailed breakdown of the pros and 
cons when compared to this approach here.

As I understand it neither requires very significant compiler 
changes, and both allow us to rewrite existing Phobos templates 
in ways that greatly improve performance.

 Andrei would you be happy to share your thoughts in more detail 
on all this, and how you feel we should balance out the different 
concerns and interests?



 Sep 30 2020



prev sibling next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 9/26/2020 9:18 AM, Andrei Alexandrescu wrote:

 [...]


It's too easy. You must have done it all wrong!



 Sep 26 2020



prev sibling next sibling parent  Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com>  writes:


On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:

 Commments and ideas for improvements are welcome.


Great idea.

1. I presume we need to benchmark this both in space and time to 
judge its merits. Have you done any such? How will the number of 
templates instances change with length of array for each 
algorithm in std.meta?

2. To reduce (all) template bloat in your new version of 
std.meta, what about adding a special handling in the compiler 
for `reify` and `dereify` like was done with `AliasSeq`?



 Sep 27 2020



prev sibling next sibling parent  Andrej Mitrovic <andrej.mitrovich gmail.com>  writes:


On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:

 I'll start with the punchline: at the link below you'll find an 
 alpha-quality reimplementation of std.meta in an iterative 
 manner:

 https://gist.github.com/andralex/6212ebf2b38c59c96cf66d0009336318


This is really cool. It looks like if we add just a tiny bit of 
compiler support (like Walter's PR) we could make this efficient. 
And we're just using the power of CTFE to do it. Sweet.



 Sep 27 2020



prev sibling next sibling parent  Mike <slavo5150 yahoo.com>  writes:


On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:


 Commments and ideas for improvements are welcome.


If you're going to go through the trouble of re-implementing 
std.meta, please move it to druntime so it can be leveraged by 
the DMD frontend and the druntime implementation, among other 
benefits.  std.meta can then forward to the druntime 
implementations for backward compatibility.



 Sep 28 2020



prev sibling  parent reply Atila Neves <atila.neves gmail.com>  writes:


On Saturday, 26 September 2020 at 16:18:27 UTC, Andrei 
Alexandrescu wrote:

 I'll start with the punchline: at the link below you'll find an 
 alpha-quality reimplementation of std.meta in an iterative 
 manner:

 https://gist.github.com/andralex/6212ebf2b38c59c96cf66d0009336318

 It's the bulk of std.meta - I stopped when it became obvious 
 doing more is just more of the same. Compiles and runs with dmd 
 2.092 and probably others.

 The implementation uses throughout a reification/dereification 
 approach. To reify a type or alias means to convert it into a 
 value. To dereify a value means to turn it back into the type 
 or alias it originated from.

 The algorithms follow the following pattern consistently:

 1. Reify the compile-time parameters into values
 2. Carry processing on these values with the usual algorithms
 3. If needed, dereify back the results into compile-time 
 parameters

 Not all artifacts need step 3. For example, allSatisfy returns 
 a bool, not a type.

 For the most part implementation has been a healthy gallop that 
 took a couple of hours yesterday and a couple of hours today. 
 Its weakest point is it uses .stringof for types, which has 
 issues with local types. Hopefully 
 https://github.com/dlang/dmd/pull/11797 could help with that.

 Most implementations are a few lines long because they get to 
 leverage algorithms as implemented in std. Here's a typical one:

 alias MostDerived(Args...) = dereify!({
     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());

 To sort an AliasSeq with most derived elements first, create a 
 lambda to reify the arguments into values, sort the values, and 
 return the sorted result. Then call the lambda and use dereify 
 against its result. And that is IT.

 Commments and ideas for improvements are welcome.


1. This is awesome.
2. I agree with everything said about keeping the core language 
simple and lowering to fundamental constructs. I agree with that 
in general, and it's a lot easier to agree with it in the context 
of D given that it's a large language already.

Lisp got mentioned in this thread as if it's some kind of 
failure, and for the life for me I have no idea why.

Users: dmd has too many bugs!
Also users: please add my bugs, err, feature!



 Oct 08 2020



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 October 2020 at 09:51:15 UTC, Atila Neves wrote:

 1. This is awesome.
 2. I agree with everything said about keeping the core language 
 simple and lowering to fundamental constructs. I agree with 
 that in general, and it's a lot easier to agree with it in the 
 context of D given that it's a large language already.

 Lisp got mentioned in this thread as if it's some kind of 
 failure, and for the life for me I have no idea why.

 Users: dmd has too many bugs!
 Also users: please add my bugs, err, feature!


---
alias MostDerived(Args...) = dereify!({

     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());


---

Note that the example I re-posted above doesn't actually work.
Because of polymorphism.



 Oct 08 2020



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/8/20 7:06 AM, Stefan Koch wrote:

 On Thursday, 8 October 2020 at 09:51:15 UTC, Atila Neves wrote:

 1. This is awesome.
 2. I agree with everything said about keeping the core language simple 
 and lowering to fundamental constructs. I agree with that in general, 
 and it's a lot easier to agree with it in the context of D given that 
 it's a large language already.

 Lisp got mentioned in this thread as if it's some kind of failure, and 
 for the life for me I have no idea why.

 Users: dmd has too many bugs!
 Also users: please add my bugs, err, feature!


 
 ---
 alias MostDerived(Args...) = dereify!({

     auto ids = reify!Args;
     sort!((a, b) => is(dereify!a : dereify!b))(ids);
     return ids;
 }());


 ---
 
 Note that the example I re-posted above doesn't actually work.
 Because of polymorphism.


The following works:

alias DerivedToFront(Args...) = dereify!({
     auto ids = reifyArray!Args;
     ids.sort;
     return ids;
}());

https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213



 Oct 08 2020



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 October 2020 at 12:05:22 UTC, Andrei Alexandrescu 
wrote:

 The following works:

 alias DerivedToFront(Args...) = dereify!({
     auto ids = reifyArray!Args;
     ids.sort;
     return ids;
 }());

 https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213


Only with the ID struct that defines the op-compare.
Which is significantly more complicated that the custom sort 
predicate.
And also more inflexible.



 Oct 08 2020



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 10/8/20 8:16 AM, Stefan Koch wrote:

 On Thursday, 8 October 2020 at 12:05:22 UTC, Andrei Alexandrescu wrote:

 The following works:

 alias DerivedToFront(Args...) = dereify!({
     auto ids = reifyArray!Args;
     ids.sort;
     return ids;
 }());

 https://gist.github.com/andralex/0d85df38be2d9ffbe89cf1fb51c44213


 
 Only with the ID struct that defines the op-compare.
 Which is significantly more complicated that the custom sort predicate.
 And also more inflexible.


Less flexible during compilation, more flexible at runtime. It is quite 
satisfying that the same information in reified form (strings I think in 
that gist) is equally usable during compilation, at run-time, or in 
mixed scenarios (Variant).

I wonder what other information would be useful to persist through 
runtime aside from the trivial (size, alignment) and the more involved 
(enough information to decide conversions). For functions, I assume that 
would be the returned type and the parameter types along with all 
adornments (ref, out, all that stuff). For classes, interfaces, and 
structs, that would include all public methods - probably on demand by 
means of an  rtti attribute so as to not waste time and space.



 Oct 08 2020



  parent reply Adam D. Ruppe <destructionator gmail.com>  writes:


On Thursday, 8 October 2020 at 13:12:22 UTC, Andrei Alexandrescu 
wrote:

 Less flexible during compilation, more flexible at runtime.


With my little lib that uses string lambdas for templates:

pragma(msg, reified!q{
         // string template lambda
         a.sizeof == 4
}.run!(types =>
         types.filter!(a => a) // normal std.algorithm!
).over!(string, int, float, Object));



 Oct 08 2020



 next sibling parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 October 2020 at 13:19:48 UTC, Adam D. Ruppe wrote:

 On Thursday, 8 October 2020 at 13:12:22 UTC, Andrei 
 Alexandrescu wrote:

 Less flexible during compilation, more flexible at runtime.


 With my little lib that uses string lambdas for templates:

 pragma(msg, reified!q{
         // string template lambda
         a.sizeof == 4
 }.run!(types =>
         types.filter!(a => a) // normal std.algorithm!
 ).over!(string, int, float, Object));


please post the source of your lib.
Benchmark is incoming.



 Oct 08 2020



  parent  Adam D. Ruppe <destructionator gmail.com>  writes:


On Thursday, 8 October 2020 at 13:27:50 UTC, Stefan Koch wrote:

 please post the source of your lib.


it is over here 
https://forum.dlang.org/post/zhaziudnbaoybmfdrzxn forum.dlang.org


 Benchmark is incoming.


I don't expect it to win, this is more to demo existing language 
api style possibilities. This style lost to existing Phobos 
std.meta when I tried.



 Oct 08 2020



prev sibling  parent  Adam D. Ruppe <destructionator gmail.com>  writes:


On Thursday, 8 October 2020 at 13:19:48 UTC, Adam D. Ruppe wrote:

         a.sizeof == 4


Of course I probably should have written it:

pragma(msg, reified!q{
         a.sizeof // just get the size
}.run!(types =>
         types.filter!(a => a == 4) // and compare it down here
).over!(string, int, float, Object));



 Oct 08 2020