digitalmars.D - Challenge: write a template-instance free dispatcher for text printing
- Richard (Rikki) Andrew Cattermole (68/68) May 25 I have a challenge for all to try!
- Adam D. Ruppe (9/12) May 25 Use D1's writeln, it was a runtime variadic instead of a template.
- Richard (Rikki) Andrew Cattermole (6/20) May 25 That works if you can lower it to a foreach, however the key requirement...
- Atila Neves (3/10) May 26 What's expensive about string mixins?
- Richard (Rikki) Andrew Cattermole (17/32) May 27 That snippet of code, needs: new file name, lexed, parsed, create parse
- H. S. Teoh (15/31) May 27 [...]
- Richard (Rikki) Andrew Cattermole (8/41) May 27 What I'm suspecting here is missing control flow capabilities and
- monkyyy (4/6) May 27 you can reassign void* enums and the ct gc does type checking
- Paul Backus (12/20) May 27 What you're describing is Lisp macros, and Walter has gone on
- H. S. Teoh (41/52) May 28 [...]
- Basile B. (4/22) May 27 We just want to insert new AST nodes during compilation, we are
I have a challenge for all to try!
For Walter says templates must be avoided in some core code!
So let's see if D can do it, but... There are two rules.
1. You cannot use ANY template instantiations to do the work
itself
2. No string mixins!
Both of these are expensive, and they need to go.
So, can you create dispatch code that:
1. Can identify and handle (in our case, writeln is fine for
this), individual arguments that are not in IES?
2. Can identify arguments that are IES, and then identify and
handle the input of literals and expressions sentinals + the
values that go with the expression sentinal?
To give an idea, here is a simple test program:
```d
import std;
void main()
{
int someVar;
bool aFlag;
string text = "Hi!";
dispatcher("prefix", i" $(someVar)/$(someVar) = $(text) ",
"suffix");
}
void dispatcher(Args...)(auto ref Args args) {
static foreach(arg; args) {
writefln!"%040s: %s"(typeof(arg).stringof, arg);
}
}
```
And its output:
```
string: prefix
InterpolationHeader:
InterpolatedLiteral!" ":
InterpolatedExpression!"someVar":
int: 0
InterpolatedLiteral!"/": /
InterpolatedExpression!"someVar":
int: 0
InterpolatedLiteral!" = ": =
InterpolatedExpression!"text":
string: Hi!
InterpolatedLiteral!" ":
InterpolationFooter:
string: suffix
```
You need to somehow find a way to replace that static foreach, so
that you have the following information:
```
string: prefix
InterpolationHeader:
InterpolatedLiteral!" ":
InterpolatedExpression!"someVar":
int: 0
InterpolatedLiteral!"/": /
InterpolatedExpression!"someVar":
int: 0
InterpolatedLiteral!" = ": =
InterpolatedExpression!"text":
string: Hi!
InterpolatedLiteral!" ":
InterpolationFooter:
string: suffix
```
It is important that you can skip arguments, and handle multiple
at a time. In essence, you need some way to do a for loop but
with enum's and at compile time.
May 25
On Monday, 25 May 2026 at 15:19:51 UTC, Richard (Rikki) Andrew Cattermole wrote:So let's see if D can do it, but... There are two rules.Use D1's writeln, it was a runtime variadic instead of a template.dispatcher("prefix", i" $(someVar)/$(someVar) = $(text) ", "suffix");something like this is what my arsd.core thing does: https://github.com/adamdruppe/arsd/blob/master/core.d#L11053 it reduces most input to small, factored helper functions to minimize generated code. also supports formatting via a wrapper: assert(toStringInternal(5.4364.formatArgs(precision: 2)) == "5.44");
May 25
"Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> On 26/05/2026 3:35 AM, Adam D. Ruppe wrote:On Monday, 25 May 2026 at 15:19:51 UTC, Richard (Rikki) Andrew Cattermole wrote:Must be at CT, no runtime shenanigans.So let's see if D can do it, but... There are two rules.Use D1's writeln, it was a runtime variadic instead of a template.That works if you can lower it to a foreach, however the key requirement is being able to see both the header, footer, and everything in between. Once you can see it, then you need to be able to manipulate it using CT constructs.dispatcher("prefix", i" $(someVar)/$(someVar) = $(text) ", "suffix");something like this is what my arsd.core thing does: https://github.com/adamdruppe/arsd/blob/master/core.d#L11053 it reduces most input to small, factored helper functions to minimize generated code. also supports formatting via a wrapper: assert(toStringInternal(5.4364.formatArgs(precision: 2)) == "5.44");
May 25
On Monday, 25 May 2026 at 15:19:51 UTC, Richard (Rikki) Andrew Cattermole wrote:I have a challenge for all to try! For Walter says templates must be avoided in some core code! So let's see if D can do it, but... There are two rules. 1. You cannot use ANY template instantiations to do the work itself 2. No string mixins! Both of these are expensive, and they need to go.What's expensive about string mixins?
May 26
On 27/05/2026 6:33 PM, Atila Neves wrote:On Monday, 25 May 2026 at 15:19:51 UTC, Richard (Rikki) Andrew Cattermole wrote:That snippet of code, needs: new file name, lexed, parsed, create parse tree, then semantic, then combined back into previous AST. Versus: copy parse tree, run semantic, append to list. Compared to the base line there is a lot of infrastructure being touched that doesn't need to be if it was modelled by the language. String mixins are fundamentally an escape hatch of last resort, and should be audited for necessity. I generally class them in one of two categories: - Unnecessary evil and should be removed. - Language needs to model it. See tuples and bit fields for PhobosV2 as an example of the latter. They also happen to encourage templates, which on top of the base line create at a minimum one new symbol that may get emitted into the object file. So avoiding string mixins is a good thing for a library to do, at least in hot code.I have a challenge for all to try! For Walter says templates must be avoided in some core code! So let's see if D can do it, but... There are two rules. 1. You cannot use ANY template instantiations to do the work itself 2. No string mixins! Both of these are expensive, and they need to go.What's expensive about string mixins?
May 27
On Thu, May 28, 2026 at 04:44:56AM +1200, Richard (Rikki) Andrew Cattermole via Digitalmars-d wrote:On 27/05/2026 6:33 PM, Atila Neves wrote:[...][...]What's expensive about string mixins?That snippet of code, needs: new file name, lexed, parsed, create parse tree, then semantic, then combined back into previous AST. Versus: copy parse tree, run semantic, append to list. Compared to the base line there is a lot of infrastructure being touched that doesn't need to be if it was modelled by the language.They also happen to encourage templates, which on top of the base line create at a minimum one new symbol that may get emitted into the object file. So avoiding string mixins is a good thing for a library to do, at least in hot code.Makes me really wish for Stefan's' CTFE engine + CT type manipulation support. Although D templates are extremely powerful and awesome for metaprogramming, templates are ultimately not the right concept for many CT tasks. What we *really* want is CTFE + type manipulation for DbI to compute the final form of a template instantiation, then a single template instantiation at the end to actualize it. The current idiom of arbitrarily-complex templates nested arbitrarily deep, while powerful, leads to needless template bloat. T -- Study gravitation, it's a field with a lot of potential.
May 27
On 28/05/2026 6:13 AM, H. S. Teoh wrote:On Thu, May 28, 2026 at 04:44:56AM +1200, Richard (Rikki) Andrew Cattermole via Digitalmars-d wrote:What I'm suspecting here is missing control flow capabilities and dynamic variables. enum assign (similar to alias assign) + static while, I suspect would solve this particular problem nicely. Oh and being able to create alias sequences without needing a template... that would be helpful. Maybe break for static foreach/static while too.On 27/05/2026 6:33 PM, Atila Neves wrote:[...][...]What's expensive about string mixins?That snippet of code, needs: new file name, lexed, parsed, create parse tree, then semantic, then combined back into previous AST. Versus: copy parse tree, run semantic, append to list. Compared to the base line there is a lot of infrastructure being touched that doesn't need to be if it was modelled by the language.They also happen to encourage templates, which on top of the base line create at a minimum one new symbol that may get emitted into the object file. So avoiding string mixins is a good thing for a library to do, at least in hot code.Makes me really wish for Stefan's' CTFE engine + CT type manipulation support. Although D templates are extremely powerful and awesome for metaprogramming, templates are ultimately not the right concept for many CT tasks. What we *really* want is CTFE + type manipulation for DbI to compute the final form of a template instantiation, then a single template instantiation at the end to actualize it. The current idiom of arbitrarily-complex templates nested arbitrarily deep, while powerful, leads to needless template bloat. T
May 27
On Wednesday, 27 May 2026 at 18:19:49 UTC, Richard (Rikki) Andrew Cattermole wrote:enum assign (similar to alias assign) + static while, I suspect would solve this particular problem nicely.you can reassign void* enums and the ct gc does type checking (dont report) so you can just test it now
May 27
On Wednesday, 27 May 2026 at 18:13:40 UTC, H. S. Teoh wrote:Although D templates are extremely powerful and awesome for metaprogramming, templates are ultimately not the right concept for many CT tasks. What we *really* want is CTFE + type manipulation for DbI to compute the final form of a template instantiation, then a single template instantiation at the end to actualize it. The current idiom of arbitrarily-complex templates nested arbitrarily deep, while powerful, leads to needless template bloat.What you're describing is Lisp macros, and Walter has gone on record many, many times that D will never have them. I think the best we can do is probably something similar to C++'s {fmt} library, which uses type erasure and runtime dispatch to avoid unnecessary template instantiations. For example, if we are formatting 5 different structs that all have toString() methods, there is no need to instantiate 5 completely separate copies of writeln--instead, we can have one instance of writeln which makes an indirect call through an interface/delegate, and 5 instances of a small helper function which that interface/delegate can dispatch to.
May 27
On Wed, May 27, 2026 at 06:50:08PM +0000, Paul Backus via Digitalmars-d wrote:On Wednesday, 27 May 2026 at 18:13:40 UTC, H. S. Teoh wrote:[...] Not necessarily. Full-out macros leads to wild things like IOCCC-style preprocessor macro abuse; what I'm talking about is in-language syntax for making compile-time decisions in the code without causing hundreds of template instantiations. We already somewhat have this in the form of `static if` and `static foreach`. What's missing is things like direct access to type operations, e.g., to construct `struct S2 { const(T) val; }` given `struct S { T val; }` as a CT argument, without using a truckload of templates to break S down into its components and reassemble it with even more templates. CT variables for expressing complex codegen tasks, like generating unique identifiers, etc.. All of this is currently already doable, but it involves convoluted templates and string mixins, with the accompanying template bloat, symbol size bloat, and weak compiler support (compiler cannot help you with checking a mixin string until the string is fully assembled; until then it's a free-for-all wild wild west of pasting code snippets together and using hacks for things like generating unique identifiers -- usually involving yet more templates). Like I said, templates are powerful and can express all of this, but they were never designed for AST manipulation of this sort, and as such, are ill-suited for the task, no matter how capable. Symptoms of this we already know: template bloat, symbol size bloat, compiler memory exhaustion, etc.. We want the metaprogramming potential of templates, but without the ugliness of recursive templates, template bloat, symbol size bloat, and high memory usage. The template instantiation really should happen only at the end, once the CT logic has decided what the output code should be. The CT logic itself shouldn't involve instantiating lots of temporary templates in between, and definitely shouldn't involve recursive templates where it's avoidable. A template function with static if / static foreach already does this, but only to a limited extent. When you start having to do more complex things like type manipulation, you quickly reach for templates as the only current way of achieving what you need. The bulk of std.traits, for example, generate truckloads of template symbols every time you want to do the simplest of CT tasks. It doesn't have to be this way. There should be a way to express most of std.traits without instantiating templates. T -- Public parking: euphemism for paid parking. -- FloraAlthough D templates are extremely powerful and awesome for metaprogramming, templates are ultimately not the right concept for many CT tasks. What we *really* want is CTFE + type manipulation for DbI to compute the final form of a template instantiation, then a single template instantiation at the end to actualize it. The current idiom of arbitrarily-complex templates nested arbitrarily deep, while powerful, leads to needless template bloat.What you're describing is Lisp macros, and Walter has gone on record many, many times that D will never have them.
May 28
On Wednesday, 27 May 2026 at 18:13:40 UTC, H. S. Teoh wrote:On Thu, May 28, 2026 at 04:44:56AM +1200, Richard (Rikki) Andrew Cattermole via Digitalmars-d wrote:We just want to insert new AST nodes during compilation, we are just not sure how to insert them... macros ? mixins ? templates ? 🤔[...][...][...][...][...]Makes me really wish for Stefan's' CTFE engine + CT type manipulation support. Although D templates are extremely powerful and awesome for metaprogramming, templates are ultimately not the right concept for many CT tasks. What we *really* want is CTFE + type manipulation for DbI to compute the final form of a template instantiation, then a single template instantiation at the end to actualize it. The current idiom of arbitrarily-complex templates nested arbitrarily deep, while powerful, leads to needless template bloat. T
May 27