www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Meta-programming - examples

reply janderson <askme me.com> writes:
A couple of people have been asking questions about pseudo-real-life 
examples for the new compile time code aspects of D.  I ask this 
question for my interest as well.

Assuming that we have a good mechanism to process the input how could 
compile time code improve your every-day proficiency?  (This is 
specifically about the Mixin Expressions aspect).

I'll start with a few of my own.  Note my background is
game-programming.  In my experience, we care constantly creating tools 
to generate code because its impracticable to write and maintain it by hand.

Statemachine (and AI logic) written by design and coders are common in 
game programming.  Normally we have to fallback on a script language 
which is never as powerful.

mixin(statemachine(
"
	state FireAtEnemy
	{
		transition EnemyRange < 10
		{
			goto ChaseEnemy
		}
		transition FacingEnemy < .2
		{
			goto ChaseEnemy
		}
	}

	state ChaseEnemy
	{
		...
	}

");

You might imagine doing the above with if statement and arrays of 
delegates however designers have a much harder time with that.

3D Object rendering

//Render a 3D model.  For instance this may generate the vertexbuffers 
and textures directly into code, or may extract some of the data from 
the 3Dmodel.obj and load the rest at run-time.
mixin(renderModel(import("3Dmodel.obj")));

Transforming Textures
DXDtexture* texture = mixin(textureToD3D(import("3Dmodel.bmp")));

Simplify saving/loading of arbitrary structures.

//Only write this once.  Things are re-ordered for best fit in memory, 
localization ect...
mixin(Serailzable("Ship",
"
	struct Part
	{
		float firePower;
		char[] Model;
	}

	struct Ship
	{
		bool Friendly;
		Part Parts[];
	}
");

... Later

Load("Ship", "Ship.asset");

GUI -> not sure about this one as I can't come up with a format thats 
much neater then straight code. I'll post it anyways.

mixin(MyGUI(
"
	Button "OK", 10, 30
	{
		Exit;
	}
");

Multi-threading

The suggestion with multi-threading I came up with a little while ago:

int[] array ...
int result = 0;
mixin(threadIt(
"
	foreach(A a; array)
	{
		result += a;
	}
	combiner //If necessary?
	{
		result = result[0] + result[1];
	}
");

Compile time checking of code for coding standards or what have u:

mixin(CheckCode("

ect...


");
//Note that there should be a way to disable CheckCode in the general 
case to save compile time


There's heaps of other stuff, but that would make this post way to long.

Personally I see this new mixin format as giving us the power to extend 
D in whatever way meets our goals.  As Walter adds more features some of 
them will be rarely used (law of demising returns) because they are 
special case.  Mixin Expressions circumvent this because now the 
programmer has the power to create the language features they need.

The only downside I see is compile-time, which I hope can be solved with 
a smart caching system.
Feb 10 2007
next sibling parent reply kris <foo bar.com> writes:
janderson wrote:

 A couple of people have been asking questions about pseudo-real-life 
 examples for the new compile time code aspects of D.  I ask this 
 question for my interest as well.
 
 Assuming that we have a good mechanism to process the input how could 
 compile time code improve your every-day proficiency?  (This is 
 specifically about the Mixin Expressions aspect).
 
 I'll start with a few of my own.  Note my background is
 game-programming.  In my experience, we care constantly creating tools 
 to generate code because its impracticable to write and maintain it by 
 hand.
 
 Statemachine (and AI logic) written by design and coders are common in 
 game programming.  Normally we have to fallback on a script language 
 which is never as powerful.
 
 mixin(statemachine(
 "
     state FireAtEnemy
     {
         transition EnemyRange < 10
         {
             goto ChaseEnemy
         }
         transition FacingEnemy < .2
         {
             goto ChaseEnemy
         }
     }
 
     state ChaseEnemy
     {
         ...
     }
 
 ");
 
 You might imagine doing the above with if statement and arrays of 
 delegates however designers have a much harder time with that.
 
 3D Object rendering
 
 //Render a 3D model.  For instance this may generate the vertexbuffers 
 and textures directly into code, or may extract some of the data from 
 the 3Dmodel.obj and load the rest at run-time.
 mixin(renderModel(import("3Dmodel.obj")));
 
 Transforming Textures
 DXDtexture* texture = mixin(textureToD3D(import("3Dmodel.bmp")));
 
 Simplify saving/loading of arbitrary structures.
 
 //Only write this once.  Things are re-ordered for best fit in memory, 
 localization ect...
 mixin(Serailzable("Ship",
 "
     struct Part
     {
         float firePower;
         char[] Model;
     }
 
     struct Ship
     {
         bool Friendly;
         Part Parts[];
     }
 ");
 
 ... Later
 
 Load("Ship", "Ship.asset");
 
 GUI -> not sure about this one as I can't come up with a format thats 
 much neater then straight code. I'll post it anyways.
 
 mixin(MyGUI(
 "
     Button "OK", 10, 30
     {
         Exit;
     }
 ");
 
 Multi-threading
 
 The suggestion with multi-threading I came up with a little while ago:
 
 int[] array ...
 int result = 0;
 mixin(threadIt(
 "
     foreach(A a; array)
     {
         result += a;
     }
     combiner //If necessary?
     {
         result = result[0] + result[1];
     }
 ");
 
 Compile time checking of code for coding standards or what have u:
 
 mixin(CheckCode("
 
 ect...
 
 
 ");
 //Note that there should be a way to disable CheckCode in the general 
 case to save compile time
 
 
 There's heaps of other stuff, but that would make this post way to long.
 
 Personally I see this new mixin format as giving us the power to extend 
 D in whatever way meets our goals.  As Walter adds more features some of 
 them will be rarely used (law of demising returns) because they are 
 special case.  Mixin Expressions circumvent this because now the 
 programmer has the power to create the language features they need.
 
 The only downside I see is compile-time, which I hope can be solved with 
 a smart caching system.



Thanks, very much. It's refreshing to see some concrete examples.

- Kris
Feb 10 2007
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
kris wrote:

 janderson wrote:

 A couple of people have been asking questions about pseudo-real-life 
 examples for the new compile time code aspects of D.  I ask this 
 question for my interest as well.

 Assuming that we have a good mechanism to process the input how could 
 compile time code improve your every-day proficiency?  (This is 
 specifically about the Mixin Expressions aspect).

 I'll start with a few of my own.  Note my background is
 game-programming.  In my experience, we care constantly creating tools 
 to generate code because its impracticable to write and maintain it by 
 hand.

 Statemachine (and AI logic) written by design and coders are common in 
 game programming.  Normally we have to fallback on a script language 
 which is never as powerful.

 mixin(statemachine(
 "
     state FireAtEnemy
     {
         transition EnemyRange < 10
         {
             goto ChaseEnemy
         }
         transition FacingEnemy < .2
         {
             goto ChaseEnemy
         }
     }

     state ChaseEnemy
     {
         ...
     }

 ");

 You might imagine doing the above with if statement and arrays of 
 delegates however designers have a much harder time with that.

 3D Object rendering

 //Render a 3D model.  For instance this may generate the vertexbuffers 
 and textures directly into code, or may extract some of the data from 
 the 3Dmodel.obj and load the rest at run-time.
 mixin(renderModel(import("3Dmodel.obj")));

 Transforming Textures
 DXDtexture* texture = mixin(textureToD3D(import("3Dmodel.bmp")));

 Simplify saving/loading of arbitrary structures.

 //Only write this once.  Things are re-ordered for best fit in memory, 
 localization ect...
 mixin(Serailzable("Ship",
 "
     struct Part
     {
         float firePower;
         char[] Model;
     }

     struct Ship
     {
         bool Friendly;
         Part Parts[];
     }
 ");

 ... Later

 Load("Ship", "Ship.asset");

 GUI -> not sure about this one as I can't come up with a format thats 
 much neater then straight code. I'll post it anyways.

 mixin(MyGUI(
 "
     Button "OK", 10, 30
     {
         Exit;
     }
 ");

 Multi-threading

 The suggestion with multi-threading I came up with a little while ago:

 int[] array ...
 int result = 0;
 mixin(threadIt(
 "
     foreach(A a; array)
     {
         result += a;
     }
     combiner //If necessary?
     {
         result = result[0] + result[1];
     }
 ");

 Compile time checking of code for coding standards or what have u:

 mixin(CheckCode("

 ect...


 ");
 //Note that there should be a way to disable CheckCode in the general 
 case to save compile time




Hmm.  Me hopes more and more that something can be done about string 
literals.  I don't really want to end up having big chunks of my code 
stashed away in quotes.  Editors don't know how to highlight it or 
format it, even if it's plain D on the inside.


 There's heaps of other stuff, but that would make this post way to long.

 Personally I see this new mixin format as giving us the power to 
 extend D in whatever way meets our goals.  As Walter adds more 
 features some of them will be rarely used (law of demising returns) 
 because they are special case.  Mixin Expressions circumvent this 
 because now the programmer has the power to create the language 
 features they need.

 The only downside I see is compile-time, which I hope can be solved 
 with a smart caching system.


 
 
 Thanks, very much. It's refreshing to see some concrete examples.
 


I had forgotten about this, but there's also the example of Qt's dynamic 
signals and slots that was discussed a while back.  Qt's moc compiler 
/sort of/ knows how to process most /garden-variety/ C++ in order to 
extract method signatures, so that it can generate string-based run-time 
stubs.  The stubs let you write code like:
     connect(object1,"notifyFoo(int)", object2,"fooChanged(int)");
To connect up object1's notifyFoo signal with object2's fooChanged 
method.  In Qt's implementation, the moc compiler generates a 
string->method_pointer lookup table in each QObject and some other 
boilerplate so that the lookup can be done dynamically at runtime.  That 
allows you to load a UI script at runtime and connect up the signals and 
slots dynamically based on just the names of methods.

Actually, that seems like it should be mostly doable already via tuples 
and mixins.  But instead of syntax like
     slot void myMethod(int x) { . . . }
you'd need
     void myMethod(int x) { . . . }
     mixin DynamicSlot!("myMethod");

But you'd probably need to declare all the dynamic slots in one place, like
     mixin DynamicSlots!("myMethod",
			"myOtherMethod",
			"aThirdMethod");
It would be nice if you could do them a method at a time, though, to 
keep them close to the place where they're declared.  Even better if it 
could be done without repeating the name.   Maybe something like

     mixin(DynamicSlot!("void myMethod(int x)")) { . . . }

Meh.  Still ugly.  It would be a lot nicer if it could just be
     DynamicSlot! void myMethod(int x) { . . . }

Still not sure how that would manage to collect all the slot info into 
one big function that does the run-time lookup, though.

--bb
Feb 10 2007
parent janderson <askme me.com> writes:
Bill Baxter wrote:

 
 Hmm.  Me hopes more and more that something can be done about string 
 literals.  I don't really want to end up having big chunks of my code 
 stashed away in quotes.  Editors don't know how to highlight it or 
 format it, even if it's plain D on the inside.


You can always use import.



 
 I had forgotten about this, but there's also the example of Qt's dynamic 
 signals and slots that was discussed a while back.  Qt's moc compiler 
 /sort of/ knows how to process most /garden-variety/ C++ in order to 
 extract method signatures, so that it can generate string-based run-time 
 stubs.  The stubs let you write code like:
     connect(object1,"notifyFoo(int)", object2,"fooChanged(int)");
 To connect up object1's notifyFoo signal with object2's fooChanged 
 method.  In Qt's implementation, the moc compiler generates a 
 string->method_pointer lookup table in each QObject and some other 
 boilerplate so that the lookup can be done dynamically at runtime.  That 
 allows you to load a UI script at runtime and connect up the signals and 
 slots dynamically based on just the names of methods.
 
 Actually, that seems like it should be mostly doable already via tuples 
 and mixins.  But instead of syntax like
     slot void myMethod(int x) { . . . }
 you'd need
     void myMethod(int x) { . . . }
     mixin DynamicSlot!("myMethod");
 
 But you'd probably need to declare all the dynamic slots in one place, like
     mixin DynamicSlots!("myMethod",
             "myOtherMethod",
             "aThirdMethod");
 It would be nice if you could do them a method at a time, though, to 
 keep them close to the place where they're declared.  Even better if it 
 could be done without repeating the name.   Maybe something like
 
     mixin(DynamicSlot!("void myMethod(int x)")) { . . . }
 
 Meh.  Still ugly.  It would be a lot nicer if it could just be
     DynamicSlot! void myMethod(int x) { . . . }


Your right, I was wondering if the mixin format itself could some how be 
improved.  Although its a good way of indicating, user-specified code 
goes here.  Of course you could do something like:

mixin(DynamicSlot!("

void myMothod(int x)
{
  ...
}

"));

or defined you own language extensions:

mixin(MyLanguageExtentions!(import(foo.d));

//foo.d
DynamicSlot! void myMothod(int x)
{
  ...
}

ect...

However I'm guessing the overhead could become insane, particularly when 
people start writing:

mixin(BobsLanguageExtentions!(MyLanguageExtentions!(import(foo.d))));

Maybe if you could somehow have an efficient tokenisor and mix bobs and 
your language extensions together in one template it may be adequate. 
The time you save with the language extensions may offset the time used 
to compile the code.


 
 Still not sure how that would manage to collect all the slot info into 
 one big function that does the run-time lookup, though.




 
 --bb
Feb 10 2007
prev sibling next sibling parent janderson <askme me.com> writes:
janderson wrote:

 
 Simplify saving/loading of arbitrary structures.
 
 //Only write this once.  Things are re-ordered for best fit in memory, 
 localization ect...
 mixin(Serailzable("Ship",
 "
     struct Part
     {
         float firePower;
         char[] Model;
     }
 
     struct Ship
     {
         bool Friendly;
         Part Parts[];
     }
 ");
 
 ... Later
 


I wrote this up pretty fast, you probably got the point.  However:


 Load("Ship", "Ship.asset");


would probably look more like:

Ship ship;
Load(ship, "Ship.asset");
...
Save(ship, "Ship.asset");

-Joel
Feb 10 2007
prev sibling next sibling parent janderson <askme me.com> writes:
janderson wrote:

 
 Personally I see this new mixin format as giving us the power to extend 
 D in whatever way meets our goals.  As Walter adds more features some of 
 them will be rarely used (law of demising returns) because they are 
 special case.  Mixin Expressions circumvent this because now the 
 programmer has the power to create the language features they need.
 
 The only downside I see is compile-time, which I hope can be solved with 
 a smart caching system.



I would also like to add:  Mixin Expressions allow us to experiment with 
new features for the D language.

Over the course of time we may discover some privative building blocks 
that are very useful.  Hopefully they would eventually make themselves 
into the core of the language.

-Joel
Feb 10 2007
prev sibling next sibling parent reply "Andrei Alexandrescu (See Website For Email)" <SeeWebsiteForEmail erdani.org> writes:
janderson wrote:

 A couple of people have been asking questions about pseudo-real-life 
 examples for the new compile time code aspects of D.  I ask this 
 question for my interest as well.
 
 Assuming that we have a good mechanism to process the input how could 
 compile time code improve your every-day proficiency?  (This is 
 specifically about the Mixin Expressions aspect).
 
 I'll start with a few of my own.  Note my background is
 game-programming.  In my experience, we care constantly creating tools 
 to generate code because its impracticable to write and maintain it by 
 hand.
 
 Statemachine (and AI logic) written by design and coders are common in 
 game programming.  Normally we have to fallback on a script language 
 which is never as powerful.
 
 mixin(statemachine(
 "
     state FireAtEnemy
     {
         transition EnemyRange < 10
         {
             goto ChaseEnemy
         }
         transition FacingEnemy < .2
         {
             goto ChaseEnemy
         }
     }
 
     state ChaseEnemy
     {
         ...
     }
 
 ");
 
 You might imagine doing the above with if statement and arrays of 
 delegates however designers have a much harder time with that.


This example is eerily on target. A while ago I've enjoyed watching a 
talk by Shriram Khrishnamurthi on... you gotta see it:

http://technetcast.ddj.com/tnc_play_stream.html?stream_id=644

Unfortunately, it looks like DDJ does not have the audio (which was 
fantastic) anymore, so you can only see the slides (which are very good 
as well). The slides 37 to 39 knocked my socks off so hard they stuck 
onto the wall, and to this day I couldn't peel them away.

The way I'm seeing defining such DSLs is a bit more conservative than 
yours: I'd let the D compiler do most of the parsing and I'd focus on 
defining the state transitions:

class GameEngine
{
   mixin automaton!(
     State!(FireAtEnemy,
       ("EnemyRange < 10", "ChaseEnemy"),
       ("FacingEnemy < .2", "ChaseEnemy")),
     State!(ChaseEnemy,
       ...)
   )
}

It's amazing to think that D will allow this (actually already allows 
with a slightly clunkier syntax). I'll make sure I'll be barefoot that 
day. :o)


 3D Object rendering
 
 //Render a 3D model.  For instance this may generate the vertexbuffers 
 and textures directly into code, or may extract some of the data from 
 the 3Dmodel.obj and load the rest at run-time.
 mixin(renderModel(import("3Dmodel.obj")));
 
 Transforming Textures
 DXDtexture* texture = mixin(textureToD3D(import("3Dmodel.bmp")));


Sounds great, but no expertise to comment on that.


 Simplify saving/loading of arbitrary structures.
 
 //Only write this once.  Things are re-ordered for best fit in memory, 
 localization ect...
 mixin(Serailzable("Ship",
 "
     struct Part
     {
         float firePower;
         char[] Model;
     }
 
     struct Ship
     {
         bool Friendly;
         Part Parts[];
     }
 ");
 
 ... Later
 
 Load("Ship", "Ship.asset");


Yah, this has been pointed out indeed. Also add remote procedure calls. 
Generating all of the marshaling/unmarshaling code (for various 
standards!) will be trivially done by libraries. I actually see PyD as a 
great first application of that concept that will spark many others.


 GUI -> not sure about this one as I can't come up with a format thats 
 much neater then straight code. I'll post it anyways.
 
 mixin(MyGUI(
 "
     Button "OK", 10, 30
     {
         Exit;
     }
 ");


Great. The problem with GUI programming has always been that programming 
language is not a natural vehicle for specifying forms and stuff. Many 
frameworks have relied on some framework for code generation and 
impedance matching (Tcl/Tk, Windows resources, ...) with various levels 
of awkwardness. Due to its compile-time manipulation abilities, D will 
make it trivial to bind GUI resources to D code.

The way I see this happening is more along the lines:

mixin(import "inputform.rc")

The file can be generated and edited with other tools (e.g. gui 
designers). That's why I insisted with Walter that strings should be 
importable from files. This effectively makes you benefit of the dual 
cake paradox, in that at the same time you have the advantage of 
separate files for separate languages, and the advantage of seamless 
integration between them.


 Multi-threading
 
 The suggestion with multi-threading I came up with a little while ago:
 
 int[] array ...
 int result = 0;
 mixin(threadIt(
 "
     foreach(A a; array)
     {
         result += a;
     }
     combiner //If necessary?
     {
         result = result[0] + result[1];
     }
 ");
 
 Compile time checking of code for coding standards or what have u:


I'm not understanding this.


 mixin(CheckCode("
 
 ect...
 
 
 ");
 //Note that there should be a way to disable CheckCode in the general 
 case to save compile time


I guess this is an obvious:

mixin(CheckCode(import "program.d"));

A file containing a mixin for each program in a project can be generated 
by the build process.


 There's heaps of other stuff, but that would make this post way to long.


There's always the opportunity of a next post :o).


 Personally I see this new mixin format as giving us the power to extend 
 D in whatever way meets our goals.  As Walter adds more features some of 
 them will be rarely used (law of demising returns) because they are 
 special case.  Mixin Expressions circumvent this because now the 
 programmer has the power to create the language features they need.


Yah indeed. This has been a source of disagreement in the past between 
Walter and myself. He used to favor compiler-wired features, while I 
envisioned an infinitely configurable language with an exceedingly small 
core. Time has past, and we both learned some of the advantages of the 
other's approach.


 The only downside I see is compile-time, which I hope can be solved with 
 a smart caching system.


Stay tuned. Compilation times will come down dramatically. Walter is up 
to something pretty cool, but it will take a while. Advice: don't wear 
socks when reading d.announce :o).


Andrei
Feb 10 2007
next sibling parent reply janderson <askme me.com> writes:
Andrei Alexandrescu (See Website For Email) wrote:

 janderson wrote:

 mixin(statemachine(
 "
     state FireAtEnemy
     {
         transition EnemyRange < 10
         {
             goto ChaseEnemy
         }
         transition FacingEnemy < .2
         {
             goto ChaseEnemy
         }
     }

     state ChaseEnemy
     {
         ...
     }

 ");

 You might imagine doing the above with if statement and arrays of 
 delegates however designers have a much harder time with that.


 
 This example is eerily on target. A while ago I've enjoyed watching a 
 talk by Shriram Khrishnamurthi on... you gotta see it:
 
 http://technetcast.ddj.com/tnc_play_stream.html?stream_id=644
 
 Unfortunately, it looks like DDJ does not have the audio (which was 
 fantastic) anymore, so you can only see the slides (which are very good 
 as well). The slides 37 to 39 knocked my socks off so hard they stuck 
 onto the wall, and to this day I couldn't peel them away.
 
 The way I'm seeing defining such DSLs is a bit more conservative than 
 yours: I'd let the D compiler do most of the parsing and I'd focus on 
 defining the state transitions:


Actually its very similar to a format that I've used in past games 
(except was a run-time thing).


 
 class GameEngine
 {
   mixin automaton!(
     State!(FireAtEnemy,
       ("EnemyRange < 10", "ChaseEnemy"),
       ("FacingEnemy < .2", "ChaseEnemy")),
     State!(ChaseEnemy,
       ...)
   )
 }



The difficulty with this approach is that its much harder to read and 
write, of course.  Also its harder to do global optimizations. 
Furthermore you can't type straight D code into it.  I'm seeing the 
statemachine as converting any code it doesn't understand into D code so 
you can use the power of D (for instance, you don't have to define 
operators for maths operations, you'd still be able to user for loops.).

Its seems like your trying to design around the limitations of the compiler.

Here's a version with mixed-in-D-Code.

mixin(statemachine(

import("Standard.behaviors") ~ //You could even reuse states like this
  "

      state FireAtEnemy
      {
          transition Me.DistanceToEnemy > 10 && Me.DistanceToEnemy < 20
          {
              goto ChaseEnemy
          }
          transition Me.FacingEnemy < .2
          {
              goto ChaseEnemy
          }
      }

      state ChaseEnemy
      {
         auto Direction = Enermy.Pos - Me.Pos;
	Me.SetFacing(Direction);

	if (EnemyRange > 20)
	{
		Me.Run();
	}
	else if (EnemyRange > 10)
	{
		Me.Walk();
	}
	else
	{
		Me.Stop();
		goto FireAtEnemy;
	}
      }

  ");

Note much of the time this would be done with an import.



 
 It's amazing to think that D will allow this (actually already allows 
 with a slightly clunkier syntax). I'll make sure I'll be barefoot that 
 day. :o)
 

 3D Object rendering

 //Render a 3D model.  For instance this may generate the vertexbuffers 
 and textures directly into code, or may extract some of the data from 
 the 3Dmodel.obj and load the rest at run-time.
 mixin(renderModel(import("3Dmodel.obj")));

 Transforming Textures
 DXDtexture* texture = mixin(textureToD3D(import("3Dmodel.bmp")));


 
 Sounds great, but no expertise to comment on that.
 

 Simplify saving/loading of arbitrary structures.

 //Only write this once.  Things are re-ordered for best fit in memory, 
 localization ect...
 mixin(Serailzable("Ship",
 "
     struct Part
     {
         float firePower;
         char[] Model;
     }

     struct Ship
     {
         bool Friendly;
         Part Parts[];
     }
 ");

 ... Later

 Load("Ship", "Ship.asset");


 
 Yah, this has been pointed out indeed. Also add remote procedure calls. 
 Generating all of the marshaling/unmarshaling code (for various 
 standards!) will be trivially done by libraries. I actually see PyD as a 
 great first application of that concept that will spark many others.
 

 GUI -> not sure about this one as I can't come up with a format thats 
 much neater then straight code. I'll post it anyways.

 mixin(MyGUI(
 "
     Button "OK", 10, 30
     {
         Exit;
     }
 ");


 
 Great. The problem with GUI programming has always been that programming 
 language is not a natural vehicle for specifying forms and stuff. Many 
 frameworks have relied on some framework for code generation and 
 impedance matching (Tcl/Tk, Windows resources, ...) with various levels 
 of awkwardness. Due to its compile-time manipulation abilities, D will 
 make it trivial to bind GUI resources to D code.
 
 The way I see this happening is more along the lines:
 
 mixin(import "inputform.rc")
 


I see both.  You can enter it directly when you want to do a little bit 
of code or proto-typing, and use import when u need that.


 The file can be generated and edited with other tools (e.g. gui 
 designers). That's why I insisted with Walter that strings should be 
 importable from files. This effectively makes you benefit of the dual 
 cake paradox, in that at the same time you have the advantage of 
 separate files for separate languages, and the advantage of seamless 
 integration between them.
 

 Multi-threading

 The suggestion with multi-threading I came up with a little while ago:

 int[] array ...
 int result = 0;
 mixin(threadIt(
 "
     foreach(A a; array)
     {
         result += a;
     }
     combiner //If necessary?
     {
         result = result[0] + result[1];
     }
 ");

 Compile time checking of code for coding standards or what have u:


 
 I'm not understanding this.


What part?


 

 mixin(CheckCode("

 ect...


 ");
 //Note that there should be a way to disable CheckCode in the general 
 case to save compile time


 
 I guess this is an obvious:
 
 mixin(CheckCode(import "program.d"));


Exactly, both would be supported.  Sometimes its useful to have code in 
the same context that its been used.  Sometimes not.


 
 A file containing a mixin for each program in a project can be generated 
 by the build process.
 

 There's heaps of other stuff, but that would make this post way to long.


 
 There's always the opportunity of a next post :o).


Actually, the real reason is I'm just lazy.


 
 Stay tuned. Compilation times will come down dramatically. Walter is up 
 to something pretty cool, but it will take a while. 


This is great news


 Advice: don't wear socks when reading d.announce :o).


I'm taking them off now.


 
 Andrei
Feb 10 2007
parent reply "Andrei Alexandrescu (See Website For Email)" <SeeWebsiteForEmail erdani.org> writes:
janderson wrote:

 Andrei Alexandrescu (See Website For Email) wrote:

 janderson wrote:

 Multi-threading

 The suggestion with multi-threading I came up with a little while ago:

 int[] array ...
 int result = 0;
 mixin(threadIt(
 "
     foreach(A a; array)
     {
         result += a;
     }
     combiner //If necessary?
     {
         result = result[0] + result[1];
     }
 ");

 Compile time checking of code for coding standards or what have u:


 I'm not understanding this.


 
 What part?


The multi-threading part.

Andrei
Feb 10 2007
parent janderson <askme me.com> writes:
Andrei Alexandrescu (See Website For Email) wrote:

 janderson wrote:

 Andrei Alexandrescu (See Website For Email) wrote:

 janderson wrote:

 Multi-threading

 The suggestion with multi-threading I came up with a little while ago:

 int[] array ...
 int result = 0;
 mixin(threadIt(
 "
     foreach(A a; array)
     {
         result += a;
     }
     combiner //If necessary?
     {
         result = result[0] + result[1];
     }
 ");

 Compile time checking of code for coding standards or what have u:


 I'm not understanding this.


 What part?


 
 The multi-threading part.


Essentially the code would be-reformated to take advantage of 
multi-cpus.  The code generator threadIt would make a copy of all 
variables used inside of the foreach and run a partial foreach on each 
thread.  The combiner bit is run on each thread until they all collapse 
together (
ie  in this case
resultA = resultofThreadA + resultofThreadB;
resultB = resultofThreadC + resultofThreadD;
result = resultA + resultB; //Final result
)

Of course it goes without saying that each iteration in for for-each 
must be independent of one another (to some extent).  The advantage 
being, you see a foreach, and you can just wrap it in "mixin(threadIt(" 
to get a performance boost.  Looking though some of my C++ code, I'd say 
30% of those loops could be mutli-threaded with very little (if any) 
changes with the above approach.

The basic thought is that you could prove ideas work before even sending 
a message to the newsgroup.


 
 Andrei
Feb 10 2007
prev sibling parent reply janderson <askme me.com> writes:
Andrei Alexandrescu (See Website For Email) wrote:

 janderson wrote:

 ect...


 ");
 //Note that there should be a way to disable CheckCode in the general 
 case to save compile time


 
 I guess this is an obvious:
 
 mixin(CheckCode(import "program.d"));


Now that I think about it, what about:

version (CheckCode) { mixin(checkCode(__FILE__)) }

//Rest of file
...

checkCode would probably make use of static asserts.  Is it possible to 
write a message to compiler output?  That would be a handy 
static-compilation tool.

Or maybe it could be in the unit-tests.  Actually you can probably do 
that in unit tests already even without mixin Expressions.

-Joel
Feb 11 2007
parent Kirk McDonald <kirklin.mcdonald gmail.com> writes:
janderson wrote:

 Andrei Alexandrescu (See Website For Email) wrote:

 janderson wrote:

 ect...


 ");
 //Note that there should be a way to disable CheckCode in the general 
 case to save compile time


 I guess this is an obvious:

 mixin(CheckCode(import "program.d"));


 
 Now that I think about it, what about:
 
 version (CheckCode) { mixin(checkCode(__FILE__)) }
 
 //Rest of file
 ...
 
 checkCode would probably make use of static asserts.  Is it possible to 
 write a message to compiler output?  That would be a handy 
 static-compilation tool.
 


There are two ways:

pragma(msg, "Compile-time output.");

And:

static assert(false, "Making the compiler die, now...");


 Or maybe it could be in the unit-tests.  Actually you can probably do 
 that in unit tests already even without mixin Expressions.
 
 -Joel



-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org
Feb 11 2007
prev sibling next sibling parent "Yauheni Akhotnikau" <eao197 intervale.ru> writes:
On Sun, 11 Feb 2007 07:39:25 +0300, janderson <askme me.com> wrote:


 Personally I see this new mixin format as giving us the power to exten=


d  =


 D in whatever way meets our goals.  As Walter adds more features some =


of  =


 them will be rarely used (law of demising returns) because they are  =



 special case.  Mixin Expressions circumvent this because now the  =



 programmer has the power to create the language features they need.


Thaks for your examples.

This is one of my own. We use an agent-oriented framework in the  =

development of our SMS-routing software. In C++ description of agents,  =

their events, messages and states requires a lot of C++-macroses:

// C++
// Agent class description
SOL4_CLASS_START( aag_3::smsc_map::a_router_t )

	// Messages ...
	MBAPI3_ROUTER_SOL4_MSG( msg_configure,
			aag_3::smsc_map::a_router_t::msg_configure )
	MBAPI3_ROUTER_SOL4_MSG( msg_send,
			aag_3::smsc_map::a_router_t::msg_send )
	MBAPI3_ROUTER_SOL4_MSG( msg_send_result,
			aag_3::smsc_map::a_router_t::msg_send_result )
	MBAPI3_ROUTER_SOL4_MSG( msg_send_finish,
			aag_3::smsc_map::a_router_t::msg_send_finish )
	...

	// Events...
	SOL4_EVENT( evt_start )
	SOL4_EVENT_STC(
			evt_configure,
			aag_3::smsc_map::a_router_t::msg_configure )
	SOL4_EVENT_STC(
			evt_send,
			aag_3::smsc_map::a_router_t::msg_send )
	SOL4_EVENT_STC(
			evt_send_result,
			aag_3::smsc_map::a_router_t::msg_send_result )
	SOL4_EVENT_STC(
			evt_send_finish,
			aag_3::smsc_map::a_router_t::msg_send_finish )
	...

	// States...
	SOL4_STATE_START( st_initial )
		SOL4_STATE_EVENT( evt_start )
		SOL4_STATE_EVENT( evt_configure )
	SOL4_STATE_FINISH()

	SOL4_STATE_START( st_configured )
		SOL4_STATE_EVENT( evt_configure )
		SOL4_STATE_EVENT( evt_send )
		SOL4_STATE_EVENT( evt_send_result )
		SOL4_STATE_EVENT( evt_send_finish )
		...
	SOL4_STATE_FINISH()

	SOL4_STATE_START( st_not_configured )
		SOL4_STATE_EVENT( evt_configure )
	SOL4_STATE_FINISH()
SOL4_CLASS_FINISH()

Each C++-macros hides a lot of auxilary code for agent support.

In D, I hope, it may be expressed as something like:

mixin( AgentClass( Aag3.SmscMap.ARouter,
`
	mbapi3_router_messages {
		MsgConfigure =3D> Aag3.SmscMap.ARouter.MsgConfigure
		MsgSend =3D> Aag3.SmscMap.ARouter.MsgSend
                      MsgSendResult =3D> Aag.SmscMap.ARouter.MsgSendResu=
lt
		MsgSendFinish =3D> Aag.SmscMap.ARouter.MsgSendFinish
		...
	}

	events {
		evtStart
		evtConfigure
		evtSend
		evtSendResult
		evtSendFinish
		...
	}

	state st_initial {
		events {
			evtStart
			evtConfigure
		}
	}

	state st_configured {
		events {
			evtConfigure
			evtSend
			evtSendResult
			evtSendFinish
			...
		}
	}

	state st_not_configured {
		events { evt_configure }
	}
` ) );




-- =

Regards,
Yauheni Akhotnikau
Feb 11 2007
prev sibling next sibling parent reply "Kristian Kilpi" <kjkilpi gmail.com> writes:
String literals with mixins are a bit awkward sometimes (editor  =

highlighting etc).

Some special marks -- I use  { }  here -- could be used to mark a part o=
f  =

a source file as a string literal, just like /* */ marks a part of code =
as  =

a comment. For example:

   mixin(
      {
       //this is a string literal block
       if(...) {
         ...
       }
     } 
   );

The  { }  marks have a close relation, of course, with quotation marks "=
".  =

But because there is a starting mark and an ending mark, you can nest  =

them. (And because they are used to mark a part of a file as a string  =

literal, they are not actually the part of the 'working code' just like =
 =

the "" literals are, if you get what I'm trying to say.)

E.g.

   alias  {
     str =3D  { foo }  ~  { bar } ;
     str ~=3D "blah";
     if(...) {
       ...
     }
   }  MyCode;

    mixin(MyCode);
Feb 11 2007
parent reply "Kristian Kilpi" <kjkilpi gmail.com> writes:
Argh, I hit the 'reply' button instead of 'new mail' button, so this mai=
l  =

was send to this thread. I'm posting this to the root thread also...  =

Sorry. (So please don't reply to this one.)



On Sun, 11 Feb 2007 17:52:20 +0200, Kristian Kilpi <kjkilpi gmail.com>  =

wrote:

 String literals with mixins are a bit awkward sometimes (editor  =



 highlighting etc).

 Some special marks -- I use  { }  here -- could be used to mark a part=


  =


 of a source file as a string literal, just like /* */ marks a part of =


 =


 code as a comment. For example:

    mixin(
       {
        //this is a string literal block
        if(...) {
          ...
        }
      } 
    );

 The  { }  marks have a close relation, of course, with quotation marks=


  =


 "". But because there is a starting mark and an ending mark, you can  =



 nest them. (And because they are used to mark a part of a file as a  =



 string literal, they are not actually the part of the 'working code'  =



 just like the "" literals are, if you get what I'm trying to say.)

 E.g.

    alias  {
      str =3D  { foo }  ~  { bar } ;
      str ~=3D "blah";
      if(...) {
        ...
      }
    }  MyCode;

     mixin(MyCode);
Feb 11 2007
parent reply "Andrei Alexandrescu (See Website For Email)" <SeeWebsiteForEmail erdani.org> writes:
Kristian Kilpi wrote:

 Argh, I hit the 'reply' button instead of 'new mail' button, so this 
 mail was send to this thread. I'm posting this to the root thread 
 also... Sorry. (So please don't reply to this one.)


With certain news clients (e.g. Thunderbird) you can cancel a message, 
and the effect in this group is immediate. In Thunderbird, just focus on 
the message you want to delete (it must originate from you) and hit Del.

Andrei
Feb 11 2007
parent reply "Kristian Kilpi" <kjkilpi gmail.com> writes:
On Sun, 11 Feb 2007 18:16:26 +0200, Andrei Alexandrescu (See Website For  
Email) <SeeWebsiteForEmail erdani.org> wrote:

 Kristian Kilpi wrote:

 Argh, I hit the 'reply' button instead of 'new mail' button, so this  
 mail was send to this thread. I'm posting this to the root thread  
 also... Sorry. (So please don't reply to this one.)


 With certain news clients (e.g. Thunderbird) you can cancel a message,  
 and the effect in this group is immediate. In Thunderbird, just focus on  
 the message you want to delete (it must originate from you) and hit Del.

 Andrei


That's a good to known. Thanks.

Hmmm, yes, in Opera (what I'm using) you can right click a message in the  
Send view and select 'Cancel post'.
Feb 11 2007
parent Sean Kelly <sean f4.ca> writes:
Kristian Kilpi wrote:

 On Sun, 11 Feb 2007 18:16:26 +0200, Andrei Alexandrescu (See Website For 
 Email) <SeeWebsiteForEmail erdani.org> wrote:

 Kristian Kilpi wrote:

 Argh, I hit the 'reply' button instead of 'new mail' button, so this 
 mail was send to this thread. I'm posting this to the root thread 
 also... Sorry. (So please don't reply to this one.)


 With certain news clients (e.g. Thunderbird) you can cancel a message, 
 and the effect in this group is immediate. In Thunderbird, just focus 
 on the message you want to delete (it must originate from you) and hit 
 Del.

 Andrei


 
 That's a good to known. Thanks.
 
 Hmmm, yes, in Opera (what I'm using) you can right click a message in 
 the Send view and select 'Cancel post'.


The only issue being that if the post is to a widely available 
newsgroup, if you wait too long then it may have already propagated to 
other servers.


Sean
Feb 11 2007
prev sibling next sibling parent reply Witold Baryluk <baryluk mpi.int.pl> writes:
Nice ideas.

I'm also thinking on few powerful things.

Compile time XML parser:
 - configuration files:

   config.xml:
     <start><tag v="aa"></start>

   mixin(buildclasses!(xml(import("config.xml")));
   writefln(xml.start.tag.v);  // will print aa

   mixin(xpath!(xml, val, "start/tag.v"));
   writefln(val);    // "aa"

   not so usefull, but:

 - glade gui:
   mixin(glade!(xml(import("gui.glade")));
   with automatic bindings?

Compile time type safe SQL (working on it!):

  create_tables.sql:
    CREATE TABLE sale (id INT, name VARCHAR, price FLOAT);

  mixin(addtables!("create_tables.sql"));

  float p = 100.00;
  auto rows = sql!("SELECT id, name, price FROM sale WHERE price < $1").sql(p);
// p MUST BE float

  foreach (r; rows) {
    writefln("%d: %s = %f", r.id, r.name, r.price); 
         // yes! p.price, not p["price"].getFloat(), or sth similiar
    char[] sid = r.id;       // error!
  }

  Note: it is very VERY complicated to parse sql select with all features in
compile
  time, but i have made some progress...

Pattern matching system (like in erlang, or nemerle):
  struct Eqn {
     enum typ {
       SYMBOL, FUNC, ENVVAR, DOUBLE
     };
     union u {
       char[] s;
       F f;
       char[] var;
       double d;
     };
  }

  Eqn e = { typ: DOUBLE, u.d: 3.4 };

  mixin(match!(
       case e of
         (SYMBOL, u.s as s) =>
            writefln(\"s=%s\", s),
         (DOUBLE, u.d as d) =>
            writefln(\"d=%f\", d),
         _ =>
            writefln(\"other\");
       end."));

   or:

   int[] a = [1,6,7];

   mixin(match!(
       case %1 of
         [Head | Tail] =>
            writefln(\"do something with head %d\", Head),
            Head + self(Tail), // recursion
         [Last] =>
            writefln(\"Last\"),
            Last
         [] =>
            writefln(\"empty\"),
            0
         _ =>
            assert(false, \"error\"),
         end.", "funk(int[]) : int"));

   writefln(funk(a));     //14



There is endless posibilities to extend syntax.
We can practicly implement some small langauages in this way,
and emit D code with some templates.

This is pretty cool, but we will need even quicker compilation time, for this
to be
fully useful.

Even after compiler optimalisations, IDE will have problems with parsing such
things.

I think we also should have regexp functions accessible in compile time. It is
painful to
parse things by hand.

-- 
Witold Baryluk
Feb 11 2007
parent janderson <askme me.com> writes:
Witold Baryluk wrote:

 Nice ideas.

 I'm also thinking on few powerful things.

 Compile time XML parser:
  - configuration files:

    config.xml:
      <start><tag v="aa"></start>

    mixin(buildclasses!(xml(import("config.xml")));
    writefln(xml.start.tag.v);  // will print aa

    mixin(xpath!(xml, val, "start/tag.v"));
    writefln(val);    // "aa"


I think XML reading will be really useful.  In my game - programming - 
experience XML (including the binary version) is good for proto-typing 
but it eventually needs to be replaced by a binary format.  It's just to 
slow to phase, you want your data in the same order as the data 
structures, not all over the place.   Compile time xml would solve that 
(sort of because you've moved the bottleneck).

However most of the time in my field design/art can't compile the 
executables themselves.  A possible work around is to compile XML dll's 
on the fly, the final version would not need to do this.



 There is endless posibilities to extend syntax.
 We can practicly implement some small langauages in this way,
 and emit D code with some templates.

 This is pretty cool, but we will need even quicker compilation time, 


for this to be

 fully useful.


Agreed.

 Even after compiler optimalisations, IDE will have problems with 


parsing such things.

They could always be put into another file if this is of concern.  That 
way xml will look fine (if the editor is smart enough).  Having said 
that most DSL scripting languages don't get syntax highlighting, and its 
not worth the effort.


 I think we also should have regexp functions accessible in compile 


time. It is painful to

 parse things by hand.


I agree, but I think they are only a sub-set of what we need.  I think 
regex extentions could be implemented by a suped-up 
template/compile-time language.
Feb 11 2007
prev sibling next sibling parent janderson <askme me.com> writes:
janderson wrote:

 A couple of people have been asking questions about pseudo-real-life 
 examples for the new compile time code aspects of D.  I ask this 
 question for my interest as well.
 
 Assuming that we have a good mechanism to process the input how could 
 compile time code improve your every-day proficiency?


What about string enums, this could be done with reflection however you 
could do something like this now:

mixin(
Enum!("Colours",
"Red",
"Blue",
"Green"
));

//Or flags
mixin(
Flags!("Colours",
"Red",
"Blue",
"Green"
));

Colours c;
...
char[] str = ToString(c);

-Joel
Feb 11 2007
prev sibling parent janderson <askme me.com> writes:
Here's another example:

.fx shaders contain shader constants which include float(1,2,3,4), 
int(1,2,3,4), textures, matrices and other constants.  To access these 
parameters you can use a bulky interface (i'm doing this from memory so 
the syntax may be 100%):

DXHANDLE ambient; //Create handle
effect->GetHandle(ambient, "ambient");
//ect..

//Get the value.
float[4] amb = effect->GetValue(ambient, sizeof(float[4]));

//Set the value
effect->SetFloat(ambient, amb, sizeof(float[4]));


Typically, you need to write this for ever shader.  This can really 
start to add up when you have 100's of shaders.  These values are 
normally data driven from some file (ie material).  Also the rendering 
code for each shader is typically boilerplate.  Its even worse if you 
try to optimize it into some sort of state-scene-graph to minimize state 
switches.  You can also get things like meta data from data types which 
are useful for building gui's which represent the shader.

If you could wrap all this into a mixin that pulls information out from 
the shader, it would be an awesome time saver.  The boilerplate C++ code 
(which would take me to long to design here) would still be written 
mostly in C++.

//In D
mixin(shader!(import("boilerplateShader.d"), import("shader1.fx")));

...

//Shader.fx
float4 k_a  <
	string UIName = "Ambient";
	> = float4( 0.47f, 0.47f, 0.47f, 1.0f );    // ambient
	

ect...


// transformations
float4x4 World      : 		WORLD;
float4x4 View       : 		VIEW;
float4x4 Projection : 		PROJECTION;
float4x4 WorldViewProj : 	WORLDVIEWPROJECTION;
float4x4 WorldView : 		WORLDVIEW;

//...

VS_OUTPUT VS(
     float3 Pos  : POSITION,
     float3  col	: COLOR,
     float3 Norm : NORMAL,
     float2 Tex  : TEXCOORD0)
{
	return ambient;
}

float4 PS(
     float4 Diff : COLOR0,
     float4 Spec : COLOR1,
     float2 Tex  : TEXCOORD0,
     float2 Tex1 : TEXCOORD1 ) : COLOR
{
     float4 color =  Diff + Spec;
     return  color;
}

technique DefaultTechnique
{
     pass P0
     {
         // shaders
         VertexShader = compile vs_1_1 VS();
         PixelShader  = compile ps_1_1 PS();
     }
}

Then you could interact with it like:

shader1 shader = StateTree.getShader1();
shader.ambient = {1, 0, 1};

//...
foreach (shadertype shtype; shader.types)
{
	AddToGui(shtype);
}

ect...
Note that this would mean, that adding/removing properties for an fx 
shader compiled at run-time would not be reflected however you normally 
don't care (and you can still modify the shader real-time).

-Joel
Feb 16 2007