www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Trying to reproduce Node.js async waterfall pattern..

reply "Christian Beaumont" <chris socialmagnetics.com> writes:
Hi,

I just started learning D, and thought I'd throw myself in at the 
deep end with some meta-programming, trying to write the 
equivalent of the commonly used, async waterfall, and also, 
because I'd like to use it...

If you aren't familiar with it, waterfall is a function that is 
passed a sequence of functions as its arguments that are to be 
executed in order... (it's a pattern for async programming).

Here is an example in Node.js...

waterfall(
   function(asyncCallback){
     asyncCallback(null, "one");
   },
   function(lastResult, asyncCallback){
     // lastResult equals "one"
     asyncCallback(null, "two");
   },
   function(lastResult, asyncCallback){
     // lastResult equals "two"
     asyncCallback(null, "done");
   }
   ,
   // final callback
   function (error, finalResult)
   {
      // result equals "done"
   }
);

Each function is given a callback, that when called, steps the 
waterfall forward on to the next function to process.  If an 
error is passed to the callback (instead of null), then the 
waterfall stops processing and calls the final callback at the 
end of the chain.

This is how I have it implemented so far, but as you can see, 
there are some issues...

import std.stdio;
import std.conv;
import std.exception;

alias Callback = void delegate(Exception error, string result);
alias AsyncFunc = void function(Callback cb);

static Waterfall(TLastResult, Funcs...)(Funcs funcs, Callback 
finalCallback, TLastResult lastResult)
{
	static if (funcs.length)
	{
		auto cb = (Exception error, string result)
		{
			if (error is null)
				Waterfall(funcs[1 .. $], finalCallback, result);
			else
				finalCallback(error, result);
		};

		funcs[0](cb);
	}
	else
		finalCallback(null, lastResult);
}

static Waterfall(Funcs...)(Funcs funcs, Callback finalCallback)
{
	static if (funcs.length)
	{
		auto cb = (Exception error, string result)
		{
			if (error is null)
				Waterfall(funcs[1 .. $], finalCallback, result);
			else
				finalCallback(error, result);
		};

		funcs[0](cb);
	}
	else
		finalCallback(null, lastResult);
}

void main()
{
	Waterfall(
		(Callback cb) { writeln("fn0"); cb(null, "one"); },
		(Callback cb) { writeln("fn1"); cb(null, "two"); },
		// (Callback cb) { writeln("fnError"); cb(new Exception("Bad 
joojoo"), "two"); },
		(Callback cb) { writeln("fn2"); cb(null, "done"); },
		(Exception error, string result)
		{
			if (error !is null)
				writeln("Error = " ~ error.to!string);

			if (result !is null)
				writeln("Result = " ~ result.to!string);
		}
	);
}

1) I can't see any way to get the compiler to deduce the type of 
"Funcs...".  I had an urge to somehow specialize the variadic 
"Funcs..." but I couldn't figure out any syntax to do that.  
Well, because of that, I have to repeatedly say (Callback cb) 
instead of (cb).

2) I can't see a path to flow the output type of the previous 
callback to the input "TLastResult" for the next... so it's stuck 
on "string" :(

3) I had to use a specialization to deal with the "head" case; 
(the first function doesn't have an input from a previous 
result). Minor, but niggly.

Any input on how to proceed would be great!

thanks!
Christian

BTW, you can read more about async-waterfall here...
https://www.npmjs.org/package/async-waterfall
Jun 23 2014
parent reply Philippe Sigaud via Digitalmars-d-learn writes:
On Mon, Jun 23, 2014 at 9:39 PM, Christian Beaumont via
Digitalmars-d-learn <digitalmars-d-learn puremagic.com> wrote:


 Each function is given a callback, that when called, steps the waterfall
 forward on to the next function to process.  If an error is passed to the
 callback (instead of null), then the waterfall stops processing and calls
 the final callback at the end of the chain.


Just to be sure: whether or not an error is passed to a callback, the
final callback is always called?
I mean, the last callback could also be called *only when something
fails*, a bit like a default case in a switch, or an error-handling
routine.



 1) I can't see any way to get the compiler to deduce the type of "Funcs...".


What do you mean? The compiler does deduce the type of Funcs.


 I had an urge to somehow specialize the variadic "Funcs..." but I couldn't
 figure out any syntax to do that.


What do you mean by 'specialize'?


  Well, because of that, I have to
 repeatedly say (Callback cb) instead of (cb).


(cb) { cb(null, "one");} is possible, but that means it's a function
template, not a function.
You can get this syntax by making the callbacks template arguments,
which means they must be known at compile-time. Is that OK with you or
do you need the possibility to define the callbacks at runtime?




 2) I can't see a path to flow the output type of the previous callback to
 the input "TLastResult" for the next... so it's stuck on "string" :(


I don't get it: none of your callbacks have a return type per se: they
all return 'void'.
Do you want callbacks that really return something?



 3) I had to use a specialization to deal with the "head" case; (the first
 function doesn't have an input from a previous result). Minor, but niggly.


You can test if func[0] takes one or two arguments:

import std.traits: isCallable, ReturnType;

static if (isCallable!(Func[0]) && ReturnType!(Func[0]).length == 1)
Jun 23 2014
parent reply "Christian Beaumont" <chris socialmagnetics.com> writes:
 Just to be sure: whether or not an error is passed to a 
 callback, the
 final callback is always called?
 I mean, the last callback could also be called *only when 
 something
 fails*, a bit like a default case in a switch, or an 
 error-handling
 routine.


Yes, the final callback is always called, but if an error is 
passed to the callback by any of the main steps in the "sequence 
ladder", it will immediately jump to the final callback and not 
execute further steps.


 What do you mean? The compiler does deduce the type of Funcs.


If you look at where I call Waterfall() in main, you'll see I had 
to manually specify (Callback cb) instead of just (cb); since it 
didn't know that the Funcs... were of type AsyncFunc


 What do you mean by 'specialize'?


That is to say, there is no way I can see, to say that the 
variadic template parameter "Funcs..." are all AsyncFunc's.  I 
think I noticed that in Swift you can say something like 
"Funcs:AsyncFunc..." to specialize the variadic.  No swift 
expert, but was just browsing around today trying to compare how 
you might do it in C++ or other languages.


 (cb) { cb(null, "one");} is possible, but that means it's a 
 function
 template, not a function.
 You can get this syntax by making the callbacks template 
 arguments,
 which means they must be known at compile-time. Is that OK with 
 you or
 do you need the possibility to define the callbacks at runtime?


The goal was to do as much as possible at compile time.  Could 
you elaborate on this a bit.  I guess the answer is, yes, it's 
okay with me.


 I don't get it: none of your callbacks have a return type per 
 se: they
 all return 'void'.
 Do you want callbacks that really return something?


Yes, the callbacks at step0 should output a type in the result 
which is specific to step0, and then that type should be fed in 
as a secondary parameter to step1.

I didn't get that far, as I was already stuck.


 You can test if func[0] takes one or two arguments:

 import std.traits: isCallable, ReturnType;

 static if (isCallable!(Func[0]) && ReturnType!(Func[0]).length 
 == 1)


Ah, yes, that sounds reasonable, I already thought something like 
that may do the trick.

thanks for your patience!
Jun 23 2014
next sibling parent reply "Christian Beaumont" <chris socialmagnetics.com> writes:
Just an idea that popped into my head... Maybe I can use variant 
for the input/output types?  I haven't looked at it yet, so I'm 
not sure what it does, or the performance costs.

I realized that because the final callback always gets called, 
and the types of the intermediate steps may be different, there 
is actually no way at all to define a completely uniformly type 
safe final callback...

although the point is a little moot, because the only way you end 
up in the final callback is either by an Exception or a Result 
from the finalCallback - 1 step.

So you can actually never get both an Exception and a Result
Jun 23 2014
next sibling parent Philippe Sigaud via Digitalmars-d-learn writes:
On Tue, Jun 24, 2014 at 2:55 AM, Christian Beaumont via
Digitalmars-d-learn <digitalmars-d-learn puremagic.com> wrote:

 Just an idea that popped into my head... Maybe I can use variant for the
 input/output types?  I haven't looked at it yet, so I'm not sure what it
 does, or the performance costs.


It imitates you standard variant type from dynamic languages. It sure
would make translating Javascript code easier.
I don't know its performance cost. I'd say the only way to know is to
compare your code with Node.js.


 I realized that because the final callback always gets called, and the types
 of the intermediate steps may be different, there is actually no way at all
 to define a completely uniformly type safe final callback...


Unless you define the final callback to be called only when an
exception is passed, as a special error-handling branch. Else the
chain would pass through the first function, then the second, up to
the Nth-1, and stop there.
Jun 24 2014
prev sibling parent Philippe Sigaud via Digitalmars-d-learn writes:
 Just an idea that popped into my head... Maybe I can use variant for the
 input/output types?  I haven't looked at it yet, so I'm not sure what it
 does, or the performance costs.


 It imitates you standard variant type from dynamic languages. It sure
 would make translating Javascript code easier.
 I don't know its performance cost. I'd say the only way to know is to
 compare your code with Node.js.


OK, replying to myself here :-)
I tried to code it with templates, but got to the point where result
types depend on runtime values (namely, is error null or not?), which
is not possible in D.
So I'd say, probably the easiest way to keep the null/error scheme is
to use Variant everywhere. That what I tend to do when converting code
from untyped languages anyway.
Jun 24 2014
prev sibling parent Philippe Sigaud via Digitalmars-d-learn writes:
 Yes, the final callback is always called, but if an error is passed to the
 callback by any of the main steps in the "sequence ladder", it will
 immediately jump to the final callback and not execute further steps.


OK.



 What do you mean? The compiler does deduce the type of Funcs.



 If you look at where I call Waterfall() in main, you'll see I had to
 manually specify (Callback cb) instead of just (cb); since it didn't know
 that the Funcs... were of type AsyncFunc


you can use std.typetuple.allSatisfy with a helper template:

enum isAsyncFunc(T) = is(T == AsyncFunc);

... (Funcs...)(Funcs funcs) if (allSatisfy!(isAsyncFunc, Funcs))
{ ... }




 (cb) { cb(null, "one");} is possible, but that means it's a function
 template, not a function.
 You can get this syntax by making the callbacks template arguments,
 which means they must be known at compile-time. Is that OK with you or
 do you need the possibility to define the callbacks at runtime?



 The goal was to do as much as possible at compile time.  Could you elaborate
 on this a bit.  I guess the answer is, yes, it's okay with me.


I mean, it's possible to get a (cb){ some code } syntax, but as that
define a function template, it has no real type: it cannot be a
runtime argument, only an alias template parameter. That means that
all callbacks must be defined in your code, none can come from user
input or runtime computation.



 I don't get it: none of your callbacks have a return type per se: they
 all return 'void'.
 Do you want callbacks that really return something?



 Yes, the callbacks at step0 should output a type in the result which is
 specific to step0, and then that type should be fed in as a secondary
 parameter to step1.

 I didn't get that far, as I was already stuck.


OK, I get it.




 thanks for your patience!


Well, we are there to explain :-)
I'll have try and code something. If I can get it to work, I'll post it there.
Jun 24 2014