www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Generators in D

reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Hi,

I've written some proof of concept code of generator pattern, example:

void genSquares(out int result, int from, int to)
{
     foreach (x; from .. to + 1)
     {
         yield!result(x * x);
     }
}

void main(string[] argv)
{
     foreach (sqr; generator(&genSquares, 10, 20))
         writeln(sqr);
}

posted on github:

https://github.com/pszturmaj/dgenerators

Thanks to Sean Kelly who wrote Fiber code.

Piotr
May 10 2011
next sibling parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
I forgot to ask. Any comments or suggestions?
May 10 2011
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 05/10/2011 07:06 PM, Piotr Szturmaj wrote:

 I forgot to ask. Any comments or suggestions?


This is solid work, and very interesting. I wonder how we can integrate 
it best with Phobos (e.g. put in its own module vs. integrating into 
std.range).

I suggest you write an article about this artifact. Don't forget that 
the deadline for the iPad contest is June 1st.

A few nits follow.

* "Fiber fiber = Fiber.getThis();" -> "auto fiber = Fiber.getThis();"

* "_generator" -> "Generator" (private names are not exempt from the 
recommended conventions)

* "staticLength!T" -> what's wrong with T.length?

* import inside a template is unusual, why did you need it that way?

* "pure nothrow ValueType front()" -> that probably should be a 
 property (sigh)

* can you implement save() to make this a forward range (e.g. by 
creating a new fiber that has its own state)?


Thanks,

Andrei
May 11 2011
next sibling parent reply %u <wfunction hotmail.com> writes:
On 05/10/2011 07:06 PM, Piotr Szturmaj wrote:

 I forgot to ask. Any comments or suggestions?


This is **beast**.

Just one thing: Would it work correctly if I was using fibers in my
own code?
May 11 2011
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
%u wrote:

 On 05/10/2011 07:06 PM, Piotr Szturmaj wrote:

 I forgot to ask. Any comments or suggestions?


 This is **beast**.

 Just one thing: Would it work correctly if I was using fibers in my
 own code?


I don't know. If fibers may call other fibers then it should work.
May 11 2011
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/11/11 5:09 PM, Piotr Szturmaj wrote:

 %u wrote:

 On 05/10/2011 07:06 PM, Piotr Szturmaj wrote:

 I forgot to ask. Any comments or suggestions?


 This is **beast**.

 Just one thing: Would it work correctly if I was using fibers in my
 own code?


 I don't know. If fibers may call other fibers then it should work.


You may want to add unittests that use two or more generators in 
interesting ways.

Andrei
May 11 2011
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On May 11, 2011, at 3:09 PM, Piotr Szturmaj wrote:


 %u wrote:

 On 05/10/2011 07:06 PM, Piotr Szturmaj wrote:

 I forgot to ask. Any comments or suggestions?


=20
 This is **beast**.
=20
 Just one thing: Would it work correctly if I was using fibers in my
 own code?


=20
 I don't know. If fibers may call other fibers then it should work.


They can.  The only issue to be aware of is that non-shared static data =
is thread-local, not fiber-local, so some code may exhibit unexpected =
behavior if not written with fibers in mind.=
May 17 2011
prev sibling parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Andrei Alexandrescu wrote:

 On 05/10/2011 07:06 PM, Piotr Szturmaj wrote:

 I forgot to ask. Any comments or suggestions?


 This is solid work, and very interesting. I wonder how we can integrate
 it best with Phobos (e.g. put in its own module vs. integrating into
 std.range).


This code is very short so integrating it with std.range might be a 
better choice.


 I suggest you write an article about this artifact. Don't forget that
 the deadline for the iPad contest is June 1st.


I'll see what I can do :-)


 A few nits follow.

 * "Fiber fiber = Fiber.getThis();" -> "auto fiber = Fiber.getThis();"

 * "_generator" -> "Generator" (private names are not exempt from the
 recommended conventions)


Fixed.


 * "staticLength!T" -> what's wrong with T.length?


I have seen this many times in the library, example from std.traits:

template ReturnType(/+   BUG4217   +/func...)
     if (/+   BUG4333   +/staticLength!(func) == 1)

And I realized that as a good idea, because bug 4333 is not closed yet.


 * import inside a template is unusual, why did you need it that way?


I needed TypeTuple only inside this template, so I though I will hide it 
there :-) But after rethinking it, I think it's better to put that in 
module beginning with other imports.


 * "pure nothrow ValueType front()" -> that probably should be a
  property (sigh)


Fixed in empty() as well.


 * can you implement save() to make this a forward range (e.g. by
 creating a new fiber that has its own state)?


It is not that simple I guess. Fiber class doesn't give any opportunity 
to save current stack and I think it might be wrong to do this, because 
of GC issues. But I don't certainly know.

I think that author of the Fiber class could answer this better than me.
If Fiber will expose some additional API, then I would certainly 
implement save().


 Thanks,

 Andrei


Thanks for the feedback!
May 11 2011
next sibling parent KennyTM~ <kennytm gmail.com> writes:
On May 12, 11 06:06, Piotr Szturmaj wrote:

 I have seen this many times in the library, example from std.traits:

 template ReturnType(/+   BUG4217   +/func...)
      if (/+   BUG4333   +/staticLength!(func) == 1)

 And I realized that as a good idea, because bug 4333 is not closed yet.


I don't think your template triggers 4333 though.
May 11 2011
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On May 11, 2011, at 3:06 PM, Piotr Szturmaj wrote:


 Andrei Alexandrescu wrote:

 * can you implement save() to make this a forward range (e.g. by
 creating a new fiber that has its own state)?


=20
 It is not that simple I guess. Fiber class doesn't give any =


opportunity to save current stack and I think it might be wrong to do =
this, because of GC issues. But I don't certainly know.

=20
 I think that author of the Fiber class could answer this better than =


me.

 If Fiber will expose some additional API, then I would certainly =


implement save().

Cloning a Fiber would essentially produce a shallow copy--if the Fiber's =
stack contains references to data elsewhere on the same stack, the =
clone's stack would reference the original's.  So while this could be =
done, I don't think it's advisable.=
May 17 2011
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Sean Kelly wrote:

 On May 11, 2011, at 3:06 PM, Piotr Szturmaj wrote:


 Andrei Alexandrescu wrote:

 * can you implement save() to make this a forward range (e.g. by
 creating a new fiber that has its own state)?


 It is not that simple I guess. Fiber class doesn't give any opportunity to save
 current stack and I think it might be wrong to do this, because of 




GC issues.

 But I don't certainly know.

 I think that author of the Fiber class could answer this better than me.
 If Fiber will expose some additional API, then I would certainly implement
save().


 Cloning a Fiber would essentially produce a shallow copy--if the Fiber's stack
contains
 references to data elsewhere on the same stack, the clone's stack > 


would reference the original's.  So while this could be done,

 I don't think it's advisable.


But couldn't Fiber's stack be scanned for references to itself and 
readjusted?
May 17 2011
parent reply Sean Kelly <sean invisibleduck.org> writes:
On May 17, 2011, at 2:37 PM, Piotr Szturmaj wrote:


 Sean Kelly wrote:

 On May 11, 2011, at 3:06 PM, Piotr Szturmaj wrote:
=20

 Andrei Alexandrescu wrote:

 * can you implement save() to make this a forward range (e.g. by
 creating a new fiber that has its own state)?


=20
 It is not that simple I guess. Fiber class doesn't give any =






opportunity to save

 current stack and I think it might be wrong to do this, because of =






GC issues.

 But I don't certainly know.
=20
 I think that author of the Fiber class could answer this better than =






me.

 If Fiber will expose some additional API, then I would certainly =






implement save().

=20
 Cloning a Fiber would essentially produce a shallow copy--if the =




Fiber's stack contains

 references to data elsewhere on the same stack, the clone's stack > =




would reference the original's.  So while this could be done,

 I don't think it's advisable.


=20
 But couldn't Fiber's stack be scanned for references to itself and =


readjusted?

Without type information, there's no way to be sure that something is =
actually a reference, so corruption could occur.=
May 17 2011
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Sean Kelly wrote:

 On May 17, 2011, at 2:37 PM, Piotr Szturmaj wrote:


 Sean Kelly wrote:

 On May 11, 2011, at 3:06 PM, Piotr Szturmaj wrote:


 Andrei Alexandrescu wrote:

 * can you implement save() to make this a forward range (e.g. by
 creating a new fiber that has its own state)?


 It is not that simple I guess. Fiber class doesn't give any opportunity to save
 current stack and I think it might be wrong to do this, because of GC issues.
 But I don't certainly know.

 I think that author of the Fiber class could answer this better than me.
 If Fiber will expose some additional API, then I would certainly implement
save().


 Cloning a Fiber would essentially produce a shallow copy--if the Fiber's stack
contains
 references to data elsewhere on the same stack, the clone's stack>  would
reference the original's.  So while this could be done,
 I don't think it's advisable.


 But couldn't Fiber's stack be scanned for references to itself and readjusted?


 Without type information, there's no way to be sure that something is actually
a reference, so corruption could occur.


Then, how GC scan Fiber's stack for references? I thought some portion 
of GC code could be used for this.
May 18 2011
parent reply Sean Kelly <sean invisibleduck.org> writes:
On May 18, 2011, at 8:00 AM, Piotr Szturmaj wrote:


 Sean Kelly wrote:

 On May 17, 2011, at 2:37 PM, Piotr Szturmaj wrote:

=20
 But couldn't Fiber's stack be scanned for references to itself and =






readjusted?

=20
 Without type information, there's no way to be sure that something is =




actually a reference, so corruption could occur.

=20
 Then, how GC scan Fiber's stack for references? I thought some portion =


of GC code could be used for this.

The GC is conservative.  If it encounters a value that matches the =
address of GCed memory, it won't collect the memory.=
May 18 2011
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Sean Kelly wrote:

 On May 18, 2011, at 8:00 AM, Piotr Szturmaj wrote:


 Sean Kelly wrote:

 On May 17, 2011, at 2:37 PM, Piotr Szturmaj wrote:

 But couldn't Fiber's stack be scanned for references to itself and readjusted?


 Without type information, there's no way to be sure that something is actually
a reference, so corruption could occur.


 Then, how GC scan Fiber's stack for references? I thought some portion of GC
code could be used for this.


 The GC is conservative.  If it encounters a value that matches the address of
GCed memory, it won't collect the memory.


Ok. Maybe a requirement that cloneable fiber's function must be marked 
as  safe would work, as  safe functions can't take the address of a 
local variable or function parameter. Also asm and pointer arithmetic 
are prohibited. This way Fiber's stack can't have references to itself.

That should be enough to implement ForwardRange's save() for generators. 
Or still, I'm missing something I don't know yet :)
May 18 2011
parent reply Sean Kelly <sean invisibleduck.org> writes:
On May 18, 2011, at 5:07 PM, Piotr Szturmaj wrote:


 Sean Kelly wrote:

=20
 On May 18, 2011, at 8:00 AM, Piotr Szturmaj wrote:
=20

 Sean Kelly wrote:

 On May 17, 2011, at 2:37 PM, Piotr Szturmaj wrote:

=20
 But couldn't Fiber's stack be scanned for references to itself and =










readjusted?

=20
 Without type information, there's no way to be sure that something =








is actually a reference, so corruption could occur.

=20
 Then, how GC scan Fiber's stack for references? I thought some =






portion of GC code could be used for this.

=20
 The GC is conservative.  If it encounters a value that matches the =




address of GCed memory, it won't collect the memory.

=20
 Ok. Maybe a requirement that cloneable fiber's function must be marked =


as  safe would work, as  safe functions can't take the address of a =
local variable or function parameter. Also asm and pointer arithmetic =
are prohibited. This way Fiber's stack can't have references to itself.

=20
 That should be enough to implement ForwardRange's save() for =


generators. Or still, I'm missing something I don't know yet :)

Referenced classes would still be shared between fibers.  I don't know =
if that would be an issue or not.=
May 18 2011
parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Sean Kelly wrote:

 On May 18, 2011, at 5:07 PM, Piotr Szturmaj wrote:


 Sean Kelly wrote:

 On May 18, 2011, at 8:00 AM, Piotr Szturmaj wrote:


 Sean Kelly wrote:

 On May 17, 2011, at 2:37 PM, Piotr Szturmaj wrote:

 But couldn't Fiber's stack be scanned for references to itself and readjusted?


 Without type information, there's no way to be sure that something is actually
a reference, so corruption could occur.


 Then, how GC scan Fiber's stack for references? I thought some portion of GC
code could be used for this.


 The GC is conservative.  If it encounters a value that matches the address of
GCed memory, it won't collect the memory.


 Ok. Maybe a requirement that cloneable fiber's function must be marked as
 safe would work, as  safe functions can't take the address of a local variable
or function parameter. Also asm and pointer arithmetic are prohibited. This way
Fiber's stack can't have references to itself.

 That should be enough to implement ForwardRange's save() for generators. Or
still, I'm missing something I don't know yet :)


 Referenced classes would still be shared between fibers.  I don't know if that
would be an issue or not.


For in-process cloning that should not be an issue. Generator coders 
should be aware of cloning behaviour.
May 19 2011
prev sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On May 11, 11 08:06, Piotr Szturmaj wrote:

 I forgot to ask. Any comments or suggestions?


The argument match assertion is too strict. The following does not compile:

---------------
void main() {
     auto a = generator( (out int result, double e) {
         yield!result(1);
     }, 4);
     writeln(a);
}
---------------
x.d(40): Error: static assert  "Arguments do not match"
x.d(104):        instantiated from here: _generator!(void delegate(out 
int result, double e),int)
x.d(108):        instantiated from here: generator!(void delegate(out 
int result, double e),int)
---------------

Perhaps you could just remove the assert? The compiler will complain if 
the arguments cannot be accepted anyway.
May 11 2011
parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
KennyTM~ wrote:

 On May 11, 11 08:06, Piotr Szturmaj wrote:

 I forgot to ask. Any comments or suggestions?


 The argument match assertion is too strict. The following does not compile:

 ---------------
 void main() {
 auto a = generator( (out int result, double e) {
 yield!result(1);
 }, 4);
 writeln(a);
 }
 ---------------
 x.d(40): Error: static assert "Arguments do not match"
 x.d(104): instantiated from here: _generator!(void delegate(out int
 result, double e),int)
 x.d(108): instantiated from here: generator!(void delegate(out int
 result, double e),int)
 ---------------

 Perhaps you could just remove the assert? The compiler will complain if
 the arguments cannot be accepted anyway.


You're right, it's better without that assert. Fixed.
May 11 2011
prev sibling next sibling parent reply so <so so.so> writes:
 void genSquares(out int result, int from, int to)
          yield!result(x * x);


Wow! i had no idea about this feature! (Variables as template parameters)
Do we have any documentation, usage example (anything!)? Pretty please.
May 12 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
 void genSquares(out int result, int from, int to)
          yield!result(x * x);


 Wow! i had no idea about this feature! (Variables as template parameters)
 Do we have any documentation, usage example (anything!)? Pretty please.


The argument is an alias, so just a symbol.

See: http://digitalmars.com/d/2.0/template.html (Template Alias Arguments)

The trick is that the template is instantiated as a local function, which has
access to the enclosing scope. (and therewith to the local symbol that is
aliased)

Timon
May 12 2011
parent so <so so.so> writes:
 The argument is an alias, so just a symbol.

 See: http://digitalmars.com/d/2.0/template.html (Template Alias  
 Arguments)

 The trick is that the template is instantiated as a local function,  
 which has
 access to the enclosing scope. (and therewith to the local symbol that  
 is aliased)

 Timon


And i was thinking that i knew all about templates.
Impressive, thanks!
May 12 2011
prev sibling parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Piotr Szturmaj wrote:

 Hi,

 I've written some proof of concept code of generator pattern, example:

 void genSquares(out int result, int from, int to)
 {
 foreach (x; from .. to + 1)
 {
 yield!result(x * x);
 }
 }

 void main(string[] argv)
 {
 foreach (sqr; generator(&genSquares, 10, 20))
 writeln(sqr);
 }

 posted on github:

 https://github.com/pszturmaj/dgenerators

 Thanks to Sean Kelly who wrote Fiber code.

 Piotr


Well, I've just found this: 
http://www.mail-archive.com/digitalmars-d puremagic.com/msg30204.html. 
It seems someone did it before ;)

In regards to serializable fibers, I've found some interesting potential 
usage - session handling in HTTP server. Fiber code could represent 
session and could be saved to disk in the middle of execution, just like 
sessions are saved in PHP. This is used to offload memory when handling 
great number of sessions.

If serialization will be clever enough, these fibers could even be 
shared across different servers! This is really a requirement in load 
balanced environments.

Piotr
May 17 2011
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/17/11 9:37 AM, Piotr Szturmaj wrote:

 Piotr Szturmaj wrote:

 Hi,

 I've written some proof of concept code of generator pattern, example:

 void genSquares(out int result, int from, int to)
 {
 foreach (x; from .. to + 1)
 {
 yield!result(x * x);
 }
 }

 void main(string[] argv)
 {
 foreach (sqr; generator(&genSquares, 10, 20))
 writeln(sqr);
 }

 posted on github:

 https://github.com/pszturmaj/dgenerators

 Thanks to Sean Kelly who wrote Fiber code.

 Piotr


 Well, I've just found this:
 http://www.mail-archive.com/digitalmars-d puremagic.com/msg30204.html.
 It seems someone did it before ;)


That's why it's important to follow through with a formal submission to 
Phobos.


 In regards to serializable fibers, I've found some interesting potential
 usage - session handling in HTTP server. Fiber code could represent
 session and could be saved to disk in the middle of execution, just like
 sessions are saved in PHP. This is used to offload memory when handling
 great number of sessions.

 If serialization will be clever enough, these fibers could even be
 shared across different servers! This is really a requirement in load
 balanced environments.


Sounds very interesting!


Andrei
May 17 2011
prev sibling next sibling parent reply Jose Armando Garcia <jsancio gmail.com> writes:
 If serialization will be clever enough, these fibers could even be shared
 across different servers! This is really a requirement in load balanced
 environments.


A very old research paper (I think it was for the amoeba project)
wrote a long time ago (I don't remember the wording) that is usually a
bad idea to migrate already running tasks (process, thread, fiber,
etc). Leading to the observation that it is probably best to do
offloading/load-balancing before the work is started. That is how most
modern scale-able architectures work. This is just an observation
since you haven't exactly said how to use Fiber to save Http sessions.

On Tue, May 17, 2011 at 11:37 AM, Piotr Szturmaj <bncrbme jadamspam.pl> wrote:

 Piotr Szturmaj wrote:

 Hi,

 I've written some proof of concept code of generator pattern, example:

 void genSquares(out int result, int from, int to)
 {
 foreach (x; from .. to + 1)
 {
 yield!result(x * x);
 }
 }

 void main(string[] argv)
 {
 foreach (sqr; generator(&genSquares, 10, 20))
 writeln(sqr);
 }

 posted on github:

 https://github.com/pszturmaj/dgenerators

 Thanks to Sean Kelly who wrote Fiber code.

 Piotr


 Well, I've just found this:
 http://www.mail-archive.com/digitalmars-d puremagic.com/msg30204.html. It
 seems someone did it before ;)

 In regards to serializable fibers, I've found some interesting potential
 usage - session handling in HTTP server. Fiber code could represent session
 and could be saved to disk in the middle of execution, just like sessions
 are saved in PHP. This is used to offload memory when handling great number
 of sessions.

 If serialization will be clever enough, these fibers could even be shared
 across different servers! This is really a requirement in load balanced
 environments.

 Piotr
May 17 2011
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Jose Armando Garcia wrote:

 If serialization will be clever enough, these fibers could even be shared
 across different servers! This is really a requirement in load balanced
 environments.


 A very old research paper (I think it was for the amoeba project)
 wrote a long time ago (I don't remember the wording) that is usually a
 bad idea to migrate already running tasks (process, thread, fiber,
 etc). Leading to the observation that it is probably best to do
 offloading/load-balancing before the work is started. That is how most
 modern scale-able architectures work. This is just an observation
 since you haven't exactly said how to use Fiber to save Http sessions.


There are some cases where it's useful. See SecondLife and EVE online. 
As for HTTP sessions, imagine some long wizard-like survey. User may 
fill half of that survey and may come back after week or so. She will 
start from the last checkpoint.
Of course such things are widely done using regular code, but fiber code 
would be certainly simpler and cleaner, and thus less error-prone.
There are of course some issues, f.i. how to deal with multiple 
requests/opened tabs or security issues. Such serialized fiber should be 
HMAC'ed and checked for authentication and integrity on deserialization.
May 18 2011
parent reply Sean Kelly <sean invisibleduck.org> writes:
On May 18, 2011, at 5:21 PM, Piotr Szturmaj wrote:


 Jose Armando Garcia wrote:

=20
 If serialization will be clever enough, these fibers could even be =






shared

 across different servers! This is really a requirement in load =






balanced

 environments.
=20


=20
 A very old research paper (I think it was for the amoeba project)
 wrote a long time ago (I don't remember the wording) that is usually =




a

 bad idea to migrate already running tasks (process, thread, fiber,
 etc). Leading to the observation that it is probably best to do
 offloading/load-balancing before the work is started. That is how =




most

 modern scale-able architectures work. This is just an observation
 since you haven't exactly said how to use Fiber to save Http =




sessions.

=20


=20
 There are some cases where it's useful. See SecondLife and EVE online. =


As for HTTP sessions, imagine some long wizard-like survey. User may =
fill half of that survey and may come back after week or so. She will =
start from the last checkpoint.

 Of course such things are widely done using regular code, but fiber =


code would be certainly simpler and cleaner, and thus less error-prone.

 There are of course some issues, f.i. how to deal with multiple =


requests/opened tabs or security issues. Such serialized fiber should be =
HMAC'ed and checked for authentication and integrity on deserialization.

Interesting issue.  For example, what if there's a code upgrade?=
May 18 2011
parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Sean Kelly wrote:

 On May 18, 2011, at 5:21 PM, Piotr Szturmaj wrote:


 Jose Armando Garcia wrote:

 If serialization will be clever enough, these fibers could even be shared
 across different servers! This is really a requirement in load balanced
 environments.


 A very old research paper (I think it was for the amoeba project)
 wrote a long time ago (I don't remember the wording) that is usually a
 bad idea to migrate already running tasks (process, thread, fiber,
 etc). Leading to the observation that it is probably best to do
 offloading/load-balancing before the work is started. That is how most
 modern scale-able architectures work. This is just an observation
 since you haven't exactly said how to use Fiber to save Http sessions.


 There are some cases where it's useful. See SecondLife and EVE online. As for
HTTP sessions, imagine some long wizard-like survey. User may fill half of that
survey and may come back after week or so. She will start from the last
checkpoint.
 Of course such things are widely done using regular code, but fiber code would
be certainly simpler and cleaner, and thus less error-prone.
 There are of course some issues, f.i. how to deal with multiple
requests/opened tabs or security issues. Such serialized fiber should be
HMAC'ed and checked for authentication and integrity on deserialization.


 Interesting issue.  For example, what if there's a code upgrade?


Yes, that's some serious problem. I guess, I was wrong with these long 
term sessions. I wonder how SecondLife authors got rid of this issue.
May 19 2011
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On May 17, 2011, at 7:37 AM, Piotr Szturmaj wrote:

=20
 In regards to serializable fibers, I've found some interesting =


potential usage - session handling in HTTP server. Fiber code could =
represent session and could be saved to disk in the middle of execution, =
just like sessions are saved in PHP. This is used to offload memory when =
handling great number of sessions.

=20
 If serialization will be clever enough, these fibers could even be =


shared across different servers! This is really a requirement in load =
balanced environments.

The trick is serializing the map of dynamic objects referenced by the =
Fiber's stack.  And more generally, dealing with any pointers on the =
stack at all.  It's the same problem as pointers in a memory-mapped =
file.  That said, you really shouldn't have to serialize Fibers just to =
offload memory.  That's what the virtual memory manager is for.=
May 17 2011
parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Sean Kelly wrote:

 On May 17, 2011, at 7:37 AM, Piotr Szturmaj wrote:

 In regards to serializable fibers, I've found some interesting potential
 usage - session handling in HTTP server. Fiber code could represent
 session and could be saved to disk in the middle of execution, just
 like sessions are saved in PHP. This is used to offload memory
 when handling great number of sessions.

 If serialization will be clever enough, these fibers could even be shared
 across different servers! This is really a requirement in load
 balanced environments.


 The trick is serializing the map of dynamic objects referenced by the Fiber's
 stack.  And more generally, dealing with any pointers on the stack
 at all.  It's the same problem as pointers in a memory-mapped file.
 That said, you really shouldn't have to serialize Fibers just to offload
 memory.  That's what the virtual memory manager is for.


While I fully agree on your opinion about virtual memory manager, I 
really meant something different. Offloading memory is not the sole reason.

I didn't state that before, but I thought of fibers which do not 
reference any external data. They should be self-contained in terms of 
referenced memory.

All external data should be passed as arguments to fiber function. Fiber 
should make a copy of all passed data.

Other ideas involve "persistent" objects that should always be there. 
Upon deserialization Fiber should be "reconnected" to these object -- it 
will just get fresh references.

Offloading memory is useful in case of sessions that may be unused for a 
long time. This include web sessions but not only them. Great examples 
are online games Second Life and EVE Online.

"SecondLife required that code be suspended at any point in time and 
that its entire state be serializable into a format suitable for storage 
into a database. Serialized state could then be restored at a different 
point in time or on a different computer (for example while moving from 
node to node)."

EVE Online uses stackless python in similar manner.

So, I find it very interesting, especially if it would be possible to 
use with D :-)

Here are some interesting links I found, about serializing 
fibers/coroutines/continuations/etc.:

http://stackoverflow.com/questions/734638/language-that-supports-serializing-coroutines
http://mikehadlow.blogspot.com/2011/04/serializing-continuations.html
http://tirania.org/blog/archive/2009/Apr-09.html

Piotr
May 17 2011