www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Fiber and Thread Communication

reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
Are there any plans to unite

fiber-to-fiber communication

with

thread-to-thread communication

in Phobos?

Does vibe.d give any solutions here?
Apr 08 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
On Friday, 8 April 2016 at 10:51:49 UTC, Nordlöw wrote:

 Are there any plans to unite

 fiber-to-fiber communication

 with

 thread-to-thread communication

 in Phobos?

 Does vibe.d give any solutions here?


Doesn't std.concurrency support both right now? I remember seeing 
PR that adds message box support to fibers ages ago.
Apr 08 2016
next sibling parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 8 April 2016 at 11:01:21 UTC, Dicebot wrote:

 Doesn't std.concurrency support both right now? I remember 
 seeing PR that adds message box support to fibers ages ago.


1. What functions provide message box communication?

2. But Fibers cannot currently be moved between threads right?
Apr 08 2016
parent reply Dicebot <public dicebot.lv> writes:
On Friday, 8 April 2016 at 11:18:11 UTC, Nordlöw wrote:

 On Friday, 8 April 2016 at 11:01:21 UTC, Dicebot wrote:

 Doesn't std.concurrency support both right now? I remember 
 seeing PR that adds message box support to fibers ages ago.




See https://issues.dlang.org/show_bug.cgi?id=12090 and 
https://github.com/D-Programming-Language/phobos/pull/1910 for 
relevant code (as you can see it was merged several releases ago)


 1. What functions provide message box communication?


The same ones as thread ones. API is completely transparent.


 2. But Fibers cannot currently be moved between threads right?


Yes, and this is by design. It harms performance of concurrent 
apps.
Apr 08 2016
parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 8 April 2016 at 13:15:07 UTC, Dicebot wrote:

 On Friday, 8 April 2016 at 11:18:11 UTC, Nordlöw wrote:

 On Friday, 8 April 2016 at 11:01:21 UTC, Dicebot wrote:

 Doesn't std.concurrency support both right now? I remember 
 seeing PR that adds message box support to fibers ages ago.




 See https://issues.dlang.org/show_bug.cgi?id=12090 and 
 https://github.com/D-Programming-Language/phobos/pull/1910 for 
 relevant code (as you can see it was merged several releases 
 ago)


 1. What functions provide message box communication?


 The same ones as thread ones. API is completely transparent.


 2. But Fibers cannot currently be moved between threads right?


 Yes, and this is by design. It harms performance of concurrent 
 apps.


So a TId can represent either a thread or a fiber?
Apr 08 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
On Friday, 8 April 2016 at 14:08:39 UTC, Nordlöw wrote:

 So a TId can represent either a thread or a fiber?


AFAIR, yes (I haven't used std.concurrency in a long while, 
telling all from memory only).
Apr 08 2016
parent reply tcak <1ltkrs+3wyh1ow7kzn1k sharklasers.com> writes:
On Friday, 8 April 2016 at 15:33:46 UTC, Dicebot wrote:

 On Friday, 8 April 2016 at 14:08:39 UTC, Nordlöw wrote:

 So a TId can represent either a thread or a fiber?


 AFAIR, yes (I haven't used std.concurrency in a long while, 
 telling all from memory only).


yes what? Thread or Fiber.

---

Anyway. Since, Fiber is not like a thread, and when a thread 
starts a Fiber, it is like calling a normal function, I guess TId 
represents the thread still.
Apr 08 2016
parent reply Dicebot <public dicebot.lv> writes:
On Friday, 8 April 2016 at 19:46:17 UTC, tcak wrote:

 On Friday, 8 April 2016 at 15:33:46 UTC, Dicebot wrote:

 On Friday, 8 April 2016 at 14:08:39 UTC, Nordlöw wrote:

 So a TId can represent either a thread or a fiber?


 AFAIR, yes (I haven't used std.concurrency in a long while, 
 telling all from memory only).


 yes what? Thread or Fiber.


Yes both :) Tid represent abstract execution context, with no 
implications about underlying executor.
Apr 08 2016
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 04/08/2016 01:16 PM, Dicebot wrote:

 On Friday, 8 April 2016 at 19:46:17 UTC, tcak wrote:

 On Friday, 8 April 2016 at 15:33:46 UTC, Dicebot wrote:

 On Friday, 8 April 2016 at 14:08:39 UTC, Nordlöw wrote:

 So a TId can represent either a thread or a fiber?


 AFAIR, yes (I haven't used std.concurrency in a long while, telling
 all from memory only).


 yes what? Thread or Fiber.


 Yes both :) Tid represent abstract execution context, with no
 implications about underlying executor.


Thanks Dicebot. I don't think the included 
std.concurrency.FiberScheduler has support for message passing because 
FiberScheduler.spawn does not return a Tid. If so, I don't see how it's 
possible to send messages between fibers.

Ali
Apr 08 2016
parent reply Dicebot <public dicebot.lv> writes:
On Friday, 8 April 2016 at 20:25:11 UTC, Ali Çehreli wrote:

 On 04/08/2016 01:16 PM, Dicebot wrote:

 On Friday, 8 April 2016 at 19:46:17 UTC, tcak wrote:

 On Friday, 8 April 2016 at 15:33:46 UTC, Dicebot wrote:

 On Friday, 8 April 2016 at 14:08:39 UTC, Nordlöw wrote:

 So a TId can represent either a thread or a fiber?


 AFAIR, yes (I haven't used std.concurrency in a long while, 
 telling
 all from memory only).


 yes what? Thread or Fiber.


 Yes both :) Tid represent abstract execution context, with no
 implications about underlying executor.


 Thanks Dicebot. I don't think the included 
 std.concurrency.FiberScheduler has support for message passing 
 because FiberScheduler.spawn does not return a Tid. If so, I 
 don't see how it's possible to send messages between fibers.

 Ali


Looks like a (funny) oversight. Note that you get it for get 
fiber via 
https://github.com/D-Programming-Language/phobos/blob/master/std
concurrency.d#L1337 (and FiberScheduler specifically extends Fiber to add
ThreadInfo to it) but there is no clear way to pass that info to spawn host. I
have a feeling that if that code is patched to simply provide Tid, message
passing will just magically work. Needs to be checked though.
Apr 08 2016
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 04/08/2016 02:42 PM, Dicebot wrote:


 Thanks Dicebot. I don't think the included
 std.concurrency.FiberScheduler has support for message passing because
 FiberScheduler.spawn does not return a Tid. If so, I don't see how
 it's possible to send messages between fibers.

 Ali


 Looks like a (funny) oversight.


Sorry, I misled you. :)


 Note that you get it for get fiber via
 


https://github.com/D-Programming-Language/phobos/blob/master/std
concurrency.d#L1337 


 (and FiberScheduler specifically extends Fiber to add ThreadInfo to it)
 but there is no clear way to pass that info to spawn host. I have a
 feeling that if that code is patched to simply provide Tid, message
 passing will just magically work. Needs to be checked though.


It turns out, instead of calling scheduler.spawn() directly, the program 
sets the __gshared 'scheduler' variable first and then calls spawn() as 
usual, which does return that fiber's Tid:

import std.stdio;
import std.concurrency;
import std.range;
import std.algorithm;

struct Done {
}

void workerTask(int id) {
     writefln("workerTask %s started", id);

     bool done = false;
     while (!done) {
         receive(
             (int message) {
                 writefln("workerTask %s received %s", id, message);
                 ownerTid.send(message * id);
             },
             (Done message) {
                 writefln("workerTask %s received Done", id);
                 done = true;
             });

         // Seems not to be needed:
         // scheduler.yield();
     }

     writefln("workerTask %s exiting", id);
}

void mainTask() {
     enum workerCount = 5;
     enum loopCount = 3;

     writeln("mainTask started");

     auto workers = iota(workerCount)
                    .map!(id => spawn(&workerTask, id))
                    .array;

     foreach (i; 0 .. loopCount) {
         foreach (id, worker; workers) {
             worker.send(i);
             auto response = receiveOnly!int();
             assert(response == i * id);
             writefln("mainTask received %s", response);
         }
     }

     writeln("mainTask sending Done messages");

     foreach (worker; workers) {
         worker.send(Done());
     }

     writeln("mainTask exiting");
}

void main() {
     scheduler = new FiberScheduler;
     scheduler.start({
         mainTask();
     });
}

Ali
Apr 09 2016
prev sibling parent Alex Parrill <initrd.gz gmail.com> writes:
On Friday, 8 April 2016 at 14:08:39 UTC, Nordlöw wrote:

 So a TId can represent either a thread or a fiber?


It represents a "logical thread", which currently consists of 
coroutines or OS threads but could theoretically be extended to, 
say, other processes or even other machines.
Apr 08 2016
prev sibling parent =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 8 April 2016 at 11:01:21 UTC, Dicebot wrote:

 Doesn't std.concurrency support both right now? I remember 
 seeing PR that adds message box support to fibers ages ago.


What progress has been since post:

http://forum.dlang.org/post/k4jsef$26h6$1 digitalmars.com
Apr 08 2016
prev sibling parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 8 April 2016 at 10:51:49 UTC, Nordlöw wrote:

 Are there any plans to unite


AFAICT, it is not clear what are the limitations of the current 
std.concurrency and, from what. An illustrating example on 
task-based parallellism (such as the ones in jin.go) should 
partly alleviate this problem. Any ideas on what such an example 
should contain and illustrate. Current limitations and plans on 
fixing should be described aswell.

References:

http://forum.dlang.org/post/mailman.776.1459177268.26339.digitalmars-d puremagic.com
http://code.dlang.org/packages/jin-go
https://github.com/nin-jin/go.d

Further, who's up for the job of add the missing parts in 
std.concurrency using ideas and code from 
https://github.com/nin-jin/go.d ? :)
Apr 09 2016
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 04/09/2016 07:45 AM, Nordlöw wrote:

 On Friday, 8 April 2016 at 10:51:49 UTC, Nordlöw wrote:



 AFAICT, it is not clear what are the limitations of the current
 std.concurrency and, from what. An illustrating example on task-based
 parallellism (such as the ones in jin.go) should partly alleviate this
 problem.


I don't know how much this helps but I was able to write an example that 
seems to work (elsewhere in this thread):

   http://forum.dlang.org/post/nec62k$26v7$1 digitalmars.com

Ali
Apr 09 2016