www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dmd-concurrency

reply "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
I sent a message to the dmd-concurrency mailing list about a 
month ago, that I'd like to contribute a set of channel-based 
functions to std.concurrency, but it was never picked up for 
inclusion nor (I don't believe) rejected. I'm not sure if I 
should make the same proposal in an alternate forum (like this 
one) or if someone can check for the message and post it? Or 
should I assume that the lack of inclusion was its own answer?

I sent the message in as a "random" rather than subscribing to 
the mailing list. Is it better for me to fully subscribe, if I 
want to contribute like this?

Similarly, is there any process for proposals for Phobos? I see 
that for the language, everything has to be a DIP first, but I 
didn't see a parallel for Phobos, hence posting to the forums.
Nov 19 2013
parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Chris Williams" <yoreanon-chrisw yahoo.co.jp> wrote in message 
news:kfmkxsgeeijwkndldrmj forum.dlang.org...

I sent a message to the dmd-concurrency mailing list about a month ago, 
that I'd like to contribute a set of channel-based functions to 
std.concurrency, but it was never picked up for inclusion nor (I don't 
believe) rejected. I'm not sure if I should make the same proposal in an 
alternate forum (like this one) or if someone can check for the message and 
post it? Or should I assume that the lack of inclusion was its own answer?


dmd-concurrency was created for the discussion of D2's new concurrency model 
to be described in TDPL.  It has not been used for years now.


 I sent the message in as a "random" rather than subscribing to the mailing 
 list. Is it better for me to fully subscribe, if I want to contribute like 
 this?

 Similarly, is there any process for proposals for Phobos? I see that for 
 the language, everything has to be a DIP first, but I didn't see a 
 parallel for Phobos, hence posting to the forums.



This is the correct forum to post phobos proposals on.
Nov 19 2013
parent reply "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Wednesday, 20 November 2013 at 04:24:14 UTC, Daniel Murphy 
wrote:

 This is the correct forum to post phobos proposals on.


Well then, here's what I had written:

A few applications I've considered implementing seem like they 
would be easier if there was a channel-based messaging system in 
std.concurrency. I'm happy to do this implementation, but I 
thought I would try to get some sort of sign-off before doing so. 
Following, I will lay out my argument for the addition, and then 
the API that I am considering.

---

One fairly common task is thread-pooling. With the standard 
send/receive model currently implemented, you have to choose a 
specific thread to target when you send a task. While it's true 
that you can simply iterate through your list of threads over and 
over, to spread the load evenly over them, that presumes that all 
tasks take even processing time. It makes more sense to be able 
to push data into a shared channel (secretly a work queue), and 
the first thread that finishes its previous task will be able to 
immediately pull the task before everyone else. This also means 
that the necessity of passing around references to your threads 
so that they can be looped over goes away.

I haven't tested it, but it looks like this sort of thing might 
be quasi-possible using the register/unregister/locate methods. 
As each thread starts, it can register itself with a named group 
(i.e. channel), and then anyone who wants to send an item to an 
arbitrary thread in that group can call locate() to retrieve one 
thread and call send() against the Tid. The target thread would 
then need to unregister itself while it is doing work, then 
re-register itself. My complaint against this is the need to 
unregister and re-register. If the thread issuing commands sends 
a large number of tasks all at once, they will all go to the same 
thread (if coded poorly) or the caller will need to use yield() 
or sleep() to allow the target thread to receive the task and 
unregister, so that locate() can find a different thread. That's 
not terribly efficient. I am also concerned that there's the 
chance that all threads will be unregistered when we call 
locate(), whereas a channeling system would be able to expand the 
mailbox during the times that all threads are busy.

The actual implementation within concurrency.d also concerns me 
as (if I read it correctly), the most recent item to register() 
will be the one which locate() finds, rather than the thread 
which has been registered the longest. While I suppose it's 
probably not too large of an issue if the same two threads keep 
taking all the tasks - that means that your load can't exceed two 
threads worth of processing power - it still seems like a LIFO 
system would be better. The registry is also based on an array 
rather than a set, which can make removal an O(n) operation, if 
the contents of the registry have to be shifted left, to fill an 
empty spot.

Overall, I think that adding a shared message box system would be 
a straightforward way to improve the handling of thread pooling 
via the actor model.

---

A less common use-case but I was also considering some 
world-simulators (e.g. for studying economics or building a game 
map) and here the ability to broadcast messages to a large set of 
other actors, based on location, interest, etc. seems useful. In 
this case, messages would need to be copied out to each 
subscriber in the channel rather than having an existence as a 
point to point connection. For a networked game, most likely you 
would want to break each channel into two, where locally all 
senders on a channel push to a single listener that pipes the 
messages over the network, and then remotely the messages would 
be broadcast to many listeners again, but that's a reasonably 
straightforward task for someone to implement on top of the 
channel functionality. I don't think that such functionality is 
needed in Phobos itself. Mostly, the presence of the broadcasting 
functionality in the standard library allows them to use the easy 
and safe actor model for more creative uses than a straight 
one-to-one pipe.

---

Overall, my hope would be to develop something that is 
conceptually no more difficult to deal with than the current 
send()/receive() model, but also able to be used in a wide 
variety of ways. The API that I would propose to develop is:

interface Channel {
	void send(T...)(T vals);
	void prioritySend(T...)(T vals);
	void receive(T...)(out Tid sender, T ops);
	receiveOnlyRet!(T) receiveOnly(T...)();
	bool receiveTimeout(T...)(Duration d, T ops);

	void setMaxMailboxSize(Tid tid, size_t messages, OnCrowding 
doThis);
	void setMaxMailboxSize(Tid tid, size_t messages, bool 
function(Tid) doThisFunc);
}

class SingleChannel : Channel {}	// Send inserts a message into a 
shared message box. Receive removes message

class DuplicateChannel(bool echo = true) : Channel {}	// Send 
inserts the message into a message box per-recipient. Receive 
removes message in the calling thread's channel message box. If 
echo is false, messages will not be sent back to the sender, even 
if they are a registered listener

void registerSend(Channel c, Tid tid = thisTid);	// used by 
function sendAll(). Channel can be of either type
void unregisterSend(Channel c, Tid tid = thisTid);
void registerReceive(Channel c, Tid tid = thisTid);	// used by 
function receiveAll(). Channel can be of either type
void unregisterReceive(Channel c, Tid tid = thisTid);

void sendAll(T...)(T ops); // Sends a copy of message to all 
channels this thread has registered for.
void receiveAll(T...)(out Channel c, out Tid sender, T ops); // 
Receives a message of type T from any channel that we are 
registered for. Returns channel and sender

I believe that the look and feel stays fairly consistent with the 
current set of functions in std.concurrency. I've added the 
ability for the recipient to infer information about the sender 
since, in the duplication model, I believe there are quite a few 
cases where this would be important information. And of course, 
I've added the option to register/unregister threads other than 
ourselves to allow a greater range of code layouts, though it's 
possible that the lack of this sort of thing in the original code 
is due to some sort of safety concern?

The most straightforward way to implement the DuplicateChannel 
would be to use the individual threads' message boxes, but this 
would mean that data put into a channel could be pulled out via 
the traditional receive() method. Currently, my intention would 
be to partition these two systems (the direct send()/receive() 
model and the channel model), unless anyone has any reason to 
think they should be merged into a single whole?

Those are my thoughts, anyways. Comments? Complaints?
Nov 19 2013
next sibling parent "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
Sticking with the concept that there needn't be any difference 
(from a programmatic view) between another thread and another 
computer, I've also considered adding innate support for 
channel-based RPC. I'm not sure if anyone would ever actually use 
such a thing for a local-threaded application, but I figure that 
creating higher levels of abstraction (like "everything is a 
file" or "everything is a range") has the possibility to open new 
paradigms.

Anyways, I'm not sure if I should just start coding and submit a 
pull request, or if someone needs to sign off on contributions 
first?
Nov 22 2013
prev sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-20 07:34:36 +0000, Chris Williams said:


 On Wednesday, 20 November 2013 at 04:24:14 UTC, Daniel Murphy wrote:

 This is the correct forum to post phobos proposals on.


 
 Well then, here's what I had written:
 
 A few applications I've considered implementing seem like they would be 
 easier if there was a channel-based messaging system in 
 std.concurrency. I'm happy to do this implementation, but I thought I 
 would try to get some sort of sign-off before doing so. Following, I 
 will lay out my argument for the addition, and then the API that I am 
 considering.
 
 ---
 
 One fairly common task is thread-pooling. With the standard 
 send/receive model currently implemented, you have to choose a specific 
 thread to target when you send a task. While it's true that you can 
 simply iterate through your list of threads over and over, to spread 
 the load evenly over them, that presumes that all tasks take even 
 processing time. It makes more sense to be able to push data into a 
 shared channel (secretly a work queue), and the first thread that 
 finishes its previous task will be able to immediately pull the task 
 before everyone else. This also means that the necessity of passing 
 around references to your threads so that they can be looped over goes 
 away.
 
 I haven't tested it, but it looks like this sort of thing might be 
 quasi-possible using the register/unregister/locate methods. As each 
 thread starts, it can register itself with a named group (i.e. 
 channel), and then anyone who wants to send an item to an arbitrary 
 thread in that group can call locate() to retrieve one thread and call 
 send() against the Tid. The target thread would then need to unregister 
 itself while it is doing work, then re-register itself. My complaint 
 against this is the need to unregister and re-register. If the thread 
 issuing commands sends a large number of tasks all at once, they will 
 all go to the same thread (if coded poorly) or the caller will need to 
 use yield() or sleep() to allow the target thread to receive the task 
 and unregister, so that locate() can find a different thread. That's 
 not terribly efficient. I am also concerned that there's the chance 
 that all threads will be unregistered when we call locate(), whereas a 
 channeling system would be able to expand the mailbox during the times 
 that all threads are busy.
 
 The actual implementation within concurrency.d also concerns me as (if 
 I read it correctly), the most recent item to register() will be the 
 one which locate() finds, rather than the thread which has been 
 registered the longest. While I suppose it's probably not too large of 
 an issue if the same two threads keep taking all the tasks - that means 
 that your load can't exceed two threads worth of processing power - it 
 still seems like a LIFO system would be better. The registry is also 
 based on an array rather than a set, which can make removal an O(n) 
 operation, if the contents of the registry have to be shifted left, to 
 fill an empty spot.
 
 Overall, I think that adding a shared message box system would be a 
 straightforward way to improve the handling of thread pooling via the 
 actor model.
 
 ---
 
 A less common use-case but I was also considering some world-simulators 
 (e.g. for studying economics or building a game map) and here the 
 ability to broadcast messages to a large set of other actors, based on 
 location, interest, etc. seems useful. In this case, messages would 
 need to be copied out to each subscriber in the channel rather than 
 having an existence as a point to point connection. For a networked 
 game, most likely you would want to break each channel into two, where 
 locally all senders on a channel push to a single listener that pipes 
 the messages over the network, and then remotely the messages would be 
 broadcast to many listeners again, but that's a reasonably 
 straightforward task for someone to implement on top of the channel 
 functionality. I don't think that such functionality is needed in 
 Phobos itself. Mostly, the presence of the broadcasting functionality 
 in the standard library allows them to use the easy and safe actor 
 model for more creative uses than a straight one-to-one pipe.
 
 ---
 
 Overall, my hope would be to develop something that is conceptually no 
 more difficult to deal with than the current send()/receive() model, 
 but also able to be used in a wide variety of ways. The API that I 
 would propose to develop is:
 
 interface Channel {
 	void send(T...)(T vals);
 	void prioritySend(T...)(T vals);
 	void receive(T...)(out Tid sender, T ops);
 	receiveOnlyRet!(T) receiveOnly(T...)();
 	bool receiveTimeout(T...)(Duration d, T ops);
 
 	void setMaxMailboxSize(Tid tid, size_t messages, OnCrowding doThis);
 	void setMaxMailboxSize(Tid tid, size_t messages, bool function(Tid) 
 doThisFunc);
 }
 
 class SingleChannel : Channel {}	// Send inserts a message into a 
 shared message box. Receive removes message
 
 class DuplicateChannel(bool echo = true) : Channel {}	// Send inserts 
 the message into a message box per-recipient. Receive removes message 
 in the calling thread's channel message box. If echo is false, messages 
 will not be sent back to the sender, even if they are a registered 
 listener
 
 void registerSend(Channel c, Tid tid = thisTid);	// used by function 
 sendAll(). Channel can be of either type
 void unregisterSend(Channel c, Tid tid = thisTid);
 void registerReceive(Channel c, Tid tid = thisTid);	// used by function 
 receiveAll(). Channel can be of either type
 void unregisterReceive(Channel c, Tid tid = thisTid);
 
 void sendAll(T...)(T ops); // Sends a copy of message to all channels 
 this thread has registered for.
 void receiveAll(T...)(out Channel c, out Tid sender, T ops); // 
 Receives a message of type T from any channel that we are registered 
 for. Returns channel and sender
 
 I believe that the look and feel stays fairly consistent with the 
 current set of functions in std.concurrency. I've added the ability for 
 the recipient to infer information about the sender since, in the 
 duplication model, I believe there are quite a few cases where this 
 would be important information. And of course, I've added the option to 
 register/unregister threads other than ourselves to allow a greater 
 range of code layouts, though it's possible that the lack of this sort 
 of thing in the original code is due to some sort of safety concern?
 
 The most straightforward way to implement the DuplicateChannel would be 
 to use the individual threads' message boxes, but this would mean that 
 data put into a channel could be pulled out via the traditional 
 receive() method. Currently, my intention would be to partition these 
 two systems (the direct send()/receive() model and the channel model), 
 unless anyone has any reason to think they should be merged into a 
 single whole?
 
 Those are my thoughts, anyways. Comments? Complaints?


How does one receive from multiple channels out-of-order?   I would 
rather this sent it to the subscribed Tid via send, rather than having 
an additional queue.   It could possible send a ChannelMessage which 
has a reference to the sending channel and the message.   I understand 
this is a different model than what Go and whatnot use, but I think 
it's more pratical in some circumstances.   Maybe both ways would be 
good?  I personally use this method in my vibe-d server.
Nov 22 2013
parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-22 22:34:19 +0000, Shammah Chancellor said:


 How does one receive from multiple channels out-of-order?   I would 
 rather this sent it to the subscribed Tid via send, rather than having 
 an additional queue.   It could possible send a ChannelMessage which 
 has a reference to the sending channel and the message.   I understand 
 this is a different model than what Go and whatnot use, but I think 
 it's more pratical in some circumstances.   Maybe both ways would be 
 good?  I personally use this method in my vibe-d server.


Edit, I see that you have receiveAll.. I didn't notice that.  However, 
that still doesn't satisfy the problem when you want to do receive() 
for your Tid, and receiveAll() from your channels.
Nov 22 2013
parent reply "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Friday, 22 November 2013 at 22:35:47 UTC, Shammah Chancellor 
wrote:

 On 2013-11-22 22:34:19 +0000, Shammah Chancellor said:
 
 Edit, I see that you have receiveAll.. I didn't notice that.  
 However, that still doesn't satisfy the problem when you want 
 to do receive() for your Tid, and receiveAll() from your 
 channels.


receiveAll() could pull from the thread's personal message box as 
well as all of its subscribed channels, so that it truly was a 
"receive ALL".

My thoughts for reasons to avoid that, however, are:

1. Threads always have their own message box allocated - even if 
it's empty - whereas with channels, at most you only have as many 
message boxes as you subscribed to. So my thinking was that if 
people are unlikely to mix channels-based systems and 
direct-sends in the same application, then every call to 
receiveAll() is having to spend an extra cycle checking the 
direct-send message box.

2. Since a Channel is just an interface, the implementation of 
which can vary, anyone who wanted to implement a 
NetworkedDuplicateChannel() class would be able to do so and pass 
it into a module that only includes std.concurency. This allows 
the actual implementation of any given channel to behave 
completely different from one another and quickly port code from 
one type to the other. send()/receive() are just innate to 
threads, however, and can't be replaced except by changing the 
import in each file to something else. Knowing that and also 
knowing that any direct-messaging system would probably be built 
like a channel (so that it had a constructor/init function where 
an IP and port could be configured, and perhaps an explicit 
startListening() method to call), I don't see anyone trying to 
override send()/receive() as their method for receiving direct 
messages over the network. They would still use the Channel 
interface, so there's not much value in trying to tie the two 
together.
Nov 22 2013
parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-22 23:14:44 +0000, Chris Williams said:


 On Friday, 22 November 2013 at 22:35:47 UTC, Shammah Chancellor wrote:

 On 2013-11-22 22:34:19 +0000, Shammah Chancellor said:
 
 Edit, I see that you have receiveAll.. I didn't notice that.  However, 
 that still doesn't satisfy the problem when you want to do receive() 
 for your Tid, and receiveAll() from your channels.


 
 receiveAll() could pull from the thread's personal message box as well 
 as all of its subscribed channels, so that it truly was a "receive ALL".
 
 My thoughts for reasons to avoid that, however, are:
 
 1. Threads always have their own message box allocated - even if it's 
 empty - whereas with channels, at most you only have as many message 
 boxes as you subscribed to. So my thinking was that if people are 
 unlikely to mix channels-based systems and direct-sends in the same 
 application, then every call to receiveAll() is having to spend an 
 extra cycle checking the direct-send message box.
 
 2. Since a Channel is just an interface, the implementation of which 
 can vary, anyone who wanted to implement a NetworkedDuplicateChannel() 
 class would be able to do so and pass it into a module that only 
 includes std.concurency. This allows the actual implementation of any 
 given channel to behave completely different from one another and 
 quickly port code from one type to the other. send()/receive() are just 
 innate to threads, however, and can't be replaced except by changing 
 the import in each file to something else. Knowing that and also 
 knowing that any direct-messaging system would probably be built like a 
 channel (so that it had a constructor/init function where an IP and 
 port could be configured, and perhaps an explicit startListening() 
 method to call), I don't see anyone trying to override send()/receive() 
 as their method for receiving direct messages over the network. They 
 would still use the Channel interface, so there's not much value in 
 trying to tie the two together.


I am suggesting you define a particular type of message to be received, 
and then send that to the Thread/Fiber's MessageQueue.   Then the 
Channel is just an interface to broadcast messages to all the 
subscribers of a particular channel.  Then each thread need only poll 
one queue.
Nov 22 2013
parent reply "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Friday, 22 November 2013 at 23:21:33 UTC, Shammah Chancellor 
wrote:

 On 2013-11-22 23:14:44 +0000, Chris Williams said:

 I am suggesting you define a particular type of message to be 
 received, and then send that to the Thread/Fiber's 
 MessageQueue.   Then the Channel is just an interface to 
 broadcast messages to all the subscribers of a particular 
 channel.  Then each thread need only poll one queue.


With a SingleChannel, that's not really an option. Theoretically, 
it could choose a random subscriber with space in its MessageBox 
as the next recipient, but there would be no guarantee that a 
thread which had just decided to shut itself down hadn't been 
selected, in which case the message would be lost. The channel 
really needs to have an internal queue that all the threads look 
at when they call receive.

For DuplicateChannel, I could indeed go and copy the data into 
each individual thread's box, so that it only had to check there 
for messages during receive (if not subscribed to any 
SingleChannel instances), but I would still need to have the 
logic in place to scan for in-channel boxes because of 
SingleChannel, in which case I might as well establish the 
presence of boxes in channels as the norm for DuplicateChannel as 
well.

This does give us the advantage that if the user requests calls 
against a single channel, there is no extra overhead associated 
with that act, because it has to scan through a large list to 
find the appropriate items. It also allows the user to customize 
the behavior of the MessageBox on a channel-by-channel basis. 
Setting a "max number of messages" per channel, when all messages 
go into the same bucket, would end up wonky.
Nov 22 2013
parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-23 01:46:22 +0000, Chris Williams said:


 On Friday, 22 November 2013 at 23:21:33 UTC, Shammah Chancellor wrote:

 On 2013-11-22 23:14:44 +0000, Chris Williams said:
 
 I am suggesting you define a particular type of message to be received, 
 and then send that to the Thread/Fiber's MessageQueue.   Then the 
 Channel is just an interface to broadcast messages to all the 
 subscribers of a particular channel.  Then each thread need only poll 
 one queue.


 
 With a SingleChannel, that's not really an option. Theoretically, it 
 could choose a random subscriber with space in its MessageBox as the 
 next recipient, but there would be no guarantee that a thread which had 
 just decided to shut itself down hadn't been selected, in which case 
 the message would be lost. The channel really needs to have an internal 
 queue that all the threads look at when they call receive.
 
 For DuplicateChannel, I could indeed go and copy the data into each 
 individual thread's box, so that it only had to check there for 
 messages during receive (if not subscribed to any SingleChannel 
 instances), but I would still need to have the logic in place to scan 
 for in-channel boxes because of SingleChannel, in which case I might as 
 well establish the presence of boxes in channels as the norm for 
 DuplicateChannel as well.
 
 This does give us the advantage that if the user requests calls against 
 a single channel, there is no extra overhead associated with that act, 
 because it has to scan through a large list to find the appropriate 
 items. It also allows the user to customize the behavior of the 
 MessageBox on a channel-by-channel basis. Setting a "max number of 
 messages" per channel, when all messages go into the same bucket, would 
 end up wonky.


In my uses of channels, I have not found customizing the message box 
size per channel to be useful.  It may be, but it's not something I 
want.    I sitll think duplicate channels should behave the way I 
described.   Take IRC for example, where I am sending messages to other 
users, but also to channels which then broadcast.   I want my clients 
to be able to simply receive() and get messages that were intended for 
them specifically, or were broadcast.  I don't want to implement 
complex logic in order to avoid my thread hanging while trying to read 
from a channel if I have messages available that are specific to my 
thread.

With regards to SingleChannel,  picking a random thread would be bad 
for a plethora of reasons, so I agree here.  I think we should continue 
to disucss this issue.  There may be some way to get Tid.receive() to 
behave the expected way when subscribed to SingleChannels.

Also, SingleChannels seem somewhat strange in general to me though -- 
What is the expected behavior when multiple threads receive different 
types of mesages from the MessageBox?  Does it consume messages it 
doesn't understand until it finds one it does?  This would prevent 
other tasks which do understand them from processing.   What is the use 
case for SingleChannel that a simple synchronized Queue does not 
acheive?

It seems you want SingleChannel and DuplicateChannel not block each 
other for the same reason I don't want DuplicateChannels to block 
Task-specific receive()'s.   However, I think SingleChannels are the 
oddity here, and should be treated as such, rather than Tid.send()

-Shammah
Nov 22 2013
parent reply "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Saturday, 23 November 2013 at 04:18:47 UTC, Shammah Chancellor 
wrote:

 In my uses of channels, I have not found customizing the 
 message box size per channel to be useful.  It may be, but it's 
 not something I want.


I agree that I have not found box size configuration to be a 
significant gain. Being able to configure crowding behavior is 
probably more useful, but there's currently nothing about those 
that would require more than one message box per thread. (And I 
could envision adding a "force in" on-crowding behavior that pops 
from the head and inserts into the tail, but I think I could 
implement that in a reasonable method, even with a shared message 
box.)


 I sitll think duplicate channels should behave the way I 
 described.   Take IRC for example, where I am sending messages 
 to other users, but also to channels which then broadcast.   I 
 want my clients to be able to simply receive() and get messages 
 that were intended for them specifically, or were broadcast.  I 
 don't want to implement complex logic in order to avoid my 
 thread hanging while trying to read from a channel if I have 
 messages available that are specific to my thread.


I'm actually pretty easy on this topic. This was the one reason I 
thought of to try and merge the two systems, but shied away for 
the previously stated reasons.


 With regards to SingleChannel,  picking a random thread would 
 be bad for a plethora of reasons, so I agree here.  I think we 
 should continue to disucss this issue.  There may be some way 
 to get Tid.receive() to behave the expected way when subscribed 
 to SingleChannels.


What I would probably do, by removing receiveAll() and instead 
making receive() grab everything, is expose some methods that 
allow one to either insert directly into a thread's message box 
from a channel or register the channel as something which 
contains its own MessageBox that needs to be checked during a 
receive().

This way, a call to receive() will by default just check the 
thread's MessageBox, and only in special cases attempt to do 
something more extravagant. The overhead for a simple check like 
this is reasonably low that I wouldn't be concerned about adding 
it.


 Also, SingleChannels seem somewhat strange in general to me 
 though -- What is the expected behavior when multiple threads 
 receive different types of mesages from the MessageBox?  Does 
 it consume messages it doesn't understand until it finds one it 
 does?  This would prevent other tasks which do understand them 
 from processing.   What is the use case for SingleChannel that 
 a simple synchronized Queue does not acheive?


I'd say that the main difference between using channels or using 
a synchronized queue are:

1. Developer preference. If you like the actor model, everything 
looks like a nail.

2. Abstraction. My SingleChannel could, for example, be sitting 
on top of something like Amazon's Simple Queue Service (a data 
queue that's maintained on one of their servers, which can have 
items removed by any number of clients). I could set up a thread 
that polls the SQS, pulls an item off, and adds it to a 
synchronized queue, but I might want to write a library that 
translates a request to receive() as a synchronous request, from 
that thread, to the SQS server. With the queue model, the library 
would always have to have its own polling thread. With the 
channel model, either implementation is possible.

3. Multiple entry points. receive() is currently able to directly 
jump to a handler based on the message type. With a queue, I have 
to make everything into a generic Object (or Variant), then 
perform type checks to decide what to do. std.concurrency already 
does all of that, so it's a waste to have to rewrite the same 
sort of handling.

4. Multiple inputs. With the ability to register to multiple data 
sources, using a queue, I have to figure out my own logic for how 
to cycle through my list of sources, so that they're getting 
checked evenly. With the channel system, that is all contained 
for me. I can call register() on 30 different pipes, and not need 
to have to know how the data is being locked, scanned for types I 
support, nor unpackaged for use. It's all just magical, upon 
calling receive().
Nov 25 2013
parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-25 21:48:09 +0000, Chris Williams said:


 On Saturday, 23 November 2013 at 04:18:47 UTC, Shammah Chancellor wrote:

 In my uses of channels, I have not found customizing the message box 
 size per channel to be useful.  It may be, but it's not something I 
 want.


 
 I agree that I have not found box size configuration to be a 
 significant gain. Being able to configure crowding behavior is probably 
 more useful, but there's currently nothing about those that would 
 require more than one message box per thread. (And I could envision 
 adding a "force in" on-crowding behavior that pops from the head and 
 inserts into the tail, but I think I could implement that in a 
 reasonable method, even with a shared message box.)
 

 I sitll think duplicate channels should behave the way I described.   
 Take IRC for example, where I am sending messages to other users, but 
 also to channels which then broadcast.   I want my clients to be able 
 to simply receive() and get messages that were intended for them 
 specifically, or were broadcast.  I don't want to implement complex 
 logic in order to avoid my thread hanging while trying to read from a 
 channel if I have messages available that are specific to my thread.


 
 I'm actually pretty easy on this topic. This was the one reason I 
 thought of to try and merge the two systems, but shied away for the 
 previously stated reasons.
 

 With regards to SingleChannel,  picking a random thread would be bad 
 for a plethora of reasons, so I agree here.  I think we should continue 
 to disucss this issue.  There may be some way to get Tid.receive() to 
 behave the expected way when subscribed to SingleChannels.


 
 What I would probably do, by removing receiveAll() and instead making 
 receive() grab everything, is expose some methods that allow one to 
 either insert directly into a thread's message box from a channel or 
 register the channel as something which contains its own MessageBox 
 that needs to be checked during a receive().
 
 This way, a call to receive() will by default just check the thread's 
 MessageBox, and only in special cases attempt to do something more 
 extravagant. The overhead for a simple check like this is reasonably 
 low that I wouldn't be concerned about adding it.
 

 Also, SingleChannels seem somewhat strange in general to me though -- 
 What is the expected behavior when multiple threads receive different 
 types of mesages from the MessageBox?  Does it consume messages it 
 doesn't understand until it finds one it does?  This would prevent 
 other tasks which do understand them from processing.   What is the use 
 case for SingleChannel that a simple synchronized Queue does not 
 acheive?


 
 I'd say that the main difference between using channels or using a 
 synchronized queue are:
 
 1. Developer preference. If you like the actor model, everything looks 
 like a nail.
 
 2. Abstraction. My SingleChannel could, for example, be sitting on top 
 of something like Amazon's Simple Queue Service (a data queue that's 
 maintained on one of their servers, which can have items removed by any 
 number of clients). I could set up a thread that polls the SQS, pulls 
 an item off, and adds it to a synchronized queue, but I might want to 
 write a library that translates a request to receive() as a synchronous 
 request, from that thread, to the SQS server. With the queue model, the 
 library would always have to have its own polling thread. With the 
 channel model, either implementation is possible.
 
 3. Multiple entry points. receive() is currently able to directly jump 
 to a handler based on the message type. With a queue, I have to make 
 everything into a generic Object (or Variant), then perform type checks 
 to decide what to do. std.concurrency already does all of that, so it's 
 a waste to have to rewrite the same sort of handling.
 
 4. Multiple inputs. With the ability to register to multiple data 
 sources, using a queue, I have to figure out my own logic for how to 
 cycle through my list of sources, so that they're getting checked 
 evenly. With the channel system, that is all contained for me. I can 
 call register() on 30 different pipes, and not need to have to know how 
 the data is being locked, scanned for types I support, nor unpackaged 
 for use. It's all just magical, upon calling receive().



What about a compromise?   Extending MessageQueue to accept subscribe 
to a SingleChannel/etc would be useful.  I think though, that 
DuplicateChannels should just poke the Tid's mbox though.

Thank you for working on this.   I think it's a great initiative.    
Also, can you get sending immutable messages working while you're at 
it? *chortle*

-Shammah
Nov 26 2013
parent reply "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Tuesday, 26 November 2013 at 12:02:18 UTC, Shammah Chancellor 
wrote:

 Thank you for working on this.   I think it's a great 
 initiative.    Also, can you get sending immutable messages 
 working while you're at it? *chortle*


This?

http://d.puremagic.com/issues/show_bug.cgi?id=4566
Nov 26 2013
parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-26 20:01:58 +0000, Chris Williams said:


 On Tuesday, 26 November 2013 at 12:02:18 UTC, Shammah Chancellor wrote:

 Thank you for working on this.   I think it's a great initiative.    
 Also, can you get sending immutable messages working while you're at 
 it? *chortle*


 
 This?
 
 http://d.puremagic.com/issues/show_bug.cgi?id=4566


Nope:

http://d.puremagic.com/issues/show_bug.cgi?id=5538
Nov 27 2013
parent "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Wednesday, 27 November 2013 at 17:42:47 UTC, Shammah 
Chancellor wrote:

 On 2013-11-26 20:01:58 +0000, Chris Williams said:


 On Tuesday, 26 November 2013 at 12:02:18 UTC, Shammah 
 Chancellor wrote:

 Thank you for working on this.   I think it's a great 
 initiative.    Also, can you get sending immutable messages 
 working while you're at it? *chortle*


 
 This?
 
 http://d.puremagic.com/issues/show_bug.cgi?id=4566


 Nope:

 http://d.puremagic.com/issues/show_bug.cgi?id=5538


Maybe I'll set that as my first task, to familiarize myself with 
all of the templating and variant nonsense that's being used in 
the module, before adding anything new.

I'll try this weekend and see where I get.
Nov 27 2013