digitalmars.D - Channels for tasks?
- Matt (14/14) Jul 18 2013 Hi guys,
- Jesse Phillips (12/17) Jul 18 2013 I don't think this is really an answer, but awhile back I was
- Matt (14/25) Jul 18 2013 Thanks for the examples, I'll have to take a look at them later!
- Sean Kelly (10/14) Jul 18 2013 thing exists for threads, but not for the lightweight tasks provided by ...
- Matt (8/13) Jul 18 2013 From what I've read about Fibers, it seems they are more similar
- Mr. Anonymous (3/17) Jul 18 2013 vibe.d is based on fibers, you might want to check it out.
- Sean Kelly (18/24) Jul 18 2013 queue in std.concurrency, but that means making TLS work at the fiber =
- Jacob Carlborg (7/8) Jul 18 2013 To be able to support dynamic libraries and TLS on Mac OS X we will most...
- Sean Kelly (9/14) Jul 22 2013 difficult to use this logic for fibers, but things get tricky once you =
- deadalnix (7/20) Jul 19 2013 My take is that we should keep fibers as they are. But change
- Sean Kelly (15/19) Jul 22 2013 execute within the context of the calling thread. Most languages that =
- deadalnix (3/5) Jul 22 2013 If so, other libraries won't be able to yield (for instance
- Sean Kelly (5/9) Jul 23 2013 performing IO).
- deadalnix (3/18) Jul 19 2013 Yes please yes.
- Dejan Lekic (2/23) Jul 23 2013 +1
Hi guys, I know this will have probably been answered before, but does D have a concept similar to Go's channels when it comes to tasks? I believe (according to a few forum posts hinting at it) that such a thing exists for threads, but not for the lightweight tasks provided by std.parallelism. Are there any plans to add such a feature to the library? If not, how would I go about implementing them in D? I imagine I would use some form of blocking list, but I've not done much research on how they are done in Go/Rust/etc. Thanks, Matt. P.S. Sorry if this is in the wrong sub-forum, I'll be happy for mods to move it if necessary.
Jul 18 2013
On Thursday, 18 July 2013 at 15:29:00 UTC, Matt wrote:Hi guys, I know this will have probably been answered before, but does D have a concept similar to Go's channels when it comes to tasks?I don't think this is really an answer, but awhile back I was looking into the difference of D and Go for parallel/concurrency and created these two examples[1][2] from the blog http://www.mprescient.com/journal/2011/1/9/concurrency-in-go-a-call-center-tutorial.html 1. https://gist.github.com/JesseKPhillips/773979 2. https://gist.github.com/JesseKPhillips/774983 The first is a translation to message passing, the second is using std.parallelism I think there is room for higher level APIs, but I don't think we will see anything that resembles Goroutines being added.P.S. Sorry if this is in the wrong sub-forum, I'll be happy for mods to move it if necessary.Should go to D.learn, but not a huge deal.
Jul 18 2013
"Matt" <MATTCA sky.com> On Thursday, 18 July 2013 at 16:41:34 UTC, Jesse Phillips wrote:I don't think this is really an answer, but awhile back I was looking into the difference of D and Go for parallel/concurrency and created these two examples[1][2] from the blog http://www.mprescient.com/journal/2011/1/9/concurrency-in-go-a-call-center-tutorial.html 1. https://gist.github.com/JesseKPhillips/773979 2. https://gist.github.com/JesseKPhillips/774983 The first is a translation to message passing, the second is using std.parallelism I think there is room for higher level APIs, but I don't think we will see anything that resembles Goroutines being added.Thanks for the examples, I'll have to take a look at them later! I guess the different backgrounds is the reason for a lack of D 'goroutines', but parallelism seemed a good alternative. Is the performance of parallelism and it's taskpools similar to goroutines? It seems they both do the same thing (adding the tasks to a queue, allocating them to a group of threads, etc), but does the level of implementation (library vs language) mean a speed difference? Furthermore, is there a limit to the number of tasks that can be spawned and performed in parallel? I know Rust allows for hundreds, even thousands, of tasks to be executed "concurrently." Thanks for the replies, Matt.
Jul 18 2013
On Jul 18, 2013, at 8:28 AM, Matt <MATTCA sky.com> wrote:Hi guys, I know this will have probably been answered before, but does D have a =concept similar to Go's channels when it comes to tasks?=20 I believe (according to a few forum posts hinting at it) that such a =thing exists for threads, but not for the lightweight tasks provided by = std.parallelism. Are there any plans to add such a feature to the = library? Not as such. I'd like to make Fibers each have their own message queue = in std.concurrency, but that means making TLS work at the fiber level, = which is tricky. I think there is value in the CSP model (ie. channels), but haven't done = anything about it in terms of library work.=
Jul 18 2013
On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:Not as such. I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky. I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.From what I've read about Fibers, it seems they are more similar to what I'm looking for (a post on stackoverflow implied they are similar to Erlang's implementation of lightweight tasks). Does this mean to say they are similar to the tasks provided by std.parallelism? Thanks for the reply, Matt.
Jul 18 2013
On Thursday, 18 July 2013 at 18:38:57 UTC, Matt wrote:On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:vibe.d is based on fibers, you might want to check it out. http://vibed.org/features#fibersNot as such. I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky. I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.From what I've read about Fibers, it seems they are more similar to what I'm looking for (a post on stackoverflow implied they are similar to Erlang's implementation of lightweight tasks). Does this mean to say they are similar to the tasks provided by std.parallelism? Thanks for the reply, Matt.
Jul 18 2013
On Jul 18, 2013, at 11:38 AM, "Matt" <MATTCA sky.com> wrote:On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:queue in std.concurrency, but that means making TLS work at the fiber = level, which is tricky.Not as such. I'd like to make Fibers each have their own message =done anything about it in terms of library work.=20 I think there is value in the CSP model (ie. channels), but haven't ==20 =46rom what I've read about Fibers, it seems they are more similar to =what I'm looking for (a post on stackoverflow implied they are similar = to Erlang's implementation of lightweight tasks). Does this mean to say = they are similar to the tasks provided by std.parallelism? Functionally, fibers are coroutines. They have their own stack and = execute within the context of the calling thread. Most languages that = scale to thousands or millions of concurrent threads/processes use = fibers (often called "green threads" in this context) to pull it off. = The obstacle for D is that global data is thread-local by default, so if = we were to use fibers in the back-end to allow the creation of a large = number of "threads", fibers would basically need their own TLS to avoid = breaking code. We already do in-library TLS for OSX and so it shouldn't = be terribly difficult to use this logic for fibers, but things get = tricky once you consider dynamic libraries. It really should happen at = some point though, for std.concurrency to operate at its full potential.=
Jul 18 2013
On 2013-07-18 21:08, Sean Kelly wrote:We already do in-library TLS for OSX and so it shouldn't be terribly difficult to use this logic for fibers, but things get tricky once you consider dynamic libraries.To be able to support dynamic libraries and TLS on Mac OS X we will most likely need to switch to the native implementation of TLS, supported in Mac OS X 10.7 and later. Or we would have to copy/mimic the TLS related code from the dynamic loader into druntime. -- /Jacob Carlborg
Jul 18 2013
On Jul 18, 2013, at 11:35 PM, Jacob Carlborg <doob me.com> wrote:On 2013-07-18 21:08, Sean Kelly wrote: =20difficult to use this logic for fibers, but things get tricky once you = consider dynamic libraries.We already do in-library TLS for OSX and so it shouldn't be terribly ==20 To be able to support dynamic libraries and TLS on Mac OS X we will =most likely need to switch to the native implementation of TLS, = supported in Mac OS X 10.7 and later. Or we would have to copy/mimic the = TLS related code from the dynamic loader into druntime. And the same for other OSes as well, if we started doing manual TLS for = fibers. Super not fun. For reference, here's how TLS works on ELF: http://www.akkadia.org/drepper/tls.pdf=
Jul 22 2013
On Thursday, 18 July 2013 at 19:08:25 UTC, Sean Kelly wrote:Functionally, fibers are coroutines. They have their own stack and execute within the context of the calling thread. Most languages that scale to thousands or millions of concurrent threads/processes use fibers (often called "green threads" in this context) to pull it off. The obstacle for D is that global data is thread-local by default, so if we were to use fibers in the back-end to allow the creation of a large number of "threads", fibers would basically need their own TLS to avoid breaking code. We already do in-library TLS for OSX and so it shouldn't be terribly difficult to use this logic for fibers, but things get tricky once you consider dynamic libraries. It really should happen at some point though, for std.concurrency to operate at its full potential.My take is that we should keep fibers as they are. But change Thread (as the class in the runtime) to be Fibers with FLS and then let the runtime operate system thread and schedule Thread on them. The read system thread can still be provided in some dark corner of the standard lib or runtime.
Jul 19 2013
On Jul 19, 2013, at 12:48 AM, deadalnix <deadalnix gmail.com> wrote:On Thursday, 18 July 2013 at 19:08:25 UTC, Sean Kelly wrote:execute within the context of the calling thread. Most languages that = scale to thousands or millions of concurrent threads/processes use = fibers (often called "green threads" in this context) to pull it off. = The obstacle for D is that global data is thread-local by default, so if = we were to use fibers in the back-end to allow the creation of a large = number of "threads", fibers would basically need their own TLS to avoid = breaking code. We already do in-library TLS for OSX and so it shouldn't = be terribly difficult to use this logic for fibers, but things get = tricky once you consider dynamic libraries. It really should happen at = some point though, for std.concurrency to operate at its full potential.Functionally, fibers are coroutines. They have their own stack and ==20 My take is that we should keep fibers as they are. But change Thread =(as the class in the runtime) to be Fibers with FLS and then let the = runtime operate system thread and schedule Thread on them. I think this would most likely happen within std.concurrency, with the = context switch occurring on send/receive.=
Jul 22 2013
On Monday, 22 July 2013 at 18:24:53 UTC, Sean Kelly wrote:I think this would most likely happen within std.concurrency, with the context switch occurring on send/receive.If so, other libraries won't be able to yield (for instance libraries performing IO).
Jul 22 2013
On Jul 22, 2013, at 11:20 PM, deadalnix <deadalnix gmail.com> wrote:On Monday, 22 July 2013 at 18:24:53 UTC, Sean Kelly wrote:the context switch occurring on send/receive.I think this would most likely happen within std.concurrency, with ==20 If so, other libraries won't be able to yield (for instance libraries =performing IO). Yes, there would have to be a yield method available for library = writers.
Jul 23 2013
On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:On Jul 18, 2013, at 8:28 AM, Matt <MATTCA sky.com> wrote:Yes please yes. Did I said yes yet ?Hi guys, I know this will have probably been answered before, but does D have a concept similar to Go's channels when it comes to tasks? I believe (according to a few forum posts hinting at it) that such a thing exists for threads, but not for the lightweight tasks provided by std.parallelism. Are there any plans to add such a feature to the library?Not as such. I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.
Jul 19 2013
On Friday, 19 July 2013 at 07:42:26 UTC, deadalnix wrote:On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:+1On Jul 18, 2013, at 8:28 AM, Matt <MATTCA sky.com> wrote:Yes please yes. Did I said yes yet ?Hi guys, I know this will have probably been answered before, but does D have a concept similar to Go's channels when it comes to tasks? I believe (according to a few forum posts hinting at it) that such a thing exists for threads, but not for the lightweight tasks provided by std.parallelism. Are there any plans to add such a feature to the library?Not as such. I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.I think there is value in the CSP model (ie. channels), but haven't done anything about it in terms of library work.
Jul 23 2013