www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Concurrency architecture for D2

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I think we are now in the position of defining a solid set of 
concurrency primitives for D. This follows many months of mulling over 
models and options.

It would be great to open the participation to the design as broadly as 
possible, but I think it's realistic to say we won't be able to get 
things done on the newsgroup. When we discuss a topic around here, 
there's plenty of good ideas but also the inevitable bikeshed 
discussions, explanations being asked, explanations being given, and 
other sources of noise. We simply don't have the time to deal with all 
that - the time is short and we only have one shot at this.

That's why I'm thinking of creating a mailing list or maybe another 
group for this. Any ideas on what would be the best approach? I also 
want to gauge interest from threading experts who'd like to participate. 
Please advise: (a) whether you would like to participate to the design; 
(b) keep discussions on the general group; (c) create a separate 
newsgroup; (d) create a mailing list. The latter would have open enrollment.


Andrei
Dec 27 2009
next sibling parent Jason House <jason.james.house gmail.com> writes:
I'd like to participate. Both email and newsgroup are fine. Even IRC or other
Internet chatroom would be ok too. Past concurrency/shared discussions on this
newsgroup have not suffered from bikeshed issues. I think it's all been too
theoretical or outside moat programmers' experience. There's almost no keywords
to debate anyway.

Andrei Alexandrescu Wrote:


 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.
 
 It would be great to open the participation to the design as broadly as 
 possible, but I think it's realistic to say we won't be able to get 
 things done on the newsgroup. When we discuss a topic around here, 
 there's plenty of good ideas but also the inevitable bikeshed 
 discussions, explanations being asked, explanations being given, and 
 other sources of noise. We simply don't have the time to deal with all 
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to participate. 
 Please advise: (a) whether you would like to participate to the design; 
 (b) keep discussions on the general group; (c) create a separate 
 newsgroup; (d) create a mailing list. The latter would have open enrollment.
 
 
 Andrei
Dec 27 2009
prev sibling next sibling parent reply retard <re tard.com.invalid> writes:
Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:


 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.
 
 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.


Have the discussions here ever led to a conclusion and get implemented? 
You should at least disallow posting via the web interface - those broken 
threads start to annoy some people.

While you're at it, I, for one, welcome a new concurrency architecture 
that scales nicely when I want optimal array processing on i486, PII/SSE, 
P4/HT, Core i7/HT/SSE 4.2, and perhaps also on Beowulf (takes care of 
network latencies). Would it use the new immutable actor/message model 
you're building?
Dec 27 2009
parent reply Jason House <jason.james.house gmail.com> writes:
retard Wrote:


 Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:
 

 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.
 
 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.


 
 Have the discussions here ever led to a conclusion and get implemented?


DIP2 is a great example of something designed on this newsgroup that got
implemented very recently. It went from newsgroup to bugzilla to DIP to
implementation. T[new] is another one that almost made it.

 

 You should at least disallow posting via the web interface - those broken 
 threads start to annoy some people.


Do my posts show up as broken threads? I use the web interface almost
exclusively.
Dec 27 2009
next sibling parent retard <re tard.com.invalid> writes:
Sun, 27 Dec 2009 18:56:09 -0500, Jason House wrote:


 retard Wrote:
 

 Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:
 

 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling
 over models and options.
 
 It would be great to open the participation to the design as broadly
 as possible, but I think it's realistic to say we won't be able to
 get things done on the newsgroup. When we discuss a topic around
 here, there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with
 all that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to
 participate. Please advise: (a) whether you would like to participate
 to the design; (b) keep discussions on the general group; (c) create
 a separate newsgroup; (d) create a mailing list. The latter would
 have open enrollment.


 
 Have the discussions here ever led to a conclusion and get implemented?


 
 DIP2 is a great example of something designed on this newsgroup that got
 implemented very recently. It went from newsgroup to bugzilla to DIP to
 implementation. T[new] is another one that almost made it.


Ok, good to know.


 You should at least disallow posting via the web interface - those
 broken threads start to annoy some people.


 
 Do my posts show up as broken threads? I use the web interface almost
 exclusively.


This one didn't break the thread. But sometimes it does break.
Dec 27 2009
prev sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 27/12/09 23:56, Jason House wrote:

 You should at least disallow posting via the web interface - those broken
 threads start to annoy some people.


 Do my posts show up as broken threads? I use the web interface almost
exclusively.


Which web interface do you use? there are 2 of them, maybe one of them 
doesn't break it?
Dec 27 2009
parent Jason House <jason.james.house gmail.com> writes:
Robert Clipsham Wrote:


 On 27/12/09 23:56, Jason House wrote:

 You should at least disallow posting via the web interface - those broken
 threads start to annoy some people.


 Do my posts show up as broken threads? I use the web interface almost
exclusively.


 
 Which web interface do you use? there are 2 of them, maybe one of them 
 doesn't break it?


Here is the URL I used to view your message: 
http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=104129
Dec 27 2009
prev sibling next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
On 27/12/09 20:32, Andrei Alexandrescu wrote:

 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.


Excellent! The last time I had a real play with D2 this is what I found 
was missing, particularly when a lot of the newer features are built 
with concurrency in mind.


 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.


You're most probably right about this. It would be nice for it to be as 
open as possible, you don't want it ending up with bikesheds all over 
the place though, particularly at this early stage. Maybe a sort of 
guided openness could be used? If you go with another newsgroup it won't 
matter about the noise, and you could go through the design process in a 
nice guided manner, getting people's initial ideas, voting on them, 
discussing pro's/cons etc, that way you get something reasonably 
workable before you get to the bikesheds. If you let everyone dive in 
straight off I don't think much will come out of it


 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.


(a) I doubt I'd be much use here, I might have a comment here and there 
to throw in though

(b..$) b or c are the preferable options for me, they're easier for me 
to follow and require little or no setup... excuse my laziness!
Dec 27 2009
prev sibling next sibling parent Kevin Bealer <kevinbealer gmail.com> writes:
Andrei Alexandrescu Wrote:


 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.
 
 It would be great to open the participation to the design as broadly as 
 possible, but I think it's realistic to say we won't be able to get 
 things done on the newsgroup. When we discuss a topic around here, 
 there's plenty of good ideas but also the inevitable bikeshed 
 discussions, explanations being asked, explanations being given, and 
 other sources of noise. We simply don't have the time to deal with all 
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to participate. 
 Please advise: (a) whether you would like to participate to the design; 
 (b) keep discussions on the general group; (c) create a separate 
 newsgroup; (d) create a mailing list. The latter would have open enrollment.
 
 
 Andrei


I'd love to follow the discussion.  I'm not sure whether I have all that much
expertise for contributing to the "core debate".  It seems like you and Walter
have had a pattern where you discuss things among yourself and post the
occasional "here's our thinking" to a group like this for the rest of us; I'd
like
to at least hear about the cool stuff periodically, even if the discussion here
is just chewing the scraps from the real table.

That said, I'll throw out one high level thought on the topic:

As for concurrency itself, my gut reaction is that I'd hope there would be
mutex + thread.  Obviously you want something better, and I look forward
to hearing all the cool stuff about it; but as clunky as we all know mutex +
thread are, it would be good to have them in some form.

One of the downsides of inventing / using a "new thing" is that when all the
existing systems use a particular programming paradigm it becomes hard
to integrate with them if you can't at least emulate that paradigm in a more
or less straightforward way.  If someone is wrapping OpenGL in their D code,
using a C library (just to pick an example, I don't know much OpenGL), it will
be very hard for them to bend OpenGL into an immutable / message passing
approach if it was designed for the land of thread + mutex.

I think that one of the big lessons of what might later be called the "silver
bullet phase" of CompSci, is that there should be a marketplace where the
various paradigms must compete.  If you build a new paradigm for this it is
best if it is a top predator in an ecosystem of ideas, not a new planet with
only a single monoculture.

All paradigms have a place; if a paradigm is used, there is probably a niche
for that approach.  If it exists, it must have value.  Except for bubble-sort.

Kevin
Dec 28 2009
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2009-12-27 15:32:52 -0500, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.
 
 It would be great to open the participation to the design as broadly as 
 possible, but I think it's realistic to say we won't be able to get 
 things done on the newsgroup. When we discuss a topic around here, 
 there's plenty of good ideas but also the inevitable bikeshed 
 discussions, explanations being asked, explanations being given, and 
 other sources of noise. We simply don't have the time to deal with all 
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to 
 participate. Please advise: (a) whether you would like to participate 
 to the design; (b) keep discussions on the general group; (c) create a 
 separate newsgroup; (d) create a mailing list. The latter would have 
 open enrollment.


I think it should be as open as possible. If done in a separate smaller 
group, it may be a good idea to post reports to the general newsgroup 
more or less regularly so that those who cannot participate in the 
detailed discussions have an idea of where it's going, and also to get 
more general input.

About the bikeshed issue, I'm not sure how much those bikeshed 
discussions are slowing down the more important ones, but they often 
start from legitimate real, often syntactic, issues. Those discussions 
shouldn't be avoided just because everyone has an opinion. But perhaps 
regular reports to the general newsgroup would help confining them 
there.

I'd be in favor of creating a newsgroup for concurrency, and I'll 
probably want to participate a little too, although I'm not sure how 
much yet.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Dec 28 2009
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Michel Fortin wrote:

 On 2009-12-27 15:32:52 -0500, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:
 

 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.

 It would be great to open the participation to the design as broadly 
 as possible, but I think it's realistic to say we won't be able to get 
 things done on the newsgroup. When we discuss a topic around here, 
 there's plenty of good ideas but also the inevitable bikeshed 
 discussions, explanations being asked, explanations being given, and 
 other sources of noise. We simply don't have the time to deal with all 
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to 
 participate. Please advise: (a) whether you would like to participate 
 to the design; (b) keep discussions on the general group; (c) create a 
 separate newsgroup; (d) create a mailing list. The latter would have 
 open enrollment.


 
 I think it should be as open as possible. If done in a separate smaller 
 group, it may be a good idea to post reports to the general newsgroup 
 more or less regularly so that those who cannot participate in the 
 detailed discussions have an idea of where it's going, and also to get 
 more general input.


That's obviously the best way to go, but there are a couple of 
circumstances that make that more difficult.

1. Chapter drafts will be the basis for discussion, but understandably 
the publisher does not allow me to freely distribute them.

2. Time. There are regulars on this group that have a "when in doubt, 
make them sweat" policy. I think it's a very good and gainful attitude 
for everyone involved, and I generally enjoy discussing this or that 
idea because it helps me and others gain a better understanding, but 
this time there won't be much time for discussions of the form:

a) Poster: "Subtle issue X sounds like a bad idea. I don't agree with it."

b) <Long argumentation back and forth.>

c) Poster: "I stay unconvinced." or "That makes sense."

There will be very little time for anything like this, particularly if 
explaining X requires a fair amount of background building.

Building a shared vision is very difficult among only a small group of 
people, and doing so for a larger group will be an enormous drag. I feel 
very lucky that Walter and I share views most of the time (except, of 
course, when he's wrong :o)).


 About the bikeshed issue, I'm not sure how much those bikeshed 
 discussions are slowing down the more important ones, but they often 
 start from legitimate real, often syntactic, issues. Those discussions 
 shouldn't be avoided just because everyone has an opinion. But perhaps 
 regular reports to the general newsgroup would help confining them there.
 
 I'd be in favor of creating a newsgroup for concurrency, and I'll 
 probably want to participate a little too, although I'm not sure how 
 much yet.


OK, let's see what Walter thinks.


Andrei
Dec 28 2009
next sibling parent retard <re tard.com.invalid> writes:
Mon, 28 Dec 2009 10:20:53 -0600, Andrei Alexandrescu wrote:


 Michel Fortin wrote:

 On 2009-12-27 15:32:52 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:
 

 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.

 It would be great to open the participation to the design as broadly
 as possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to
 participate. Please advise: (a) whether you would like to participate
 to the design; (b) keep discussions on the general group; (c) create a
 separate newsgroup; (d) create a mailing list. The latter would have
 open enrollment.


 
 I think it should be as open as possible. If done in a separate smaller
 group, it may be a good idea to post reports to the general newsgroup
 more or less regularly so that those who cannot participate in the
 detailed discussions have an idea of where it's going, and also to get
 more general input.


 
 That's obviously the best way to go, but there are a couple of
 circumstances that make that more difficult.
 
 1. Chapter drafts will be the basis for discussion, but understandably
 the publisher does not allow me to freely distribute them.
 
 2. Time. There are regulars on this group that have a "when in doubt,
 make them sweat" policy. I think it's a very good and gainful attitude
 for everyone involved, and I generally enjoy discussing this or that
 idea because it helps me and others gain a better understanding, but
 this time there won't be much time for discussions of the form:
 
 a) Poster: "Subtle issue X sounds like a bad idea. I don't agree with
 it."
 
 b) <Long argumentation back and forth.>
 
 c) Poster: "I stay unconvinced." or "That makes sense."
 
 There will be very little time for anything like this, particularly if
 explaining X requires a fair amount of background building.
 
 Building a shared vision is very difficult among only a small group of
 people, and doing so for a larger group will be an enormous drag. I feel
 very lucky that Walter and I share views most of the time (except, of
 course, when he's wrong :o)).


I think a shared vision is easier to achieve in a small group. If the 
group size is 1, you only need to convince yourself, if 2, you have only 
one other person who needs to agree and so forth. In groups like c++ 
standards committee every player in the software industry wants to be 
heard. That's why it's not gonna be c++0x anymore, maybe c++1x (c++0x0a) 
or c++2x.

I don't like how in D community new features are added. We often first 
get a binary dmd release containing the new feature, after that some 
obscure half-official explanation about the corner cases that will be 
fixed in the next release. Only after that Walter might answer 'Yes.' to 
some well laid questions and finally after 2-3 years a very terse 
informal text is added to the specification.

What kind of experts are you exactly seeking now? Guys who build massive 
and scalable native/green thread pools to serve http requests? Guys who 
compute weather forecasts with 10.000 node MPI setups? Is it going to be 
a multi-threaded, multi-core, or multi-computer architecture?
Dec 28 2009
prev sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2009-12-28 11:20:53 -0500, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 Michel Fortin wrote:
 

 I think it should be as open as possible. If done in a separate smaller 
 group, it may be a good idea to post reports to the general newsgroup 
 more or less regularly so that those who cannot participate in the 
 detailed discussions have an idea of where it's going, and also to get 
 more general input.


 
 That's obviously the best way to go, but there are a couple of 
 circumstances that make that more difficult.
 
 1. Chapter drafts will be the basis for discussion, but understandably 
 the publisher does not allow me to freely distribute them.


I see. I find that unfortunate, because this seems to imply it'll need 
to be a closed group, not only closed regarding the participants but 
also regarding external observers (because the basis of the discussion 
will be hidden from them).

Often you can evaluate how good something is not only by looking at the 
final result, but by looking at the process that led to it. Having an 
archive of the past is often very useful to check why some things have 
been made that way. But that works best if the discussion are done in 
the open.



 2. Time. There are regulars on this group that have a "when in doubt, 
 make them sweat" policy. I think it's a very good and gainful attitude 
 for everyone involved, and I generally enjoy discussing this or that 
 idea because it helps me and others gain a better understanding, but 
 this time there won't be much time for discussions of the form:
 
 a) Poster: "Subtle issue X sounds like a bad idea. I don't agree with it."
 
 b) <Long argumentation back and forth.>
 
 c) Poster: "I stay unconvinced." or "That makes sense."
 
 There will be very little time for anything like this, particularly if 
 explaining X requires a fair amount of background building.
 
 Building a shared vision is very difficult among only a small group of 
 people, and doing so for a larger group will be an enormous drag. I 
 feel very lucky that Walter and I share views most of the time (except, 
 of course, when he's wrong :o)).


I think it'd be a good idea to set an objective for the group for that 
limited time frame. This way if the discussion deviates towards 
something off-topic it's easy to suggest posting to the general or the 
D.learn newsgroup instead, or if the discussion is draining taking too 
much time in explanations it's easy to justify postponing that to later 
in order to fulfill the more pressing objectives. Just be careful not 
setting the goal too broadly.

Also keep in mind that we don't really need a shared vision among 
everyone. What's needed is someone who takes the decisions. Discussion 
is only needed to help that person take the right decisions. Although 
consensus among all members certainly boosts the decider 
self-confidence, it is not required, and not necessarily desirable 
either. A consensus among only a few key people is all that is needed, 
and this has little to do with who is allowed to raise issues and 
propose solutions.

And those few key people are the implementors, because if they don't 
agree the result isn't gonna be pretty.

(This reminds me pretty much of how HTML5 is being developed right now: 
Ian Hickson, the spec editor, decides what goes in and out. But he 
always seeks the broadest consensus among the major browser vendors 
because in the end it's them who decide what they want to implement, 
and if they refuse to implement the spec, the spec is useless. This 
doesn't prevent the WHATWG mailing list from being open to all. I did 
some research a few years ago, posted the results on the list and after 
some discussion it was used to make the <figure> element what it is 
now. I wouldn't have bothered if the list wasn't open.)

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Dec 28 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Michel Fortin wrote:

 Also keep in mind that we don't really need a shared vision among 
 everyone. What's needed is someone who takes the decisions. Discussion 
 is only needed to help that person take the right decisions. Although 
 consensus among all members certainly boosts the decider 
 self-confidence, it is not required, and not necessarily desirable 
 either. A consensus among only a few key people is all that is needed, 
 and this has little to do with who is allowed to raise issues and 
 propose solutions.


The real problem with a concurrency model is that very few programmers 
understand the issues. The failed Java concurrency model is an example 
of this shortage. For another, about 5 years ago I attended a panel of 
30 of the top C++ experts in the world to discuss a concurrency model 
for C++0x.

It didn't take long for it to become obvious that exactly two people 
understood the issues - Hans Boehm and Herb Sutter. The rest of us sat 
there slack-jawed and drooling, asking endless inane questions. I wish I 
had the patience Hans and Herb showed in dealing with this.

Since then I have tried to master this topic, but I don't have much 
experience writing complex multithreaded code. So what we need are 
people who are experienced with MT code who can evaluate the design to 
see if we missed the boat or not. I'd rather not shoot at the moon yet 
wind up orbiting some asteroid.
Jan 08 2010
next sibling parent Jason House <jason.james.house gmail.com> writes:
Walter Bright Wrote:


 Since then I have tried to master this topic, but I don't have much 
 experience writing complex multithreaded code. So what we need are 
 people who are experienced with MT code who can evaluate the design to 
 see if we missed the boat or not. I'd rather not shoot at the moon yet 
 wind up orbiting some asteroid.


I can't comment on the future threading model, but I have been trying to use
the current one by converting an existing multithreaded app. I've gotten a lot
further than my last update on this list. Bugzilla 3660 is quite severe. My
current understanding of the issue is that matching of function templates to
attempted function calls isn't aware of the distinction between shared and
non-shared functions.
Jan 08 2010
prev sibling next sibling parent reply Brien <reply tolist.com> writes:
Walter Bright Wrote:

 Since then I have tried to master this topic, but I don't have much 
 experience writing complex multithreaded code. So what we need are 
 people who are experienced with MT code who can evaluate the design to 
 see if we missed the boat or not. I'd rather not shoot at the moon yet 
 wind up orbiting some asteroid.


Walter,

How would you characterize D1.0 in terms of it's use for big multithreaded
applications?   Does the compiler produce safe/performant code?

Thanks,
Brien
Jan 08 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Brien wrote:

 How would you characterize D1.0 in terms of it's use for big
 multithreaded applications?   Does the compiler produce
 safe/performant code?


D 1.0 is about where Java is in its support for MT programming. You can
do it, but you get little or no support from the type system.
Jan 08 2010
prev sibling next sibling parent reply "Steven E. Harris" <seh panix.com> writes:
Walter Bright <newshound1 digitalmars.com> writes:


 The failed Java concurrency model is an example of this shortage.


Can you elaborate? Are you talking pre-JSR-000133=C2=B9?


Footnotes:=20
=C2=B9 http://jcp.org/aboutJava/communityprocess/review/jsr133/index.html

--=20
Steven E. Harris
Jan 08 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Steven E. Harris wrote:

 Walter Bright <newshound1 digitalmars.com> writes:
 

 The failed Java concurrency model is an example of this shortage.


 
 Can you elaborate? Are you talking pre-JSR-000133¹?
 
 
 Footnotes: 
 ¹ http://jcp.org/aboutJava/communityprocess/review/jsr133/index.html
 



In Java, you can share data between threads without restriction. Hence, 
code is vulnerable to all the sequential consistency bugs.
Jan 08 2010
prev sibling next sibling parent reply retard <re tard.com.invalid> writes:
Fri, 08 Jan 2010 00:50:26 -0800, Walter Bright wrote:


 Michel Fortin wrote:

 Also keep in mind that we don't really need a shared vision among
 everyone. What's needed is someone who takes the decisions. Discussion
 is only needed to help that person take the right decisions. Although
 consensus among all members certainly boosts the decider
 self-confidence, it is not required, and not necessarily desirable
 either. A consensus among only a few key people is all that is needed,
 and this has little to do with who is allowed to raise issues and
 propose solutions.


 
 The real problem with a concurrency model is that very few programmers
 understand the issues. The failed Java concurrency model is an example
 of this shortage. For another, about 5 years ago I attended a panel of
 30 of the top C++ experts in the world to discuss a concurrency model
 for C++0x.
 
 It didn't take long for it to become obvious that exactly two people
 understood the issues - Hans Boehm and Herb Sutter. The rest of us sat
 there slack-jawed and drooling, asking endless inane questions. I wish I
 had the patience Hans and Herb showed in dealing with this.
 
 Since then I have tried to master this topic, but I don't have much
 experience writing complex multithreaded code. So what we need are
 people who are experienced with MT code who can evaluate the design to
 see if we missed the boat or not. I'd rather not shoot at the moon yet
 wind up orbiting some asteroid.


If only 2 of the top 30 c++ experts understand concurrency models, what 
makes you think that D programming community, mostly consisting of novice/
hobbyist/student programmers have any better luck? There are only a 
handful of such experts as Sean, Andrei, Bartosz, Brad et al here. The D 
community is maybe 3 to 4 orders of magnitude smaller than the global C++ 
community. Only a handful of enterprises use D and there is basically no 
academic research happening around D.
Jan 08 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
retard wrote:

 Fri, 08 Jan 2010 00:50:26 -0800, Walter Bright wrote:
 

 Michel Fortin wrote:

 Also keep in mind that we don't really need a shared vision among
 everyone. What's needed is someone who takes the decisions. Discussion
 is only needed to help that person take the right decisions. Although
 consensus among all members certainly boosts the decider
 self-confidence, it is not required, and not necessarily desirable
 either. A consensus among only a few key people is all that is needed,
 and this has little to do with who is allowed to raise issues and
 propose solutions.


 The real problem with a concurrency model is that very few programmers
 understand the issues. The failed Java concurrency model is an example
 of this shortage. For another, about 5 years ago I attended a panel of
 30 of the top C++ experts in the world to discuss a concurrency model
 for C++0x.

 It didn't take long for it to become obvious that exactly two people
 understood the issues - Hans Boehm and Herb Sutter. The rest of us sat
 there slack-jawed and drooling, asking endless inane questions. I wish I
 had the patience Hans and Herb showed in dealing with this.

 Since then I have tried to master this topic, but I don't have much
 experience writing complex multithreaded code. So what we need are
 people who are experienced with MT code who can evaluate the design to
 see if we missed the boat or not. I'd rather not shoot at the moon yet
 wind up orbiting some asteroid.


 
 If only 2 of the top 30 c++ experts understand concurrency models, what 
 makes you think that D programming community, mostly consisting of novice/
 hobbyist/student programmers have any better luck? There are only a 
 handful of such experts as Sean, Andrei, Bartosz, Brad et al here. The D 
 community is maybe 3 to 4 orders of magnitude smaller than the global C++ 
 community. Only a handful of enterprises use D and there is basically no 
 academic research happening around D.


Our flagship for communicating across concurrent entities is message 
passing, which is much more palatable than classic share-the-world 
concurrency models.

I'm very excited about how this is coming together.


Andrei
Jan 08 2010
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
retard wrote:

 If only 2 of the top 30 c++ experts understand concurrency models, what 
 makes you think that D programming community, mostly consisting of novice/
 hobbyist/student programmers have any better luck?


That is why the flagship of doing concurrency in D will be message 
passing, which is how Erlang, Scala, and Go do concurrency. Using 
message passing will avoid any need by the user to understand sharing, 
sequential consistency, and synchronization issues.

Sharing and synchronization will be for the advanced programmer.



 Only a handful of enterprises use D and there is basically no 
 academic research happening around D.


That'll change!
Jan 08 2010
parent reply retard <re tard.com.invalid> writes:
Fri, 08 Jan 2010 12:33:06 -0800, Walter Bright wrote:


 retard wrote:

 If only 2 of the top 30 c++ experts understand concurrency models, what
 makes you think that D programming community, mostly consisting of
 novice/ hobbyist/student programmers have any better luck?


 
 That is why the flagship of doing concurrency in D will be message
 passing, which is how Erlang, Scala, and Go do concurrency. Using
 message passing will avoid any need by the user to understand sharing,
 sequential consistency, and synchronization issues.
 
 Sharing and synchronization will be for the advanced programmer.


My point was, what makes D community members more capable of reasoning 
about these issues. I bet it's obvious to the top 30 C++ experts that 
other kinds of concurrency models exist than multi-threaded shared memory 
abstraction or SIMD. The problem isn't just about coming up with a model 
-- if it helps you in any way, go ahead and browse old academic papers. 
E.g. CSP was invented 32 years ago.

There are message passing libs for C++, too, for instance MPI. Making the 
feature built-in doesn't automatically make creating concurrent 
applications easier. For instance message passing is harder to use than 
implicit parallelism and auto-vectorization when doing parallel 
processing in loops.
Jan 08 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
retard wrote:

 My point was, what makes D community members more capable of reasoning 
 about these issues.


Nothing. What makes D special, though, is better language support for 
sound concurrent programming. But you'll still need a mental model of 
what's going on, just like you cannot program very successfully in C if 
you have no mental model of what a pointer is.


 I bet it's obvious to the top 30 C++ experts that 
 other kinds of concurrency models exist than multi-threaded shared memory 
 abstraction or SIMD. The problem isn't just about coming up with a model 
 -- if it helps you in any way, go ahead and browse old academic papers. 
 E.g. CSP was invented 32 years ago.
 
 There are message passing libs for C++, too, for instance MPI. Making the 
 feature built-in doesn't automatically make creating concurrent 
 applications easier.


Yes it does. C++'s (and Java's, Scala's, and Go's) gaping problem is 
there's no way to ensure that you don't have implicit sharing going on. 
In D, such can be guaranteed.


 For instance message passing is harder to use than 
 implicit parallelism and auto-vectorization when doing parallel 
 processing in loops.


Sure. Except that implicit parallelism is inherently unsound unless the 
type system can support it.

I'll go further than that. Any language that supports implicit sharing 
of mutable data is inherently unsound for multithreaded programming.
Jan 08 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Walter Bright (newshound1 digitalmars.com)'s article

 Sure. Except that implicit parallelism is inherently unsound unless the
 type system can support it.
 I'll go further than that. Any language that supports implicit sharing
 of mutable data is inherently unsound for multithreaded programming.


One thing that I think needs to be considered in the concurrency architecture is
the case of performance-critical massive data parallelism.  In these cases
concurrency is actually not very hard and a simple parallel foreach loop covers
a
lot of cases.  As far as safety, the amount of code being executed inside a
parallel foreach loop is generally small enough that it's easy to reason about,
and thus it's ok not to have any hard static guarantees and leave safety up to
the
programmer, as long as the programmer understands at least the basics of
concurrency.

I would like to see a situation where OpenMP/ParallelFuture-style concurrency is
still implementable in D without unreasonable performance or syntactic overhead
after the new concurrency system is fully in place.
Jan 08 2010
next sibling parent retard <re tard.com.invalid> writes:
Fri, 08 Jan 2010 23:12:38 +0000, dsimcha wrote:


 == Quote from Walter Bright (newshound1 digitalmars.com)'s article

 Sure. Except that implicit parallelism is inherently unsound unless the
 type system can support it.
 I'll go further than that. Any language that supports implicit sharing
 of mutable data is inherently unsound for multithreaded programming.


 
 One thing that I think needs to be considered in the concurrency
 architecture is the case of performance-critical massive data
 parallelism.  In these cases concurrency is actually not very hard and a
 simple parallel foreach loop covers a lot of cases.  As far as safety,
 the amount of code being executed inside a parallel foreach loop is
 generally small enough that it's easy to reason about, and thus it's ok
 not to have any hard static guarantees and leave safety up to the
 programmer, as long as the programmer understands at least the basics of
 concurrency.
 
 I would like to see a situation where OpenMP/ParallelFuture-style
 concurrency is still implementable in D without unreasonable performance
 or syntactic overhead after the new concurrency system is fully in
 place.


These systems also solve different kinds of problems. MMX/SSE/Altivec and 
now GPU hardware solves problems where you run a single operation on a 
large set of data simultaneously. The communication is really cheap 
because e.g. in SSE data is in "ordinary" registers which are the fastest 
memory units available.

Message passing system scales well if you have many computers that form a 
computing cluster. It also performs well when the computation can be 
split into distinct tasks which run in separate threads and communicate 
rarely. Unless messages are translated into native SIMD primitives in 
data parallel cases, message passing will be much slower when you do that 
kind of processing. Parallel loops don't scale that well on multiple 
computing nodes - the computation needs to be split into larger tasks to 
minimize communication delays since nodes simply can't share the same 
thread state.
Jan 08 2010
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
dsimcha wrote:

 == Quote from Walter Bright (newshound1 digitalmars.com)'s article

 Sure. Except that implicit parallelism is inherently unsound unless the
 type system can support it.
 I'll go further than that. Any language that supports implicit sharing
 of mutable data is inherently unsound for multithreaded programming.


 
 One thing that I think needs to be considered in the concurrency architecture
is
 the case of performance-critical massive data parallelism.  In these cases
 concurrency is actually not very hard and a simple parallel foreach loop
covers a
 lot of cases.  As far as safety, the amount of code being executed inside a
 parallel foreach loop is generally small enough that it's easy to reason about,
 and thus it's ok not to have any hard static guarantees and leave safety up to
the
 programmer, as long as the programmer understands at least the basics of
concurrency.


It's hard to reason about even small examples. Consider the double 
checked locking bug. I even thought I understood it, until Bartosz 
discovered I'd written one into Phobos.

As for understanding the basics of concurrency, I defy anyone to read 
the Intel CPU manuals and discover how memory read/write barriers work 
on Intel chips and how and when they should be used.

With C++, everyone using memory barriers has to figure it out, and the 
consequences of getting it wrong are very, very, very obscure bugs. With 
D, only the compiler writer has to get it right.



 I would like to see a situation where OpenMP/ParallelFuture-style concurrency
is
 still implementable in D without unreasonable performance or syntactic overhead
 after the new concurrency system is fully in place.


There's no reason it cannot be.

At the worst, you can cast away "shared" and cowboy it exactly as you 
would in C++.

There is no circumstance where you are worse off with D than with C++, 
and quite a lot of cases where you're much better off.
Jan 08 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Walter Bright (newshound1 digitalmars.com)'s article

 dsimcha wrote:

 == Quote from Walter Bright (newshound1 digitalmars.com)'s article

 Sure. Except that implicit parallelism is inherently unsound unless the
 type system can support it.
 I'll go further than that. Any language that supports implicit sharing
 of mutable data is inherently unsound for multithreaded programming.


 One thing that I think needs to be considered in the concurrency architecture
is
 the case of performance-critical massive data parallelism.  In these cases
 concurrency is actually not very hard and a simple parallel foreach loop
covers a
 lot of cases.  As far as safety, the amount of code being executed inside a
 parallel foreach loop is generally small enough that it's easy to reason about,
 and thus it's ok not to have any hard static guarantees and leave safety up to
the
 programmer, as long as the programmer understands at least the basics of




concurrency.

 It's hard to reason about even small examples. Consider the double
 checked locking bug. I even thought I understood it, until Bartosz
 discovered I'd written one into Phobos.
 As for understanding the basics of concurrency, I defy anyone to read
 the Intel CPU manuals and discover how memory read/write barriers work
 on Intel chips and how and when they should be used.
 With C++, everyone using memory barriers has to figure it out, and the
 consequences of getting it wrong are very, very, very obscure bugs. With
 D, only the compiler writer has to get it right.

 I would like to see a situation where OpenMP/ParallelFuture-style concurrency
is
 still implementable in D without unreasonable performance or syntactic overhead
 after the new concurrency system is fully in place.


 There's no reason it cannot be.
 At the worst, you can cast away "shared" and cowboy it exactly as you
 would in C++.
 There is no circumstance where you are worse off with D than with C++,
 and quite a lot of cases where you're much better off.


Ok, one thing that leads to a lot of misunderstanding and confusion is that
there's (as far as I can tell) absolutely no documentation of what shared
does/will do or what its semantics are/will be, other than a few newsgroup posts
from years ago,which were tentative, and the "migrating to shared" article,
which
is mostly about thread-local storage.  Also, as far as I can tell, shared is not
fully implemented yet, because core.thread doesn't require a shared delegate. 
Is
this correct?

Here are some questions, and this is just what I've been able to think of off
the
top of my head in a few minutes:

What guarantees is shared supposed to provide?
What does shared have to do with synchronization?
What does shared have to do with memory barriers?
What are the semantics of casting FROM unshared TO shared?
What are the semantics of casting FROM shared TO unshared?
Why doesn't shared appear anywhere in core.thread?
Jan 08 2010
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
dsimcha wrote:

 Ok, one thing that leads to a lot of misunderstanding and confusion is that
 there's (as far as I can tell) absolutely no documentation of what shared
 does/will do or what its semantics are/will be, other than a few newsgroup
posts
 from years ago,which were tentative, and the "migrating to shared" article,
which
 is mostly about thread-local storage.  Also, as far as I can tell, shared is
not
 fully implemented yet, because core.thread doesn't require a shared delegate. 
Is
 this correct?


Yes.


 Here are some questions, and this is just what I've been able to think of off
the
 top of my head in a few minutes:
 
 What guarantees is shared supposed to provide?


Shared means that multiple threads can access the data. The guarantee is 
that if it is not shared, and not immutable, that only the current 
thread can see it.


 What does shared have to do with synchronization?


Only shared data can be synchronized. It makes no sense to synchronize 
thread local data.


 What does shared have to do with memory barriers?


Reading/writing shared data emits memory barriers to ensure sequential 
consistency (not implemented).


 What are the semantics of casting FROM unshared TO shared?


You'd better be sure there are no other unshared references to that same 
data.


 What are the semantics of casting FROM shared TO unshared?


You'd better be sure there are no other shared references to that same data.


 Why doesn't shared appear anywhere in core.thread?


Because nobody has spent the effort to do it.
Jan 08 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter Bright wrote:

 dsimcha wrote:

 Ok, one thing that leads to a lot of misunderstanding and confusion is 
 that
 there's (as far as I can tell) absolutely no documentation of what shared
 does/will do or what its semantics are/will be, other than a few 
 newsgroup posts
 from years ago,which were tentative, and the "migrating to shared" 
 article, which
 is mostly about thread-local storage.  Also, as far as I can tell, 
 shared is not
 fully implemented yet, because core.thread doesn't require a shared 
 delegate.  Is
 this correct?


 
 Yes.
 

 Here are some questions, and this is just what I've been able to think 
 of off the
 top of my head in a few minutes:

 What guarantees is shared supposed to provide?


 
 Shared means that multiple threads can access the data. The guarantee is 
 that if it is not shared, and not immutable, that only the current 
 thread can see it.
 

 What does shared have to do with synchronization?


 
 Only shared data can be synchronized. It makes no sense to synchronize 
 thread local data.
 

 What does shared have to do with memory barriers?


 
 Reading/writing shared data emits memory barriers to ensure sequential 
 consistency (not implemented).
 

 What are the semantics of casting FROM unshared TO shared?


 
 You'd better be sure there are no other unshared references to that same 
 data.
 

 What are the semantics of casting FROM shared TO unshared?


 
 You'd better be sure there are no other shared references to that same 
 data.
 

 Why doesn't shared appear anywhere in core.thread?


 
 Because nobody has spent the effort to do it.


Thanks Walter for doing this. These questions and their answers could 
start an FAQ page on D's website.

Andrei
Jan 09 2010
parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Andrei Alexandrescu wrote:


 Thanks Walter for doing this. These questions and their answers could 
 start an FAQ page on D's website.

 Andrei


In the mean time:
http://www.prowiki.org/wiki4d/wiki.cgi?DevelopmentWithD/FAQ/Shared
Jan 09 2010
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
dsimcha Wrote:

 Why doesn't shared appear anywhere in core.thread?


Until the semantics of shared are finalized and the critical bugs are worked
out I don't want to integrate shared with core.thread--it's the centerpoint of
too much stuff (GC, etc).  But general purpose concurrency should never touch
core.thread anyway.  There will be a higher-level interface available that's
safer and easier to use.  One could also argue that core.thread should accept
unshared delegates, since it's possible to do so safely, you just don't have
any language support to help.  The design philosophy with the modules in core
is quite different from std.
Jan 08 2010
prev sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2010-01-08 03:50:26 -0500, Walter Bright <newshound1 digitalmars.com> said:


 Michel Fortin wrote:

 Also keep in mind that we don't really need a shared vision among 
 everyone. What's needed is someone who takes the decisions. Discussion 
 is only needed to help that person take the right decisions. Although 
 consensus among all members certainly boosts the decider 
 self-confidence, it is not required, and not necessarily desirable 
 either. A consensus among only a few key people is all that is needed, 
 and this has little to do with who is allowed to raise issues and 
 propose solutions.


 
 The real problem with a concurrency model is that very few programmers 
 understand the issues.
 [...]
 
 Since then I have tried to master this topic, but I don't have much 
 experience writing complex multithreaded code. So what we need are 
 people who are experienced with MT code who can evaluate the design to 
 see if we missed the boat or not. I'd rather not shoot at the moon yet 
 wind up orbiting some asteroid.


I'm not sure what you're replying to in the above message. I was simply 
saying that to accomplish progress in something, the only necessary 
condition is to have an understanding among those who will actually 
implement that thing. Other opinions can be very useful, they can also 
be irrelevant, but you don't absolutely need a consensus among every 
participant in the discussion for something to advance.

You're making a case for making "shared" help programmers because most 
don't grasp all the implications of shared memory between processors. 
That's good but I don't really see how it relates.

By the way, I have seen my lot of multithreaded code in C++, sometime 
very well written, sometime just wrong, and sometime correct but 
obscure enough that takes help from the original author to understand. 
I've written code that probably enters in all these categories, and 
debugged my share of hard to track concurrency issues too. Were I was 
working previously, the most time-consuming bugs were always 
concurrency issues, generally race conditions happening every few 
hours/days or in certain rare conditions but not always, causing either 
deadlocks or other problematic behaviors. Fortunately, we also had 
failsafes to stop things in case the software would crash as it had 
safety implications.

Although I'm no longer working there and dealing much less with 
concurrency issues now, I'm still interested in the subject.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Jan 09 2010
prev sibling parent Tim Matthews <tim.matthews7 gmail.com> writes:
On 29/12/2009 3:48 a.m., Michel Fortin wrote:

 On 2009-12-27 15:32:52 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:


 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.

 It would be great to open the participation to the design as broadly
 as possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to
 participate. Please advise: (a) whether you would like to participate
 to the design; (b) keep discussions on the general group; (c) create a
 separate newsgroup; (d) create a mailing list. The latter would have
 open enrollment.


 I think it should be as open as possible. If done in a separate smaller
 group, it may be a good idea to post reports to the general newsgroup
 more or less regularly so that those who cannot participate in the
 detailed discussions have an idea of where it's going, and also to get
 more general input.

 About the bikeshed issue, I'm not sure how much those bikeshed
 discussions are slowing down the more important ones, but they often
 start from legitimate real, often syntactic, issues. Those discussions
 shouldn't be avoided just because everyone has an opinion. But perhaps
 regular reports to the general newsgroup would help confining them there.

 I'd be in favor of creating a newsgroup for concurrency, and I'll
 probably want to participate a little too, although I'm not sure how
 much yet.


I agree with the open as possible +1. A closed model would likely only 
permit fully committed and those that have no say without letting 
anything in between where as the open allows some feedback.

As for off topic: This can be avoided by using another form of 
communication. Maybe just add a few announcements here and put all 
proposals with for/against arguments on the wiki. Or anything that 
supports some sort of moderation/editing but I think wiki is perfect for 
this.
Dec 28 2009
prev sibling next sibling parent reply "Phil Deets" <pjdeets2 gmail.com> writes:
On Sun, 27 Dec 2009 15:32:52 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 That's why I'm thinking of creating a mailing list or maybe another  
 group for this. Any ideas on what would be the best approach? I also  
 want to gauge interest from threading experts who'd like to participate.  
 Please advise: (a) whether you would like to participate to the design;  
 (b) keep discussions on the general group; (c) create a separate  
 newsgroup; (d) create a mailing list. The latter would have open  
 enrollment.


I probably wouldn't contribute much, but I'm satisfied as long as I can  
view the discussion. I have learned a lot from the D newsgroups.

I would strongly prefer a newsgroup to a mailing list. I wish every  
mailing list on the Internet would convert to a newsgroup.

The rest of this message is just a dump of my limited experience with  
threading in case its useful for you to see the perspective of someone  
without a lot of threading experience.

My only major experience with concurrency involved a program which was  


threads in the C++ side were in while(true) loops which checked volatile  
boolean flags for work and slept briefly when there was nothing to do.  
(See below for a general idea of what the code looked like.) I didn't use  
any synchronization except the volatile flags and global volatile sets of  
parameters. This worked for me since only one thread handled any one piece  
of functionality. So some built-in way of low overhead cross-thread  
function calling would be nice. Or if there was a better way to do it, a  
design that would nudge me in the right direction would be good.

Actually, after looking at the code below, I'm not convinced I got it  
right since TakeData could possibly be called before thread1_buffer got  
filled.

globals:
volatile bool take_data_flag = false;
volatile char* take_data_buffer = null; // constant buffer size (something  
like 32KiB)

thread1:
void TakeData(char* buffer) {
	take_data_buffer = buffer;
	take_data_flag = true;
	take_data_called = true;
}

bool take_data_called = false;
char thread1_buffer[BUFFER_SIZE];
while (true) {
	if (!take_data_called) {
		// Add data to thread1_buffer
		if (<buffer is full>) {
			TakeData(thread1_buffer);
		}
	} else {
		if (take_data_flag == false) {
			// ok to use buffer again
			take_data_called = false;
		}
	}
	sleep(50);
}

thread2:
char thread2_buffer[BUFFER_SIZE];
while (true) {
	if (take_data_flag) {
		// copy take_data_buffer to thread2_buffer
		take_data_flag = false;
		// process thread2_buffer
	}
	else
		sleep(50);
}
Dec 28 2009
parent reply Leandro Lucarella <llucax gmail.com> writes:
Phil Deets, el 28 de diciembre a las 11:09 me escribiste:

 On Sun, 27 Dec 2009 15:32:52 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 

That's why I'm thinking of creating a mailing list or maybe
another group for this. Any ideas on what would be the best
approach? I also want to gauge interest from threading experts
who'd like to participate. Please advise: (a) whether you would
like to participate to the design; (b) keep discussions on the
general group; (c) create a separate newsgroup; (d) create a
mailing list. The latter would have open enrollment.


 
 I probably wouldn't contribute much, but I'm satisfied as long as I
 can view the discussion. I have learned a lot from the D newsgroups.
 
 I would strongly prefer a newsgroup to a mailing list. I wish every
 mailing list on the Internet would convert to a newsgroup.


Then you will looove Gmane, because it is exactly that a NNTP gateway for
almost any ML:
http://www.gmane.org/

(you can add missing mailing lists)

BTW, I think all the discussion should be here, if there is no time to
discuss, there is no point in creating another ML/NG, just discuss it
privately and be done with it (I really think it would be a very bad idea
though).

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
"Somos testigos de Jaimito, venimos a traer la salvación, el mundo va a
desaparecer, somos testigos de Jaimito!". Nos enyoguizamos... Así que
"somos testigos"? Te dejo el culo hecho un vino, y la conch'itumá, y la
conch'itumá!
	-- Sidharta Kiwi
Dec 28 2009
next sibling parent "Phil Deets" <pjdeets2 gmail.com> writes:
On Mon, 28 Dec 2009 12:46:25 -0500, Leandro Lucarella <llucax gmail.com>  
wrote:


 Then you will looove Gmane, because it is exactly that a NNTP gateway for
 almost any ML:
 http://www.gmane.org/

 (you can add missing mailing lists)


Thanks for the link. It looks awesome.
Dec 28 2009
prev sibling parent "Steven E. Harris" <seh panix.com> writes:
Leandro Lucarella <llucax gmail.com> writes:


 Then you will looove Gmane, because it is exactly that a NNTP gateway for
 almost any ML


As an example, this mailing list is available there as group

  gmane.comp.lang.d.general

However, the mailing list doesn't seem to accept posts through Gmane,
despite the reported status of "posting allowed" here:

  http://dir.gmane.org/gmane.comp.lang.d.general

-- 
Steven E. Harris
Dec 30 2009
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 27 Dec 2009 12:32:52 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 I think we are now in the position of defining a solid set of  
 concurrency primitives for D. This follows many months of mulling over  
 models and options.

 It would be great to open the participation to the design as broadly as  
 possible, but I think it's realistic to say we won't be able to get  
 things done on the newsgroup. When we discuss a topic around here,  
 there's plenty of good ideas but also the inevitable bikeshed  
 discussions, explanations being asked, explanations being given, and  
 other sources of noise. We simply don't have the time to deal with all  
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another  
 group for this. Any ideas on what would be the best approach? I also  
 want to gauge interest from threading experts who'd like to participate.  
 Please advise: (a) whether you would like to participate to the design;  
 (b) keep discussions on the general group; (c) create a separate  
 newsgroup; (d) create a mailing list. The latter would have open  
 enrollment.


 Andrei


I'd like to participate and prefer newsgroups to mailing lists. I think a  
separate newsgroup, although some work to setup, would make it easier to  
follow the discussions.
Dec 29 2009
prev sibling next sibling parent reply Nick B <"nick_NOSPAM_.barbalich" gmail.com> writes:
Andrei Alexandrescu wrote:

 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.
 


[snip]

 
 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to participate. 
 Please advise: (a) whether you would like to participate to the design; 
 (b) keep discussions on the general group; (c) create a separate 
 newsgroup; (d) create a mailing list. The latter would have open 
 enrollment.
 
 
 Andrei



Andrei

Will you be inviting Bartosz to participate, or have you already has 
discussions with him ? He has written a number of  blogs around this issue.

Nick B
Dec 29 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Nick B wrote:

 Andrei Alexandrescu wrote:

 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.


 [snip]

 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to 
 participate. Please advise: (a) whether you would like to participate 
 to the design; (b) keep discussions on the general group; (c) create a 
 separate newsgroup; (d) create a mailing list. The latter would have 
 open enrollment.


 Andrei


 
 
 Andrei
 
 Will you be inviting Bartosz to participate, or have you already has 
 discussions with him ? He has written a number of  blogs around this issue.
 
 Nick B


I'm not sure about Bartosz' and Walter's thoughts. I personally think 
that past experience suggests that collaboration would be difficult. No 
less than three attempts to work together on threads ranging from design 
to implementation have failed, in spite of everybody's best intentions.

Andrei
Dec 29 2009
parent reply Nick B <"nick_NOSPAM_.barbalich" gmail.com> writes:
Andrei Alexandrescu wrote:

 Nick B wrote:





 Andrei

 Will you be inviting Bartosz to participate, or have you already has 
 discussions with him ? He has written a number of  blogs around this 
 issue.

 Nick B


 
 I'm not sure about Bartosz' and Walter's thoughts. I personally think 
 that past experience suggests that collaboration would be difficult. No 
 less than three attempts to work together on threads ranging from design 
 to implementation have failed, in spite of everybody's best intentions.
 
 Andrei


Andrei

Very interesting to see that you and Walter tried three times !!

In Bartosz reply on 14 Oct 2009, [subject: Re: Revamped concurrency 
API], detailing his detachment from the D community, he gave this 
opinion: " I'm a bit of a perfectionist and it's hard for me to 
subscribe to the "good enough" philosophy (as long as it's better that 
C++, it's fine for D)".  If the three of you, can't agree on approach 
and architecture then I suspect, there is little to be said further.

All I can ask is:

1.  if you could re-read his blog posts on this subject.

2.   Can you explain further this comment (see above ref) by Bartosz:
" The semantics of "shared". I can live with postponing the 
implementation of the race-free type system, but not with the compiler 
inserting barriers around all shared reads and writes, even inside 
synchronized sections. "

3. Finally, is this a requirement to be finished for your book, or is 
this a separate issue ?

cheers
Nick B
Jan 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Nick B wrote:

 Andrei Alexandrescu wrote:

 Nick B wrote:




 

 Andrei

 Will you be inviting Bartosz to participate, or have you already has 
 discussions with him ? He has written a number of  blogs around this 
 issue.

 Nick B


 I'm not sure about Bartosz' and Walter's thoughts. I personally think 
 that past experience suggests that collaboration would be difficult. 
 No less than three attempts to work together on threads ranging from 
 design to implementation have failed, in spite of everybody's best 
 intentions.

 Andrei


 
 Andrei
 
 Very interesting to see that you and Walter tried three times !!


Bartosz too. We all tried, and if we failed I can say for sure it wasn't
for the lack of good intentions.


 In Bartosz reply on 14 Oct 2009, [subject: Re: Revamped concurrency 
 API], detailing his detachment from the D community, he gave this 
 opinion: " I'm a bit of a perfectionist and it's hard for me to 
 subscribe to the "good enough" philosophy (as long as it's better that 
 C++, it's fine for D)".  If the three of you, can't agree on approach 
 and architecture then I suspect, there is little to be said further.


There's more than one side to each story. My choice of words describing
the situation would be quite different, but please allow me to keep that
to myself. Let me just emphasize that the "good enough" philosophy and 
using C++ as the sole yardstick does not characterize D's development.


 All I can ask is:
 
 1.  if you could re-read his blog posts on this subject.


I am familiar with Bartosz's posts and with the work he is building on.


 2.   Can you explain further this comment (see above ref) by Bartosz:
 " The semantics of "shared". I can live with postponing the 
 implementation of the race-free type system, but not with the compiler 
 inserting barriers around all shared reads and writes, even inside 
 synchronized sections. "


A shared object allows synchronized calls. However, shared is deep and
the effect of synchronized is shallow, so there is a conflict. What
synchronized effects is a sort of "tail-shared". Walter's and my
suggestion was to have synchronized _not_ change the type of the object.
That way, the type of each field of the object remains shared-qualified
inside a synchronized method, which is correct but overly conservative.
In theory the compiler emits barriers whenever reading and writing
shared fields. However, inside a synchronized method, the compiler can
elide barriers because it understands shared is really tail-shared. Yet
there will still be more barriers than strictly necessary because e.g.
you could pass the address of a field to another function, which doesn't
realize the field is lock-protected. Walter and I deemed the situation
rare enough to not complicate the language with the likes of "tail-shared".


 3. Finally, is this a requirement to be finished for your book, or is 
 this a separate issue ?


If TDPL comes without addressing concurrency, or if if comes with
something, ahem, not-so-good a la Python, we may as well gather our toys
and go home.


Andrei
Jan 02 2010
next sibling parent reply Jason House <jason.james.house gmail.com> writes:
Andrei Alexandrescu Wrote:


 Nick B wrote:



 3. Finally, is this a requirement to be finished for your book, or is 
 this a separate issue ?


 
 If TDPL comes without addressing concurrency, or if if comes with
 something, ahem, not-so-good a la Python, we may as well gather our toys
 and go home.


The way things are going, I can't see things turning out any other way.
Everything involving concurrency/shared seems to get pushed off. This thread is
a recent example: asking for participants and then having no meaningful
follow-up. When I was part of the sausage-making emails you and Walter were
nearly silent, even in response to messages such as "can we at least agree to
X?"
Jan 02 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Jason House wrote:

 Andrei Alexandrescu Wrote:
 

 Nick B wrote:


 

 3. Finally, is this a requirement to be finished for your book,
 or is this a separate issue ?


 If TDPL comes without addressing concurrency, or if if comes with 
 something, ahem, not-so-good a la Python, we may as well gather our
 toys and go home.


 
 The way things are going, I can't see things turning out any other
 way. Everything involving concurrency/shared seems to get pushed off.


We already have a compelling message-passing model designed and to a 
large extent implemented by Sean Kelly, so you can discount that risk.


 This thread is a recent example: asking for participants and then
 having no meaningful follow-up. When I was part of the sausage-making
 emails you and Walter were nearly silent, even in response to
 messages such as "can we at least agree to X?"


There are a few clerical issue to take care of, and I'm not sure what 
Walter's preferred method of communication is. I will make you part of 
whatever transport we'll use.

I am sure that some questions weren't answered simply because there was 
no clear answer. Besides, email communication is imperfect.


Andrei
Jan 02 2010
prev sibling parent Nick B <"nick_NOSPAM_.barbalich" gmail.com> writes:
Andrei Alexandrescu wrote:


 

 In Bartosz reply on 14 Oct 2009, [subject: Re: Revamped concurrency 
 API], detailing his detachment from the D community, he gave this 
 opinion: " I'm a bit of a perfectionist and it's hard for me to 
 subscribe to the "good enough" philosophy (as long as it's better that 
 C++, it's fine for D)".  If the three of you, can't agree on approach 
 and architecture then I suspect, there is little to be said further.


 
 There's more than one side to each story. My choice of words describing
 the situation would be quite different, but please allow me to keep that
 to myself. Let me just emphasize that the "good enough" philosophy and 
 using C++ as the sole yardstick does not characterize D's development.


Agree,  everyone does have their own point of view.
[snip]

 

 2.   Can you explain further this comment (see above ref) by Bartosz:
 " The semantics of "shared". I can live with postponing the 
 implementation of the race-free type system, but not with the compiler 
 inserting barriers around all shared reads and writes, even inside 
 synchronized sections. "


 
 A shared object allows synchronized calls. However, shared is deep and
 the effect of synchronized is shallow, so there is a conflict. What
 synchronized effects is a sort of "tail-shared". Walter's and my
 suggestion was to have synchronized _not_ change the type of the object.
 That way, the type of each field of the object remains shared-qualified
 inside a synchronized method, which is correct but overly conservative.
 In theory the compiler emits barriers whenever reading and writing
 shared fields. However, inside a synchronized method, the compiler can
 elide barriers because it understands shared is really tail-shared. Yet
 there will still be more barriers than strictly necessary because e.g.
 you could pass the address of a field to another function, which doesn't
 realize the field is lock-protected. Walter and I deemed the situation
 rare enough to not complicate the language with the likes of "tail-shared".
 


Thank you for this clarification.



 3. Finally, is this a requirement to be finished for your book, or is 
 this a separate issue ?


 
 If TDPL comes without addressing concurrency, or if if comes with
 something, ahem, not-so-good a la Python, we may as well gather our toys
 and go home.
 


I am sure you and Walter will deliver something good and robust.

Nick B
Jan 02 2010
prev sibling parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
On Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:


 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.
 
 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.
 
 
 Andrei


I would love to be involved too.
Jan 03 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Graham St Jack wrote:

 On Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:
 

 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.

 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.


 Andrei


 
 I would love to be involved too.


Subscribe to dmd-concurrency from http://lists.puremagic.com/.

Andrei
Jan 03 2010