www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [OT] Senders and Receivers

reply Sergey <kornburn yandex.ru> writes:
Wanted to share with everyone series of podcasts about the topic.
The topic is quite complicated, so maybe this helps you to better 
understand what is it all about.
They discussing it mostly from C++ perspective, with some Haskell 
ideas.

D package with this approach: 
https://code.dlang.org/packages/concurrency

Podcast series:
https://adspthepodcast.com/2025/04/25/Episode-231.html
https://adspthepodcast.com/2025/05/23/Episode-235.html
https://adspthepodcast.com/2025/05/30/Episode-236.html
Jun 02
next sibling parent reply Derek Fawcus <dfawcus+dlang employees.org> writes:
On Monday, 2 June 2025 at 19:22:47 UTC, Sergey wrote:

 Wanted to share with everyone series of podcasts about the 
 topic.
 The topic is quite complicated, so maybe this helps you to 
 better understand what is it all about.
 They discussing it mostly from C++ perspective, with some 
 Haskell ideas.

 D package with this approach: 
 https://code.dlang.org/packages/concurrency

 Podcast series:
 https://adspthepodcast.com/2025/04/25/Episode-231.html
 https://adspthepodcast.com/2025/05/23/Episode-235.html
 https://adspthepodcast.com/2025/05/30/Episode-236.html


Why should "Structured Concurrency" be viewed as a better 
approach than CSP (or Actors for that matter)?

DF
Jun 02
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 2 June 2025 at 19:32:03 UTC, Derek Fawcus wrote:

 Why should "Structured Concurrency" be viewed as a better 
 approach than CSP (or Actors for that matter)?


Here's the most popular explanation:

https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/

The main thesis (quoted from the article's conclusion) is this:


 These [unstructured concurrency] primitives are dangerous even 
 if we don't use them directly, because they undermine our 
 ability to reason about control flow and compose complex 
 systems out of abstract modular parts, and they interfere with 
 useful language features like automatic resource cleanup and 
 error propagation.
Jun 02
next sibling parent Derek Fawcus <dfawcus+dlang employees.org> writes:
On Tuesday, 3 June 2025 at 06:06:24 UTC, Paul Backus wrote:

 On Monday, 2 June 2025 at 19:32:03 UTC, Derek Fawcus wrote:

 Why should "Structured Concurrency" be viewed as a better 
 approach than CSP (or Actors for that matter)?


 Here's the most popular explanation:

 https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/

 The main thesis (quoted from the article's conclusion) is this:


 These [unstructured concurrency] primitives are dangerous even 
 if we don't use them directly, because they undermine our 
 ability to reason about control flow and compose complex 
 systems out of abstract modular parts, and they interfere with 
 useful language features like automatic resource cleanup and 
 error propagation.




Thanks.  I'll have a read of that later.

The flaw I've seen to date in the arguments against CSP (and 
Actors) in favour of 'Structured Concurrency', but which I do 
view as valid against raw threads, async/await type abstractions 
is the scope of application.

The way I approach CSP and Actors is to apply them to 'large' 
chunks of natural concurrency, and hence generally avoid shared 
data access.

Whereas the raw threads + locks, async/await and friends and 
implicitly 'Structured Concurrency' seem to be targetting 'fine 
grained' conncurrency decomposition, even to the level of 
individual functions calls.  This I view as inherently difficult 
to reason about, and possibly SC then provides some form of aid 
in reasoning.

Now one could use CSP and Actors in a similar way, and there they 
would likewise be equally difficult to reason about, but I 
suggest they tend to encourage different and higher level form of 
decomposition.

There the same reasoning issues don't arise, and moreover there 
are tools one to apply to prove the network/graph one has for 
CSP. (I'm not sure if the same applies for Actors).

The obvious trade off which one makes when using CSP is the risk 
of deadlock.

However the above tools are supposed to avoid that. I've not yet 
tried them, since I've yet to hit/create a sufficiently difficult 
graph which challenges manual analysis.  However it may still be 
worth using such tools if the graph may change under long term 
support of the program.

Even without such use I find that with sufficient logging (which 
is not really a lot) it is easy to reason through any remaining 
deadlocks once the arise during testing. Whereas it is incredibly 
difficult to reason through the trigger cause with shared data 
update (even under locks) when it occurs from multiple concurrent 
call graphs, which would happen under the other methodologies 
(including, I believe, SC).


So I generally see SC as tackling the wrong problem, however I 
shall give your reference a fair crack of the whip.

(I listened to the first podcast above; I've yet to get to the 
other two)
Jun 03
prev sibling parent reply =?UTF-8?Q?Ali_=C3=87ehreli?= <acehreli yahoo.com> writes:
On 6/2/25 11:06 PM, Paul Backus wrote:


 The main thesis (quoted from the article's conclusion) is this:


 These [unstructured concurrency] primitives are dangerous even if we
 don't use them directly, because they undermine our ability to reason
 about control flow and compose complex systems out of abstract modular
 parts, and they interfere with useful language features like automatic
 resource cleanup and error propagation.




The garbage collector wins in that scenario because it removes the need 
for "automatic resource cleanup" e.g. of C++.

That fact was pointed out to me by a colleague years ago: The sender 
threads generate messages, the receiver threads use those messages, and 
thanks to the garbage collector, nobody cares about the ownership of 
messages.

Ali
Jun 04
next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 4 June 2025 at 19:59:27 UTC, Ali Çehreli wrote:

 On 6/2/25 11:06 PM, Paul Backus wrote:


 The main thesis (quoted from the article's conclusion) is


 this:

 These [unstructured concurrency] primitives are dangerous




 even if we

 don't use them directly, because they undermine our ability




 to reason

 about control flow and compose complex systems out of




 abstract modular

 parts, and they interfere with useful language features like




 automatic

 resource cleanup and error propagation.




 The garbage collector wins in that scenario because it removes 
 the need for "automatic resource cleanup" e.g. of C++.

 That fact was pointed out to me by a colleague years ago: The 
 sender threads generate messages, the receiver threads use 
 those messages, and thanks to the garbage collector, nobody 
 cares about the ownership of messages.

 Ali


It is not the lifetimes of the messages that is interesting, it's 
that of the producer and consumer themselves instead.

This is because they are different sides of the same coin; one 
without the other wouldn't work. The essential point is thus to 
model that relationship correctly - to ensure their lifetimes are 
coupled and avoid one going off without the other. Getting that 
guarantee is the big benefit of SC. Just look at the link I 
posted and read the papers it links to. Almost all those bugs are 
variations of mishaps in ensuring that invariant manually, not a 
good way to go about it.

Now, it must be said, the GC does help a lot in lock-free 
algorithms. Just look into hazard pointers to see what madness 
you have to resort to without a GC.
Jun 04
parent reply Araq <rumpf_a web.de> writes:
On Wednesday, 4 June 2025 at 21:48:07 UTC, Sebastiaan Koppe wrote:

 Now, it must be said, the GC does help a lot in lock-free 
 algorithms. Just look into hazard pointers to see what madness 
 you have to resort to without a GC.


It's not a big deal, you can use a library for it (C++ is getting 
it in its stdlib: 
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p1121r3.pdf) just like
your GC which lives in the code space you didn't write yourself.
Jun 05
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 5 June 2025 at 08:00:35 UTC, Araq wrote:

 On Wednesday, 4 June 2025 at 21:48:07 UTC, Sebastiaan Koppe 
 wrote:

 Now, it must be said, the GC does help a lot in lock-free 
 algorithms. Just look into hazard pointers to see what madness 
 you have to resort to without a GC.


 It's not a big deal, you can use a library for it (C++ is 
 getting it in its stdlib: 
 https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p1121r3.pdf) just
like your GC which lives in the code space you didn't write yourself.


There is probably a parallel universe where I did actually end up 
writing a GC for D.
Jun 05
parent reply Araq <rumpf_a web.de> writes:
On Thursday, 5 June 2025 at 14:30:09 UTC, Sebastiaan Koppe wrote:

 There is probably a parallel universe where I did actually end 
 up writing a GC for D.


On Thursday, 5 June 2025 at 14:30:09 UTC, Sebastiaan Koppe wrote:

I fail to see the point. So write a hazard pointer implementation 
too and see what is easier.
Jun 05
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 5 June 2025 at 14:41:09 UTC, Araq wrote:

 On Thursday, 5 June 2025 at 14:30:09 UTC, Sebastiaan Koppe 
 wrote:

 There is probably a parallel universe where I did actually end 
 up writing a GC for D.


 On Thursday, 5 June 2025 at 14:30:09 UTC, Sebastiaan Koppe 
 wrote:

 I fail to see the point. So write a hazard pointer 
 implementation too and see what is easier.


My point was that I am grateful there is a GC and that I don't 
have to write a hazard pointer library because of it.

It would be a bit odd to follow that up by implementing one, so 
I'm not going to. Maybe in the nogc-verse though ;)
Jun 05
prev sibling parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 4 June 2025 at 19:59:27 UTC, Ali Çehreli wrote:

 On 6/2/25 11:06 PM, Paul Backus wrote:


 The main thesis (quoted from the article's conclusion) is


 this:

 These [unstructured concurrency] primitives are dangerous




 even if we

 don't use them directly, because they undermine our ability




 to reason

 about control flow and compose complex systems out of




 abstract modular

 parts, and they interfere with useful language features like




 automatic

 resource cleanup and error propagation.




 The garbage collector wins in that scenario because it removes 
 the need for "automatic resource cleanup" e.g. of C++.

 That fact was pointed out to me by a colleague years ago: The 
 sender threads generate messages, the receiver threads use 
 those messages, and thanks to the garbage collector, nobody 
 cares about the ownership of messages.


The linked article mostly talks about Python and Go, which are 
both garbage-collected languages. The "resources" referred to 
here are things like open files and network connections, which 
are not reclaimed by the GC.
Jun 04
prev sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 2 June 2025 at 19:32:03 UTC, Derek Fawcus wrote:

 On Monday, 2 June 2025 at 19:22:47 UTC, Sergey wrote:

 Wanted to share with everyone series of podcasts about the 
 topic.
 The topic is quite complicated, so maybe this helps you to 
 better understand what is it all about.
 They discussing it mostly from C++ perspective, with some 
 Haskell ideas.

 D package with this approach: 
 https://code.dlang.org/packages/concurrency

 Podcast series:
 https://adspthepodcast.com/2025/04/25/Episode-231.html
 https://adspthepodcast.com/2025/05/23/Episode-235.html
 https://adspthepodcast.com/2025/05/30/Episode-236.html


 Why should "Structured Concurrency" be viewed as a better 
 approach than CSP (or Actors for that matter)?

 DF


Some more reading: https://skoppe.github.io/dconf-2022/6

In short: "Structured concurrency organizes concurrent tasks as 
tree-like hierarchies. This means that any child task is 
logically nested within its parent. When a parent task finishes, 
it waits for all its child tasks to complete or is automatically 
cancelled if the parent task fails."

This tree-like hierarchy achieves 2 important things:

- It improves reasoning of the code, like structured programming 
did;
- It ensures lifetimes of concurrent tasks follow the same 
structure, instead of the fire-and-forget approach.
Jun 03
prev sibling next sibling parent reply f <f abc.com> writes:
On Monday, 2 June 2025 at 19:22:47 UTC, Sergey wrote:

 Wanted to share with everyone series of podcasts about the 
 topic.
 The topic is quite complicated, so maybe this helps you to 
 better understand what is it all about.
 They discussing it mostly from C++ perspective, with some 
 Haskell ideas.

 D package with this approach: 
 https://code.dlang.org/packages/concurrency

 Podcast series:
 https://adspthepodcast.com/2025/04/25/Episode-231.html
 https://adspthepodcast.com/2025/05/23/Episode-235.html
 https://adspthepodcast.com/2025/05/30/Episode-236.html


another OT idea . perhaps D needs some kind of 2nd tier package 
bundle or even 3rd tier package. I mean that this kind of package 
must be somewhat impoertant that it gets downloaded for 137217  
times.

Otherwise it get lost somewhere in the forum, and int 
code.dlang.org, never be found and be used. Especially for newbie 
. Like me.

So maybe the download has some extra batteries
Jun 03
parent monkyyy <crazymonkyyy gmail.com> writes:
On Tuesday, 3 June 2025 at 16:19:50 UTC, f wrote:

 
 another OT idea . perhaps D needs some kind of 2nd tier package 
 bundle


come join the dark side of opend
Jun 03
prev sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 2 June 2025 at 19:22:47 UTC, Sergey wrote:

 Wanted to share with everyone series of podcasts about the 
 topic.
 The topic is quite complicated, so maybe this helps you to 
 better understand what is it all about.
 They discussing it mostly from C++ perspective, with some 
 Haskell ideas.

 D package with this approach: 
 https://code.dlang.org/packages/concurrency

 Podcast series:
 https://adspthepodcast.com/2025/04/25/Episode-231.html
 https://adspthepodcast.com/2025/05/23/Episode-235.html
 https://adspthepodcast.com/2025/05/30/Episode-236.html


Implementer here. Thanks for the link and ama.
Jun 03