www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Does D have equivalent of Java util.concurrent?

reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
Java programmers have benefited hugely from Doug Lea's work 
(http://gee.cs.oswego.edu/dl/concurrency-interest/index.html). I 
was wondering if D's library has anything equivalent. If not, I 
am interested in porting Doug Lea's library from Java to D; I 
think the license allows this, but I will check with Doug Lea.
Apr 27
next sibling parent Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Saturday, 27 April 2019 at 14:15:01 UTC, Dibyendu Majumdar 
wrote:
 Java programmers have benefited hugely from Doug Lea's work 
 (http://gee.cs.oswego.edu/dl/concurrency-interest/index.html). 
 I was wondering if D's library has anything equivalent. If not, 
 I am interested in porting Doug Lea's library from Java to D; I 
 think the license allows this, but I will check with Doug Lea.
I checked with Doug - the core concurrent library code at his site is in the public domain so it can be freely ported to another language.
Apr 27
prev sibling next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Saturday, 27 April 2019 at 14:15:01 UTC, Dibyendu Majumdar 
wrote:
 Java programmers have benefited hugely from Doug Lea's work 
 (http://gee.cs.oswego.edu/dl/concurrency-interest/index.html). 
 I was wondering if D's library has anything equivalent. If not, 
 I am interested in porting Doug Lea's library from Java to D; I 
 think the license allows this, but I will check with Doug Lea.
Concurrent features included in Phobos are nicely written down here http://ddili.org/ders/d.en/concurrency.html http://ddili.org/ders/d.en/parallelism.html http://ddili.org/ders/d.en/concurrency_shared.html You may have a look to see wheter there is an overlap and what is missing. Kind regards Andre
Apr 27
parent reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Saturday, 27 April 2019 at 18:26:53 UTC, Andre Pany wrote:
 Concurrent features included in Phobos are nicely written down 
 here

 http://ddili.org/ders/d.en/concurrency.html
 http://ddili.org/ders/d.en/parallelism.html
 http://ddili.org/ders/d.en/concurrency_shared.html
That's interesting, thanks. I also found this: http://www.informit.com/articles/article.aspx?p=1609144 I need to understand the model used by D. As a systems language it should not be imposing a programming model but it appears to do so here.
Apr 27
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 28/04/2019 12:13 PM, Dibyendu Majumdar wrote:
 On Saturday, 27 April 2019 at 18:26:53 UTC, Andre Pany wrote:
 Concurrent features included in Phobos are nicely written down here

 http://ddili.org/ders/d.en/concurrency.html
 http://ddili.org/ders/d.en/parallelism.html
 http://ddili.org/ders/d.en/concurrency_shared.html
That's interesting, thanks. I also found this: http://www.informit.com/articles/article.aspx?p=1609144 I need to understand the model used by D. As a systems language it should not be imposing a programming model but it appears to do so here.
What model are you thinking it is imposing?
Apr 27
parent reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Sunday, 28 April 2019 at 03:54:43 UTC, rikki cattermole wrote:
 On 28/04/2019 12:13 PM, Dibyendu Majumdar wrote:
 On Saturday, 27 April 2019 at 18:26:53 UTC, Andre Pany wrote:
 Concurrent features included in Phobos are nicely written 
 down here

 http://ddili.org/ders/d.en/concurrency.html
 http://ddili.org/ders/d.en/parallelism.html
 http://ddili.org/ders/d.en/concurrency_shared.html
 I need to understand the model used by D. As a systems 
 language it should not be imposing a programming model but it 
 appears to do so here.
What model are you thinking it is imposing?
I have to understand the model. Suppose some mutable data is passed around between threads. I don't really know as I haven't read this in detail - but my question is this: is there a cost to accessing data across threads? As in C or C++ I would expect the cost to be zero unless marked volatile. I admit I don't understand the model of memory access in D. It seems that like Go, D is attempting to push a message passing model - that's what I meant by 'imposing a model'.
Apr 28
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 28/04/2019 8:34 PM, Dibyendu Majumdar wrote:
 I have to understand the model. Suppose some mutable data is passed 
 around between threads. I don't really know as I haven't read this in 
 detail - but my question is this: is there a cost to accessing data 
 across threads?
Same cost compared to C.
Apr 28
parent reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Sunday, 28 April 2019 at 08:47:06 UTC, rikki cattermole wrote:
 On 28/04/2019 8:34 PM, Dibyendu Majumdar wrote:
 I have to understand the model. Suppose some mutable data is 
 passed around between threads. I don't really know as I 
 haven't read this in detail - but my question is this: is 
 there a cost to accessing data across threads?
Same cost compared to C.
Is there a document that describes the rules regarding concurrent memory access, such as the Java Memory Model or C++ Memory Model (https://en.cppreference.com/w/cpp/language/memory_model)? Suppose some data needs to be shared between threads, and it is not global data. Does it need to be marked 'shared'? Reading Andrei's doc I got the impression that the 'shared' mark has a cost. Thanks
Apr 28
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 28/04/2019 9:05 PM, Dibyendu Majumdar wrote:
 On Sunday, 28 April 2019 at 08:47:06 UTC, rikki cattermole wrote:
 On 28/04/2019 8:34 PM, Dibyendu Majumdar wrote:
 I have to understand the model. Suppose some mutable data is passed 
 around between threads. I don't really know as I haven't read this in 
 detail - but my question is this: is there a cost to accessing data 
 across threads?
Same cost compared to C.
Is there a document that describes the rules regarding concurrent memory access, such as the Java Memory Model or C++ Memory Model (https://en.cppreference.com/w/cpp/language/memory_model)? Suppose some data needs to be shared between threads, and it is not global data. Does it need to be marked 'shared'? Reading Andrei's doc I got the impression that the 'shared' mark has a cost. Thanks
shared does nothing. There is strong desire to remove it, most people cast it away since its just a pain in the type system (ignoring its behavior of __gshared for globals). I think you're over thinking it. Yes you do need to use some form of memory synchronization strategy. No, the language does not force it. Yes the standard library does offer some solutions to this end. No, the language is by default unsafe and you're free to do anything you want including BSOD your computer. This is what native languages offer you.
Apr 28
parent Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Sunday, 28 April 2019 at 09:24:52 UTC, rikki cattermole wrote:
 shared does nothing.
 There is strong desire to remove it, most people cast it away 
 since its just a pain in the type system (ignoring its behavior 
 of __gshared for globals).

 I think you're over thinking it.

 Yes you do need to use some form of memory synchronization 
 strategy.
 No, the language does not force it.
 Yes the standard library does offer some solutions to this end.
 No, the language is by default unsafe and you're free to do 
 anything you want including BSOD your computer. This is what 
 native languages offer you.
Okay that's good )
Apr 28
prev sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Sun, 2019-04-28 at 08:34 +0000, Dibyendu Majumdar via Digitalmars-d wrot=
e:
 [=E2=80=A6]
=20
 I have to understand the model. Suppose some mutable data is=20
 passed around between threads. I don't really know as I haven't=20
 read this in detail - but my question is this: is there a cost to=20
 accessing data across threads?
It depends what you mean by passing here: passing by value, i.e. making a c= opy of the value and losing all reference to the original, can be fairly overhe= ad free; passing by reference always imposes some sort of overhead. This is fo= r all programming languages.
 As in C or C++ I would expect the cost to be zero unless marked=20
 volatile.
I think you need to offer some examples here. As far as I am aware C++ has overhead for dealing with mutable data between threads: unique_ptr and shared_ptr are not zero cost. Arguably Rust gets closest to zero cost by having a move by default and bor= row system that is checked at compile time. (For heap values referenced using a= n Arc instance there is a runtime cost of type and accessibility checking.) C= ++ hasn't quite got there as yet.
 I admit I don't understand the model of memory access in D. It=20
 seems that like Go, D is attempting to push a message passing=20
 model - that's what I meant by 'imposing a model'.
Message passing is the One True Way=E2=84=A2 of handling data in a parallel= or concurrent system =E2=80=93 obviously safety requires pass by value rather = than pass by reference. Go has this right in the language except that it allows passi= ng pointers. Rust mostly has this right in the language with the futures packa= ge (or equivalent in gtk-rs). D doesn't really have this right but is well on = the way. C++17 I haven't investigated properly, but C++14 was far, far, far fro= m doing the right thing, so I doubt C++17 has it right. So you meant memory model and not programming model. C++, D, Rust, Go all h= ave memory models, as Java, Kotlin, etc. have. Indeed a programming language th= at has concurrency and parallelism features in the language or standard librar= y that doesn't have a memory model has so many "undefined behaviours" as to b= e nigh on useless. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Apr 28
next sibling parent Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Sunday, 28 April 2019 at 09:03:00 UTC, Russel Winder wrote:

 So you meant memory model and not programming model. C++, D, 
 Rust, Go all have memory models, as Java, Kotlin, etc. have. 
 Indeed a programming language that has concurrency and 
 parallelism features in the language or standard library that 
 doesn't have a memory model has so many "undefined behaviours" 
 as to be nigh on useless.
I meant both. By programming model I meant the idea that a language pushes for certain type of programming. Go is a good example. It cannot be called a systems language (I believe it no longer is).
Apr 28
prev sibling next sibling parent reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Sunday, 28 April 2019 at 09:03:00 UTC, Russel Winder wrote:
 On Sun, 2019-04-28 at 08:34 +0000, Dibyendu Majumdar via 
 Digitalmars-d wrote:
 […]
 
 I have to understand the model. Suppose some mutable data is 
 passed around between threads. I don't really know as I 
 haven't read this in detail - but my question is this: is 
 there a cost to accessing data across threads?
It depends what you mean by passing here: passing by value, i.e. making a copy of the value and losing all reference to the original, can be fairly overhead free; passing by reference always imposes some sort of overhead. This is for all programming languages.
 As in C or C++ I would expect the cost to be zero unless 
 marked volatile.
I think you need to offer some examples here. As far as I am aware C++ has overhead for dealing with mutable data between threads: unique_ptr and shared_ptr are not zero cost.
I wasn't talking about the cost of passing data, more about the cost of operations with this. Say we have something like this. SomeFunc(data) various ops with data return data There should not be any extra overhead in the 'ops' part above. In C++ if data was marked volatile, then there would be an impact, but not otherwise, apart from the standard optimisation rules to do with data escaping, or aliasing something.
Apr 28
parent Russel Winder <russel winder.org.uk> writes:
On Sun, 2019-04-28 at 09:21 +0000, Dibyendu Majumdar via Digitalmars-d wrot=
e:
=20
[=E2=80=A6]
 I wasn't talking about the cost of passing data, more about the=20
 cost of operations with this.
=20
 Say we have something like this.
=20
=20
 SomeFunc(data)
      various ops with data
      return data
=20
 There should not be any extra overhead in the 'ops' part above.=20
 In C++ if data was marked volatile, then there would be an=20
 impact, but not otherwise, apart from the standard optimisation=20
 rules to do with data escaping, or aliasing something.
In a single threaded context this is entirely right. In a multi-threaded context then it is very likely that the "various ops with data" would be wrapped with a std::mutex or similar. The overhead is before and after the "various ops with data", the ops themselves should not have any overhead. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
May 22
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 28 April 2019 at 09:03:00 UTC, Russel Winder wrote:
 I think you need to offer some examples here. As far as I am 
 aware C++ has overhead for dealing with mutable data between 
 threads: unique_ptr and shared_ptr are not zero cost.
A bit late reply, but unique_ptr is near zero cost, however, neither are thread safe. For that you need to wrap the smart-pointer like std::atomic<std::shared_ptr<T>> (C++20). Anyway, I'd argue that irrespective of language you might as well consider implementing your own shared store using CAS instructions and avoid locking altogether. If it fits what you want to do. In D's typesystem that means that only the encapsulating datastructure would be shared and all the nodes would be nonshared (you acquire them using CAS).
May 17
prev sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Sun, 2019-04-28 at 00:13 +0000, Dibyendu Majumdar via Digitalmars-d wrot=
e:
 On Saturday, 27 April 2019 at 18:26:53 UTC, Andre Pany wrote:
 Concurrent features included in Phobos are nicely written down=20
 here
=20
 http://ddili.org/ders/d.en/concurrency.html
 http://ddili.org/ders/d.en/parallelism.html
 http://ddili.org/ders/d.en/concurrency_shared.html
=20
std::parallelism uses a threadpool and tasks approach and provide paralleli= sm features harmonious with the D programming model. The ForkJoin model at the heart of Doug's (and lots of other people now) framework is not the only concurrency and parallelism tool in JDK, Streams make use of the framework underneath but they provide a lot that has nothin= g to do with the framework. It could be argued that there is nothing in the JDK that cannot already be done in D, that there is no point in porting the ForkJoin framework to D si= nce it would provide no new facility, that everything the port of ForkJoin woul= d offer is already available in D.
=20
[=E2=80=A6]
=20
 I need to understand the model used by D. As a systems language=20
 it should not be imposing a programming model but it appears to=20
 do so here.
I do not understand this final sentence. All programming language from assembly language, C, C++, D, Rust, Go (all of which claim to be systems programming languages) provide a programming model. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
Apr 28
parent reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Sunday, 28 April 2019 at 08:05:48 UTC, Russel Winder wrote:

 The ForkJoin model at the heart of Doug's (and lots of other 
 people now) framework is not the only concurrency and 
 parallelism tool in JDK, Streams make use of the framework 
 underneath but they provide a lot that has nothing to do with 
 the framework.
That's not true - it isn't the heart of the concurrency library - in fact it came later.
 It could be argued that there is nothing in the JDK that cannot 
 already be done in D, that there is no point in porting the 
 ForkJoin framework to D since it would provide no new facility, 
 that everything the port of ForkJoin would offer is already 
 available in D.
I have not used the ForkJoin framework much - so I don't know what benefits it may or may not have over D's threadpool. I read it has something called work stealing but that's the limit of my knowledge about that. The main advantages of Java's library are lock free / concurrent data structures and synchronisation methods that are high performance.
 
[…]
 
 I need to understand the model used by D. As a systems 
 language it should not be imposing a programming model but it 
 appears to do so here.
I do not understand this final sentence. All programming language from assembly language, C, C++, D, Rust, Go (all of which claim to be systems programming languages) provide a programming model.
My question is: is there a cost to accessing data across threads? The document I was reading (from Andrei's book) appeared to imply there is. I would expect the cost to be zero unless some data was marked volatile as in C or C++. Maybe I misunderstood and this is the case.
Apr 28
parent Russel Winder <russel winder.org.uk> writes:
On Sun, 2019-04-28 at 08:40 +0000, Dibyendu Majumdar via Digitalmars-d wrot=
e:
 On Sunday, 28 April 2019 at 08:05:48 UTC, Russel Winder wrote:
=20
 The ForkJoin model at the heart of Doug's (and lots of other=20
 people now) framework is not the only concurrency and=20
 parallelism tool in JDK, Streams make use of the framework=20
 underneath but they provide a lot that has nothing to do with=20
 the framework.
=20 That's not true - it isn't the heart of the concurrency library -=20 in fact it came later.
I am sure we can have endless debates on the history of concurrency and parallelism in Java, threads, JSR166, memory model, what came when, etc. an= d what is now in java.util.concurrent. Whilst I was present all the way throu= gh it, memory of history is never exact. And of course, in this context, it is irrelevant. What matters is that D is missing something that means std.concurrency and std.parallelism need review and integration, and possib= ly extension.=20 =20
 It could be argued that there is nothing in the JDK that cannot=20
 already be done in D, that there is no point in porting the=20
 ForkJoin framework to D since it would provide no new facility,=20
 that everything the port of ForkJoin would offer is already=20
 available in D.
=20
=20 I have not used the ForkJoin framework much - so I don't know=20 what benefits it may or may not have over D's threadpool. I read=20 it has something called work stealing but that's the limit of my=20 knowledge about that.
Java's Fork/Join Framework (based on Doug Lea's work) does indeed do work stealing. If I remember correctly the threadpool in std.parallelism also implements work stealing, which is why the threadpool needs to be opaque. However, std.parallelism is not a system analogous to Fork/Join. Fork/Join = is a concurrency/parallelism architecture that D doesn't have support for. Hen= ce some people are looking to a port of java.util.concurrency and various othe= r bits. I believe this will replicate a lot of what is in std.parallelism.= =20
 The main advantages of Java's library are lock free / concurrent=20
 data structures and synchronisation methods that are high=20
 performance.
As far as I am aware only a few of the concurrent data structures in the Ja= va standard library are fully lock free. I could be wrong though, I haven't us= ed things from java.util.concurrent in a long while.=20 [=E2=80=A6]
=20
 My question is: is there a cost to accessing data across threads?=20
 The document I was reading (from Andrei's book) appeared to imply=20
 there is. I would expect the cost to be zero unless some data was=20
 marked volatile as in C or C++. Maybe I misunderstood and this is=20
 the case.
=20
I am not an expert in the D data structures, but there are some on this lis= t who are. Having said that, in general, accessing data across threads genera= lly requires some form of overhead. This is part of why Go obsesses over sharin= g data by sending it over channels.=20 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
May 22
prev sibling parent reply Heromyth <bitworld qq.com> writes:
On Saturday, 27 April 2019 at 14:15:01 UTC, Dibyendu Majumdar 
wrote:
 Java programmers have benefited hugely from Doug Lea's work 
 (http://gee.cs.oswego.edu/dl/concurrency-interest/index.html). 
 I was wondering if D's library has anything equivalent. If not, 
 I am interested in porting Doug Lea's library from Java to D; I 
 think the license allows this, but I will check with Doug Lea.
We have done somthing about this. See here: Source: https://github.com/huntlabs/hunt/tree/master/source/hunt/concurrency Examples: https://github.com/huntlabs/hunt/tree/master/examples/UnitTest/source/test Current stage: We are trying to port the CompletableFuture.
Apr 27
parent reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Sunday, 28 April 2019 at 01:22:32 UTC, Heromyth wrote:
 On Saturday, 27 April 2019 at 14:15:01 UTC, Dibyendu Majumdar 
 wrote:
 Java programmers have benefited hugely from Doug Lea's work 
 (http://gee.cs.oswego.edu/dl/concurrency-interest/index.html). 
 I was wondering if D's library has anything equivalent. If 
 not, I am interested in porting Doug Lea's library from Java 
 to D; I think the license allows this, but I will check with 
 Doug Lea.
We have done somthing about this. See here: Source: https://github.com/huntlabs/hunt/tree/master/source/hunt/concurrency Examples: https://github.com/huntlabs/hunt/tree/master/examples/UnitTest/source/test Current stage: We are trying to port the CompletableFuture.
Nice. Some questions: Have you based this on the Java8 version from Doug Lea's repository? Presumably this isn't a port of Oracle's version? The concurrent collections aren't ported, am I right? BTW Doug Lea's copyright notice should be retained in the sources if you have ported his code.
Apr 28
parent reply Heromyth <bitworld qq.com> writes:
On Sunday, 28 April 2019 at 08:27:31 UTC, Dibyendu Majumdar wrote:
 On Sunday, 28 April 2019 at 01:22:32 UTC, Heromyth wrote:
 On Saturday, 27 April 2019 at 14:15:01 UTC, Dibyendu Majumdar 
 wrote:
 Java programmers have benefited hugely from Doug Lea's work 
 (http://gee.cs.oswego.edu/dl/concurrency-interest/index.html). I was wondering
if D's library has anything equivalent. If not, I am interested in porting Doug
Lea's library from Java to D; I think the license allows this, but I will check
with Doug Lea.
We have done somthing about this. See here: Source: https://github.com/huntlabs/hunt/tree/master/source/hunt/concurrency Examples: https://github.com/huntlabs/hunt/tree/master/examples/UnitTest/source/test Current stage: We are trying to port the CompletableFuture.
Nice. Some questions: Have you based this on the Java8 version from Doug Lea's repository? Presumably this isn't a port of Oracle's version? The concurrent collections aren't ported, am I right? BTW Doug Lea's copyright notice should be retained in the sources if you have ported his code.
The most copyright notices are retained. All the code are porting from OpenJDK 8/11. The porting of CompletableFuture done. The 20 examples from [1] are passed, see [2]. The next step is make CompletableFuture support value type like int, string etc. [1] https://mahmoudanouti.wordpress.com/2018/01/26/20-examples-of-using-javas-completablefuture/ [2] https://github.com/huntlabs/hunt/blob/master/examples/UnitTest/source/test/CompletableFutureTest.d
May 16
parent Tony <tonytdominguez aol.com> writes:
On Friday, 17 May 2019 at 06:36:05 UTC, Heromyth wrote:

 The most copyright notices are retained. All the code are 
 porting from OpenJDK 8/11.
I have wondered what the copyright requirements are in porting a library from one language to another. I notice that for String.java http://www.docjar.com/html/api/java/lang/String.java.html to String.d, https://github.com/huntlabs/hunt/blob/master/source/hunt/String.d you have removed the copyright notices of Oracle while trying to maintain some amount of their API. And also retaining their comments, for instance: /** * The { code String} class represents character strings. All * string literals in Java programs, such as { code "abc"}, are * implemented as instances of this class. * <p> * Strings are constant; their values cannot be changed after they * are created. String buffers support mutable strings. * Because String objects are immutable they can be shared. For example: In asking about a port like this, people have claimed that the Oracle copyright should be in your source files.
May 17