www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How you guys go about -BetterC Multithreading?

reply ParticlePeter <ParticlePeter gmx.de> writes:
Any experience reports or general suggestions?
I've used only D threads so far.
Nov 09 2017
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter wrote:

 Any experience reports or general suggestions?
 I've used only D threads so far.


It would be far easier if you use druntime +  nogc and/or 
de-register latency-sensitive threads from druntime [1], so 
they're not interrupted even if some other thread calls the GC. 
Probably the path of least resistance is to call [2] and queue 
 nogc tasks on [3].

If you really want to pursue the version(D_BetterC) route, then 
you're essentially on your own to use the threading facilities 
provided by your target OS, e.g.:

https://linux.die.net/man/3/pthread_create
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx

Though you need to be extra careful not to use thread-local 
storage (e.g. only shared static and __gshared) and not to rely 
on (shared) static {con|de}structors, dynamic arrays, associative 
arrays, exceptions, classes, RAII, etc., which is really not 
worth it, unless you're writing very low-level code (e.g. OS 
kernels and drivers).

[1]: https://dlang.org/phobos/core_thread#.thread_detachThis
[2]: https://dlang.org/phobos/core_memory#.GC.disable
[3]: https://dlang.org/phobos/std_parallelism#.taskPool
Nov 09 2017
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 09/11/2017 12:19 PM, Petar Kirov [ZombineDev] wrote:

 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter wrote:

 Any experience reports or general suggestions?
 I've used only D threads so far.


 
 It would be far easier if you use druntime +  nogc and/or de-register 
 latency-sensitive threads from druntime [1], so they're not interrupted 
 even if some other thread calls the GC. Probably the path of least 
 resistance is to call [2] and queue  nogc tasks on [3].
 
 If you really want to pursue the version(D_BetterC) route, then you're 
 essentially on your own to use the threading facilities provided by your 
 target OS, e.g.:
 
 https://linux.die.net/man/3/pthread_create
 https://msdn.microsoft.com/en-us/library/windows/desktop/ms
82516(v=vs.85).aspx 


You can use a library like libuv to handle threads (non-language based 
TLS too, not sure that it can be tied in unfortunately).
Nov 09 2017
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 12:30:49 UTC, rikki cattermole 
wrote:

 On 09/11/2017 12:19 PM, Petar Kirov [ZombineDev] wrote:

 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter 
 wrote:

 Any experience reports or general suggestions?
 I've used only D threads so far.


 
 It would be far easier if you use druntime +  nogc and/or 
 de-register latency-sensitive threads from druntime [1], so 
 they're not interrupted even if some other thread calls the 
 GC. Probably the path of least resistance is to call [2] and 
 queue  nogc tasks on [3].
 
 If you really want to pursue the version(D_BetterC) route, 
 then you're essentially on your own to use the threading 
 facilities provided by your target OS, e.g.:
 
 https://linux.die.net/man/3/pthread_create
 https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx


 You can use a library like libuv to handle threads 
 (non-language based TLS too, not sure that it can be tied in 
 unfortunately).


Yeah, any cross-platform thread-pool / event loop library with C 
interface should obviously be preferred than manual use of raw 
thread primitives.

Essentially, try to follow Sean Parent's advice on "No 
Raw/Incidental *":
https://www.youtube.com/watch?v=zULU6Hhp42w
Nov 09 2017
parent ParticlePeter <ParticlePeter gmx.de> writes:
On Thursday, 9 November 2017 at 12:43:54 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Thursday, 9 November 2017 at 12:30:49 UTC, rikki cattermole 
 wrote:

 On 09/11/2017 12:19 PM, Petar Kirov [ZombineDev] wrote:

 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter 
 wrote:

 Any experience reports or general suggestions?
 I've used only D threads so far.


 
 It would be far easier if you use druntime +  nogc and/or 
 de-register latency-sensitive threads from druntime [1], so 
 they're not interrupted even if some other thread calls the 
 GC. Probably the path of least resistance is to call [2] and 
 queue  nogc tasks on [3].
 
 If you really want to pursue the version(D_BetterC) route, 
 then you're essentially on your own to use the threading 
 facilities provided by your target OS, e.g.:
 
 https://linux.die.net/man/3/pthread_create
 https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx


 You can use a library like libuv to handle threads 
 (non-language based TLS too, not sure that it can be tied in 
 unfortunately).


 Yeah, any cross-platform thread-pool / event loop library with 
 C interface should obviously be preferred than manual use of 
 raw thread primitives.

 Essentially, try to follow Sean Parent's advice on "No 
 Raw/Incidental *":
 https://www.youtube.com/watch?v=zULU6Hhp42w


This all is good input, thanks.
I was looking into:
https://github.com/GerHobbelt/pthread-win32

Anyone used this?
Nov 09 2017
prev sibling next sibling parent reply ParticlePeter <ParticlePeter gmx.de> writes:
On Thursday, 9 November 2017 at 12:19:00 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter 
 wrote:

 Any experience reports or general suggestions?
 I've used only D threads so far.


 It would be far easier if you use druntime +  nogc and/or 
 de-register latency-sensitive threads from druntime [1], so 
 they're not interrupted even if some other thread calls the GC. 
 Probably the path of least resistance is to call [2] and queue 
  nogc tasks on [3].

 If you really want to pursue the version(D_BetterC) route, then 
 you're essentially on your own to use the threading facilities 
 provided by your target OS, e.g.:

 https://linux.die.net/man/3/pthread_create
 https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx

 Though you need to be extra careful not to use thread-local 
 storage (e.g. only shared static and __gshared) and not to rely 
 on (shared) static {con|de}structors, dynamic arrays, 
 associative arrays, exceptions, classes, RAII, etc., which is 
 really not worth it, unless you're writing very low-level code 
 (e.g. OS kernels and drivers).

 [1]: https://dlang.org/phobos/core_thread#.thread_detachThis
 [2]: https://dlang.org/phobos/core_memory#.GC.disable
 [3]: https://dlang.org/phobos/std_parallelism#.taskPool


Forgot to mention, I'll try this first, I think its a good first 
step towards -BetterC usage. But in the end I want to see how far 
I can get with the -BetterC feature.
Nov 09 2017
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 13:00:15 UTC, ParticlePeter wrote:

 On Thursday, 9 November 2017 at 12:19:00 UTC, Petar Kirov 
 [ZombineDev] wrote:

 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter 
 wrote:

 Any experience reports or general suggestions?
 I've used only D threads so far.


 It would be far easier if you use druntime +  nogc and/or 
 de-register latency-sensitive threads from druntime [1], so 
 they're not interrupted even if some other thread calls the 
 GC. Probably the path of least resistance is to call [2] and 
 queue  nogc tasks on [3].

 If you really want to pursue the version(D_BetterC) route, 
 then you're essentially on your own to use the threading 
 facilities provided by your target OS, e.g.:

 https://linux.die.net/man/3/pthread_create
 https://msdn.microsoft.com/en-us/library/windows/desktop/ms682516(v=vs.85).aspx

 Though you need to be extra careful not to use thread-local 
 storage (e.g. only shared static and __gshared) and not to 
 rely on (shared) static {con|de}structors, dynamic arrays, 
 associative arrays, exceptions, classes, RAII, etc., which is 
 really not worth it, unless you're writing very low-level code 
 (e.g. OS kernels and drivers).

 [1]: https://dlang.org/phobos/core_thread#.thread_detachThis
 [2]: https://dlang.org/phobos/core_memory#.GC.disable
 [3]: https://dlang.org/phobos/std_parallelism#.taskPool


 Forgot to mention, I'll try this first, I think its a good 
 first step towards -BetterC usage. But in the end I want to see 
 how far I can get with the -BetterC feature.


In short, the cost / benefit of going all the way 
version(D_BetterC) is incredibly poor for regular applications, 
as you end up a bit more limited than with modern C++ (> 11) for 
prototyping. For example, even writers of D real-time audio 
plugins don't go as far.
If you're writing libraries, especially math-heavy template code, 
CTFE and generic code in general, then version(D_BetterC) is a 
useful tool for verifying that your library doesn't need 
unnecessary dependencies preventing it from being trivially 
integrated in foreign language environments.

Well if you like generic code as much as I do, you can surely do 
great with version(D_BetterC) even for application code, but you 
would have to make alomst every non-builtin type that you use in 
your code a template parameter (or alternatively an extern 
(C++/COM) interface if that works in -betterC), so you can easily 
swap druntime/phobos-based implementations for your custom ones, 
one by one, but I guess few people would be interested in 
following this path.
Nov 09 2017
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 09/11/2017 4:00 PM, Petar Kirov [ZombineDev] wrote:

 On Thursday, 9 November 2017 at 13:00:15 UTC, ParticlePeter wrote:

 On Thursday, 9 November 2017 at 12:19:00 UTC, Petar Kirov [ZombineDev] 
 wrote:

 On Thursday, 9 November 2017 at 11:08:21 UTC, ParticlePeter wrote:

 Any experience reports or general suggestions?
 I've used only D threads so far.


 It would be far easier if you use druntime +  nogc and/or de-register 
 latency-sensitive threads from druntime [1], so they're not 
 interrupted even if some other thread calls the GC. Probably the path 
 of least resistance is to call [2] and queue  nogc tasks on [3].

 If you really want to pursue the version(D_BetterC) route, then 
 you're essentially on your own to use the threading facilities 
 provided by your target OS, e.g.:

 https://linux.die.net/man/3/pthread_create
 https://msdn.microsoft.com/en-us/library/windows/desktop/ms
82516(v=vs.85).aspx 


 Though you need to be extra careful not to use thread-local storage 
 (e.g. only shared static and __gshared) and not to rely on (shared) 
 static {con|de}structors, dynamic arrays, associative arrays, 
 exceptions, classes, RAII, etc., which is really not worth it, unless 
 you're writing very low-level code (e.g. OS kernels and drivers).

 [1]: https://dlang.org/phobos/core_thread#.thread_detachThis
 [2]: https://dlang.org/phobos/core_memory#.GC.disable
 [3]: https://dlang.org/phobos/std_parallelism#.taskPool


 Forgot to mention, I'll try this first, I think its a good first step 
 towards -BetterC usage. But in the end I want to see how far I can get 
 with the -BetterC feature.


 
 In short, the cost / benefit of going all the way version(D_BetterC) is 
 incredibly poor for regular applications, as you end up a bit more 
 limited than with modern C++ (> 11) for prototyping. For example, even 
 writers of D real-time audio plugins don't go as far.


I just did some work for Guillaume Piolat (p0nce author of dplug), guess 
what is going to be used again, druntime!
Nov 09 2017
prev sibling parent reply Guillaume Piolat <contact spam.com> writes:
On Thursday, 9 November 2017 at 16:00:36 UTC, Petar Kirov 
[ZombineDev] wrote:

 In short, the cost / benefit of going all the way 
 version(D_BetterC) is incredibly poor for regular applications, 
 as you end up a bit more limited than with modern C++ (> 11) 
 for prototyping. For example, even writers of D real-time audio 
 plugins don't go as far.


For now we do have some  nogc alternatives for mutex, condition 
variables, thread-pool, file reading, etc... (dplug:core package) 
for use with the runtime disabled - the middle ground that's way 
more usable than -betterC. They may, or not, be applicable to 
-betterC.
Nov 10 2017
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 10 November 2017 at 11:55:57 UTC, Guillaume Piolat 
wrote:

 On Thursday, 9 November 2017 at 16:00:36 UTC, Petar Kirov 
 [ZombineDev] wrote:

 In short, the cost / benefit of going all the way 
 version(D_BetterC) is incredibly poor for regular 
 applications, as you end up a bit more limited than with 
 modern C++ (> 11) for prototyping. For example, even writers 
 of D real-time audio plugins don't go as far.


 For now we do have some  nogc alternatives for mutex, condition 
 variables, thread-pool, file reading, etc... (dplug:core 
 package) for use with the runtime disabled - the middle ground 
 that's way more usable than -betterC. They may, or not, be 
 applicable to -betterC.


Interesting, thanks for the info. It looks you made good progress 
since the last time I read about your project (quite a while 
ago). I'll be sure to check what you have in dplug:core ;)
Nov 10 2017
prev sibling parent ParticlePeter <ParticlePeter gmx.de> writes:
On Friday, 10 November 2017 at 11:55:57 UTC, Guillaume Piolat 
wrote:


 For now we do have some  nogc alternatives for mutex, condition 
 variables, thread-pool, file reading, etc... (dplug:core 
 package) for use with the runtime disabled - the middle ground 
 that's way more usable than -betterC. They may, or not, be 
 applicable to -betterC.


Your thread module (among others) is an amazing read and a very 
nice starting point for my endeavor. Thanks for pointing me in 
this direction!
Nov 27 2017
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-11-09 13:19, Petar Kirov [ZombineDev] wrote:


 Though you need to be extra careful not to use thread-local storage


I think TLS should work, it's the OS that handles TLS, not druntime.

-- 
/Jacob Carlborg
Nov 09 2017
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 16:08:20 UTC, Jacob Carlborg 
wrote:

 On 2017-11-09 13:19, Petar Kirov [ZombineDev] wrote:


 Though you need to be extra careful not to use thread-local 
 storage


 I think TLS should work, it's the OS that handles TLS, not 
 druntime.


Thanks for reminding me, I keep forgetting that it should just 
work (minus initialization?).
Nov 09 2017
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-11-09 17:52, Petar Kirov [ZombineDev] wrote:


 Thanks for reminding me, I keep forgetting that it should just work 
 (minus initialization?).


What do you mean "initialization"? Any type that can be used in C in TLS 
should work in D as well (except for macOS 32bit, if anyone cares).

-- 
/Jacob Carlborg
Nov 09 2017
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 9 November 2017 at 19:42:55 UTC, Jacob Carlborg 
wrote:

 On 2017-11-09 17:52, Petar Kirov [ZombineDev] wrote:


 Thanks for reminding me, I keep forgetting that it should just 
 work (minus initialization?).


 What do you mean "initialization"?


static constructors
Nov 10 2017
parent Jacob Carlborg <doob me.com> writes:
On 2017-11-10 14:30, Petar Kirov [ZombineDev] wrote:


 static constructors


Yeah, those won't work. I don't think that's really related to TLS, 
hence my confusion.

-- 
/Jacob Carlborg
Nov 10 2017