www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - is(Mutex == shared) == false?

reply "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
Ditto for other core.sync primitives.

This has been haunting me for a while now. Currently all those 
guys are not qualified shared at all. What that means is that we 
cannot use them in any shared classes:

shared class C
{
     Mutex m;

     this()
     {
         m = new Mutex; // error, cannot implicitly convert Mutex 
to shared(Mutex)
         m = new shared Mutex; // error, Mutex.this is not 
callable using a shared object
     }
}

Same goes for shared methods too. So the only possible ways to 
use Mutex et al. are to either declare them __gshared, which 
implies static, or apply casts. So we can't have per-instance 
mutexes, condition variables, etc, unless using casts everywhere.

As far as I can see this state is the same between dmd, ldc and 
gdc.

So, question number 1:

Is this at all intentional or just inherent and no one got around 
to adding support for shared? I can't at all see why would they 
be non-shared.

Because if that is to change, it'd better be sooner than later, 
right?

With mutexes the roots run as deep as object_.{d,di} where the 
Monitor interface is declared (having both lock and unlock 
methods non-shared).

I've been able to coerce my local pull of druntime to define 
Mutex et al. as shared by default. Aside from sorcery with casts 
inside e.g. src/gc/gc.d and combating segfaults in GC and Thread 
initialization, it involved:

- qualifying Monitor's methods as shared
- renaming existing classes (e.g. class Mutex -> shared class 
Mutex_)
- providing default-shared aliases (e.g. alias Mutex = 
shared(Mutex))

The renaming and alias are needed to (a) not break existing code 
and (b) because in my understanding they should be shared anyway 
:)

Surprisingly, this didn't take all that long, though I suspect 
there are some underwater rocks still remaining. But at least 
both druntime and Phobos pass their unittests.

Thus, question number 2:

Am I going in the right direction, or is there something already 
planned regarding this?

The complete set of changes would be rather large, as not only it 
spans currently supported OSs, but I imagine also would concern 
both druntime and Phobos (e.g. std.concurrency, std.parallelism).

I am primarily on Linux, but given time I can also edit the 
relevant files for Windows too. However, I don't have access to 
any other OSs, so I still won't be able to create a complete pull 
request.

Hence, question number 3:

Provided you've read to this point, and question number 2 yields 
positive answer, is there anybody willing to collaborate on this? 
E.g. complete/test the changes on Windows, OSX, etc?

I understand that the community is currently battling in the 
fields of GC vs ARC vs manual memory management, but I still hope 
to hear your feedback :)
Feb 07 2014
next sibling parent "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
 - providing default-shared aliases (e.g. alias Mutex = 
 shared(Mutex))


This should've been 'alias Mutex = shared(Mutex_)'.
Feb 07 2014
prev sibling parent reply "tcak" <tcak pcak.com> writes:
On Saturday, 8 February 2014 at 03:11:17 UTC, Stanislav Blinov 
wrote:

 Ditto for other core.sync primitives.

 This has been haunting me for a while now. Currently all those 
 guys are not qualified shared at all. What that means is that 
 we cannot use them in any shared classes:

 shared class C
 {
     Mutex m;

     this()
     {
         m = new Mutex; // error, cannot implicitly convert 
 Mutex to shared(Mutex)
         m = new shared Mutex; // error, Mutex.this is not 
 callable using a shared object
     }
 }

 Same goes for shared methods too. So the only possible ways to 
 use Mutex et al. are to either declare them __gshared, which 
 implies static, or apply casts. So we can't have per-instance 
 mutexes, condition variables, etc, unless using casts 
 everywhere.

 As far as I can see this state is the same between dmd, ldc and 
 gdc.

 So, question number 1:

 Is this at all intentional or just inherent and no one got 
 around to adding support for shared? I can't at all see why 
 would they be non-shared.

 Because if that is to change, it'd better be sooner than later, 
 right?

 With mutexes the roots run as deep as object_.{d,di} where the 
 Monitor interface is declared (having both lock and unlock 
 methods non-shared).

 I've been able to coerce my local pull of druntime to define 
 Mutex et al. as shared by default. Aside from sorcery with 
 casts inside e.g. src/gc/gc.d and combating segfaults in GC and 
 Thread initialization, it involved:

 - qualifying Monitor's methods as shared
 - renaming existing classes (e.g. class Mutex -> shared class 
 Mutex_)
 - providing default-shared aliases (e.g. alias Mutex = 
 shared(Mutex))

 The renaming and alias are needed to (a) not break existing 
 code and (b) because in my understanding they should be shared 
 anyway :)

 Surprisingly, this didn't take all that long, though I suspect 
 there are some underwater rocks still remaining. But at least 
 both druntime and Phobos pass their unittests.

 Thus, question number 2:

 Am I going in the right direction, or is there something 
 already planned regarding this?

 The complete set of changes would be rather large, as not only 
 it spans currently supported OSs, but I imagine also would 
 concern both druntime and Phobos (e.g. std.concurrency, 
 std.parallelism).

 I am primarily on Linux, but given time I can also edit the 
 relevant files for Windows too. However, I don't have access to 
 any other OSs, so I still won't be able to create a complete 
 pull request.

 Hence, question number 3:

 Provided you've read to this point, and question number 2 
 yields positive answer, is there anybody willing to collaborate 
 on this? E.g. complete/test the changes on Windows, OSX, etc?

 I understand that the community is currently battling in the 
 fields of GC vs ARC vs manual memory management, but I still 
 hope to hear your feedback :)


When I first time saw D language, I called it as my dream 
language. Well, every nice thing has its problems. I hated that 
"shared" keyword since first day. It ruins my codes whenever I 
need to write multiple thread programs.
Feb 07 2014
next sibling parent reply "Sean Kelly" <sean invisibleduck.org> writes:
I tried making Mutex shared once, and ended up down a rabbit hole 
of needing to make various Posix and Windows types shared, which 
in turn meant changing function signatures...  I reverted the 
change and decided to revisit it later... Which never happened.  
I suppose it's time to revisit this and see how it goes.
Feb 07 2014
parent reply "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
On Saturday, 8 February 2014 at 06:48:50 UTC, Sean Kelly wrote:

 I tried making Mutex shared once, and ended up down a rabbit 
 hole of needing to make various Posix and Windows types shared, 
 which in turn meant changing function signatures...


I think most of those can be resolved with casts (or methods that 
return unshared-casted pointers)?


 I reverted the change and decided to revisit it later... Which 
 never happened.  I suppose it's time to revisit this and see 
 how it goes.


I've created a branch here:

https://github.com/radcapricorn/druntime/tree/shared_sync_primitives

There are currently changes to make core.sync primitives shared 
on Posix.
Feb 08 2014
next sibling parent reply "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
Also, on a related note, is there any benefit in having core.sync 
primitives not final? What would be a use case to inherit from 
e.g. Mutex or Condition? GC does that with Mutex, but only to 
devirtualize all methods :)

Maybe we should also consider making them final along the way?
Feb 08 2014
next sibling parent reply "Sean Kelly" <sean invisibleduck.org> writes:
On Saturday, 8 February 2014 at 16:46:26 UTC, Stanislav Blinov 
wrote:

 Also, on a related note, is there any benefit in having 
 core.sync primitives not final? What would be a use case to 
 inherit from e.g. Mutex or Condition? GC does that with Mutex, 
 but only to devirtualize all methods :)

 Maybe we should also consider making them final along the way?


Probably.  The new std.concurrency patch overrides Condition, but 
mostly for convenience.  I think those methods not being virtual 
was an oversight on my part.
Feb 10 2014
next sibling parent reply "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
On Tuesday, 11 February 2014 at 05:58:15 UTC, Sean Kelly wrote:

 On Saturday, 8 February 2014 at 16:46:26 UTC, Stanislav Blinov



 Maybe we should also consider making [core.sync primitives] 
 final along the way?


 Probably.  The new std.concurrency patch overrides Condition, 
 but mostly for convenience.  I think those methods not being 
 virtual was an oversight on my part.


Great :)

Now to another issue, or possible enhancement. It would seem that 
we may benefit from some sort of relaxed shared ops.

For example, Condition on Windows has several int fields, which 
of course would become shared(int) due to transitivity. 
Currently, those ints are modified directly using ++, --, += and 
so on. As per 
https://d.puremagic.com/issues/show_bug.cgi?id=3672, such code 
would be illegal. A straightforward hack would be to move such 
code into non-shared methods and then call those methods by 
casting away shared on the this reference. A better, but still 
naive replacement would be to use atomicOp() for all those 
operations. The problem is that many of those operations happen 
under a lock, where a strong do {} while (!cas()) loop, to which 
atomicOp() currently translates, would be unnecessary 
pessimization.
Of course, they could be replaced manually with something like:

// atomicOp!"-="(m_numWaitersBlocked, 1);
atomicStore!(MemoryOrder.rel)(m_numWaitersBlocked, 
atomicLoad!(MemoryOrder.acq)(m_numWaitersBlocked) - 1);

But that is just tedious to type, hard to read, and still may be 
less efficient that a straightforward --.

Maybe an additional function like this could be helpful:

--8<--

HeadUnshared!T localOp(string op,T,V)(T what, V mod) if (is(T == 
shared))
{ ... }

-->8--

Or even something like this:

--8<--

ref auto assumeLocal(T)(ref T v) if (is(T == shared))
{
     // Cast via pointer to preserve lvalue
     return *cast(HeadUnshared!T*)&v;
}

-->8--

With which we can perform this:

--8<--

--assumeLocal(m_numWaitersBlocked);

-->8--

What do you think?
Feb 11 2014
next sibling parent "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
On Tuesday, 11 February 2014 at 17:09:39 UTC, Stanislav Blinov 
wrote:


 ref auto assumeLocal(T)(ref T v) if (is(T == shared))
 {
     // Cast via pointer to preserve lvalue
     return *cast(HeadUnshared!T*)&v;
 }


Well, maybe not HeadUnshared!T but some other template that 
removes shared qualifier.
Feb 11 2014
prev sibling parent reply "Sean Kelly" <sean invisibleduck.org> writes:
On Tuesday, 11 February 2014 at 17:09:39 UTC, Stanislav Blinov
wrote:

 Maybe an additional function like this could be helpful:

 --8<--

 HeadUnshared!T localOp(string op,T,V)(T what, V mod) if (is(T 
 == shared))
 { ... }

 -->8--

 Or even something like this:

 --8<--

 ref auto assumeLocal(T)(ref T v) if (is(T == shared))
 {
     // Cast via pointer to preserve lvalue
     return *cast(HeadUnshared!T*)&v;
 }

 -->8--

 With which we can perform this:

 --8<--

 --assumeLocal(m_numWaitersBlocked);

 -->8--

 What do you think?


I think it's a good idea.  Please file bugzilla requests for this
and the final method issue :-)

Regarding making Mutex methods final, this may require some care
because existing projects might rely on them being virtual.  I'm
pretty sure vibe.d, for example, overrides Condition as well.  I
don't know what the best process is for changing these to final
in terms of breaking code.
Feb 11 2014
parent reply "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
On Tuesday, 11 February 2014 at 21:08:24 UTC, Sean Kelly wrote:

 On Tuesday, 11 February 2014 at 17:09:39 UTC, Stanislav Blinov
 wrote:



 What do you think?


 I think it's a good idea.  Please file bugzilla requests for 
 this
 and the final method issue :-)


Will do.


 Regarding making Mutex methods final, this may require some care
 because existing projects might rely on them being virtual.  I'm
 pretty sure vibe.d, for example, overrides Condition as well.  I
 don't know what the best process is for changing these to final
 in terms of breaking code.


Heh. Yeah, I've just browsed through vibe.d code... such breakage 
would look pretty gruesome. Maybe it's not such a good idea after 
all. But I'll open the enhancement request anyway, if only to 
have a centralized place to gather all pros and cons on this 
specific issue.
Feb 11 2014
parent "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
Here's an ER for those functions: 
http://forum.dlang.org/thread/bug-12133-3 https.d.puremagic.com%2Fissues%2F

I guess in regards to final there will be room to maneuver 
anyway: 
http://forum.dlang.org/thread/bug-11616-3 https.d.puremagic.com%2Fissues%2F

:)
Feb 12 2014
prev sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig+dforum outerproduct.org> writes:
Am 11.02.2014 06:58, schrieb Sean Kelly:

 On Saturday, 8 February 2014 at 16:46:26 UTC, Stanislav Blinov wrote:

 Also, on a related note, is there any benefit in having core.sync
 primitives not final? What would be a use case to inherit from e.g.
 Mutex or Condition? GC does that with Mutex, but only to devirtualize
 all methods :)

 Maybe we should also consider making them final along the way?


 Probably.  The new std.concurrency patch overrides Condition, but mostly
 for convenience.  I think those methods not being virtual was an
 oversight on my part.


vibe.d currently (ab)uses this fact to be able to use its own mutex 
class as object monitors and for use in synchronized blocks. I agree 
that they should generally be final, but please let's add another way to 
plug in there when doing such a change.
Feb 17 2014
parent "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
On Monday, 17 February 2014 at 08:29:41 UTC, Sönke Ludwig wrote:


 vibe.d currently (ab)uses this fact to be able to use its own 
 mutex class as object monitors and for use in synchronized 
 blocks. I agree that they should generally be final, but please 
 let's add another way to plug in there when doing such a change.


I'd suggest putting this on hold until the time for that 
transition to default-final starts. It'd be an ideal moment to 
re-evaluate this need of inheritance. :)
Feb 17 2014
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 08 Feb 2014 16:46:25 +0000
schrieb "Stanislav Blinov" <stanislav.blinov gmail.com>:


 Also, on a related note, is there any benefit in having core.sync 
 primitives not final? What would be a use case to inherit from 
 e.g. Mutex or Condition? GC does that with Mutex, but only to 
 devirtualize all methods :)
 
 Maybe we should also consider making them final along the way?


I did once inherit from Mutex to add a reentrancy counter.
This can be used in a Logger, to prevent one thread to
endlessly recurse into logging functions. Like when logging
causes and error that is logged which causes the same error
again, which is logged, ...
If the reentrancy counter is > 1 after locking, you can
special case around that and just print something to stderr
directly or delegate to a less error prone logger.

-- 
Marco
Oct 20 2014
prev sibling parent "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
On Saturday, 8 February 2014 at 13:59:59 UTC, Stanislav Blinov 
wrote:


 I've created a branch here:

 https://github.com/radcapricorn/druntime/tree/shared_sync_primitives


I've committed a considerable update. Still haven't tested on 
anything but Linux though, so if anyone is able to pull that 
branch and run druntime tests, I'd be obliged.

Criticism, suggestions and improvements are also welcome :)
Feb 15 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/7/14, 10:21 PM, tcak wrote:

 When I first time saw D language, I called it as my dream language.
 Well, every nice thing has its problems. I hated that "shared" keyword
 since first day. It ruins my codes whenever I need to write multiple
 thread programs.


We should add finalizing shared to our H1 goals.

Andrei
Feb 08 2014
next sibling parent "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
Here's bugzilla enhancement request: 
https://d.puremagic.com/issues/show_bug.cgi?id=12132
Feb 10 2014
prev sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Saturday, 8 February 2014 at 16:46:47 UTC, Andrei Alexandrescu 
wrote:

 We should add finalizing shared to our H1 goals.

 Andrei


Not sure what an H1 goal is, but please put this on the Agenda, 
so that we can reserve time for it. http://wiki.dlang.org/Agenda
Feb 15 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/15/14, 12:27 AM, Martin Nowak wrote:

 On Saturday, 8 February 2014 at 16:46:47 UTC, Andrei Alexandrescu wrote:

 We should add finalizing shared to our H1 goals.

 Andrei


 Not sure what an H1 goal is, but please put this on the Agenda, so that
 we can reserve time for it. http://wiki.dlang.org/Agenda


I meant first half of the year. I don't think it fits the Agenda, which 
is release-focused.

Andrei
Feb 15 2014
next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Saturday, 15 February 2014 at 14:55:44 UTC, Andrei
Alexandrescu wrote:

 I don't think it fits the Agenda, which is release-focused.


So we're talking about a DIP here?
Feb 15 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/15/14, 7:54 AM, Martin Nowak wrote:

 On Saturday, 15 February 2014 at 14:55:44 UTC, Andrei
 Alexandrescu wrote:

 I don't think it fits the Agenda, which is release-focused.


 So we're talking about a DIP here?


Not a DIP, it's more like a direction/focus thing. In my opinion we must 
focus this year on:

* Less garbage

* Address null pointers at language level

* Finish the language, in particular shared & comp

* Do something about this(this) that is less complicated than the Space 
Shuttle

* And of course quality, quality, quality



Andrei
Feb 15 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/15/14, 12:07 PM, Andrei Alexandrescu wrote:

 On 2/15/14, 7:54 AM, Martin Nowak wrote:

 On Saturday, 15 February 2014 at 14:55:44 UTC, Andrei
 Alexandrescu wrote:

 I don't think it fits the Agenda, which is release-focused.


 So we're talking about a DIP here?


 Not a DIP, it's more like a direction/focus thing. In my opinion we must
 focus this year on:

 * Less garbage

 * Address null pointers at language level

 * Finish the language, in particular shared & comp

 * Do something about this(this) that is less complicated than the Space
 Shuttle

 * And of course quality, quality, quality


Slipped my mind:

* Finalize portable shared lib support on Unixen, OSX, and Windows


Andrei
Feb 15 2014
parent reply "Martin Nowak" <code dawg.eu> writes:
On Saturday, 15 February 2014 at 20:31:07 UTC, Andrei 
Alexandrescu wrote:

 Not a DIP, it's more like a direction/focus thing. In my 
 opinion we must
 focus this year on:

 * Less garbage

 * Address null pointers at language level

 * Finish the language, in particular shared & comp

 * Do something about this(this) that is less complicated than 
 the Space
 Shuttle

 * And of course quality, quality, quality


 Slipped my mind:

 * Finalize portable shared lib support on Unixen, OSX, and 
 Windows


 Andrei


So how do we communicate/achieve these?
I think having a newsgroup thread about this year's goals would 
be a good start, to get everyone on the same page.
Feb 16 2014
next sibling parent "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
On Sunday, 16 February 2014 at 18:02:24 UTC, Martin Nowak wrote:


 So how do we communicate/achieve these?
 I think having a newsgroup thread about this year's goals would 
 be a good start, to get everyone on the same page.


Maybe with some additional reflection i.e. in the Wiki? You know 
how these threads can be derailed or otherwise become a mess to 
search for anything :)
Feb 16 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/16/14, 10:02 AM, Martin Nowak wrote:

 On Saturday, 15 February 2014 at 20:31:07 UTC, Andrei Alexandrescu wrote:

 Not a DIP, it's more like a direction/focus thing. In my opinion we must
 focus this year on:

 * Less garbage

 * Address null pointers at language level

 * Finish the language, in particular shared & comp

 * Do something about this(this) that is less complicated than the Space
 Shuttle

 * And of course quality, quality, quality


 Slipped my mind:

 * Finalize portable shared lib support on Unixen, OSX, and Windows


 Andrei


 So how do we communicate/achieve these?
 I think having a newsgroup thread about this year's goals would be a
 good start, to get everyone on the same page.


Feel free to paste these in a wiki page. Experience suggests that me 
telling people what I think is important and must be done has very 
limited impact.

I've made no secret about what I think are important topics for D at any 
time, and often each had one or a few people who'd be obviously best 
positioned to work on. Yet each person continued to work on whatever 
itch they wanted scratched, which is entirely understandable.


Andrei
Feb 16 2014
parent reply "Stanislav Blinov" <stanislav.blinov gmail.com> writes:
On Monday, 17 February 2014 at 02:22:10 UTC, Andrei Alexandrescu 
wrote:


 Feel free to paste these in a wiki page. Experience suggests 
 that me telling people what I think is important and must be 
 done has very limited impact.


I'd hazard a guess that this is in part because whatever 
consensus reached at any given point doesn't always find its way 
to publicity and only stays in mind of people involved in the 
discussion and buried in NG postings. Wiki at least can help in 
that regard.


 I've made no secret about what I think are important topics for 
 D at any time, and often each had one or a few people who'd be 
 obviously best positioned to work on. Yet each person continued 
 to work on whatever itch they wanted scratched, which is 
 entirely understandable.


Not everyone is fond of taking initiative, that's a known fact. 
Me, I'm tired of seeing new big features and ideas popping up 
when there's so much unfinished business in the language, runtime 
and Phobos. When I left this NG (two years ago?) 'shared' was an 
infant, today it's just a tad more than that. Without clearly 
defined (and supported by the runtime) 'shared' I can't see how 
allocators can be finalized, which in turn also influences that 
GC/ARC/whatever shennanigans... So I'm willing to help make it 
better to the extent of my mental capacity. But again, this 
thread will soon be buried deep, a more concrete and focused 
discussion begs for another type of media.
Feb 17 2014
parent "Tolga Cakiroglu (tcak)" <tcak pcak.com> writes:
 Not everyone is fond of taking initiative, that's a known fact. 
 Me, I'm tired of seeing new big features and ideas popping up 
 when there's so much unfinished business in the language, 
 runtime and Phobos. When I left this NG (two years ago?) 
 'shared' was an infant, today it's just a tad more than that. 
 Without clearly defined (and supported by the runtime) 'shared' 
 I can't see how allocators can be finalized, which in turn also 
 influences that GC/ARC/whatever shennanigans... So I'm willing 
 to help make it better to the extent of my mental capacity. But 
 again, this thread will soon be buried deep, a more concrete 
 and focused discussion begs for another type of media.


Let me tell you the people why that is happening. It is because 
we are trying to solve problems without dividing into small 
chunks. This is a problem in my life, many people lives, and also 
businesses. If we just say that let's solve X problems (shared, 
GC, etc. whatever it is), without properly "defining" what small 
steps to be taken, what are trying to achieve, who will work on 
which part, people will continue talking without any achievement, 
and same topics come front of us again and again. The reason why 
project managers exist is this.

We need better documentation as Todo's, charts etc. Exempli 
gratia, I haven't seen any gantt chart like thing. It is not 
obvious where we are going. If a ship doesn't have a destination, 
no wind can help it.
Feb 17 2014
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 08 Feb 2014 08:46:47 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 We should add finalizing shared to our H1 goals.


Yes.

Am Sat, 15 Feb 2014 06:55:46 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 I meant first half of the year. I don't think it fits the Agenda, which 
 is release-focused.
 
 Andrei


I take your word on it!

I am really trying to _use_ shared, but when not even a Mutex
or Condition is shared and making it so leads to horrible
head-aches few people will use it, even fewer will understand
it and there is no one there to do something about it.

-- 
Marco
Oct 20 2014