=?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

Hi,

I've seen several occurrences of synchronized (this) and synchronized 
(this.classinfo) in both druntime and phobos by now. I propose that we 
officially ban these patterns with extreme prejudice.

1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happens 
to use the object for locking will most likely end up with a deadlock on 
their hands.
2) Locking on something as fundamental as type info means that any 
arbitrary part of the application could cause a deadlock by doing the same.

The solution to (1) is to simply use a Mutex internally (or an Object 
with synchronized statements), and for (2), to simply use private global 
objects.

(Now, regarding (1), you might argue that anyone who locks on an 
arbitrary object is doing it wrong, but you can't really blame them - 
it's frankly D's fault for allowing monitors on arbitrary objects, which 
is a horrible mess.)

Anyone against this?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Jonathan M Davis <jmdavisProg gmx.com> writes:

On Tuesday, May 29, 2012 00:36:13 Alex R=C3=B8nne Petersen wrote:
 Hi,
=20
 I've seen several occurrences of synchronized (this) and synchronized=

 (this.classinfo) in both druntime and phobos by now. I propose that w=

e
 officially ban these patterns with extreme prejudice.
=20
 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happ=

ens
 to use the object for locking will most likely end up with a deadlock=

 on
 their hands.
 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the=

 same.
=20
 The solution to (1) is to simply use a Mutex internally (or an Object=

 with synchronized statements), and for (2), to simply use private glo=

bal
 objects.
=20
 (Now, regarding (1), you might argue that anyone who locks on an
 arbitrary object is doing it wrong, but you can't really blame them -=

 it's frankly D's fault for allowing monitors on arbitrary objects, wh=

ich
 is a horrible mess.)
=20
 Anyone against this?

Don't synchronized classes synchronize on the object for all of their f=
unction=20
calls? How is this any different?

- Jonathan M Davis

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 00:42, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 00:36:13 Alex Rønne Petersen wrote:
 Hi,

 I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happens
 to use the object for locking will most likely end up with a deadlock on
 their hands.
 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the same.

 The solution to (1) is to simply use a Mutex internally (or an Object
 with synchronized statements), and for (2), to simply use private global
 objects.

 (Now, regarding (1), you might argue that anyone who locks on an
 arbitrary object is doing it wrong, but you can't really blame them -
 it's frankly D's fault for allowing monitors on arbitrary objects, which
 is a horrible mess.)

 Anyone against this?

 Don't synchronized classes synchronize on the object for all of their function
 calls? How is this any different?

 - Jonathan M Davis

I have no idea how synchronized classes work; they are not a documented 
feature of the language. We have synchronized functions which 
synchronize on the this reference. Perhaps synchronized classes just 
make all functions in a class do this.

Either way, this is a fundamental language design fallacy. This is 
anti-pattern number one when it comes to locking:

* http://stackoverflow.com/a/251668/438034
* http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and 
SyncLock Keywords section)

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Jonathan M Davis <jmdavisProg gmx.com> writes:

On Tuesday, May 29, 2012 01:11:49 Alex R=C3=B8nne Petersen wrote:
 I have no idea how synchronized classes work; they are not a document=

ed
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either=
 _all_=20
of the functions in a class are synchronized or _none_ of them are. Put=
ting=20
synchronized on a function should actually be illegal. It belongs only =
on=20
classes or in synchronized blocks, never on functions.

Now, unfortuntately, I don't believe that the compiler enforces any of =
that=20
right now, so you end up synchronizing indivdual functions rather than =
whole=20
classes, but the synchronized functions themselves should function the =
same.



- Jonathan M Davis

Jonathan M Davis <jmdavisProg gmx.com> writes:

On Tuesday, May 29, 2012 01:11:49 Alex R=C3=B8nne Petersen wrote:
 I have no idea how synchronized classes work; they are not a document=

ed
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

Per TDPL, having individually synchronized functions is illegal. Either=
 all=20
of the functions in a class are synchronized or none of them are. Putti=
ng=20
synchronized on a function should actually be illegal. It belongs only =
on=20
classes or in synchronized blocks, never on functions.

Now, unfortuntately, I don't believe that the compiler enforces any of =
that=20
right now, so you end up synchronizing indivdual functions rather than =
whole=20
classes, but the synchronized functions themselves should function the =
same.

 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:
=20
 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and=

 SyncLock Keywords section)

Well, then you should probably be arguing about the design of synchroni=
zed=20
classes/functions rather than synchronized(this). However, given the de=
sign of=20
synchronized classes, synchronized(this) would probably never be approp=
riate.=20
If you're using a synchronized class, then you don't need it. And if yo=
u're=20
not using a synchronized class, then you shouldn't be synchronizing you=
r=20
class. I would definitely think that synchronized blocks should synchro=
nize on=20
something else, since they're trying to lock a much smaller area than a=
n=20
entire class.

- Jonathan M Davis

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

 Per TDPL, having individually synchronized functions is illegal. Either all
 of the functions in a class are synchronized or none of them are. Putting
 synchronized on a function should actually be illegal. It belongs only on
 classes or in synchronized blocks, never on functions.

 Now, unfortuntately, I don't believe that the compiler enforces any of that
 right now, so you end up synchronizing indivdual functions rather than whole
 classes, but the synchronized functions themselves should function the same.

 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

 Well, then you should probably be arguing about the design of synchronized
 classes/functions rather than synchronized(this). However, given the design of
 synchronized classes, synchronized(this) would probably never be appropriate.
 If you're using a synchronized class, then you don't need it. And if you're
 not using a synchronized class, then you shouldn't be synchronizing your
 class. I would definitely think that synchronized blocks should synchronize on
 something else, since they're trying to lock a much smaller area than an
 entire class.

 - Jonathan M Davis

I don't think arguing about them makes sense at this point. Way too much 
code would break if we changed the semantics. I'd consider it a mistake 
and a lesson learned, rather.

But I take it you agree that synchronized (this) and similar 
"uncontrollable" synchronization blocks should be avoided?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 01:35, Alex Rønne Petersen wrote:
 On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

 Per TDPL, having individually synchronized functions is illegal.
 Either all
 of the functions in a class are synchronized or none of them are. Putting
 synchronized on a function should actually be illegal. It belongs only on
 classes or in synchronized blocks, never on functions.

 Now, unfortuntately, I don't believe that the compiler enforces any of
 that
 right now, so you end up synchronizing indivdual functions rather than
 whole
 classes, but the synchronized functions themselves should function the
 same.

 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

 Well, then you should probably be arguing about the design of
 synchronized
 classes/functions rather than synchronized(this). However, given the
 design of
 synchronized classes, synchronized(this) would probably never be
 appropriate.
 If you're using a synchronized class, then you don't need it. And if
 you're
 not using a synchronized class, then you shouldn't be synchronizing your
 class. I would definitely think that synchronized blocks should
 synchronize on
 something else, since they're trying to lock a much smaller area than an
 entire class.

 - Jonathan M Davis

 I don't think arguing about them makes sense at this point. Way too much
 code would break if we changed the semantics. I'd consider it a mistake
 and a lesson learned, rather.

I should probably add that Java learned it long ago, and yet we adopted 
it anyway... blergh.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Jonathan M Davis <jmdavisProg gmx.com> writes:

On Tuesday, May 29, 2012 01:38:25 Alex R=C3=B8nne Petersen wrote:
 I should probably add that Java learned it long ago, and yet we adopt=

ed
 it anyway... blergh.

The "lesson learned" from Java that TDPL enumerates is the mistake of h=
aving=20
synchronized on functions rather than entire classes, but clearly even =
that is=20
currently TDPL-only and not actually properly implemented yet.

- Jonathan M Davis

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

 The "lesson learned" from Java that TDPL enumerates is the mistake of having
 synchronized on functions rather than entire classes, but clearly even that is
 currently TDPL-only and not actually properly implemented yet.

 - Jonathan M Davis

But synchronized on entire classes is exactly equally flawed... the 
fundamental problem is still that they synchronize on a public resource.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

"Nathan M. Swan" <nathanmswan gmail.com> writes:

On Monday, 28 May 2012 at 23:55:04 UTC, Alex Rønne Petersen 
wrote:
 On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet 
 we adopted
 it anyway... blergh.

 The "lesson learned" from Java that TDPL enumerates is the 
 mistake of having
 synchronized on functions rather than entire classes, but 
 clearly even that is
 currently TDPL-only and not actually properly implemented yet.

 - Jonathan M Davis

 But synchronized on entire classes is exactly equally flawed... 
 the fundamental problem is still that they synchronize on a 
 public resource.

The fundamental problem is shared memory ;)
NMS

"Jess" <Jess.Sandgren hotmail.com> writes:

On Monday, 28 May 2012 at 23:55:04 UTC, Alex Rønne Petersen 
wrote:
 On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet 
 we adopted
 it anyway... blergh.

 The "lesson learned" from Java that TDPL enumerates is the 
 mistake of having
 synchronized on functions rather than entire classes, but 
 clearly even that is
 currently TDPL-only and not actually properly implemented yet.

 - Jonathan M Davis

 But synchronized on entire classes is exactly equally flawed... 
 the fundamental problem is still that they synchronize on a 
 public resource.

Sorry, I don't quite follow...

1) locking on a public resource
=> BAD, high risk.

2) Adding GC.callLocked(), allowing a generic delegate to claim 
the GC lock
https://github.com/D-Programming-Language/druntime/pull/213
=> no-brainer, i.e. no risk?

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 18:41, Jess wrote:
 On Monday, 28 May 2012 at 23:55:04 UTC, Alex Rønne Petersen wrote:
 On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

 The "lesson learned" from Java that TDPL enumerates is the mistake of
 having
 synchronized on functions rather than entire classes, but clearly
 even that is
 currently TDPL-only and not actually properly implemented yet.

 - Jonathan M Davis

 But synchronized on entire classes is exactly equally flawed... the
 fundamental problem is still that they synchronize on a public resource.

 Sorry, I don't quite follow...

 1) locking on a public resource
 => BAD, high risk.

 2) Adding GC.callLocked(), allowing a generic delegate to claim the GC lock
 https://github.com/D-Programming-Language/druntime/pull/213
 => no-brainer, i.e. no risk?

druntime is the low-level runtime library of D. This discussion is about 
high-level class design.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 18:57, Alex Rønne Petersen wrote:
 On 29-05-2012 18:41, Jess wrote:
 On Monday, 28 May 2012 at 23:55:04 UTC, Alex Rønne Petersen wrote:
 On 29-05-2012 01:46, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:38:25 Alex Rønne Petersen wrote:
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

 The "lesson learned" from Java that TDPL enumerates is the mistake of
 having
 synchronized on functions rather than entire classes, but clearly
 even that is
 currently TDPL-only and not actually properly implemented yet.

 - Jonathan M Davis

 But synchronized on entire classes is exactly equally flawed... the
 fundamental problem is still that they synchronize on a public resource.

 Sorry, I don't quite follow...

 1) locking on a public resource
 => BAD, high risk.

 2) Adding GC.callLocked(), allowing a generic delegate to claim the GC
 lock
 https://github.com/D-Programming-Language/druntime/pull/213
 => no-brainer, i.e. no risk?

 druntime is the low-level runtime library of D. This discussion is about
 high-level class design.

I should also add that the reason I started this thread was to prevent 
common locking errors induced by the anti-patterns synchronized causes. 
If you're calling *anything* in the core.memory module, I sure hope you 
know what you're doing in general.

I really don't think these two issues are connected at all.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

"Jess" <Jess.Sandgren hotmail.com> writes:

On Tuesday, 29 May 2012 at 17:01:01 UTC, Alex Rønne Petersen 
wrote:

 druntime is the low-level runtime library of D. This 
 discussion is about
 high-level class design.

 I should also add that the reason I started this thread was to 
 prevent common locking errors induced by the anti-patterns 
 synchronized causes. If you're calling *anything* in the 
 core.memory module, I sure hope you know what you're doing in 
 general.

 I really don't think these two issues are connected at all.

ok, maybe I read too much into your statement below? it sounded 
as if you wanted to ban this usage from druntime?

Alex Rønne Petersen wrote:
"I've seen several occurrences of synchronized (this) and 
synchronized
(this.classinfo) in both druntime and phobos by now. I propose 
that we
officially ban these patterns with extreme prejudice."

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 19:37, Jess wrote:
 On Tuesday, 29 May 2012 at 17:01:01 UTC, Alex Rønne Petersen wrote:

 druntime is the low-level runtime library of D. This discussion is about
 high-level class design.

 I should also add that the reason I started this thread was to prevent
 common locking errors induced by the anti-patterns synchronized
 causes. If you're calling *anything* in the core.memory module, I sure
 hope you know what you're doing in general.

 I really don't think these two issues are connected at all.

 ok, maybe I read too much into your statement below? it sounded as if
 you wanted to ban this usage from druntime?

 Alex Rønne Petersen wrote:
 "I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice."

Yes, I do. But I'm not sure what that has to do with my GC.callLocked() PR?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 On 29-05-2012 01:35, Alex Rønne Petersen wrote:
 On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

 Per TDPL, having individually synchronized functions is illegal.
 Either all
 of the functions in a class are synchronized or none of them are.
 Putting
 synchronized on a function should actually be illegal. It belongs
 only on
 classes or in synchronized blocks, never on functions.

 Now, unfortuntately, I don't believe that the compiler enforces any of
 that
 right now, so you end up synchronizing indivdual functions rather than
 whole
 classes, but the synchronized functions themselves should function the
 same.

 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

 Well, then you should probably be arguing about the design of
 synchronized
 classes/functions rather than synchronized(this). However, given the
 design of
 synchronized classes, synchronized(this) would probably never be
 appropriate.
 If you're using a synchronized class, then you don't need it. And if
 you're
 not using a synchronized class, then you shouldn't be synchronizing your
 class. I would definitely think that synchronized blocks should
 synchronize on
 something else, since they're trying to lock a much smaller area than an
 entire class.

 - Jonathan M Davis

 I don't think arguing about them makes sense at this point. Way too much
 code would break if we changed the semantics. I'd consider it a mistake
 and a lesson learned, rather.

 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

That is what I was about to say. No point of doing D if it is to repeat 
previously done errors.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to repeat
 previously done errors.

So what is the lesson Java learned, and how does it address 
multithreaded programming in wake of that lesson?

Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to repeat
 previously done errors.

 So what is the lesson Java learned, and how does it address
 multithreaded programming in wake of that lesson?

 Andrei

It learned that allowing locking on arbitrary objects makes controlling 
locking (and thus reducing the chance for deadlocks) impossible.

It goes against our TLS-by-default and message-passing concurrency design.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to repeat
 previously done errors.

 So what is the lesson Java learned, and how does it address
 multithreaded programming in wake of that lesson?

 Andrei

 It learned that allowing locking on arbitrary objects makes controlling
 locking (and thus reducing the chance for deadlocks) impossible.

And how does Java address multithreading in general, and those issues in 
particular, today?

Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to repeat
 previously done errors.

 So what is the lesson Java learned, and how does it address
 multithreaded programming in wake of that lesson?

 Andrei

 It learned that allowing locking on arbitrary objects makes controlling
 locking (and thus reducing the chance for deadlocks) impossible.

 And how does Java address multithreading in general, and those issues in
 particular, today?

 Andrei



figures in those programming language communities have written blog 
posts on why these language constructs are evil and should be avoided.

Besides, it seems to me that D can't quite make up its mind. We have TLS 
by default, and we encourage message-passing (through a library 
mechanism), and then we have the synchronized statement and attribute. 
It just seems so incredibly inconsistent. synchronized encourages doing 
the wrong thing (locks and synchronization).

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

"Jonathan M Davis" <jmdavisProg gmx.com> writes:

On Wednesday, May 30, 2012 00:01:49 Alex Rønne Petersen wrote:
 Besides, it seems to me that D can't quite make up its mind. We have TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

It allows multiple approaches at multiple levels. TLS and message passing is 
the preferred way, and shared combined with synchronized or mutexes to protect 
it is available when you need it. synchronized classes are more 
straightforward approach to locking when you need to protect an entire object 
(and less error-prone in some respects, because you then have the guarantee 
that it happens for all functions and don't run the risk of not locking in 
some functions; it also works with inheritance that way, unlike mutexes). 
sychronized blocks on the other hand are a quick and easy way to protect 
specific sections of code without having to bother creating mutexes for it. And 
mutexes are available when you need full control. Which approach you go with 
depends on what you're doing and what your needs are.

Now, I could definitely see an argument that using shared is more low level and 
that it would be just simpler to only have mutexes and no synchronized, but we 
ended up with a more tiered approach, and that's not all bad. Each approach 
has its advantages and disadvantages, and D gives you all of them, so you can 
pick and choose what works best for you.

- Jonathan M Davis

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 00:35, Jonathan M Davis wrote:
 On Wednesday, May 30, 2012 00:01:49 Alex Rønne Petersen wrote:
 Besides, it seems to me that D can't quite make up its mind. We have TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

 It allows multiple approaches at multiple levels. TLS and message passing is
 the preferred way, and shared combined with synchronized or mutexes to protect
 it is available when you need it. synchronized classes are more
 straightforward approach to locking when you need to protect an entire object
 (and less error-prone in some respects, because you then have the guarantee
 that it happens for all functions and don't run the risk of not locking in
 some functions; it also works with inheritance that way, unlike mutexes).
 sychronized blocks on the other hand are a quick and easy way to protect
 specific sections of code without having to bother creating mutexes for it. And
 mutexes are available when you need full control. Which approach you go with
 depends on what you're doing and what your needs are.

I agree on your point about synchronized blocks, but not about 
synchronized classes. Synchronized classes lock on the this reference, 
which is exactly what should be avoided.

 Now, I could definitely see an argument that using shared is more low level and
 that it would be just simpler to only have mutexes and no synchronized, but we
 ended up with a more tiered approach, and that's not all bad. Each approach
 has its advantages and disadvantages, and D gives you all of them, so you can
 pick and choose what works best for you.

 - Jonathan M Davis

To be clear, I have nothing against synchronized blocks per se. I think 
it's fine to have them in the language. But I think that making *every* 
object a monitor is a horrible mistake (for reasons I outlined in 
another post in this thread). Synchronized blocks are good because they 
operate on an implicit, hidden, global mutex. You can't screw up with that.

Synchronized blocks that operate on a specific object should be limited 
somehow so not every object has to have a monitor field.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
  Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with that.

I think there's quite some disconnect here. If there's any anti-pattern 
in this discussion, it's operating on an implicit, hidden, global mutex. 
Walter agreed to eliminate that from D, but never got around to it.

Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

 I think there's quite some disconnect here. If there's any anti-pattern
 in this discussion, it's operating on an implicit, hidden, global mutex.
 Walter agreed to eliminate that from D, but never got around to it.

 Andrei

I'd love to hear why you think this design is problematic as opposed to 
one that lets users accidentally expose synchronization issues to 
consumers of their API surface, which is what many people end up doing 
since synchronized (this) or even synchronized (this.classinfo) are 
allowed at all.

(I've seen countless cases of those two horrible abuses of synchronized 
especially from people asking questions on e.g. IRC.)

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 4:17 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

 I think there's quite some disconnect here. If there's any anti-pattern
 in this discussion, it's operating on an implicit, hidden, global mutex.
 Walter agreed to eliminate that from D, but never got around to it.

 Andrei

 I'd love to hear why you think this design is problematic as opposed to
 one that lets users accidentally expose synchronization issues to
 consumers of their API surface, which is what many people end up doing
 since synchronized (this) or even synchronized (this.classinfo) are
 allowed at all.

 (I've seen countless cases of those two horrible abuses of synchronized
 especially from people asking questions on e.g. IRC.)

I think the most egregious example is synchronization by global lock 
done in Python and other languages. It has very nice semantics (stop the 
world, do something, resume the world), but scales poorly enough to be 
universally considered a failure. Quite honest I'm shocked that you're 
advocating it.

Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 01:22, Andrei Alexandrescu wrote:
 On 5/29/12 4:17 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

 I think there's quite some disconnect here. If there's any anti-pattern
 in this discussion, it's operating on an implicit, hidden, global mutex.
 Walter agreed to eliminate that from D, but never got around to it.

 Andrei

 I'd love to hear why you think this design is problematic as opposed to
 one that lets users accidentally expose synchronization issues to
 consumers of their API surface, which is what many people end up doing
 since synchronized (this) or even synchronized (this.classinfo) are
 allowed at all.

 (I've seen countless cases of those two horrible abuses of synchronized
 especially from people asking questions on e.g. IRC.)

 I think the most egregious example is synchronization by global lock
 done in Python and other languages. It has very nice semantics (stop the
 world, do something, resume the world), but scales poorly enough to be
 universally considered a failure. Quite honest I'm shocked that you're
 advocating it.

 Andrei

I'm not advocating it. I don't use it myself. I'm just saying that I 
find it to be less of an issue than synchronizing on arbitrary objects, 
because the latter is error-prone.

And yes, Python is a textbook example of synchronization gone completely 
wrong. But that doesn't mean that you don't sometimes have to 
synchronize on some global resource, and in those cases, it can be 
useful syntactic sugar (but certainly not essential).

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 01:25, Alex Rønne Petersen a écrit :
 On 30-05-2012 01:22, Andrei Alexandrescu wrote:
 On 5/29/12 4:17 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

 I think there's quite some disconnect here. If there's any anti-pattern
 in this discussion, it's operating on an implicit, hidden, global
 mutex.
 Walter agreed to eliminate that from D, but never got around to it.

 Andrei

 I'd love to hear why you think this design is problematic as opposed to
 one that lets users accidentally expose synchronization issues to
 consumers of their API surface, which is what many people end up doing
 since synchronized (this) or even synchronized (this.classinfo) are
 allowed at all.

 (I've seen countless cases of those two horrible abuses of synchronized
 especially from people asking questions on e.g. IRC.)

 I think the most egregious example is synchronization by global lock
 done in Python and other languages. It has very nice semantics (stop the
 world, do something, resume the world), but scales poorly enough to be
 universally considered a failure. Quite honest I'm shocked that you're
 advocating it.

 Andrei

 I'm not advocating it. I don't use it myself. I'm just saying that I
 find it to be less of an issue than synchronizing on arbitrary objects,
 because the latter is error-prone.

 And yes, Python is a textbook example of synchronization gone completely
 wrong. But that doesn't mean that you don't sometimes have to
 synchronize on some global resource, and in those cases, it can be
 useful syntactic sugar (but certainly not essential).

I have to agree with Andrei here. The global mutex isn't the solution 
either. The mutex should be explicit, and under control (not this).

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 11:15, deadalnix wrote:
 Le 30/05/2012 01:25, Alex Rønne Petersen a écrit :
 On 30-05-2012 01:22, Andrei Alexandrescu wrote:
 On 5/29/12 4:17 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 01:10, Andrei Alexandrescu wrote:
 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

 I think there's quite some disconnect here. If there's any
 anti-pattern
 in this discussion, it's operating on an implicit, hidden, global
 mutex.
 Walter agreed to eliminate that from D, but never got around to it.

 Andrei

 I'd love to hear why you think this design is problematic as opposed to
 one that lets users accidentally expose synchronization issues to
 consumers of their API surface, which is what many people end up doing
 since synchronized (this) or even synchronized (this.classinfo) are
 allowed at all.

 (I've seen countless cases of those two horrible abuses of synchronized
 especially from people asking questions on e.g. IRC.)

 I think the most egregious example is synchronization by global lock
 done in Python and other languages. It has very nice semantics (stop the
 world, do something, resume the world), but scales poorly enough to be
 universally considered a failure. Quite honest I'm shocked that you're
 advocating it.

 Andrei

 I'm not advocating it. I don't use it myself. I'm just saying that I
 find it to be less of an issue than synchronizing on arbitrary objects,
 because the latter is error-prone.

 And yes, Python is a textbook example of synchronization gone completely
 wrong. But that doesn't mean that you don't sometimes have to
 synchronize on some global resource, and in those cases, it can be
 useful syntactic sugar (but certainly not essential).

 I have to agree with Andrei here. The global mutex isn't the solution
 either. The mutex should be explicit, and under control (not this).

I've been arguing for being explicit all along. :)

What I've been saying in this particular part of the thread is just that 
synchronized blocks that don't operate on a specific monitor object are 
less error-prone. Nothing else.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

"Martin Nowak" <dawg dawgfoto.de> writes:

On Wed, 30 May 2012 01:10:41 +0200, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/29/12 4:06 PM, Alex R=C3=B8nne Petersen wrote:
  Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with=


  =

 that.

 I think there's quite some disconnect here. If there's any anti-patter=

n  =

 in this discussion, it's operating on an implicit, hidden, global mute=

x.  =

 Walter agreed to eliminate that from D, but never got around to it.

They're not really global, it's one per synchronized block.
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/cr=
itical_.d
They're actually pretty limited/safe because you can't access/compose th=
e  =

underlying lock.

import std.stdio, core.thread;

void foo()
{
     synchronized
     {
         auto t =3D new Thread(&bar);
         t.start();
         t.join();
         writeln("foo");
     }
}

void bar()
{
     synchronized writeln("bar");
}

void main()
{
     foo();
}

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 10:20 AM, Martin Nowak wrote:
 On Wed, 30 May 2012 01:10:41 +0200, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

 I think there's quite some disconnect here. If there's any
 anti-pattern in this discussion, it's operating on an implicit,
 hidden, global mutex. Walter agreed to eliminate that from D, but
 never got around to it.

 They're not really global, it's one per synchronized block.

That means global. They're terrible and should be eliminated from the 
language.


Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 19:22, Andrei Alexandrescu wrote:
 On 5/30/12 10:20 AM, Martin Nowak wrote:
 On Wed, 30 May 2012 01:10:41 +0200, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

 I think there's quite some disconnect here. If there's any
 anti-pattern in this discussion, it's operating on an implicit,
 hidden, global mutex. Walter agreed to eliminate that from D, but
 never got around to it.

 They're not really global, it's one per synchronized block.

 That means global. They're terrible and should be eliminated from the
 language.


 Andrei

So you do agree that explicit synchronization is better? Or only in this 
particular case?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 10:34 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 19:22, Andrei Alexandrescu wrote:
 On 5/30/12 10:20 AM, Martin Nowak wrote:
 On Wed, 30 May 2012 01:10:41 +0200, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/29/12 4:06 PM, Alex Rønne Petersen wrote:
 Synchronized blocks are good because they
 operate on an implicit, hidden, global mutex. You can't screw up with
 that.

 I think there's quite some disconnect here. If there's any
 anti-pattern in this discussion, it's operating on an implicit,
 hidden, global mutex. Walter agreed to eliminate that from D, but
 never got around to it.

 They're not really global, it's one per synchronized block.

 That means global. They're terrible and should be eliminated from the
 language.


 Andrei

 So you do agree that explicit synchronization is better?

No. It's worse most of the time, and usefully more flexible sometimes.

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 3:01 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to
 repeat
 previously done errors.

 So what is the lesson Java learned, and how does it address
 multithreaded programming in wake of that lesson?

 Andrei

 It learned that allowing locking on arbitrary objects makes controlling
 locking (and thus reducing the chance for deadlocks) impossible.

 And how does Java address multithreading in general, and those issues in
 particular, today?

 Andrei



 figures in those programming language communities have written blog
 posts on why these language constructs are evil and should be avoided.

Some citations (beyond the known fallacies of Java 1.0) would be great. 
Thanks.

 Besides, it seems to me that D can't quite make up its mind. We have TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

Each paradigm has its place. Lock-based programming is definitely here 
to stay, and when the paradigm is needed, doing it with synchronized 
objects is better than most alternatives.


Andrei

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 00:53, Andrei Alexandrescu a écrit :
 On 5/29/12 3:01 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to
 repeat
 previously done errors.

 So what is the lesson Java learned, and how does it address
 multithreaded programming in wake of that lesson?

 Andrei

 It learned that allowing locking on arbitrary objects makes controlling
 locking (and thus reducing the chance for deadlocks) impossible.

 And how does Java address multithreading in general, and those issues in
 particular, today?

 Andrei



 figures in those programming language communities have written blog
 posts on why these language constructs are evil and should be avoided.

 Some citations (beyond the known fallacies of Java 1.0) would be great.
 Thanks.

 Besides, it seems to me that D can't quite make up its mind. We have TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

 Each paradigm has its place. Lock-based programming is definitely here
 to stay, and when the paradigm is needed, doing it with synchronized
 objects is better than most alternatives.


 Andrei

No !

You don't want to synchronize on ANY object. You want to synchronize on 
explicit mutexes.

See that link for instance : 
http://msdn.microsoft.com/en-us/library/ms173179.aspx


"Generally, it is best to avoid locking on a public type, or on object 
instances beyond the control of your application. For example, 
lock(this) can be problematic if the instance can be accessed publicly, 
because code beyond your control may lock on the object as well. This 
could create deadlock situations where two or more threads wait for the 
release of the same object. Locking on a public data type, as opposed to 
an object, can cause problems for the same reason."

Given example also create dumb object to lock on them instead of using 
this/any object.

Time has proven it is the way to go.

I have tracked race condition in large java codebase, and I conclude the 
same thing as microsoft's guys.

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com> wrot=
e:

 Le 30/05/2012 00:53, Andrei Alexandrescu a =E9crit :
 On 5/29/12 3:01 PM, Alex R=F8nne Petersen wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex R=F8nne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex R=F8nne Petersen a =E9crit :
 I should probably add that Java learned it long ago, and yet we=








 adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to=







 repeat
 previously done errors.

 So what is the lesson Java learned, and how does it address
 multithreaded programming in wake of that lesson?

 Andrei

 It learned that allowing locking on arbitrary objects makes  =





 controlling
 locking (and thus reducing the chance for deadlocks) impossible.

 And how does Java address multithreading in general, and those issu=




es  =

 in
 particular, today?

 Andrei






 figures in those programming language communities have written blog
 posts on why these language constructs are evil and should be avoide=



d.
 Some citations (beyond the known fallacies of Java 1.0) would be grea=


t.
 Thanks.

 Besides, it seems to me that D can't quite make up its mind. We have=



  =

 TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribut=



e.
 It just seems so incredibly inconsistent. synchronized encourages do=



ing
 the wrong thing (locks and synchronization).

 Each paradigm has its place. Lock-based programming is definitely her=


e
 to stay, and when the paradigm is needed, doing it with synchronized
 objects is better than most alternatives.


 Andrei

 No !

 You don't want to synchronize on ANY object. You want to synchronize o=

n  =

 explicit mutexes.

+1 .. this is the key point/issue.  If all objects can be locked, and if=
  =

synchronized classes/methods use the /same/ mutex you can  =

easily/accidentally have hard to find deadlocks.  The deadlocks in Q are=
  =

usually (at least) two threads locking (at least) two objects in the  =

opposite order and with synchronized classes/methods the fact that a loc=
k  =

is being taken is not always obvious from the code (at the call site) so=
  =

the problem code can look fairly innocuous.

**[Example 1]**

class C
{
   synchronized void ccc() { }
}

class D
{
   synchronized void ddd() { }
}

C c =3D new C();
D d =3D new D();

[thread1]
..lock(c)  // locks c
{          // deadlock window
   d.ddd(); // locks d
}

[thread2]
..lock(d)  // locks d
{          // deadlock window
   c.ccc(); // locks c
}

**[Example 2]**

class A
{
   B b;
   ..
   synchronized void foo()  // locks a
   {                        // deadlock window
     b.bar();               // locks b
   }
}

class B
{
   A a;
   ..
   synchronized void bar()  // locks b
   {                        // deadlock window
     a.foo();               // locks a
   }
}

A a =3D new A();
B b =3D new B();

a.b =3D b;
b.a =3D a;

[thread1]
a.foo(); // locks a then b

[thread2]
b.bar(); // locks b then a

So, the root cause of the problem is that synchronized classes/methods u=
se  =

a publicly visible mutex (the object/this pointer) to perform their  =

locking.  The solution is to lock on a private mutex, which no-one else =
 =

can lock unexpectedly as described in the link below:

 See that link for instance :  =

 http://msdn.microsoft.com/en-us/library/ms173179.aspx

Now, ideally we want to make it hard for people to screw this up in this=
  =

manner and there are several ways to do it.

1. Prevent locking on any/every object.  People would have to create a  =

separate mutex object for locking.  This is a little tedious, but it doe=
s  =

make people think about the scope of the locking (where to put the mutex=
,  =

who can see/use it, etc).  On the flipside it can make people re-use a  =

mutex for multiple conceptual critical section areas of code, which is  =

less than ideal, but at least it's not 'broken'.

2. Allow locking on any/every object but use a different mutex for  =

synchronized class/methods.  So 'synchronized' on a method call locks a =
 =

private mutex object, not the object itself.  And synchronized(object)  =

locks the public mutex object.  The downside here is that now every obje=
ct  =

potentially has 2 mutex objects/handles internally - a public and a  =

private one.



especially as thread-local is the default storage type now.  100% of  =

objects in a single threaded application don't need a built-in mutex, mo=
st  =

objects (80%??) in a threaded application will be thread-local and don't=
  =

need one.  Only those few shared objects, which are actually manually  =

locked with synchronized or have synchronized class/methods actually nee=
d  =

a mutex.  It seems that removing this will save memory and avoid tricky =
 =

deadlock bugs and the cost is having to manually create a mutex object  =

(1/2 lines of code) to lock instead.. it seems like a good idea to me.

R

-- =

Using Opera's revolutionary email client: http://www.opera.com/mail/

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 14:32, Regan Heath a écrit :
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com> wrote:

 Le 30/05/2012 00:53, Andrei Alexandrescu a écrit :
 On 5/29/12 3:01 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to
 repeat
 previously done errors.

 So what is the lesson Java learned, and how does it address
 multithreaded programming in wake of that lesson?

 Andrei

 It learned that allowing locking on arbitrary objects makes
 controlling
 locking (and thus reducing the chance for deadlocks) impossible.

 And how does Java address multithreading in general, and those
 issues in
 particular, today?

 Andrei



 figures in those programming language communities have written blog
 posts on why these language constructs are evil and should be avoided.

 Some citations (beyond the known fallacies of Java 1.0) would be great.
 Thanks.

 Besides, it seems to me that D can't quite make up its mind. We have
 TLS
 by default, and we encourage message-passing (through a library
 mechanism), and then we have the synchronized statement and attribute.
 It just seems so incredibly inconsistent. synchronized encourages doing
 the wrong thing (locks and synchronization).

 Each paradigm has its place. Lock-based programming is definitely here
 to stay, and when the paradigm is needed, doing it with synchronized
 objects is better than most alternatives.


 Andrei

 No !

 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

 +1 .. this is the key point/issue. If all objects can be locked, and if
 synchronized classes/methods use the /same/ mutex you can
 easily/accidentally have hard to find deadlocks. The deadlocks in Q are
 usually (at least) two threads locking (at least) two objects in the
 opposite order and with synchronized classes/methods the fact that a
 lock is being taken is not always obvious from the code (at the call
 site) so the problem code can look fairly innocuous.

 **[Example 1]**

 class C
 {
 synchronized void ccc() { }
 }

 class D
 {
 synchronized void ddd() { }
 }

 C c = new C();
 D d = new D();

 [thread1]
 ..lock(c) // locks c
 { // deadlock window
 d.ddd(); // locks d
 }

 [thread2]
 ..lock(d) // locks d
 { // deadlock window
 c.ccc(); // locks c
 }

 **[Example 2]**

 class A
 {
 B b;
 ..
 synchronized void foo() // locks a
 { // deadlock window
 b.bar(); // locks b
 }
 }

 class B
 {
 A a;
 ..
 synchronized void bar() // locks b
 { // deadlock window
 a.foo(); // locks a
 }
 }

 A a = new A();
 B b = new B();

 a.b = b;
 b.a = a;

 [thread1]
 a.foo(); // locks a then b

 [thread2]
 b.bar(); // locks b then a

 So, the root cause of the problem is that synchronized classes/methods
 use a publicly visible mutex (the object/this pointer) to perform their
 locking. The solution is to lock on a private mutex, which no-one else
 can lock unexpectedly as described in the link below:

 See that link for instance :
 http://msdn.microsoft.com/en-us/library/ms173179.aspx

 Now, ideally we want to make it hard for people to screw this up in this
 manner and there are several ways to do it.

 1. Prevent locking on any/every object. People would have to create a
 separate mutex object for locking. This is a little tedious, but it does
 make people think about the scope of the locking (where to put the
 mutex, who can see/use it, etc). On the flipside it can make people
 re-use a mutex for multiple conceptual critical section areas of code,
 which is less than ideal, but at least it's not 'broken'.

It is suboptimal, but correct.

 2. Allow locking on any/every object but use a different mutex for
 synchronized class/methods. So 'synchronized' on a method call locks a
 private mutex object, not the object itself. And synchronized(object)
 locks the public mutex object. The downside here is that now every
 object potentially has 2 mutex objects/handles internally - a public and
 a private one.

It doesn't address most of the drawback cited. Notably the fact that 
every object have a monitor field, but most of them will not use it.

"Thiez" <thiezz gmail.com> writes:

On Wednesday, 30 May 2012 at 12:43:15 UTC, deadalnix wrote:
 It doesn't address most of the drawback cited. Notably the fact 
 that every object have a monitor field, but most of them will 
 not use it.

That could be solved without abandoning object locks. If locking 
on all objects is allowed, objects in TLS could simply omit the 
monitor field. Locking would then be performed as:

if (object not in TLS) {
   insert expensive locking operation here
}
return

Yes, this would introduce an extra conditional jump whenever a 
lock is acquired/released, but the overhead is very small 
compared to the relatively expensive lock/unlock operation.

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 14:54, Thiez wrote:
 On Wednesday, 30 May 2012 at 12:43:15 UTC, deadalnix wrote:
 It doesn't address most of the drawback cited. Notably the fact that
 every object have a monitor field, but most of them will not use it.

 That could be solved without abandoning object locks. If locking on all
 objects is allowed, objects in TLS could simply omit the monitor field.
 Locking would then be performed as:

 if (object not in TLS) {
 insert expensive locking operation here
 }
 return

 Yes, this would introduce an extra conditional jump whenever a lock is
 acquired/released, but the overhead is very small compared to the
 relatively expensive lock/unlock operation.

I don't understand how that would work. Where would you store the 
monitor reference?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 13:43:14 +0100, deadalnix <deadalnix gmail.com> wrot=
e:
 Le 30/05/2012 14:32, Regan Heath a =E9crit :
 1. Prevent locking on any/every object. People would have to create a=


 separate mutex object for locking. This is a little tedious, but it d=


oes
 make people think about the scope of the locking (where to put the
 mutex, who can see/use it, etc). On the flipside it can make people
 re-use a mutex for multiple conceptual critical section areas of code=


,
 which is less than ideal, but at least it's not 'broken'.

 It is suboptimal, but correct.

Indeed, and can be made optimal with some fine-tuning/work.

 2. Allow locking on any/every object but use a different mutex for
 synchronized class/methods. So 'synchronized' on a method call locks =


a
 private mutex object, not the object itself. And synchronized(object)=


 locks the public mutex object. The downside here is that now every
 object potentially has 2 mutex objects/handles internally - a public =


and
 a private one.

 It doesn't address most of the drawback cited. Notably the fact that  =

 every object have a monitor field, but most of them will not use it.

True, I was just mentioning it as an option that solves the key  =


  =

better.

Ultimately the "problem" that needs solving is the fact that synchronize=
d  =

classes/methods use a public mutex and this exposure makes deadlocks mor=
e  =

likely.  I was hoping that by giving some examples in code, it would  =

become clearer to the people who don't see what all the fuss is about.

Yes, having to manually create a mutex is a pain, and as Andrei has  =

mentioned a few times having 2 separate things (one object to lock, one =
 =

mutex) which conceptually should be one thing is worse from an  =

understanding/maintenance point of view, however..

1. If the mutex is intended to lock a single object then we can make the=
  =

following options available:
   a. A Lockable class to derive from.
   b. A Lockable struct to compose with (assuming here that the struct w=
ill  =

be created/destroyed with the object).
   c. A Lockable!(T) wrapper template/struct/class to wrap objects to be=
  =

locked.
   d. A Lockable interface which classes may implement.
   e. Some sort of compiler magic where it supplies the mutex for  =

objects/methods marked with "synchronized" or involved in a  =

synchronized(object) statement.

2. If the scope of the locking is greater than a single object, then a  =

separate mutex is required/desired anyway.

1a. would contain a mutex primitive and lock/tryLock/unlock methods (it =
 =

would implement the interface in 1d)

1b. would also contain a mutex primitive and alias could be used to expo=
se  =

the lock/tryLock/unlock methods (it would implement the interface in 1d)=


1c. is actually a technique I have used a lot in C++ where the template =
 =

class is created on the stack of a function/critical section and it  =

(optionally) locks on creation (or later when lock() is called) and alwa=
ys  =

unlocks on destruction as it leaves scope (by any means).  It's a very  =

robust pattern, and is similar (the inspiration/or result) of/to the  =

synchronized block itself.

1d. could be a requirement for classes which are to be used in  =

synchronized statements i.e.

class A : Lockable
{
   void lock() {}
   void unlock() {}
}

A a =3D new A();
synchronized(a)
{ // calls a.lock()
   ...
} // calls a.unlock()

1e. I leave as a opener to Walter/Andrei/the reader .. I've seen many a =
 =

genius idea pop out of nowhere on this forum.

IMO, the wasted space used by the monitor isn't ideal, but it's not a  =

"problem" for the general case.  It is a problem for people with limited=
  =

memory environments but that's another topic.  Talking about both is onl=
y  =

clouding the discussional waters so to speak.

R

-- =

Using Opera's revolutionary email client: http://www.opera.com/mail/

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 15:10, Regan Heath a écrit :
 IMO, the wasted space used by the monitor isn't ideal, but it's not a
 "problem" for the general case. It is a problem for people with limited
 memory environments but that's another topic. Talking about both is only
 clouding the discussional waters so to speak.

 R

As the GC allocate by power of 2, this can go pretty bad. It did happen 
to me (once to be fair, it is probably not the every day problem) that 
my memory consumption exploded because of the monitor field.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 6:10 AM, Regan Heath wrote:
 Ultimately the "problem" that needs solving is the fact that
 synchronized classes/methods use a public mutex and this exposure makes
 deadlocks more likely. I was hoping that by giving some examples in
 code, it would become clearer to the people who don't see what all the
 fuss is about.

How do free-floating mutexes that are not explicitly associated with 
_any_ data make deadlocks less likely? In my experience the less 
structured mutexes are worse, not better.

 Yes, having to manually create a mutex is a pain, and as Andrei has
 mentioned a few times having 2 separate things (one object to lock, one
 mutex) which conceptually should be one thing is worse from an
 understanding/maintenance point of view, however..

 1. If the mutex is intended to lock a single object then we can make the
 following options available:
 a. A Lockable class to derive from.
 b. A Lockable struct to compose with (assuming here that the struct will
 be created/destroyed with the object).
 c. A Lockable!(T) wrapper template/struct/class to wrap objects to be
 locked.
 d. A Lockable interface which classes may implement.
 e. Some sort of compiler magic where it supplies the mutex for
 objects/methods marked with "synchronized" or involved in a
 synchronized(object) statement.

 2. If the scope of the locking is greater than a single object, then a
 separate mutex is required/desired anyway.

But all of these are available in the current design. You have 
synchronized classes for the likes of 1a and the core mutex for creating 
all others except e.

 1a. would contain a mutex primitive and lock/tryLock/unlock methods (it
 would implement the interface in 1d)

 1b. would also contain a mutex primitive and alias could be used to
 expose the lock/tryLock/unlock methods (it would implement the interface
 in 1d)

 1c. is actually a technique I have used a lot in C++ where the template
 class is created on the stack of a function/critical section and it
 (optionally) locks on creation (or later when lock() is called) and
 always unlocks on destruction as it leaves scope (by any means). It's a
 very robust pattern, and is similar (the inspiration/or result) of/to
 the synchronized block itself.

Yah, we use it a lot at Facebook. We probably should have such a wrapper 
in the standard library.

 IMO, the wasted space used by the monitor isn't ideal, but it's not a
 "problem" for the general case. It is a problem for people with limited
 memory environments but that's another topic. Talking about both is only
 clouding the discussional waters so to speak.

I agree.


Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 30-05-2012 15:10, Regan Heath wrote:
 On Wed, 30 May 2012 13:43:14 +0100, deadalnix <deadalnix gmail.com> wrote:
 Le 30/05/2012 14:32, Regan Heath a écrit :
 1. Prevent locking on any/every object. People would have to create a
 separate mutex object for locking. This is a little tedious, but it does
 make people think about the scope of the locking (where to put the
 mutex, who can see/use it, etc). On the flipside it can make people
 re-use a mutex for multiple conceptual critical section areas of code,
 which is less than ideal, but at least it's not 'broken'.

 It is suboptimal, but correct.

 Indeed, and can be made optimal with some fine-tuning/work.

 2. Allow locking on any/every object but use a different mutex for
 synchronized class/methods. So 'synchronized' on a method call locks a
 private mutex object, not the object itself. And synchronized(object)
 locks the public mutex object. The downside here is that now every
 object potentially has 2 mutex objects/handles internally - a public and
 a private one.

 It doesn't address most of the drawback cited. Notably the fact that
 every object have a monitor field, but most of them will not use it.

 True, I was just mentioning it as an option that solves the key

 better.

 Ultimately the "problem" that needs solving is the fact that
 synchronized classes/methods use a public mutex and this exposure makes
 deadlocks more likely. I was hoping that by giving some examples in
 code, it would become clearer to the people who don't see what all the
 fuss is about.

 Yes, having to manually create a mutex is a pain, and as Andrei has
 mentioned a few times having 2 separate things (one object to lock, one
 mutex) which conceptually should be one thing is worse from an
 understanding/maintenance point of view, however..

 1. If the mutex is intended to lock a single object then we can make the
 following options available:
 a. A Lockable class to derive from.
 b. A Lockable struct to compose with (assuming here that the struct will
 be created/destroyed with the object).
 c. A Lockable!(T) wrapper template/struct/class to wrap objects to be
 locked.
 d. A Lockable interface which classes may implement.
 e. Some sort of compiler magic where it supplies the mutex for
 objects/methods marked with "synchronized" or involved in a
 synchronized(object) statement.

 2. If the scope of the locking is greater than a single object, then a
 separate mutex is required/desired anyway.

 1a. would contain a mutex primitive and lock/tryLock/unlock methods (it
 would implement the interface in 1d)

 1b. would also contain a mutex primitive and alias could be used to
 expose the lock/tryLock/unlock methods (it would implement the interface
 in 1d)

 1c. is actually a technique I have used a lot in C++ where the template
 class is created on the stack of a function/critical section and it
 (optionally) locks on creation (or later when lock() is called) and
 always unlocks on destruction as it leaves scope (by any means). It's a
 very robust pattern, and is similar (the inspiration/or result) of/to
 the synchronized block itself.

In D, you can kind of already do that:

void foo()
{
     mtx.lock();
     scope (exit) mtx.unlock();

     // ...
}

It does involve that one extra scope (exit) line, but I'm of the opinion 
that being explicit about that is a good thing.

 1d. could be a requirement for classes which are to be used in
 synchronized statements i.e.

 class A : Lockable
 {
 void lock() {}
 void unlock() {}
 }

 A a = new A();
 synchronized(a)
 { // calls a.lock()
 ...
 } // calls a.unlock()

 1e. I leave as a opener to Walter/Andrei/the reader .. I've seen many a
 genius idea pop out of nowhere on this forum.

 IMO, the wasted space used by the monitor isn't ideal, but it's not a
 "problem" for the general case. It is a problem for people with limited
 memory environments but that's another topic. Talking about both is only
 clouding the discussional waters so to speak.

 R


-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 5:43 AM, deadalnix wrote:
 It doesn't address most of the drawback cited. Notably the fact that
 every object have a monitor field, but most of them will not use it.

Once the TDPL design is implemented, we could look into eliminating that 
field for all but "synchronized" class. In the TDPL design, 
"synchronized" is a class-level attribute.

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com> wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

 +1 .. this is the key point/issue.

TDPL's design only allows for entire synchronized classes (not separate 
synchronized and unsynchronized methods), which pair mutexes with the 
data they protect. This is more restrictive than exposing mutexes, but 
in a good way. We use such a library artifact in C++ at Facebook all the 
time, to great success.

People shouldn't create designs that have synchronized classes referring 
to one another naively. Designing with mutexes (explicit or implicit) 
will always create the possibility of deadlock, so examples how that 
could happen are easy to come across.

TDPL improves on deadlocks by introducing synchronized statements with 
more than one argument, see 13.15.

The implicit mutexes used by synchronized classes are recursive.


Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>  
 wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

 +1 .. this is the key point/issue.

 TDPL's design only allows for entire synchronized classes (not separate  
 synchronized and unsynchronized methods), which pair mutexes with the  
 data they protect. This is more restrictive than exposing mutexes, but  
 in a good way. We use such a library artifact in C++ at Facebook all the  
 time, to great success.

Can you call pass them to a synchronized statement?  i.e.

TDPLStyleSynchClass a = new TDPLStyleSynchClass();
synchronized(a) {
}

.. because, if you can, then you're exposing the mutex.  If you can't, how  
do you do the above with these classes?  By explicitly calling a lock() or  
tryLock() method?  (do they implement a known interface?)

 People shouldn't create designs that have synchronized classes referring  
 to one another naively. Designing with mutexes (explicit or implicit)  
 will always create the possibility of deadlock, so examples how that  
 could happen are easy to come across.

True.  But in my Example 1 the classes could be "entire" synchronized  
classes, and they do not refer to each other naively.  Instead, two  
threads have shared access to them and in each case they explicitly  
acquire one mutex (via a synchronized() statement), but implicitly acquire  
another (by calling a method).  It's the implicit acquisition which makes  
the bug hard to see and easy to do accidentally.

 TDPL improves on deadlocks by introducing synchronized statements with  
 more than one argument, see 13.15.

Is there anywhere I can see this online? (for free :p)

 The implicit mutexes used by synchronized classes are recursive.

:) why would you want anything else.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 9:43 AM, Regan Heath wrote:
 On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>
 wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

 +1 .. this is the key point/issue.

 TDPL's design only allows for entire synchronized classes (not
 separate synchronized and unsynchronized methods), which pair mutexes
 with the data they protect. This is more restrictive than exposing
 mutexes, but in a good way. We use such a library artifact in C++ at
 Facebook all the time, to great success.

 Can you call pass them to a synchronized statement? i.e.

 TDPLStyleSynchClass a = new TDPLStyleSynchClass();
 synchronized(a) {
 }

Yes. Well I recommend acquiring the text! :o)

 ... because, if you can, then you're exposing the mutex.

No.

 People shouldn't create designs that have synchronized classes
 referring to one another naively. Designing with mutexes (explicit or
 implicit) will always create the possibility of deadlock, so examples
 how that could happen are easy to come across.

 True. But in my Example 1

Your example 1 should not compile.

 TDPL improves on deadlocks by introducing synchronized statements with
 more than one argument, see 13.15.

 Is there anywhere I can see this online? (for free :p)

http://goo.gl/ZhPM2

 The implicit mutexes used by synchronized classes are recursive.

 :) why would you want anything else.

Efficiency.


Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 18:16:38 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 9:43 AM, Regan Heath wrote:
 On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>
 wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

 +1 .. this is the key point/issue.

 TDPL's design only allows for entire synchronized classes (not
 separate synchronized and unsynchronized methods), which pair mutexes
 with the data they protect. This is more restrictive than exposing
 mutexes, but in a good way. We use such a library artifact in C++ at
 Facebook all the time, to great success.

 Can you call pass them to a synchronized statement? i.e.

 TDPLStyleSynchClass a = new TDPLStyleSynchClass();
 synchronized(a) {
 }

 Yes. Well I recommend acquiring the text! :o)

 ... because, if you can, then you're exposing the mutex.

 No.

For the purposes of this thread, and the anti-pattern/problem we're  
discussing, you are.  It is the combination of synchronized  
classes/methods (implicit locking) and external synchronized statements  
(explicit locking) which result in the unexpected, accidental, and hard to  
see deadlocks we're talking about here.

 People shouldn't create designs that have synchronized classes
 referring to one another naively. Designing with mutexes (explicit or
 implicit) will always create the possibility of deadlock, so examples
 how that could happen are easy to come across.

 True. But in my Example 1

 Your example 1 should not compile.

o.. k.. I expected you would get my meaning with the simplified example.   
Here you are:

import core.thread;
import std.random;
import std.stdio;

class C
{
   synchronized void ccc() { writefln("c.ccc()"); }
}

class D
{
   synchronized void ddd() { writefln("d.ddd()"); }
}

shared C c;
shared D d;

void main()
{
   c = new shared(C)();
   d = new shared(D)();
   Thread thread1 = new Thread( &threadFunc1 );
   Thread thread2 = new Thread( &threadFunc2 );
   thread1.start();
   thread2.start();
   Thread.sleep(dur!("seconds")(60));
}

void threadFunc1()
{
   Random gen = rndGen();

   while(1)
   {
     synchronized(c)
     {
       printf("threadFunc1 sync(c) %p, obtain d.. %p\n", c, d);
       d.ddd();
     }
	
     Thread.sleep(dur!("usecs")(gen.front()));
     gen.popFront();
   }
}

void threadFunc2()
{
   Random gen = rndGen();

   while(1)
   {
     synchronized(d)
     {
       printf("threadFunc1 sync(d) %p, obtain c.. %p\n", d, c);
       c.ccc();
     }
	
     Thread.sleep(dur!("usecs")(gen.front()));
     gen.popFront();
   }
}

Runs and deadlocks immediately.

 TDPL improves on deadlocks by introducing synchronized statements with
 more than one argument, see 13.15.

 Is there anywhere I can see this online? (for free :p)

 http://goo.gl/ZhPM2

Thanks.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 10:40 AM, Regan Heath wrote:
 On Wed, 30 May 2012 18:16:38 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 9:43 AM, Regan Heath wrote:
 On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>
 wrote:
 You don't want to synchronize on ANY object. You want to synchronize
 on explicit mutexes.

 +1 .. this is the key point/issue.

 TDPL's design only allows for entire synchronized classes (not
 separate synchronized and unsynchronized methods), which pair mutexes
 with the data they protect. This is more restrictive than exposing
 mutexes, but in a good way. We use such a library artifact in C++ at
 Facebook all the time, to great success.

 Can you call pass them to a synchronized statement? i.e.

 TDPLStyleSynchClass a = new TDPLStyleSynchClass();
 synchronized(a) {
 }

 Yes. Well I recommend acquiring the text! :o)

 ... because, if you can, then you're exposing the mutex.

 No.

 For the purposes of this thread, and the anti-pattern/problem we're
 discussing, you are.

No. I explained in my previous post that the synchronized statement does 
not expose locks. This is not a matter of opinion.

 It is the combination of synchronized
 classes/methods (implicit locking) and external synchronized statements
 (explicit locking) which result in the unexpected, accidental, and hard
 to see deadlocks we're talking about here.

You can have deadlocks but with synchronized you can't leak locks or 
doubly-unlock them. With free mutexes you have all of the above.

 People shouldn't create designs that have synchronized classes
 referring to one another naively. Designing with mutexes (explicit or
 implicit) will always create the possibility of deadlock, so examples
 how that could happen are easy to come across.

 True. But in my Example 1

 Your example 1 should not compile.

 o.. k.. I expected you would get my meaning with the simplified example.
 Here you are:

[snip]

 Runs and deadlocks immediately.

Sure. As I said, synchronized helps with scoping the locks and unlocks, 
but not with deadlocks. You can rewrite the example with two bare 
mutexes just as well.



Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 10:40 AM, Regan Heath wrote:
 On Wed, 30 May 2012 18:16:38 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 9:43 AM, Regan Heath wrote:
 On Wed, 30 May 2012 17:00:43 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 5:32 AM, Regan Heath wrote:
 On Wed, 30 May 2012 10:21:00 +0100, deadalnix <deadalnix gmail.com>
 wrote:
 You don't want to synchronize on ANY object. You want to  
 synchronize
 on explicit mutexes.

 +1 .. this is the key point/issue.

 TDPL's design only allows for entire synchronized classes (not
 separate synchronized and unsynchronized methods), which pair mutexes
 with the data they protect. This is more restrictive than exposing
 mutexes, but in a good way. We use such a library artifact in C++ at
 Facebook all the time, to great success.

 Can you call pass them to a synchronized statement? i.e.

 TDPLStyleSynchClass a = new TDPLStyleSynchClass();
 synchronized(a) {
 }

 Yes. Well I recommend acquiring the text! :o)

 ... because, if you can, then you're exposing the mutex.

 No.

 For the purposes of this thread, and the anti-pattern/problem we're
 discussing, you are.

 No. I explained in my previous post that the synchronized statement does  
 not expose locks. This is not a matter of opinion.

Well, it seems some people disagree here.  You are exposing the ability to  
lock and unlock them, which is the cause of more frequent deadlocks (this  
assertion comes from the M$ article, my experience with them and it seems  
 from other peoples experience of them as well).  You're not exposing the  
mutex primitive in any other way, but that's irrelevant for this  
discussion.  Now, you and I have probably done enough multi-threaded  
programming with mutexes that we no longer fall into this trap easily, but  
someone just starting in the world of threading/locking is going to,  
repeatedly.  If we make it impossible to do it "by default", so they have  
to think about when/where to apply the mutex, and make the code doing the  
locking more obvious, then I think we'd go a long way to making it much  
nicer.

 It is the combination of synchronized
 classes/methods (implicit locking) and external synchronized statements
 (explicit locking) which result in the unexpected, accidental, and hard
 to see deadlocks we're talking about here.

 You can have deadlocks but with synchronized you can't leak locks or  
 doubly-unlock them. With free mutexes you have all of the above.

I'm not suggesting using free mutexes.  I'm suggesting keeping the mutex  
private inside the object.  Private in this case means cannot be locked by  
external code.  Which basically means synchronized() is a no go "by  
default".  This will avoid all the careless accidental deadlock cases  
involving synchronized w/ synchronized classes/methods and make people  
actually think about when/where to use their mutexes, instead of just  
throwing synchronized all about the place.  It is sad, because I really  
like the scopedness of synchronized but I think it encourages lazy  
programming in this case.  Note that I said "by default" above, we can  
definitely provide library interfaces, structs, classes, etc to make  
synchronized work and if we did it would become a conscious choice the  
programmer makes to use them, and that should hopefully (because I cannot  
predict the future) mean less accidental deadlocks.

 People shouldn't create designs that have synchronized classes
 referring to one another naively. Designing with mutexes (explicit or
 implicit) will always create the possibility of deadlock, so examples
 how that could happen are easy to come across.

 True. But in my Example 1

 Your example 1 should not compile.

 o.. k.. I expected you would get my meaning with the simplified example.
 Here you are:

 [snip]

 Runs and deadlocks immediately.

 Sure. As I said, synchronized helps with scoping the locks and unlocks,  
 but not with deadlocks. You can rewrite the example with two bare  
 mutexes just as well.

Again, I'm not suggesting using "bare" or "free" mutexes.  I'm suggesting  
keeping the mutex private, so that it cannot be locked without it being  
obvious that that's whats happening.  If it were more obvious there would  
be less chance of an accidental deadlock, and more chance people would  
actually use an idea like an "atomic lock of multiple sync primitives"  
(your synchronized with multiple mutexes and lock ordering), where  
possible.  Sadly even that idea only works when both locks are taken in  
the same method/function and only if they can and should be taken at the  
same time .. which I think is a fairly rare case.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

 I'm not suggesting using free mutexes. I'm suggesting keeping the mutex
 private inside the object.

Ergo, you are suggesting using free mutexes. Your second sentence 
destroys the first.


Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 10:48:51 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

 I'm not suggesting using free mutexes. I'm suggesting keeping the mutex
 private inside the object.

 Ergo, you are suggesting using free mutexes. Your second sentence  
 destroys the first.

Depends on your definition of "free".  You appear to have meant as an  
instance/pointer/object even one in a class, I initially read it as  
meaning as a separate object from the class you're locking.  In any case,  
you're right the compiler doesn't get synchronized()  
statements/classes/methods wrong and a programmer can.  The trade-off is  
the cause of this thread of discussion.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

deadalnix <deadalnix gmail.com> writes:

Le 31/05/2012 11:55, Regan Heath a écrit :
 On Thu, 31 May 2012 10:48:51 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

 I'm not suggesting using free mutexes. I'm suggesting keeping the mutex
 private inside the object.

 Ergo, you are suggesting using free mutexes. Your second sentence
 destroys the first.

 Depends on your definition of "free". You appear to have meant as an
 instance/pointer/object even one in a class, I initially read it as
 meaning as a separate object from the class you're locking. In any case,
 you're right the compiler doesn't get synchronized()
 statements/classes/methods wrong and a programmer can. The trade-off is
 the cause of this thread of discussion.

 R

I think you misunderstand each other. Here is what I nderstand :

Andrei is talking about free mutex in the sense of a mutex with lock and 
unlock function available. Such a mutex is error prone, because you can 
lock/unlock in an incorrect way.

synchronized is here a better alternative, because lock and unlock 
always goes by pair.

Regan, by free mutex, you mean a mutex that can be locked and unlocked 
by everybody. This is problematic too.

The ideal is to lock in a way that isn't a free mutex in both definition 
of the term.

Sean Kelly <sean invisibleduck.org> writes:

On May 31, 2012, at 2:48 AM, Andrei Alexandrescu <SeeWebsiteForEmail erdani.=
org> wrote:

 On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

=20
 I'm not suggesting using free mutexes. I'm suggesting keeping the mutex
 private inside the object.

=20
 Ergo, you are suggesting using free mutexes. Your second sentence destroys=

 the first.

To be fair:

auto m =3D new Mutex;
synchronized (m) {...}

Free mutexes but still safe. Scope guards obviously work too. That said, I t=
hink the point of contention here is that because synchronized can take an a=
rbitrary object, it's possible to lock on a class for stuff completely unrel=
ated to what the class does internally. It's certainly bad form though, and I=
 don't know a good way to prevent it without crippling synchronized.=20=

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Thu, 31 May 2012 11:39:06 -0400, Sean Kelly <sean invisibleduck.org>  
wrote:

 On May 31, 2012, at 2:48 AM, Andrei Alexandrescu  
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:36 AM, Regan Heath wrote:
 On Wed, 30 May 2012 19:29:39 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 You can have deadlocks but with synchronized you can't leak locks or
 doubly-unlock them. With free mutexes you have all of the above.

 I'm not suggesting using free mutexes. I'm suggesting keeping the mutex
 private inside the object.

 Ergo, you are suggesting using free mutexes. Your second sentence  
 destroys the first.

 To be fair:

 auto m = new Mutex;
 synchronized (m) {...}

 Free mutexes but still safe. Scope guards obviously work too. That said,  
 I think the point of contention here is that because synchronized can  
 take an arbitrary object, it's possible to lock on a class for stuff  
 completely unrelated to what the class does internally. It's certainly  
 bad form though, and I don't know a good way to prevent it without  
 crippling synchronized.

See my suggestion and Dmitry's in other parts of this thread.  The idea is  
that you still allow synchronized(x), but only if the author of x allows  
it.

If we use lowering, and make the lock and unlock methods of some entity,  
then you can also use attributes to limit scope of synchronized.

For example, if we make synchronized(x) equivalent to calling x.lock() and  
x.unlock() correctly, if lock() and unlock() are private, then you can  
only call synchronized(x) from inside the instance.

-Steve

"SomeDude" <lovelydear mailmetrash.com> writes:

On Tuesday, 29 May 2012 at 22:01:50 UTC, Alex Rønne Petersen 
wrote:
 On 29-05-2012 23:54, Andrei Alexandrescu wrote:
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:



 prominent figures in those programming language communities 
 have written blog posts on why these language constructs are 
 evil and should be avoided.

 Besides, it seems to me that D can't quite make up its mind. We 
 have TLS by default, and we encourage message-passing (through 
 a library mechanism), and then we have the synchronized 
 statement and attribute. It just seems so incredibly 
 inconsistent. synchronized encourages doing the wrong thing 
 (locks and synchronization).

I can't remember having read those blog posts. I agree message 
passing is a better abstraction, but I haven't read Josh Bloch, 
for example, saying "don't use synchronized".

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 23:54, Andrei Alexandrescu a écrit :
 On 5/29/12 2:49 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:32, Andrei Alexandrescu wrote:
 On 5/29/12 1:35 AM, deadalnix wrote:
 Le 29/05/2012 01:38, Alex Rønne Petersen a écrit :
 I should probably add that Java learned it long ago, and yet we
 adopted
 it anyway... blergh.

 That is what I was about to say. No point of doing D if it is to repeat
 previously done errors.

 So what is the lesson Java learned, and how does it address
 multithreaded programming in wake of that lesson?

 Andrei

 It learned that allowing locking on arbitrary objects makes controlling
 locking (and thus reducing the chance for deadlocks) impossible.

 And how does Java address multithreading in general, and those issues in
 particular, today?

 Andrei

In java, you basically have no concurency built-in. Everything is shared 
by default, you can lock on anything (which is deadlock and liquid lock 
prone), you have dumb primitive like wait and notify that are perfect to 
cause race conditions.

Don't get me wrong, I do a lot of java code, and it isn't that bad in 
general. But for concurrency, this isn't the model we want.

In a more general scope, the problem of java is to believe that 
everything is OOP, including concurrency, when they are, in fact, 
different topics.

D already have much better tools that the one java provide 
(std.concurency, std.parallelism, TLS by default, transitive type 
qualifiers, . . .) that most these thing taken from java don't make any 
sense now.

For instance, what is the point of being able to lock on any object when 
most of them are thread local ??

mta`chrono <chrono mta-international.net> writes:

Am 30.05.2012 11:11, schrieb deadalnix:
 
 D already have much better tools that the one java provide
 (std.concurency, std.parallelism, TLS by default, transitive type
 qualifiers, . . .) that most these thing taken from java don't make any
 sense now.
 
 For instance, what is the point of being able to lock on any object when
 most of them are thread local ??

Right! Locking on non-TLS objects doesn't make sense. Perhaps only
shared objects should be synchronizeable and thus contain a monitor /
pointer to a monitor.

Jonathan M Davis <jmdavisProg gmx.com> writes:

On Tuesday, May 29, 2012 01:35:23 Alex R=C3=B8nne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way too m=

uch
 code would break if we changed the semantics. I'd consider it a mista=

ke
 and a lesson learned, rather.
=20
 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

I'm not an expert on threading stuf, but it would be my opinion that if=
 you're=20
not intending to protect the entire class with locks that it makes no s=
ense to=20
lock on the class itself. You're locking for something specific, in whi=
ch case,=20
your sychronized block should be locking on something else specific to =
what=20
you're trying to protect. Certainly, that's how I'd approach it with mu=
texes.=20
You don't have a mutex for an entire class unless it's actually used fo=
r all=20
of the class' functions. Rather, you use mutexes specific to what you'r=
e trying=20
to protect.

- Jonathan M Davis

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 01:41, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:35:23 Alex Rønne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way too much
 code would break if we changed the semantics. I'd consider it a mistake
 and a lesson learned, rather.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

 I'm not an expert on threading stuf, but it would be my opinion that if you're
 not intending to protect the entire class with locks that it makes no sense to
 lock on the class itself. You're locking for something specific, in which case,
 your sychronized block should be locking on something else specific to what
 you're trying to protect. Certainly, that's how I'd approach it with mutexes.
 You don't have a mutex for an entire class unless it's actually used for all
 of the class' functions. Rather, you use mutexes specific to what you're trying
 to protect.

 - Jonathan M Davis

Right, but even if you really *are* protecting the entire class, you can 
still create mysterious deadlocks if users of your code lock on your 
class. So I'm arguing that no matter the use case, never lock on a 
'this' reference exposed outside of some API layer.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Jonathan M Davis <jmdavisProg gmx.com> writes:

On Tuesday, May 29, 2012 01:54:59 Alex R=C3=B8nne Petersen wrote:
 On 29-05-2012 01:41, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:35:23 Alex R=C3=B8nne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way to=



o much
 code would break if we changed the semantics. I'd consider it a mi=



stake
 and a lesson learned, rather.
=20
 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

=20
 I'm not an expert on threading stuf, but it would be my opinion tha=


t if
 you're not intending to protect the entire class with locks that it=


 makes
 no sense to lock on the class itself. You're locking for something
 specific, in which case, your sychronized block should be locking o=


n
 something else specific to what you're trying to protect. Certainly=


,
 that's how I'd approach it with mutexes. You don't have a mutex for=


 an
 entire class unless it's actually used for all of the class' functi=


ons.
 Rather, you use mutexes specific to what you're trying to protect.
=20
 - Jonathan M Davis

=20
 Right, but even if you really *are* protecting the entire class, you =

can
 still create mysterious deadlocks if users of your code lock on your
 class. So I'm arguing that no matter the use case, never lock on a
 'this' reference exposed outside of some API layer.

Well, it seems pretty abysmal to me to be locking on something that you=
 don't=20
control. Making a mutex that your class used public would just be stupi=
d. With=20
synchronized classes/functions, you're basically creating and using an=20=

implicit mutex for the whole class, which would then be the same as if =
you=20
locked it at the beginning of every member function call and unlocked i=
t at=20
its end, which doesn't expose the mutex at all. So, I don't really see =
te=20
problem there would have to study the matter more to see what the issue=
 there=20
is. But locking on another class rather than something specifically int=
ended as=20
a mutex does seem to me like it's asking for trouble. But maybe that mi=
ndset=20
comes from using mutexes primarily as opposed to synchronized statement=
s (as=20
most of that sort of programming that I've done has been in C++), and I=
 don't=20
necessarily understand all of the ways that programmers screw up with=20=

synchronized blocks.

- Jonathan M Davis

"Regan Heath" <regan netmail.co.nz> writes:

On Tue, 29 May 2012 01:08:59 +0100, Jonathan M Davis <jmdavisProg gmx.co=
m>  =

wrote:

 On Tuesday, May 29, 2012 01:54:59 Alex R=F8nne Petersen wrote:
 On 29-05-2012 01:41, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:35:23 Alex R=F8nne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way to=




o  =

 much
 code would break if we changed the semantics. I'd consider it a  =




 mistake
 and a lesson learned, rather.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

 I'm not an expert on threading stuf, but it would be my opinion tha=



t  =

 if
 you're not intending to protect the entire class with locks that it=



  =

 makes
 no sense to lock on the class itself. You're locking for something
 specific, in which case, your sychronized block should be locking o=



n
 something else specific to what you're trying to protect. Certainly=



,
 that's how I'd approach it with mutexes. You don't have a mutex for=



 an
 entire class unless it's actually used for all of the class'  =



 functions.
 Rather, you use mutexes specific to what you're trying to protect.

 - Jonathan M Davis

 Right, but even if you really *are* protecting the entire class, you =


can
 still create mysterious deadlocks if users of your code lock on your
 class. So I'm arguing that no matter the use case, never lock on a
 'this' reference exposed outside of some API layer.

 Well, it seems pretty abysmal to me to be locking on something that yo=

u  =

 don't control. Making a mutex that your class used public would just b=

e  =

 stupid.

Interestingly this is what C#s Array type does with SyncRoot  =

(intentionally).

 With synchronized classes/functions, you're basically creating and usi=

ng  =

 an
 implicit mutex for the whole class, which would then be the same as if=

  =

 you locked it at the beginning of every member function call and  =

 unlocked it at its end, which doesn't expose the mutex at all. So, I  =

 don't really see te
 problem there would have to study the matter more to see what the issu=

e  =

 there is.

According to the docs here:
http://dlang.org/class.html#synchronized-functions

A synchronized function locks "this" and as "this" is exposed publicly..=
.   =

In the following code the "lock" statement and "synchronized void bar"  =

lock the same mutex.

class Foo {
   synchronized void bar() { ...statements... }
}

void main()
{
   Foo foo =3D new Foo();
   lock(foo)
   {
     ...statements...
   }
}

 But locking on another class rather than something specifically intend=

ed  =

 as a mutex does seem to me like it's asking for trouble.

Yep.  It commonly arises where you have a class/object which is either n=
ot  =

synchronized itself (because it might be used in single threaded  =

situations and you want performance) like a collection class for example=
,  =

or is synchronized but you need to lock a sequence of member function  =

calls i.e. a lookup and insert on the collection.  What happens then, is=
  =

people just call lock(<object>) and end up locking the same mutex as the=
  =

class does internally.  What happens next to cause a deadlock is that  =

inside that lock they call one or more functions or methods which  =

internally call methods on another synchronized object.  This results in=
 a  =

locking pattern of object1, object2.  In another piece of code, running =
in  =

another thread, they do something similar which results in a locking  =

pattern of object2, object1 and eventually these threads will deadlock  =

each other.

R

-- =

Using Opera's revolutionary email client: http://www.opera.com/mail/

"Regan Heath" <regan netmail.co.nz> writes:

On Tue, 29 May 2012 13:07:29 +0100, Regan Heath <regan netmail.co.nz>  
wrote:
 void main()
 {
    Foo foo = new Foo();
    lock(foo)
    {
      ...statements...
    }
 }


be "synchronized" for D.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 29.05.2012 16:07, Regan Heath wrote:
 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly... In the following code the "lock" statement and "synchronized
 void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.


I'd be darned but every Object in D has monitor fields. If I'm not 
mistaken it's the mutex you are looking for ;)


-- 
Dmitry Olshansky

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 29-05-2012 14:19, Dmitry Olshansky wrote:
 On 29.05.2012 16:07, Regan Heath wrote:
 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly... In the following code the "lock" statement and "synchronized
 void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.


 I'd be darned but every Object in D has monitor fields. If I'm not
 mistaken it's the mutex you are looking for ;)

Indeed they do, and therefore each object must eat an entire word of 
memory for questionable gain.

Generalized object monitors is the worst idea in programming language 
and virtual machine design, ever.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 29.05.2012 16:26, Alex Rønne Petersen wrote:
 On 29-05-2012 14:19, Dmitry Olshansky wrote:
 On 29.05.2012 16:07, Regan Heath wrote:
 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly... In the following code the "lock" statement and "synchronized
 void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.


 I'd be darned but every Object in D has monitor fields. If I'm not
 mistaken it's the mutex you are looking for ;)

 Indeed they do, and therefore each object must eat an entire word of
 memory for questionable gain.

 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

Agreed, awfuly crippled design for a language with Thread-local by default.
Looks like we have 'oh my god, what were they thinking' moment here.
If anything I'd rather re-implement the whole v-table infrastructure via 
mixins, templates & friends.

-- 
Dmitry Olshansky

Jacob Carlborg <doob me.com> writes:

On 2012-05-29 14:29, Dmitry Olshansky wrote:

 Agreed, awfuly crippled design for a language with Thread-local by default.
 Looks like we have 'oh my god, what were they thinking' moment here.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

I'm pretty sure that was decided long before D 1.0 and way way longer 
before D2 where TLS was introduced.

-- 
/Jacob Carlborg

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 29.05.2012 17:59, Jacob Carlborg wrote:
 On 2012-05-29 14:29, Dmitry Olshansky wrote:

 Agreed, awfuly crippled design for a language with Thread-local by
 default.
 Looks like we have 'oh my god, what were they thinking' moment here.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

 I'm pretty sure that was decided long before D 1.0 and way way longer
 before D2 where TLS was introduced.

I know. If anything it's hardly a good excuse, it should have been 
revisited once TLS was introduced. It still can be now.

-- 
Dmitry Olshansky

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 12:41 PM, Dmitry Olshansky wrote:
 On 29.05.2012 17:59, Jacob Carlborg wrote:
 On 2012-05-29 14:29, Dmitry Olshansky wrote:

 Agreed, awfuly crippled design for a language with Thread-local by
 default.
 Looks like we have 'oh my god, what were they thinking' moment here.
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

 I'm pretty sure that was decided long before D 1.0 and way way longer
 before D2 where TLS was introduced.

 I know. If anything it's hardly a good excuse, it should have been
 revisited once TLS was introduced. It still can be now.

Don't be hatin'. You'd appreciate the matter considerably more after you 
defined your own language and had its users yell at you for changing 
even a small detail.

The situation is at it is, and there's no need to get agitated. We're 
all on the same boat, trying to make things work out best.


Andrei

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 1:48, Andrei Alexandrescu wrote:

 I know. If anything it's hardly a good excuse, it should have been
 revisited once TLS was introduced. It still can be now.

 Don't be hatin'. You'd appreciate the matter considerably more after you
 defined your own language and had its users yell at you for changing
 even a small detail.

 The situation is at it is, and there's no need to get agitated. We're
 all on the same boat, trying to make things work out best.

Yeah, no problem. I mean I wasn't bashing D for bugs or flaws (before 
this point at least) anyway. And it's not like I'm cursing here ;)

I just wish it was different. To set things straight I still believe 
that OOP part of language is not what I want it to be, and thus I just 
don't use it. Like in the old saying: there is always a choice.

Things that stopped me from using it are:
- hidden v-table design like C++ and Java, doesn't help things much. 
Encapsulation is good, but if it has no advantages other then hiding 
things it's bad.
- GCed by default, no provision for custom allocation until very recently
- monitor field, and who knows what else (v-table obviously, maybe 
something else?)
- no tail-const, no solutions aside from rebindable and casts. Pointers 
to struct look so much better in this regard ;)
- special slow built-in protocols for equality (it's robust, but I don't 
need it)
- opEquals signature madness (probably fixed?)
- final vs virtual, no inlining of known ahead-of-time virtual call
- nowdays enforced purity,  safe, nothrow whatever. Why should I follow 
these restrictions - dubious.

Observe that all of the above have very few advantages brought to me by 
compiler:
- polymorphism
- inheritance
- contract inheritance

Of the above I only ever needed the first 2. Call me dinosaur.

-- 
Dmitry Olshansky

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 2:27, Dmitry Olshansky wrote:
 I just wish it was different. To set things straight I still believe
 that OOP part of language is not what I want it to be, and thus I just
 don't use it. Like in the old saying: there is always a choice.

Minor correction, when I first come to try D I used classes, I confess 
:). The toy project of 2D graphics engine needed resource manager, thus 
I used Flyweight pattern with lazy-loading/unloding behind the scenes. 
The concrete resource were struct on C heap but handles were  small 
objects were derived from Resource fro obvious reasons.

Brings us back to extra word per object problem, flyweight is exactly 
kind of pattern that is hit by collateral damage of this decision!

OT: Actually algorithms to lazily cache resources are very similar to 
virtual memory management, same strategies, same action just replace 
"swap-out page" with "unload resource".

TL;DR all of this worked out quite well then I lost interest in the 
project but certainly not D itself.

 Things that stopped me from using it are:
 - hidden v-table design like C++ and Java, doesn't help things much.
 Encapsulation is good, but if it has no advantages other then hiding
 things it's bad.
 - GCed by default, no provision for custom allocation until very recently
 - monitor field, and who knows what else (v-table obviously, maybe
 something else?)
 - no tail-const, no solutions aside from rebindable and casts. Pointers
 to struct look so much better in this regard ;)
 - special slow built-in protocols for equality (it's robust, but I don't
 need it)
 - opEquals signature madness (probably fixed?)
 - final vs virtual, no inlining of known ahead-of-time virtual call
 - nowdays enforced purity,  safe, nothrow whatever. Why should I follow
 these restrictions - dubious.

 Observe that all of the above have very few advantages brought to me by
 compiler:
 - polymorphism
 - inheritance
 - contract inheritance

 Of the above I only ever needed the first 2. Call me dinosaur.

Throwing in this one:
	-useless RTTi that still occupies space.
(the useless part might have been fixed recently? Some customizable 
field added to Typeinfo - dunno)

-- 
Dmitry Olshansky

Jacob Carlborg <doob me.com> writes:

On 2012-05-29 23:48, Andrei Alexandrescu wrote:

 Don't be hatin'. You'd appreciate the matter considerably more after you
 defined your own language and had its users yell at you for changing
 even a small detail.

 The situation is at it is, and there's no need to get agitated. We're
 all on the same boat, trying to make things work out best.

It seems more and more that D2 is not a designed language. Instead new 
features are just slapped on without considering how it would impact the 
rest of the language.

-- 
/Jacob Carlborg

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 2:14 AM, Jacob Carlborg wrote:
 On 2012-05-29 23:48, Andrei Alexandrescu wrote:

 Don't be hatin'. You'd appreciate the matter considerably more after you
 defined your own language and had its users yell at you for changing
 even a small detail.

 The situation is at it is, and there's no need to get agitated. We're
 all on the same boat, trying to make things work out best.

 It seems more and more that D2 is not a designed language. Instead new
 features are just slapped on without considering how it would impact the
 rest of the language.

What features are you referring to?

Andrei

"Jacob Carlborg" <doob me.com> writes:

On Wednesday, 30 May 2012 at 15:45:05 UTC, Andrei Alexandrescu 
wrote:
 On 5/30/12 2:14 AM, Jacob Carlborg wrote:
 It seems more and more that D2 is not a designed language. 
 Instead new
 features are just slapped on without considering how it would 
 impact the
 rest of the language.

 What features are you referring to?

The concurrency model as this thread shows. There is no bridge 
between shared and unshared data like const is to immutable and 
mutable. Perhaps the monitor on every object should have been 
removed when the new concurrency model was designed, as this 
thread suggests.

"inout" was added long after the const system was added to D. If 
done correctly this should have come up as a problem when 
designing the const system. You cannot apply const/immutable to 
an object reference in the same way as you can to a pointer.

It took some pretty good convincing for you to accept that 
strings weren't enough as a substitute for lambdas. Fortunately 
we have proper lambdas now :)

"const" doesn't play nice with ranges.

All these points have been mentioned before.

--
/Jacob Carlborg

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 11:41 AM, Jacob Carlborg wrote:
 On Wednesday, 30 May 2012 at 15:45:05 UTC, Andrei Alexandrescu wrote:
 On 5/30/12 2:14 AM, Jacob Carlborg wrote:
 It seems more and more that D2 is not a designed language. Instead new
 features are just slapped on without considering how it would impact the
 rest of the language.

 What features are you referring to?

 The concurrency model as this thread shows. There is no bridge between
 shared and unshared data like const is to immutable and mutable.

We considered that (maybe_synchronized) , but decided not to go with it 
amid fear of overcomplicating things.

 Perhaps
 the monitor on every object should have been removed when the new
 concurrency model was designed, as this thread suggests.

This thread does a good job at arguing that scoped locking does not 
prevent deadlocks, but this is not new or interesting. Unfortunately I 
fail to derive significant proposed value.

There exist type systems that avoid locks. They are very restrictive and 
difficult to work with.

 "inout" was added long after the const system was added to D. If done
 correctly this should have come up as a problem when designing the const
 system. You cannot apply const/immutable to an object reference in the
 same way as you can to a pointer.

 "const" doesn't play nice with ranges.

I see how these can be annoying, but they're not the result of us not 
designing things. We designed things best we could.


Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 21:10, Andrei Alexandrescu wrote:
 On 5/30/12 11:41 AM, Jacob Carlborg wrote:
 On Wednesday, 30 May 2012 at 15:45:05 UTC, Andrei Alexandrescu wrote:
 On 5/30/12 2:14 AM, Jacob Carlborg wrote:
 It seems more and more that D2 is not a designed language. Instead new
 features are just slapped on without considering how it would impact
 the
 rest of the language.

 What features are you referring to?

 The concurrency model as this thread shows. There is no bridge between
 shared and unshared data like const is to immutable and mutable.

 We considered that (maybe_synchronized) , but decided not to go with it
 amid fear of overcomplicating things.

The result is a feature that is arguably only useful in small laboratory 
cases. Are you sure we shouldn't revisit shared and try to work out a 
bridge between the unshared and shared world?

Not to mention that shared is fundamentally x86-biased, which seems to 
be annoying trend in D development in general...

 Perhaps
 the monitor on every object should have been removed when the new
 concurrency model was designed, as this thread suggests.

 This thread does a good job at arguing that scoped locking does not
 prevent deadlocks, but this is not new or interesting. Unfortunately I
 fail to derive significant proposed value.

 There exist type systems that avoid locks. They are very restrictive and
 difficult to work with.

I can think of 3 languages that have monitors in the "type" system: D, 

interoperability), and Java. Are you really saying all other type 
systems are difficult to work with just because of this little thing?

 "inout" was added long after the const system was added to D. If done
 correctly this should have come up as a problem when designing the const
 system. You cannot apply const/immutable to an object reference in the
 same way as you can to a pointer.

 "const" doesn't play nice with ranges.

 I see how these can be annoying, but they're not the result of us not
 designing things. We designed things best we could.


 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 12:17 PM, Alex Rønne Petersen wrote:
 The result is a feature that is arguably only useful in small laboratory
 cases. Are you sure we shouldn't revisit shared and try to work out a
 bridge between the unshared and shared world?

We're as always open to suggestions.

 There exist type systems that avoid locks. They are very restrictive and
 difficult to work with.

 I can think of 3 languages that have monitors in the "type" system: D,

 interoperability), and Java. Are you really saying all other type
 systems are difficult to work with just because of this little thing?

Apologies, meant "avoid deadlocks" instead of "avoid locks".


Andrei

Jacob Carlborg <doob me.com> writes:

On 2012-05-30 21:10, Andrei Alexandrescu wrote:

 I see how these can be annoying, but they're not the result of us not
 designing things. We designed things best we could.

I would say it's not good enough. The whole approach of designing the 
language is wrong. This is how it works today, which is bad:

1. Designing feature X
2. Implementing in the compiler
3. Shipped with the next release

Andrei, Walter: "Hey look at this new feature X in the next release, 
it's great".

Community: "Say what now. What did that come from?".

Some time later

Community: "X is broken in these different ways. X can't integrate with 
Y, Z, W. Since Phobos doesn't use the feature we can't use it, making it 
useless"
Andrei, Walter: "No, it's perfectly designed"
Community: "But it's not working in practice"
Andrei, Walter: "Sure it is, end of discussion"

This is the bad approach, but there's also a worse approach:

1. Designing feature X
2. Document the feature in TDPL
3. Wait wait wait
4. Community: "Where is feature X, it's in TDPL"
5. Andrei, Walter: "It's not implemented yet/correctly"
6. Repeat step 3-5 a couple of times
7. Implement feature X in the compiler
8. Ship with the next release

Community: "X is broken in these different ways. X can't integrate with 
Y, Z, W. Since Phobos doesn't use the feature we can't use it, making it 
useless"
Andrei, Walter: "No, it's perfectly designed"
Community: "But it's not working in practice"
Andrei, Walter: "It can't be changed, it's already in TDPL, it's written 
in stone"

What should have been done is something like this:

1. Designing feature X
2. Show the new feature for the community
3. Consider the feedback and possible tweak/redesign
4. Implementing in an experimental branch of the compiler
5. Release an experimental version with just this feature
6. Repeat step 3-5 until satisfied or put on hold/drop the idea
7. Prepare Phobos and druntime for the new feature
8. Move the implementation to the main branch
9. Ship feature X with the next release
10. wait
11. Fix bugs for feature X
12. Repeat step 10-11 a couple of times
13. Write about it in TDPL

-- 
/Jacob Carlborg

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 11:47 PM, Jacob Carlborg wrote:
 On 2012-05-30 21:10, Andrei Alexandrescu wrote:

 I see how these can be annoying, but they're not the result of us not
 designing things. We designed things best we could.

 I would say it's not good enough. The whole approach of designing the
 language is wrong.

I understand how frustrating this is. In fact even the way you consider 
"good" is not nearly good enough. What we need is really more 
formalization of the language design, something that we're sorely 
missing. I am sometimes frustrated out of my mind at the lack of rigor 
and discipline in the process. On the other hand, we march with the 
troops we have.


Andrei

"foobar" <foo bar.com> writes:

On Thursday, 31 May 2012 at 08:01:14 UTC, Andrei Alexandrescu 
wrote:
 On 5/30/12 11:47 PM, Jacob Carlborg wrote:
 On 2012-05-30 21:10, Andrei Alexandrescu wrote:

 I see how these can be annoying, but they're not the result 
 of us not
 designing things. We designed things best we could.

 I would say it's not good enough. The whole approach of 
 designing the
 language is wrong.

 I understand how frustrating this is. In fact even the way you 
 consider "good" is not nearly good enough. What we need is 
 really more formalization of the language design, something 
 that we're sorely missing. I am sometimes frustrated out of my 
 mind at the lack of rigor and discipline in the process. On the 
 other hand, we march with the troops we have.


 Andrei

Please no. This is how C++ is designed and we all know how fucked 
up that is.

Writing a [rigorous] spec is almost always incorrect since 
requirements change and unforeseen things come about. Jacob's 
post illustrates this when the spec is written [in TDPL] before 
implementing, testing and integrating it.

By making a rigorous spec you exacerbate the problem - it takes 
more time to write such a spec thus making the time-frame for 
unforeseen changes larger.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 2:12 AM, foobar wrote:
 On Thursday, 31 May 2012 at 08:01:14 UTC, Andrei Alexandrescu wrote:
 On 5/30/12 11:47 PM, Jacob Carlborg wrote:
 On 2012-05-30 21:10, Andrei Alexandrescu wrote:

 I see how these can be annoying, but they're not the result of us not
 designing things. We designed things best we could.

 I would say it's not good enough. The whole approach of designing the
 language is wrong.

 I understand how frustrating this is. In fact even the way you
 consider "good" is not nearly good enough. What we need is really more
 formalization of the language design, something that we're sorely
 missing. I am sometimes frustrated out of my mind at the lack of rigor
 and discipline in the process. On the other hand, we march with the
 troops we have.


 Andrei

 Please no. This is how C++ is designed and we all know how fucked up
 that is.

Not at all. This is either a misunderstanding, or you lack the faintest 
idea about the history of C++.

 Writing a [rigorous] spec is almost always incorrect since requirements
 change and unforeseen things come about. Jacob's post illustrates this
 when the spec is written [in TDPL] before implementing, testing and
 integrating it.

 By making a rigorous spec you exacerbate the problem - it takes more
 time to write such a spec thus making the time-frame for unforeseen
 changes larger.

No.


Andrei

deadalnix <deadalnix gmail.com> writes:

Le 31/05/2012 11:24, Andrei Alexandrescu a écrit :
 By making a rigorous spec you exacerbate the problem - it takes more
 time to write such a spec thus making the time-frame for unforeseen
 changes larger.

 No.

Can you elaborate on what you are thinking as a process ?

Jacob Carlborg <doob me.com> writes:

On 2012-05-31 10:01, Andrei Alexandrescu wrote:

 I understand how frustrating this is. In fact even the way you consider
 "good" is not nearly good enough. What we need is really more
 formalization of the language design, something that we're sorely
 missing. I am sometimes frustrated out of my mind at the lack of rigor
 and discipline in the process. On the other hand, we march with the
 troops we have.

Yeah, "good" should have been "better". I'm not sure what you mean with 
"we march with the troops we have" but it would be fairly easy to at 
least improve the process somewhat. It would require any extra manpower. 
It's just that you, Walter and the community needs to be willing to do 
the changes.

-- 
/Jacob Carlborg

mta`chrono <chrono mta-international.net> writes:

Am 31.05.2012 08:47, schrieb Jacob Carlborg:
 What should have been done is something like this:
 
 1. Designing feature X
 2. Show the new feature for the community
 3. Consider the feedback and possible tweak/redesign
 4. Implementing in an experimental branch of the compiler
 5. Release an experimental version with just this feature
 6. Repeat step 3-5 until satisfied or put on hold/drop the idea
 7. Prepare Phobos and druntime for the new feature
 8. Move the implementation to the main branch
 9. Ship feature X with the next release
 10. wait
 11. Fix bugs for feature X
 12. Repeat step 10-11 a couple of times
 13. Write about it in TDPL
 

+1

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 5:29 AM, Dmitry Olshansky wrote:
 Agreed, awfuly crippled design for a language with Thread-local by default.
 Looks like we have 'oh my god, what were they thinking' moment here.

The synchronized class feature was copied by Walter from Java while he 
at the time had only little understanding of multithreading. The 
thread-local by default notion was added much later.

Indeed we'd design things very differently if synchronized came about 
later. The design of "synchronized" described in TDPL and not yet 
implemented is a conciliation of the two.

 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

Could you please elaborate how that would help multithreading?


Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 29-05-2012 23:41, Andrei Alexandrescu wrote:
 On 5/29/12 5:29 AM, Dmitry Olshansky wrote:
 Agreed, awfuly crippled design for a language with Thread-local by
 default.
 Looks like we have 'oh my god, what were they thinking' moment here.

 The synchronized class feature was copied by Walter from Java while he
 at the time had only little understanding of multithreading. The
 thread-local by default notion was added much later.

 Indeed we'd design things very differently if synchronized came about
 later. The design of "synchronized" described in TDPL and not yet
 implemented is a conciliation of the two.

 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

 Could you please elaborate how that would help multithreading?


 Andrei

The whole concept of synchronization needs to be ripped out of Object 
with *extreme prejudice*. We have core.sync.mutex for a reason.

I think the vtable infrastructure as templates/mixins is an orthogonal 
point.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 2:50 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:41, Andrei Alexandrescu wrote:
 On 5/29/12 5:29 AM, Dmitry Olshansky wrote:
 Agreed, awfuly crippled design for a language with Thread-local by
 default.
 Looks like we have 'oh my god, what were they thinking' moment here.

 The synchronized class feature was copied by Walter from Java while he
 at the time had only little understanding of multithreading. The
 thread-local by default notion was added much later.

 Indeed we'd design things very differently if synchronized came about
 later. The design of "synchronized" described in TDPL and not yet
 implemented is a conciliation of the two.

 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

 Could you please elaborate how that would help multithreading?


 Andrei

 The whole concept of synchronization needs to be ripped out of Object
 with *extreme prejudice*. We have core.sync.mutex for a reason.

That's more of a restating than an elaboration.

Andrei

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 1:41, Andrei Alexandrescu wrote:
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

 Could you please elaborate how that would help multithreading?

It's unrelated to "ease multithreading part strictly speaking.
My observation is that it leads to at least removal of 1 word per 
object. Not a small thing if you happen to use GC, that "sells his 
memory" in 16 byte chunks, chances are you'd waste 2 instead of one.

Again strictly speaking I'm of an opinion that having mutex together 
with object guarded by it is at least better then 2 separate entities 
bound together by virtue of code comments :)
In any case if mutex is desired, object could have had some other base 
type say SyncObject. Or use "synchronized class" to that end, what it 
does now by the way - locks on each method?

More about the actual point is that I've come to believe that there is 
satisfactory way to implement whatever scheme of polymorphism* we want 
within the language on top of structs without 'class' and 'interface' 
keywords, special "flawed" pointer type (i.e. tail-const anyone?), and 
last but not least without new/delete/finalizes (new/delete are still 
overridable, hint-hint) madness.

Ideally I think it should be possible to lower the whole 
interface/object/class infrastructure to code that uses structs with 
direct function pointer tables, etc. Multiple alias this is the key, 
sadly so, otherwise subtyping to multiple interfaces seem not likely.
Then some future compiler may even chose to not provide OOP as built-in 
but lower to this manual implementation on top of struct(!).

*I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as 
ordinal (index inside one global master v-table) instead of pointless 
_hidden_ v-table pointer could be interesting in certain designs. 
Another idea is to try tackling multi-methods via some form of 
compressed 2-stage v-table. (my recent work on generalized tries in D 
sparked some ideas)

-- 
Dmitry Olshansky

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 3:06 PM, Dmitry Olshansky wrote:
 On 30.05.2012 1:41, Andrei Alexandrescu wrote:
 If anything I'd rather re-implement the whole v-table infrastructure via
 mixins, templates & friends.

 Could you please elaborate how that would help multithreading?

 It's unrelated to "ease multithreading part strictly speaking.
 My observation is that it leads to at least removal of 1 word per
 object. Not a small thing if you happen to use GC, that "sells his
 memory" in 16 byte chunks, chances are you'd waste 2 instead of one.

So there is concern about the word per object wasted by the possible 
mutex. I understand.

 Again strictly speaking I'm of an opinion that having mutex together
 with object guarded by it is at least better then 2 separate entities
 bound together by virtue of code comments :)

Absolutely. I hope you agree that this essentially means you're 
advocating a Java-style approach in which the mutex is implicitly present...

 In any case if mutex is desired, object could have had some other base
 type say SyncObject.

... albeit not in all objects, only a subhierarchy thereof. I posted the 
same thing. Nice :o).

 Or use "synchronized class" to that end, what it does now by the way
 - locks on each method?

TDPL's design of synchronized still hasn't been implemented. The design 
indeed prescribes that all public access to the resource is synchronized.

 More about the actual point is that I've come to believe that there is
 satisfactory way to implement whatever scheme of polymorphism* we want
 within the language on top of structs without 'class' and 'interface'
 keywords, special "flawed" pointer type (i.e. tail-const anyone?), and
 last but not least without new/delete/finalizes (new/delete are still
 overridable, hint-hint) madness.

 Ideally I think it should be possible to lower the whole
 interface/object/class infrastructure to code that uses structs with
 direct function pointer tables, etc. Multiple alias this is the key,
 sadly so, otherwise subtyping to multiple interfaces seem not likely.
 Then some future compiler may even chose to not provide OOP as built-in
 but lower to this manual implementation on top of struct(!).

Well all of these are nice thoughts but at some point we must 
acknowledge we're operating within the confines of an already-defined 
language.

 *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as
 ordinal (index inside one global master v-table) instead of pointless
 _hidden_ v-table pointer could be interesting in certain designs.
 Another idea is to try tackling multi-methods via some form of
 compressed 2-stage v-table. (my recent work on generalized tries in D
 sparked some ideas)

Any post that starts with taking an issue against the waste of one word 
and ends advocating Smalltalk and Obj-C is... ho-hum.


Andrei

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 3:02, Andrei Alexandrescu wrote:
 On 5/29/12 3:06 PM, Dmitry Olshansky wrote:
 Again strictly speaking I'm of an opinion that having mutex together
 with object guarded by it is at least better then 2 separate entities
 bound together by virtue of code comments :)

 Absolutely. I hope you agree that this essentially means you're
 advocating a Java-style approach in which the mutex is implicitly
 present...

 In any case if mutex is desired, object could have had some other base
 type say SyncObject.

 ... albeit not in all objects, only a subhierarchy thereof. I posted the
 same thing. Nice :o).

Great.

 Or use "synchronized class" to that end, what it does now by the way
 - locks on each method?

 TDPL's design of synchronized still hasn't been implemented. The design
 indeed prescribes that all public access to the resource is synchronized.

So sad, I recall when I was reading about it it made a lot of sense.

 More about the actual point is that I've come to believe that there is
 satisfactory way to implement whatever scheme of polymorphism* we want
 within the language on top of structs without 'class' and 'interface'
 keywords, special "flawed" pointer type (i.e. tail-const anyone?), and
 last but not least without new/delete/finalizes (new/delete are still
 overridable, hint-hint) madness.

 Ideally I think it should be possible to lower the whole
 interface/object/class infrastructure to code that uses structs with
 direct function pointer tables, etc. Multiple alias this is the key,
 sadly so, otherwise subtyping to multiple interfaces seem not likely.
 Then some future compiler may even chose to not provide OOP as built-in
 but lower to this manual implementation on top of struct(!).

 Well all of these are nice thoughts but at some point we must
 acknowledge we're operating within the confines of an already-defined
 language.

 *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as
 ordinal (index inside one global master v-table) instead of pointless
 _hidden_ v-table pointer could be interesting in certain designs.
 Another idea is to try tackling multi-methods via some form of
 compressed 2-stage v-table. (my recent work on generalized tries in D
 sparked some ideas)

 Any post that starts with taking an issue against the waste of one word
 and ends advocating Smalltalk and Obj-C is... ho-hum.

Well going from practical matters to personal dreams is remarkably easy 
at 2 AM :)


-- 
Dmitry Olshansky

Jacob Carlborg <doob me.com> writes:

On 2012-05-30 00:06, Dmitry Olshansky wrote:

 *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as
 ordinal (index inside one global master v-table) instead of pointless
 _hidden_ v-table pointer could be interesting in certain designs.
 Another idea is to try tackling multi-methods via some form of
 compressed 2-stage v-table. (my recent work on generalized tries in D
 sparked some ideas)

I like that classes are first class citizens in Objective-C and Ruby. 
Class methods instead of static methods. Easily extendable from outside 
the class. But I still like to have the "class" and "interface" keywords.

Would this solve the problem with const:

https://github.com/D-Programming-Language/dmd/pull/3

-- 
/Jacob Carlborg

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 13:28, Jacob Carlborg wrote:
 On 2012-05-30 00:06, Dmitry Olshansky wrote:

 *I like the one in Smalltalk or Obj-C. Also I think exposing type-tag as
 ordinal (index inside one global master v-table) instead of pointless
 _hidden_ v-table pointer could be interesting in certain designs.
 Another idea is to try tackling multi-methods via some form of
 compressed 2-stage v-table. (my recent work on generalized tries in D
 sparked some ideas)

 I like that classes are first class citizens in Objective-C and Ruby.
 Class methods instead of static methods. Easily extendable from outside
 the class. But I still like to have the "class" and "interface" keywords.

Yup. Even Java has Class as object. If D copied most of OOP from Java it 
seems strange that it was replaced by halfhearted factory method in 
object.di
kind of saying "you have RTTI but it's crippled just in case if it is of 
no use".

Still I'd think it's reasonable course to enact slow but steady 
evolution of existing OOP design, changing synchronized/monitors seems 
as good start as any.

 Would this solve the problem with const:

 https://github.com/D-Programming-Language/dmd/pull/3

I know, and hoped it would happen sooner. But at least it's possible.

-- 
Dmitry Olshansky

Jacob Carlborg <doob me.com> writes:

On 2012-05-30 12:21, Dmitry Olshansky wrote:

 Yup. Even Java has Class as object. If D copied most of OOP from Java it
 seems strange that it was replaced by halfhearted factory method in
 object.di
 kind of saying "you have RTTI but it's crippled just in case if it is of
 no use".

I would not compare Java's Class with classes in Objective-C and Ruby. 
In Objective-C and Ruby classes are objects with their own hierarchy of 
inheritance. Example in Ruby:

class Foo
   def bar
      instance_variable = 1
       class_variable = 2
   end


   def self.bar
      instance_variable_on_the_class = 1
       class_variable = 3
   end
end

a = Foo
instance_of_foo = a.new

 Still I'd think it's reasonable course to enact slow but steady
 evolution of existing OOP design, changing synchronized/monitors seems
 as good start as any.

I agree.

-- 
/Jacob Carlborg

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 17:58, Jacob Carlborg wrote:
 On 2012-05-30 12:21, Dmitry Olshansky wrote:

 Yup. Even Java has Class as object. If D copied most of OOP from Java it
 seems strange that it was replaced by halfhearted factory method in
 object.di
 kind of saying "you have RTTI but it's crippled just in case if it is of
 no use".

 I would not compare Java's Class with classes in Objective-C and Ruby.
 In Objective-C and Ruby classes are objects with their own hierarchy of
 inheritance. Example in Ruby:

 class Foo
 def bar
  instance_variable = 1
   class_variable = 2
 end


 def self.bar
  instance_variable_on_the_class = 1
   class_variable = 3
 end
 end

 a = Foo
 instance_of_foo = a.new

Yes, it's OOP classics I think. Dynamic languages are easier in this 
regard. But let's not detract from original topic - locking.


-- 
Dmitry Olshansky

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 14:26, Alex Rønne Petersen a écrit :
 On 29-05-2012 14:19, Dmitry Olshansky wrote:
 On 29.05.2012 16:07, Regan Heath wrote:
 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly... In the following code the "lock" statement and "synchronized
 void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.


 I'd be darned but every Object in D has monitor fields. If I'm not
 mistaken it's the mutex you are looking for ;)

 Indeed they do, and therefore each object must eat an entire word of
 memory for questionable gain.

 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

Every fucking things MUST BE AN OBJECT :D So let's lock on ANY OBJECT !

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 5:26 AM, Alex Rønne Petersen wrote:
 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

I think that's an exaggeration. Care to elaborate a bit?

Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 29-05-2012 23:38, Andrei Alexandrescu wrote:
 On 5/29/12 5:26 AM, Alex Rønne Petersen wrote:
 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

 I think that's an exaggeration. Care to elaborate a bit?

 Andrei

1) You waste an entire word of memory in *every single object you ever 
create*. And for the majority of objects, this word will always be 
zero/null. Not to mention that when you do allocate a monitor, you also 
get the actual memory of that monitor in addition.
2) Anyone can lock on any object meaning it's near impossible to see 
where a deadlock might come from.
3) Encapsulation is completely broken as a result of (2).
4) There's a whole bunch of runtime and GC plumbing (which is even 
broken in druntime right now) to support object monitors. To fix this 
entire mess, we'd need weak references.
5) This whole object monitor model goes against our "thread-local by 
default" model. Synchronization is evil and should be explicit with 
core.sync.mutex.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 2:56 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:38, Andrei Alexandrescu wrote:
 On 5/29/12 5:26 AM, Alex Rønne Petersen wrote:
 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

 I think that's an exaggeration. Care to elaborate a bit?

 Andrei

 1) You waste an entire word of memory in *every single object you ever
 create*. And for the majority of objects, this word will always be
 zero/null. Not to mention that when you do allocate a monitor, you also
 get the actual memory of that monitor in addition.

I agree. It would be better if synchronizable objects would be part of a 
sub-hierarchy such that not everyone pays for the word.

 2) Anyone can lock on any object meaning it's near impossible to see
 where a deadlock might come from.

What would be the alternative? Deadlocks are a natural consequence of 
the fact that indeed any thread can wait on a resource.

 3) Encapsulation is completely broken as a result of (2).

Again, what would be the alternative?

 4) There's a whole bunch of runtime and GC plumbing (which is even
 broken in druntime right now) to support object monitors. To fix this
 entire mess, we'd need weak references.

That's more like an argument the implementation is imperfect, not that 
the idea is the worst ever.

 5) This whole object monitor model goes against our "thread-local by
 default" model. Synchronization is evil and should be explicit with
 core.sync.mutex.

But doing things with core.sync.mutex is a definite step backwards as it 
is prey to all of the issues above. For example, "anyone can lock any 
mutex". How is that progress?


Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:56 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:38, Andrei Alexandrescu wrote:
 On 5/29/12 5:26 AM, Alex Rønne Petersen wrote:
 Generalized object monitors is the worst idea in programming language
 and virtual machine design, ever.

 I think that's an exaggeration. Care to elaborate a bit?

 Andrei

 1) You waste an entire word of memory in *every single object you ever
 create*. And for the majority of objects, this word will always be
 zero/null. Not to mention that when you do allocate a monitor, you also
 get the actual memory of that monitor in addition.

 I agree. It would be better if synchronizable objects would be part of a
 sub-hierarchy such that not everyone pays for the word.

 2) Anyone can lock on any object meaning it's near impossible to see
 where a deadlock might come from.

 What would be the alternative? Deadlocks are a natural consequence of
 the fact that indeed any thread can wait on a resource.

But mutexes allow proper encapsulation by hiding the mutex resource. As 
I've proven in the very OP of this thread, both druntime and phobos 
suffer from the anti-pattern that is locking on a public resource when 
you shouldn't. The language encourages doing it with this synchronized 
business.

 3) Encapsulation is completely broken as a result of (2).

 Again, what would be the alternative?

 4) There's a whole bunch of runtime and GC plumbing (which is even
 broken in druntime right now) to support object monitors. To fix this
 entire mess, we'd need weak references.

 That's more like an argument the implementation is imperfect, not that
 the idea is the worst ever.

Of course. But it's enough reason for me to have banned synchronized 
from my source base entirely. This point is a practical matter more than 
it's a design matter.

 5) This whole object monitor model goes against our "thread-local by
 default" model. Synchronization is evil and should be explicit with
 core.sync.mutex.

 But doing things with core.sync.mutex is a definite step backwards as it
 is prey to all of the issues above. For example, "anyone can lock any
 mutex". How is that progress?

You encapsulate the mutex and thereby avoid the anti-pattern I pointed 
out in the OP, which has evidently found its way into both druntime and 
phobos.

 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

"Jonathan M Davis" <jmdavisProg gmx.com> writes:

On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

From your comments, it sounds to me like the problem is entirely with 

synchronized blocks, not sychronized classes. Synchronized classes don't have 
the issue of having externals lock on them without the explicit use of 
synchronized blocks. They're the equivalent of locking a mutex in every public 
function and releasing it afterwards. The problem is that that mutex is 
effectively public such that when a synchronized block is used on it, it locks 
on the same mutex. If no synchronized blocks are used, as far as I can tell, 
it's a non-issue.

As such, wouldn't simply making the use of a sychronized block on a 
synchronized object illegal fix the problem?

- Jonathan M Davis

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 01:10, Jonathan M Davis wrote:
 On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

 From your comments, it sounds to me like the problem is entirely with

 synchronized blocks, not sychronized classes. Synchronized classes don't have
 the issue of having externals lock on them without the explicit use of
 synchronized blocks. They're the equivalent of locking a mutex in every public
 function and releasing it afterwards. The problem is that that mutex is
 effectively public such that when a synchronized block is used on it, it locks
 on the same mutex. If no synchronized blocks are used, as far as I can tell,
 it's a non-issue.

 As such, wouldn't simply making the use of a sychronized block on a
 synchronized object illegal fix the problem?

 - Jonathan M Davis

Possibly. I haven't thought that through, but I think that that could 
work. In fact, it would probably be a good way to ensure safety 
statically, since a synchronized class tells its user "I take care of 
synchronization".

We still have the issue of every object having a monitor, but that is 
probably a separate issue...

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Michel Fortin <michel.fortin michelf.com> writes:

On 2012-05-29 23:22:38 +0000, Alex Rønne Petersen <alex lycus.org> said:

 Possibly. I haven't thought that through, but I think that that could 
 work. In fact, it would probably be a good way to ensure safety 
 statically, since a synchronized class tells its user "I take care of 
 synchronization".

More or less.

My biggest gripe about synchronized functions is that calling other 
functions within them is prone to create deadlocks. Can you see the 
deadlock in this example?

	synchronized class Node
	{
		private Node[] peers;

		void addPeer(Node peer)    // implicitly synchronized
		{
			peers ~= peer;
		}

		void poke()    // implicitly synchronized
		{
			writeln("blip!");
		}

		void pokeNext()    // implicitly synchronized
		{
			if (!peers.empty)
				peers.front.poke();
		}
	}

	shared Node node1;
	shared Node node2;

	void pokeThread(Node node)
	{
		node.pokeNext();
	}

	void main()
	{
		// setup
		node1 = new Node;
		node2 = new Node;
		node1.addPeer(node2);
		node2.addPeer(node1);

		// start two threads
		spawn(&pokeThread, node1);
		spawn(&pokeThread, node2);
	}

The correct way to write the pokeNext function would be this, assuming 
you could remove the implicit synchronization:

		void pokeNext()     // not implicitly synchronized
		{
			Node next;
			synchronized (this)
			{
				if (!peers.empty)
					next = peers.front;
			}
			if (next)
				next.poke();
		}

Here you only synchronize access to the variable, which cannot deadlock 
in any way. Keeping the lock while calling other functions is dangerous 
(other functions can take their own lock and thus deadlock) and should 
be avoided as long as you hold a lock. The easiest way to avoid 
deadlocks is to never take two locks at the same time. The problem is 
that implicit synchronized blocks makes it too easy to take many locks 
at the same time without even noticing, which is deadlock prone.

I think that is a reason enough not to use synchronized classes in the 
general case. They're fine as long as your member functions (which are 
implicitly synchronized) only access what is contained in the class, 
but you must be sure not to call another function that takes another 
lock, or then you need to be very careful about what those functions do.

(Also, being forced to take locks longer than necessary because the 
whole function is always synchronized can be a performance problem, but 
that's a relatively minor issue compared to deadlocks.)

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis <jmdavisProg gmx.co=
m>  =

wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex R=F8nne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. =


As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource whe=


n
 you shouldn't. The language encourages doing it with this synchronize=


d
 business.

 From your comments, it sounds to me like the problem is entirely with=


 synchronized blocks, not sychronized classes. Synchronized classes don=

't  =

 have
 the issue of having externals lock on them without the explicit use of=

 synchronized blocks. They're the equivalent of locking a mutex in ever=

y  =

 public
 function and releasing it afterwards. The problem is that that mutex i=

s
 effectively public such that when a synchronized block is used on it, =

it  =

 locks
 on the same mutex. If no synchronized blocks are used, as far as I can=

  =

 tell,
 it's a non-issue.

 As such, wouldn't simply making the use of a sychronized block on a
 synchronized object illegal fix the problem?

Not in all cases.  If you have a synchronized class, say a thread-safe  =

container, which has a synchronized lookup and synchronized insert  =

function, code like..

if (!o.lookup(...))
   o.insert(...)

obtains and releases the mutex twice each, and in between another thread=
  =

could call o.insert() and insert the missing object - resulting in 2  =

copies being inserted (or a duplicate insert exception being throw, or..=
.).

Now, the above is bad design, but the only other choice is to expose  =

lock() and unlock() methods (or the mutex - back to square 1one) to allo=
w  =

locking around multiple class, but now the caller has to be careful to  =

call unlock in all code paths.  scope helps us in D, but this is actuall=
y  =

the whole point of synchronized blocks - 1. ensuring the unlock always  =

happens and 2. making the scope of the locking visible in the code.

R

-- =

Using Opera's revolutionary email client: http://www.opera.com/mail/

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 16:43, Regan Heath wrote:
 On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

 From your comments, it sounds to me like the problem is entirely with

 synchronized blocks, not sychronized classes. Synchronized classes
 don't have
 the issue of having externals lock on them without the explicit use of
 synchronized blocks. They're the equivalent of locking a mutex in
 every public
 function and releasing it afterwards. The problem is that that mutex is
 effectively public such that when a synchronized block is used on it,
 it locks
 on the same mutex. If no synchronized blocks are used, as far as I can
 tell,
 it's a non-issue.

 As such, wouldn't simply making the use of a sychronized block on a
 synchronized object illegal fix the problem?

 Not in all cases. If you have a synchronized class, say a thread-safe
 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

 obtains and releases the mutex twice each, and in between another thread
 could call o.insert() and insert the missing object - resulting in 2
 copies being inserted (or a duplicate insert exception being throw, or...).

 Now, the above is bad design, but the only other choice is to expose
 lock() and unlock() methods (or the mutex - back to square 1one) to
 allow locking around multiple class, but now the caller has to be
 careful to call unlock in all code paths. scope helps us in D, but this
 is actually the whole point of synchronized blocks - 1. ensuring the
 unlock always happens and 2. making the scope of the locking visible in
 the code.

synchronized class Container
{

void applyLocked(scope delegate void(Container _this));
...
}


-- 
Dmitry Olshansky

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 13:55:10 +0100, Dmitry Olshansky  =

<dmitry.olsh gmail.com> wrote:

 On 30.05.2012 16:43, Regan Heath wrote:
 On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex R=F8nne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource=




.  =

 As
 I've proven in the very OP of this thread, both druntime and phobos=




 suffer from the anti-pattern that is locking on a public resource w=




hen
 you shouldn't. The language encourages doing it with this synchroni=




zed
 business.

 From your comments, it sounds to me like the problem is entirely wi=




th
 synchronized blocks, not sychronized classes. Synchronized classes
 don't have
 the issue of having externals lock on them without the explicit use =



of
 synchronized blocks. They're the equivalent of locking a mutex in
 every public
 function and releasing it afterwards. The problem is that that mutex=



 is
 effectively public such that when a synchronized block is used on it=



,
 it locks
 on the same mutex. If no synchronized blocks are used, as far as I c=



an
 tell,
 it's a non-issue.

 As such, wouldn't simply making the use of a sychronized block on a
 synchronized object illegal fix the problem?

 Not in all cases. If you have a synchronized class, say a thread-safe=


 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

 obtains and releases the mutex twice each, and in between another thr=


ead
 could call o.insert() and insert the missing object - resulting in 2
 copies being inserted (or a duplicate insert exception being throw,  =


 or...).

 Now, the above is bad design, but the only other choice is to expose
 lock() and unlock() methods (or the mutex - back to square 1one) to
 allow locking around multiple class, but now the caller has to be
 careful to call unlock in all code paths. scope helps us in D, but th=


is
 is actually the whole point of synchronized blocks - 1. ensuring the
 unlock always happens and 2. making the scope of the locking visible =


in
 the code.

 synchronized class Container
 {

 void applyLocked(scope delegate void(Container _this));
 ...
 }

Neat :)

R

-- =

Using Opera's revolutionary email client: http://www.opera.com/mail/

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 17:14, Regan Heath wrote:
 synchronized class Container
 {

 void applyLocked(scope delegate void(Container _this));
 ...
 }

 Neat :)

Rewritten and almost complete:

synchronized class Locked!(C)
{
	void applyLocked(scope delegate void(C _this) dg)
	{//should have lock/unlock already, as every sync'ed class
		gd(_this); 	
	}

	final auto opDispatch(string mtd, Args...)(Args args)
	{//can't go alias this, it won't be locked I think
		return mixin("_this."~mtd~"("~args~")");
	}
private:
	C _this;
}


-- 
Dmitry Olshansky

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 14:55, Dmitry Olshansky a écrit :
 On 30.05.2012 16:43, Regan Heath wrote:
 On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

 From your comments, it sounds to me like the problem is entirely with

 synchronized blocks, not sychronized classes. Synchronized classes
 don't have
 the issue of having externals lock on them without the explicit use of
 synchronized blocks. They're the equivalent of locking a mutex in
 every public
 function and releasing it afterwards. The problem is that that mutex is
 effectively public such that when a synchronized block is used on it,
 it locks
 on the same mutex. If no synchronized blocks are used, as far as I can
 tell,
 it's a non-issue.

 As such, wouldn't simply making the use of a sychronized block on a
 synchronized object illegal fix the problem?

 Not in all cases. If you have a synchronized class, say a thread-safe
 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

 obtains and releases the mutex twice each, and in between another thread
 could call o.insert() and insert the missing object - resulting in 2
 copies being inserted (or a duplicate insert exception being throw,
 or...).

 Now, the above is bad design, but the only other choice is to expose
 lock() and unlock() methods (or the mutex - back to square 1one) to
 allow locking around multiple class, but now the caller has to be
 careful to call unlock in all code paths. scope helps us in D, but this
 is actually the whole point of synchronized blocks - 1. ensuring the
 unlock always happens and 2. making the scope of the locking visible in
 the code.

 synchronized class Container
 {

 void applyLocked(scope delegate void(Container _this));
 ...
 }

I often ends up doing similar stuff when manipulating pair of actions. 
lock/unlock, connect/disconect, begin/commit/rollback, etc . . .

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 14:43, Regan Heath a écrit :
 On Wed, 30 May 2012 00:10:19 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Wednesday, May 30, 2012 01:02:53 Alex Rønne Petersen wrote:
 But mutexes allow proper encapsulation by hiding the mutex resource. As
 I've proven in the very OP of this thread, both druntime and phobos
 suffer from the anti-pattern that is locking on a public resource when
 you shouldn't. The language encourages doing it with this synchronized
 business.

 From your comments, it sounds to me like the problem is entirely with

 synchronized blocks, not sychronized classes. Synchronized classes
 don't have
 the issue of having externals lock on them without the explicit use of
 synchronized blocks. They're the equivalent of locking a mutex in
 every public
 function and releasing it afterwards. The problem is that that mutex is
 effectively public such that when a synchronized block is used on it,
 it locks
 on the same mutex. If no synchronized blocks are used, as far as I can
 tell,
 it's a non-issue.

 As such, wouldn't simply making the use of a sychronized block on a
 synchronized object illegal fix the problem?

 Not in all cases. If you have a synchronized class, say a thread-safe
 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

 obtains and releases the mutex twice each, and in between another thread
 could call o.insert() and insert the missing object - resulting in 2
 copies being inserted (or a duplicate insert exception being throw, or...).

 Now, the above is bad design, but the only other choice is to expose
 lock() and unlock() methods (or the mutex - back to square 1one) to
 allow locking around multiple class, but now the caller has to be
 careful to call unlock in all code paths. scope helps us in D, but this
 is actually the whole point of synchronized blocks - 1. ensuring the
 unlock always happens and 2. making the scope of the locking visible in
 the code.

 R

The correct solution is an insertIfNotPresent function, or to 
throw/return an error when inserting an item.

Exposing the lock is just spreading the mess.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 6:39 AM, deadalnix wrote:
 Le 30/05/2012 14:43, Regan Heath a écrit :
 Not in all cases. If you have a synchronized class, say a thread-safe
 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

 The correct solution is an insertIfNotPresent function, or to
 throw/return an error when inserting an item.

 Exposing the lock is just spreading the mess.

Agreed on both counts.


Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 5:43 AM, Regan Heath wrote:
 Not in all cases. If you have a synchronized class, say a thread-safe
 container, which has a synchronized lookup and synchronized insert
 function, code like..

 if (!o.lookup(...))
 o.insert(...)

 obtains and releases the mutex twice each, and in between another thread
 could call o.insert() and insert the missing object - resulting in 2
 copies being inserted (or a duplicate insert exception being throw,
 or....).

 Now, the above is bad design, but the only other choice is to expose
 lock() and unlock() methods (or the mutex - back to square 1one) to
 allow locking around multiple class, but now the caller has to be
 careful to call unlock in all code paths. scope helps us in D, but this
 is actually the whole point of synchronized blocks - 1. ensuring the
 unlock always happens and 2. making the scope of the locking visible in
 the code.

There's a hybrid design available - a lock() operation may return a 
Locked!Type object which has a richer set of primitives. The implied 
understanding is that for the lifetime of that object, the underlying 
mutex is locked.

Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 29-05-2012 14:07, Regan Heath wrote:
 On Tue, 29 May 2012 01:08:59 +0100, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:

 On Tuesday, May 29, 2012 01:54:59 Alex Rønne Petersen wrote:
 On 29-05-2012 01:41, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:35:23 Alex Rønne Petersen wrote:
 I don't think arguing about them makes sense at this point. Way


 too much
 code would break if we changed the semantics. I'd consider it a


 mistake
 and a lesson learned, rather.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

 I'm not an expert on threading stuf, but it would be my opinion

 that if
 you're not intending to protect the entire class with locks that it

 makes
 no sense to lock on the class itself. You're locking for something
 specific, in which case, your sychronized block should be locking on
 something else specific to what you're trying to protect. Certainly,
 that's how I'd approach it with mutexes. You don't have a mutex for an
 entire class unless it's actually used for all of the class'

 functions.
 Rather, you use mutexes specific to what you're trying to protect.

 - Jonathan M Davis

 Right, but even if you really *are* protecting the entire class, you can
 still create mysterious deadlocks if users of your code lock on your
 class. So I'm arguing that no matter the use case, never lock on a
 'this' reference exposed outside of some API layer.

 Well, it seems pretty abysmal to me to be locking on something that
 you don't control. Making a mutex that your class used public would
 just be stupid.

 Interestingly this is what C#s Array type does with SyncRoot
 (intentionally).

Microsoft has officially dismissed the SyncRoot approach to 
synchronization. I don't have a link handy, but there's an article 
*somewhere* on MSDN about it.

 With synchronized classes/functions, you're basically creating and
 using an
 implicit mutex for the whole class, which would then be the same as if
 you locked it at the beginning of every member function call and
 unlocked it at its end, which doesn't expose the mutex at all. So, I
 don't really see te
 problem there would have to study the matter more to see what the
 issue there is.

 According to the docs here:
 http://dlang.org/class.html#synchronized-functions

 A synchronized function locks "this" and as "this" is exposed
 publicly.... In the following code the "lock" statement and
 "synchronized void bar" lock the same mutex.

 class Foo {
 synchronized void bar() { ...statements... }
 }

 void main()
 {
 Foo foo = new Foo();
 lock(foo)
 {
 ...statements...
 }
 }

 But locking on another class rather than something specifically
 intended as a mutex does seem to me like it's asking for trouble.

 Yep. It commonly arises where you have a class/object which is either
 not synchronized itself (because it might be used in single threaded
 situations and you want performance) like a collection class for
 example, or is synchronized but you need to lock a sequence of member
 function calls i.e. a lookup and insert on the collection. What happens
 then, is people just call lock(<object>) and end up locking the same
 mutex as the class does internally. What happens next to cause a
 deadlock is that inside that lock they call one or more functions or
 methods which internally call methods on another synchronized object.
 This results in a locking pattern of object1, object2. In another piece
 of code, running in another thread, they do something similar which
 results in a locking pattern of object2, object1 and eventually these
 threads will deadlock each other.

 R

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 01:35, Alex Rønne Petersen a écrit :
 On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

 Per TDPL, having individually synchronized functions is illegal.
 Either all
 of the functions in a class are synchronized or none of them are. Putting
 synchronized on a function should actually be illegal. It belongs only on
 classes or in synchronized blocks, never on functions.

 Now, unfortuntately, I don't believe that the compiler enforces any of
 that
 right now, so you end up synchronizing indivdual functions rather than
 whole
 classes, but the synchronized functions themselves should function the
 same.

 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

 Well, then you should probably be arguing about the design of
 synchronized
 classes/functions rather than synchronized(this). However, given the
 design of
 synchronized classes, synchronized(this) would probably never be
 appropriate.
 If you're using a synchronized class, then you don't need it. And if
 you're
 not using a synchronized class, then you shouldn't be synchronizing your
 class. I would definitely think that synchronized blocks should
 synchronize on
 something else, since they're trying to lock a much smaller area than an
 entire class.

 - Jonathan M Davis

 I don't think arguing about them makes sense at this point. Way too much
 code would break if we changed the semantics. I'd consider it a mistake
 and a lesson learned, rather.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

I would say that breaking things here, with the right deprecation 
process, is the way to go.

shared isn't working properly ATM, so anyway, things will have to change 
in regard of shared memory support in the language.

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 10:37, deadalnix wrote:
 Le 29/05/2012 01:35, Alex Rønne Petersen a écrit :
 On 29-05-2012 01:24, Jonathan M Davis wrote:
 On Tuesday, May 29, 2012 01:11:49 Alex Rønne Petersen wrote:
 I have no idea how synchronized classes work; they are not a documented
 feature of the language. We have synchronized functions which
 synchronize on the this reference. Perhaps synchronized classes just
 make all functions in a class do this.

 Per TDPL, having individually synchronized functions is illegal.
 Either all
 of the functions in a class are synchronized or none of them are.
 Putting
 synchronized on a function should actually be illegal. It belongs
 only on
 classes or in synchronized blocks, never on functions.

 Now, unfortuntately, I don't believe that the compiler enforces any of
 that
 right now, so you end up synchronizing indivdual functions rather than
 whole
 classes, but the synchronized functions themselves should function the
 same.

 Either way, this is a fundamental language design fallacy. This is
 anti-pattern number one when it comes to locking:

 * http://stackoverflow.com/a/251668/438034
 * http://msdn.microsoft.com/en-us/library/ms173179.aspx (The lock and
 SyncLock Keywords section)

 Well, then you should probably be arguing about the design of
 synchronized
 classes/functions rather than synchronized(this). However, given the
 design of
 synchronized classes, synchronized(this) would probably never be
 appropriate.
 If you're using a synchronized class, then you don't need it. And if
 you're
 not using a synchronized class, then you shouldn't be synchronizing your
 class. I would definitely think that synchronized blocks should
 synchronize on
 something else, since they're trying to lock a much smaller area than an
 entire class.

 - Jonathan M Davis

 I don't think arguing about them makes sense at this point. Way too much
 code would break if we changed the semantics. I'd consider it a mistake
 and a lesson learned, rather.

 But I take it you agree that synchronized (this) and similar
 "uncontrollable" synchronization blocks should be avoided?

 I would say that breaking things here, with the right deprecation
 process, is the way to go.

Well, we could deprecate it over time, but it certainly can't be a 
day-to-day thing.

 shared isn't working properly ATM, so anyway, things will have to change
 in regard of shared memory support in the language.

And it never will. ;)

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 14:26, Alex Rønne Petersen a écrit :
 shared isn't working properly ATM, so anyway, things will have to change
 in regard of shared memory support in the language.

 And it never will. ;)

It has been successful done in other languages and has proven itself 
useful. This is a difficult backend problem, but it can be solved.

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 17:52, deadalnix wrote:
 Le 29/05/2012 14:26, Alex Rønne Petersen a écrit :
 shared isn't working properly ATM, so anyway, things will have to change
 in regard of shared memory support in the language.

 And it never will. ;)

 It has been successful done in other languages and has proven itself
 useful. This is a difficult backend problem, but it can be solved.

The problem is not in the compiler back end, the problem is that shared 
is completely useless as it is. There's no bridge from shared to 
unshared, making it impossible to work with any real world code base.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 18:56, Alex Rønne Petersen a écrit :
 On 29-05-2012 17:52, deadalnix wrote:
 Le 29/05/2012 14:26, Alex Rønne Petersen a écrit :
 shared isn't working properly ATM, so anyway, things will have to
 change
 in regard of shared memory support in the language.

 And it never will. ;)

 It has been successful done in other languages and has proven itself
 useful. This is a difficult backend problem, but it can be solved.

 The problem is not in the compiler back end, the problem is that shared
 is completely useless as it is. There's no bridge from shared to
 unshared, making it impossible to work with any real world code base.

I'd argue that shared shouldn't contains big stuff. Integers, pointer to 
immutable data, flags, stuff like that.

For such usage, this would work. Still, you comment show that the 
current state of things needs some work, and it is likely to impact both 
shared and synchronized.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

So what should we use for mutex-based synchronization if we deprecate 
synchronized classes?

Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

core.sync.mutex

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 core.sync.mutex

That's worse.

Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 core.sync.mutex

 That's worse.

 Andrei

I don't agree. Also, it is faster than object monitors. This was proven 
by David Simcha recently where he sped up GC allocations by some 40%-ish 
factor just by changing to a final class derived from core.sync.mutex. 
And no, that's not a bug in the monitor implementation. It's a 
limitation of the design.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 3:58 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 core.sync.mutex

 That's worse.

 Andrei

 I don't agree.

One simple thing to understand is that core.sync.mutex does everything 
synchronized objects do, in a much less structured way. So it's tenuous 
to build an argument that synchronized classes do something wrong but 
bare, unstructured mutexes do something good.

 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

We'd need to take a look at that. I recall at a point Bartosz was 
working on cheap locks using the mutex word as a spin lock in certain 
circumstances. Anyhow, it's something that would be interesting to look at.


Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 01:07, Andrei Alexandrescu wrote:
 On 5/29/12 3:58 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 core.sync.mutex

 That's worse.

 Andrei

 I don't agree.

 One simple thing to understand is that core.sync.mutex does everything
 synchronized objects do, in a much less structured way. So it's tenuous
 to build an argument that synchronized classes do something wrong but
 bare, unstructured mutexes do something good.

Okay, let's get something straight here: The Mutex class in 
core.sync.mutex is *not* final. It does *not* force you to use it in a 
compositional fashion. You can inherit it just as any other non-final 
class. And when you use synchronized (mtx), where mtx is an instance of 
Mutex or a derived class, it will even do the Right Thing (TM) and lock 
on the mutex.

Mutex is effectively the synchronized object type that you want. Why not 
just use it?

 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

 We'd need to take a look at that. I recall at a point Bartosz was
 working on cheap locks using the mutex word as a spin lock in certain
 circumstances. Anyhow, it's something that would be interesting to look at.


 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Sean Kelly <sean invisibleduck.org> writes:

On May 29, 2012, at 4:07 PM, Andrei Alexandrescu <SeeWebsiteForEmail erdani.=
org> wrote:

 On 5/29/12 3:58 PM, Alex R=C3=B8nne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex R=C3=B8nne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

=20
 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?
=20
 Andrei

=20
 core.sync.mutex

=20
 That's worse.
=20
 Andrei
=20

=20
 I don't agree.

=20
 One simple thing to understand is that core.sync.mutex does everything syn=

chronized objects do, in a much less structured way. So it's tenuous to buil=
d an argument that synchronized classes do something wrong but bare, unstruc=
tured mutexes do something good.
=20
 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

=20
 We'd need to take a look at that. I recall at a point Bartosz was working o=

n cheap locks using the mutex word as a spin lock in certain circumstances. A=
nyhow, it's something that would be interesting to look at.

I bet this is because monitors are lazily initialized, so the cost of acquir=
ing a lock is more than just locking the underlying mutex. The implementatio=
n for built-in monitors really isnt great. I've been meaning to do something=
 about that.=20=

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 08:23, Sean Kelly a écrit :
 On May 29, 2012, at 4:07 PM, Andrei
Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:

 On 5/29/12 3:58 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 core.sync.mutex

 That's worse.

 Andrei

 I don't agree.

 One simple thing to understand is that core.sync.mutex does everything
synchronized objects do, in a much less structured way. So it's tenuous to
build an argument that synchronized classes do something wrong but bare,
unstructured mutexes do something good.

 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

 We'd need to take a look at that. I recall at a point Bartosz was working on
cheap locks using the mutex word as a spin lock in certain circumstances.
Anyhow, it's something that would be interesting to look at.

 I bet this is because monitors are lazily initialized, so the cost of
acquiring a lock is more than just locking the underlying mutex. The
implementation for built-in monitors really isnt great. I've been meaning to do
something about that.

And it use double check locking the wrong way, so can eventually explode 
sometime.

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 08:23, Sean Kelly wrote:
 On May 29, 2012, at 4:07 PM, Andrei
Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:

 On 5/29/12 3:58 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 00:45, Andrei Alexandrescu wrote:
 On 5/29/12 2:57 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:33, Andrei Alexandrescu wrote:
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 core.sync.mutex

 That's worse.

 Andrei

 I don't agree.

 One simple thing to understand is that core.sync.mutex does everything
synchronized objects do, in a much less structured way. So it's tenuous to
build an argument that synchronized classes do something wrong but bare,
unstructured mutexes do something good.

 Also, it is faster than object monitors. This was proven
 by David Simcha recently where he sped up GC allocations by some 40%-ish
 factor just by changing to a final class derived from core.sync.mutex.
 And no, that's not a bug in the monitor implementation. It's a
 limitation of the design.

 We'd need to take a look at that. I recall at a point Bartosz was working on
cheap locks using the mutex word as a spin lock in certain circumstances.
Anyhow, it's something that would be interesting to look at.

 I bet this is because monitors are lazily initialized, so the cost of
acquiring a lock is more than just locking the underlying mutex. The
implementation for built-in monitors really isnt great. I've been meaning to do
something about that.

We also need to fix the monitor memory leak that sometimes manifests 
itself...

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

I think something similar to range design here is the way to go.

It is easy to define something like

template isLockable(T) {
	enum isLockable = isShared!T && is(typeof(T.init.lock())) && 
is(typeof(T.init.release()));
}

And allow locking only if(isLockable!Type) .

Now we can create SyncObject or any structure we want. The point is that 
we lock explicit stuff.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable = isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is that
 we lock explicit stuff.

But in this design anyone can lock such an object, which was something 
you advocated against.

Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a =E9crit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecat=



e
 synchronized classes?

 Andrei

 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable =3D isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is t=


hat
 we lock explicit stuff.

 But in this design anyone can lock such an object, which was something=

  =

 you advocated against.

I think there is some confusion here as to what the "problem" is and is =
 =

not.

The problem is /not/ that you can lock any object.
The problem is /not/ that we have synchronized(object) {}
The problem is /not/ that we have synchronized classes/methods.

The problem /is/ that synchronized classes/methods use a mutex which is =
 =

exposed publicly, and it the same mutex as used by synchronized(object) =
 =

{}.  This exposure/re-use makes deadlocks more likely to happen, and  =

harder to spot.

Removal of this "problem" will not stop deadlocks from happening as  =

they're a problem multi-threaded/lock based code will always have (*).  =
It  =

will however make them far less likely to happen accidentally.  It will =
 =

make programmers think about what/where/when and how to lock things rath=
er  =

than blithely using synchronized everywhere.  It will mean using locks i=
s  =

not as easy as currently, though I think we can make it fairly nice with=
  =

good library code and/or some co-operation between the runtime and  =

synchronized() statements i.e. requiring the class implement a common  =

Lockable interface for example.

The fact that every object can be locked, and this means they all use mo=
re  =

memory is a side issue - arguably as important for some.

R

(*) the best you can do to "solve" the deadlock problem is to impose loc=
k  =

acquisition ordering on your code, as in all locks must be acquired in a=
  =

defined order, and if not an assertion/error/exception is thrown.  This =
 =

requires some sort of static/singelton/overlord class which is aware of =
 =

all mutexes and is involved in all acquisitions/releases.

-- =

Using Opera's revolutionary email client: http://www.opera.com/mail/

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 9:03 AM, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable = isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is that
 we lock explicit stuff.

 But in this design anyone can lock such an object, which was something
 you advocated against.

 I think there is some confusion here as to what the "problem" is and is
 not.

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

Several posts in this thread assert that such are problems.

 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.

This is news to me. How do you publicly access the mutex of a 
synchronized class object?


Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 On 5/30/12 9:03 AM, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable = isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is
 that
 we lock explicit stuff.

 But in this design anyone can lock such an object, which was something
 you advocated against.

 I think there is some confusion here as to what the "problem" is and is
 not.

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

 Several posts in this thread assert that such are problems.

 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.

 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

Generally in two ways:

1) synchronized (obj)
2) obj.__monitor

(1) accesses it in order to actually do locking, but if obj is an 
instantiation of a class that uses the this reference internally for 
locking, you end up with potential deadlocks. Therefore, you can say 
that obj's mutex is exposed. This isn't necessarily bad, because if obj 
does *not* use its this reference for synchronized statements, nothing 
will blow up. This is why the OP was about banning synchronized (this).

(2) is an undocumented feature of the language that druntime (and maybe 
some phobos code) makes use of to create/alter/delete monitors.

 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Iain Buclaw <ibuclaw ubuntu.com> writes:

On 30 May 2012 17:23, Alex R=F8nne Petersen <alex lycus.org> wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 On 5/30/12 9:03 AM, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a =E9crit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.


 So what should we use for mutex-based synchronization if we deprecat=






e
 synchronized classes?

 Andrei


 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable =3D isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is
 that
 we lock explicit stuff.


 But in this design anyone can lock such an object, which was something
 you advocated against.


 I think there is some confusion here as to what the "problem" is and is
 not.

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.


 Several posts in this thread assert that such are problems.

 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.


 This is news to me. How do you publicly access the mutex of a
 synchronized class object?


 Generally in two ways:

 1) synchronized (obj)
 2) obj.__monitor

 (1) accesses it in order to actually do locking, but if obj is an
 instantiation of a class that uses the this reference internally for
 locking, you end up with potential deadlocks. Therefore, you can say that
 obj's mutex is exposed. This isn't necessarily bad, because if obj does
 *not* use its this reference for synchronized statements, nothing will bl=

ow
 up. This is why the OP was about banning synchronized (this).

 (2) is an undocumented feature of the language that druntime (and maybe s=

ome
 phobos code) makes use of to create/alter/delete monitors.

I'm pretty certain the use of .__monitor is stricted only to:
object_.d - The frontend implementation which contains the underlying
functions that are invoked when you synchronise(obj).
monitor_.d - The backend implementation which contains the platform
separated bits.


Regards
--=20
Iain Buclaw

*(p < e ? p++ : p) =3D (c & 0x0f) + '0';

=?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 30-05-2012 18:42, Iain Buclaw wrote:
 On 30 May 2012 17:23, Alex Rønne Petersen<alex lycus.org>  wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 On 5/30/12 9:03 AM, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org>  wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.


 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei


 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable = isShared!T&&  is(typeof(T.init.lock()))&&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is
 that
 we lock explicit stuff.


 But in this design anyone can lock such an object, which was something
 you advocated against.


 I think there is some confusion here as to what the "problem" is and is
 not.

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.


 Several posts in this thread assert that such are problems.

 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.


 This is news to me. How do you publicly access the mutex of a
 synchronized class object?


 Generally in two ways:

 1) synchronized (obj)
 2) obj.__monitor

 (1) accesses it in order to actually do locking, but if obj is an
 instantiation of a class that uses the this reference internally for
 locking, you end up with potential deadlocks. Therefore, you can say that
 obj's mutex is exposed. This isn't necessarily bad, because if obj does
 *not* use its this reference for synchronized statements, nothing will blow
 up. This is why the OP was about banning synchronized (this).

 (2) is an undocumented feature of the language that druntime (and maybe some
 phobos code) makes use of to create/alter/delete monitors.

 I'm pretty certain the use of .__monitor is stricted only to:
 object_.d - The frontend implementation which contains the underlying
 functions that are invoked when you synchronise(obj).
 monitor_.d - The backend implementation which contains the platform
 separated bits.


 Regards

I use it in my source base currently to implement weak references.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 9:23 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

 Generally in two ways:

 1) synchronized (obj)

This is not accessing the mutex for arbitrary operations.

 2) obj.__monitor

All symbols starting with two underscores are reserved by the 
implementation.

 (1) accesses it in order to actually do locking, but if obj is an
 instantiation of a class that uses the this reference internally for
 locking, you end up with potential deadlocks.

Deadlocks are endemic to mutex-based programming and are not avoided 
inherently or preferentially by the alternative you discussed (exposing 
unrestricted mutex).

 Therefore, you can say
 that obj's mutex is exposed.

Nothing could stop one, but that claim is incorrect.


Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 30-05-2012 19:12, Andrei Alexandrescu wrote:
 On 5/30/12 9:23 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

 Generally in two ways:

 1) synchronized (obj)

 This is not accessing the mutex for arbitrary operations.

No, it is indeed not. You didn't explicitly say you wanted to do 
"arbitrary operations", so I assumed that by accessing the mutex you 
meant doing the only two things that are sensible for a mutex: Locking 
and unlocking it. What else would you want to do? This discussion has 
always been about locking and unlocking publicly exposed mutexes. I 
don't think arguing about "arbitrary operations" is going to get the 
discussion anywhere exactly because of what you proceed to say ...

 2) obj.__monitor

 All symbols starting with two underscores are reserved by the
 implementation.

... since that effectively means there is no way, from an 
implementation-independent standpoint, to alter the memory of a mutex 
(or whatever) (see my answer below), thus making the only sensible 
operations on a mutex locking and unlocking.

But I may be misunderstanding you. If so, please clarify and/or correct me.

(Just so we're clear, I included (2) only for completeness. I know it's 
a dirty hack and I agree that it should never be in the language 'spec' 
or TDPL.)

 (1) accesses it in order to actually do locking, but if obj is an
 instantiation of a class that uses the this reference internally for
 locking, you end up with potential deadlocks.

 Deadlocks are endemic to mutex-based programming and are not avoided
 inherently or preferentially by the alternative you discussed (exposing
 unrestricted mutex).

 Therefore, you can say
 that obj's mutex is exposed.

 Nothing could stop one, but that claim is incorrect.

It is exposed as far as using it for anything sensible goes. No, 
synchronized (obj) does not let you delete the monitor of obj or alter 
it or something along these lines, but I don't think that's worth 
bringing into this argument. This has always been about locking and 
unlocking a publicly exposed mutex, and about preventing common deadlock 
sources.

 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 10:28 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 19:12, Andrei Alexandrescu wrote:
 On 5/30/12 9:23 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

 Generally in two ways:

 1) synchronized (obj)

 This is not accessing the mutex for arbitrary operations.

 No, it is indeed not. You didn't explicitly say you wanted to do
 "arbitrary operations", so I assumed that by accessing the mutex you
 meant doing the only two things that are sensible for a mutex: Locking
 and unlocking it. What else would you want to do?

For example, locking it in one function and unlocking it in another. 
This is impossible with synchronized(obj).

To summarize, the synchronized(obj) operation is not exposing the mutex, 
only manipulate it in a specific way.


Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 30-05-2012 20:25, Andrei Alexandrescu wrote:
 On 5/30/12 10:28 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 19:12, Andrei Alexandrescu wrote:
 On 5/30/12 9:23 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

 Generally in two ways:

 1) synchronized (obj)

 This is not accessing the mutex for arbitrary operations.

 No, it is indeed not. You didn't explicitly say you wanted to do
 "arbitrary operations", so I assumed that by accessing the mutex you
 meant doing the only two things that are sensible for a mutex: Locking
 and unlocking it. What else would you want to do?

 For example, locking it in one function and unlocking it in another.
 This is impossible with synchronized(obj).

 To summarize, the synchronized(obj) operation is not exposing the mutex,
 only manipulate it in a specific way.

It clearly exposes it *enough* to cause potential for deadlocks by 
manipulating it in this specific way, which is what this is all about.

I don't think arguing about what "exposed" means in this specific 
context (or perhaps in general?) or what levels of manipulation justify 
the use of the term "expose" will get the discussion anywhere...

 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 11:38 AM, Alex Rønne Petersen wrote:
 On 30-05-2012 20:25, Andrei Alexandrescu wrote:
 To summarize, the synchronized(obj) operation is not exposing the mutex,
 only manipulate it in a specific way.

 It clearly exposes it *enough* to cause potential for deadlocks by
 manipulating it in this specific way, which is what this is all about.

Nobody claimed synchronized protects against deadlocks. Unfortunately, 
raw mutexes are just as bad in that regard.

 I don't think arguing about what "exposed" means in this specific
 context (or perhaps in general?) or what levels of manipulation justify
 the use of the term "expose" will get the discussion anywhere...

I simply pointed out a factual mistake in your argument.


Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 30 May 2012 13:12:47 -0400, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 9:23 AM, Alex R=C3=B8nne Petersen wrote:
 On 30-05-2012 18:14, Andrei Alexandrescu wrote:
 This is news to me. How do you publicly access the mutex of a
 synchronized class object?

 Generally in two ways:

 1) synchronized (obj)

 This is not accessing the mutex for arbitrary operations.

If I might interject, I think the issue is that as the author of a class=
,  =

you cannot enforce that *only* the data in your class is protected by th=
e  =

mutex.

Yes, you can enforce that the data is protected, but there is nothing  =

stopping a 3rd party caller from hijacking your mutex to protect other  =

things it shouldn't be involved with.

For instance, let's say you have:

synchronized class A
{
   private int _foo;
    property int foo() { return _foo;}
}

synchronized class B
{
    private A _a;
     property A a() { return _a;}
    void fun() {}
}

void thread(B b)
{
    synchronized(b.a)
    {
       b.fun();
    }
}

Run multiple instances of this thread, and it will deadlock.  As the  =

author of A, you have no control for preventing external code from  =

hijacking your mutex for its own purposes in an incorrect way.

But take out this public access to your mutex, and you *can* enforce A's=
  =

locking semantics correctly, and therefore B's semantics.  Absolutely no=
  =

way can A or B's lock participate in a deadlock.

I think it is worth making this enforceable.  In other words, the  =

synchronized(b.a) statement shouldn't compile.

Yes, you can just use a private mutex.  But doesn't that just lead to  =

recommending not using a feature of the language?  Besides, I really lik=
e  =

the synchronized block keyword thing.  It's a great mechanism for lockin=
g.

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

I don't think so. Synchronized classes are the unit of scoped locking in 
D. If you want to do all scoped locking internally, make the class private.

 Besides, I really like
 the synchronized block keyword thing. It's a great mechanism for locking.

Me too!


Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

 I don't think so. Synchronized classes are the unit of scoped locking in  
 D. If you want to do all scoped locking internally, make the class  
 private.

Maybe you didn't read thoroughly the first part of my post (the example  
that shows a deadlock that is easily preventable if the mutex isn't  
exposed).  The issue is that it's possible to hijack the mutex that you  
are using to protect the class data in order to protect *external* data.   
Hijacking is not possible if the mutex is private (i.e. a private member).

So great!  Just don't use the builtin object mutex, because it exposes  
possible race conditions.  But then why do we have it (the object hidden  
mutex)?  And now I can't use synchronized classes to ensure the whole  
thing is protected, I have to write all my functions like this:

void foo()
{
    synchronized(this.mutex) // lock a private mutex
    {
    }
}

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

 I don't think so. Synchronized classes are the unit of scoped locking
 in D. If you want to do all scoped locking internally, make the class
 private.

 Maybe you didn't read thoroughly the first part of my post (the example
 that shows a deadlock that is easily preventable if the mutex isn't
 exposed).

The mutex is not exposed.

 The issue is that it's possible to hijack the mutex that you
 are using to protect the class data in order to protect *external* data.
 Hijacking is not possible if the mutex is private (i.e. a private member).

 So great! Just don't use the builtin object mutex, because it exposes
 possible race conditions. But then why do we have it (the object hidden
 mutex)? And now I can't use synchronized classes to ensure the whole
 thing is protected, I have to write all my functions like this:

 void foo()
 {
 synchronized(this.mutex) // lock a private mutex
 {
 }
 }

This is a good example. But many synchronized classes are actually 
meaningful (e.g. the classic synchronized queue etc).


Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 21:12, Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

 I don't think so. Synchronized classes are the unit of scoped locking
 in D. If you want to do all scoped locking internally, make the class
 private.

 Maybe you didn't read thoroughly the first part of my post (the example
 that shows a deadlock that is easily preventable if the mutex isn't
 exposed).

 The mutex is not exposed.

I'm trying really hard to not to be rather impolite here, but I don't 
know how else to put it: You seem to be the only one who subscribes to 
this definition of "exposed". I like to think that there are varying 
degrees of exposing resources in programming. The mutex may not be 
directly exposed in the sense that you can obtain a reference (though in 
reality in all compiler implementations, you can), but it is exposed in 
the sense that you can lock and unlock it, which is a mutation of state 
and program flow.

I think we need to set aside this matter about what exposing something 
really means. The mere fact is that you can lock and unlock the mutex of 
any object.

 The issue is that it's possible to hijack the mutex that you
 are using to protect the class data in order to protect *external* data.
 Hijacking is not possible if the mutex is private (i.e. a private
 member).

 So great! Just don't use the builtin object mutex, because it exposes
 possible race conditions. But then why do we have it (the object hidden
 mutex)? And now I can't use synchronized classes to ensure the whole
 thing is protected, I have to write all my functions like this:

 void foo()
 {
 synchronized(this.mutex) // lock a private mutex
 {
 }
 }

 This is a good example. But many synchronized classes are actually
 meaningful (e.g. the classic synchronized queue etc).


 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 21:25, Alex Rønne Petersen a écrit :
 On 30-05-2012 21:12, Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

 I don't think so. Synchronized classes are the unit of scoped locking
 in D. If you want to do all scoped locking internally, make the class
 private.

 Maybe you didn't read thoroughly the first part of my post (the example
 that shows a deadlock that is easily preventable if the mutex isn't
 exposed).

 The mutex is not exposed.

 I'm trying really hard to not to be rather impolite here, but I don't
 know how else to put it: You seem to be the only one who subscribes to
 this definition of "exposed". I like to think that there are varying
 degrees of exposing resources in programming. The mutex may not be
 directly exposed in the sense that you can obtain a reference (though in
 reality in all compiler implementations, you can), but it is exposed in
 the sense that you can lock and unlock it, which is a mutation of state
 and program flow.

 I think we need to set aside this matter about what exposing something
 really means. The mere fact is that you can lock and unlock the mutex of
 any object.

Expliciting terms is always useful, even if it seems sometime irritating.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 12:25 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 21:12, Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

 I don't think so. Synchronized classes are the unit of scoped locking
 in D. If you want to do all scoped locking internally, make the class
 private.

 Maybe you didn't read thoroughly the first part of my post (the example
 that shows a deadlock that is easily preventable if the mutex isn't
 exposed).

 The mutex is not exposed.

 I'm trying really hard to not to be rather impolite here, but I don't
 know how else to put it: You seem to be the only one who subscribes to
 this definition of "exposed".

I don't care as long as it's the correct one. The mutex is not exposed.

Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Wed, 30 May 2012 20:46:17 +0100, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 12:25 PM, Alex R=F8nne Petersen wrote:
 On 30-05-2012 21:12, Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead=






 to
 recommending not using a feature of the language?

 I don't think so. Synchronized classes are the unit of scoped lock=





ing
 in D. If you want to do all scoped locking internally, make the cl=





ass
 private.

 Maybe you didn't read thoroughly the first part of my post (the  =




 example
 that shows a deadlock that is easily preventable if the mutex isn't=




 exposed).

 The mutex is not exposed.

 I'm trying really hard to not to be rather impolite here, but I don't=


 know how else to put it: You seem to be the only one who subscribes t=


o
 this definition of "exposed".

 I don't care as long as it's the correct one. The mutex is not exposed=

.

Ok, how about you tell us what term you want to use for the current stat=
e  =

of affairs then, because arguing about this is pointless and I'm happy t=
o  =

use whatever term you deem appropriate (because I couldn't care less wha=
t  =

the term is - as long as we all know what is meant, which I believe is t=
he  =

case here).

R

-- =

Using Opera's revolutionary email client: http://www.opera.com/mail/

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 2:45 AM, Regan Heath wrote:
 Ok, how about you tell us what term you want to use for the current
 state of affairs then, because arguing about this is pointless and I'm
 happy to use whatever term you deem appropriate (because I couldn't care
 less what the term is - as long as we all know what is meant, which I
 believe is the case here).

The idiom is called "scoped locking".

Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 10:49:27 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:45 AM, Regan Heath wrote:
 Ok, how about you tell us what term you want to use for the current
 state of affairs then, because arguing about this is pointless and I'm
 happy to use whatever term you deem appropriate (because I couldn't care
 less what the term is - as long as we all know what is meant, which I
 believe is the case here).

 The idiom is called "scoped locking".

So.. the mutex is "scoped locking"?  Doesn't sound right to me.  I think  
the mutex is "available for locking and unlocking" <- need a word for  
that, which is not "exposed".  How about accessible, available, usable, or  
just plain lockable .. So, the problem here is that the mutex is lockable  
by external code via synchronized() and this means a certain type of  
deadlock is possible.

Moving on..

I think the change mentioned in TDPL to restrict synchronized to  
synchronized classes is a step in the right direction WRT wasted monitor  
space and people freely locking anything.  But, it is exactly the case  
which results in more possible deadlocks (the cause of this thread) AND I  
think it's actually far more likely people will want to use a synchronized  
statement on a class which is not itself synchronized, like for example an  
existing container class.

Given that, restricting synchronized statements to synchronized classes  
seems entirely wrong to me.  In fact, I would say you almost want to stop  
people using synchronized statements on synchronized classes because:
1. If a synchronized class is written correctly it should not be necessary  
in the general case(*)
2. It raises the chances of deadlocks (the cause of this thread).
3. It means that classes in general will be simpler to write (no need to  
worry about synchronization) and also more lightweight for use in  
non-threaded/non-shared cases.  (because we'd provide a template wrapper  
to make them synchronizable)

So, more and more I'm thinking it would be better to provide a  
library/runtime co-operative solution where we have an interface which is  
required for synchronized statements, and a wrapper template to implement  
that interface for any existing non-synchronized class.  Meaning, the  
choice is either to write a synchronized class (rare) or a  
non-synchronized class - knowing it can easily be synchronized if/when  
needed.

The "liquid lock" problem mentioned earlier is an interesting one that I  
have not personally experienced, perhaps because I don't lock anything but  
mutex primitives and I never to re-assign these.

(*) Locking on a larger scope can be achieved by providing a method taking  
a delegate (see earlier thread/replies) and/or exposing lock/unlock for  
those few situations where the delegate method cannot be used.  These are  
the few cases in which this cannot be avoided as the lock/unlock are  
separated by more than a single scope (so synchronized statements don't  
help in these cases either).

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 3:27 AM, Regan Heath wrote:
 On Thu, 31 May 2012 10:49:27 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:45 AM, Regan Heath wrote:
 Ok, how about you tell us what term you want to use for the current
 state of affairs then, because arguing about this is pointless and I'm
 happy to use whatever term you deem appropriate (because I couldn't care
 less what the term is - as long as we all know what is meant, which I
 believe is the case here).

 The idiom is called "scoped locking".

 So.. the mutex is "scoped locking"?

No, the Java-style approach with the synchronized statement etc.

Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 12:06:45 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 3:27 AM, Regan Heath wrote:
 On Thu, 31 May 2012 10:49:27 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 2:45 AM, Regan Heath wrote:
 Ok, how about you tell us what term you want to use for the current
 state of affairs then, because arguing about this is pointless and I'm
 happy to use whatever term you deem appropriate (because I couldn't  
 care
 less what the term is - as long as we all know what is meant, which I
 believe is the case here).

 The idiom is called "scoped locking".

 So.. the mutex is "scoped locking"?

 No, the Java-style approach with the synchronized statement etc.

*sigh* I give up, this is pointless.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 3:27 AM, Regan Heath wrote:
 I think
 the mutex is "available for locking and unlocking" <- need a word for
 that, which is not "exposed". How about accessible, available, usable,
 or just plain lockable .. So, the problem here is that the mutex is
 lockable by external code via synchronized() and this means a certain
 type of deadlock is possible.

But this is a protection/visibility issue, which is orthogonal on the 
locking capability. It's as if you say "int is not good because anyone 
can overflow it." Okay! Make it private inside a CheckedInt class.

 Moving on..

 I think the change mentioned in TDPL to restrict synchronized to
 synchronized classes is a step in the right direction WRT wasted monitor
 space and people freely locking anything. But, it is exactly the case
 which results in more possible deadlocks (the cause of this thread) AND
 I think it's actually far more likely people will want to use a
 synchronized statement on a class which is not itself synchronized, like
 for example an existing container class.

 Given that, restricting synchronized statements to synchronized classes
 seems entirely wrong to me.

So where's the mutex that would be used to synchronize objects that are 
not synchronizable?

 In fact, I would say you almost want to stop
 people using synchronized statements on synchronized classes because:
 1. If a synchronized class is written correctly it should not be
 necessary in the general case(*)
 2. It raises the chances of deadlocks (the cause of this thread).
 3. It means that classes in general will be simpler to write (no need to
 worry about synchronization) and also more lightweight for use in
 non-threaded/non-shared cases. (because we'd provide a template wrapper
 to make them synchronizable)

There are cases in which you want to do multiple operations under a 
single critical section, even though the API is otherwise well-designed. 
That may be for correctness, efficiency, or both. I don't see why we'd 
want to disallow that, it's a good idiom.

 So, more and more I'm thinking it would be better to provide a
 library/runtime co-operative solution where we have an interface which
 is required for synchronized statements, and a wrapper template to
 implement that interface for any existing non-synchronized class.
 Meaning, the choice is either to write a synchronized class (rare) or a
 non-synchronized class - knowing it can easily be synchronized if/when
 needed.

Looking forward for a fleshed out proposal. Make sure you motivate it 
properly.

 The "liquid lock" problem mentioned earlier is an interesting one that I
 have not personally experienced, perhaps because I don't lock anything
 but mutex primitives and I never to re-assign these.

 (*) Locking on a larger scope can be achieved by providing a method
 taking a delegate (see earlier thread/replies) and/or exposing
 lock/unlock for those few situations where the delegate method cannot be
 used. These are the few cases in which this cannot be avoided as the
 lock/unlock are separated by more than a single scope (so synchronized
 statements don't help in these cases either).

On first look, the inversion of control using delegates delegates has 
similar liabilities as straight scoped locking.


Andrei

deadalnix <deadalnix gmail.com> writes:

Le 31/05/2012 13:13, Andrei Alexandrescu a écrit :
 On 5/31/12 3:27 AM, Regan Heath wrote:
 I think
 the mutex is "available for locking and unlocking" <- need a word for
 that, which is not "exposed". How about accessible, available, usable,
 or just plain lockable .. So, the problem here is that the mutex is
 lockable by external code via synchronized() and this means a certain
 type of deadlock is possible.

 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

 Moving on..

 I think the change mentioned in TDPL to restrict synchronized to
 synchronized classes is a step in the right direction WRT wasted monitor
 space and people freely locking anything. But, it is exactly the case
 which results in more possible deadlocks (the cause of this thread) AND
 I think it's actually far more likely people will want to use a
 synchronized statement on a class which is not itself synchronized, like
 for example an existing container class.

 Given that, restricting synchronized statements to synchronized classes
 seems entirely wrong to me.

 So where's the mutex that would be used to synchronize objects that are
 not synchronizable?

 In fact, I would say you almost want to stop
 people using synchronized statements on synchronized classes because:
 1. If a synchronized class is written correctly it should not be
 necessary in the general case(*)
 2. It raises the chances of deadlocks (the cause of this thread).
 3. It means that classes in general will be simpler to write (no need to
 worry about synchronization) and also more lightweight for use in
 non-threaded/non-shared cases. (because we'd provide a template wrapper
 to make them synchronizable)

 There are cases in which you want to do multiple operations under a
 single critical section, even though the API is otherwise well-designed.
 That may be for correctness, efficiency, or both. I don't see why we'd
 want to disallow that, it's a good idiom.

Because it is now unclear who is controlling the lock.

The solution consisting in passing a delegate as parameter or as 
template is superior, because it is now clear who is in charge of the 
synchronization, reducing greatly chances of deadlock.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

It can also be a lot clunkier for certain abstractions. Say I want a 
ProducerConsumerQueue. It's much more convenient to simply make it a 
synchronized class with the classic primitives, instead of primitives 
that accept delegates etc.

Nevertheless I think there's merit in this idea. One thing to point out 
is that the idiom can easily be done today with a regular class holding 
a synchronized class private member.

So we got everything we need.


Andrei

deadalnix <deadalnix gmail.com> writes:

Le 31/05/2012 20:17, Andrei Alexandrescu a écrit :
 On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

 It can also be a lot clunkier for certain abstractions. Say I want a
 ProducerConsumerQueue. It's much more convenient to simply make it a
 synchronized class with the classic primitives, instead of primitives
 that accept delegates etc.

 Nevertheless I think there's merit in this idea. One thing to point out
 is that the idiom can easily be done today with a regular class holding
 a synchronized class private member.

 So we got everything we need.


 Andrei

I was thinking about that. Here is what I ended up to think is the best 
solution :

synchronized classes exists. By default, they can't be use as parameter 
for synchronized(something) .

synchronized(something) will be valid is something provide 
opSynchronized(scope delegate void()) or something similar. Think 
opApply here. The synchronized statement is rewritten in a call to that 
delegate.

Here are the benefit of such an approach :
1/ Lock and unlock are not exposed. You can only use them by pair.
2/ You cannot lock on any object, so you avoid most liquid locks and 
don't waste memory.
3/ synchronized classes ensure that a class can be shared and it 
internal are protected from concurrent access.
4/ It is not possible possible by default to lock on synchronized 
classes's instances. It grant better control over the lock and it is now 
clear which piece of code is responsible of it.
5/ The design allow the programmer to grant the permission to lock on 
synchronized classes's instances if he/she want to.
6/ It is now possible to synchronize on a broader range of user defined 
stuffs.

The main drawback is the same as opApply : return (and break/continue 
but it is less relevant for opSynchronized). Solution to this problem 
have been proposed in the past using compiler and stack magic.

It open door for stuff like :
ReadWriteLock rw;
synchronized(rw.read) {

}

synchronized(rw.write) {

}

And many types of lock : spin lock, interprocesses locks, semaphores, . 
. . And all can be used with the synchronized syntax, and without 
exposing locking and unlocking primitives.

What do people think ?

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 01.06.2012 16:26, deadalnix wrote:
  Here is what I ended up to think is the best
 solution :

 synchronized classes exists. By default, they can't be use as parameter
 for synchronized(something) .

 synchronized(something) will be valid is something provide
 opSynchronized(scope delegate void()) or something similar. Think
 opApply here. The synchronized statement is rewritten in a call to that
 delegate.

 Here are the benefit of such an approach :
 1/ Lock and unlock are not exposed. You can only use them by pair.
 2/ You cannot lock on any object, so you avoid most liquid locks and
 don't waste memory.
 3/ synchronized classes ensure that a class can be shared and it
 internal are protected from concurrent access.
 4/ It is not possible possible by default to lock on synchronized
 classes's instances. It grant better control over the lock and it is now
 clear which piece of code is responsible of it.
 5/ The design allow the programmer to grant the permission to lock on
 synchronized classes's instances if he/she want to.
 6/ It is now possible to synchronize on a broader range of user defined
 stuffs.

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem
 have been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 And many types of lock : spin lock, interprocesses locks, semaphores, .
 . . And all can be used with the synchronized syntax, and without
 exposing locking and unlocking primitives.

 What do people think ?

+1. Works for me.

It refines what I believe the shadow cabinet (loosely: me, you, Alex, 
Regan Heath and Steven) propose.

P.S. Removing monitor from non-synced/shared classes would be good too. 
As a separate matter.

-- 
Dmitry Olshansky

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Fri, 01 Jun 2012 08:38:45 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 On 01.06.2012 16:26, deadalnix wrote:
  Here is what I ended up to think is the best
 solution :

 synchronized classes exists. By default, they can't be use as parameter
 for synchronized(something) .

 synchronized(something) will be valid is something provide
 opSynchronized(scope delegate void()) or something similar. Think
 opApply here. The synchronized statement is rewritten in a call to that
 delegate.

 Here are the benefit of such an approach :
 1/ Lock and unlock are not exposed. You can only use them by pair.
 2/ You cannot lock on any object, so you avoid most liquid locks and
 don't waste memory.
 3/ synchronized classes ensure that a class can be shared and it
 internal are protected from concurrent access.
 4/ It is not possible possible by default to lock on synchronized
 classes's instances. It grant better control over the lock and it is now
 clear which piece of code is responsible of it.
 5/ The design allow the programmer to grant the permission to lock on
 synchronized classes's instances if he/she want to.
 6/ It is now possible to synchronize on a broader range of user defined
 stuffs.

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem
 have been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 And many types of lock : spin lock, interprocesses locks, semaphores, .
 . . And all can be used with the synchronized syntax, and without
 exposing locking and unlocking primitives.

 What do people think ?

 +1. Works for me.

 It refines what I believe the shadow cabinet (loosely: me, you, Alex,  
 Regan Heath and Steven) propose.

Is this really necessary?  When is opSynchronized going to be written any  
way other than:

_mutex.lock();
scope(exit) _mutex.unlock();
dg();

I'll note that it's easier to forget to lock or unlock if the compiler  
isn't enforcing it.  You might even naively do this:

_mutex.lock();
dg();
_mutex.unlock(); // not called on exception thrown!

I kind of like the __lock() __unlock() pair that the compiler always calls  
both in the right place/way.  Yes, you could just leave those  
implementations blank, but very unlikely.

Plus, we already have issues with inout and delegates for opApply, this  
would have the same issues.

 P.S. Removing monitor from non-synced/shared classes would be good too.  
 As a separate matter.

I think at this point, we should leave it there until we can really figure  
out a detailed plan on how to deal with it.  It currently affects all  
runtime code which does virtual function lookups or interface lookups, and  
alignment.  We would have to change a lot of compiler and runtime code to  
remove it.

I personally don't see it as a huge issue, we are already allocating on  
power-of-two boundaries which can almost double required space.  I feel  
class size is really one of those things you should be oblivious to.  That  
being said, I'm all for performance improvement, even small ones, if they  
are free and someone is willing to do all the leg work :)

-Steve

deadalnix <deadalnix gmail.com> writes:

Le 01/06/2012 14:52, Steven Schveighoffer a écrit :
 On Fri, 01 Jun 2012 08:38:45 -0400, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:

 On 01.06.2012 16:26, deadalnix wrote:
 Here is what I ended up to think is the best
 solution :

 synchronized classes exists. By default, they can't be use as parameter
 for synchronized(something) .

 synchronized(something) will be valid is something provide
 opSynchronized(scope delegate void()) or something similar. Think
 opApply here. The synchronized statement is rewritten in a call to that
 delegate.

 Here are the benefit of such an approach :
 1/ Lock and unlock are not exposed. You can only use them by pair.
 2/ You cannot lock on any object, so you avoid most liquid locks and
 don't waste memory.
 3/ synchronized classes ensure that a class can be shared and it
 internal are protected from concurrent access.
 4/ It is not possible possible by default to lock on synchronized
 classes's instances. It grant better control over the lock and it is now
 clear which piece of code is responsible of it.
 5/ The design allow the programmer to grant the permission to lock on
 synchronized classes's instances if he/she want to.
 6/ It is now possible to synchronize on a broader range of user defined
 stuffs.

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem
 have been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 And many types of lock : spin lock, interprocesses locks, semaphores, .
 . . And all can be used with the synchronized syntax, and without
 exposing locking and unlocking primitives.

 What do people think ?

 +1. Works for me.

 It refines what I believe the shadow cabinet (loosely: me, you, Alex,
 Regan Heath and Steven) propose.

 Is this really necessary? When is opSynchronized going to be written any
 way other than:

 _mutex.lock();
 scope(exit) _mutex.unlock();
 dg();

 I'll note that it's easier to forget to lock or unlock if the compiler
 isn't enforcing it. You might even naively do this:

 _mutex.lock();
 dg();
 _mutex.unlock(); // not called on exception thrown!

 I kind of like the __lock() __unlock() pair that the compiler always
 calls both in the right place/way. Yes, you could just leave those
 implementations blank, but very unlikely.

 Plus, we already have issues with inout and delegates for opApply, this
 would have the same issues.

 P.S. Removing monitor from non-synced/shared classes would be good
 too. As a separate matter.

 I think at this point, we should leave it there until we can really
 figure out a detailed plan on how to deal with it. It currently affects
 all runtime code which does virtual function lookups or interface
 lookups, and alignment. We would have to change a lot of compiler and
 runtime code to remove it.

A lot of work have to be done, for sure. But concurency is the next big 
thing D will have to face IMO. When const/immutable are fixed :D

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 01-06-2012 14:26, deadalnix wrote:
 Le 31/05/2012 20:17, Andrei Alexandrescu a écrit :
 On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

 It can also be a lot clunkier for certain abstractions. Say I want a
 ProducerConsumerQueue. It's much more convenient to simply make it a
 synchronized class with the classic primitives, instead of primitives
 that accept delegates etc.

 Nevertheless I think there's merit in this idea. One thing to point out
 is that the idiom can easily be done today with a regular class holding
 a synchronized class private member.

 So we got everything we need.


 Andrei

 I was thinking about that. Here is what I ended up to think is the best
 solution :

 synchronized classes exists. By default, they can't be use as parameter
 for synchronized(something) .

 synchronized(something) will be valid is something provide
 opSynchronized(scope delegate void()) or something similar. Think
 opApply here. The synchronized statement is rewritten in a call to that
 delegate.

 Here are the benefit of such an approach :
 1/ Lock and unlock are not exposed. You can only use them by pair.
 2/ You cannot lock on any object, so you avoid most liquid locks and
 don't waste memory.
 3/ synchronized classes ensure that a class can be shared and it
 internal are protected from concurrent access.
 4/ It is not possible possible by default to lock on synchronized
 classes's instances. It grant better control over the lock and it is now
 clear which piece of code is responsible of it.
 5/ The design allow the programmer to grant the permission to lock on
 synchronized classes's instances if he/she want to.
 6/ It is now possible to synchronize on a broader range of user defined
 stuffs.

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem
 have been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 And many types of lock : spin lock, interprocesses locks, semaphores, .
 .. . And all can be used with the synchronized syntax, and without
 exposing locking and unlocking primitives.

 What do people think ?

Your idea is great, but it has one (or arguably many) fundamental flaw, 
the same one that opApply does: The delegate's type is fixed. That is, 
you can't call opSynchronized() in a pure function, for example (same 
goes for nothrow,  safe, ...).

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 01/06/2012 14:52, Alex Rønne Petersen a écrit :
 On 01-06-2012 14:26, deadalnix wrote:
 Le 31/05/2012 20:17, Andrei Alexandrescu a écrit :
 On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

 It can also be a lot clunkier for certain abstractions. Say I want a
 ProducerConsumerQueue. It's much more convenient to simply make it a
 synchronized class with the classic primitives, instead of primitives
 that accept delegates etc.

 Nevertheless I think there's merit in this idea. One thing to point out
 is that the idiom can easily be done today with a regular class holding
 a synchronized class private member.

 So we got everything we need.


 Andrei

 I was thinking about that. Here is what I ended up to think is the best
 solution :

 synchronized classes exists. By default, they can't be use as parameter
 for synchronized(something) .

 synchronized(something) will be valid is something provide
 opSynchronized(scope delegate void()) or something similar. Think
 opApply here. The synchronized statement is rewritten in a call to that
 delegate.

 Here are the benefit of such an approach :
 1/ Lock and unlock are not exposed. You can only use them by pair.
 2/ You cannot lock on any object, so you avoid most liquid locks and
 don't waste memory.
 3/ synchronized classes ensure that a class can be shared and it
 internal are protected from concurrent access.
 4/ It is not possible possible by default to lock on synchronized
 classes's instances. It grant better control over the lock and it is now
 clear which piece of code is responsible of it.
 5/ The design allow the programmer to grant the permission to lock on
 synchronized classes's instances if he/she want to.
 6/ It is now possible to synchronize on a broader range of user defined
 stuffs.

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem
 have been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 And many types of lock : spin lock, interprocesses locks, semaphores, .
 .. . And all can be used with the synchronized syntax, and without
 exposing locking and unlocking primitives.

 What do people think ?

 Your idea is great, but it has one (or arguably many) fundamental flaw,
 the same one that opApply does: The delegate's type is fixed. That is,
 you can't call opSynchronized() in a pure function, for example (same
 goes for nothrow,  safe, ...).

I was also thinking about passing the delegate as template parameter but 
wanted to keep design proposal consistent with what already exists. It 
solve some issues.

Maybe opApply have to be modified that way, or, better, both could work.

Anyway, some issue are known and have to be fixed with opApply, and the 
same issue are present here.I did mention that in my proposal. Still, I 
think it is better to have consistent mecanisms in the language, and it 
is no more work to fix that for opApply than to fix that for both 
opApply and opSynchronized .

Artur Skawina <art.08.09 gmail.com> writes:

On 06/01/12 14:26, deadalnix wrote:
 Le 31/05/2012 20:17, Andrei Alexandrescu a écrit :
 On 5/31/12 5:19 AM, deadalnix wrote:
 The solution consisting in passing a delegate as parameter or as
 template is superior, because it is now clear who is in charge of the
 synchronization, reducing greatly chances of deadlock.

 It can also be a lot clunkier for certain abstractions. Say I want a
 ProducerConsumerQueue. It's much more convenient to simply make it a
 synchronized class with the classic primitives, instead of primitives
 that accept delegates etc.

 Nevertheless I think there's merit in this idea. One thing to point out
 is that the idiom can easily be done today with a regular class holding
 a synchronized class private member.

 So we got everything we need.


 Andrei

 
 I was thinking about that. Here is what I ended up to think is the best
solution :
 
 synchronized classes exists. By default, they can't be use as parameter for
synchronized(something) .
 
 synchronized(something) will be valid is something provide
opSynchronized(scope delegate void()) or something similar. Think opApply here.
The synchronized statement is rewritten in a call to that delegate.

This has similar issues as opApply. It would have to be a template and
always inlined; The opLock/opUnlock approach lets you do the same things
w/o the delegate overhead and signature restrictions while making it a
bit harder to screw up the locking.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {
 
 }
 
 synchronized(rw.write) {
 
 }
 
 And many types of lock : spin lock, interprocesses locks, semaphores, . . .
And all can be used with the synchronized syntax, and without exposing locking
and unlocking primitives.
 
 What do people think ?

It can already be done using 'synchronized', it's *only* an issue
of efficiency and syntax. Eg right now i'm doing

      { scope s = somesemaphore.sync;
         whatever();
      }

and a properly lowered 'synchronized' would turn that into

      synchronized (somesemaphore) {
         whatever();
      }

Currently, the latter is probably possible, but not w/o a huge
perf hit; the former is practically free, just as if it was written
as:
    somesemaphore.wait();
    try whatever();
    finally somesemaphore.post();

Lowering 'synchronized' is about making the second form possible,
for anything resembling some kind of synchronization primitive.
OpSynchronized isn't necessary.

artur 

Sean Kelly <sean invisibleduck.org> writes:

On Jun 1, 2012, at 5:26 AM, deadalnix wrote:
=20
 The main drawback is the same as opApply : return (and break/continue =

but it is less relevant for opSynchronized). Solution to this problem =
have been proposed in the past using compiler and stack magic.
=20
 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {
=20
 }
=20
 synchronized(rw.write) {
=20
 }

Opens the door?  This works today exactly as outlined above.  Or am I =
missing a part of your argument?

 And many types of lock : spin lock, interprocesses locks, semaphores, =

. . . And all can be used with the synchronized syntax, and without =
exposing locking and unlocking primitives.

All works today.=

deadalnix <deadalnix gmail.com> writes:

Le 01/06/2012 22:55, Sean Kelly a écrit :
 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:
 The main drawback is the same as opApply : return (and break/continue but it
is less relevant for opSynchronized). Solution to this problem have been
proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 Opens the door?  This works today exactly as outlined above.  Or am I missing
a part of your argument?

 And many types of lock : spin lock, interprocesses locks, semaphores, . . .
And all can be used with the synchronized syntax, and without exposing locking
and unlocking primitives.

 All works today.

Unless you do some monitor magic, it doesn't.

Andrew Wiley <wiley.andrew.j gmail.com> writes:

On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com> wrote:

 Le 01/06/2012 22:55, Sean Kelly a =E9crit :

 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:

 The main drawback is the same as opApply : return (and break/continue
 but it is less relevant for opSynchronized). Solution to this problem h=



ave
 been proposed in the past using compiler and stack magic.

 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {

 }

 synchronized(rw.write) {

 }

 Opens the door?  This works today exactly as outlined above.  Or am I
 missing a part of your argument?

  And many types of lock : spin lock, interprocesses locks, semaphores, .
 . . And all can be used with the synchronized syntax, and without expos=



ing
 locking and unlocking primitives.

 All works today.

 Unless you do some monitor magic, it doesn't.

Yes, it does.
-----
class Something {
    private:
        ReadWriteLock _rw;
    public:
        this() {
            _rw =3D new ReadWriteLock();
        }
        void doSomething() shared {
            synchronized(_rw.read) {
                // do things
            }
        }
}
-----

I've used this pattern in code. There might be some casting required
because the core synchronization primitives haven't been updated to use
shared yet.

deadalnix <deadalnix gmail.com> writes:

Le 03/06/2012 21:40, Andrew Wiley a écrit :
 On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     Le 01/06/2012 22:55, Sean Kelly a écrit :

         On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


             The main drawback is the same as opApply : return (and
             break/continue but it is less relevant for opSynchronized).
             Solution to this problem have been proposed in the past
             using compiler and stack magic.

             It open door for stuff like :
             ReadWriteLock rw;
             synchronized(rw.read) {

             }

             synchronized(rw.write) {

             }


         Opens the door?  This works today exactly as outlined above.  Or
         am I missing a part of your argument?

             And many types of lock : spin lock, interprocesses locks,
             semaphores, . . . And all can be used with the synchronized
             syntax, and without exposing locking and unlocking primitives.


         All works today.


     Unless you do some monitor magic, it doesn't.

 Yes, it does.
 -----
 class Something {
      private:
          ReadWriteLock _rw;
      public:
          this() {
              _rw = new ReadWriteLock();
          }
          void doSomething() shared {
              synchronized(_rw.read) {
                  // do things
              }
          }
 }
 -----
 I've used this pattern in code. There might be some casting required
 because the core synchronization primitives haven't been updated to use
 shared yet.

And where is that ReadWriteLock ?

Andrew Wiley <wiley.andrew.j gmail.com> writes:

On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com> wrote:

 Le 03/06/2012 21:40, Andrew Wiley a =E9crit :

 On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

    Le 01/06/2012 22:55, Sean Kelly a =E9crit :

        On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


            The main drawback is the same as opApply : return (and
            break/continue but it is less relevant for opSynchronized).
            Solution to this problem have been proposed in the past
            using compiler and stack magic.

            It open door for stuff like :
            ReadWriteLock rw;
            synchronized(rw.read) {

            }

            synchronized(rw.write) {

            }


        Opens the door?  This works today exactly as outlined above.  Or
        am I missing a part of your argument?

            And many types of lock : spin lock, interprocesses locks,
            semaphores, . . . And all can be used with the synchronized
            syntax, and without exposing locking and unlocking primitives=


.
        All works today.


    Unless you do some monitor magic, it doesn't.

 Yes, it does.
 -----
 class Something {
     private:
         ReadWriteLock _rw;
     public:
         this() {
             _rw =3D new ReadWriteLock();
         }
         void doSomething() shared {
             synchronized(_rw.read) {
                 // do things
             }
         }
 }
 -----
 I've used this pattern in code. There might be some casting required
 because the core synchronization primitives haven't been updated to use
 shared yet.

 And where is that ReadWriteLock ?

On the GC heap, just like the Monitor object pointed to by __monitor if you
mark a method or class as synchronized.

deadalnix <deadalnix gmail.com> writes:

Le 04/06/2012 02:03, Andrew Wiley a écrit :
 On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     Le 03/06/2012 21:40, Andrew Wiley a écrit :

         On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
         <mailto:deadalnix gmail.com>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>> wrote:

             Le 01/06/2012 22:55, Sean Kelly a écrit :

                 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


                     The main drawback is the same as opApply : return (and
                     break/continue but it is less relevant for
         opSynchronized).
                     Solution to this problem have been proposed in the past
                     using compiler and stack magic.

                     It open door for stuff like :
                     ReadWriteLock rw;
                     synchronized(rw.read) {

                     }

                     synchronized(rw.write) {

                     }


                 Opens the door?  This works today exactly as outlined
         above.  Or
                 am I missing a part of your argument?

                     And many types of lock : spin lock, interprocesses
         locks,
                     semaphores, . . . And all can be used with the
         synchronized
                     syntax, and without exposing locking and unlocking
         primitives.


                 All works today.


             Unless you do some monitor magic, it doesn't.

         Yes, it does.
         -----
         class Something {
              private:
                  ReadWriteLock _rw;
              public:
                  this() {
                      _rw = new ReadWriteLock();
                  }
                  void doSomething() shared {
                      synchronized(_rw.read) {
                          // do things
                      }
                  }
         }
         -----
         I've used this pattern in code. There might be some casting required
         because the core synchronization primitives haven't been updated
         to use
         shared yet.


     And where is that ReadWriteLock ?

 On the GC heap, just like the Monitor object pointed to by __monitor if
 you mark a method or class as synchronized.

I meant where is the code ?

Andrew Wiley <wiley.andrew.j gmail.com> writes:

On Sun, Jun 3, 2012 at 5:13 PM, deadalnix <deadalnix gmail.com> wrote:

 Le 04/06/2012 02:03, Andrew Wiley a =E9crit :

 On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com

 <mailto:deadalnix gmail.com>> wrote:

    Le 03/06/2012 21:40, Andrew Wiley a =E9crit :

        On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
        <mailto:deadalnix gmail.com>
        <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>> wrote:

            Le 01/06/2012 22:55, Sean Kelly a =E9crit :

                On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


                    The main drawback is the same as opApply : return (an=


d
                    break/continue but it is less relevant for
        opSynchronized).
                    Solution to this problem have been proposed in the pa=


st
                    using compiler and stack magic.

                    It open door for stuff like :
                    ReadWriteLock rw;
                    synchronized(rw.read) {

                    }

                    synchronized(rw.write) {

                    }


                Opens the door?  This works today exactly as outlined
        above.  Or
                am I missing a part of your argument?

                    And many types of lock : spin lock, interprocesses
        locks,
                    semaphores, . . . And all can be used with the
        synchronized
                    syntax, and without exposing locking and unlocking
        primitives.


                All works today.


            Unless you do some monitor magic, it doesn't.

        Yes, it does.
        -----
        class Something {
             private:
                 ReadWriteLock _rw;
             public:
                 this() {
                     _rw =3D new ReadWriteLock();
                 }
                 void doSomething() shared {
                     synchronized(_rw.read) {
                         // do things
                     }
                 }
        }
        -----
        I've used this pattern in code. There might be some casting
 required
        because the core synchronization primitives haven't been updated
        to use
        shared yet.


    And where is that ReadWriteLock ?

 On the GC heap, just like the Monitor object pointed to by __monitor if
 you mark a method or class as synchronized.

 I meant where is the code ?

My apologies, it's actually ReadWriteMutex:
http://dlang.org/phobos/core_sync_rwmutex.html

deadalnix <deadalnix gmail.com> writes:

Le 04/06/2012 02:21, Andrew Wiley a écrit :
 On Sun, Jun 3, 2012 at 5:13 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     Le 04/06/2012 02:03, Andrew Wiley a écrit :

         On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com
         <mailto:deadalnix gmail.com>

         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>> wrote:

             Le 03/06/2012 21:40, Andrew Wiley a écrit :

                 On Sun, Jun 3, 2012 at 12:29 PM, deadalnix
         <deadalnix gmail.com <mailto:deadalnix gmail.com>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>>> wrote:

                     Le 01/06/2012 22:55, Sean Kelly a écrit :

                         On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


                             The main drawback is the same as opApply :
         return (and
                             break/continue but it is less relevant for
                 opSynchronized).
                             Solution to this problem have been proposed
         in the past
                             using compiler and stack magic.

                             It open door for stuff like :
                             ReadWriteLock rw;
                             synchronized(rw.read) {

                             }

                             synchronized(rw.write) {

                             }


                         Opens the door?  This works today exactly as
         outlined
                 above.  Or
                         am I missing a part of your argument?

                             And many types of lock : spin lock,
         interprocesses
                 locks,
                             semaphores, . . . And all can be used with the
                 synchronized
                             syntax, and without exposing locking and
         unlocking
                 primitives.


                         All works today.


                     Unless you do some monitor magic, it doesn't.

                 Yes, it does.
                 -----
                 class Something {
                      private:
                          ReadWriteLock _rw;
                      public:
                          this() {
                              _rw = new ReadWriteLock();
                          }
                          void doSomething() shared {
                              synchronized(_rw.read) {
                                  // do things
                              }
                          }
                 }
                 -----
                 I've used this pattern in code. There might be some
         casting required
                 because the core synchronization primitives haven't been
         updated
                 to use
                 shared yet.


             And where is that ReadWriteLock ?

         On the GC heap, just like the Monitor object pointed to by
         __monitor if
         you mark a method or class as synchronized.


     I meant where is the code ?

 My apologies, it's actually ReadWriteMutex:
 http://dlang.org/phobos/core_sync_rwmutex.html

Which does use monitor magic.

=?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 04-06-2012 02:03, Andrew Wiley wrote:
 On Sun, Jun 3, 2012 at 4:39 PM, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     Le 03/06/2012 21:40, Andrew Wiley a écrit :

         On Sun, Jun 3, 2012 at 12:29 PM, deadalnix <deadalnix gmail.com
         <mailto:deadalnix gmail.com>
         <mailto:deadalnix gmail.com <mailto:deadalnix gmail.com>>> wrote:

             Le 01/06/2012 22:55, Sean Kelly a écrit :

                 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:


                     The main drawback is the same as opApply : return (and
                     break/continue but it is less relevant for
         opSynchronized).
                     Solution to this problem have been proposed in the past
                     using compiler and stack magic.

                     It open door for stuff like :
                     ReadWriteLock rw;
                     synchronized(rw.read) {

                     }

                     synchronized(rw.write) {

                     }


                 Opens the door?  This works today exactly as outlined
         above.  Or
                 am I missing a part of your argument?

                     And many types of lock : spin lock, interprocesses
         locks,
                     semaphores, . . . And all can be used with the
         synchronized
                     syntax, and without exposing locking and unlocking
         primitives.


                 All works today.


             Unless you do some monitor magic, it doesn't.

         Yes, it does.
         -----
         class Something {
              private:
                  ReadWriteLock _rw;
              public:
                  this() {
                      _rw = new ReadWriteLock();
                  }
                  void doSomething() shared {
                      synchronized(_rw.read) {
                          // do things
                      }
                  }
         }
         -----
         I've used this pattern in code. There might be some casting required
         because the core synchronization primitives haven't been updated
         to use
         shared yet.


     And where is that ReadWriteLock ?

 On the GC heap, just like the Monitor object pointed to by __monitor if
 you mark a method or class as synchronized.

The monitor in obj.__monitor is actually allocated on the native C heap, 
and, in some cases, leaked... one of the many reasons I want it gone.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Sean Kelly <sean invisibleduck.org> writes:

On Jun 3, 2012, at 5:24 PM, Alex R=F8nne Petersen wrote:
=20
 The monitor in obj.__monitor is actually allocated on the native C =

heap, and, in some cases, leaked... one of the many reasons I want it =
gone.

The benefit of it being on the native C heap is that you can use =
synchronized{} in the object's dtor.  You can't do this if the monitor =
is on the GC heap, as Mutex is.=

Sean Kelly <sean invisibleduck.org> writes:

On Jun 3, 2012, at 12:29 PM, deadalnix wrote:

 Le 01/06/2012 22:55, Sean Kelly a =E9crit :
 On Jun 1, 2012, at 5:26 AM, deadalnix wrote:
=20
 The main drawback is the same as opApply : return (and =



break/continue but it is less relevant for opSynchronized). Solution to =
this problem have been proposed in the past using compiler and stack =
magic.
=20
 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {
=20
 }
=20
 synchronized(rw.write) {
=20
 }

=20
 Opens the door?  This works today exactly as outlined above.  Or am I =


missing a part of your argument?
=20
 And many types of lock : spin lock, interprocesses locks, =



semaphores, . . . And all can be used with the synchronized syntax, and =
without exposing locking and unlocking primitives.
=20
 All works today.

=20
 Unless you do some monitor magic, it doesn't.

There is a bit of cleverness in there.

travert phare.normalesup.org (Christophe Travert) writes:

deadalnix , dans le message (digitalmars.D:169136), a écrit :
 It open door for stuff like :
 ReadWriteLock rw;
 synchronized(rw.read) {
 
 }
 
 synchronized(rw.write) {
 
 }
 
 And many types of lock : spin lock, interprocesses locks, semaphores, . 
 . . And all can be used with the synchronized syntax, and without 
 exposing locking and unlocking primitives.
 

synchronize (foo) { ... } could do anything.

You might as well propose a syntax sugar to call any function or method 
taking a delegate. You could make it do any operation with the idiom:

struct MyType {
  void foo(void delegate() dg) {
    this.open();
    scope(exit) this.close();
    dg();
  }
  ...
}

a.foo { // maybe some keyword should be necessary
  ...
}

Same problem as opApply, should take different type of delegates, etc...

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 12:13:00 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 On 5/31/12 3:27 AM, Regan Heath wrote:
 I think
 the mutex is "available for locking and unlocking" <- need a word for
 that, which is not "exposed". How about accessible, available, usable,
 or just plain lockable .. So, the problem here is that the mutex is
 lockable by external code via synchronized() and this means a certain
 type of deadlock is possible.

 But this is a protection/visibility issue, which is orthogonal on the  
 locking capability. It's as if you say "int is not good because anyone  
 can overflow it." Okay! Make it private inside a CheckedInt class.

Sorry, that's a bad comparison.  CheckedInt is to int, what CheckedMutex  
is to mutex - but I'm not suggesting anything like a CheckedMutex.  I'm  
suggesting "mutex" but kept private inside the class /that it locks/.   
Yes, it's a visibility issue, the issue is that the mutex used by  
synchronized classes/methods is too visible/accessible and this opens it  
up for deadlocks which are otherwise impossible.

 I think the change mentioned in TDPL to restrict synchronized to
 synchronized classes is a step in the right direction WRT wasted monitor
 space and people freely locking anything. But, it is exactly the case
 which results in more possible deadlocks (the cause of this thread) AND
 I think it's actually far more likely people will want to use a
 synchronized statement on a class which is not itself synchronized, like
 for example an existing container class.

 Given that, restricting synchronized statements to synchronized classes
 seems entirely wrong to me.

 So where's the mutex that would be used to synchronize objects that are  
 not synchronizable?

In the wrapper class/struct/object which derives a synchronized  
class/struct from the original.  My D foo is not strong enough to just  
come up with valid D code for the idiom on the fly, but essentially you  
wrap the original object in a new object using a template which adds the  
mutex member and the interface methods (lock, tryLock, and unlock)  
required.  No, this doesn't work with "final" classes.. but it shouldn't,  
they're final after all.  For them you need to add/manage the mutex  
manually - the price you pay for "final".

 In fact, I would say you almost want to stop
 people using synchronized statements on synchronized classes because:
 1. If a synchronized class is written correctly it should not be
 necessary in the general case(*)
 2. It raises the chances of deadlocks (the cause of this thread).
 3. It means that classes in general will be simpler to write (no need to
 worry about synchronization) and also more lightweight for use in
 non-threaded/non-shared cases. (because we'd provide a template wrapper
 to make them synchronizable)

 There are cases in which you want to do multiple operations under a  
 single critical section, even though the API is otherwise well-designed.  
 That may be for correctness, efficiency, or both. I don't see why we'd  
 want to disallow that, it's a good idiom.

Who suggested disallowing this?  No-one.  There are 3 main use cases I see  
for this;

1. Several disparate objects locked by a single mutex - in which case the  
correct solution is a separate mutex/monitor object.
2. A single object, locked for serveral method calls - in which case the  
method-passed-a-delegate idea (mentioned below/ described in a separate  
thread) works, unless..
3. The calls span several scopes, in which case good-old manual  
lock/unlock is required (synchronized blocks don't help here)

 So, more and more I'm thinking it would be better to provide a
 library/runtime co-operative solution where we have an interface which
 is required for synchronized statements, and a wrapper template to
 implement that interface for any existing non-synchronized class.
 Meaning, the choice is either to write a synchronized class (rare) or a
 non-synchronized class - knowing it can easily be synchronized if/when
 needed.

 Looking forward for a fleshed out proposal. Make sure you motivate it  
 properly.

Sorry, I have no spare time to spare.  You're getting free ideas/thoughts  
 from me, feel free to ignore them.

 The "liquid lock" problem mentioned earlier is an interesting one that I
 have not personally experienced, perhaps because I don't lock anything
 but mutex primitives and I never to re-assign these.

 (*) Locking on a larger scope can be achieved by providing a method
 taking a delegate (see earlier thread/replies) and/or exposing
 lock/unlock for those few situations where the delegate method cannot be
 used. These are the few cases in which this cannot be avoided as the
 lock/unlock are separated by more than a single scope (so synchronized
 statements don't help in these cases either).

 On first look, the inversion of control using delegates delegates has  
 similar liabilities as straight scoped locking.

True, it's basically the same as a synchronized block in that respect.   
What we actually want is a way to limit the calls made by the delegate to  
methods of the object itself.  If it could not call a synchronized method  
on a 2nd object, you could not possibly deadlock.  Except, that is to say,  
unless you held a separate lock beforehand - but, the important point here  
is that you would have to take both locks explicitly, rather than by an  
implicit synchronized method call, making the bug far more obvious to code  
inspection.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

deadalnix <deadalnix gmail.com> writes:

Le 31/05/2012 16:01, Regan Heath a écrit :
 True, it's basically the same as a synchronized block in that respect.
 What we actually want is a way to limit the calls made by the delegate
 to methods of the object itself. If it could not call a synchronized
 method on a 2nd object, you could not possibly deadlock. Except, that is
 to say, unless you held a separate lock beforehand - but, the important
 point here is that you would have to take both locks explicitly, rather
 than by an implicit synchronized method call, making the bug far more
 obvious to code inspection.

 R

I'm not sure I follow you here. Can you elaborate on that ?

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 15:06:14 +0100, deadalnix <deadalnix gmail.com> wrot=
e:

 Le 31/05/2012 16:01, Regan Heath a =E9crit :
 True, it's basically the same as a synchronized block in that respect=


.
 What we actually want is a way to limit the calls made by the delegat=


e
 to methods of the object itself. If it could not call a synchronized
 method on a 2nd object, you could not possibly deadlock. Except, that=


 is
 to say, unless you held a separate lock beforehand - but, the importa=


nt
 point here is that you would have to take both locks explicitly, rath=


er
 than by an implicit synchronized method call, making the bug far more=


 obvious to code inspection.

 R

 I'm not sure I follow you here. Can you elaborate on that ?

Apologies in advance, my D code skillz have rusted significantly..

class A
{
   synchronized void foo()
   {
   }
}

class B {}

void main()
{
   A a =3D new A();
   B b =3D new B();

   synchronized(b)  // locks B
   {
     a.foo();       // locks A
   }
}

.. this deadlocks if another thread locks A then B anywhere.  The  =

"problem" is that this can be hard to spot and easy to do accidentally a=
s  =

a.foo() does not scream "I'm going to lock something".

This is the same problem with the synchronized method calling a supplied=
  =

delegate..

class A
{
   synchronized void foo()
   {
   }
}

class B
{
   synchronized void syncDelegate(void delegate(void) dg) { .. }
}

void main()
{
   A a =3D new A();
   B b =3D new B();

   b.syncDelegate(.. code which calls a.foo() .. );  // locks B  =

(syncDelegate), then locks A (a.foo)
}

The only way to avoid the deadlock is to prevent calls to synchronized  =

methods inside the delegate.

R

-- =

Using Opera's revolutionary email client: http://www.opera.com/mail/

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

 Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex
 is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm
 suggesting "mutex" but kept private inside the class /that it locks/.
 Yes, it's a visibility issue, the issue is that the mutex used by
 synchronized classes/methods is too visible/accessible and this opens it
 up for deadlocks which are otherwise impossible.

Sure it's awful comparison.


 So where's the mutex that would be used to synchronize objects that
 are not synchronizable?

 In the wrapper class/struct/object which derives a synchronized
 class/struct from the original. My D foo is not strong enough to just
 come up with valid D code for the idiom on the fly, but essentially you
 wrap the original object in a new object using a template which adds the
 mutex member and the interface methods (lock, tryLock, and unlock)
 required. No, this doesn't work with "final" classes.. but it shouldn't,
 they're final after all. For them you need to add/manage the mutex
 manually - the price you pay for "final".

OK let me land you a hand here. My proposal, that I think fits your 
ideas quite favorably.

I'll enumerate certain assumptions beforehand since it's one of most 
confusing threads I ever followed.

1. synchronized class means: always allocate hidden monitor mutex, lock 
it on every public method of this class.

2. synchronized(x,y,z) is lowered to (I use Steven's idea):
	auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z 
sorted
	FOR EACH item IN sorted tuple add code in [[ ... ]]
	[[// conceptual
		item.__lock();
		scope(exit) item.__unlock();
	]]
In other words it works for every object that defines lock/unlock. 
Multiple object version works only if there is opCmp defined. (by 
address whatever, any total ordering should work)


Per definition above synchronized classes AS IS can't be *synchronized* 
on. Instead their public methods are implicitly synchronized.

The end result is:

User can synchronize on various synchronization entities and even plug 
in custom synchronization primitives (say OS Y provides have this fancy 
lock type Z). It's explicit in as sense that object supports it via 
__lock__/unlock. Obviously one still can use __lock/__unlock explicitly 
just don't forget to wear big HERE BE DRAGONS banner.

Synchronized class encapsulate mutex completely - nobody aside from 
designer of the class may (a)use it.

Does it makes sense?

-- 
Dmitry Olshansky

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 15:49:52 +0100, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your  
 ideas quite favorably.

 I'll enumerate certain assumptions beforehand since it's one of most  
 confusing threads I ever followed.

 1. synchronized class means: always allocate hidden monitor mutex, lock  
 it on every public method of this class.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 	auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z  
 sorted
 	FOR EACH item IN sorted tuple add code in [[ ... ]]
 	[[// conceptual
 		item.__lock();
 		scope(exit) item.__unlock();
 	]]
 In other words it works for every object that defines lock/unlock.  
 Multiple object version works only if there is opCmp defined. (by  
 address whatever, any total ordering should work)


 Per definition above synchronized classes AS IS can't be *synchronized*  
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug  
 in custom synchronization primitives (say OS Y provides have this fancy  
 lock type Z). It's explicit in as sense that object supports it via  
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly  
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from  
 designer of the class may (a)use it.

 Does it makes sense?

Yes, and it's all more intentional/flexible than what we have now, but.. :)

Does it address what I thought was the main "problem" case.  That is, as  
soon as you lock 2 objects, one via a synchronized() statement and the  
other via a synchronized method you can get a non-obvious deadlock. e.g.

synchronized class A
{
   void foo() {} // implicitly locks instance of A
}

class B
{
   void __lock() {}   // locks instance of B
   void __unlock() {} // unlocks instance of B
}

shared A a;
shared B b;

void main()
{
   a = new shared(A)();
   b = new shared(B)();
   ..start thread which locks a then b..
   synchronized(b)  // locks b 'explicitly'
   {
     a.foo();       // locks a 'implicitly'
   }
}

.. but, hang on, can a thread actually lock a and then b?  If 'a' cannot  
participate in a synchronized statement (which it can't under this  
proposal) then no, there is no way to lock 'a' except by calling a  
member.  So, provided 'a' does not have a member which locks 'b' - were  
deadlock safe!

So.. problem solved; by preventing external/public lock/unlock on a  
synchronized class.  (I think the proposal should enforce this  
restriction; synchronized classes cannot define __lock/__unlock).

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 31.05.2012 19:05, Regan Heath wrote:
 Yes, and it's all more intentional/flexible than what we have now, but.. :)

 Does it address what I thought was the main "problem" case. That is, as
 soon as you lock 2 objects, one via a synchronized() statement and the
 other via a synchronized method you can get a non-obvious deadlock. e.g.

 synchronized class A
 {
 void foo() {} // implicitly locks instance of A
 }

 class B
 {
 void __lock() {} // locks instance of B
 void __unlock() {} // unlocks instance of B
 }

 shared A a;
 shared B b;

 void main()
 {
 a = new shared(A)();
 b = new shared(B)();
 ..start thread which locks a then b..
 synchronized(b) // locks b 'explicitly'
 {
 a.foo(); // locks a 'implicitly'
 }
 }

 .. but, hang on, can a thread actually lock a and then b? If 'a' cannot
 participate in a synchronized statement (which it can't under this
 proposal) then no, there is no way to lock 'a' except by calling a
 member. So, provided 'a' does not have a member which locks 'b' - were
 deadlock safe!


 So.. problem solved; by preventing external/public lock/unlock on a
 synchronized class. (I think the proposal should enforce this
 restriction; synchronized classes cannot define __lock/__unlock).

Surprisingly it is. And if A can't access B it's fool-proof.
As to __lock/__unlock defined for synchronized classes, yes I believe 
compiler should catch it.


-- 
Dmitry Olshansky

mta`chrono <chrono mta-international.net> writes:

Am 31.05.2012 17:05, schrieb Regan Heath:
 .. but, hang on, can a thread actually lock a and then b?  If 'a' cannot
 participate in a synchronized statement (which it can't under this
 proposal) then no, there is no way to lock 'a' except by calling a
 member.  So, provided 'a' does not have a member which locks 'b' - were
 deadlock safe!
 
 So.. problem solved; by preventing external/public lock/unlock on a
 synchronized class.  (I think the proposal should enforce this
 restriction; synchronized classes cannot define __lock/__unlock).
 
 R
 

I think it doesn't matter whether you expose your mointor / locking /
unlocking to the public or not. You can always unhappily create
deadlocks that are hard to debug between tons of spaghetti code.

shared A a;
shared B b;

void thread1()
{
     synchronized(a)  // locks A
     {
         synchronized(b)  // ... then B
         {
             // ..... code ....
         }
     }
}

void thread2()
{
     synchronized(b) // locks B
     {
         synchronized(a) // ... then A
         {
             // ..... code ....
         }
     }
}

Jonathan M Davis <jmdavisProg gmx.com> writes:

On Monday, June 04, 2012 10:51:08 mta`chrono wrote:
 Am 31.05.2012 17:05, schrieb Regan Heath:
 .. but, hang on, can a thread actually lock a and then b?  If 'a' cannot
 participate in a synchronized statement (which it can't under this
 proposal) then no, there is no way to lock 'a' except by calling a
 member.  So, provided 'a' does not have a member which locks 'b' - were
 deadlock safe!
 
 So.. problem solved; by preventing external/public lock/unlock on a
 synchronized class.  (I think the proposal should enforce this
 restriction; synchronized classes cannot define __lock/__unlock).
 
 R

 
 I think it doesn't matter whether you expose your mointor / locking /
 unlocking to the public or not. You can always unhappily create
 deadlocks that are hard to debug between tons of spaghetti code.

You can always create deadlocks, but if there's something which gives you 
little benefit but significantly increases the risk of deadlocks (e.g. making
it 
easy to lock on a synchronized class' internal mutex via a synchronized 
block), then it's valuable to make it illegal. Because while it won't prevent 
all deadlocking, it _does_ eliminate one case where it's overly easy to 
deadlock.

- Jonathan M Davis

deadalnix <deadalnix gmail.com> writes:

Le 04/06/2012 10:56, Jonathan M Davis a écrit :
 On Monday, June 04, 2012 10:51:08 mta`chrono wrote:
 Am 31.05.2012 17:05, schrieb Regan Heath:
 .. but, hang on, can a thread actually lock a and then b?  If 'a' cannot
 participate in a synchronized statement (which it can't under this
 proposal) then no, there is no way to lock 'a' except by calling a
 member.  So, provided 'a' does not have a member which locks 'b' - were
 deadlock safe!

 So.. problem solved; by preventing external/public lock/unlock on a
 synchronized class.  (I think the proposal should enforce this
 restriction; synchronized classes cannot define __lock/__unlock).

 R

 I think it doesn't matter whether you expose your mointor / locking /
 unlocking to the public or not. You can always unhappily create
 deadlocks that are hard to debug between tons of spaghetti code.

 You can always create deadlocks, but if there's something which gives you
 little benefit but significantly increases the risk of deadlocks (e.g. making
it
 easy to lock on a synchronized class' internal mutex via a synchronized
 block), then it's valuable to make it illegal. Because while it won't prevent
 all deadlocking, it _does_ eliminate one case where it's overly easy to
 deadlock.

 - Jonathan M Davis

At least illegal by default. The programmer may enable it by 
him/herself, but not fall in the trap inadvertently.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Thu, 31 May 2012 10:49:52 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your  
 ideas quite favorably.

[snip]

 Does it makes sense?

Everything but the ordering.  I like the idea of ordering the locks based  
on an intrinsic value, but opCmp isn't it.  Objects can mutate, and opCmp  
may produce different results.  Imagine:

synchronized(a, b) // at this point a < b
{
    a.makeGreaterThan(b);
    assert(a > b);
}

You *never* want the ordering to change.

But I think we can probably work that out.  What about comparing handles  
of the mutexes?  So you sort based on some property __mutex_id() which  
must return a unique size_t that can be used to always define an ordering  
of locking mutexes?  Most mutexes are allocated by the OS, and so their  
handles won't be affected by a moving GC, so you likely will use the  
handle value.

-Steve

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 31.05.2012 19:11, Steven Schveighoffer wrote:
 On Thu, 31 May 2012 10:49:52 -0400, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your
 ideas quite favorably.

 [snip]

 Does it makes sense?

 Everything but the ordering. I like the idea of ordering the locks based
 on an intrinsic value, but opCmp isn't it. Objects can mutate, and opCmp
 may produce different results. Imagine:

 synchronized(a, b) // at this point a < b
 {
 a.makeGreaterThan(b);
 assert(a > b);
 }

No way! I live in world where victim's hand is cut off as a punishment 
for mutating a lock after creation ;)

 You *never* want the ordering to change.

 But I think we can probably work that out. What about comparing handles
 of the mutexes? So you sort based on some property __mutex_id() which
 must return a unique size_t that can be used to always define an
 ordering of locking mutexes? Most mutexes are allocated by the OS, and
 so their handles won't be affected by a moving GC, so you likely will
 use the handle value.

This could work. In fact I imagined comparing handles ...
As with math at large you may either project (biject) your domain to a 
well-known one (like numbers, mutex_id in your case) and work with it or 
define all relevant properties for whatever your values in this domain 
are (providing custom ordering for user defined types).



-- 
Dmitry Olshansky

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 16:23:29 +0100, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 On 31.05.2012 19:11, Steven Schveighoffer wrote:
 On Thu, 31 May 2012 10:49:52 -0400, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your
 ideas quite favorably.

 [snip]

 Does it makes sense?

 Everything but the ordering. I like the idea of ordering the locks based
 on an intrinsic value, but opCmp isn't it. Objects can mutate, and opCmp
 may produce different results. Imagine:

 synchronized(a, b) // at this point a < b
 {
 a.makeGreaterThan(b);
 assert(a > b);
 }

 No way! I live in world where victim's hand is cut off as a punishment  
 for mutating a lock after creation ;)

This is related to the "liquid lock" problem raised/mentioned elsewhere in  
the thread.  If the reference you lock can change, then one  
thread/iteration may lock one instance, the lock instance mutates, and a  
2nd thread/iteration locks the new instance and both execute more or less  
in parallel over code which was supposed to be serialized.  It's not a  
problem if the lock object is the object being mutated/used by the code,  
but it is a problem if the lock object is protecting other shared  
objects.  It's not something I have ever bumped into myself, because the  
bulk of my locking experience is in C/C++ and I use a locking primitive  
which is initialised once and never mutates.

 You *never* want the ordering to change.

 But I think we can probably work that out. What about comparing handles
 of the mutexes? So you sort based on some property __mutex_id() which
 must return a unique size_t that can be used to always define an
 ordering of locking mutexes? Most mutexes are allocated by the OS, and
 so their handles won't be affected by a moving GC, so you likely will
 use the handle value.

 This could work. In fact I imagined comparing handles ...
 As with math at large you may either project (biject) your domain to a  
 well-known one (like numbers, mutex_id in your case) and work with it or  
 define all relevant properties for whatever your values in this domain  
 are (providing custom ordering for user defined types).

This ordering idea is present in TDPL.. have you both read the link Andrei  
posted?
http://goo.gl/ZhPM2
(see 13.15)


R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 31.05.2012 20:54, Regan Heath wrote:

 No way! I live in world where victim's hand is cut off as a punishment
 for mutating a lock after creation ;)

 This is related to the "liquid lock" problem raised/mentioned elsewhere
 in the thread. If the reference you lock can change, then one
 thread/iteration may lock one instance, the lock instance mutates, and a
 2nd thread/iteration locks the new instance and both execute more or
 less in parallel over code which was supposed to be serialized. It's not
 a problem if the lock object is the object being mutated/used by the
 code, but it is a problem if the lock object is protecting other shared
 objects. It's not something I have ever bumped into myself, because the
 bulk of my locking experience is in C/C++ and I use a locking primitive
 which is initialised once and never mutates.

Yup, C/C++, locking only special OS provided stuff, it's my background 
exactly.

 You *never* want the ordering to change.

 But I think we can probably work that out. What about comparing handles
 of the mutexes? So you sort based on some property __mutex_id() which
 must return a unique size_t that can be used to always define an
 ordering of locking mutexes? Most mutexes are allocated by the OS, and
 so their handles won't be affected by a moving GC, so you likely will
 use the handle value.

 This could work. In fact I imagined comparing handles ...
 As with math at large you may either project (biject) your domain to a
 well-known one (like numbers, mutex_id in your case) and work with it
 or define all relevant properties for whatever your values in this
 domain are (providing custom ordering for user defined types).

 This ordering idea is present in TDPL.. have you both read the link
 Andrei posted?
 http://goo.gl/ZhPM2
 (see 13.15)

Yes, in fact I was going by TDPL. I just propose to extend it to 
entities other then class instances ( more specifically only those with 
__lock/__unlock ).

-- 
Dmitry Olshansky

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Thu, 31 May 2012 12:54:07 -0400, Regan Heath <regan netmail.co.nz>  
wrote:

 On Thu, 31 May 2012 16:23:29 +0100, Dmitry Olshansky  
 <dmitry.olsh gmail.com> wrote:

 On 31.05.2012 19:11, Steven Schveighoffer wrote:
 On Thu, 31 May 2012 10:49:52 -0400, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:
 OK let me land you a hand here. My proposal, that I think fits your
 ideas quite favorably.

 [snip]

 Does it makes sense?

 Everything but the ordering. I like the idea of ordering the locks  
 based
 on an intrinsic value, but opCmp isn't it. Objects can mutate, and  
 opCmp
 may produce different results. Imagine:

 synchronized(a, b) // at this point a < b
 {
 a.makeGreaterThan(b);
 assert(a > b);
 }

 No way! I live in world where victim's hand is cut off as a punishment  
 for mutating a lock after creation ;)

 This is related to the "liquid lock" problem raised/mentioned elsewhere  
 in the thread.  If the reference you lock can change, then one  
 thread/iteration may lock one instance, the lock instance mutates, and a  
 2nd thread/iteration locks the new instance and both execute more or  
 less in parallel over code which was supposed to be serialized.  It's  
 not a problem if the lock object is the object being mutated/used by the  
 code, but it is a problem if the lock object is protecting other shared  
 objects.  It's not something I have ever bumped into myself, because the  
 bulk of my locking experience is in C/C++ and I use a locking primitive  
 which is initialised once and never mutates.

Well, I thought the original proposal was to use the opCmp of the *data*,  
i.e. the object being protected.  At least that's what it seems like from  
the example.

Apologies if this was not the case.

 You *never* want the ordering to change.

 But I think we can probably work that out. What about comparing handles
 of the mutexes? So you sort based on some property __mutex_id() which
 must return a unique size_t that can be used to always define an
 ordering of locking mutexes? Most mutexes are allocated by the OS, and
 so their handles won't be affected by a moving GC, so you likely will
 use the handle value.

 This could work. In fact I imagined comparing handles ...
 As with math at large you may either project (biject) your domain to a  
 well-known one (like numbers, mutex_id in your case) and work with it  
 or define all relevant properties for whatever your values in this  
 domain are (providing custom ordering for user defined types).

 This ordering idea is present in TDPL.. have you both read the link  
 Andrei posted?
 http://goo.gl/ZhPM2
 (see 13.15)

No, I haven't read it yet.  I did review an early version of TDPL, but I  
have not re-read the released version (though I do have a copy).  Whether  
it was in there or not, it's been a while :)

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 7:49 AM, Dmitry Olshansky wrote:
 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

 Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex
 is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm
 suggesting "mutex" but kept private inside the class /that it locks/.
 Yes, it's a visibility issue, the issue is that the mutex used by
 synchronized classes/methods is too visible/accessible and this opens it
 up for deadlocks which are otherwise impossible.

 Sure it's awful comparison.

It's a great comparison.

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 7:49 AM, Dmitry Olshansky wrote:
 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

Great.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)

Great. What are regular calls to synchronized class methods lowered into?

 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

 Does it makes sense?

It does make sense, but I think we need a Lock struct type that makes 
sure code cannot screw up the number of lock and unlock calls. We 
shouldn't just expose bare lock() and unlock().


Andrei

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 31.05.2012 22:33, Andrei Alexandrescu wrote:
 On 5/31/12 7:49 AM, Dmitry Olshansky wrote:
 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

 Great.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)

 Great. What are regular calls to synchronized class methods lowered into?

Posted in another reply, basically same lock(); scope(exit)unlock() of 
hidden monitor.
 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

 Does it makes sense?

 It does make sense, but I think we need a Lock struct type that makes
 sure code cannot screw up the number of lock and unlock calls. We
 shouldn't just expose bare lock() and unlock().

Dunno. In fact __lock/__unlock could be artificially hidden from user 
but not compiler with his magic rewrites.
But I believe there is value in having access to lock/unlock for those 
select cases where it's beneficial.


-- 
Dmitry Olshansky

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Thu, 31 May 2012 14:55:31 -0400, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:

 Dunno. In fact __lock/__unlock could be artificially hidden from user  
 but not compiler with his magic rewrites.
 But I believe there is value in having access to lock/unlock for those  
 select cases where it's beneficial.

__ctor and __dtor are accessible, but nobody abuses them.

I think if we use the double-underscore terminology, this would be fine.

-Steve

Sean Kelly <sean invisibleduck.org> writes:

On May 31, 2012, at 11:33 AM, Andrei Alexandrescu wrote:
=20
 It does make sense, but I think we need a Lock struct type that makes =

sure code cannot screw up the number of lock and unlock calls. We =
shouldn't just expose bare lock() and unlock().

synchronized already works with classes that implement Object.Monitor, =
and we have scope guards.  Do we really need an RTTI Lock struct as =
well?=

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 12:27 PM, Sean Kelly wrote:
 On May 31, 2012, at 11:33 AM, Andrei Alexandrescu wrote:
 It does make sense, but I think we need a Lock struct type that
 makes sure code cannot screw up the number of lock and unlock
 calls. We shouldn't just expose bare lock() and unlock().

 synchronized already works with classes that implement
 Object.Monitor, and we have scope guards.  Do we really need an RTTI
 Lock struct as well?

At this point it's unclear to me what different branches of this 
sprawling thread are proposing. I have extracted one thing that we 
should definitely look into - formalizing the lowering of the 
synchronized statement. Other than that I failed to derive much signal.

Andrei

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 31-05-2012 16:49, Dmitry Olshansky wrote:
 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

 Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex
 is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm
 suggesting "mutex" but kept private inside the class /that it locks/.
 Yes, it's a visibility issue, the issue is that the mutex used by
 synchronized classes/methods is too visible/accessible and this opens it
 up for deadlocks which are otherwise impossible.

 Sure it's awful comparison.


 So where's the mutex that would be used to synchronize objects that
 are not synchronizable?

 In the wrapper class/struct/object which derives a synchronized
 class/struct from the original. My D foo is not strong enough to just
 come up with valid D code for the idiom on the fly, but essentially you
 wrap the original object in a new object using a template which adds the
 mutex member and the interface methods (lock, tryLock, and unlock)
 required. No, this doesn't work with "final" classes.. but it shouldn't,
 they're final after all. For them you need to add/manage the mutex
 manually - the price you pay for "final".

 OK let me land you a hand here. My proposal, that I think fits your
 ideas quite favorably.

 I'll enumerate certain assumptions beforehand since it's one of most
 confusing threads I ever followed.

 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

If this means that the monitor field in regular objects goes away, then 
I'm all for this, at least.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)


 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

 Does it makes sense?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 31.05.2012 22:34, Alex Rønne Petersen wrote:
 On 31-05-2012 16:49, Dmitry Olshansky wrote:
 But this is a protection/visibility issue, which is orthogonal on the
 locking capability. It's as if you say "int is not good because anyone
 can overflow it." Okay! Make it private inside a CheckedInt class.

 Sorry, that's a bad comparison. CheckedInt is to int, what CheckedMutex
 is to mutex - but I'm not suggesting anything like a CheckedMutex. I'm
 suggesting "mutex" but kept private inside the class /that it locks/.
 Yes, it's a visibility issue, the issue is that the mutex used by
 synchronized classes/methods is too visible/accessible and this opens it
 up for deadlocks which are otherwise impossible.

 Sure it's awful comparison.


 So where's the mutex that would be used to synchronize objects that
 are not synchronizable?

 In the wrapper class/struct/object which derives a synchronized
 class/struct from the original. My D foo is not strong enough to just
 come up with valid D code for the idiom on the fly, but essentially you
 wrap the original object in a new object using a template which adds the
 mutex member and the interface methods (lock, tryLock, and unlock)
 required. No, this doesn't work with "final" classes.. but it shouldn't,
 they're final after all. For them you need to add/manage the mutex
 manually - the price you pay for "final".

 OK let me land you a hand here. My proposal, that I think fits your
 ideas quite favorably.

 I'll enumerate certain assumptions beforehand since it's one of most
 confusing threads I ever followed.

 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

 If this means that the monitor field in regular objects goes away, then
 I'm all for this, at least.

Yes, it removes monitor in non-synchronized object. Glad you we are in 
agreement here.

And all public methods lower to :

<signature>
{
this.hidden__monitor.lock();
scope(exit) this.hidden__monitor.unlock();
... //code
}


-- 
Dmitry Olshansky

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 31.05.2012 18:49, Dmitry Olshansky wrote:
 I'll enumerate certain assumptions beforehand since it's one of most
 confusing threads I ever followed.

 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)


 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

It could even go one step forward. Though it require a great deal of 
though, here is an initial idea.

So with the above we have  synchronized class with monitor that is 
eagerly initialized (it's the propose of sync-ed class after all to use 
this mutex safely) and ordinary class that don't have it.

Now if we let shared classes to have monitor field lazily initialized 
something interesting shows up. For the moment let's skip painful detail 
of cast(shared).

It's then possible to introduce a restricted version of ownership 
system. Since ever member of shared class is shared or synchronized(?) 
it should be OK to access member of any depth in it by following the 
chain of monitor locks:
(i.e. I'm assuming composition implies "owns" relationship,
this ownership relation comes only for shared types)

shared class A
{
	shared B b;
	...
}

shared class B
{
	shared C c;
	...
}


then:
{
	...
	A a = ....;
	// locks A's monitor, then B's monitor, unlocks A's
	// then C's monitor, unlocks B's
	// then calls do_smth, then unlocks C's
	//all of the above lock/unlock sequence has to be exception proof
	a.b.c.do_smth();

}


Grunted assuming compositon == ownership is wrong in general, but could 
serve as a sensible default. Then weak non-owning references would be 
provided via dirty casting in special wrappers like
{
	NotOwning!(shared T) x;//shared but doesn't own it, thus if
//calling any resource inside of x, we should keep x.monitor locked 
//through out the operation. (unlike in the above case)
}	
...

Obviously it's iteration 1 of idea, it needs to deal with possible 
atomic modifications, cast(shared), etc.


-- 
Dmitry Olshansky

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 01.06.2012 0:39, Dmitry Olshansky wrote:
 On 31.05.2012 18:49, Dmitry Olshansky wrote:
 I'll enumerate certain assumptions beforehand since it's one of most
 confusing threads I ever followed.

 1. synchronized class means: always allocate hidden monitor mutex, lock
 it on every public method of this class.

 2. synchronized(x,y,z) is lowered to (I use Steven's idea):
 auto sorted = total_order(x,y,z);//conceptual, sorted is tuple of x,y,z
 sorted
 FOR EACH item IN sorted tuple add code in [[ ... ]]
 [[// conceptual
 item.__lock();
 scope(exit) item.__unlock();
 ]]
 In other words it works for every object that defines lock/unlock.
 Multiple object version works only if there is opCmp defined. (by
 address whatever, any total ordering should work)


 Per definition above synchronized classes AS IS can't be *synchronized*
 on. Instead their public methods are implicitly synchronized.

 The end result is:

 User can synchronize on various synchronization entities and even plug
 in custom synchronization primitives (say OS Y provides have this fancy
 lock type Z). It's explicit in as sense that object supports it via
 __lock__/unlock. Obviously one still can use __lock/__unlock explicitly
 just don't forget to wear big HERE BE DRAGONS banner.

 Synchronized class encapsulate mutex completely - nobody aside from
 designer of the class may (a)use it.

 It could even go one step forward. Though it require a great deal of
 though, here is an initial idea.

 So with the above we have synchronized class with monitor that is
 eagerly initialized (it's the propose of sync-ed class after all to use
 this mutex safely) and ordinary class that don't have it.

 Now if we let shared classes to have monitor field lazily initialized
 something interesting shows up. For the moment let's skip painful detail
 of cast(shared).

 It's then possible to introduce a restricted version of ownership
 system. Since ever member of shared class is shared or synchronized(?)
 it should be OK to access member of any depth in it by following the
 chain of monitor locks:
 (i.e. I'm assuming composition implies "owns" relationship,
 this ownership relation comes only for shared types)

 shared class A
 {
 shared B b;
 ...
 }

 shared class B
 {
 shared C c;
 ...
 }


 then:
 {
 ...
 A a = ....;
 // locks A's monitor, then B's monitor, unlocks A's
 // then C's monitor, unlocks B's
 // then calls do_smth, then unlocks C's
 //all of the above lock/unlock sequence has to be exception proof
 a.b.c.do_smth();

 }


 Grunted assuming compositon == ownership is wrong in general, but could

s/Grunted/Granted

 serve as a sensible default. Then weak non-owning references would be
 provided via dirty casting in special wrappers like
 {
 NotOwning!(shared T) x;//shared but doesn't own it, thus if
 //calling any resource inside of x, we should keep x.monitor locked
 //through out the operation. (unlike in the above case)
 }
 ...

 Obviously it's iteration 1 of idea, it needs to deal with possible
 atomic modifications, cast(shared), etc.


-- 
Dmitry Olshansky

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 7:01 AM, Regan Heath wrote:
 I'm
 suggesting "mutex" but kept private inside the class /that it locks/.
 Yes, it's a visibility issue, the issue is that the mutex used by
 synchronized classes/methods is too visible/accessible and this opens it
 up for deadlocks which are otherwise impossible.

You're suggesting an idiom based on an unrestricted abstraction 
("mutex") combined with a language feature ("private"). That's no 
progress because:

1. People can still misuse the unrestricted "mutex" as they have for the 
past 50 years.

2. The idiom can be done TODAY with a regular class that has a member of 
type synchronized class. Alternatively, you could use the mutex type in 
core directly.

So your suggestion actually makes the language worse and adds no power. 
Your impression being that "synchronized" is too deadlock-prone, and you 
believe the solution is taking one step BACK and eliminate it in favor 
of the "private"-based idiom in conjunction with the "mutex" type which 
is deadlock-prone PLUS prone to a variety of other issues.

 So where's the mutex that would be used to synchronize objects that
 are not synchronizable?

 In the wrapper class/struct/object which derives a synchronized
 class/struct from the original. My D foo is not strong enough to just
 come up with valid D code for the idiom on the fly, but essentially you
 wrap the original object in a new object using a template which adds the
 mutex member and the interface methods (lock, tryLock, and unlock)
 required. No, this doesn't work with "final" classes.. but it shouldn't,
 they're final after all. For them you need to add/manage the mutex
 manually - the price you pay for "final".

Is there anything here that can't be done today, and pronto?

 There are cases in which you want to do multiple operations under a
 single critical section, even though the API is otherwise
 well-designed. That may be for correctness, efficiency, or both. I
 don't see why we'd want to disallow that, it's a good idiom.

 Who suggested disallowing this? No-one. There are 3 main use cases I see
 for this;

 1. Several disparate objects locked by a single mutex - in which case
 the correct solution is a separate mutex/monitor object.

There's also "13.14.3 Forcing Identical Mutexes" in TDPL.

 2. A single object, locked for serveral method calls - in which case the
 method-passed-a-delegate idea (mentioned below/ described in a separate
 thread) works, unless..

Here the synchronized-based idiom works well if the cost of reacquiring 
an already-owned mutex is sufficiently low.

 3. The calls span several scopes, in which case good-old manual
 lock/unlock is required (synchronized blocks don't help here)

Right. In these rare cases core.sync.mutex would be recommendable.

 Looking forward for a fleshed out proposal. Make sure you motivate it
 properly.

 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

Thanks. Let me know if I understand correctly that your idea boils down 
to "I don't like synchronized, let's deprecate it and get back to 
core.sync.mutex and recommend the private thingamaroo." In that case, I 
disagree. I believe synchronized has good merits that are being ignored.

 On first look, the inversion of control using delegates delegates has
 similar liabilities as straight scoped locking.

 True, it's basically the same as a synchronized block in that respect.
 What we actually want is a way to limit the calls made by the delegate
 to methods of the object itself. If it could not call a synchronized
 method on a 2nd object, you could not possibly deadlock. Except, that is
 to say, unless you held a separate lock beforehand - but, the important
 point here is that you would have to take both locks explicitly, rather
 than by an implicit synchronized method call, making the bug far more
 obvious to code inspection.

Right. This is very D-unlike. I don't see anything in the current 
language that can limit the things a lambda/delegate can do.


Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Thu, 31 May 2012 14:29:27 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 7:01 AM, Regan Heath wrote:

 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

 Thanks. Let me know if I understand correctly that your idea boils down  
 to "I don't like synchronized, let's deprecate it and get back to  
 core.sync.mutex and recommend the private thingamaroo." In that case, I  
 disagree. I believe synchronized has good merits that are being ignored.

No, this is definitely *not* what we are saying.  The idea is that  
synchronized(x) is still present, but what objects you can call this on,  
and more importantly, *who* can do this is restricted.

Nobody is advocating abandoning synchronized in favor of manual locks.  In  
fact, I think we all want to *avoid* manual locks as much as possible.   
It's all about controlling access.  If it comes down to "you must use a  
private, error-prone mutex member in order to prevent deadlocks," then I  
think we have room for improvement.

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 11:35 AM, Steven Schveighoffer wrote:
 On Thu, 31 May 2012 14:29:27 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 7:01 AM, Regan Heath wrote:

 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

 Thanks. Let me know if I understand correctly that your idea boils
 down to "I don't like synchronized, let's deprecate it and get back to
 core.sync.mutex and recommend the private thingamaroo." In that case,
 I disagree. I believe synchronized has good merits that are being
 ignored.

 No, this is definitely *not* what we are saying.

Then probably a more formal proposal is in order.

Andrei

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 19:35:50 +0100, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Thu, 31 May 2012 14:29:27 -0400, Andrei Alexandrescu  
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/12 7:01 AM, Regan Heath wrote:

 Sorry, I have no spare time to spare. You're getting free  
 ideas/thoughts
 from me, feel free to ignore them.

 Thanks. Let me know if I understand correctly that your idea boils down  
 to "I don't like synchronized, let's deprecate it and get back to  
 core.sync.mutex and recommend the private thingamaroo." In that case, I  
 disagree. I believe synchronized has good merits that are being ignored.

 No, this is definitely *not* what we are saying.  The idea is that  
 synchronized(x) is still present, but what objects you can call this on,  
 and more importantly, *who* can do this is restricted.

Exactly.

 Nobody is advocating abandoning synchronized in favor of manual locks.   
 In fact, I think we all want to *avoid* manual locks as much as  
 possible.  It's all about controlling access.  If it comes down to "you  
 must use a private, error-prone mutex member in order to prevent  
 deadlocks," then I think we have room for improvement.

Indeed.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

"Regan Heath" <regan netmail.co.nz> writes:

On Thu, 31 May 2012 19:29:27 +0100, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 On 5/31/12 7:01 AM, Regan Heath wrote:
 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

 Thanks. Let me know if I understand correctly that your idea boils down  
 to "I don't like synchronized, let's deprecate it and get back to  
 core.sync.mutex and recommend the private thingamaroo." In that case, I  
 disagree. I believe synchronized has good merits that are being ignored.

To present this another way..

Your motivation for the construct: "synchronized(a, b, ...)" was to  
prevent deadlocks caused by:

[thread1]
synchronized(a)
{
   synchronized(b)
   {
   }
}

[thread2]
synchronized(b)
{
   synchronized(a)
   {
   }
}

right?

Well, this is the same problem expressed several other less-obvious (to  
code inspection) ways:

1.
[thread1]
synchronized(a)
{
   b.foo();  // where b.foo is { synchronized(this) { ... } }
}

[thread2]
synchronized(b)
{
   a.foo();  // where a.foo is { synchronized(this) { ... } }
}

2.
[thread1]
synchronized(a)
{
   b.foo();  // where b.foo is synchronized void foo() { ... }
}

[thread2]
synchronized(b)
{
   a.foo();  // where a.foo is synchronized void foo() { ... }
}


disallowing that idiom completely in favour of synchronized classes/class  
methods (which I think TDPL does?), and second by adding more control as  



can participate in synchronized statements.  If either 'a' or 'b' were not  
allowed to participate in a synchronized statement, then either thread1 or  
thread2 would be invalid code and a deadlock involving these 2 objects  
would be impossible(*).

There will still exist some synchronized classes which want to participate  
in synchronized statements but I'm thinking/hoping this is rare and if the  
default for D is 'not allowed' then it becomes a conscious choice and we  
can supply the developer with a warning in the docs which describe how to  
do it, introduce the synchronized(a, b, ...) construct, etc.

 From another angle.. I'm guessing it's either impossible or very hard to  
detect the 2 cases presented above at compile time?  Essentially the  
compiler would need to know which code could execute in separate threads,  
then determine lock ordering for all shared/lockable objects, then detect  
cases of both (lock a, b) and (lock b, a) in separate threads.  Sounds  
tricky.

R

(*)using synchronized statements - one could still keep a reference to the  
other internally and call a synchronized member function from within a  
synchronized member function

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

deadalnix <deadalnix gmail.com> writes:

Le 01/06/2012 14:55, Regan Heath a écrit :
 On Thu, 31 May 2012 19:29:27 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 On 5/31/12 7:01 AM, Regan Heath wrote:
 Sorry, I have no spare time to spare. You're getting free ideas/thoughts
 from me, feel free to ignore them.

 Thanks. Let me know if I understand correctly that your idea boils
 down to "I don't like synchronized, let's deprecate it and get back to
 core.sync.mutex and recommend the private thingamaroo." In that case,
 I disagree. I believe synchronized has good merits that are being
 ignored.

 To present this another way..

 Your motivation for the construct: "synchronized(a, b, ...)" was to
 prevent deadlocks caused by:

 [thread1]
 synchronized(a)
 {
 synchronized(b)
 {
 }
 }

 [thread2]
 synchronized(b)
 {
 synchronized(a)
 {
 }
 }

 right?

 Well, this is the same problem expressed several other less-obvious (to
 code inspection) ways:

 1.
 [thread1]
 synchronized(a)
 {
 b.foo(); // where b.foo is { synchronized(this) { ... } }
 }

 [thread2]
 synchronized(b)
 {
 a.foo(); // where a.foo is { synchronized(this) { ... } }
 }

 2.
 [thread1]
 synchronized(a)
 {
 b.foo(); // where b.foo is synchronized void foo() { ... }
 }

 [thread2]
 synchronized(b)
 {
 a.foo(); // where a.foo is synchronized void foo() { ... }
 }


 disallowing that idiom completely in favour of synchronized
 classes/class methods (which I think TDPL does?), and second by adding



 who can participate in synchronized statements. If either 'a' or 'b'
 were not allowed to participate in a synchronized statement, then either
 thread1 or thread2 would be invalid code and a deadlock involving these
 2 objects would be impossible(*).

 There will still exist some synchronized classes which want to
 participate in synchronized statements but I'm thinking/hoping this is
 rare and if the default for D is 'not allowed' then it becomes a
 conscious choice and we can supply the developer with a warning in the
 docs which describe how to do it, introduce the synchronized(a, b, ...)
 construct, etc.

  From another angle.. I'm guessing it's either impossible or very hard
 to detect the 2 cases presented above at compile time? Essentially the
 compiler would need to know which code could execute in separate
 threads, then determine lock ordering for all shared/lockable objects,
 then detect cases of both (lock a, b) and (lock b, a) in separate
 threads. Sounds tricky.

 R

 (*)using synchronized statements - one could still keep a reference to
 the other internally and call a synchronized member function from within
 a synchronized member function

I think it is unrealistic to prevent all deadlock, unless you can come 
up with a radically new approach.

It is still possible to provide interface that prevent common traps.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Fri, 01 Jun 2012 10:09:01 -0400, deadalnix <deadalnix gmail.com> wrote:


 I think it is unrealistic to prevent all deadlock, unless you can come  
 up with a radically new approach.

 It is still possible to provide interface that prevent common traps.

Right, the idea is not to exterminate all deadlock, it's to *make it  
possible to* prevent deadlock.  Currently, it's not possible unless you  
avoid using synchronized(this).

-Steve

"Regan Heath" <regan netmail.co.nz> writes:

On Fri, 01 Jun 2012 15:34:20 +0100, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Fri, 01 Jun 2012 10:09:01 -0400, deadalnix <deadalnix gmail.com>  
 wrote:


 I think it is unrealistic to prevent all deadlock, unless you can come  
 up with a radically new approach.

 It is still possible to provide interface that prevent common traps.

 Right, the idea is not to exterminate all deadlock, it's to *make it  
 possible to* prevent deadlock.  Currently, it's not possible unless you  
 avoid using synchronized(this).

What he ^ said :)

R


-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

To clarify:

On 5/30/12 12:25 PM, Alex Rønne Petersen wrote:
  The mutex may not be
 directly exposed in the sense that you can obtain a reference (though in
 reality in all compiler implementations, you can), but it is exposed in
 the sense that you can lock and unlock it, which is a mutation of state
 and program flow.

synchronized (object) {
     writeln("about to unlock the object");
     XXX
     writeln("unlocked the object");
}

Replace "XXX" with a construct that unlocks the object.


Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 To clarify:

 On 5/30/12 12:25 PM, Alex R=C3=B8nne Petersen wrote:
  The mutex may not be
 directly exposed in the sense that you can obtain a reference (though=


 in
 reality in all compiler implementations, you can), but it is exposed =


in
 the sense that you can lock and unlock it, which is a mutation of sta=


te
 and program flow.

 synchronized (object) {
      writeln("about to unlock the object");
      XXX
      writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

This is not what we are talking about.

I guess the easiest way to say it is, the API used to "lock and then  =

subsequently unlock" the mutex.  That is, even this:

Object o =3D new Object;

synchronized(o)
{
}

is exposing the mutex in such a way that you can interfere with the  =

semantic meaning of the mutex, and cause preventable deadlocks.

It would be preferrable for:

synchronized(o)
{
}

to be an error, unless typeof(o) allows it explicitly.

I gave you a case where you can have a deadlock, even with simple fully =
 =

synchronized classes.  You might say, "yeah, but all mutexes can have  =

deadlocks!".

My contention is that if the default is you do *not* expose the mutex to=
  =

external callers, there *cannot* be a deadlock.

Here is the example again:

synchronized class A
{
   private int _foo;
    property int foo() { return _foo;}
}

synchronized class B
{
    private A _a;
     property A a() { return _a;}
    void fun() {}
}

void thread(B b)
{
    synchronized(b.a)
    {
       b.fun();
    }
}

If synchronized(b.a) is valid, deadlock can occur.  If it's invalid,  =

deadlock *cannot* occur.

-Steve

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 22:31, Steven Schveighoffer a écrit :
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 To clarify:

 On 5/30/12 12:25 PM, Alex Rønne Petersen wrote:
 The mutex may not be
 directly exposed in the sense that you can obtain a reference (though in
 reality in all compiler implementations, you can), but it is exposed in
 the sense that you can lock and unlock it, which is a mutation of state
 and program flow.

 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

 This is not what we are talking about.

 I guess the easiest way to say it is, the API used to "lock and then
 subsequently unlock" the mutex. That is, even this:

 Object o = new Object;

 synchronized(o)
 {
 }

 is exposing the mutex in such a way that you can interfere with the
 semantic meaning of the mutex, and cause preventable deadlocks.

 It would be preferrable for:

 synchronized(o)
 {
 }

 to be an error, unless typeof(o) allows it explicitly.

 I gave you a case where you can have a deadlock, even with simple fully
 synchronized classes. You might say, "yeah, but all mutexes can have
 deadlocks!".

 My contention is that if the default is you do *not* expose the mutex to
 external callers, there *cannot* be a deadlock.

 Here is the example again:

 synchronized class A
 {
 private int _foo;
  property int foo() { return _foo;}
 }

 synchronized class B
 {
 private A _a;
  property A a() { return _a;}
 void fun() {}
 }

 void thread(B b)
 {
 synchronized(b.a)
 {
 b.fun();
 }
 }

 If synchronized(b.a) is valid, deadlock can occur. If it's invalid,
 deadlock *cannot* occur.

 -Steve

This cannot remove all deadlocks, but yes, it goes in the right direction.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 30 May 2012 16:56:51 -0400, deadalnix <deadalnix gmail.com> wrot=
e:

 Le 30/05/2012 22:31, Steven Schveighoffer a =C3=A9crit :
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 To clarify:

 On 5/30/12 12:25 PM, Alex R=C3=B8nne Petersen wrote:
 The mutex may not be
 directly exposed in the sense that you can obtain a reference (thou=




gh  =

 in
 reality in all compiler implementations, you can), but it is expose=




d  =

 in
 the sense that you can lock and unlock it, which is a mutation of  =




 state
 and program flow.

 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

 This is not what we are talking about.

 I guess the easiest way to say it is, the API used to "lock and then
 subsequently unlock" the mutex. That is, even this:

 Object o =3D new Object;

 synchronized(o)
 {
 }

 is exposing the mutex in such a way that you can interfere with the
 semantic meaning of the mutex, and cause preventable deadlocks.

 It would be preferrable for:

 synchronized(o)
 {
 }

 to be an error, unless typeof(o) allows it explicitly.

 I gave you a case where you can have a deadlock, even with simple ful=


ly
 synchronized classes. You might say, "yeah, but all mutexes can have
 deadlocks!".

 My contention is that if the default is you do *not* expose the mutex=


 to
 external callers, there *cannot* be a deadlock.

 Here is the example again:

 synchronized class A
 {
 private int _foo;
  property int foo() { return _foo;}
 }

 synchronized class B
 {
 private A _a;
  property A a() { return _a;}
 void fun() {}
 }

 void thread(B b)
 {
 synchronized(b.a)
 {
 b.fun();
 }
 }

 If synchronized(b.a) is valid, deadlock can occur. If it's invalid,
 deadlock *cannot* occur.

 -Steve

 This cannot remove all deadlocks, but yes, it goes in the right  =

 direction.

All deadlocks involving A and B mutexes.  Or if not, can you show how?  =
I  =

can't see it.

-Steve

=?ISO-8859-1?Q?Jos=E9_Armando_Garc=EDa_Sancio?= <jsancio gmail.com> writes:

On Wed, May 30, 2012 at 5:30 PM, Steven Schveighoffer
<schveiguy yahoo.com> wrote:
 All deadlocks involving A and B mutexes. =A0Or if not, can you show how? =

=A0I
 can't see it.

 -Steve

This can cause a deadlock because locking order is not guaranteed, no?

synchronized class A {
  void delegate(B b) { b.call() }
  void call() {}
}

synchronized class B {
  void delegate(A a) { a.call() }
  void call() {}
}

Thanks,
-Jose
PS. This may not compile not sure if it should be 'void
delegate(shared B b)', etc.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 30 May 2012 17:42:47 -0400, Jos=C3=A9 Armando Garc=C3=ADa Sancio=
  =

<jsancio gmail.com> wrote:

 On Wed, May 30, 2012 at 5:30 PM, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:
 All deadlocks involving A and B mutexes.  Or if not, can you show how=


?  =

  I
 can't see it.

 -Steve

 This can cause a deadlock because locking order is not guaranteed, no?=

 synchronized class A {
   void delegate(B b) { b.call() }
   void call() {}
 }

 synchronized class B {
   void delegate(A a) { a.call() }
   void call() {}
 }

 Thanks,
 -Jose
 PS. This may not compile not sure if it should be 'void
 delegate(shared B b)', etc.

A and B are not modifiable.  Given the implementation of A and B that I =
 =

gave, how can you achieve deadlock?

The big problem I see with allowing anyone to synchronize on A and B is =
 =

that one cannot achieve the property of "not possible to use me in a  =

deadlock."

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 1:31 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

 This is not what we are talking about.

Oh yes it is. "Expose the mutex" means "make the two mutex primitive 
operations lock() and unlock() freely usable against the mutex object".

 I guess the easiest way to say it is, the API used to "lock and then
 subsequently unlock" the mutex. That is, even this:

 Object o = new Object;

 synchronized(o)
 {
 }

 is exposing the mutex in such a way that you can interfere with the
 semantic meaning of the mutex, and cause preventable deadlocks.

 It would be preferrable for:

 synchronized(o)
 {
 }

 to be an error, unless typeof(o) allows it explicitly.

Yah, it should be only allowed for synchronized classes.

 I gave you a case where you can have a deadlock, even with simple fully
 synchronized classes. You might say, "yeah, but all mutexes can have
 deadlocks!".

Not only I might say that, I'm actually gonna say it. Yeah, but all 
mutexes can have deadlocks!

 My contention is that if the default is you do *not* expose the mutex to
 external callers, there *cannot* be a deadlock.

That's shuffling the fundamental issue from client code to library code. 
This can be done in either approach (synchronized vs. raw mutex), except 
it's clunkier with raw mutex.

 Here is the example again:

 synchronized class A
 {
 private int _foo;
  property int foo() { return _foo;}
 }

 synchronized class B
 {
 private A _a;
  property A a() { return _a;}
 void fun() {}
 }

 void thread(B b)
 {
 synchronized(b.a)
 {
 b.fun();
 }
 }

 If synchronized(b.a) is valid, deadlock can occur. If it's invalid,
 deadlock *cannot* occur.

I don't get what this is supposed to prove.


Andrei

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 23:45, Andrei Alexandrescu a écrit :
 On 5/30/12 1:31 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

 This is not what we are talking about.

 Oh yes it is. "Expose the mutex" means "make the two mutex primitive
 operations lock() and unlock() freely usable against the mutex object".

The more I think of it, the more I think it should go throw a 
 synchrnizable property or something.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 30 May 2012 17:45:32 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 1:31 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

 This is not what we are talking about.

 Oh yes it is. "Expose the mutex" means "make the two mutex primitive  
 operations lock() and unlock() freely usable against the mutex object".

Stop that!  You are blatantly arguing minutia when you should be trying to  
understand what we are saying!  This is like dealing with a stubborn child.

 I guess the easiest way to say it is, the API used to "lock and then
 subsequently unlock" the mutex. That is, even this:

 Object o = new Object;

 synchronized(o)
 {
 }

 is exposing the mutex in such a way that you can interfere with the
 semantic meaning of the mutex, and cause preventable deadlocks.

 It would be preferrable for:

 synchronized(o)
 {
 }

 to be an error, unless typeof(o) allows it explicitly.

 Yah, it should be only allowed for synchronized classes.

Not good enough.  Whether I want all accesses to methods and members to be  
synchronized should be orthogonal to whether I want to allow arbitrary  
scoped locking and unlocking of my object.

 I gave you a case where you can have a deadlock, even with simple fully
 synchronized classes. You might say, "yeah, but all mutexes can have
 deadlocks!".

 Not only I might say that, I'm actually gonna say it. Yeah, but all  
 mutexes can have deadlocks!

Right, if used incorrectly.  In my example, you cannot have a deadlock  
unless the ability to scope-lock the object is exposed to the public.

 My contention is that if the default is you do *not* expose the mutex to
 external callers, there *cannot* be a deadlock.

 That's shuffling the fundamental issue from client code to library code.  
 This can be done in either approach (synchronized vs. raw mutex), except  
 it's clunkier with raw mutex.

Not talking about raw mutexes, my code is based on synchronized classes.   
Period.

 Here is the example again:

 synchronized class A
 {
 private int _foo;
  property int foo() { return _foo;}
 }

 synchronized class B
 {
 private A _a;
  property A a() { return _a;}
 void fun() {}
 }

 void thread(B b)
 {
 synchronized(b.a)
 {
 b.fun();
 }
 }

 If synchronized(b.a) is valid, deadlock can occur. If it's invalid,
 deadlock *cannot* occur.

 I don't get what this is supposed to prove.

If you cannot synchronize on b.a, as an *external* entity, then you cannot  
deadlock.  This is an important property.  The author of A and B can say  
"deadlock-proof".  How do you not see this as valuable?

Right now, we have a choice:

a. use synchronized -> prone to deadlocks that you cannot control or  
prevent.
b. roll your own locking -> possible to prevent deadlocks in all cases,  
but prone to error since you could accidentally miss an unlock or lock.

I want:

c. use synchronized with *no* public access -> possible to prevent  
deadlocks in all cases, safe to use without forgetting to unlock/lock.

-Steve

"foobar" <foo bar.com> writes:

On Thursday, 31 May 2012 at 11:24:56 UTC, Steven Schveighoffer 
wrote:
 On Wed, 30 May 2012 17:45:32 -0400, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 1:31 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 15:48:47 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 synchronized (object) {
 writeln("about to unlock the object");
 XXX
 writeln("unlocked the object");
 }

 Replace "XXX" with a construct that unlocks the object.

 This is not what we are talking about.

 Oh yes it is. "Expose the mutex" means "make the two mutex 
 primitive operations lock() and unlock() freely usable against 
 the mutex object".

 Stop that!  You are blatantly arguing minutia when you should 
 be trying to understand what we are saying!  This is like 
 dealing with a stubborn child.

[snip]
 -Steve

This reminds me one of my favorite Zen stories:

Nan-in, a Japanese master during the Meiji era (1868-1912), 
received a university professor who came to inquire about Zen.

Nan-in served tea. He poured his visitor's cup full, and then 
kept on pouring.

The professor watched the overflow until he no longer could 
restrain himself. "It is overfull. No more will go in!"

"Like this cup," Nan-in said, "you are full of your own opinions 
and speculations. How can I show you Zen unless you first empty 
your cup?"

"Jonathan M Davis" <jmdavisProg gmx.com> writes:

On Wednesday, May 30, 2012 12:12:15 Andrei Alexandrescu wrote:
 On 5/30/12 12:03 PM, Steven Schveighoffer wrote:
 On Wed, 30 May 2012 14:32:59 -0400, Andrei Alexandrescu
 
 <SeeWebsiteForEmail erdani.org> wrote:
 On 5/30/12 10:47 AM, Steven Schveighoffer wrote:
 Yes, you can just use a private mutex. But doesn't that just lead to
 recommending not using a feature of the language?

 
 I don't think so. Synchronized classes are the unit of scoped locking
 in D. If you want to do all scoped locking internally, make the class
 private.

 
 Maybe you didn't read thoroughly the first part of my post (the example
 that shows a deadlock that is easily preventable if the mutex isn't
 exposed).

 
 The mutex is not exposed.

No, you can't directly access the mutex externally, but a synchronized block 
used on a synchronized object uses the same mutex that the object does.

So, if you have

synchronized class C
{
}

then this synchronized block

C c;
synchronized(c)
{
}

will use the exact same mutex that any functions on C itself use. So, it's 
possible for code outside of your class to lock that mutex using a 
synchronized statement. They do _not_ have direct access to the mutex to do 
whatever they want with it, but they _can_ lock via synchronized statements, 
which means that the synchronized class does _not_ have total control over 
what code locks its mutex.

All that's required to fix this particular problem is to make it illegal to use 
a synchronized block on a synchronized object - either that or give 
synchronized object's two mutexes (one to use internally for its functions and 
one to be locked externally using synchronized blocks). The doesn't 
necessarily solve all of the issues being discussed here, but it _does_ solve 
the issue of a synchronized class not being the only entity with the ability 
to lock its mutex.

- Jonathan M Davis

=?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 30-05-2012 18:03, Regan Heath wrote:
 On Wed, 30 May 2012 16:46:54 +0100, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable = isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is that
 we lock explicit stuff.

 But in this design anyone can lock such an object, which was something
 you advocated against.

 I think there is some confusion here as to what the "problem" is and is
 not.

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

 The problem /is/ that synchronized classes/methods use a mutex which is
 exposed publicly, and it the same mutex as used by synchronized(object)
 {}. This exposure/re-use makes deadlocks more likely to happen, and
 harder to spot.

Thanks. I think the discussion was getting a bit derailed, myself being 
guilty of going off on tangential issues. Yes, the issue is exactly what 
you point out here: Exposing a locking resource publicly. The issue with 
a monitor on every object can probably be solved completely separately.

 Removal of this "problem" will not stop deadlocks from happening as
 they're a problem multi-threaded/lock based code will always have (*).
 It will however make them far less likely to happen accidentally. It
 will make programmers think about what/where/when and how to lock things
 rather than blithely using synchronized everywhere. It will mean using
 locks is not as easy as currently, though I think we can make it fairly
 nice with good library code and/or some co-operation between the runtime
 and synchronized() statements i.e. requiring the class implement a
 common Lockable interface for example.

One gripe I have with using interfaces is speed. David Simcha showed 
that virtual mutex calls actually *do* impact speed significantly, 
particularly in the GC in his case.

As I pointed out earlier in this thread, core.sync.mutex.Mutex allows 
both composition and inheritance, and works with the synchronized 
statement. I still think it satisfies what everyone wants, given those 
two facts, and I have yet to see any counter-arguments to that (but 
would love to hear some to get a better understanding of why some people 
are against mutexes).

 The fact that every object can be locked, and this means they all use
 more memory is a side issue - arguably as important for some.

 R

 (*) the best you can do to "solve" the deadlock problem is to impose
 lock acquisition ordering on your code, as in all locks must be acquired
 in a defined order, and if not an assertion/error/exception is thrown.
 This requires some sort of static/singelton/overlord class which is
 aware of all mutexes and is involved in all acquisitions/releases.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 17:46, Andrei Alexandrescu a écrit :
 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable = isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is that
 we lock explicit stuff.

 But in this design anyone can lock such an object, which was something
 you advocated against.

 Andrei

I was advocating on being able to lock ANY object, which is out of control.

Such an object is known to be lockable, and most object will not be. It 
will avoid liquid lock, because most thing will not be lockable.

Combined with encapsulation capabilities that D already have, exposing 
the lock to the entire worlds is easy. It will avoid unexpected lock 
from 3rd party code, which is a source of tedious deadlock.

It also open door for locking on struct, that is easily embeded in a 
class as value (so constructed and destructed with the class).

The fact that anyone or not can lock/unlock a mutex is a encapsulation 
problem and we already have solution in D for that.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 9:56 AM, deadalnix wrote:
 Le 30/05/2012 17:46, Andrei Alexandrescu a écrit :
 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable = isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is that
 we lock explicit stuff.

 But in this design anyone can lock such an object, which was something
 you advocated against.

 Andrei

 I was advocating on being able to lock ANY object, which is out of control.

Must've been another poster. Anyhow, in TDPL's design only synchronized 
classes can be used with the synchronized statement.

 Such an object is known to be lockable, and most object will not be. It
 will avoid liquid lock, because most thing will not be lockable.

I'm celebrating day 2 of having no idea what a liquid lock is.

 Combined with encapsulation capabilities that D already have, exposing
 the lock to the entire worlds is easy. It will avoid unexpected lock
 from 3rd party code, which is a source of tedious deadlock.

 It also open door for locking on struct, that is easily embeded in a
 class as value (so constructed and destructed with the class).

 The fact that anyone or not can lock/unlock a mutex is a encapsulation
 problem and we already have solution in D for that.

... which can be used with synchronized classes.


Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 19:19, Andrei Alexandrescu wrote:
 On 5/30/12 9:56 AM, deadalnix wrote:
 Le 30/05/2012 17:46, Andrei Alexandrescu a écrit :
 On 5/30/12 2:34 AM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I think something similar to range design here is the way to go.

 It is easy to define something like

 template isLockable(T) {
 enum isLockable = isShared!T && is(typeof(T.init.lock())) &&
 is(typeof(T.init.release()));
 }

 And allow locking only if(isLockable!Type) .

 Now we can create SyncObject or any structure we want. The point is
 that
 we lock explicit stuff.

 But in this design anyone can lock such an object, which was something
 you advocated against.

 Andrei

 I was advocating on being able to lock ANY object, which is out of
 control.

 Must've been another poster. Anyhow, in TDPL's design only synchronized
 classes can be used with the synchronized statement.

 Such an object is known to be lockable, and most object will not be. It
 will avoid liquid lock, because most thing will not be lockable.

 I'm celebrating day 2 of having no idea what a liquid lock is.

I have to confess to this as well. I'm starting to suspect that he 
really means deadlock. Searching on Google for "liquid lock mutex" 
literally brings up this very NG thread. :)

 Combined with encapsulation capabilities that D already have, exposing
 the lock to the entire worlds is easy. It will avoid unexpected lock
 from 3rd party code, which is a source of tedious deadlock.

 It also open door for locking on struct, that is easily embeded in a
 class as value (so constructed and destructed with the class).

 The fact that anyone or not can lock/unlock a mutex is a encapsulation
 problem and we already have solution in D for that.

 .... which can be used with synchronized classes.


 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

[snip]
 Such an object is known to be lockable, and most object will not be. It
 will avoid liquid lock, because most thing will not be lockable.

 I'm celebrating day 2 of having no idea what a liquid lock is.

 I have to confess to this as well. I'm starting to suspect that he
 really means deadlock. Searching on Google for "liquid lock mutex"
 literally brings up this very NG thread. :)

Maybe it's livelock?

P.S. Just trying to snatched the prize in this little guess-game :)


-- 
Dmitry Olshansky

"cal" <callumenator gmail.com> writes:

On Wednesday, 30 May 2012 at 17:46:11 UTC, Dmitry Olshansky wrote:
 [snip]
 Such an object is known to be lockable, and most object will 
 not be. It will avoid liquid lock, because most thing will 
 not be lockable.

 I'm celebrating day 2 of having no idea what a liquid lock is.

 I have to confess to this as well. I'm starting to suspect 
 that he really means deadlock. Searching on Google for "liquid 
 lock mutex" literally brings up this very NG thread. :)

 Maybe it's livelock?

 P.S. Just trying to snatched the prize in this little 
 guess-game :)

FWIW, I recently came across the term here:
http://schneide.wordpress.com/tag/liquid-lock/

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 20:03, cal a écrit :
 On Wednesday, 30 May 2012 at 17:46:11 UTC, Dmitry Olshansky wrote:
 [snip]
 Such an object is known to be lockable, and most object will not
 be. It will avoid liquid lock, because most thing will not be
 lockable.

 I'm celebrating day 2 of having no idea what a liquid lock is.

 I have to confess to this as well. I'm starting to suspect that he
 really means deadlock. Searching on Google for "liquid lock mutex"
 literally brings up this very NG thread. :)

 Maybe it's livelock?

 P.S. Just trying to snatched the prize in this little guess-game :)

 FWIW, I recently came across the term here:
 http://schneide.wordpress.com/tag/liquid-lock/

I explained that in another post a few minutes ago, and yes, this is it.

Maybe i should write an article on that, I though it was more well known.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 12:22 PM, deadalnix wrote:
 Le 30/05/2012 20:03, cal a écrit :
 On Wednesday, 30 May 2012 at 17:46:11 UTC, Dmitry Olshansky wrote:
 [snip]
 Such an object is known to be lockable, and most object will not
 be. It will avoid liquid lock, because most thing will not be
 lockable.

 I'm celebrating day 2 of having no idea what a liquid lock is.

 I have to confess to this as well. I'm starting to suspect that he
 really means deadlock. Searching on Google for "liquid lock mutex"
 literally brings up this very NG thread. :)

 Maybe it's livelock?

 P.S. Just trying to snatched the prize in this little guess-game :)

 FWIW, I recently came across the term here:
 http://schneide.wordpress.com/tag/liquid-lock/

 I explained that in another post a few minutes ago, and yes, this is it.

 Maybe i should write an article on that, I though it was more well known.

As a side note, I'm highly weary of inventing new terminology. This is a 
general remark, not related to this particular instance. Most often when 
I personally found the need to invent a new term, it was because of not 
having done my homework of looking at related work.

Andrei

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 21:45, Andrei Alexandrescu a écrit :
 On 5/30/12 12:22 PM, deadalnix wrote:
 Le 30/05/2012 20:03, cal a écrit :
 On Wednesday, 30 May 2012 at 17:46:11 UTC, Dmitry Olshansky wrote:
 [snip]
 Such an object is known to be lockable, and most object will not
 be. It will avoid liquid lock, because most thing will not be
 lockable.

 I'm celebrating day 2 of having no idea what a liquid lock is.

 I have to confess to this as well. I'm starting to suspect that he
 really means deadlock. Searching on Google for "liquid lock mutex"
 literally brings up this very NG thread. :)

 Maybe it's livelock?

 P.S. Just trying to snatched the prize in this little guess-game :)

 FWIW, I recently came across the term here:
 http://schneide.wordpress.com/tag/liquid-lock/

 I explained that in another post a few minutes ago, and yes, this is it.

 Maybe i should write an article on that, I though it was more well known.

 As a side note, I'm highly weary of inventing new terminology. This is a
 general remark, not related to this particular instance. Most often when
 I personally found the need to invent a new term, it was because of not
 having done my homework of looking at related work.

 Andrei

I didn't invented that term. But this is off topic. How would you call 
that phenomena ?

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 12:51 PM, deadalnix wrote:
 http://schneide.wordpress.com/tag/liquid-lock/

 I explained that in another post a few minutes ago, and yes, this is it.

 Maybe i should write an article on that, I though it was more well
 known.

 As a side note, I'm highly weary of inventing new terminology. This is a
 general remark, not related to this particular instance. Most often when
 I personally found the need to invent a new term, it was because of not
 having done my homework of looking at related work.

 Andrei

 I didn't invented that term. But this is off topic.

I know. Apparently Daniel Lindner invented it in a blog post that made 
no inroads on the net in five months.

 How would you call
 that phenomena ?

"Researching related work". It's very common outside the academic 
circles, and very frowned upon inside them.


Andrei

"Martin Nowak" <dawg dawgfoto.de> writes:

 FWIW, I recently came across the term here:
 http://schneide.wordpress.com/tag/liquid-lock/

 I explained that in another post a few minutes ago, and yes, this is i=

t.
 Maybe i should write an article on that, I though it was more well kno=

wn.

 The main problem here is the object the lock is acquired upon: the  =

 reference of lastMessage is mutable! We call this a liquid lock, becau=

se  =

 the lock isn=E2=80=99t as solid as it should be. It=E2=80=99s one of t=

he more hideous  =

 multithreading pitfalls as it looks like everything=E2=80=99s fine at =

first  =

 glance.

You should try to name the real root of the bug.
It's trying to serialize access to a channel by locking the messages.

There is nothing fancy about this, it has nothing to do with mutability
it's just about locking the wrong resource.

deadalnix <deadalnix gmail.com> writes:

Le 31/05/2012 07:18, Martin Nowak a écrit :
 FWIW, I recently came across the term here:
 http://schneide.wordpress.com/tag/liquid-lock/

 I explained that in another post a few minutes ago, and yes, this is it.

 Maybe i should write an article on that, I though it was more well known.

 The main problem here is the object the lock is acquired upon: the
 reference of lastMessage is mutable! We call this a liquid lock,
 because the lock isn’t as solid as it should be. It’s one of the more
 hideous multithreading pitfalls as it looks like everything’s fine at
 first glance.

 You should try to name the real root of the bug.
 It's trying to serialize access to a channel by locking the messages.

 There is nothing fancy about this, it has nothing to do with mutability
 it's just about locking the wrong resource.

When you lock on a non final resource, you ends up with trouble at some 
point. A have wasted enough time on that type of problem to know it for 
sure.

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 19:19, Andrei Alexandrescu a écrit :
 Must've been another poster. Anyhow, in TDPL's design only synchronized
 classes can be used with the synchronized statement.

Which is good.

 Such an object is known to be lockable, and most object will not be. It
 will avoid liquid lock, because most thing will not be lockable.

 I'm celebrating day 2 of having no idea what a liquid lock is.

I just explained that in another post.

 Combined with encapsulation capabilities that D already have, exposing
 the lock to the entire worlds is easy. It will avoid unexpected lock
 from 3rd party code, which is a source of tedious deadlock.

 It also open door for locking on struct, that is easily embeded in a
 class as value (so constructed and destructed with the class).

 The fact that anyone or not can lock/unlock a mutex is a encapsulation
 problem and we already have solution in D for that.

 ... which can be used with synchronized classes.

As said, I have nothing against synchronized classes.

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

I missed that message. I don't think synchronized classes should go 
away. I think synchronize on non synchronized classes should go away.

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 21:17, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I missed that message. I don't think synchronized classes should go
 away. I think synchronize on non synchronized classes should go away.

This is something I can agree on.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 12:26 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 21:17, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I missed that message. I don't think synchronized classes should go
 away. I think synchronize on non synchronized classes should go away.

 This is something I can agree on.

Then I'm unclear what we disagree about.

Andrei

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 21:49, Andrei Alexandrescu wrote:
 On 5/30/12 12:26 PM, Alex Rønne Petersen wrote:
 On 30-05-2012 21:17, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I missed that message. I don't think synchronized classes should go
 away. I think synchronize on non synchronized classes should go away.

 This is something I can agree on.

 Then I'm unclear what we disagree about.

 Andrei

OK, so we agree that synchronizing on non-synchronized classes is a Bad 
Thing (TM) and should be disallowed. We also agree that (while it is a 
somewhat tangential issue) the extra word used in every object is 
wasteful (right?).

Now, it's clear that a synchronized class has to store its mutex 
somewhere. We could have synchronized classes add an implicit, hidden 
field that holds the mutex. Personally, I feel that this might add a 
little too much magic (and also forces using the GC), but maybe it's 
just me. What I pointed out earlier in this thread is that 
core.sync.mutex.Mutex can be used both in a compositional style and 
through inheritance. For example:

class MySynchronizedObject : Mutex
{
}

auto mso = new MySynchronizedObject;

synchronized (mso) // this calls lock and unlock on the mutex that mso 
*is* since it inherits Mutex directly, i.e. no implicit monitor is created.
{
     // ...
}

So basically, I think the question is this: Should synchronized classes 
implicitly have core.sync.mutex.Mutex as their base class instead of 
object.Object, or should they contain a hidden field? Both options 
involve magic, which might be undesirable. The inheritance solution 
obviously places restrictions on what base type can be used for 
synchronized classes. The hidden field solution means forcing use of the GC.

Am I making sense? Thoughts on this?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 12:17 PM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I missed that message. I don't think synchronized classes should go
 away. I think synchronize on non synchronized classes should go away.

I agree.

Andrei

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 21:42, Andrei Alexandrescu a écrit :
 On 5/30/12 12:17 PM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I missed that message. I don't think synchronized classes should go
 away. I think synchronize on non synchronized classes should go away.

 I agree.

 Andrei

Ha, this is getting somewhere :D

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 12:52 PM, deadalnix wrote:
 Le 30/05/2012 21:42, Andrei Alexandrescu a écrit :
 On 5/30/12 12:17 PM, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 I missed that message. I don't think synchronized classes should go
 away. I think synchronize on non synchronized classes should go away.

 I agree.

 Andrei

 Ha, this is getting somewhere :D

It's getting where I was the whole time.

Andrei

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 21:58, Andrei Alexandrescu a écrit :
 It's getting where I was the whole time.

 Andrei

So we have a communication problem.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 1:13 PM, deadalnix wrote:
 Le 30/05/2012 21:58, Andrei Alexandrescu a écrit :
 It's getting where I was the whole time.

 Andrei

 So we have a communication problem.

Not me! :o)

Andrei

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 23:41, Andrei Alexandrescu a écrit :
 On 5/30/12 1:13 PM, deadalnix wrote:
 Le 30/05/2012 21:58, Andrei Alexandrescu a écrit :
 It's getting where I was the whole time.

 Andrei

 So we have a communication problem.

 Not me! :o)

 Andrei

Happy to know that you understand yourself <:o) . Now let's simply work 
on a mind to mind broadcasting system !

Jacob Carlborg <doob me.com> writes:

On 2012-05-30 21:58, Andrei Alexandrescu wrote:
 On 5/30/12 12:52 PM, deadalnix wrote:
 Ha, this is getting somewhere :D

 It's getting where I was the whole time.

 Andrei

No, the original problem was that you can synchronize on ALL objects, 
regardless if they're marked with "synchronized" or not. This it how it 
works today.

-- 
/Jacob Carlborg

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/31/12 12:28 AM, Jacob Carlborg wrote:
 On 2012-05-30 21:58, Andrei Alexandrescu wrote:
 On 5/30/12 12:52 PM, deadalnix wrote:
 Ha, this is getting somewhere :D

 It's getting where I was the whole time.

 Andrei

 No, the original problem was that you can synchronize on ALL objects,
 regardless if they're marked with "synchronized" or not. This it how it
 works today.

I was the whole time where TDPL is, which is not where the 
implementation is today.

Andrei

deadalnix <deadalnix gmail.com> writes:

Le 31/05/2012 09:49, Andrei Alexandrescu a écrit :
 On 5/31/12 12:28 AM, Jacob Carlborg wrote:
 On 2012-05-30 21:58, Andrei Alexandrescu wrote:
 On 5/30/12 12:52 PM, deadalnix wrote:
 Ha, this is getting somewhere :D

 It's getting where I was the whole time.

 Andrei

 No, the original problem was that you can synchronize on ALL objects,
 regardless if they're marked with "synchronized" or not. This it how it
 works today.

 I was the whole time where TDPL is, which is not where the
 implementation is today.

 Andrei

And which isn't what dlang.org say too. You can agree that this is 
confusing for everybody.

deadalnix <deadalnix gmail.com> writes:

Le 29/05/2012 00:36, Alex Rønne Petersen a écrit :
 Hi,

 I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happens
 to use the object for locking will most likely end up with a deadlock on
 their hands.
 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the same.

 The solution to (1) is to simply use a Mutex internally (or an Object
 with synchronized statements), and for (2), to simply use private global
 objects.

 (Now, regarding (1), you might argue that anyone who locks on an
 arbitrary object is doing it wrong, but you can't really blame them -
 it's frankly D's fault for allowing monitors on arbitrary objects, which
 is a horrible mess.)

 Anyone against this?

I already did some comment about this.

Making any object synchronization is a very bad design decision. It is 
deadlock prone, it is liquid lock prone, it cause an overhead on any 
object, and even worse, it is useless most of the time as D promote 
thread locality (which is very good).

OOP and concurrency are 2 orthogonal topics, and should be handled as such.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

Actually I think such a characterization is superficial and biased to 
the extent it becomes wrong.

Locks are deadlock-prone by design, whether used with objects or with 
classic procedural programs. In fact they are better confined to objects 
because the association between the protected data and the 
synchronization object is clear.

 OOP and concurrency are 2 orthogonal topics, and should be handled as
 such.

Well there would be active objects that contradict that. But really it 
is very natural to encapsulate together some data along with the mutex 
that must be acquired to manipulate it. There are known issues with 
inheritance, but they are not introduced by the approach, only exposed 
by it.


Andrei

=?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

 Actually I think such a characterization is superficial and biased to
 the extent it becomes wrong.

Really ? I think he's spot on.

 Locks are deadlock-prone by design, whether used with objects or with
 classic procedural programs. In fact they are better confined to objects
 because the association between the protected data and the
 synchronization object is clear.

There is no doubt that synchronization of any kind is hard and 
error-prone no matter how you do it. But do you really think that the 
situation is made better by allowing locking on *anything*, meaning that 
locks are effectively resources shared publicly?

Besides, what's wrong with core.sync.mutex?

 OOP and concurrency are 2 orthogonal topics, and should be handled as
 such.

 Well there would be active objects that contradict that. But really it
 is very natural to encapsulate together some data along with the mutex
 that must be acquired to manipulate it. There are known issues with
 inheritance, but they are not introduced by the approach, only exposed
 by it.


 Andrei

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

 Actually I think such a characterization is superficial and biased to
 the extent it becomes wrong.

 Really ? I think he's spot on.

I'd be glad to think the same. Good arguments might be helpful. So far 
I've seen exaggerated statements and posturing. Such are entertaining 
but are of limited effectiveness in furthering a point.

 Locks are deadlock-prone by design, whether used with objects or with
 classic procedural programs. In fact they are better confined to objects
 because the association between the protected data and the
 synchronization object is clear.

 There is no doubt that synchronization of any kind is hard and
 error-prone no matter how you do it. But do you really think that the
 situation is made better by allowing locking on *anything*, meaning that
 locks are effectively resources shared publicly?

 Besides, what's wrong with core.sync.mutex?

Your very argument that anyone can lock on it. To that, I'll add that 
core.sync.mutex is not properly associated with the resource it 
protects, so coding with bare mutexes is in fact inferior to coding with 
synchronized objects.


Andrei

=?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:

On 30-05-2012 00:50, Andrei Alexandrescu wrote:
 On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

 Actually I think such a characterization is superficial and biased to
 the extent it becomes wrong.

 Really ? I think he's spot on.

 I'd be glad to think the same. Good arguments might be helpful. So far
 I've seen exaggerated statements and posturing. Such are entertaining
 but are of limited effectiveness in furthering a point.

 Locks are deadlock-prone by design, whether used with objects or with
 classic procedural programs. In fact they are better confined to objects
 because the association between the protected data and the
 synchronization object is clear.

 There is no doubt that synchronization of any kind is hard and
 error-prone no matter how you do it. But do you really think that the
 situation is made better by allowing locking on *anything*, meaning that
 locks are effectively resources shared publicly?

 Besides, what's wrong with core.sync.mutex?

 Your very argument that anyone can lock on it. To that, I'll add that
 core.sync.mutex is not properly associated with the resource it
 protects, so coding with bare mutexes is in fact inferior to coding with
 synchronized objects.


 Andrei

I'm not sure I follow.

A mutex can be stored privately. Any object can be locked on, meaning no 
lock is effectively protected or private or... anything. It encourages 
shared locks, which is seriously the worst deadlock inducing 
anti-pattern to have ever manifested in multithreaded programming.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/29/12 3:56 PM, Alex Rønne Petersen wrote:
 A mutex can be stored privately.

It can, but that doesn't mean it will.

 Any object can be locked on, meaning no
 lock is effectively protected or private or... anything.

To paraphrase you, "An object can be stored privately".

 It encourages
 shared locks, which is seriously the worst deadlock inducing
 anti-pattern to have ever manifested in multithreaded programming.

I'll ignore the hyperbole and continued posturing. Please understand it 
does absolutely nothing in carrying your point. A technical question I 
have is - what are shared locks, and what are superior alternatives to them?


Andrei

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 3:04, Andrei Alexandrescu wrote:
 On 5/29/12 3:56 PM, Alex Rønne Petersen wrote:
 A mutex can be stored privately.

 It can, but that doesn't mean it will.

 Any object can be locked on, meaning no
 lock is effectively protected or private or... anything.

 To paraphrase you, "An object can be stored privately".

 It encourages
 shared locks, which is seriously the worst deadlock inducing
 anti-pattern to have ever manifested in multithreaded programming.

 I'll ignore the hyperbole and continued posturing. Please understand it
 does absolutely nothing in carrying your point. A technical question I
 have is - what are shared locks, and what are superior alternatives to
 them?

I'll intervene. Following famous nerd motto "talk is cheap, show me the 
code" I strongly suggest discussing concrete scenarios. All "prone to 
deadlock sentiment" is trivia as in buyer beware.

That being said:

class iMutexed
{//implicit intent, explicit details
	Mutex mutex;
}

vs

class iMutexed
{//implicit intent, implicit details

//implicit mutex
}

Makes little to no difference. EXCEPT that designer of the class has no 
control what so ever over hidden mutex! Thus I believe the best way to 
fix it is to (say) require obj be a subclass of Mutex in order to use 
synchronized(obj). i.e.

class iMutexed: Mutex
{//explicit intent, implicit details

}

-- 
Dmitry Olshansky

"Thiez" <thiezz gmail.com> writes:

On Wednesday, 30 May 2012 at 06:31:32 UTC, Dmitry Olshansky wrote:
 I'll intervene. Following famous nerd motto "talk is cheap, 
 show me the code" I strongly suggest discussing concrete 
 scenarios. All "prone to deadlock sentiment" is trivia as in 
 buyer beware.

 That being said:

 class iMutexed
 {//implicit intent, explicit details
 	Mutex mutex;
 }

 vs

 class iMutexed
 {//implicit intent, implicit details

 //implicit mutex
 }

 Makes little to no difference. EXCEPT that designer of the 
 class has no control what so ever over hidden mutex! Thus I 
 believe the best way to fix it is to (say) require obj be a 
 subclass of Mutex in order to use synchronized(obj). i.e.

 class iMutexed: Mutex
 {//explicit intent, implicit details

 }

Forcing synchronized classes to extend Mutex would make it 
impossible to create a synchronized subclass of a pre-existing 
unsynchronized class, would it not? Unless you want to introduce 
multiple inheritence. If you want to go this way at least make it 
an interface that has special meaning to the compiler, instead of 
a class that must be inherited from.

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 12:17, Thiez wrote:
 On Wednesday, 30 May 2012 at 06:31:32 UTC, Dmitry Olshansky wrote:
 I'll intervene. Following famous nerd motto "talk is cheap, show me
 the code" I strongly suggest discussing concrete scenarios. All "prone
 to deadlock sentiment" is trivia as in buyer beware.

 That being said:

 class iMutexed
 {//implicit intent, explicit details
 Mutex mutex;
 }

 vs

 class iMutexed
 {//implicit intent, implicit details

 //implicit mutex
 }

 Makes little to no difference. EXCEPT that designer of the class has
 no control what so ever over hidden mutex! Thus I believe the best way
 to fix it is to (say) require obj be a subclass of Mutex in order to
 use synchronized(obj). i.e.

 class iMutexed: Mutex
 {//explicit intent, implicit details

 }

 Forcing synchronized classes to extend Mutex would make it impossible to
 create a synchronized subclass of a pre-existing unsynchronized class,
 would it not?

Composition. Obviously synchronized subclasses of existing class sounds 
strange. The other way around even worse ;)
Wrapping synchronized classes obviously fine.

  Unless you want to introduce multiple inheritence. If you
 want to go this way at least make it an interface that has special
 meaning to the compiler, instead of a class that must be inherited from.

No thanks, multiple inheritance brings too much "coolness". Interface is 
no go, locks should be  effective.

-- 
Dmitry Olshansky

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

On 30.05.2012 12:33, Dmitry Olshansky wrote:
 On 30.05.2012 12:17, Thiez wrote:
 On Wednesday, 30 May 2012 at 06:31:32 UTC, Dmitry Olshansky wrote:
 I'll intervene. Following famous nerd motto "talk is cheap, show me
 the code" I strongly suggest discussing concrete scenarios. All "prone
 to deadlock sentiment" is trivia as in buyer beware.

 That being said:

 class iMutexed
 {//implicit intent, explicit details
 Mutex mutex;
 }

 vs

 class iMutexed
 {//implicit intent, implicit details

 //implicit mutex
 }

 Makes little to no difference. EXCEPT that designer of the class has
 no control what so ever over hidden mutex! Thus I believe the best way
 to fix it is to (say) require obj be a subclass of Mutex in order to
 use synchronized(obj). i.e.

 class iMutexed: Mutex
 {//explicit intent, implicit details

 }

 Forcing synchronized classes to extend Mutex would make it impossible to
 create a synchronized subclass of a pre-existing unsynchronized class,
 would it not?

 Composition. Obviously synchronized subclasses of existing class sounds
 strange. The other way around even worse ;)
 Wrapping synchronized classes obviously fine.

Wrapping by synchronized class....



-- 
Dmitry Olshansky

Artur Skawina <art.08.09 gmail.com> writes:

On 05/30/12 10:17, Thiez wrote:
 On Wednesday, 30 May 2012 at 06:31:32 UTC, Dmitry Olshansky wrote:
 I'll intervene. Following famous nerd motto "talk is cheap, show me the code"
I strongly suggest discussing concrete scenarios. All "prone to deadlock
sentiment" is trivia as in buyer beware.

 That being said:

 class iMutexed
 {//implicit intent, explicit details
     Mutex mutex;
 }

 vs

 class iMutexed
 {//implicit intent, implicit details

 //implicit mutex
 }

 Makes little to no difference. EXCEPT that designer of the class has no
control what so ever over hidden mutex! Thus I believe the best way to fix it
is to (say) require obj be a subclass of Mutex in order to use
synchronized(obj). i.e.

 class iMutexed: Mutex
 {//explicit intent, implicit details

 }

 
 Forcing synchronized classes to extend Mutex would make it impossible to
create a synchronized subclass of a pre-existing unsynchronized class, would it
not? Unless you want to introduce multiple inheritence. If you want to go this
way at least make it an interface that has special meaning to the compiler,
instead of a class that must be inherited from.
 

Yeah, that leads to multiple inheritance, or cheap imitations thereof.

Just create lock+unlock (or acquire+release, whatever) operators that
can be overloaded, ie

void opUnary!("lock")() {}
void opUnary!("unlock")() {}

and make 'synchronize (struct|class){...}' use them, ditto for implicitly
synchronized public methods (note: by lowering directly. You can already
do something like this right now, but it would have a large runtime
overhead.) Full backward compatibility retained, and the user can actually
implement the locking in a sane manner - something that is not likely to
happen any time soon as a built-in language feature. And yes, removing the
monitors from classes is desirable and would have back-compat issues, but
it is then just an optimization, and does not need to happen immediately.


On 05/30/12 11:34, deadalnix wrote:
 Le 29/05/2012 23:33, Andrei Alexandrescu a écrit :
 On 5/29/12 1:37 AM, deadalnix wrote:
 I would say that breaking things here, with the right deprecation
 process, is the way to go.

 So what should we use for mutex-based synchronization if we deprecate
 synchronized classes?

 Andrei

 
 I think something similar to range design here is the way to go.
 
 It is easy to define something like
 
 template isLockable(T) {
     enum isLockable = isShared!T && is(typeof(T.init.lock())) &&
is(typeof(T.init.release()));
 }
 
 And allow locking only if(isLockable!Type) .
 
 Now we can create SyncObject or any structure we want. The point is that we
lock explicit stuff.

Yep, a way to mark objects usable for locking would also help. I'd drop the
'isShared' check - it's redundant and doesn't play well with D 'shared' concept
in its current form; it can always be introduced back, when 'shared' evolves to
something more sane.


On 05/30/12 11:14, Jacob Carlborg wrote:
 On 2012-05-29 23:48, Andrei Alexandrescu wrote:
 
 Don't be hatin'. You'd appreciate the matter considerably more after you
 defined your own language and had its users yell at you for changing
 even a small detail.

 The situation is at it is, and there's no need to get agitated. We're
 all on the same boat, trying to make things work out best.

 
 It seems more and more that D2 is not a designed language. Instead new
features are just slapped on without considering how it would impact the rest
of the language.

Yes. But it is not so much the adhoc features that are the problem, as long
as they don't significantly break the rest of the language. Not fixing known
deficiencies in the name of backward compatibility is. The gain from having
a new, useful and working feature more than makes up for the cost of changes
to app code. Having features that work for trivial hello-world programs, but
fail for most real code, does much more harm than good.


artur

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 00:50, Andrei Alexandrescu a écrit :
 On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

 Actually I think such a characterization is superficial and biased to
 the extent it becomes wrong.

 Really ? I think he's spot on.

 I'd be glad to think the same. Good arguments might be helpful. So far
 I've seen exaggerated statements and posturing. Such are entertaining
 but are of limited effectiveness in furthering a point.

I have provided link in other posts to document the point. Not to 
mention that my personal experience back up this too.

I don't like what you are doing here, because you don't provide anything 
and simply discard what don't fit the current design. For instance, you 
didn't even considered the liquid lock problem.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 5/30/12 2:40 AM, deadalnix wrote:
 Le 30/05/2012 00:50, Andrei Alexandrescu a écrit :
 On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision. It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

 Actually I think such a characterization is superficial and biased to
 the extent it becomes wrong.

 Really ? I think he's spot on.

 I'd be glad to think the same. Good arguments might be helpful. So far
 I've seen exaggerated statements and posturing. Such are entertaining
 but are of limited effectiveness in furthering a point.

 I have provided link in other posts to document the point. Not to
 mention that my personal experience back up this too.

 I don't like what you are doing here, because you don't provide anything
 and simply discard what don't fit the current design. For instance, you
 didn't even considered the liquid lock problem.

It would help if I knew what a liquid lock is. The first page of Google 
search doesn't help.

I'm only discarding content-less posturing a la "worst idea ever", 
"useless", "very bad design decision" etc. Such is just immature. It is 
encouraging you have started sending content (the msdn paper), thanks, 
and please keep it coming.


Thanks,

Andrei

deadalnix <deadalnix gmail.com> writes:

Le 30/05/2012 17:50, Andrei Alexandrescu a écrit :
 On 5/30/12 2:40 AM, deadalnix wrote:
 Le 30/05/2012 00:50, Andrei Alexandrescu a écrit :
 On 5/29/12 2:59 PM, Alex Rønne Petersen wrote:
 On 29-05-2012 23:31, Andrei Alexandrescu wrote:
 On 5/29/12 1:32 AM, deadalnix wrote:
 I already did some comment about this.

 Making any object synchronization is a very bad design decision.
 It is
 deadlock prone, it is liquid lock prone, it cause an overhead on any
 object, and even worse, it is useless most of the time as D promote
 thread locality (which is very good).

 Actually I think such a characterization is superficial and biased to
 the extent it becomes wrong.

 Really ? I think he's spot on.

 I'd be glad to think the same. Good arguments might be helpful. So far
 I've seen exaggerated statements and posturing. Such are entertaining
 but are of limited effectiveness in furthering a point.

 I have provided link in other posts to document the point. Not to
 mention that my personal experience back up this too.

 I don't like what you are doing here, because you don't provide anything
 and simply discard what don't fit the current design. For instance, you
 didn't even considered the liquid lock problem.

 It would help if I knew what a liquid lock is. The first page of Google
 search doesn't help.

 I'm only discarding content-less posturing a la "worst idea ever",
 "useless", "very bad design decision" etc. Such is just immature. It is
 encouraging you have started sending content (the msdn paper), thanks,
 and please keep it coming.


 Thanks,

 Andrei

Ok let's me explain what liquid lock is. It is a common error in 
languages that allow lock on any object.

The phenomena is fairly simple, but lead to really tedious debugging 
session. It happen when you use an object that is useful for some 
functionality of your program as a mutex. This is OK because the 
instance is always the same. Then, later on, when adding new 
functionality/refactoring/whatever, you introduce a possibility to 
change the instance you lock on.

In such situation, you seems to lock in an appropriate manner, but you 
fail to ensure correct locking when it occur at the exact moment the 
object referred is changed.

This is really tedious to track down, and quite easy to create when 
several developers are working on the same codebase for an extended 
period of time. To avoid this, you must ensure that any reference on 
object you mutate isn't used somewhere as a mutex.

It is considered good practice in Java to not lock on any object and use 
a specific object to lock on. The reference to that object will be 
final, so it is guaranteed to not change.

For instance, Intellij issue a warning about that : 
http://www.coderanch.com/t/233943/threads/java/Synchronization-non-final-field

At the end, locking on any object cause trouble.

Locking on any object has proven itself to be a very bad design decision 

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Mon, 28 May 2012 18:36:13 -0400, Alex R=C3=B8nne Petersen <alex lycus=
.org>  =

wrote:

 Hi,

 I've seen several occurrences of synchronized (this) and synchronized =

 =

 (this.classinfo) in both druntime and phobos by now. I propose that we=

  =

 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happe=

ns  =

 to use the object for locking will most likely end up with a deadlock =

on  =

 their hands.
 2) Locking on something as fundamental as type info means that any  =

 arbitrary part of the application could cause a deadlock by doing the =

 =

 same.

Reading all of this thread, I think I agree with you.

Now, I'll point out a couple things:

1. Most code that is written with synchronized is *not* abused.
2. using synchronized(x) should not go away.

Here is what I'm thinking:

First, we give control of the mutex to the author of the type.  So inste=
ad  =

of an implicit location, and implicit lazy construction, we make  =

everything explicit:

Currently:

synchronized(x)
{
}

translates to

if(x.mutex =3D=3D null)
    x.mutex =3D createmutex();
lock(x.mutex)
try
{
}
finally
{
    unlock(x.mutex);
}

So I propose it does this instead (entirely via lowering):

synchronized(x)
{
}

translates to:

auto l =3D x.__getMutex();
l.lock();
try
{
}
finally
{
    l.unlock();
}

So we have a few benefits here:

1. if typeof(x) does not define __getMutex, the above is a compiler erro=
r.
2. We can control the visibility of the mutex, because we can attribute =
 =

private or protected to the __getMutex function.
3. We can control what kinds of objects of typeof(x) use mutexes.  e.g. =
 =

define __getMutex() shared only.
4. We have control over the type of the lock, it can be anything and doe=
s  =

not need to fit into some runtime function's interface.

Second, I propose a somewhat gradual transition.  I think on first  =

release, we *still* keep the allocated word in the class instance, and  =

keep the druntime function which lazily allocates the mutex.  Therefore,=
  =

changing an object that currently *properly* can use synchronized(this) =
 =

can be made to work:

class Synchronizable
{
    xxx __getMutex() shared { return _d_getObjectMonitor(this);}
}

We can deprecate that functionality, and the allocated space for the mut=
ex  =

later if desired.

-Steve

Artur Skawina <art.08.09 gmail.com> writes:

On 05/30/12 17:50, Steven Schveighoffer wrote:
 So I propose it does this instead (entirely via lowering):
 
 synchronized(x)
 {
 }
 
 translates to:
 
 auto l = x.__getMutex();
 l.lock();
 try
 {
 }
 finally
 {
    l.unlock();
 }

I like this, it's slightly more complex, but can reduce the amount of
boilerplate
(by not requiring lock/unlock forwarders etc) and is more powerful than my
proposal.

 So we have a few benefits here:
 
 1. if typeof(x) does not define __getMutex, the above is a compiler error.
 2. We can control the visibility of the mutex, because we can attribute
private or protected to the __getMutex function.
 3. We can control what kinds of objects of typeof(x) use mutexes.  e.g. define
__getMutex() shared only.
 4. We have control over the type of the lock, it can be anything and does not
need to fit into some runtime function's interface.

5. The lock doesn't have to be a real lock, but eg a lock manager. Hmm,
lowering to
l.lock(x) and l.unlock(x) might be better; inlining will make it zero-cost for
the
simple case, but this will make certain schemes much simplier (no need to encode
'x' inside the returned 'l').

 Second, I propose a somewhat gradual transition.  I think on first release, we
*still* keep the allocated word in the class instance, and keep the druntime
function which lazily allocates the mutex.  Therefore, changing an object that
currently *properly* can use synchronized(this) can be made to work:
 
 class Synchronizable
 {
    xxx __getMutex() shared { return _d_getObjectMonitor(this);}
 }
 
 We can deprecate that functionality, and the allocated space for the mutex
later if desired.

Yes, UFCS could help too.

artur

"Martin Nowak" <dawg dawgfoto.de> writes:

On Tue, 29 May 2012 00:36:13 +0200, Alex R=C3=B8nne Petersen <alex lycus=
.org>  =

wrote:

 Hi,

 I've seen several occurrences of synchronized (this) and synchronized =

 =

 (this.classinfo) in both druntime and phobos by now. I propose that we=

  =

 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who happe=

ns  =

 to use the object for locking will most likely end up with a deadlock =

on  =

 their hands.

 2) Locking on something as fundamental as type info means that any  =

 arbitrary part of the application could cause a deadlock by doing the =

 =

 same.

All occurences of this pattern use it to obtain a global unique nameable=
  =

mutex.
It's a little ugly but I fail to see fundamental issues.

synchronized(Stacktrace.classinfo)
synchronized(typeid(ArrayAllocLengthLock)) // uses an exclusive type as =
 =

named mutex
synchronized(TaskPool.classinfo)
synchronized(this.classinfo) // this a.k.a. Socket

Then there is std.concurrency.registryLock which is used in the exact sa=
me  =

ways as the occurences above. To do that it requires 6 lines of code, ha=
s  =

to successfully avoid 2 pitfalls and allocates the mutex eagerly.

Then there is core.thread which has a lot of issues with locking.

There is some suspicious code in object.rt_attachDisposeEvent
which synchronizes on it's argument.

I'm inclined to say that this is a very thin backing for such a rant.


There was one interesting argument by Regan Heath:

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

 The problem /is/ that synchronized classes/methods use a mutex which i=

s  =

 exposed publicly, and it the same mutex as used by synchronized(object=

)  =

 {}.  This exposure/re-use makes deadlocks more likely to happen, and  =

 harder to spot.

Now I do not see anyone synchronizing it's code on arbitrary objects or =
any
other kind of abuse that would increase the inherent possibility for  =

deadlocks.

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 20:26, Martin Nowak wrote:
 On Tue, 29 May 2012 00:36:13 +0200, Alex Rønne Petersen
 <alex lycus..org> wrote:

 Hi,

 I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who
 happens to use the object for locking will most likely end up with a
 deadlock on their hands.

 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the
 same.

 All occurences of this pattern use it to obtain a global unique nameable
 mutex.
 It's a little ugly but I fail to see fundamental issues.

 synchronized(Stacktrace.classinfo)
 synchronized(typeid(ArrayAllocLengthLock)) // uses an exclusive type as
 named mutex
 synchronized(TaskPool.classinfo)

And what happens if I synchronize on TaskPool.classinfo in my code? The 
point here is that deadlocks *can* happen. If we went by the logic that 
"it's not likely to go wrong, so let's not care", why do we have slices? 
scope statements? exception handling?

 synchronized(this.classinfo) // this a.k.a. Socket

 Then there is std.concurrency.registryLock which is used in the exact
 same ways as the occurences above. To do that it requires 6 lines of
 code, has to successfully avoid 2 pitfalls and allocates the mutex eagerly.

 Then there is core.thread which has a lot of issues with locking.

What issues specifically...?

 There is some suspicious code in object.rt_attachDisposeEvent
 which synchronizes on it's argument.

 I'm inclined to say that this is a very thin backing for such a rant.


 There was one interesting argument by Regan Heath:

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

 The problem /is/ that synchronized classes/methods use a mutex which
 is exposed publicly, and it the same mutex as used by
 synchronized(object) {}. This exposure/re-use makes deadlocks more
 likely to happen, and harder to spot.

 Now I do not see anyone synchronizing it's code on arbitrary objects or any
 other kind of abuse that would increase the inherent possibility for
 deadlocks.

As I pointed out in another post in this thread, I've seen the pattern 
used several times by people asking questions on media such as IRC. IRC 
has much higher traffic than e.g. D.learn.

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

"Martin Nowak" <dawg dawgfoto.de> writes:

 And what happens if I synchronize on TaskPool.classinfo in my code?

You don't own that class, you haven't written it and it's not meant to be

it'd be a bad idea to lock a literal string and the liquid-lock bug [2]
just locks nearby data to synchronize it's code. That this should be a
common source of bugs strikes me as rather odd, but then one could already
wonder why 'lock' is used intransitively. Saying 'lock on sth.' when you  
actually
want to lock sth. (data or resource) just pinpoints the underlying issue.

For example the analysis in [2] misses the actual point of the bug. He  
wanted to achieve
mutual exclusive usage of a channel but instead of locking the channel he  
locked a message.

[1] http://msdn.microsoft.com/en-us/library/ms173179.aspx
[2] http://schneide.wordpress.com/tag/liquid-lock/

For me this whole discussion boils down to the following.
Locks are quite safe if you use them to actually protect a resource.
Locks are very unsafe if you use them as part of an implicit protocol.

=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:

On 30-05-2012 20:26, Martin Nowak wrote:
 On Tue, 29 May 2012 00:36:13 +0200, Alex Rønne Petersen
 <alex lycus..org> wrote:

 Hi,

 I've seen several occurrences of synchronized (this) and synchronized
 (this.classinfo) in both druntime and phobos by now. I propose that we
 officially ban these patterns with extreme prejudice.

 1) Locking on the object instance is a HORRIBLE IDEA. Anyone who
 happens to use the object for locking will most likely end up with a
 deadlock on their hands.

 2) Locking on something as fundamental as type info means that any
 arbitrary part of the application could cause a deadlock by doing the
 same.

 All occurences of this pattern use it to obtain a global unique nameable
 mutex.
 It's a little ugly but I fail to see fundamental issues.

 synchronized(Stacktrace.classinfo)
 synchronized(typeid(ArrayAllocLengthLock)) // uses an exclusive type as
 named mutex
 synchronized(TaskPool.classinfo)
 synchronized(this.classinfo) // this a.k.a. Socket

 Then there is std.concurrency.registryLock which is used in the exact
 same ways as the occurences above. To do that it requires 6 lines of
 code, has to successfully avoid 2 pitfalls and allocates the mutex eagerly.

 Then there is core.thread which has a lot of issues with locking.

 There is some suspicious code in object.rt_attachDisposeEvent
 which synchronizes on it's argument.

 I'm inclined to say that this is a very thin backing for such a rant.

Also, I don't think calling this a rant is fair. I gave specific reasons 
for why the patterns I mentioned are bad in _libraries_ of all things 
(notice that I was talking about druntime and phobos!).

 There was one interesting argument by Regan Heath:

 The problem is /not/ that you can lock any object.
 The problem is /not/ that we have synchronized(object) {}
 The problem is /not/ that we have synchronized classes/methods.

 The problem /is/ that synchronized classes/methods use a mutex which
 is exposed publicly, and it the same mutex as used by
 synchronized(object) {}. This exposure/re-use makes deadlocks more
 likely to happen, and harder to spot.

 Now I do not see anyone synchronizing it's code on arbitrary objects or any
 other kind of abuse that would increase the inherent possibility for
 deadlocks.


-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org

"Martin Nowak" <dawg dawgfoto.de> writes:

On Wed, 30 May 2012 20:45:51 +0200, Alex R=C3=B8nne Petersen <alex lycus=
.org>  =

wrote:

 Also, I don't think calling this a rant is fair. I gave specific reaso=

ns  =

 for why the patterns I mentioned are bad in _libraries_ of all things =

 =

 (notice that I was talking about druntime and phobos!).

Sorry, I didn't meant to be harsh.
But please reread your initial post.
It's important to emphasize the specific reasons in
an objective way to start off a solid discussion.
Words like extreme, horrible, mess, ban, blame or fault
are counterproductive.