www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - synchronized - shared but actually useful

reply FeepingCreature <feepingcreature gmail.com> writes:
1. Make synchronized an attribute of fields, classes and methods
2. synchronized methods are synchronized(this) wrapping 
inconditions, outconditions, invariant checks and the main body. 
They are thus actually safe, as opposed to the status quo where 
invariants have to be synchronized separately and are thus Worse 
Than Useless™.
3. synchronized classes consist of synchronized fields and 
synchronized public methods
3.1. Only synchronized subclasses may inherit from synchronized 
classes, and vice versa.
4. immutable data is implicitly synchronized. Local variables are 
implicitly synchronized.
5. Synchronized methods must not allow mutable references to 
escape, unless they are to synchronized objects.
6. in  safe, only synchronized methods may access synchronized 
fields.

Advantages:
* simple
* does something useful
Disadvantages:
* inability to generate giant forum debates (fingers crossed)
Oct 23 2018
next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
Addendum: synchronized methods must be pure.

No setting a global variable from two different threads for you!
Oct 23 2018
prev sibling next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature 
wrote:

 1. Make synchronized an attribute of fields, classes and methods
 ...

 Advantages:
 * simple
 * does something useful
 Disadvantages:
 * inability to generate giant forum debates (fingers crossed)


:) What about structs? Or all the cases where you don't need 
'synchronized'? You can write a lock-free queue with a few 
integers and an array.
Oct 23 2018
parent FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 23 October 2018 at 12:50:30 UTC, Stanislav Blinov 
wrote:

 On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature 
 wrote:

 1. Make synchronized an attribute of fields, classes and 
 methods
 ...

 Advantages:
 * simple
 * does something useful
 Disadvantages:
 * inability to generate giant forum debates (fingers crossed)


 :) What about structs? Or all the cases where you don't need 
 'synchronized'? You can write a lock-free queue with a few 
 integers and an array.


That's why I tied it to  safe, so you can always write 
synchronized  trusted code that needs manual thread safety. That 
said, clever threading solutions, atomics and the like are mostly 
out of scope for this proposal, which is primarily aimed at 
simple oop-based code.
Oct 23 2018
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
ping

On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature 
wrote:

 Disadvantages:
 * inability to generate giant forum debates (fingers crossed)


I consider the point proven, but the downside of being unable to 
generate giant flamewars is that you eventually drop off the 
frontpage.

I really think the shared viewpoint on thread-owned data and how 
to move data from thread to thread is very alien to how thread 
safety is handled in practice. In practice, a lot of data is 
shared; I would argue a large fraction of *all* mutable data is 
shared, and access is protected using synchronization in the 
owning class. That is, objects, not threads, are the owners and 
overseers of mutable data.

Whatever solution for thread safety we arrive at should account 
for this.
Oct 26 2018
next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
ping
Oct 30 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, October 26, 2018 7:21:50 AM MDT FeepingCreature via Digitalmars-d 
wrote:

 ping

 On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature

 wrote:

 Disadvantages:
 * inability to generate giant forum debates (fingers crossed)


 I consider the point proven, but the downside of being unable to
 generate giant flamewars is that you eventually drop off the
 frontpage.

 I really think the shared viewpoint on thread-owned data and how
 to move data from thread to thread is very alien to how thread
 safety is handled in practice. In practice, a lot of data is
 shared; I would argue a large fraction of *all* mutable data is
 shared, and access is protected using synchronization in the
 owning class. That is, objects, not threads, are the owners and
 overseers of mutable data.

 Whatever solution for thread safety we arrive at should account
 for this.


Really, all shared is about is preventing common bugs related to
multithreading by segregating data shared across threads and making
operations which aren't guaranteed to be thread-safe illegal (the main
benefit then being that you know that everything else then isn't shared
across threads). The language doesn't do that entirely correctly at the
moment (e.g. copying and assigning shared data is currently legal in spite
of not being thread-safe) but ultimately, when you use shared, what you're
doing is pretty much exactly what you'd do in C/C++ except that the compiler
prevents many bugs with regards to data shared across threads, and because
of the extra type information in place to enable the restrictions to give
you that protection, you're often forced to cast away shared in  trusted
code to then do what you would have done in C++. But all of the atomics and
mutexes and whatnot are the same as what you'd be doing in C++. The basic
design patterns are the same. It's the fact that you you're getting compiler
errors when you do something that's not thread-safe that's different, and
it's the fact that you have to cast away shared after locking a mutex that's
different (since C++ doesn't have shared).

But aside from that, what you're doing is the same. The fact that the
compiler prevents you from doing basic stuff while the variable is shared
and requires casting (unless you use atomics) frequently causes folks to get
mad at shared and misunderstand it, but that's really the compiler
preventing you from shooting yourself in the foot. Once you understand
what's going on, it's really just the same as C++ but with some extra casts.

So, shared is very low level ultimately. It's about how the data is stored
and accessed, not about how higher level constructs are written. We can then
build other, higher level mechanisms on top of that - such as
std.concurrency - but at the language level, whether you're passing data
across threads or having multiple threads access data on the same thread
isn't really part of the design. The language is just concerned about the
fact that the data is shared.

The only higher level threading mechanism that the language has is
synchronized, and all that really is is a built-in mutex, which in
conjunction whith synchronized classes (if they're ever fully implemented)
would allow for the implicit casting away of the top level of shared inside
of member functions, but that's really as far as it goes, and without some
kind of ownership model in the language, it's _very_ hard to go much farther
than that - either with implicitly removing shared in code or with having
passing ownership across threads work without explicit casts.

Certainly, I think that any attempt to move shared away from being a low
level construct indicating simply that the data is shared across threads is
mis-guided. It's what other constructs for sharing data across threads are
built upon. So, if someone wants to push for some new mechanism that
improves sharing data across threads (rather than passing it across threads)
that's built on top of shared, that's fine. If someone wants to push for
some new mechanism that improves sharing data by passing it across threads,
that's fine. But shared is a low level construct and should stay that way.
It's the base for everything else.

And to be honest, I don't really expect it to be reasonable to add much in
the way of higher level mechanisms for sharing data to D at the language
level - not with it being a systems language. Most stuff would require
restricting it further and/or introducing a serious ownership model to it,
which we really don't want to do. Even synchronized classes are pretty
questionable, because they can only safely remove the outer layer of shared,
which is arguably better than nothing, but it doesn't get you very far.
Without a full-blown ownership model, we're pretty much reduced to manual
casting when mutexes or synchronized are used, and the language really isn't
going to be much help. If someone can come up with some bright ideas, great.
All the more power to them. Maybe we can improve the situation. But again,
they should be built on top of shared as a low level mechanism, not try to
change what shared is.

- Jonathan M Davis
Oct 30 2018
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis 
wrote:

 But shared is a low level construct and should stay that way. 
 It's the base for everything else.


This is simply not correct. People didn't stop having a need for 
threading while D was figuring out how to unbreak shared for ten 
years and counting. Fact is, the base for everything else is 
__gshared and synchronized, and it will *continue* to be 
__gshared and synchronized unless __gshared is removed and the 
Thread constructor is changed to only take shared delegates.

Why not do something useful with synchronized instead?


 And to be honest, I don't really expect it to be reasonable to 
 add much in the way of higher level mechanisms for sharing data 
 to D at the language level - not with it being a systems 
 language. Most stuff would require restricting it further 
 and/or introducing a serious ownership model to it, which we 
 really don't want to do. Even synchronized classes are pretty 
 questionable, because they can only safely remove the outer 
 layer of shared, which is arguably better than nothing, but it 
 doesn't get you very far. Without a full-blown ownership model, 
 we're pretty much reduced to manual casting when mutexes or 
 synchronized are used, and the language really isn't going to 
 be much help. If someone can come up with some bright ideas, 
 great. All the more power to them. Maybe we can improve the 
 situation. But again, they should be built on top of shared as 
 a low level mechanism, not try to change what shared is.

 - Jonathan M Davis



Sorry, but can you please actually comment on my proposal in 
concrete terms rather than handwave about how we're reduced to 
manual casting (which is not necessary for concurrency at the 
moment, so I'm not sure why you think it's unavoidable), and how 
the language surely cannot be of much help without even 
commenting on a proposal for how it *could* actually be of help?
Oct 31 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, October 31, 2018 1:00:32 AM MDT FeepingCreature via 
Digitalmars-d wrote:

 On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis

 wrote:

 But shared is a low level construct and should stay that way.
 It's the base for everything else.


 This is simply not correct. People didn't stop having a need for
 threading while D was figuring out how to unbreak shared for ten
 years and counting. Fact is, the base for everything else is
 __gshared and synchronized, and it will *continue* to be
 __gshared and synchronized unless __gshared is removed and the
 Thread constructor is changed to only take shared delegates.

 Why not do something useful with synchronized instead?


__gshared is intended for interacting with C globals and that's it. If it's
used for anything else, it's not being used for its intended purpose.
Certainly, using it for D objects is just begging for bugs. __gshared should
_never_ be used for D objects. I know that some folks do, and sometimes
folks get away with it, but it's just asking for trouble, and there's no
guarantee that such code will continue to work, because it's violating
compiler guarantees when it does so. __gshared is designed solely as a
backdoor for dealing with C and that is it. On the whole, shared really is
_not_ broken and _does_ work. Yes, it has some rough edges that need fixing,
and not everything in core.sync has been updated to use it properly,
necessitating more casts than should be necessary when using those
constructs than should be necessary, but shared as a whole is sound and
always has been. The biggest problem with it is that most folks don't
understand how to use it properly.


 And to be honest, I don't really expect it to be reasonable to
 add much in the way of higher level mechanisms for sharing data
 to D at the language level - not with it being a systems
 language. Most stuff would require restricting it further
 and/or introducing a serious ownership model to it, which we
 really don't want to do. Even synchronized classes are pretty
 questionable, because they can only safely remove the outer
 layer of shared, which is arguably better than nothing, but it
 doesn't get you very far. Without a full-blown ownership model,
 we're pretty much reduced to manual casting when mutexes or
 synchronized are used, and the language really isn't going to
 be much help. If someone can come up with some bright ideas,
 great. All the more power to them. Maybe we can improve the
 situation. But again, they should be built on top of shared as
 a low level mechanism, not try to change what shared is.


 Sorry, but can you please actually comment on my proposal in
 concrete terms rather than handwave about how we're reduced to
 manual casting (which is not necessary for concurrency at the
 moment, so I'm not sure why you think it's unavoidable), and how
 the language surely cannot be of much help without even
 commenting on a proposal for how it *could* actually be of help?


Manual casting is required if you're actually using shared with mutexes or
synchronized. If you're using __gshared, it's not, but that's only because
you're lying to the compiler about whether the variables are shared are not,
which means that you're circumventing the type system, and you're risking
subtle bugs in your programs, because the guarantees that the type system is
suppose make aren't actually in place.

Your synchronized field proposal is basically just a variation of
synchronized classes with the extra caveat that somehow mutable references
to the member variables can't escape, and for some reason, not everything in
the class has to be synchronized. It's like you're trying to have
synchronized classes without talking about shared. But you can't take shared
out of the mix. Per the type system, anything that isn't shared is
considered thread-local by the compiler, and trying to do anything like this
without shared would be a serious violation of the type system and the
compiler's guarantees.

Instead of talking about safely casting away the outer layer of shared
inside the class, you're talking about preventing escaping. synchronized
classes as described in TDPL already guaranteed that the outer layer can't
escape, since it lives directly in the class. What they can't guarantee is
that the rest can't escape, and the language doesn't provide any way for
such a guarantee. scope (even with DIP 1000) isn't transitive, so it doesn't
really do the trick. So, I don't know how we could guarantee that _nothing_
escapes, though if nothing escapes, then all levels of shared could be
implicitly cast away instead of just the outer layer, which would be nice.

Also, even if we _could_ somehow prevent anything from escaping, I would
point out that mutability has nothing to do with this (or at least constness
doesn't), because reading has many of the same threading issues as writing.
In order to avoid threading issues, the data must actually be immutable. So,
allowing const references to escape would still violate thread-safety.
Access to data must be synchronized unless it is immutable - whether we're
talking about reading or writing is irrelevant.

synchronized classes work (or would work if fully implemented) by
guaranteeing that a single level can't escape - the one level that it _can_
guarantee can't escape - and thus are able to properly synchronize that
level. If scope were transitive, then maybe we could do better, but since it
isn't (and Walter insists that it can't reasonbly be transitive), I'm not
sure that it's really reasonable to prevent escaping transitively.

In any case, your proposal is just a variation of synchronized classes. So,
while it may state it in a new way, the basic concept really isn't anything
new. And my point still stands that anything that we do with regards to
thread safety must be built on top of shared, because that is the building
block of sharing data across threads in D. Whether you like the specifics of
shared or not, the idea that the compiler can rely on the assumption that
data is _not_ shared across threads unless it's marked as such is a key
feature of D. And attempting to use __gshared to get around it is just
shooting yourself in the foot.

- Jonathan M Davis
Oct 31 2018
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 07:57:59 UTC, Jonathan M Davis 
wrote:

 On Wednesday, October 31, 2018 1:00:32 AM MDT FeepingCreature 
 via Digitalmars-d wrote:

 On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis

 wrote:

 But shared is a low level construct and should stay that 
 way. It's the base for everything else.


 This is simply not correct. People didn't stop having a need 
 for threading while D was figuring out how to unbreak shared 
 for ten years and counting. Fact is, the base for everything 
 else is __gshared and synchronized, and it will *continue* to 
 be __gshared and synchronized unless __gshared is removed and 
 the Thread constructor is changed to only take shared 
 delegates.

 Why not do something useful with synchronized instead?


 __gshared is intended for interacting with C globals and that's 
 it. If it's used for anything else, it's not being used for its 
 intended purpose. Certainly, using it for D objects is just 
 begging for bugs. __gshared should _never_ be used for D 
 objects. I know that some folks do, and sometimes folks get 
 away with it, but it's just asking for trouble, and there's no 
 guarantee that such code will continue to work, because it's 
 violating compiler guarantees when it does so. __gshared is 
 designed solely as a backdoor for dealing with C and that is 
 it. On the whole, shared really is _not_ broken and _does_ 
 work. Yes, it has some rough edges that need fixing, and not 
 everything in core.sync has been updated to use it properly, 
 necessitating more casts than should be necessary when using 
 those constructs than should be necessary, but shared as a 
 whole is sound and always has been. The biggest problem with it 
 is that most folks don't understand how to use it properly.


 And to be honest, I don't really expect it to be reasonable 
 to add much in the way of higher level mechanisms for 
 sharing data to D at the language level - not with it being 
 a systems language. Most stuff would require restricting it 
 further and/or introducing a serious ownership model to it, 
 which we really don't want to do. Even synchronized classes 
 are pretty questionable, because they can only safely remove 
 the outer layer of shared, which is arguably better than 
 nothing, but it doesn't get you very far. Without a 
 full-blown ownership model, we're pretty much reduced to 
 manual casting when mutexes or synchronized are used, and 
 the language really isn't going to be much help. If someone 
 can come up with some bright ideas, great. All the more 
 power to them. Maybe we can improve the situation. But 
 again, they should be built on top of shared as a low level 
 mechanism, not try to change what shared is.


 Sorry, but can you please actually comment on my proposal in 
 concrete terms rather than handwave about how we're reduced to 
 manual casting (which is not necessary for concurrency at the 
 moment, so I'm not sure why you think it's unavoidable), and 
 how the language surely cannot be of much help without even 
 commenting on a proposal for how it *could* actually be of 
 help?


 Manual casting is required if you're actually using shared with 
 mutexes or synchronized. If you're using __gshared, it's not, 
 but that's only because you're lying to the compiler about 
 whether the variables are shared are not, which means that 
 you're circumventing the type system, and you're risking subtle 
 bugs in your programs, because the guarantees that the type 
 system is suppose make aren't actually in place.


This does not match what I'm actually seeing in practice. If this 
is how it is, then this admonishment should be written in the D 
style guide in big, bolded letters.
Generally we try to write cast-free code if possible, so casting 
away shared feels wrong. That's not really an argument, of 
course, but to me semantic casting indicates "no compiler, you 
are mistaken about the properties of this data." The compiler 
should never *require* casting.


 Your synchronized field proposal is basically just a variation 
 of synchronized classes with the extra caveat that somehow 
 mutable references to the member variables can't escape, and 
 for some reason, not everything in the class has to be 
 synchronized.


Everything in the class has to be synchronized. I'm not sure 
where I said differently.


 It's like you're trying to have synchronized classes without 
 talking about shared. But you can't take shared out of the mix. 
 Per the type system, anything that isn't shared is considered 
 thread-local by the compiler, and trying to do anything like 
 this without shared would be a serious violation of the type 
 system and the compiler's guarantees.


Again, imo this belongs in either the spec or the DStyle guide in 
big bold letters.


 Instead of talking about safely casting away the outer layer of 
 shared inside the class, you're talking about preventing 
 escaping. synchronized classes as described in TDPL already 
 guaranteed that the outer layer can't escape, since it lives 
 directly in the class. What they can't guarantee is that the 
 rest can't escape, and the language doesn't provide any way for 
 such a guarantee.


Not sure what you mean by "outer layer" here or this entire 
paragraph.


 scope (even with DIP 1000) isn't transitive, so it doesn't 
 really do the trick. So, I don't know how we could guarantee 
 that _nothing_ escapes, though if nothing escapes, then all 
 levels of shared could be implicitly cast away instead of just 
 the outer layer, which would be nice.

 Also, even if we _could_ somehow prevent anything from 
 escaping, I would point out that mutability has nothing to do 
 with this (or at least constness doesn't), because reading has 
 many of the same threading issues as writing. In order to avoid 
 threading issues, the data must actually be immutable. So, 
 allowing const references to escape would still violate 
 thread-safety. Access to data must be synchronized unless it is 
 immutable - whether we're talking about reading or writing is 
 irrelevant.


Indeed, const is useless for threading, which is why my proposal 
explicitly only notes immutable data as implicitly synchronized. 
We're on the same page.


 synchronized classes work (or would work if fully implemented) 
 by guaranteeing that a single level can't escape - the one 
 level that it _can_ guarantee can't escape - and thus are able 
 to properly synchronize that level. If scope were transitive, 
 then maybe we could do better, but since it isn't (and Walter 
 insists that it can't reasonbly be transitive), I'm not sure 
 that it's really reasonable to prevent escaping transitively.


The point of the proposal is not precisely to prevent escaping, 
but to prevent the escape of *data structures/references that 
offer the ability to mutate data unprotected.* Remember that in 
this model, the object is the unit of data ownership. So the two 
things that are allowed to escape are immutable data (and 
non-reference copied data), and synchronized classes, because 
they can guarantee that they're taking care of protecting their 
data.


 In any case, your proposal is just a variation of synchronized 
 classes. So, while it may state it in a new way, the basic 
 concept really isn't anything new.


I didn't say it was! I said it was *useful* and *straightforward*.


 And my point still stands that anything that we do with regards 
 to thread safety must be built on top of shared, because that 
 is the building block of sharing data across threads in D. 
 Whether you like the specifics of shared or not, the idea that 
 the compiler can rely on the assumption that data is _not_ 
 shared across threads unless it's marked as such is a key 
 feature of D. And attempting to use __gshared to get around it 
 is just shooting yourself in the foot.

 - Jonathan M Davis



And yet, there's no way to even *state* that the Thread 
constructor must not take a delegate that can access unshared 
data, and none of the current proposals list one. Should we 
demand that only shared data can be passed to new threads? That's 
going to limit their usefulness. With synchronized, we can 
deprecate the delegate constructor, then pass the thread data as 
synchronized objects to a pure function, thereby actually 
guaranteeing thread safety.
Oct 31 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, October 31, 2018 2:08:58 AM MDT FeepingCreature via 
Digitalmars-d wrote:

 On Wednesday, 31 October 2018 at 07:57:59 UTC, Jonathan M Davis

 Manual casting is required if you're actually using shared with
 mutexes or synchronized. If you're using __gshared, it's not,
 but that's only because you're lying to the compiler about
 whether the variables are shared are not, which means that
 you're circumventing the type system, and you're risking subtle
 bugs in your programs, because the guarantees that the type
 system is suppose make aren't actually in place.


 This does not match what I'm actually seeing in practice. If this
 is how it is, then this admonishment should be written in the D
 style guide in big, bolded letters.


Probably. There are places in the documentation that talk about it, but it's
almost certainly not clear enough given how many folks keep using it.
Certainly, far too many folks run to __gshared when they get annoyed with
shared.

In general, we've done a terrible job messaging how shared works, in part
because it's been 95% complete for ages, and it's never been a priority. In
general, what shared must do (and _mostly_ does) is guarantee that no
operations that are not thread-safe are illegal. Earlier on, there was a
larger push to find ways to get the compiler to then do stuff for you to
make stuff thread-safe (such as introducing write barriers), and Andrei
tends to prefer that approach, but that approach is hard to do (if possible
at all), and pieces that we _can_ do (like write barriers) tend to be
inefficient. Walter prefers the approach of just making more operations
illegal (which unfortunately, then tends to require more casts and more
 trusted code). So, some stuff has been left in limbo that really shouldn't
have been left in limbo, and so we have shared - which works - but where
some operations aren't actually thread-safe but are still legal (hence why I
say it's 95% complete). And that still needs to be sorted out. And since it
hasn't been a priority, and it's perpetually mostly finished, you don't tend
to get much in the way of stuff like documentation write-ups about it. That
really should be fixed.


 Generally we try to write cast-free code if possible, so casting
 away shared feels wrong. That's not really an argument, of
 course, but to me semantic casting indicates "no compiler, you
 are mistaken about the properties of this data." The compiler
 should never *require* casting.


In the case of shared, it does require casting, and there really isn't a
good way around it. The two major exceptions are atomics and synchronized
classes (though those aren't fully implemented). synchronized classes still
do the casting; they're just able to cast away shared on the outer layer
implicitly. It would be fantastic to be able to do it implicitly in more
cases, but it can only be done implicitly in cases where the compiler can
actually guarantee that it's thread-safe, and that's really, really hard to
do. synchronized classes are the best proposal that we've had for a
mechanism for doing so. More would likely be possible with a more
complicated ownership model, but without that, the compiler simply doesn't
have enough information. It needs to know stuff like that a mutex is
associated with a particular variable or set of variables, that it's
definitely locked in a section of code, and that it's absolutely impossible
that there are any other references to that data which are not currently
protected by that mutex. The biggest obstacle there is probably the bit
about there being no references not currently protected. Without a real
ownership model, that just doesn't work. Maybe someone will come up with a
bright idea, but everything I've seen at best has subtle holes in it, and
ultimately, it comes down to basically writing the same kind of code that
you'd write in C/C++ except that you need a few extra casts within the
sections of code where the shared variables are protected by a locked mutex.


 Your synchronized field proposal is basically just a variation
 of synchronized classes with the extra caveat that somehow
 mutable references to the member variables can't escape, and
 for some reason, not everything in the class has to be
 synchronized.


 Everything in the class has to be synchronized. I'm not sure
 where I said differently.


You were talking about marking individual fields and variables as
synchronized rather than simply marking the entire class as synchronized.


 It's like you're trying to have synchronized classes without
 talking about shared. But you can't take shared out of the mix.
 Per the type system, anything that isn't shared is considered
 thread-local by the compiler, and trying to do anything like
 this without shared would be a serious violation of the type
 system and the compiler's guarantees.


 Again, imo this belongs in either the spec or the DStyle guide in
 big bold letters.


 Instead of talking about safely casting away the outer layer of
 shared inside the class, you're talking about preventing
 escaping. synchronized classes as described in TDPL already
 guaranteed that the outer layer can't escape, since it lives
 directly in the class. What they can't guarantee is that the
 rest can't escape, and the language doesn't provide any way for
 such a guarantee.


 Not sure what you mean by "outer layer" here or this entire
 paragraph.


The part directly embedded in the class and which therefore cannot possibly
escape. Unlike regular classes, synchronized classes do not allow direct
access to their member variables - even from anything else in the same
module - in order to be able to provide that guarantee. If you don't have
your own copy of TDPL, I'd suggest reading the concurrency chapter which is
available for free online, since it talks about synchronized classes:

http://www.informit.com/articles/article.aspx?p=1609144

Unfortunately, they're only partially implemented. IIRC, restricting access
to the member variables has been implemented, but beyond that, right now
they're mostly just synchronized functions like in Java.


 And my point still stands that anything that we do with regards
 to thread safety must be built on top of shared, because that
 is the building block of sharing data across threads in D.
 Whether you like the specifics of shared or not, the idea that
 the compiler can rely on the assumption that data is _not_
 shared across threads unless it's marked as such is a key
 feature of D. And attempting to use __gshared to get around it
 is just shooting yourself in the foot.


 And yet, there's no way to even *state* that the Thread
 constructor must not take a delegate that can access unshared
 data, and none of the current proposals list one. Should we
 demand that only shared data can be passed to new threads? That's
 going to limit their usefulness. With synchronized, we can
 deprecate the delegate constructor, then pass the thread data as
 synchronized objects to a pure function, thereby actually
 guaranteeing thread safety.


Well, in principle if you're passing data across threads, it should be
shared while it's passed across threads, and the type system requires that.
That's why you have to cast to immutable when passing reference types with
std.concurrency (shared would work too except for an issue with
std.variant's implementation). You can cast to thread-local again on the
other side, but to get it across, it needs to be typed as shared /
immutable, because that's what it really is at that point. And what you're
doing is inherently  system, hence why the casting is required. It's up to
the programmer to ensure that no references to the data remain on the
original thread. If we had a full-own ownership model in the type system,
then we might be able to avoid that, but we don't so we can't.

Thread is a bit different, but the concept is basically the same.
Conceptually, it's being passed off to another thread, so it really should
be shared or immutable, but C doesn't actually _have_ shared (hence
__gshared), so everything involved _has_ to be  trusted internally. Looking
at Thread's constructor, having it take a delegate probably isn't a problem,
but what _is_ a problem is the fact that it's  safe. I don't know how we can
reasonably fix that without breaking code, but it's definitely a problem.
All data passed through that delegate must either be shared or it must be
the only reference to that data so that when it ends up on that new thread,
and it's thread-local, no other thread has reference to it, and it really is
properly thread-local. That constructor can't guarantee that. It's up to the
programmer to guarantee that. So, it needs to be  system.

If we could somehow require that the delegate only accepted shared data,
then we could reasonably make the constructor  safe, but since AFAIK, that's
not possible, it needs to be  system.

- Jonathan M Davis
Oct 31 2018
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 09:04:43 UTC, Jonathan M Davis 
wrote:

 Everything in the class has to be synchronized. I'm not sure 
 where I said differently.


 You were talking about marking individual fields and variables 
 as synchronized rather than simply marking the entire class as 
 synchronized.


Sorry- I see your point. There is indeed no reason for 
non-synchronized fields. Strike that from the proposal. I'm not 
sure why I had it in there.


 The part directly embedded in the class and which therefore 
 cannot possibly escape. Unlike regular classes, synchronized 
 classes do not allow direct access to their member variables - 
 even from anything else in the same module - in order to be 
 able to provide that guarantee. If you don't have your own copy 
 of TDPL, I'd suggest reading the concurrency chapter which is 
 available for free online, since it talks about synchronized 
 classes:

 http://www.informit.com/articles/article.aspx?p=1609144


Wow, I straight up hadn't known that synchronized classes were a 
thing. I think it's because synchronized is not listed as an 
attribute on the Attributes page, but rather as a separate syntax 
on the Class page.


 And yet, there's no way to even *state* that the Thread 
 constructor must not take a delegate that can access unshared 
 data, and none of the current proposals list one. Should we 
 demand that only shared data can be passed to new threads? 
 That's going to limit their usefulness. With synchronized, we 
 can deprecate the delegate constructor, then pass the thread 
 data as synchronized objects to a pure function, thereby 
 actually guaranteeing thread safety.


 
 [ snip std.concurrency which we're not using ]
 
 Thread is a bit different, but the concept is basically the 
 same. Conceptually, it's being passed off to another thread, so 
 it really should be shared or immutable, but C doesn't actually 
 _have_ shared (hence __gshared), so everything involved _has_ 
 to be  trusted internally. Looking at Thread's constructor, 
 having it take a delegate probably isn't a problem, but what 
 _is_ a problem is the fact that it's  safe. I don't know how we 
 can reasonably fix that without breaking code, but it's 
 definitely a problem. All data passed through that delegate 
 must either be shared or it must be the only reference to that 
 data so that when it ends up on that new thread, and it's 
 thread-local, no other thread has reference to it, and it 
 really is properly thread-local. That constructor can't 
 guarantee that. It's up to the programmer to guarantee that. 
 So, it needs to be  system.

 If we could somehow require that the delegate only accepted 
 shared data, then we could reasonably make the constructor 
  safe, but since AFAIK, that's not possible, it needs to be 
  system.

 - Jonathan M Davis


There doesn't seem to be a trait to detect a synchronized class. 
Having learnt that synchronized classes are actually mostly 
implemented (how did I not know this! Though I still think the 
fact that they can be impure is an issue, but that's statically 
checkable at least), it seems like it should at least be possible 
to require that the Thread runner only accesses shared data, by 
limiting it to take a pure function that only has 
synchronized-class parameters. Well, it would be, if there was a 
trait for synchronized classes, but that seems like it'd be much 
easier to add.
Oct 31 2018
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 31 October 2018 at 09:36:22 UTC, FeepingCreature 
wrote:


 it seems like it should at least be possible to require that 
 the Thread runner only accesses shared data, by limiting it to 
 take a pure function that only has synchronized-class 
 parameters. Well, it would be, if there was a trait for 
 synchronized classes, but that seems like it'd be much easier 
 to add.


And so you're jumping from your "simple oop cases" to "let's 
infect the runtime with a narrow-minded view of thread-safety and 
inflict synchronization on everyone using threads"? And what if 
anyone, deity forbid, wants to pass data that is not actually 
mutex-protected, because they happen to know that there's a lot 
more to synchronizing data access than that?
Oct 31 2018
parent FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 09:41:58 UTC, Stanislav Blinov 
wrote:

 On Wednesday, 31 October 2018 at 09:36:22 UTC, FeepingCreature 
 wrote:


 it seems like it should at least be possible to require that 
 the Thread runner only accesses shared data, by limiting it to 
 take a pure function that only has synchronized-class 
 parameters. Well, it would be, if there was a trait for 
 synchronized classes, but that seems like it'd be much easier 
 to add.


 And so you're jumping from your "simple oop cases" to "let's 
 infect the runtime with a narrow-minded view of thread-safety 
 and inflict synchronization on everyone using threads"? And 
 what if anyone, deity forbid, wants to pass data that is not 
 actually mutex-protected, because they happen to know that 
 there's a lot more to synchronizing data access than that?


Sorry, allow me to restate this.

It seems like it should be possible that we, in our utility 
libraries, write a Thread wrapper that actually enforces that 
<see above>.
Oct 31 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, October 31, 2018 3:36:22 AM MDT FeepingCreature via 
Digitalmars-d wrote:

 There doesn't seem to be a trait to detect a synchronized class.
 Having learnt that synchronized classes are actually mostly
 implemented (how did I not know this! Though I still think the
 fact that they can be impure is an issue, but that's statically
 checkable at least),


synchronized classes are only partially implemented, and the key feature
that they implicitly remove the outer layer of shared in their member
functions is one of the things that isn't implemented. I don't know exactly
what is and isn't implemented at the moment. I recall restrictions on
accessing member variables from outside the class being implemented, but I
don't think that anyone has really sat down and worked through what's left
to do (though I could be wrong - I don't follow the list of dmd PR's
closely). I don't see why the member functions being impure is a problem.
Direct access to the member variables is protected, and I don't think that
you're supposed to be allowed to pass a pointer to a member variable out of
the class, but you can escape stuff that the member variables refer to, so I
don't see why it matters particularly that they would have access to
module-level variables. synchronized classes really only protect the stuff
that sit directly inside the class itself, not what the class' member
variables refer to. If they were to fully protect everything inside of it,
then you couldn't really pass anything in or out of it.

- Jonathan M Davis
Oct 31 2018
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 10:25:52 UTC, Jonathan M Davis 
wrote:

 synchronized classes really only protect the stuff that sit 
 directly inside the class itself, not what the class' member 
 variables refer to. If they were to fully protect everything 
 inside of it, then you couldn't really pass anything in or out 
 of it.

 - Jonathan M Davis


You could pass other synchronized objects, immutable data and 
data by value.

In many project designs, that's really all you need.
Oct 31 2018
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 10:36:35 UTC, FeepingCreature 
wrote:

 On Wednesday, 31 October 2018 at 10:25:52 UTC, Jonathan M Davis 
 wrote:

 synchronized classes really only protect the stuff that sit 
 directly inside the class itself, not what the class' member 
 variables refer to. If they were to fully protect everything 
 inside of it, then you couldn't really pass anything in or out 
 of it.

 - Jonathan M Davis


 You could pass other synchronized objects, immutable data and 
 data by value.

 In many project designs, that's really all you need.


Apropos of running into a production threading error that would 
have been fixed by this synchronized proposal, let me just bump 
this thread to note that it's four years later and this code 
still fails to compile for absolutely zero reason:

```
synchronized class A
{
     private int e;
     void foo() { e++; }
}
```
Jul 01 2022
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 1 July 2022 at 07:13:23 UTC, FeepingCreature wrote:

 Apropos of running into a production threading error that would 
 have been fixed by this synchronized proposal, let me just bump 
 this thread to note that it's four years later and this code 
 still fails to compile for absolutely zero reason:

 ```
 synchronized class A
 {
     private int e;
     void foo() { e++; }
 }
 ```


That is a weird bug, it works for ‘e=e+1’?
Jul 01 2022
parent FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 1 July 2022 at 07:42:07 UTC, Ola Fosheim Grøstad wrote:

 On Friday, 1 July 2022 at 07:13:23 UTC, FeepingCreature wrote:

 Apropos of running into a production threading error that 
 would have been fixed by this synchronized proposal, let me 
 just bump this thread to note that it's four years later and 
 this code still fails to compile for absolutely zero reason:

 ```
 synchronized class A
 {
     private int e;
     void foo() { e++; }
 }
 ```


 That is a weird bug, it works for ‘e=e+1’?


Huh... cool!

I've ran into some issues with trying it in a somewhat nontrivial 
class (it doesn't really interact with associative arrays 
correctly), but if this *should* work now, I'm going to try it on 
simpler classes. Maybe it's good now for some cases!
Jul 01 2022
prev sibling parent FeepingCreature <feepingcreature gmail.com> writes:
Update to previous post:

synchronized class Class {
     void foo()  safe { }
}

void main() { auto object = new Class; object.foo(); }

This errors because object is not shared. Why is that a problem? 
It's not shared, but it is synchronized; there is no possibility 
of a threading issue here.

Analogously, I cannot do

synchronized class Class {
     private Class object;
     Class foo()  safe { return this.object; }
}

either, despite the fact that Class is again synchronized and 
thus should be able to be treated as shared implicitly.
Oct 31 2018