www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - synchronized - shared but actually useful

reply FeepingCreature <feepingcreature gmail.com> writes:
1. Make synchronized an attribute of fields, classes and methods
2. synchronized methods are synchronized(this) wrapping 
inconditions, outconditions, invariant checks and the main body. 
They are thus actually safe, as opposed to the status quo where 
invariants have to be synchronized separately and are thus Worse 
Than Useless™.
3. synchronized classes consist of synchronized fields and 
synchronized public methods
3.1. Only synchronized subclasses may inherit from synchronized 
classes, and vice versa.
4. immutable data is implicitly synchronized. Local variables are 
implicitly synchronized.
5. Synchronized methods must not allow mutable references to 
escape, unless they are to synchronized objects.
6. in  safe, only synchronized methods may access synchronized 
fields.

Advantages:
* simple
* does something useful
Disadvantages:
* inability to generate giant forum debates (fingers crossed)
Oct 23 2018
next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
Addendum: synchronized methods must be pure.

No setting a global variable from two different threads for you!
Oct 23 2018
prev sibling next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature 
wrote:
 1. Make synchronized an attribute of fields, classes and methods
 ...

 Advantages:
 * simple
 * does something useful
 Disadvantages:
 * inability to generate giant forum debates (fingers crossed)
:) What about structs? Or all the cases where you don't need 'synchronized'? You can write a lock-free queue with a few integers and an array.
Oct 23 2018
parent FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 23 October 2018 at 12:50:30 UTC, Stanislav Blinov 
wrote:
 On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature 
 wrote:
 1. Make synchronized an attribute of fields, classes and 
 methods
 ...

 Advantages:
 * simple
 * does something useful
 Disadvantages:
 * inability to generate giant forum debates (fingers crossed)
:) What about structs? Or all the cases where you don't need 'synchronized'? You can write a lock-free queue with a few integers and an array.
That's why I tied it to safe, so you can always write synchronized trusted code that needs manual thread safety. That said, clever threading solutions, atomics and the like are mostly out of scope for this proposal, which is primarily aimed at simple oop-based code.
Oct 23 2018
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
ping

On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature 
wrote:
 Disadvantages:
 * inability to generate giant forum debates (fingers crossed)
I consider the point proven, but the downside of being unable to generate giant flamewars is that you eventually drop off the frontpage. I really think the shared viewpoint on thread-owned data and how to move data from thread to thread is very alien to how thread safety is handled in practice. In practice, a lot of data is shared; I would argue a large fraction of *all* mutable data is shared, and access is protected using synchronization in the owning class. That is, objects, not threads, are the owners and overseers of mutable data. Whatever solution for thread safety we arrive at should account for this.
Oct 26 2018
next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
ping
Oct 30 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, October 26, 2018 7:21:50 AM MDT FeepingCreature via Digitalmars-d 
wrote:
 ping

 On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature

 wrote:
 Disadvantages:
 * inability to generate giant forum debates (fingers crossed)
I consider the point proven, but the downside of being unable to generate giant flamewars is that you eventually drop off the frontpage. I really think the shared viewpoint on thread-owned data and how to move data from thread to thread is very alien to how thread safety is handled in practice. In practice, a lot of data is shared; I would argue a large fraction of *all* mutable data is shared, and access is protected using synchronization in the owning class. That is, objects, not threads, are the owners and overseers of mutable data. Whatever solution for thread safety we arrive at should account for this.
Really, all shared is about is preventing common bugs related to multithreading by segregating data shared across threads and making operations which aren't guaranteed to be thread-safe illegal (the main benefit then being that you know that everything else then isn't shared across threads). The language doesn't do that entirely correctly at the moment (e.g. copying and assigning shared data is currently legal in spite of not being thread-safe) but ultimately, when you use shared, what you're doing is pretty much exactly what you'd do in C/C++ except that the compiler prevents many bugs with regards to data shared across threads, and because of the extra type information in place to enable the restrictions to give you that protection, you're often forced to cast away shared in trusted code to then do what you would have done in C++. But all of the atomics and mutexes and whatnot are the same as what you'd be doing in C++. The basic design patterns are the same. It's the fact that you you're getting compiler errors when you do something that's not thread-safe that's different, and it's the fact that you have to cast away shared after locking a mutex that's different (since C++ doesn't have shared). But aside from that, what you're doing is the same. The fact that the compiler prevents you from doing basic stuff while the variable is shared and requires casting (unless you use atomics) frequently causes folks to get mad at shared and misunderstand it, but that's really the compiler preventing you from shooting yourself in the foot. Once you understand what's going on, it's really just the same as C++ but with some extra casts. So, shared is very low level ultimately. It's about how the data is stored and accessed, not about how higher level constructs are written. We can then build other, higher level mechanisms on top of that - such as std.concurrency - but at the language level, whether you're passing data across threads or having multiple threads access data on the same thread isn't really part of the design. The language is just concerned about the fact that the data is shared. The only higher level threading mechanism that the language has is synchronized, and all that really is is a built-in mutex, which in conjunction whith synchronized classes (if they're ever fully implemented) would allow for the implicit casting away of the top level of shared inside of member functions, but that's really as far as it goes, and without some kind of ownership model in the language, it's _very_ hard to go much farther than that - either with implicitly removing shared in code or with having passing ownership across threads work without explicit casts. Certainly, I think that any attempt to move shared away from being a low level construct indicating simply that the data is shared across threads is mis-guided. It's what other constructs for sharing data across threads are built upon. So, if someone wants to push for some new mechanism that improves sharing data across threads (rather than passing it across threads) that's built on top of shared, that's fine. If someone wants to push for some new mechanism that improves sharing data by passing it across threads, that's fine. But shared is a low level construct and should stay that way. It's the base for everything else. And to be honest, I don't really expect it to be reasonable to add much in the way of higher level mechanisms for sharing data to D at the language level - not with it being a systems language. Most stuff would require restricting it further and/or introducing a serious ownership model to it, which we really don't want to do. Even synchronized classes are pretty questionable, because they can only safely remove the outer layer of shared, which is arguably better than nothing, but it doesn't get you very far. Without a full-blown ownership model, we're pretty much reduced to manual casting when mutexes or synchronized are used, and the language really isn't going to be much help. If someone can come up with some bright ideas, great. All the more power to them. Maybe we can improve the situation. But again, they should be built on top of shared as a low level mechanism, not try to change what shared is. - Jonathan M Davis
Oct 30 2018
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis 
wrote:
 But shared is a low level construct and should stay that way. 
 It's the base for everything else.
This is simply not correct. People didn't stop having a need for threading while D was figuring out how to unbreak shared for ten years and counting. Fact is, the base for everything else is __gshared and synchronized, and it will *continue* to be __gshared and synchronized unless __gshared is removed and the Thread constructor is changed to only take shared delegates. Why not do something useful with synchronized instead?
 And to be honest, I don't really expect it to be reasonable to 
 add much in the way of higher level mechanisms for sharing data 
 to D at the language level - not with it being a systems 
 language. Most stuff would require restricting it further 
 and/or introducing a serious ownership model to it, which we 
 really don't want to do. Even synchronized classes are pretty 
 questionable, because they can only safely remove the outer 
 layer of shared, which is arguably better than nothing, but it 
 doesn't get you very far. Without a full-blown ownership model, 
 we're pretty much reduced to manual casting when mutexes or 
 synchronized are used, and the language really isn't going to 
 be much help. If someone can come up with some bright ideas, 
 great. All the more power to them. Maybe we can improve the 
 situation. But again, they should be built on top of shared as 
 a low level mechanism, not try to change what shared is.

 - Jonathan M Davis
Sorry, but can you please actually comment on my proposal in concrete terms rather than handwave about how we're reduced to manual casting (which is not necessary for concurrency at the moment, so I'm not sure why you think it's unavoidable), and how the language surely cannot be of much help without even commenting on a proposal for how it *could* actually be of help?
Oct 31 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, October 31, 2018 1:00:32 AM MDT FeepingCreature via 
Digitalmars-d wrote:
 On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis

 wrote:
 But shared is a low level construct and should stay that way.
 It's the base for everything else.
This is simply not correct. People didn't stop having a need for threading while D was figuring out how to unbreak shared for ten years and counting. Fact is, the base for everything else is __gshared and synchronized, and it will *continue* to be __gshared and synchronized unless __gshared is removed and the Thread constructor is changed to only take shared delegates. Why not do something useful with synchronized instead?
__gshared is intended for interacting with C globals and that's it. If it's used for anything else, it's not being used for its intended purpose. Certainly, using it for D objects is just begging for bugs. __gshared should _never_ be used for D objects. I know that some folks do, and sometimes folks get away with it, but it's just asking for trouble, and there's no guarantee that such code will continue to work, because it's violating compiler guarantees when it does so. __gshared is designed solely as a backdoor for dealing with C and that is it. On the whole, shared really is _not_ broken and _does_ work. Yes, it has some rough edges that need fixing, and not everything in core.sync has been updated to use it properly, necessitating more casts than should be necessary when using those constructs than should be necessary, but shared as a whole is sound and always has been. The biggest problem with it is that most folks don't understand how to use it properly.
 And to be honest, I don't really expect it to be reasonable to
 add much in the way of higher level mechanisms for sharing data
 to D at the language level - not with it being a systems
 language. Most stuff would require restricting it further
 and/or introducing a serious ownership model to it, which we
 really don't want to do. Even synchronized classes are pretty
 questionable, because they can only safely remove the outer
 layer of shared, which is arguably better than nothing, but it
 doesn't get you very far. Without a full-blown ownership model,
 we're pretty much reduced to manual casting when mutexes or
 synchronized are used, and the language really isn't going to
 be much help. If someone can come up with some bright ideas,
 great. All the more power to them. Maybe we can improve the
 situation. But again, they should be built on top of shared as
 a low level mechanism, not try to change what shared is.
Sorry, but can you please actually comment on my proposal in concrete terms rather than handwave about how we're reduced to manual casting (which is not necessary for concurrency at the moment, so I'm not sure why you think it's unavoidable), and how the language surely cannot be of much help without even commenting on a proposal for how it *could* actually be of help?
Manual casting is required if you're actually using shared with mutexes or synchronized. If you're using __gshared, it's not, but that's only because you're lying to the compiler about whether the variables are shared are not, which means that you're circumventing the type system, and you're risking subtle bugs in your programs, because the guarantees that the type system is suppose make aren't actually in place. Your synchronized field proposal is basically just a variation of synchronized classes with the extra caveat that somehow mutable references to the member variables can't escape, and for some reason, not everything in the class has to be synchronized. It's like you're trying to have synchronized classes without talking about shared. But you can't take shared out of the mix. Per the type system, anything that isn't shared is considered thread-local by the compiler, and trying to do anything like this without shared would be a serious violation of the type system and the compiler's guarantees. Instead of talking about safely casting away the outer layer of shared inside the class, you're talking about preventing escaping. synchronized classes as described in TDPL already guaranteed that the outer layer can't escape, since it lives directly in the class. What they can't guarantee is that the rest can't escape, and the language doesn't provide any way for such a guarantee. scope (even with DIP 1000) isn't transitive, so it doesn't really do the trick. So, I don't know how we could guarantee that _nothing_ escapes, though if nothing escapes, then all levels of shared could be implicitly cast away instead of just the outer layer, which would be nice. Also, even if we _could_ somehow prevent anything from escaping, I would point out that mutability has nothing to do with this (or at least constness doesn't), because reading has many of the same threading issues as writing. In order to avoid threading issues, the data must actually be immutable. So, allowing const references to escape would still violate thread-safety. Access to data must be synchronized unless it is immutable - whether we're talking about reading or writing is irrelevant. synchronized classes work (or would work if fully implemented) by guaranteeing that a single level can't escape - the one level that it _can_ guarantee can't escape - and thus are able to properly synchronize that level. If scope were transitive, then maybe we could do better, but since it isn't (and Walter insists that it can't reasonbly be transitive), I'm not sure that it's really reasonable to prevent escaping transitively. In any case, your proposal is just a variation of synchronized classes. So, while it may state it in a new way, the basic concept really isn't anything new. And my point still stands that anything that we do with regards to thread safety must be built on top of shared, because that is the building block of sharing data across threads in D. Whether you like the specifics of shared or not, the idea that the compiler can rely on the assumption that data is _not_ shared across threads unless it's marked as such is a key feature of D. And attempting to use __gshared to get around it is just shooting yourself in the foot. - Jonathan M Davis
Oct 31 2018
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 07:57:59 UTC, Jonathan M Davis 
wrote:
 On Wednesday, October 31, 2018 1:00:32 AM MDT FeepingCreature 
 via Digitalmars-d wrote:
 On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis

 wrote:
 But shared is a low level construct and should stay that 
 way. It's the base for everything else.
This is simply not correct. People didn't stop having a need for threading while D was figuring out how to unbreak shared for ten years and counting. Fact is, the base for everything else is __gshared and synchronized, and it will *continue* to be __gshared and synchronized unless __gshared is removed and the Thread constructor is changed to only take shared delegates. Why not do something useful with synchronized instead?
__gshared is intended for interacting with C globals and that's it. If it's used for anything else, it's not being used for its intended purpose. Certainly, using it for D objects is just begging for bugs. __gshared should _never_ be used for D objects. I know that some folks do, and sometimes folks get away with it, but it's just asking for trouble, and there's no guarantee that such code will continue to work, because it's violating compiler guarantees when it does so. __gshared is designed solely as a backdoor for dealing with C and that is it. On the whole, shared really is _not_ broken and _does_ work. Yes, it has some rough edges that need fixing, and not everything in core.sync has been updated to use it properly, necessitating more casts than should be necessary when using those constructs than should be necessary, but shared as a whole is sound and always has been. The biggest problem with it is that most folks don't understand how to use it properly.
 And to be honest, I don't really expect it to be reasonable 
 to add much in the way of higher level mechanisms for 
 sharing data to D at the language level - not with it being 
 a systems language. Most stuff would require restricting it 
 further and/or introducing a serious ownership model to it, 
 which we really don't want to do. Even synchronized classes 
 are pretty questionable, because they can only safely remove 
 the outer layer of shared, which is arguably better than 
 nothing, but it doesn't get you very far. Without a 
 full-blown ownership model, we're pretty much reduced to 
 manual casting when mutexes or synchronized are used, and 
 the language really isn't going to be much help. If someone 
 can come up with some bright ideas, great. All the more 
 power to them. Maybe we can improve the situation. But 
 again, they should be built on top of shared as a low level 
 mechanism, not try to change what shared is.
Sorry, but can you please actually comment on my proposal in concrete terms rather than handwave about how we're reduced to manual casting (which is not necessary for concurrency at the moment, so I'm not sure why you think it's unavoidable), and how the language surely cannot be of much help without even commenting on a proposal for how it *could* actually be of help?
Manual casting is required if you're actually using shared with mutexes or synchronized. If you're using __gshared, it's not, but that's only because you're lying to the compiler about whether the variables are shared are not, which means that you're circumventing the type system, and you're risking subtle bugs in your programs, because the guarantees that the type system is suppose make aren't actually in place.
This does not match what I'm actually seeing in practice. If this is how it is, then this admonishment should be written in the D style guide in big, bolded letters. Generally we try to write cast-free code if possible, so casting away shared feels wrong. That's not really an argument, of course, but to me semantic casting indicates "no compiler, you are mistaken about the properties of this data." The compiler should never *require* casting.
 Your synchronized field proposal is basically just a variation 
 of synchronized classes with the extra caveat that somehow 
 mutable references to the member variables can't escape, and 
 for some reason, not everything in the class has to be 
 synchronized.
Everything in the class has to be synchronized. I'm not sure where I said differently.
 It's like you're trying to have synchronized classes without 
 talking about shared. But you can't take shared out of the mix. 
 Per the type system, anything that isn't shared is considered 
 thread-local by the compiler, and trying to do anything like 
 this without shared would be a serious violation of the type 
 system and the compiler's guarantees.
Again, imo this belongs in either the spec or the DStyle guide in big bold letters.
 Instead of talking about safely casting away the outer layer of 
 shared inside the class, you're talking about preventing 
 escaping. synchronized classes as described in TDPL already 
 guaranteed that the outer layer can't escape, since it lives 
 directly in the class. What they can't guarantee is that the 
 rest can't escape, and the language doesn't provide any way for 
 such a guarantee.
Not sure what you mean by "outer layer" here or this entire paragraph.
 scope (even with DIP 1000) isn't transitive, so it doesn't 
 really do the trick. So, I don't know how we could guarantee 
 that _nothing_ escapes, though if nothing escapes, then all 
 levels of shared could be implicitly cast away instead of just 
 the outer layer, which would be nice.

 Also, even if we _could_ somehow prevent anything from 
 escaping, I would point out that mutability has nothing to do 
 with this (or at least constness doesn't), because reading has 
 many of the same threading issues as writing. In order to avoid 
 threading issues, the data must actually be immutable. So, 
 allowing const references to escape would still violate 
 thread-safety. Access to data must be synchronized unless it is 
 immutable - whether we're talking about reading or writing is 
 irrelevant.
Indeed, const is useless for threading, which is why my proposal explicitly only notes immutable data as implicitly synchronized. We're on the same page.
 synchronized classes work (or would work if fully implemented) 
 by guaranteeing that a single level can't escape - the one 
 level that it _can_ guarantee can't escape - and thus are able 
 to properly synchronize that level. If scope were transitive, 
 then maybe we could do better, but since it isn't (and Walter 
 insists that it can't reasonbly be transitive), I'm not sure 
 that it's really reasonable to prevent escaping transitively.
The point of the proposal is not precisely to prevent escaping, but to prevent the escape of *data structures/references that offer the ability to mutate data unprotected.* Remember that in this model, the object is the unit of data ownership. So the two things that are allowed to escape are immutable data (and non-reference copied data), and synchronized classes, because they can guarantee that they're taking care of protecting their data.
 In any case, your proposal is just a variation of synchronized 
 classes. So, while it may state it in a new way, the basic 
 concept really isn't anything new.
I didn't say it was! I said it was *useful* and *straightforward*.
 And my point still stands that anything that we do with regards 
 to thread safety must be built on top of shared, because that 
 is the building block of sharing data across threads in D. 
 Whether you like the specifics of shared or not, the idea that 
 the compiler can rely on the assumption that data is _not_ 
 shared across threads unless it's marked as such is a key 
 feature of D. And attempting to use __gshared to get around it 
 is just shooting yourself in the foot.

 - Jonathan M Davis
And yet, there's no way to even *state* that the Thread constructor must not take a delegate that can access unshared data, and none of the current proposals list one. Should we demand that only shared data can be passed to new threads? That's going to limit their usefulness. With synchronized, we can deprecate the delegate constructor, then pass the thread data as synchronized objects to a pure function, thereby actually guaranteeing thread safety.
Oct 31 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, October 31, 2018 2:08:58 AM MDT FeepingCreature via 
Digitalmars-d wrote:
 On Wednesday, 31 October 2018 at 07:57:59 UTC, Jonathan M Davis
 Manual casting is required if you're actually using shared with
 mutexes or synchronized. If you're using __gshared, it's not,
 but that's only because you're lying to the compiler about
 whether the variables are shared are not, which means that
 you're circumventing the type system, and you're risking subtle
 bugs in your programs, because the guarantees that the type
 system is suppose make aren't actually in place.
This does not match what I'm actually seeing in practice. If this is how it is, then this admonishment should be written in the D style guide in big, bolded letters.
Probably. There are places in the documentation that talk about it, but it's almost certainly not clear enough given how many folks keep using it. Certainly, far too many folks run to __gshared when they get annoyed with shared. In general, we've done a terrible job messaging how shared works, in part because it's been 95% complete for ages, and it's never been a priority. In general, what shared must do (and _mostly_ does) is guarantee that no operations that are not thread-safe are illegal. Earlier on, there was a larger push to find ways to get the compiler to then do stuff for you to make stuff thread-safe (such as introducing write barriers), and Andrei tends to prefer that approach, but that approach is hard to do (if possible at all), and pieces that we _can_ do (like write barriers) tend to be inefficient. Walter prefers the approach of just making more operations illegal (which unfortunately, then tends to require more casts and more trusted code). So, some stuff has been left in limbo that really shouldn't have been left in limbo, and so we have shared - which works - but where some operations aren't actually thread-safe but are still legal (hence why I say it's 95% complete). And that still needs to be sorted out. And since it hasn't been a priority, and it's perpetually mostly finished, you don't tend to get much in the way of stuff like documentation write-ups about it. That really should be fixed.
 Generally we try to write cast-free code if possible, so casting
 away shared feels wrong. That's not really an argument, of
 course, but to me semantic casting indicates "no compiler, you
 are mistaken about the properties of this data." The compiler
 should never *require* casting.
In the case of shared, it does require casting, and there really isn't a good way around it. The two major exceptions are atomics and synchronized classes (though those aren't fully implemented). synchronized classes still do the casting; they're just able to cast away shared on the outer layer implicitly. It would be fantastic to be able to do it implicitly in more cases, but it can only be done implicitly in cases where the compiler can actually guarantee that it's thread-safe, and that's really, really hard to do. synchronized classes are the best proposal that we've had for a mechanism for doing so. More would likely be possible with a more complicated ownership model, but without that, the compiler simply doesn't have enough information. It needs to know stuff like that a mutex is associated with a particular variable or set of variables, that it's definitely locked in a section of code, and that it's absolutely impossible that there are any other references to that data which are not currently protected by that mutex. The biggest obstacle there is probably the bit about there being no references not currently protected. Without a real ownership model, that just doesn't work. Maybe someone will come up with a bright idea, but everything I've seen at best has subtle holes in it, and ultimately, it comes down to basically writing the same kind of code that you'd write in C/C++ except that you need a few extra casts within the sections of code where the shared variables are protected by a locked mutex.
 Your synchronized field proposal is basically just a variation
 of synchronized classes with the extra caveat that somehow
 mutable references to the member variables can't escape, and
 for some reason, not everything in the class has to be
 synchronized.
Everything in the class has to be synchronized. I'm not sure where I said differently.
You were talking about marking individual fields and variables as synchronized rather than simply marking the entire class as synchronized.
 It's like you're trying to have synchronized classes without
 talking about shared. But you can't take shared out of the mix.
 Per the type system, anything that isn't shared is considered
 thread-local by the compiler, and trying to do anything like
 this without shared would be a serious violation of the type
 system and the compiler's guarantees.
Again, imo this belongs in either the spec or the DStyle guide in big bold letters.
 Instead of talking about safely casting away the outer layer of
 shared inside the class, you're talking about preventing
 escaping. synchronized classes as described in TDPL already
 guaranteed that the outer layer can't escape, since it lives
 directly in the class. What they can't guarantee is that the
 rest can't escape, and the language doesn't provide any way for
 such a guarantee.
Not sure what you mean by "outer layer" here or this entire paragraph.
The part directly embedded in the class and which therefore cannot possibly escape. Unlike regular classes, synchronized classes do not allow direct access to their member variables - even from anything else in the same module - in order to be able to provide that guarantee. If you don't have your own copy of TDPL, I'd suggest reading the concurrency chapter which is available for free online, since it talks about synchronized classes: http://www.informit.com/articles/article.aspx?p=1609144 Unfortunately, they're only partially implemented. IIRC, restricting access to the member variables has been implemented, but beyond that, right now they're mostly just synchronized functions like in Java.
 And my point still stands that anything that we do with regards
 to thread safety must be built on top of shared, because that
 is the building block of sharing data across threads in D.
 Whether you like the specifics of shared or not, the idea that
 the compiler can rely on the assumption that data is _not_
 shared across threads unless it's marked as such is a key
 feature of D. And attempting to use __gshared to get around it
 is just shooting yourself in the foot.
And yet, there's no way to even *state* that the Thread constructor must not take a delegate that can access unshared data, and none of the current proposals list one. Should we demand that only shared data can be passed to new threads? That's going to limit their usefulness. With synchronized, we can deprecate the delegate constructor, then pass the thread data as synchronized objects to a pure function, thereby actually guaranteeing thread safety.
Well, in principle if you're passing data across threads, it should be shared while it's passed across threads, and the type system requires that. That's why you have to cast to immutable when passing reference types with std.concurrency (shared would work too except for an issue with std.variant's implementation). You can cast to thread-local again on the other side, but to get it across, it needs to be typed as shared / immutable, because that's what it really is at that point. And what you're doing is inherently system, hence why the casting is required. It's up to the programmer to ensure that no references to the data remain on the original thread. If we had a full-own ownership model in the type system, then we might be able to avoid that, but we don't so we can't. Thread is a bit different, but the concept is basically the same. Conceptually, it's being passed off to another thread, so it really should be shared or immutable, but C doesn't actually _have_ shared (hence __gshared), so everything involved _has_ to be trusted internally. Looking at Thread's constructor, having it take a delegate probably isn't a problem, but what _is_ a problem is the fact that it's safe. I don't know how we can reasonably fix that without breaking code, but it's definitely a problem. All data passed through that delegate must either be shared or it must be the only reference to that data so that when it ends up on that new thread, and it's thread-local, no other thread has reference to it, and it really is properly thread-local. That constructor can't guarantee that. It's up to the programmer to guarantee that. So, it needs to be system. If we could somehow require that the delegate only accepted shared data, then we could reasonably make the constructor safe, but since AFAIK, that's not possible, it needs to be system. - Jonathan M Davis
Oct 31 2018
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 09:04:43 UTC, Jonathan M Davis 
wrote:
 Everything in the class has to be synchronized. I'm not sure 
 where I said differently.
You were talking about marking individual fields and variables as synchronized rather than simply marking the entire class as synchronized.
Sorry- I see your point. There is indeed no reason for non-synchronized fields. Strike that from the proposal. I'm not sure why I had it in there.
 The part directly embedded in the class and which therefore 
 cannot possibly escape. Unlike regular classes, synchronized 
 classes do not allow direct access to their member variables - 
 even from anything else in the same module - in order to be 
 able to provide that guarantee. If you don't have your own copy 
 of TDPL, I'd suggest reading the concurrency chapter which is 
 available for free online, since it talks about synchronized 
 classes:

 http://www.informit.com/articles/article.aspx?p=1609144
Wow, I straight up hadn't known that synchronized classes were a thing. I think it's because synchronized is not listed as an attribute on the Attributes page, but rather as a separate syntax on the Class page.
 And yet, there's no way to even *state* that the Thread 
 constructor must not take a delegate that can access unshared 
 data, and none of the current proposals list one. Should we 
 demand that only shared data can be passed to new threads? 
 That's going to limit their usefulness. With synchronized, we 
 can deprecate the delegate constructor, then pass the thread 
 data as synchronized objects to a pure function, thereby 
 actually guaranteeing thread safety.
[ snip std.concurrency which we're not using ] Thread is a bit different, but the concept is basically the same. Conceptually, it's being passed off to another thread, so it really should be shared or immutable, but C doesn't actually _have_ shared (hence __gshared), so everything involved _has_ to be trusted internally. Looking at Thread's constructor, having it take a delegate probably isn't a problem, but what _is_ a problem is the fact that it's safe. I don't know how we can reasonably fix that without breaking code, but it's definitely a problem. All data passed through that delegate must either be shared or it must be the only reference to that data so that when it ends up on that new thread, and it's thread-local, no other thread has reference to it, and it really is properly thread-local. That constructor can't guarantee that. It's up to the programmer to guarantee that. So, it needs to be system. If we could somehow require that the delegate only accepted shared data, then we could reasonably make the constructor safe, but since AFAIK, that's not possible, it needs to be system. - Jonathan M Davis
There doesn't seem to be a trait to detect a synchronized class. Having learnt that synchronized classes are actually mostly implemented (how did I not know this! Though I still think the fact that they can be impure is an issue, but that's statically checkable at least), it seems like it should at least be possible to require that the Thread runner only accesses shared data, by limiting it to take a pure function that only has synchronized-class parameters. Well, it would be, if there was a trait for synchronized classes, but that seems like it'd be much easier to add.
Oct 31 2018
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 31 October 2018 at 09:36:22 UTC, FeepingCreature 
wrote:

 it seems like it should at least be possible to require that 
 the Thread runner only accesses shared data, by limiting it to 
 take a pure function that only has synchronized-class 
 parameters. Well, it would be, if there was a trait for 
 synchronized classes, but that seems like it'd be much easier 
 to add.
And so you're jumping from your "simple oop cases" to "let's infect the runtime with a narrow-minded view of thread-safety and inflict synchronization on everyone using threads"? And what if anyone, deity forbid, wants to pass data that is not actually mutex-protected, because they happen to know that there's a lot more to synchronizing data access than that?
Oct 31 2018
parent FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 09:41:58 UTC, Stanislav Blinov 
wrote:
 On Wednesday, 31 October 2018 at 09:36:22 UTC, FeepingCreature 
 wrote:

 it seems like it should at least be possible to require that 
 the Thread runner only accesses shared data, by limiting it to 
 take a pure function that only has synchronized-class 
 parameters. Well, it would be, if there was a trait for 
 synchronized classes, but that seems like it'd be much easier 
 to add.
And so you're jumping from your "simple oop cases" to "let's infect the runtime with a narrow-minded view of thread-safety and inflict synchronization on everyone using threads"? And what if anyone, deity forbid, wants to pass data that is not actually mutex-protected, because they happen to know that there's a lot more to synchronizing data access than that?
Sorry, allow me to restate this. It seems like it should be possible that we, in our utility libraries, write a Thread wrapper that actually enforces that <see above>.
Oct 31 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, October 31, 2018 3:36:22 AM MDT FeepingCreature via 
Digitalmars-d wrote:
 There doesn't seem to be a trait to detect a synchronized class.
 Having learnt that synchronized classes are actually mostly
 implemented (how did I not know this! Though I still think the
 fact that they can be impure is an issue, but that's statically
 checkable at least),
synchronized classes are only partially implemented, and the key feature that they implicitly remove the outer layer of shared in their member functions is one of the things that isn't implemented. I don't know exactly what is and isn't implemented at the moment. I recall restrictions on accessing member variables from outside the class being implemented, but I don't think that anyone has really sat down and worked through what's left to do (though I could be wrong - I don't follow the list of dmd PR's closely). I don't see why the member functions being impure is a problem. Direct access to the member variables is protected, and I don't think that you're supposed to be allowed to pass a pointer to a member variable out of the class, but you can escape stuff that the member variables refer to, so I don't see why it matters particularly that they would have access to module-level variables. synchronized classes really only protect the stuff that sit directly inside the class itself, not what the class' member variables refer to. If they were to fully protect everything inside of it, then you couldn't really pass anything in or out of it. - Jonathan M Davis
Oct 31 2018
parent FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 31 October 2018 at 10:25:52 UTC, Jonathan M Davis 
wrote:
 synchronized classes really only protect the stuff that sit 
 directly inside the class itself, not what the class' member 
 variables refer to. If they were to fully protect everything 
 inside of it, then you couldn't really pass anything in or out 
 of it.

 - Jonathan M Davis
You could pass other synchronized objects, immutable data and data by value. In many project designs, that's really all you need.
Oct 31 2018
prev sibling parent FeepingCreature <feepingcreature gmail.com> writes:
Update to previous post:

synchronized class Class {
     void foo()  safe { }
}

void main() { auto object = new Class; object.foo(); }

This errors because object is not shared. Why is that a problem? 
It's not shared, but it is synchronized; there is no possibility 
of a threading issue here.

Analogously, I cannot do

synchronized class Class {
     private Class object;
     Class foo()  safe { return this.object; }
}

either, despite the fact that Class is again synchronized and 
thus should be able to be treated as shared implicitly.
Oct 31 2018