www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Shared

reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
I've followed the AGM a little and think, why not go with the 
"disable access to shared" approach, but instead of allowing it 
to be casted away, introduce a "locked scope" lock { } which need 
to be within a function and allow access to shared variables only 
within such blocks?
It's still up to the user to ensure that such locked blocks 
encapsulate the semantic blocks which need to be kept together to 
safely modify shared variables, but at least the compiler can 
make sure they are not modified outside locked blocks and that at 
any time only one such block is executed on any thread. And such 
blocks should be scarce so they can be examined carefully like 
trusted and everything build upon is verifiable safe.
May 11 2019
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 11 May 2019 at 09:51:41 UTC, Dominikus Dittes 
Scherkl wrote:
 I've followed the AGM a little and think, why not go with the 
 "disable access to shared" approach, but instead of allowing it 
 to be casted away, introduce a "locked scope" lock { } which 
 need to be within a function and allow access to shared 
 variables only within such blocks?
 It's still up to the user to ensure that such locked blocks 
 encapsulate the semantic blocks which need to be kept together 
 to safely modify shared variables, but at least the compiler 
 can make sure they are not modified outside locked blocks and 
 that at any time only one such block is executed on any thread. 
 And such blocks should be scarce so they can be examined 
 carefully like trusted and everything build upon is verifiable 
 safe.
And leave shared in the useless state that it currently has? + introducing a new syncronized keyword? (locked)? doesn't sound too compelling.
May 11 2019
parent Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Saturday, 11 May 2019 at 12:08:00 UTC, Stefan Koch wrote:
 On Saturday, 11 May 2019 at 09:51:41 UTC, Dominikus Dittes 
 Scherkl wrote:
 I've followed the AGM a little and think, why not go with the 
 "disable access to shared" approach, but instead of allowing 
 it to be casted away, introduce a "locked scope" lock { } 
 which need to be within a function and allow access to shared 
 variables only within such blocks?
 It's still up to the user to ensure that such locked blocks 
 encapsulate the semantic blocks which need to be kept together 
 to safely modify shared variables, but at least the compiler 
 can make sure they are not modified outside locked blocks and 
 that at any time only one such block is executed on any 
 thread. And such blocks should be scarce so they can be 
 examined carefully like trusted and everything build upon is 
 verifiable safe.
And leave shared in the useless state that it currently has?
No, as I said: use the "access disabled" (outside of locked blocks) approach, as several people advocated for.
 + introducing a new synchronized keyword? (locked)?
Need not be a keyword. It's also possible to use Lock() and Unlock() functions, but they need to be present in the same scope, so that the compiler can check, that what was locked really gets unlocked again close by (to keep it easily parsable). Or the block reuses the shared keyword, just to not burn another word for identifiers... This is to avoid to need a cast and trusted stuff. I think shared and trusted should not be conflated, so that they can use a different review process (e.g. someone specialized in looking for synchronization problems)
 doesn't sound too compelling.
May 11 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, May 11, 2019 3:51:41 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:
 I've followed the AGM a little and think, why not go with the
 "disable access to shared" approach, but instead of allowing it
 to be casted away, introduce a "locked scope" lock { } which need
 to be within a function and allow access to shared variables only
 within such blocks?
 It's still up to the user to ensure that such locked blocks
 encapsulate the semantic blocks which need to be kept together to
 safely modify shared variables, but at least the compiler can
 make sure they are not modified outside locked blocks and that at
 any time only one such block is executed on any thread. And such
 blocks should be scarce so they can be examined carefully like
 trusted and everything build upon is verifiable safe.
All that really would do is add a bit of extra syntax around locking a mutex and casting away shared. It doesn't really change anything with regards to what happens. The operations are the same, just wrapped. It's also arguably worse in that it implies that the compiler is actually making something safe for you, and it isn't. You're still doing the cast. It's still up to you to make sure that the concurrency aspects are sorted out correctly. It's just that what's actually happening is less obvious. I'd argue that we really don't want anything to just remove shared for us if the compiler can't actually guarantee that doing so is safe, which it can't do in this case. Also, mutexes aren't the only concurrency mechanism. You have to deal with stuff like condition variables and semaphores. And even with mutexes, the actual code flow may not be as simple as * lock mutex * cast away shared * do stuff as thread-local * make sure no thread-local references to the data exist * unlock mutex That's the basic case and what your suggestion targets, but if your code flow needs to be even slightly different, it won't work, whereas casting gives a lot more freedom. It might make sense to add such a helper on top of casting, but casting is still needed. Regardless, it's clear that Walter and Andrei want to fully look at the issue - with concurrency and memory model experts - to make sure that they're doing the right thing rather than simply doing something now that might be wrong. It's almost certainly the case that part of whatever we get will involve disabling reading from and writing to shared objects, and casting is pretty much going to have to be involved with that, but the details still need to be worked out, and working out all of those details (like the memory model) could very well affect what we get in unexpected ways. It may ultimately make sense to add helper stuff that hides casts, but we really need to figure out the details of how shared really works before looking at adding more user-friendly helpers on top of it. - Jonathan M Davis
May 11 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis wrote:


 All that really would do is add a bit of extra syntax around 
 locking a mutex and casting away shared.
No, it makes it possible to use mutex in safe functions without needing to cast and which the compiler CAN guarantee to really be safe.
 It doesn't really change anything with regards to what happens.
correct. It's a safe wrapper that actually make shared usefull in one way.
 Also, mutexes aren't the only concurrency mechanism. You have 
 to deal with stuff like condition variables and semaphores. And 
 even with mutexes, the actual code flow may not be as simple as

 * lock mutex
 * cast away shared
No, that's exactly NOT necessary anymore, because the compiler can deal with this in a safe and checked manner for this special case.
 * do stuff as thread-local
 * make sure no thread-local references to the data exist
 * unlock mutex
Of course. But that would still be possible if you are willing to cast and deal with trusted functions. But it would not be required anymore for the 99% of usecases where simple mutex would be enough.
 That's the basic case and what your suggestion targets, but if 
 your code flow needs to be even slightly different, it won't 
 work, whereas casting gives a lot more freedom. It might make 
 sense to add such a helper on top of casting, but casting is 
 still needed.
Yes and it would be far from my opinion to suggest otherwise. Casting is always possible and need be in a systems language. But its not safe, and shared should be possible to be used in at least one way without getting into unsafe territory, I think.
 It may ultimately make sense to add helper stuff that hides 
 casts, but we really need to figure out the details of how 
 shared really works before looking at adding more user-friendly 
 helpers on top of it.
And that I think is simple: forbid access without cast (outside locked blocks). Its the best (and most KISS) idea that has come up so far - and adding locked blocks seems logical to me, to allow one safe way to use shared.
May 13 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:
 On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis wrote:
 All that really would do is add a bit of extra syntax around
 locking a mutex and casting away shared.
No, it makes it possible to use mutex in safe functions without needing to cast and which the compiler CAN guarantee to really be safe.
Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time. And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code, thereby making the mutex useless. The type system has no concept of ownership and no concept of a mutex being associated with any particular object aside from synchronized classes (and those don't even currently require that the mutex always be used). And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist. So, with the construct you've proposed, there's no way for the compiler to guarantee that no other thread is accessing the data at the same time. All it guarantees is that a mutex has been locked, not that it's actually protecting the data. The only proposal that we've had thus far that is actually able to make enough guarantees to safely remove shared is the synchronized classes proposal in TDPL (which has never been implemented). The compiler would only be able to remove shared at all with TDPL synchronized classes, because it would guarantee that no references to the data directly in the class escaped the class, and even then, it could only safely remove the outer layer of shared, because anything that isn't directly in the class could still have references to it elsewhere. So, TDPL synchronized classes would actually be very limited in usefulness, because they simply can't guarantee enough to strip much of shared away, and they're only able to do that much because of how restrictive they are. Something as free-form as simply indicating that a mutex is locked in a block of code is nowhere near enough to guarantee that accessing any variables in that code is thread-safe. - Jonathan M Davis
May 13 2019
next sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl 
 via Digitalmars-d wrote:
 On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis 
 wrote:
 All that really would do is add a bit of extra syntax around 
 locking a mutex and casting away shared.
No, it makes it possible to use mutex in safe functions without needing to cast and which the compiler CAN guarantee to really be safe.
Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time.
But that piece of code is system, which explicitly allows you to shoot into your foot. If you want to stay safe, don't do that.
 And even if this were the only mechanism for removing shared, 
 you could easily use it with the same object and a completely 
 different mutex in another piece of code
Yes, the lock block need a list of vars that it allows to be modified lock(var1, var2, ...) { } two mutexes can only be executed at parallel if their parameter set is disjunct.
 And even if the compiler could check that all uses of a 
 particular variable were locked with a mutex, that still 
 wouldn't be enough, because other references to the same data 
 could exist.
ok, so we need in addition that a reference to a shared var need not be lived beyond the end of the locked block or be immutable. Bad, but seems necessary.
May 14 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, May 14, 2019 6:22:20 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:
 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl

 via Digitalmars-d wrote:
 On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis

 wrote:
 All that really would do is add a bit of extra syntax around
 locking a mutex and casting away shared.
No, it makes it possible to use mutex in safe functions without needing to cast and which the compiler CAN guarantee to really be safe.
Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time.
But that piece of code is system, which explicitly allows you to shoot into your foot. If you want to stay safe, don't do that.
The point is that if that code can legally exist, then the compiler simply cannot guarantee that removing shared from the object is thread-safe even with the locking mechanism you're proposing.
 And even if this were the only mechanism for removing shared,
 you could easily use it with the same object and a completely
 different mutex in another piece of code
Yes, the lock block need a list of vars that it allows to be modified lock(var1, var2, ...) { } two mutexes can only be executed at parallel if their parameter set is disjunct.
Sure, but another thread could be using a completely different mutex with one or more of those variables. So, even if just your locking mechanism is used, there's no guarantee that the same mutex is always used with the same set of variables, meaning that the compiler can't guarantee that the data is actually protected.
 And even if the compiler could check that all uses of a
 particular variable were locked with a mutex, that still
 wouldn't be enough, because other references to the same data
 could exist.
ok, so we need in addition that a reference to a shared var need not be lived beyond the end of the locked block or be immutable. Bad, but seems necessary.
A reference could already exist before your proposed locking mechanism was reached in the code. If the type is a class or pointer, then there could be other class references or pointers to the same data in safe code. And in system/ trusted code, the address of the object could have been taken to create a pointer to the object (and that could have been done in code for removed from the code that's using the lock with all of the code around the lock being safe). Heck, there could even be references to data within the object rather than to the object itself which are available elsewhere, meaning that part of the object is protected by the lock and part isn't. If any reference to any part of the data exists anywhere in the program, then it's possible for another thread to access the data at the same time that it's locked by the mechanism that you've proposed. In order for the compiler to be able to actually guarantee that it's safe to remove shared from an object, it has to be able to guarantee that there are no other references to any part of that object which exist in the program. That's why TDPL synchronized classes are so locked down. Without that, other references to the data could exist. And even with all of the restrictions that they have, the compiler would still only be able to remove the outer layer of shared - the layer directly in the class - not any more than that. With something as free-form as you're proposing, the compiler can't guarantee anything, let alone that it's thread-safe to completely remove shared from an object within that block. If D had ownership semantics baked into its type system, then we could probably do more, but as it is, the compiler is _very_ limited in its ability to know that no other references to any portion of an object exist. All it's dealing with are the types, not what owns them or what else could refer to them. Even scope is only able to do its job by restricting the operations that are allowed on a scope object, not by actually tracking an object's lifetime. So, it's incredibly difficult for the compiler to have enough information to know that an object is actually fully protected by a mutex when it's locked. And if the compiler can't guarantee that accessing the shared object is thread-safe within a particular piece of code, then it can't safely remove shared. For any proposal you might have with regards to how we might be able to have the compiler safely remove shared from an object for us, you're going to have to be able to prove that no other references to any piece of that object could possibly exist or that there's no way that any reference to any portion of that object could be accessed without the same mutex being locked at every point that it's accessed. And it wouldn't surprise me if someone else were able to point out why even that wasn't enough because of some detail I'm not thinking of at the moment. Having the compiler be able to prove that a piece of code is thread-safe such that shared can be safely removed automatically from anything is incredibly difficult. - Jonathan M Davis
May 14 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Tuesday, 14 May 2019 at 21:31:57 UTC, Jonathan M Davis wrote:
 On Tuesday, May 14, 2019 6:22:20 AM MDT Dominikus Dittes 
 Scherkl via Digitalmars-d wrote:
 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes 
 Scherkl

 via Digitalmars-d wrote:
 On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis

 wrote:
 All that really would do is add a bit of extra syntax 
 around locking a mutex and casting away shared.
No, it makes it possible to use mutex in safe functions without needing to cast and which the compiler CAN guarantee to really be safe.
Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time.
But that piece of code is system, which explicitly allows you to shoot into your foot. If you want to stay safe, don't do that.
The point is that if that code can legally exist, then the compiler simply cannot guarantee that removing shared from the object is thread-safe even with the locking mechanism you're proposing.
with system code you can always destroy safety assumptions of any other written code. This is why system code should be avoided where ever possible and the unavoidable remains need to be reviewed very carefully to not spoil the guarantees that are valid otherwise.
 And even if this were the only mechanism for removing 
 shared, you could easily use it with the same object and a 
 completely different mutex in another piece of code
Yes, the lock block need a list of vars that it allows to be modified lock(var1, var2, ...) { } two mutexes can only be executed at parallel if their parameter set is disjunct.
Sure, but another thread could be using a completely different mutex with one or more of those variables.
No, it can't. Disjunct means: It cannot be called unless all of the given variables are free (not locked by any other mutex).
 ok, so we need in addition that a reference to a shared var 
 need not be lived beyond the end of the locked block or be 
 immutable. Bad, but seems necessary.
A reference could already exist before your proposed locking mechanism was reached in the code. If the type is a class or pointer, then there could be other class references or pointers to the same data in safe code. And in system/ trusted code, the address of the object could have been taken to create a pointer to the object (and that could have been done in code for removed from the code that's using the lock with all of the code around the lock being safe). Heck, there could even be references to data within the object rather than to the object itself which are available elsewhere, meaning that part of the object is protected by the lock and part isn't. If any reference to any part of the data exists anywhere in the program, then it's possible for another thread to access the data at the same time that it's locked by the mechanism that you've proposed.
Ok, that whole reference stuff is always a problem. Why not simply forbid it? You can't reference shared variables (outside locked blocks), you can only copy them. We can later relax that rule if some safe ways to allow that are found. I can't see why that should hinder us to make the more practical usecases safe for now.
 In order for the compiler to be able to actually guarantee that 
 it's safe to remove shared from an object, it has to be able to 
 guarantee that there are no other references to any part of 
 that object which exist in the program. That's why TDPL 
 synchronized classes are so locked down. Without that, other 
 references to the data could exist. And even with all of the 
 restrictions that they have, the compiler would still only be 
 able to remove the outer layer of shared - the layer directly 
 in the class - not any more than that. With something as 
 free-form as you're proposing, the compiler can't guarantee 
 anything, let alone that it's thread-safe to completely remove 
 shared from an object within that block.

 If D had ownership semantics baked into its type system, then 
 we could probably do more, but as it is, the compiler is _very_ 
 limited in its ability to know that no other references to any 
 portion of an object exist.
If we forbid them (for now), the compiler is well able to.
 Even scope is only able to do its job by restricting the 
 operations that are allowed on a scope object, not by actually 
 tracking an object's lifetime.
Yes, and that's fine. It isn't necessary that everything is possible with an object, but it should at least be useful for SOME task.
 For any proposal you might have with regards to how we might be 
 able to have the compiler safely remove shared from an object 
 for us, you're going to have to be able to prove that no other 
 references to any piece of that object could possibly exist or 
 that there's no way that any reference to any portion of that 
 object could be accessed without the same mutex being locked at 
 every point that it's accessed.
Understood.
 And it wouldn't surprise me if someone else were able to point 
 out why even that wasn't enough because of some detail I'm not 
 thinking of at the moment. Having the compiler be able to prove 
 that a piece of code is thread-safe such that shared can be 
 safely removed automatically from anything is incredibly 
 difficult.
shared shouldn't be removed from an object, but it can only be modyfied if it is locked. Removing shared (with a cast) is system stuff and should be out of scope for any safety related proposal (including mine), because with system stuff you can destroy any kind of safety. If you don't remove shared, you can easily apply rules like forbid to take it's address or such. If you remove it, that makes it much harder (and isn't useful anyway). I still think my proposal could work (provide provable thread-safety for shared objects) in a limited but useful way (only mutex, no references), and should be relatively easy to implement. If you want more complex stuff, that's still possible in the same way it currently is: cast shared away together with all guarantees and verify manually that it works, just like in C++.
May 14 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 15, 2019 12:59:00 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:
 On Tuesday, 14 May 2019 at 21:31:57 UTC, Jonathan M Davis wrote:
 The point is that if that code can legally exist, then the
 compiler simply cannot guarantee that removing shared from the
 object is thread-safe even with the locking mechanism you're
 proposing.
with system code you can always destroy safety assumptions of any other written code. This is why system code should be avoided where ever possible and the unavoidable remains need to be reviewed very carefully to not spoil the guarantees that are valid otherwise.
safety and thread-safety are very different. system mechanisms such as casting are involved with thread-safety, meaning that system and trusted get involved, but what they mean and how they're verified are very different. For safety, if you want to mark something as trusted, you just need to look at that piece of code to verify that what it's doing is memory safe. You don't have to look at other safe parts of the program to verify that what it's doing is correct. On the other hand, with thread-safety, you have to look at _everywhere_ that a particular piece of shared data is accessed if you want to be able to guarantee that it's accessed in a thread-safe manner. You can't just assume that other code is doing the right thing and just verify that one piece of code that's using system mechanisms to interact with shared data. So, you can't rely on safe to tell you whether anything is thread-safe.
 And even if this were the only mechanism for removing
 shared, you could easily use it with the same object and a
 completely different mutex in another piece of code
Yes, the lock block need a list of vars that it allows to be modified lock(var1, var2, ...) { } two mutexes can only be executed at parallel if their parameter set is disjunct.
Sure, but another thread could be using a completely different mutex with one or more of those variables.
No, it can't. Disjunct means: It cannot be called unless all of the given variables are free (not locked by any other mutex).
So, you're proposing that something in the runtime keeps track of which variables are currently associated with a locked mutex in order to guarantee that no other lock block is able to access any of those variables at the same time? That would probably require adding a global lock used by the runtime when any code enters or exists a lock block. I'd be _very_ surprised if anything like that were deemed acceptable for D. And it still doesn't solve the problem of other references referring to any part of the objects referred to by those variables existing and potentially being used elsewhere. If you had lock(mutex, var1) { } and elsewhere lock(mutex, var2) { } when var1 and var2 were references to the same object or when var2 referred to a piece of data inside of var1, then the lock wouldn't be providing thread-safety. If you only had one mutex for the entire program, and casting away shared were illegal, then something like this could probaly work, but having one mutex for the entire program would clearly be unworkable - especially for a systems language - and since mutexes (let alone this particular use pattern for mutexes) aren't the only way to protect shared data when accessing it, requiring that this particular construct be used for accessing shared data wouldn't work anyway.
 ok, so we need in addition that a reference to a shared var
 need not be lived beyond the end of the locked block or be
 immutable. Bad, but seems necessary.
A reference could already exist before your proposed locking mechanism was reached in the code. If the type is a class or pointer, then there could be other class references or pointers to the same data in safe code. And in system/ trusted code, the address of the object could have been taken to create a pointer to the object (and that could have been done in code for removed from the code that's using the lock with all of the code around the lock being safe). Heck, there could even be references to data within the object rather than to the object itself which are available elsewhere, meaning that part of the object is protected by the lock and part isn't. If any reference to any part of the data exists anywhere in the program, then it's possible for another thread to access the data at the same time that it's locked by the mechanism that you've proposed.
Ok, that whole reference stuff is always a problem. Why not simply forbid it? You can't reference shared variables (outside locked blocks), you can only copy them. We can later relax that rule if some safe ways to allow that are found. I can't see why that should hinder us to make the more practical usecases safe for now.
You can't forbid references to the same data. All that would be required would be something like shared foo = new shared(Foo)(42); auto bar = foo; and you have two references to the same object without doing anything unsafe. You could also have stuff like shared baz = foo.getBaz(); resulting in a reference to some piece of data that the foo object contains (possibly simply a shared class object that it has a reference to as a member). In general, to be able to verify that no other references to data existed, we'd probably have to add some sort of ownership semantics to the language so that the compiler could know that nothing else can possibly have access to the data (as I understand it, Rust manages something along those lines, but they have a much more restrictive type system).
 And it wouldn't surprise me if someone else were able to point
 out why even that wasn't enough because of some detail I'm not
 thinking of at the moment. Having the compiler be able to prove
 that a piece of code is thread-safe such that shared can be
 safely removed automatically from anything is incredibly
 difficult.
shared shouldn't be removed from an object, but it can only be modyfied if it is locked. Removing shared (with a cast) is system stuff and should be out of scope for any safety related proposal (including mine), because with system stuff you can destroy any kind of safety. If you don't remove shared, you can easily apply rules like forbid to take it's address or such. If you remove it, that makes it much harder (and isn't useful anyway). I still think my proposal could work (provide provable thread-safety for shared objects) in a limited but useful way (only mutex, no references), and should be relatively easy to implement. If you want more complex stuff, that's still possible in the same way it currently is: cast shared away together with all guarantees and verify manually that it works, just like in C++.
As soon as casting away shared is legal (and I think that it has to be for many common thread-synchronization idioms to be used), any mechanism like you're suggesting isn't enough to guarantee that it's safe to remove shared even temporarily. Either way, the ability to get multiple references to the same data defeats what you're proposing, and I don't see how it would be possible to make it so that they're can't be multiple references to the data given D's type system. TDPL synchronized classes are only able to do it with the data that lives directly in the class, because they're a very restrictive construct. But even then, what the member variables refer to can't have shared removed, because references to the same data could have escaped the class or be passed into the class from elsewhere. And if something as restrictive as TDPL synchronized classes aren't able to restrict references sufficiently to be able to just outright remove shared from the class' member variables, there's no way that something as free-form as locking a mutex on a set of variables that aren't encapsulated in anything is going to be able to guarantee that other references to the same data don't exist. - Jonathan M Davis
May 15 2019
next sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Wednesday, 15 May 2019 at 09:33:19 UTC, Jonathan M Davis wrote:
 You can't forbid references to the same data. All that would be 
 required would be something like

 shared foo = new shared(Foo)(42);
 auto bar = foo;
Meep. This is either a compile error outside locked block (cannot read shared) or bar would be scope (it's lifetime will end after lock block is left).
 and you have two references to the same object without doing 
 anything unsafe. You could also have stuff like

 shared baz = foo.getBaz();
This will be forced to a copy of the returned value, not a reference.
May 16 2019
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 4:17:15 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:
 On Wednesday, 15 May 2019 at 09:33:19 UTC, Jonathan M Davis wrote:
 You can't forbid references to the same data. All that would be
 required would be something like

 shared foo = new shared(Foo)(42);
 auto bar = foo;
Meep. This is either a compile error outside locked block (cannot read shared) or bar would be scope (it's lifetime will end after lock block is left).
If you can't pass shared class references or pointers around like this, then it wouldn't even be possible to call shared functions on shared objects, because that's what happens when you call a function. And that really would make shared ridiculously hard to use. It would make it impossible to encapsulate thread synchronization primitives and force the programmer to handle it directly. Things like a shared class or struct that handled its own locking would be impossible, because you couldn't even call any of its methods. Code like shared foo = new shared(Foo)(42); auto bar = foo; needs to work as long as the type being passed around is a pointer or class reference, or shared simply won't be useable.
 and you have two references to the same object without doing
 anything unsafe. You could also have stuff like

 shared baz = foo.getBaz();
This will be forced to a copy of the returned value, not a reference.
No matter what the return type is, a line like that would either copy or move the return value (which one would depend on the implementation of getBaz and what exactly it was returning), but for a pointer or class reference, that just means that you have a pointer or class reference referring to the same thing as something that was in getBaz. So, copying is the normal semantics. And if you're trying to claim that it should be a deep copy, D doesn't even have that capability for classes - not without there being user-defind functions which do it, and the compiler won't know that that's what they do and couldn't use them automatically. So, that wouldn't work even if it were desirable - and it wouldn't be, because you really don't want to be deep copying classe normally. And you _really_ don't want to be doing it if shared is involved; that would require locking in a way that copying a simple pointer or reference shouldn't (at least not if we want to be able to actually do stuff like call shared member functions). Also, just in general, trying to prevent other references to the same data with shared actually makes no sense. If you don't have multiple references to the same data, then there's no point in it being shared in the first place, because you won't have multiple threads accessing the data. Even with TDPL synchronized classes, which prevent references to the member variables from escaping, you still have multiple references to the class itself - and the class itself is shared. Multiple references to the same data is a key part of data being shared. At best, you can avoid it by having only a single reference to some shared data with multiple references to a shared object which has the single reference to the other shared data (i.e. what you get with TDPL synchronized classes). And if that's what you're doing, then you might as well just have TDPL synchronized classes rather than trying to do something more free-from. - Jonathan M Davis
May 16 2019
prev sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Wednesday, 15 May 2019 at 09:33:19 UTC, Jonathan M Davis wrote:
 On Wednesday, May 15, 2019 12:59:00 AM MDT Dominikus Dittes 
 Scherkl via Digitalmars-d wrote:
 No, it can't. Disjunct means: It cannot be called unless all 
 of the given variables are free (not locked by any other 
 mutex).
So, you're proposing that something in the runtime keeps track of which variables are currently associated with a locked mutex in order to guarantee that no other lock block is able to access any of those variables at the same time? That would probably require adding a global lock used by the runtime when any code enters or exists a lock block. I'd be _very_ surprised if anything like that were deemed acceptable for D.
Ok, that can be a problem. But yes, this is what I suggest. But why the resistence? Nobody is forced to use mutexes, and if you don't that part of the runtime could (should!) be eliminated from the program.
 And it still doesn't solve the problem of other references 
 referring to any part of the objects referred to by those 
 variables existing and potentially being used elsewhere. If you 
 had

 lock(mutex, var1)
 {
 }

 and elsewhere

 lock(mutex, var2)
 {
 }

 when var1 and var2 were references to the same object
Forbidden. See last post.
 or when var2 referred to a piece of data inside of var1,
This would also be impossible. shared vars are no references.
 If you only had one mutex for the entire program, and casting 
 away shared were illegal, then something like this could 
 probably work,
Thank you, but why this restriction? There can be as many mutexes as you like. The runtime has only to ensure that any running locked block doesn't modify any of the vars in the other running locked blocks.
May 16 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 4:35:52 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:
 On Wednesday, 15 May 2019 at 09:33:19 UTC, Jonathan M Davis wrote:
 On Wednesday, May 15, 2019 12:59:00 AM MDT Dominikus Dittes

 Scherkl via Digitalmars-d wrote:
 No, it can't. Disjunct means: It cannot be called unless all
 of the given variables are free (not locked by any other
 mutex).
So, you're proposing that something in the runtime keeps track of which variables are currently associated with a locked mutex in order to guarantee that no other lock block is able to access any of those variables at the same time? That would probably require adding a global lock used by the runtime when any code enters or exists a lock block. I'd be _very_ surprised if anything like that were deemed acceptable for D.
Ok, that can be a problem. But yes, this is what I suggest. But why the resistence? Nobody is forced to use mutexes, and if you don't that part of the runtime could (should!) be eliminated from the program.
For the locking mechanism you're proposing to work, you basically have to disallow a lot of stuff that is required for other types of thread synchronization (e.g. casting away shared is needed), and having a global lock that's used whenever a mutex is used would be terrible for performance. In general, we want to avoid global locks, and it seems particularly bad to have a global mutex that needs to be used just to use a normal mutex.
 And it still doesn't solve the problem of other references
 referring to any part of the objects referred to by those
 variables existing and potentially being used elsewhere. If you
 had

 lock(mutex, var1)
 {
 }

 and elsewhere

 lock(mutex, var2)
 {
 }

 when var1 and var2 were references to the same object
Forbidden. See last post.
 or when var2 referred to a piece of data inside of var1,
This would also be impossible. shared vars are no references.
I discussed this in another post, but you _have_ to be able to have shared references or shared is basically useless.
 If you only had one mutex for the entire program, and casting
 away shared were illegal, then something like this could
 probably work,
Thank you, but why this restriction? There can be as many mutexes as you like. The runtime has only to ensure that any running locked block doesn't modify any of the vars in the other running locked blocks.
If it's at all possible to access any of the shared data that your lock blocks protect without using the lock block, then your lock block would not be enough for the compiler to know that it was safe to remove shared from the data within the lock block. And without a global mutex that's used to check that different locks aren't being used for the same variable, then different mutexes could be used for the same varible, defeating the protection - and even that is assuming that it's not possible to have any other references to the same data, which would be too restrictive to be useful. Having multiple references to the same data is a core concept of sharing data across threads. - Jonathan M Davis
May 16 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Thursday, 16 May 2019 at 11:15:13 UTC, Jonathan M Davis wrote:
 [...] that is assuming that it's not possible to have any other 
 references to the same data, which would be too restrictive to 
 be useful. Having multiple references to the same data is a 
 core concept of sharing data across threads.
Hm. I don't understand the necessity of this. If I have a shared symbol x, this same symbol can be used in different threads, no? I can't see why multiple threads need to have different references to the same data. They just can use the same reference.
May 16 2019
parent reply Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On Thursday, 16 May 2019 at 15:47:37 UTC, Dominikus Dittes 
Scherkl wrote:
 On Thursday, 16 May 2019 at 11:15:13 UTC, Jonathan M Davis 
 wrote:
 [...] that is assuming that it's not possible to have any 
 other references to the same data, which would be too 
 restrictive to be useful. Having multiple references to the 
 same data is a core concept of sharing data across threads.
Hm. I don't understand the necessity of this. If I have a shared symbol x, this same symbol can be used in different threads, no? I can't see why multiple threads need to have different references to the same data. They just can use the same reference.
Well, no. The reference is the part that gets copied around. It's the address of the shared object. When you do 'auto ref2 = ref1;', you're creating a new reference to an already existing object, but leaving the object itself unchanged. If more than one thread can access the same object, that means they each have a copy of the reference*. In the same way, when you call a function with a class argument, that function receives a copy of the reference (but not the object). Clearly then, the ability to create multiple references to the same object must be possible without breaking shared. -- Simen * barring __gshared tricks
May 16 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 11:11:19 AM MDT Simen Kjærås via Digitalmars-d 
wrote:
 On Thursday, 16 May 2019 at 15:47:37 UTC, Dominikus Dittes

 Scherkl wrote:
 On Thursday, 16 May 2019 at 11:15:13 UTC, Jonathan M Davis

 wrote:
 [...] that is assuming that it's not possible to have any
 other references to the same data, which would be too
 restrictive to be useful. Having multiple references to the
 same data is a core concept of sharing data across threads.
Hm. I don't understand the necessity of this. If I have a shared symbol x, this same symbol can be used in different threads, no? I can't see why multiple threads need to have different references to the same data. They just can use the same reference.
Well, no. The reference is the part that gets copied around. It's the address of the shared object. When you do 'auto ref2 = ref1;', you're creating a new reference to an already existing object, but leaving the object itself unchanged. If more than one thread can access the same object, that means they each have a copy of the reference*. In the same way, when you call a function with a class argument, that function receives a copy of the reference (but not the object). Clearly then, the ability to create multiple references to the same object must be possible without breaking shared.
Exactly. And since everything in D is thread-local by default, it's usually the case that a variable only exists on one thread. shared changes that if the variable is static, but otherwise, you're just passing around a pointer or reference to shared objects within thread-local objects. So, while you might have a pointer to shared data contained by a variable with the same name on multiple threads, because that variable is thread-local, it's different on different threads. e.g. struct S { int i; shared Foo* foo; } // Every thread will have it's own variable for this S s; - Jonathan M Davis
May 16 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Thursday, 16 May 2019 at 17:29:25 UTC, Jonathan M Davis wrote:
 On Thursday, May 16, 2019 11:11:19 AM MDT Simen Kjærås via 
 Digitalmars-d wrote:
 On Thursday, 16 May 2019 at 15:47:37 UTC, Dominikus Dittes

 Scherkl wrote:
 On Thursday, 16 May 2019 at 11:15:13 UTC, Jonathan M Davis

 wrote:
 [...] that is assuming that it's not possible to have any 
 other references to the same data, which would be too 
 restrictive to be useful. Having multiple references to the 
 same data is a core concept of sharing data across threads.
Hm. I don't understand the necessity of this. If I have a shared symbol x, this same symbol can be used in different threads, no? I can't see why multiple threads need to have different references to the same data. They just can use the same reference.
Well, no. The reference is the part that gets copied around. It's the address of the shared object. When you do 'auto ref2 = ref1;', you're creating a new reference to an already existing object, but leaving the object itself unchanged. If more than one thread can access the same object, that means they each have a copy of the reference*. In the same way, when you call a function with a class argument, that function receives a copy of the reference (but not the object). Clearly then, the ability to create multiple references to the same object must be possible without breaking shared.
Exactly. And since everything in D is thread-local by default, it's usually the case that a variable only exists on one thread. shared changes that if the variable is static, but otherwise, you're just passing around a pointer or reference to shared objects within thread-local objects. So, while you might have a pointer to shared data contained by a variable with the same name on multiple threads, because that variable is thread-local, it's different on different threads. e.g.
Ok, so then a shared object is in fact a specific memory segment. So something the GC is already aware of. This makes implementing the locking mechanism just more easy. We only need to prevent writing to that address, no matter how many references are pointing there. And that mechanism is well known in GC's - a write barrier. Only new thing is that no other locked block pointing in that address area can be executed. I just can't see what fundamental, all blocking problem you see with that approach. Seems pretty strait forward to me.
May 16 2019
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 16, 2019 at 07:53:56PM +0000, Dominikus Dittes Scherkl via
Digitalmars-d wrote:
 On Thursday, 16 May 2019 at 17:29:25 UTC, Jonathan M Davis wrote:
[...]
 Ok, so then a shared object is in fact a specific memory segment. So
 something the GC is already aware of. This makes implementing the
 locking mechanism just more easy. We only need to prevent writing to
 that address, no matter how many references are pointing there. And
 that mechanism is well known in GC's - a write barrier.
[...] D does not have write barriers, however. And it's unlikely to get one in the foreseeable future. T -- What do you get if you drop a piano down a mineshaft? A flat minor.
May 16 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Thursday, 16 May 2019 at 20:04:13 UTC, H. S. Teoh wrote:
 What do you get if you drop a piano down a mineshaft? A flat 
 minor.
I love you're signatures. You don't choose them at random, do you? At least I accept now, it's not as easy as one might think. Arimasen.
May 17 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 17, 2019 at 10:20:32AM +0000, Dominikus Dittes Scherkl via
Digitalmars-d wrote:
 On Thursday, 16 May 2019 at 20:04:13 UTC, H. S. Teoh wrote:
 What do you get if you drop a piano down a mineshaft? A flat minor.
I love you're signatures. You don't choose them at random, do you?
[...] It's chosen by a Perl script. It's supposed to be random, but the Perl RNG seems to have a mind of its own sometimes. :-D T -- Study gravitation, it's a field with a lot of potential.
May 17 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 2:04:13 PM MDT H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 16, 2019 at 07:53:56PM +0000, Dominikus Dittes Scherkl via 
Digitalmars-d wrote:
 On Thursday, 16 May 2019 at 17:29:25 UTC, Jonathan M Davis wrote:
[...]
 Ok, so then a shared object is in fact a specific memory segment. So
 something the GC is already aware of. This makes implementing the
 locking mechanism just more easy. We only need to prevent writing to
 that address, no matter how many references are pointing there. And
 that mechanism is well known in GC's - a write barrier.
[...] D does not have write barriers, however. And it's unlikely to get one in the foreseeable future.
That and not all memory comes from the GC. It's perfectly legitimate to use malloced memory with shared. The type system doesn't care where the memory came from. - Jonathan M Davis
May 16 2019
prev sibling parent Manu <turkeyman gmail.com> writes:
On Wed, May 15, 2019 at 2:34 AM Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 [... everything that jonathan said ...]
 ...there's no way that something as free-form
 as locking a mutex on a set of variables that aren't encapsulated in
 anything is going to be able to guarantee that other references to the same
 data don't exist.
This is hard-facts. We talked about this topic a lot at dconf. Here's where we got to. there's any future design where this isn't true. We should do this now. have enough context to know. start building some serious constructions above. Any such construction necessarily requires tight encapsulation to keep references contained and only issue leases appropriately. shared away (take a lease), the resulting pointer must be `scope`, otherwise it could be sequestered away, and all bets are off! Constructions are too varied, and too complex. But I think we know how to start to write some useful libraries that can safely share data I spent some time with Amaury talking through designs for `shared` shared pointers. We concluded that with DIP1000, there is enough language available to write a safe shared object as a library (and no mutex-es!). It is clear that any such design requires lease-tracking. It's not hard to imagine a shared shared-pointer with the same rules as Rust; "write access may only have 1 lease", "read access many have N leases", "read/write access are mutually exclusive". The container would have functions to capture a lease into some raii object, and the returned object must be DIP1000 `scope`.
May 15 2019
prev sibling parent reply Radu <void null.pt> writes:
On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl 
 via Digitalmars-d wrote:
 [...]
Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time. And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code, thereby making the mutex useless. The type system has no concept of ownership and no concept of a mutex being associated with any particular object aside from synchronized classes (and those don't even currently require that the mutex always be used). And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist. So, with the construct you've proposed, there's no way for the compiler to guarantee that no other thread is accessing the data at the same time. All it guarantees is that a mutex has been locked, not that it's actually protecting the data. [...]
I had this idea for some time, not sure I can best articulated now, but I think a workable solution for shared is closely linked with the dip1000 - scoped pointers. What I mean is that something like: void doSharedStuff(scope shared(Foo) foo) { } Will be able to safely lock/unlock foo and cast away shared'ness in the function's scope. The compiler can provide guarantees here that foo will not escape.
May 14 2019
next sibling parent Kagamin <spam here.lot> writes:
Sounds like what Atila did 
https://forum.dlang.org/post/vocustlarsadbankufjk forum.dlang.org
May 14 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via Digitalmars-d wrote:
 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl

 via Digitalmars-d wrote:
 [...]
Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time. And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code, thereby making the mutex useless. The type system has no concept of ownership and no concept of a mutex being associated with any particular object aside from synchronized classes (and those don't even currently require that the mutex always be used). And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist. So, with the construct you've proposed, there's no way for the compiler to guarantee that no other thread is accessing the data at the same time. All it guarantees is that a mutex has been locked, not that it's actually protecting the data. [...]
I had this idea for some time, not sure I can best articulated now, but I think a workable solution for shared is closely linked with the dip1000 - scoped pointers. What I mean is that something like: void doSharedStuff(scope shared(Foo) foo) { } Will be able to safely lock/unlock foo and cast away shared'ness in the function's scope. The compiler can provide guarantees here that foo will not escape.
Sure, it can guarantee that no reference will escape that function, but all that's required is that another reference to the same data exist elsewhere, and another thread could muck with the object while the mutex was locked. There's no question that helpers could be created which would help users avoid mistakes when casting when casting away shared, but the compiler can't actually make the guarantee that casting away shared is thread-safe. - Jonathan M Davis
May 14 2019
parent reply Radu <void null.pt> writes:
On Tuesday, 14 May 2019 at 21:02:10 UTC, Jonathan M Davis wrote:
 On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via Digitalmars-d 
 wrote:
 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes 
 Scherkl

 via Digitalmars-d wrote:
 [...]
Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time. And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code, thereby making the mutex useless. The type system has no concept of ownership and no concept of a mutex being associated with any particular object aside from synchronized classes (and those don't even currently require that the mutex always be used). And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist. So, with the construct you've proposed, there's no way for the compiler to guarantee that no other thread is accessing the data at the same time. All it guarantees is that a mutex has been locked, not that it's actually protecting the data. [...]
I had this idea for some time, not sure I can best articulated now, but I think a workable solution for shared is closely linked with the dip1000 - scoped pointers. What I mean is that something like: void doSharedStuff(scope shared(Foo) foo) { } Will be able to safely lock/unlock foo and cast away shared'ness in the function's scope. The compiler can provide guarantees here that foo will not escape.
Sure, it can guarantee that no reference will escape that function, but all that's required is that another reference to the same data exist elsewhere, and another thread could muck with the object while the mutex was locked. There's no question that helpers could be created which would help users avoid mistakes when casting when casting away shared, but the compiler can't actually make the guarantee that casting away shared is thread-safe. - Jonathan M Davis
My view is that the compiler could automatically insert locking logic (ala synchronized) when the shared parameters gets references inside the function, and also automatically cast away shared so for the function internals it would be like working with local non-shared data. Given that compiler inferes lifetime, it could safely elide locking if the parameter is passed to other functions that have the same signature (scope is explicit or inferred). The idea is to provide the tools that simplify concurrent programming, the compiler will need to insert all the checks automatically using the lifetime tracking.
May 15 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 15, 2019 1:56:12 AM MDT Radu via Digitalmars-d wrote:
 On Tuesday, 14 May 2019 at 21:02:10 UTC, Jonathan M Davis wrote:
 On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via Digitalmars-d

 wrote:
 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes
 Scherkl

 via Digitalmars-d wrote:
 [...]
Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time. And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code, thereby making the mutex useless. The type system has no concept of ownership and no concept of a mutex being associated with any particular object aside from synchronized classes (and those don't even currently require that the mutex always be used). And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist. So, with the construct you've proposed, there's no way for the compiler to guarantee that no other thread is accessing the data at the same time. All it guarantees is that a mutex has been locked, not that it's actually protecting the data. [...]
I had this idea for some time, not sure I can best articulated now, but I think a workable solution for shared is closely linked with the dip1000 - scoped pointers. What I mean is that something like: void doSharedStuff(scope shared(Foo) foo) { } Will be able to safely lock/unlock foo and cast away shared'ness in the function's scope. The compiler can provide guarantees here that foo will not escape.
Sure, it can guarantee that no reference will escape that function, but all that's required is that another reference to the same data exist elsewhere, and another thread could muck with the object while the mutex was locked. There's no question that helpers could be created which would help users avoid mistakes when casting when casting away shared, but the compiler can't actually make the guarantee that casting away shared is thread-safe. - Jonathan M Davis
My view is that the compiler could automatically insert locking logic (ala synchronized) when the shared parameters gets references inside the function, and also automatically cast away shared so for the function internals it would be like working with local non-shared data. Given that compiler inferes lifetime, it could safely elide locking if the parameter is passed to other functions that have the same signature (scope is explicit or inferred). The idea is to provide the tools that simplify concurrent programming, the compiler will need to insert all the checks automatically using the lifetime tracking.
The compiler does not have enough information to know which mutexes to use when even if we wanted it to insert locks. TDPL synchronized classes are a special case in that they would provide a way in the language to associate a specific lock with a specific set of variables in a way that the compiler could then guarantee that it's safe to remove the outer layer of shared within a synchronized function. Without a similar mechanism to associate a mutex with one or more variables and guarantee that that mutex is always locked when they're accessed, the compiler won't be able to even know what to lock, let alone that it's safe to remove shared within a particular section of code. And the issue with what you're proposing with scope is that unless the compiler can actually guarantee that no other references to the shared object exist which could be used to access the object at the same time, then the compiler cannot safely remove shared even temporarily. scope is just enough to guarantee that that particular function can't escape any references, not enough to guarantee that they don't exist. So, while features like scope can be used to make it easier to reason about the code and cast away shared in a way that your code is thread-safe, I fully expect that it's going to have to be up to the programmer to know when it's safe to remove shared, thus requiring a cast or some other trusted mechanism to temporarily remove shared. TDPL synchronized classes are the only proposal I've seen that would be able to guarantee thread-safety, and it can only do it for what's directly in the class, making TDPL synchronized classes arguably pretty useless (on top of the fact that it means requiring classes when most D code wouldn't normally use classes for something like this). - Jonathan M Davis
May 15 2019
parent reply Radu <void null.pt> writes:
On Wednesday, 15 May 2019 at 08:52:23 UTC, Jonathan M Davis wrote:
 On Wednesday, May 15, 2019 1:56:12 AM MDT Radu via 
 Digitalmars-d wrote:
 On Tuesday, 14 May 2019 at 21:02:10 UTC, Jonathan M Davis 
 wrote:
 On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via 
 Digitalmars-d

 wrote:
 [...]
Sure, it can guarantee that no reference will escape that function, but all that's required is that another reference to the same data exist elsewhere, and another thread could muck with the object while the mutex was locked. There's no question that helpers could be created which would help users avoid mistakes when casting when casting away shared, but the compiler can't actually make the guarantee that casting away shared is thread-safe. - Jonathan M Davis
My view is that the compiler could automatically insert locking logic (ala synchronized) when the shared parameters gets references inside the function, and also automatically cast away shared so for the function internals it would be like working with local non-shared data. Given that compiler inferes lifetime, it could safely elide locking if the parameter is passed to other functions that have the same signature (scope is explicit or inferred). The idea is to provide the tools that simplify concurrent programming, the compiler will need to insert all the checks automatically using the lifetime tracking.
The compiler does not have enough information to know which mutexes to use when even if we wanted it to insert locks. TDPL synchronized classes are a special case in that they would provide a way in the language to associate a specific lock with a specific set of variables in a way that the compiler could then guarantee that it's safe to remove the outer layer of shared within a synchronized function. Without a similar mechanism to associate a mutex with one or more variables and guarantee that that mutex is always locked when they're accessed, the compiler won't be able to even know what to lock, let alone that it's safe to remove shared within a particular section of code. And the issue with what you're proposing with scope is that unless the compiler can actually guarantee that no other references to the shared object exist which could be used to access the object at the same time, then the compiler cannot safely remove shared even temporarily. scope is just enough to guarantee that that particular function can't escape any references, not enough to guarantee that they don't exist. So, while features like scope can be used to make it easier to reason about the code and cast away shared in a way that your code is thread-safe, I fully expect that it's going to have to be up to the programmer to know when it's safe to remove shared, thus requiring a cast or some other trusted mechanism to temporarily remove shared. TDPL synchronized classes are the only proposal I've seen that would be able to guarantee thread-safety, and it can only do it for what's directly in the class, making TDPL synchronized classes arguably pretty useless (on top of the fact that it means requiring classes when most D code wouldn't normally use classes for something like this). - Jonathan M Davis
Locks for classes can be done using the class monitor field (like synchronized works today). For other types the compiler could probably use the typeinfo class for locking, that might be to much for some cases. But there may be optimization opportunities, like for example use atomics when the target supports it and the type is a primitive, or assignment for the reference/pointer, same for binary ops when dereferencing a pointer to a primitive. scope role is to ensure lock/unlock semantics and to help ellide superfluous locking when chaining the calls. I'm not proposing a ownership system here. The shared castaway should be performed only inside the locked scope, the difference is that the compiler will automatically do that for you. I think this should be enabled only for safe functions, and not for system.
May 15 2019
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 15, 2019 3:09:36 AM MDT Radu via Digitalmars-d wrote:
 On Wednesday, 15 May 2019 at 08:52:23 UTC, Jonathan M Davis wrote:
 On Wednesday, May 15, 2019 1:56:12 AM MDT Radu via

 Digitalmars-d wrote:
 On Tuesday, 14 May 2019 at 21:02:10 UTC, Jonathan M Davis

 wrote:
 On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via
 Digitalmars-d

 wrote:
 [...]
Sure, it can guarantee that no reference will escape that function, but all that's required is that another reference to the same data exist elsewhere, and another thread could muck with the object while the mutex was locked. There's no question that helpers could be created which would help users avoid mistakes when casting when casting away shared, but the compiler can't actually make the guarantee that casting away shared is thread-safe. - Jonathan M Davis
My view is that the compiler could automatically insert locking logic (ala synchronized) when the shared parameters gets references inside the function, and also automatically cast away shared so for the function internals it would be like working with local non-shared data. Given that compiler inferes lifetime, it could safely elide locking if the parameter is passed to other functions that have the same signature (scope is explicit or inferred). The idea is to provide the tools that simplify concurrent programming, the compiler will need to insert all the checks automatically using the lifetime tracking.
The compiler does not have enough information to know which mutexes to use when even if we wanted it to insert locks. TDPL synchronized classes are a special case in that they would provide a way in the language to associate a specific lock with a specific set of variables in a way that the compiler could then guarantee that it's safe to remove the outer layer of shared within a synchronized function. Without a similar mechanism to associate a mutex with one or more variables and guarantee that that mutex is always locked when they're accessed, the compiler won't be able to even know what to lock, let alone that it's safe to remove shared within a particular section of code. And the issue with what you're proposing with scope is that unless the compiler can actually guarantee that no other references to the shared object exist which could be used to access the object at the same time, then the compiler cannot safely remove shared even temporarily. scope is just enough to guarantee that that particular function can't escape any references, not enough to guarantee that they don't exist. So, while features like scope can be used to make it easier to reason about the code and cast away shared in a way that your code is thread-safe, I fully expect that it's going to have to be up to the programmer to know when it's safe to remove shared, thus requiring a cast or some other trusted mechanism to temporarily remove shared. TDPL synchronized classes are the only proposal I've seen that would be able to guarantee thread-safety, and it can only do it for what's directly in the class, making TDPL synchronized classes arguably pretty useless (on top of the fact that it means requiring classes when most D code wouldn't normally use classes for something like this). - Jonathan M Davis
Locks for classes can be done using the class monitor field (like synchronized works today). For other types the compiler could probably use the typeinfo class for locking, that might be to much for some cases. But there may be optimization opportunities, like for example use atomics when the target supports it and the type is a primitive, or assignment for the reference/pointer, same for binary ops when dereferencing a pointer to a primitive. scope role is to ensure lock/unlock semantics and to help ellide superfluous locking when chaining the calls. I'm not proposing a ownership system here. The shared castaway should be performed only inside the locked scope, the difference is that the compiler will automatically do that for you. I think this should be enabled only for safe functions, and not for system.
The problem is that unless the compiler can absolutely guarantee that no other references to the data exist, having the compiler automatically removing shared at all - including in safe code - can't be safely done. Without an ownership model, the compiler basically doesn't have the information that it needs to know that such references don't exist. Just because code is safe doesn't mean that it's thread-safe, and scope does not provide enough information for the compiler to know that it's thread-safe. Ultimately, you need a programmer to examine the code to verify the thread-safety - which arguably means that anything that removes shared needs to be system so that the programmer knows that they need to verify the code. - Jonathan M Davis
May 15 2019