www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Shared

reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
I've followed the AGM a little and think, why not go with the 
"disable access to shared" approach, but instead of allowing it 
to be casted away, introduce a "locked scope" lock { } which need 
to be within a function and allow access to shared variables only 
within such blocks?
It's still up to the user to ensure that such locked blocks 
encapsulate the semantic blocks which need to be kept together to 
safely modify shared variables, but at least the compiler can 
make sure they are not modified outside locked blocks and that at 
any time only one such block is executed on any thread. And such 
blocks should be scarce so they can be examined carefully like 
trusted and everything build upon is verifiable safe.
May 11 2019
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 11 May 2019 at 09:51:41 UTC, Dominikus Dittes 
Scherkl wrote:

 I've followed the AGM a little and think, why not go with the 
 "disable access to shared" approach, but instead of allowing it 
 to be casted away, introduce a "locked scope" lock { } which 
 need to be within a function and allow access to shared 
 variables only within such blocks?
 It's still up to the user to ensure that such locked blocks 
 encapsulate the semantic blocks which need to be kept together 
 to safely modify shared variables, but at least the compiler 
 can make sure they are not modified outside locked blocks and 
 that at any time only one such block is executed on any thread. 
 And such blocks should be scarce so they can be examined 
 carefully like trusted and everything build upon is verifiable 
 safe.


And leave shared in the useless state that it currently has?
+ introducing a new syncronized keyword? (locked)?

doesn't sound too compelling.
May 11 2019
parent Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Saturday, 11 May 2019 at 12:08:00 UTC, Stefan Koch wrote:

 On Saturday, 11 May 2019 at 09:51:41 UTC, Dominikus Dittes 
 Scherkl wrote:

 I've followed the AGM a little and think, why not go with the 
 "disable access to shared" approach, but instead of allowing 
 it to be casted away, introduce a "locked scope" lock { } 
 which need to be within a function and allow access to shared 
 variables only within such blocks?
 It's still up to the user to ensure that such locked blocks 
 encapsulate the semantic blocks which need to be kept together 
 to safely modify shared variables, but at least the compiler 
 can make sure they are not modified outside locked blocks and 
 that at any time only one such block is executed on any 
 thread. And such blocks should be scarce so they can be 
 examined carefully like trusted and everything build upon is 
 verifiable safe.


 And leave shared in the useless state that it currently has?


No, as I said: use the "access disabled" (outside of locked 
blocks) approach, as several people advocated for.


 + introducing a new synchronized keyword? (locked)?


Need not be a keyword. It's also possible to use Lock() and 
Unlock() functions, but they need to be present in the same 
scope, so that the compiler can check, that what was locked 
really gets unlocked again close by (to keep it easily parsable). 
Or the block reuses the shared keyword, just to not burn another 
word for identifiers...

This is to avoid to need a cast and trusted stuff. I think shared 
and trusted should not be conflated, so that they can use a 
different review process (e.g. someone specialized in looking for 
synchronization problems)

 doesn't sound too compelling.
May 11 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, May 11, 2019 3:51:41 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:

 I've followed the AGM a little and think, why not go with the
 "disable access to shared" approach, but instead of allowing it
 to be casted away, introduce a "locked scope" lock { } which need
 to be within a function and allow access to shared variables only
 within such blocks?
 It's still up to the user to ensure that such locked blocks
 encapsulate the semantic blocks which need to be kept together to
 safely modify shared variables, but at least the compiler can
 make sure they are not modified outside locked blocks and that at
 any time only one such block is executed on any thread. And such
 blocks should be scarce so they can be examined carefully like
 trusted and everything build upon is verifiable safe.


All that really would do is add a bit of extra syntax around locking a mutex
and casting away shared. It doesn't really change anything with regards to
what happens. The operations are the same, just wrapped. It's also arguably
worse in that it implies that the compiler is actually making something safe
for you, and it isn't. You're still doing the cast. It's still up to you to
make sure that the concurrency aspects are sorted out correctly. It's just
that what's actually happening is less obvious. I'd argue that we really
don't want anything to just remove shared for us if the compiler can't
actually guarantee that doing so is safe, which it can't do in this case.

Also, mutexes aren't the only concurrency mechanism. You have to deal with
stuff like condition variables and semaphores. And even with mutexes, the
actual code flow may not be as simple as

* lock mutex
* cast away shared
* do stuff as thread-local
* make sure no thread-local references to the data exist
* unlock mutex

That's the basic case and what your suggestion targets, but if your code
flow needs to be even slightly different, it won't work, whereas casting
gives a lot more freedom. It might make sense to add such a helper on top of
casting, but casting is still needed.

Regardless, it's clear that Walter and Andrei want to fully look at the
issue - with concurrency and memory model experts - to make sure that
they're doing the right thing rather than simply doing something now that
might be wrong. It's almost certainly the case that part of whatever we get
will involve disabling reading from and writing to shared objects, and
casting is pretty much going to have to be involved with that, but the
details still need to be worked out, and working out all of those details
(like the memory model) could very well affect what we get in unexpected
ways.

It may ultimately make sense to add helper stuff that hides casts, but we
really need to figure out the details of how shared really works before
looking at adding more user-friendly helpers on top of it.

- Jonathan M Davis
May 11 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis wrote:



 All that really would do is add a bit of extra syntax around 
 locking a mutex and casting away shared.


No, it makes it possible to use mutex in safe functions without 
needing to cast and which the compiler CAN guarantee to really be 
safe.


 It doesn't really change anything with regards to what happens.


correct. It's a safe wrapper that actually make shared usefull in 
one way.


 Also, mutexes aren't the only concurrency mechanism. You have 
 to deal with stuff like condition variables and semaphores. And 
 even with mutexes, the actual code flow may not be as simple as

 * lock mutex
 * cast away shared


No, that's exactly NOT necessary anymore, because the compiler 
can deal with this in a safe and checked manner for this special 
case.

 * do stuff as thread-local
 * make sure no thread-local references to the data exist
 * unlock mutex


Of course. But that would still be possible if you are willing to 
cast and deal with trusted functions. But it would not be 
required anymore for the 99% of usecases where simple mutex would 
be enough.


 That's the basic case and what your suggestion targets, but if 
 your code flow needs to be even slightly different, it won't 
 work, whereas casting gives a lot more freedom. It might make 
 sense to add such a helper on top of casting, but casting is 
 still needed.


Yes and it would be far from my opinion to suggest otherwise.
Casting is always possible and need be in a systems language.
But its not safe, and shared should be possible to be used in at 
least one way without getting into unsafe territory, I think.

 It may ultimately make sense to add helper stuff that hides 
 casts, but we really need to figure out the details of how 
 shared really works before looking at adding more user-friendly 
 helpers on top of it.


And that I think is simple: forbid access without cast (outside 
locked blocks). Its the best (and most KISS) idea that has come 
up so far - and adding locked blocks seems logical to me, to 
allow one safe way to use shared.
May 13 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:

 On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis wrote:

 All that really would do is add a bit of extra syntax around
 locking a mutex and casting away shared.


 No, it makes it possible to use mutex in safe functions without
 needing to cast and which the compiler CAN guarantee to really be
 safe.


Actually, it doesn't. All you've done is lock a mutex and cast away shared.
There is no guarantee that that mutex is always used when that object is
accessed. There could easily be another piece of code somewhere that casts
away shared without locking anything at the same time. And even if this were
the only mechanism for removing shared, you could easily use it with the
same object and a completely different mutex in another piece of code,
thereby making the mutex useless. The type system has no concept of
ownership and no concept of a mutex being associated with any particular
object aside from synchronized classes (and those don't even currently
require that the mutex always be used). And even if the compiler could check
that all uses of a particular variable were locked with a mutex, that still
wouldn't be enough, because other references to the same data could exist.
So, with the construct you've proposed, there's no way for the compiler to
guarantee that no other thread is accessing the data at the same time. All
it guarantees is that a mutex has been locked, not that it's actually
protecting the data.

The only proposal that we've had thus far that is actually able to make
enough guarantees to safely remove shared is the synchronized classes
proposal in TDPL (which has never been implemented). The compiler would only
be able to remove shared at all with TDPL synchronized classes, because it
would guarantee that no references to the data directly in the class escaped
the class, and even then, it could only safely remove the outer layer of
shared, because anything that isn't directly in the class could still have
references to it elsewhere. So, TDPL synchronized classes would actually be
very limited in usefulness, because they simply can't guarantee enough to
strip much of shared away, and they're only able to do that much because of
how restrictive they are. Something as free-form as simply indicating that a
mutex is locked in a block of code is nowhere near enough to guarantee that
accessing any variables in that code is thread-safe.

- Jonathan M Davis
May 13 2019
next sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:

 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl 
 via Digitalmars-d wrote:

 On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis 
 wrote:

 All that really would do is add a bit of extra syntax around 
 locking a mutex and casting away shared.


 No, it makes it possible to use mutex in safe functions 
 without needing to cast and which the compiler CAN guarantee 
 to really be safe.


 Actually, it doesn't. All you've done is lock a mutex and cast 
 away shared. There is no guarantee that that mutex is always 
 used when that object is accessed. There could easily be 
 another piece of code somewhere that casts away shared without 
 locking anything at the same time.


But that piece of code is system, which explicitly allows you to 
shoot into your foot. If you want to stay safe, don't do that.


 And even if this were the only mechanism for removing shared, 
 you could easily use it with the same object and a completely 
 different mutex in another piece of code


Yes, the lock block need a list of vars that it allows to be 
modified

lock(var1, var2, ...)
{
}

two mutexes can only be executed at parallel if their parameter 
set is disjunct.



 And even if the compiler could check that all uses of a 
 particular variable were locked with a mutex, that still 
 wouldn't be enough, because other references to the same data 
 could exist.


ok, so we need in addition that a reference to a shared var need 
not be lived beyond the end of the locked block or be immutable. 
Bad, but seems necessary.
May 14 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, May 14, 2019 6:22:20 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:

 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:

 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl

 via Digitalmars-d wrote:

 On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis

 wrote:

 All that really would do is add a bit of extra syntax around
 locking a mutex and casting away shared.


 No, it makes it possible to use mutex in safe functions
 without needing to cast and which the compiler CAN guarantee
 to really be safe.


 Actually, it doesn't. All you've done is lock a mutex and cast
 away shared. There is no guarantee that that mutex is always
 used when that object is accessed. There could easily be
 another piece of code somewhere that casts away shared without
 locking anything at the same time.


 But that piece of code is system, which explicitly allows you to
 shoot into your foot. If you want to stay safe, don't do that.


The point is that if that code can legally exist, then the compiler simply
cannot guarantee that removing shared from the object is thread-safe even
with the locking mechanism you're proposing.


 And even if this were the only mechanism for removing shared,
 you could easily use it with the same object and a completely
 different mutex in another piece of code


 Yes, the lock block need a list of vars that it allows to be
 modified

 lock(var1, var2, ...)
 {
 }

 two mutexes can only be executed at parallel if their parameter
 set is disjunct.


Sure, but another thread could be using a completely different mutex with
one or more of those variables. So, even if just your locking mechanism is
used, there's no guarantee that the same mutex is always used with the same
set of variables, meaning that the compiler can't guarantee that the data is
actually protected.


 And even if the compiler could check that all uses of a
 particular variable were locked with a mutex, that still
 wouldn't be enough, because other references to the same data
 could exist.


 ok, so we need in addition that a reference to a shared var need
 not be lived beyond the end of the locked block or be immutable.
 Bad, but seems necessary.


A reference could already exist before your proposed locking mechanism was
reached in the code. If the type is a class or pointer, then there could be
other class references or pointers to the same data in  safe code. And in
 system/ trusted code, the address of the object could have been taken to
create a pointer to the object (and that could have been done in code for
removed from the code that's using the lock with all of the code around the
lock being  safe). Heck, there could even be references to data within the
object rather than to the object itself which are available elsewhere,
meaning that part of the object is protected by the lock and part isn't. If
any reference to any part of the data exists anywhere in the program, then
it's possible for another thread to access the data at the same time that
it's locked by the mechanism that you've proposed.

In order for the compiler to be able to actually guarantee that it's safe to
remove shared from an object, it has to be able to guarantee that there are
no other references to any part of that object which exist in the program.
That's why TDPL synchronized classes are so locked down. Without that, other
references to the data could exist. And even with all of the restrictions
that they have, the compiler would still only be able to remove the outer
layer of shared - the layer directly in the class - not any more than that.
With something as free-form as you're proposing, the compiler can't
guarantee anything, let alone that it's thread-safe to completely remove
shared from an object within that block.

If D had ownership semantics baked into its type system, then we could
probably do more, but as it is, the compiler is _very_ limited in its
ability to know that no other references to any portion of an object exist.
All it's dealing with are the types, not what owns them or what else could
refer to them. Even scope is only able to do its job by restricting the
operations that are allowed on a scope object, not by actually tracking an
object's lifetime. So, it's incredibly difficult for the compiler to have
enough information to know that an object is actually fully protected by a
mutex when it's locked. And if the compiler can't guarantee that accessing
the shared object is thread-safe within a particular piece of code, then it
can't safely remove shared.

For any proposal you might have with regards to how we might be able to have
the compiler safely remove shared from an object for us, you're going to
have to be able to prove that no other references to any piece of that
object could possibly exist or that there's no way that any reference to any
portion of that object could be accessed without the same mutex being locked
at every point that it's accessed. And it wouldn't surprise me if someone
else were able to point out why even that wasn't enough because of some
detail I'm not thinking of at the moment. Having the compiler be able to
prove that a piece of code is thread-safe such that shared can be safely
removed automatically from anything is incredibly difficult.

- Jonathan M Davis
May 14 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Tuesday, 14 May 2019 at 21:31:57 UTC, Jonathan M Davis wrote:

 On Tuesday, May 14, 2019 6:22:20 AM MDT Dominikus Dittes 
 Scherkl via Digitalmars-d wrote:

 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:

 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes 
 Scherkl

 via Digitalmars-d wrote:

 On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis

 wrote:

 All that really would do is add a bit of extra syntax 
 around locking a mutex and casting away shared.


 No, it makes it possible to use mutex in safe functions 
 without needing to cast and which the compiler CAN 
 guarantee to really be safe.


 Actually, it doesn't. All you've done is lock a mutex and 
 cast away shared. There is no guarantee that that mutex is 
 always used when that object is accessed. There could easily 
 be another piece of code somewhere that casts away shared 
 without locking anything at the same time.


 But that piece of code is system, which explicitly allows you 
 to shoot into your foot. If you want to stay safe, don't do 
 that.


 The point is that if that code can legally exist, then the 
 compiler simply cannot guarantee that removing shared from the 
 object is thread-safe even with the locking mechanism you're 
 proposing.


with system code you can always destroy safety assumptions of any 
other written code. This is why system code should be avoided 
where ever possible and the unavoidable remains need to be 
reviewed very carefully to not spoil the guarantees that are 
valid otherwise.


 And even if this were the only mechanism for removing 
 shared, you could easily use it with the same object and a 
 completely different mutex in another piece of code


 Yes, the lock block need a list of vars that it allows to be 
 modified

 lock(var1, var2, ...)
 {
 }

 two mutexes can only be executed at parallel if their 
 parameter set is disjunct.


 Sure, but another thread could be using a completely different 
 mutex with one or more of those variables.


No, it can't. Disjunct means: It cannot be called unless all of 
the given variables are free (not locked by any other mutex).


 ok, so we need in addition that a reference to a shared var 
 need not be lived beyond the end of the locked block or be 
 immutable. Bad, but seems necessary.


 A reference could already exist before your proposed locking 
 mechanism was reached in the code. If the type is a class or 
 pointer, then there could be other class references or pointers 
 to the same data in  safe code. And in  system/ trusted code, 
 the address of the object could have been taken to create a 
 pointer to the object (and that could have been done in code 
 for removed from the code that's using the lock with all of the 
 code around the lock being  safe). Heck, there could even be 
 references to data within the object rather than to the object 
 itself which are available elsewhere, meaning that part of the 
 object is protected by the lock and part isn't. If any 
 reference to any part of the data exists anywhere in the 
 program, then it's possible for another thread to access the 
 data at the same time that it's locked by the mechanism that 
 you've proposed.


Ok, that whole reference stuff is always a problem. Why not 
simply forbid it? You can't reference shared variables (outside 
locked blocks), you can only copy them. We can later relax that 
rule if some safe ways to allow that are found. I can't see why 
that should hinder us to make the more practical usecases safe 
for now.


 In order for the compiler to be able to actually guarantee that 
 it's safe to remove shared from an object, it has to be able to 
 guarantee that there are no other references to any part of 
 that object which exist in the program. That's why TDPL 
 synchronized classes are so locked down. Without that, other 
 references to the data could exist. And even with all of the 
 restrictions that they have, the compiler would still only be 
 able to remove the outer layer of shared - the layer directly 
 in the class - not any more than that. With something as 
 free-form as you're proposing, the compiler can't guarantee 
 anything, let alone that it's thread-safe to completely remove 
 shared from an object within that block.

 If D had ownership semantics baked into its type system, then 
 we could probably do more, but as it is, the compiler is _very_ 
 limited in its ability to know that no other references to any 
 portion of an object exist.


If we forbid them (for now), the compiler is well able to.


 Even scope is only able to do its job by restricting the 
 operations that are allowed on a scope object, not by actually 
 tracking an object's lifetime.


Yes, and that's fine. It isn't necessary that everything is 
possible with an object, but it should at least be useful for 
SOME task.


 For any proposal you might have with regards to how we might be 
 able to have the compiler safely remove shared from an object 
 for us, you're going to have to be able to prove that no other 
 references to any piece of that object could possibly exist or 
 that there's no way that any reference to any portion of that 
 object could be accessed without the same mutex being locked at 
 every point that it's accessed.


Understood.


 And it wouldn't surprise me if someone else were able to point 
 out why even that wasn't enough because of some detail I'm not 
 thinking of at the moment. Having the compiler be able to prove 
 that a piece of code is thread-safe such that shared can be 
 safely removed automatically from anything is incredibly 
 difficult.


shared shouldn't be removed from an object, but it can only be 
modyfied if it is locked. Removing shared (with a cast) is system 
stuff and should be out of scope for any safety related proposal 
(including mine), because with system stuff you can destroy any 
kind of safety.

If you don't remove shared, you can easily apply rules like 
forbid to take it's address or such. If you remove it, that makes 
it much harder (and isn't useful anyway).

I still think my proposal could work (provide provable 
thread-safety for shared objects) in a limited but useful way 
(only mutex, no references), and should be relatively easy to 
implement.
If you want more complex stuff, that's still possible in the same 
way it currently is: cast shared away together with all 
guarantees and verify manually that it works, just like in C++.
May 14 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 15, 2019 12:59:00 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:

 On Tuesday, 14 May 2019 at 21:31:57 UTC, Jonathan M Davis wrote:

 The point is that if that code can legally exist, then the
 compiler simply cannot guarantee that removing shared from the
 object is thread-safe even with the locking mechanism you're
 proposing.


 with system code you can always destroy safety assumptions of any
 other written code. This is why system code should be avoided
 where ever possible and the unavoidable remains need to be
 reviewed very carefully to not spoil the guarantees that are
 valid otherwise.


 safety and thread-safety are very different.  system mechanisms such as
casting are involved with thread-safety, meaning that  system and  trusted
get involved, but what they mean and how they're verified are very
different. For  safety, if you want to mark something as  trusted, you just
need to look at that piece of code to verify that what it's doing is memory
safe. You don't have to look at other  safe parts of the program to verify
that what it's doing is correct.

On the other hand, with thread-safety, you have to look at _everywhere_ that
a particular piece of shared data is accessed if you want to be able to
guarantee that it's accessed in a thread-safe manner. You can't just assume
that other code is doing the right thing and just verify that one piece of
code that's using  system mechanisms to interact with shared data. So, you
can't rely on  safe to tell you whether anything is thread-safe.


 And even if this were the only mechanism for removing
 shared, you could easily use it with the same object and a
 completely different mutex in another piece of code


 Yes, the lock block need a list of vars that it allows to be
 modified

 lock(var1, var2, ...)
 {
 }

 two mutexes can only be executed at parallel if their
 parameter set is disjunct.


 Sure, but another thread could be using a completely different
 mutex with one or more of those variables.


 No, it can't. Disjunct means: It cannot be called unless all of
 the given variables are free (not locked by any other mutex).


So, you're proposing that something in the runtime keeps track of which
variables are currently associated with a locked mutex in order to guarantee
that no other lock block is able to access any of those variables at the
same time? That would probably require adding a global lock used by the
runtime when any code enters or exists a lock block. I'd be _very_ surprised
if anything like that were deemed acceptable for D. And it still doesn't
solve the problem of other references referring to any part of the objects
referred to by those variables existing and potentially being used
elsewhere. If you had

lock(mutex, var1)
{
}

and elsewhere

lock(mutex, var2)
{
}

when var1 and var2 were references to the same object or when var2 referred
to a piece of data inside of var1, then the lock wouldn't be providing
thread-safety. If you only had one mutex for the entire program, and casting
away shared were illegal, then something like this could probaly work, but
having one mutex for the entire program would clearly be unworkable -
especially for a systems language - and since mutexes (let alone this
particular use pattern for mutexes) aren't the only way to protect shared
data when accessing it, requiring that this particular construct be used for
accessing shared data wouldn't work anyway.


 ok, so we need in addition that a reference to a shared var
 need not be lived beyond the end of the locked block or be
 immutable. Bad, but seems necessary.


 A reference could already exist before your proposed locking
 mechanism was reached in the code. If the type is a class or
 pointer, then there could be other class references or pointers
 to the same data in  safe code. And in  system/ trusted code,
 the address of the object could have been taken to create a
 pointer to the object (and that could have been done in code
 for removed from the code that's using the lock with all of the
 code around the lock being  safe). Heck, there could even be
 references to data within the object rather than to the object
 itself which are available elsewhere, meaning that part of the
 object is protected by the lock and part isn't. If any
 reference to any part of the data exists anywhere in the
 program, then it's possible for another thread to access the
 data at the same time that it's locked by the mechanism that
 you've proposed.


 Ok, that whole reference stuff is always a problem. Why not
 simply forbid it? You can't reference shared variables (outside
 locked blocks), you can only copy them. We can later relax that
 rule if some safe ways to allow that are found. I can't see why
 that should hinder us to make the more practical usecases safe
 for now.


You can't forbid references to the same data. All that would be required
would be something like

shared foo = new shared(Foo)(42);
auto bar = foo;

and you have two references to the same object without doing anything
unsafe. You could also have stuff like

shared baz = foo.getBaz();

resulting in a reference to some piece of data that the foo object contains
(possibly simply a shared class object that it has a reference to as a
member). In general, to be able to verify that no other references to data
existed, we'd probably have to add some sort of ownership semantics to the
language so that the compiler could know that nothing else can possibly have
access to the data (as I understand it, Rust manages something along those
lines, but they have a much more restrictive type system).


 And it wouldn't surprise me if someone else were able to point
 out why even that wasn't enough because of some detail I'm not
 thinking of at the moment. Having the compiler be able to prove
 that a piece of code is thread-safe such that shared can be
 safely removed automatically from anything is incredibly
 difficult.


 shared shouldn't be removed from an object, but it can only be
 modyfied if it is locked. Removing shared (with a cast) is system
 stuff and should be out of scope for any safety related proposal
 (including mine), because with system stuff you can destroy any
 kind of safety.

 If you don't remove shared, you can easily apply rules like
 forbid to take it's address or such. If you remove it, that makes
 it much harder (and isn't useful anyway).

 I still think my proposal could work (provide provable
 thread-safety for shared objects) in a limited but useful way
 (only mutex, no references), and should be relatively easy to
 implement.
 If you want more complex stuff, that's still possible in the same
 way it currently is: cast shared away together with all
 guarantees and verify manually that it works, just like in C++.


As soon as casting away shared is legal (and I think that it has to be for
many common thread-synchronization idioms to be used), any mechanism like
you're suggesting isn't enough to guarantee that it's safe to remove shared
even temporarily. Either way, the ability to get multiple references to the
same data defeats what you're proposing, and I don't see how it would be
possible to make it so that they're can't be multiple references to the data
given D's type system. TDPL synchronized classes are only able to do it with
the data that lives directly in the class, because they're a very
restrictive construct. But even then, what the member variables refer to
can't have shared removed, because references to the same data could have
escaped the class or be passed into the class from elsewhere. And if
something as restrictive as TDPL synchronized classes aren't able to
restrict references sufficiently to be able to just outright remove shared
from the class' member variables, there's no way that something as free-form
as locking a mutex on a set of variables that aren't encapsulated in
anything is going to be able to guarantee that other references to the same
data don't exist.

- Jonathan M Davis
May 15 2019
next sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Wednesday, 15 May 2019 at 09:33:19 UTC, Jonathan M Davis wrote:

 You can't forbid references to the same data. All that would be 
 required would be something like

 shared foo = new shared(Foo)(42);
 auto bar = foo;


Meep. This is either a compile error outside locked block (cannot 
read shared) or bar would be scope (it's lifetime will end after 
lock block is left).


 and you have two references to the same object without doing 
 anything unsafe. You could also have stuff like

 shared baz = foo.getBaz();


This will be forced to a copy of the returned value, not a 
reference.
May 16 2019
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 4:17:15 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:

 On Wednesday, 15 May 2019 at 09:33:19 UTC, Jonathan M Davis wrote:

 You can't forbid references to the same data. All that would be
 required would be something like

 shared foo = new shared(Foo)(42);
 auto bar = foo;


 Meep. This is either a compile error outside locked block (cannot
 read shared) or bar would be scope (it's lifetime will end after
 lock block is left).


If you can't pass shared class references or pointers around like this, then
it wouldn't even be possible to call shared functions on shared objects,
because that's what happens when you call a function. And that really would
make shared ridiculously hard to use. It would make it impossible to
encapsulate thread synchronization primitives and force the programmer to
handle it directly. Things like a shared class or struct that handled its
own locking would be impossible, because you couldn't even call any of its
methods. Code like

shared foo = new shared(Foo)(42);
auto bar = foo;

needs to work as long as the type being passed around is a pointer or class
reference, or shared simply won't be useable.


 and you have two references to the same object without doing
 anything unsafe. You could also have stuff like

 shared baz = foo.getBaz();


 This will be forced to a copy of the returned value, not a
 reference.


No matter what the return type is, a line like that would either copy or
move the return value (which one would depend on the implementation of
getBaz and what exactly it was returning), but for a pointer or class
reference, that just means that you have a pointer or class reference
referring to the same thing as something that was in getBaz. So, copying is
the normal semantics. And if you're trying to claim that it should be a deep
copy, D doesn't even have that capability for classes - not without there
being user-defind functions which do it, and the compiler won't know that
that's what they do and couldn't use them automatically. So, that wouldn't
work even if it were desirable - and it wouldn't be, because you really
don't want to be deep copying classe normally. And you _really_ don't want
to be doing it if shared is involved; that would require locking in a way
that copying a simple pointer or reference shouldn't (at least not if we
want to be able to actually do stuff like call shared member functions).

Also, just in general, trying to prevent other references to the same data
with shared actually makes no sense. If you don't have multiple references
to the same data, then there's no point in it being shared in the first
place, because you won't have multiple threads accessing the data. Even with
TDPL synchronized classes, which prevent references to the member variables
from escaping, you still have multiple references to the class itself - and
the class itself is shared. Multiple references to the same data is a key
part of data being shared. At best, you can avoid it by having only a single
reference to some shared data with multiple references to a shared object
which has the single reference to the other shared data (i.e. what you get
with TDPL synchronized classes). And if that's what you're doing, then you
might as well just have TDPL synchronized classes rather than trying to do
something more free-from.

- Jonathan M Davis
May 16 2019
prev sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Wednesday, 15 May 2019 at 09:33:19 UTC, Jonathan M Davis wrote:

 On Wednesday, May 15, 2019 12:59:00 AM MDT Dominikus Dittes 
 Scherkl via Digitalmars-d wrote:

 No, it can't. Disjunct means: It cannot be called unless all 
 of the given variables are free (not locked by any other 
 mutex).


 So, you're proposing that something in the runtime keeps track 
 of which variables are currently associated with a locked mutex 
 in order to guarantee that no other lock block is able to 
 access any of those variables at the same time? That would 
 probably require adding a global lock used by the runtime when 
 any code enters or exists a lock block. I'd be _very_ surprised 
 if anything like that were deemed acceptable for D.


Ok, that can be a problem. But yes, this is what I suggest.
But why the resistence? Nobody is forced to use mutexes, and if 
you don't that part of the runtime could (should!) be eliminated 
from the program.


 And it still doesn't solve the problem of other references 
 referring to any part of the objects referred to by those 
 variables existing and potentially being used elsewhere. If you 
 had

 lock(mutex, var1)
 {
 }

 and elsewhere

 lock(mutex, var2)
 {
 }

 when var1 and var2 were references to the same object


Forbidden. See last post.


 or when var2 referred to a piece of data inside of var1,


This would also be impossible. shared vars are no references.


 If you only had one mutex for the entire program, and casting 
 away shared were illegal, then something like this could 
 probably work,


Thank you, but why this restriction? There can be as many mutexes 
as you like. The runtime has only to ensure that any running 
locked block doesn't modify any of the vars in the other running 
locked blocks.
May 16 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 4:35:52 AM MDT Dominikus Dittes Scherkl via 
Digitalmars-d wrote:

 On Wednesday, 15 May 2019 at 09:33:19 UTC, Jonathan M Davis wrote:

 On Wednesday, May 15, 2019 12:59:00 AM MDT Dominikus Dittes

 Scherkl via Digitalmars-d wrote:

 No, it can't. Disjunct means: It cannot be called unless all
 of the given variables are free (not locked by any other
 mutex).


 So, you're proposing that something in the runtime keeps track
 of which variables are currently associated with a locked mutex
 in order to guarantee that no other lock block is able to
 access any of those variables at the same time? That would
 probably require adding a global lock used by the runtime when
 any code enters or exists a lock block. I'd be _very_ surprised
 if anything like that were deemed acceptable for D.


 Ok, that can be a problem. But yes, this is what I suggest.
 But why the resistence? Nobody is forced to use mutexes, and if
 you don't that part of the runtime could (should!) be eliminated
 from the program.


For the locking mechanism you're proposing to work, you basically have to
disallow a lot of stuff that is required for other types of thread
synchronization (e.g. casting away shared is needed), and having a global
lock that's used whenever a mutex is used would be terrible for performance.
In general, we want to avoid global locks, and it seems particularly bad to
have a global mutex that needs to be used just to use a normal mutex.


 And it still doesn't solve the problem of other references
 referring to any part of the objects referred to by those
 variables existing and potentially being used elsewhere. If you
 had

 lock(mutex, var1)
 {
 }

 and elsewhere

 lock(mutex, var2)
 {
 }

 when var1 and var2 were references to the same object


 Forbidden. See last post.


 or when var2 referred to a piece of data inside of var1,


 This would also be impossible. shared vars are no references.


I discussed this in another post, but you _have_ to be able to have shared
references or shared is basically useless.


 If you only had one mutex for the entire program, and casting
 away shared were illegal, then something like this could
 probably work,


 Thank you, but why this restriction? There can be as many mutexes
 as you like. The runtime has only to ensure that any running
 locked block doesn't modify any of the vars in the other running
 locked blocks.


If it's at all possible to access any of the shared data that your lock
blocks protect without using the lock block, then your lock block would not
be enough for the compiler to know that it was safe to remove shared from
the data within the lock block. And without a global mutex that's used to
check that different locks aren't being used for the same variable, then
different mutexes could be used for the same varible, defeating the
protection - and even that is assuming that it's not possible to have any
other references to the same data, which would be too restrictive to be
useful. Having multiple references to the same data is a core concept of
sharing data across threads.

- Jonathan M Davis
May 16 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Thursday, 16 May 2019 at 11:15:13 UTC, Jonathan M Davis wrote:

 [...] that is assuming that it's not possible to have any other 
 references to the same data, which would be too restrictive to 
 be useful. Having multiple references to the same data is a 
 core concept of sharing data across threads.


Hm. I don't understand the necessity of this. If I have a shared 
symbol x,
this same symbol can be used in different threads, no? I can't 
see why multiple threads need to have different references to the 
same data. They just can use the same reference.
May 16 2019
parent reply Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On Thursday, 16 May 2019 at 15:47:37 UTC, Dominikus Dittes 
Scherkl wrote:

 On Thursday, 16 May 2019 at 11:15:13 UTC, Jonathan M Davis 
 wrote:

 [...] that is assuming that it's not possible to have any 
 other references to the same data, which would be too 
 restrictive to be useful. Having multiple references to the 
 same data is a core concept of sharing data across threads.


 Hm. I don't understand the necessity of this. If I have a 
 shared symbol x,
 this same symbol can be used in different threads, no? I can't 
 see why multiple threads need to have different references to 
 the same data. They just can use the same reference.


Well, no. The reference is the part that gets copied around. It's 
the address of the shared object. When you do 'auto ref2 = 
ref1;', you're creating a new reference to an already existing 
object, but leaving the object itself unchanged.

If more than one thread can access the same object, that means 
they each have a copy of the reference*. In the same way, when 
you call a function with a class argument, that function receives 
a copy of the reference (but not the object). Clearly then, the 
ability to create multiple references to the same object must be 
possible without breaking shared.

--
   Simen

* barring __gshared tricks
May 16 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 11:11:19 AM MDT Simen Kjærås via Digitalmars-d 
wrote:

 On Thursday, 16 May 2019 at 15:47:37 UTC, Dominikus Dittes

 Scherkl wrote:

 On Thursday, 16 May 2019 at 11:15:13 UTC, Jonathan M Davis

 wrote:

 [...] that is assuming that it's not possible to have any
 other references to the same data, which would be too
 restrictive to be useful. Having multiple references to the
 same data is a core concept of sharing data across threads.


 Hm. I don't understand the necessity of this. If I have a
 shared symbol x,
 this same symbol can be used in different threads, no? I can't
 see why multiple threads need to have different references to
 the same data. They just can use the same reference.


 Well, no. The reference is the part that gets copied around. It's
 the address of the shared object. When you do 'auto ref2 =
 ref1;', you're creating a new reference to an already existing
 object, but leaving the object itself unchanged.

 If more than one thread can access the same object, that means
 they each have a copy of the reference*. In the same way, when
 you call a function with a class argument, that function receives
 a copy of the reference (but not the object). Clearly then, the
 ability to create multiple references to the same object must be
 possible without breaking shared.


Exactly. And since everything in D is thread-local by default, it's usually
the case that a variable only exists on one thread. shared changes that if
the variable is static, but otherwise, you're just passing around a pointer
or reference to shared objects within thread-local objects. So, while  you
might have a pointer to shared data contained by a variable with the same
name on multiple threads, because that variable is thread-local, it's
different on different threads. e.g.

struct S
{
    int i;
    shared Foo* foo;
}

// Every thread will have it's own variable for this
S s;

- Jonathan M Davis
May 16 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Thursday, 16 May 2019 at 17:29:25 UTC, Jonathan M Davis wrote:

 On Thursday, May 16, 2019 11:11:19 AM MDT Simen Kjærås via 
 Digitalmars-d wrote:

 On Thursday, 16 May 2019 at 15:47:37 UTC, Dominikus Dittes

 Scherkl wrote:

 On Thursday, 16 May 2019 at 11:15:13 UTC, Jonathan M Davis

 wrote:

 [...] that is assuming that it's not possible to have any 
 other references to the same data, which would be too 
 restrictive to be useful. Having multiple references to the 
 same data is a core concept of sharing data across threads.


 Hm. I don't understand the necessity of this. If I have a
 shared symbol x,
 this same symbol can be used in different threads, no? I 
 can't
 see why multiple threads need to have different references to
 the same data. They just can use the same reference.


 Well, no. The reference is the part that gets copied around. 
 It's the address of the shared object. When you do 'auto ref2 
 = ref1;', you're creating a new reference to an already 
 existing object, but leaving the object itself unchanged.

 If more than one thread can access the same object, that means 
 they each have a copy of the reference*. In the same way, when 
 you call a function with a class argument, that function 
 receives a copy of the reference (but not the object). Clearly 
 then, the ability to create multiple references to the same 
 object must be possible without breaking shared.


 Exactly. And since everything in D is thread-local by default, 
 it's usually the case that a variable only exists on one 
 thread. shared changes that if the variable is static, but 
 otherwise, you're just passing around a pointer or reference to 
 shared objects within thread-local objects. So, while  you 
 might have a pointer to shared data contained by a variable 
 with the same name on multiple threads, because that variable 
 is thread-local, it's different on different threads. e.g.


Ok, so then a shared object is in fact a specific memory segment. 
So something the GC is already aware of. This makes implementing 
the locking mechanism just more easy. We only need to prevent 
writing to that address, no matter how many references are 
pointing there. And that mechanism is well known in GC's - a 
write barrier. Only new thing is that no other locked block 
pointing in that address area can be executed. I just can't see 
what fundamental, all blocking problem you see with that 
approach. Seems pretty strait forward to me.
May 16 2019
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 16, 2019 at 07:53:56PM +0000, Dominikus Dittes Scherkl via
Digitalmars-d wrote:

 On Thursday, 16 May 2019 at 17:29:25 UTC, Jonathan M Davis wrote:


[...]

 Ok, so then a shared object is in fact a specific memory segment. So
 something the GC is already aware of. This makes implementing the
 locking mechanism just more easy. We only need to prevent writing to
 that address, no matter how many references are pointing there. And
 that mechanism is well known in GC's - a write barrier.


[...]

D does not have write barriers, however. And it's unlikely to get one in
the foreseeable future.


T

-- 
What do you get if you drop a piano down a mineshaft? A flat minor.
May 16 2019
parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Thursday, 16 May 2019 at 20:04:13 UTC, H. S. Teoh wrote:

 What do you get if you drop a piano down a mineshaft? A flat 
 minor.


I love you're signatures. You don't choose them at random, do you?

At least I accept now, it's not as easy as one might think.

Arimasen.
May 17 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 17, 2019 at 10:20:32AM +0000, Dominikus Dittes Scherkl via
Digitalmars-d wrote:

 On Thursday, 16 May 2019 at 20:04:13 UTC, H. S. Teoh wrote:

 What do you get if you drop a piano down a mineshaft? A flat minor.


 
 I love you're signatures. You don't choose them at random, do you?


[...]

It's chosen by a Perl script.  It's supposed to be random, but the Perl
RNG seems to have a mind of its own sometimes. :-D


T

-- 
Study gravitation, it's a field with a lot of potential.
May 17 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 16, 2019 2:04:13 PM MDT H. S. Teoh via Digitalmars-d wrote:

 On Thu, May 16, 2019 at 07:53:56PM +0000, Dominikus Dittes Scherkl via 


Digitalmars-d wrote:

 On Thursday, 16 May 2019 at 17:29:25 UTC, Jonathan M Davis wrote:


 [...]


 Ok, so then a shared object is in fact a specific memory segment. So
 something the GC is already aware of. This makes implementing the
 locking mechanism just more easy. We only need to prevent writing to
 that address, no matter how many references are pointing there. And
 that mechanism is well known in GC's - a write barrier.


 [...]

 D does not have write barriers, however. And it's unlikely to get one in
 the foreseeable future.


That and not all memory comes from the GC. It's perfectly legitimate to use
malloced memory with shared. The type system doesn't care where the memory
came from.

- Jonathan M Davis
May 16 2019
prev sibling parent Manu <turkeyman gmail.com> writes:
On Wed, May 15, 2019 at 2:34 AM Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 [... everything that jonathan said ...]
 ...there's no way that something as free-form
 as locking a mutex on a set of variables that aren't encapsulated in
 anything is going to be able to guarantee that other references to the same
 data don't exist.


This is hard-facts.
We talked about this topic a lot at dconf. Here's where we got to.


there's any future design where this isn't true. We should do this
now.

have enough context to know.

start building some serious constructions above. Any such construction
necessarily requires tight encapsulation to keep references contained
and only issue leases appropriately.

shared away (take a lease), the resulting pointer must be `scope`,
otherwise it could be sequestered away, and all bets are off!


Constructions are too varied, and too complex. But I think we know how
to start to write some useful libraries that can safely share data


I spent some time with Amaury talking through designs for `shared`
shared pointers. We concluded that with DIP1000, there is enough
language available to write a  safe shared object as a library (and no
mutex-es!).
It is clear that any such design requires lease-tracking. It's not
hard to imagine a shared shared-pointer with the same rules as Rust;
"write access may only have 1 lease", "read access many have N
leases", "read/write access are mutually exclusive". The container
would have functions to capture a lease into some raii object, and the
returned object must be DIP1000 `scope`.
May 15 2019
prev sibling parent reply Radu <void null.pt> writes:
On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:

 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl 
 via Digitalmars-d wrote:

 [...]


 Actually, it doesn't. All you've done is lock a mutex and cast 
 away shared. There is no guarantee that that mutex is always 
 used when that object is accessed. There could easily be 
 another piece of code somewhere that casts away shared without 
 locking anything at the same time. And even if this were the 
 only mechanism for removing shared, you could easily use it 
 with the same object and a completely different mutex in 
 another piece of code, thereby making the mutex useless. The 
 type system has no concept of ownership and no concept of a 
 mutex being associated with any particular object aside from 
 synchronized classes (and those don't even currently require 
 that the mutex always be used). And even if the compiler could 
 check that all uses of a particular variable were locked with a 
 mutex, that still wouldn't be enough, because other references 
 to the same data could exist. So, with the construct you've 
 proposed, there's no way for the compiler to guarantee that no 
 other thread is accessing the data at the same time. All it 
 guarantees is that a mutex has been locked, not that it's 
 actually protecting the data.

 [...]


I had this idea for some time, not sure I can best articulated 
now, but I think a workable solution for shared is closely linked 
with the dip1000 - scoped pointers.

What I mean is that something like:

void doSharedStuff(scope shared(Foo) foo)
{
}

Will be able to safely lock/unlock foo and cast away shared'ness 
in the function's scope.
The compiler can provide guarantees here that foo will not escape.
May 14 2019
next sibling parent Kagamin <spam here.lot> writes:
Sounds like what Atila did 
https://forum.dlang.org/post/vocustlarsadbankufjk forum.dlang.org
May 14 2019
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via Digitalmars-d wrote:

 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:

 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl

 via Digitalmars-d wrote:

 [...]


 Actually, it doesn't. All you've done is lock a mutex and cast
 away shared. There is no guarantee that that mutex is always
 used when that object is accessed. There could easily be
 another piece of code somewhere that casts away shared without
 locking anything at the same time. And even if this were the
 only mechanism for removing shared, you could easily use it
 with the same object and a completely different mutex in
 another piece of code, thereby making the mutex useless. The
 type system has no concept of ownership and no concept of a
 mutex being associated with any particular object aside from
 synchronized classes (and those don't even currently require
 that the mutex always be used). And even if the compiler could
 check that all uses of a particular variable were locked with a
 mutex, that still wouldn't be enough, because other references
 to the same data could exist. So, with the construct you've
 proposed, there's no way for the compiler to guarantee that no
 other thread is accessing the data at the same time. All it
 guarantees is that a mutex has been locked, not that it's
 actually protecting the data.

 [...]


 I had this idea for some time, not sure I can best articulated
 now, but I think a workable solution for shared is closely linked
 with the dip1000 - scoped pointers.

 What I mean is that something like:

 void doSharedStuff(scope shared(Foo) foo)
 {
 }

 Will be able to safely lock/unlock foo and cast away shared'ness
 in the function's scope.
 The compiler can provide guarantees here that foo will not escape.


Sure, it can guarantee that no reference will escape that function, but all
that's required is that another reference to the same data exist elsewhere,
and another thread could muck with the object while the mutex was locked.
There's no question that helpers could be created which would help users
avoid mistakes when casting when casting away shared, but the compiler can't
actually make the guarantee that casting away shared is thread-safe.

- Jonathan M Davis
May 14 2019
parent reply Radu <void null.pt> writes:
On Tuesday, 14 May 2019 at 21:02:10 UTC, Jonathan M Davis wrote:

 On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via Digitalmars-d 
 wrote:

 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:

 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes 
 Scherkl

 via Digitalmars-d wrote:

 [...]


 Actually, it doesn't. All you've done is lock a mutex and 
 cast away shared. There is no guarantee that that mutex is 
 always used when that object is accessed. There could easily 
 be another piece of code somewhere that casts away shared 
 without locking anything at the same time. And even if this 
 were the only mechanism for removing shared, you could 
 easily use it with the same object and a completely 
 different mutex in another piece of code, thereby making the 
 mutex useless. The type system has no concept of ownership 
 and no concept of a mutex being associated with any 
 particular object aside from synchronized classes (and those 
 don't even currently require that the mutex always be used). 
 And even if the compiler could check that all uses of a 
 particular variable were locked with a mutex, that still 
 wouldn't be enough, because other references to the same 
 data could exist. So, with the construct you've proposed, 
 there's no way for the compiler to guarantee that no other 
 thread is accessing the data at the same time. All it 
 guarantees is that a mutex has been locked, not that it's 
 actually protecting the data.

 [...]


 I had this idea for some time, not sure I can best articulated 
 now, but I think a workable solution for shared is closely 
 linked with the dip1000 - scoped pointers.

 What I mean is that something like:

 void doSharedStuff(scope shared(Foo) foo)
 {
 }

 Will be able to safely lock/unlock foo and cast away 
 shared'ness
 in the function's scope.
 The compiler can provide guarantees here that foo will not 
 escape.


 Sure, it can guarantee that no reference will escape that 
 function, but all that's required is that another reference to 
 the same data exist elsewhere, and another thread could muck 
 with the object while the mutex was locked. There's no question 
 that helpers could be created which would help users avoid 
 mistakes when casting when casting away shared, but the 
 compiler can't actually make the guarantee that casting away 
 shared is thread-safe.

 - Jonathan M Davis


My view is that the compiler could automatically insert locking 
logic (ala synchronized) when the shared parameters gets 
references inside the function, and also automatically cast away 
shared so for the function internals it would be like working 
with local non-shared data.

Given that compiler inferes lifetime, it could safely elide 
locking if the parameter is passed to other functions that have 
the same signature (scope is explicit or inferred).

The idea is to provide the tools that simplify concurrent 
programming, the compiler will need to insert all the checks 
automatically using the lifetime tracking.
May 15 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 15, 2019 1:56:12 AM MDT Radu via Digitalmars-d wrote:

 On Tuesday, 14 May 2019 at 21:02:10 UTC, Jonathan M Davis wrote:

 On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via Digitalmars-d

 wrote:

 On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:

 On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes
 Scherkl

 via Digitalmars-d wrote:

 [...]


 Actually, it doesn't. All you've done is lock a mutex and
 cast away shared. There is no guarantee that that mutex is
 always used when that object is accessed. There could easily
 be another piece of code somewhere that casts away shared
 without locking anything at the same time. And even if this
 were the only mechanism for removing shared, you could
 easily use it with the same object and a completely
 different mutex in another piece of code, thereby making the
 mutex useless. The type system has no concept of ownership
 and no concept of a mutex being associated with any
 particular object aside from synchronized classes (and those
 don't even currently require that the mutex always be used).
 And even if the compiler could check that all uses of a
 particular variable were locked with a mutex, that still
 wouldn't be enough, because other references to the same
 data could exist. So, with the construct you've proposed,
 there's no way for the compiler to guarantee that no other
 thread is accessing the data at the same time. All it
 guarantees is that a mutex has been locked, not that it's
 actually protecting the data.

 [...]


 I had this idea for some time, not sure I can best articulated
 now, but I think a workable solution for shared is closely
 linked with the dip1000 - scoped pointers.

 What I mean is that something like:

 void doSharedStuff(scope shared(Foo) foo)
 {
 }

 Will be able to safely lock/unlock foo and cast away
 shared'ness
 in the function's scope.
 The compiler can provide guarantees here that foo will not
 escape.


 Sure, it can guarantee that no reference will escape that
 function, but all that's required is that another reference to
 the same data exist elsewhere, and another thread could muck
 with the object while the mutex was locked. There's no question
 that helpers could be created which would help users avoid
 mistakes when casting when casting away shared, but the
 compiler can't actually make the guarantee that casting away
 shared is thread-safe.

 - Jonathan M Davis


 My view is that the compiler could automatically insert locking
 logic (ala synchronized) when the shared parameters gets
 references inside the function, and also automatically cast away
 shared so for the function internals it would be like working
 with local non-shared data.

 Given that compiler inferes lifetime, it could safely elide
 locking if the parameter is passed to other functions that have
 the same signature (scope is explicit or inferred).

 The idea is to provide the tools that simplify concurrent
 programming, the compiler will need to insert all the checks
 automatically using the lifetime tracking.


The compiler does not have enough information to know which mutexes to use
when even if we wanted it to insert locks. TDPL synchronized classes are a
special case in that they would provide a way in the language to associate a
specific lock with a specific set of variables in a way that the compiler
could then guarantee that it's safe to remove the outer layer of shared
within a synchronized function. Without a similar mechanism to associate a
mutex with one or more variables and guarantee that that mutex is always
locked when they're accessed, the compiler won't be able to even know what
to lock, let alone that it's safe to remove shared within a particular
section of code.

And the issue with what you're proposing with scope is that unless the
compiler can actually guarantee that no other references to the shared
object exist which could be used to access the object at the same time, then
the compiler cannot safely remove shared even temporarily. scope is just
enough to guarantee that that particular function can't escape any
references, not enough to guarantee that they don't exist. So, while
features like scope can be used to make it easier to reason about the code
and cast away shared in a way that your code is thread-safe, I fully expect
that it's going to have to be up to the programmer to know when it's safe to
remove shared, thus requiring a cast or some other  trusted mechanism to
temporarily remove shared. TDPL synchronized classes are the only proposal
I've seen that would be able to guarantee thread-safety, and it can only do
it for what's directly in the class, making TDPL synchronized classes
arguably pretty useless (on top of the fact that it means requiring classes
when most D code wouldn't normally use classes for something like this).

- Jonathan M Davis
May 15 2019
parent reply Radu <void null.pt> writes:
On Wednesday, 15 May 2019 at 08:52:23 UTC, Jonathan M Davis wrote:

 On Wednesday, May 15, 2019 1:56:12 AM MDT Radu via 
 Digitalmars-d wrote:

 On Tuesday, 14 May 2019 at 21:02:10 UTC, Jonathan M Davis 
 wrote:

 On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via 
 Digitalmars-d

 wrote:

 [...]


 Sure, it can guarantee that no reference will escape that 
 function, but all that's required is that another reference 
 to the same data exist elsewhere, and another thread could 
 muck with the object while the mutex was locked. There's no 
 question that helpers could be created which would help 
 users avoid mistakes when casting when casting away shared, 
 but the compiler can't actually make the guarantee that 
 casting away shared is thread-safe.

 - Jonathan M Davis


 My view is that the compiler could automatically insert 
 locking logic (ala synchronized) when the shared parameters 
 gets references inside the function, and also automatically 
 cast away shared so for the function internals it would be 
 like working with local non-shared data.

 Given that compiler inferes lifetime, it could safely elide 
 locking if the parameter is passed to other functions that 
 have the same signature (scope is explicit or inferred).

 The idea is to provide the tools that simplify concurrent 
 programming, the compiler will need to insert all the checks 
 automatically using the lifetime tracking.


 The compiler does not have enough information to know which 
 mutexes to use when even if we wanted it to insert locks. TDPL 
 synchronized classes are a special case in that they would 
 provide a way in the language to associate a specific lock with 
 a specific set of variables in a way that the compiler could 
 then guarantee that it's safe to remove the outer layer of 
 shared within a synchronized function. Without a similar 
 mechanism to associate a mutex with one or more variables and 
 guarantee that that mutex is always locked when they're 
 accessed, the compiler won't be able to even know what to lock, 
 let alone that it's safe to remove shared within a particular 
 section of code.

 And the issue with what you're proposing with scope is that 
 unless the compiler can actually guarantee that no other 
 references to the shared object exist which could be used to 
 access the object at the same time, then the compiler cannot 
 safely remove shared even temporarily. scope is just enough to 
 guarantee that that particular function can't escape any 
 references, not enough to guarantee that they don't exist. So, 
 while features like scope can be used to make it easier to 
 reason about the code and cast away shared in a way that your 
 code is thread-safe, I fully expect that it's going to have to 
 be up to the programmer to know when it's safe to remove 
 shared, thus requiring a cast or some other  trusted mechanism 
 to temporarily remove shared. TDPL synchronized classes are the 
 only proposal I've seen that would be able to guarantee 
 thread-safety, and it can only do it for what's directly in the 
 class, making TDPL synchronized classes arguably pretty useless 
 (on top of the fact that it means requiring classes when most D 
 code wouldn't normally use classes for something like this).

 - Jonathan M Davis


Locks for classes can be done using the class monitor field (like 
synchronized works today). For other types the compiler could 
probably use the typeinfo class for locking, that might be to 
much for some cases. But there may be optimization opportunities, 
like for example use atomics when the target supports it and the 
type is a primitive, or assignment for the reference/pointer, 
same for binary ops when dereferencing a pointer to a primitive.

scope role is to ensure lock/unlock semantics and to help ellide 
superfluous locking when chaining the calls. I'm not proposing a 
ownership system here.
The shared castaway should be performed only inside the locked 
scope, the difference is that the compiler will automatically do 
that for you.

I think this should be enabled only for  safe functions, and not 
for  system.
May 15 2019
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, May 15, 2019 3:09:36 AM MDT Radu via Digitalmars-d wrote:

 On Wednesday, 15 May 2019 at 08:52:23 UTC, Jonathan M Davis wrote:

 On Wednesday, May 15, 2019 1:56:12 AM MDT Radu via

 Digitalmars-d wrote:

 On Tuesday, 14 May 2019 at 21:02:10 UTC, Jonathan M Davis

 wrote:

 On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via
 Digitalmars-d

 wrote:

 [...]


 Sure, it can guarantee that no reference will escape that
 function, but all that's required is that another reference
 to the same data exist elsewhere, and another thread could
 muck with the object while the mutex was locked. There's no
 question that helpers could be created which would help
 users avoid mistakes when casting when casting away shared,
 but the compiler can't actually make the guarantee that
 casting away shared is thread-safe.

 - Jonathan M Davis


 My view is that the compiler could automatically insert
 locking logic (ala synchronized) when the shared parameters
 gets references inside the function, and also automatically
 cast away shared so for the function internals it would be
 like working with local non-shared data.

 Given that compiler inferes lifetime, it could safely elide
 locking if the parameter is passed to other functions that
 have the same signature (scope is explicit or inferred).

 The idea is to provide the tools that simplify concurrent
 programming, the compiler will need to insert all the checks
 automatically using the lifetime tracking.


 The compiler does not have enough information to know which
 mutexes to use when even if we wanted it to insert locks. TDPL
 synchronized classes are a special case in that they would
 provide a way in the language to associate a specific lock with
 a specific set of variables in a way that the compiler could
 then guarantee that it's safe to remove the outer layer of
 shared within a synchronized function. Without a similar
 mechanism to associate a mutex with one or more variables and
 guarantee that that mutex is always locked when they're
 accessed, the compiler won't be able to even know what to lock,
 let alone that it's safe to remove shared within a particular
 section of code.

 And the issue with what you're proposing with scope is that
 unless the compiler can actually guarantee that no other
 references to the shared object exist which could be used to
 access the object at the same time, then the compiler cannot
 safely remove shared even temporarily. scope is just enough to
 guarantee that that particular function can't escape any
 references, not enough to guarantee that they don't exist. So,
 while features like scope can be used to make it easier to
 reason about the code and cast away shared in a way that your
 code is thread-safe, I fully expect that it's going to have to
 be up to the programmer to know when it's safe to remove
 shared, thus requiring a cast or some other  trusted mechanism
 to temporarily remove shared. TDPL synchronized classes are the
 only proposal I've seen that would be able to guarantee
 thread-safety, and it can only do it for what's directly in the
 class, making TDPL synchronized classes arguably pretty useless
 (on top of the fact that it means requiring classes when most D
 code wouldn't normally use classes for something like this).

 - Jonathan M Davis


 Locks for classes can be done using the class monitor field (like
 synchronized works today). For other types the compiler could
 probably use the typeinfo class for locking, that might be to
 much for some cases. But there may be optimization opportunities,
 like for example use atomics when the target supports it and the
 type is a primitive, or assignment for the reference/pointer,
 same for binary ops when dereferencing a pointer to a primitive.

 scope role is to ensure lock/unlock semantics and to help ellide
 superfluous locking when chaining the calls. I'm not proposing a
 ownership system here.
 The shared castaway should be performed only inside the locked
 scope, the difference is that the compiler will automatically do
 that for you.

 I think this should be enabled only for  safe functions, and not
 for  system.


The problem is that unless the compiler can absolutely guarantee that no
other references to the data exist, having the compiler automatically
removing shared at all - including in  safe code - can't be safely done.
Without an ownership model, the compiler basically doesn't have the
information that it needs to know that such references don't exist. Just
because code is  safe doesn't mean that it's thread-safe, and scope does not
provide enough information for the compiler to know that it's thread-safe.
Ultimately, you need a programmer to examine the code to verify the
thread-safety - which arguably means that anything that removes shared needs
to be  system so that the programmer knows that they need to verify the
code.

- Jonathan M Davis
May 15 2019