www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Proposal to make "shared" (more) useful

reply Arafel <er.krali gmail.com> writes:
Hi all,

I know that many (most?) D users don't like using classes or old, 
manually controlled, concurrency using "shared" & co., but still, since 
they *are* in the language, I think they should at least be usable.

After having had my share (no pun intended) of problems using shared, 
I've finally settled for the following:

* Encapsulate all the shared stuff in classes (personal preference, 
easier to pass around).
* When possible, try to use "shared synchronized" classes, because even 
if there are potential losses of performance, the simplicity is often 
worth it. This mean that the classed is declared:

```
shared synchronized class A { }
```

and now, the important point:

* Make all _private non-reference fields_ of shared, synchronized 
classes __gshared.

AIUI the access of those fields is already guaranteed to be safe by the 
fact that *all* the methods of the class are already synchronized on 
"this", and nothing else can access them.

Of course, assuming you then don't escape references to them, but I 
think that would be a *really* silly thing to do, at least in the most 
common case... why on earth are they then private in the first place?.

Now, the question is, would it make sense to have the compiler do this 
for me in a transparent way? i.e. the compiler would automatically store 
private fields of shared *and* synchronized classes in the global storage.

Bonus points if it detects and forbids escaping references to them, 
although it could also be enough to warn the user.

This way I think there would an easy and sane way of using shared, 
because many of its worst quirks (for one, try using a struct like 
SysTime that overrides OpAssign, but not for shared objects, as a field) 
would be transparently dealt with.

A.
Sep 13 2018
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 13 September 2018 at 13:53:49 UTC, Arafel wrote:

 * Make all _private non-reference fields_ of shared, 
 synchronized classes __gshared.


so __gshared implies static. Are you sure that's what you want?
Sep 13 2018
parent reply Arafel <er.krali gmail.com> writes:
On 09/13/2018 04:27 PM, Adam D. Ruppe wrote:

 On Thursday, 13 September 2018 at 13:53:49 UTC, Arafel wrote:

 * Make all _private non-reference fields_ of shared, synchronized 
 classes __gshared.


 
 so __gshared implies static. Are you sure that's what you want?


Indeed it isn't! Why must __gshared be static?? (BTW, thanks a lot, you 
have just saved me a lot of debugging!!).

Then, how on earth are we supposed to have a struct like SysTime as a 
field in a shared class? Other than the "fun" of having a shared 
*pointer* to such a struct that then you can cast as non-shared as needed...

But let's say that this solution seems quite...... sub-optimal... at so 
many levels...
Sep 13 2018
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 13 September 2018 at 14:43:51 UTC, Arafel wrote:

  Why must __gshared be static?? (BTW, thanks a lot, you have 
 just saved me a lot of debugging!!).


The memory location differences of shared doesn't apply to class 
members. All members are stored with the instance, and shared 
only changes the type. (Contrast to global variables, where 
shared changes where they are stored - the default is to put them 
in thread-local storage, and shared moves it back out of that.)

Class static variables btw follow the same TLS rules. A static 
member is really the same as a global thing, just in a different 
namespace.


Now, the rule of __gshared is it puts it in that global memory 
storage using the unshared type. Unshared type you like here, but 
also, since normally, class members are stored in the object, 
changing the memory storage to the global shared area means it is 
no longer with the object... thus it becomes static.


 Then, how on earth are we supposed to have a struct like 
 SysTime as a field in a shared class? Other than the "fun" of 
 having a shared *pointer* to such a struct that then you can 
 cast as non-shared as needed...


What you want is an unshared type without changing the memory 
layout. There's no syntax for this at declaration, but there is 
one at usage point: you can cast away attributes on an lvalue:

shared class Foo {
         void update() {
                 // the cast below strips it of all attributes, 
including shared,
                 // allowing the assignment to succeed
                 cast() s = Clock.currTime;
         }
         SysTime s;
}



Using the private field with a public/protected/whatever accessor 
method, you can encapsulate this assignment a little and make it 
more sane to the outside world.
Sep 13 2018
parent Arafel <er.krali gmail.com> writes:
On 09/13/2018 05:11 PM, Adam D. Ruppe wrote:

 On Thursday, 13 September 2018 at 14:43:51 UTC, Arafel wrote:

  Why must __gshared be static?? (BTW, thanks a lot, you have just 
 saved me a lot of debugging!!).


 
 The memory location differences of shared doesn't apply to class 
 members. All members are stored with the instance, and shared only 
 changes the type. (Contrast to global variables, where shared changes 
 where they are stored - the default is to put them in thread-local 
 storage, and shared moves it back out of that.)
 
 Class static variables btw follow the same TLS rules. A static member is 
 really the same as a global thing, just in a different namespace.
 
 
 Now, the rule of __gshared is it puts it in that global memory storage 
 using the unshared type. Unshared type you like here, but also, since 
 normally, class members are stored in the object, changing the memory 
 storage to the global shared area means it is no longer with the 
 object... thus it becomes static.
 

 Then, how on earth are we supposed to have a struct like SysTime as a 
 field in a shared class? Other than the "fun" of having a shared 
 *pointer* to such a struct that then you can cast as non-shared as 
 needed...


 
 What you want is an unshared type without changing the memory layout. 
 There's no syntax for this at declaration, but there is one at usage 
 point: you can cast away attributes on an lvalue:
 
 shared class Foo {
          void update() {
                  // the cast below strips it of all attributes, 
 including shared,
                  // allowing the assignment to succeed
                  cast() s = Clock.currTime;
          }
          SysTime s;
 }
 
 
 
 Using the private field with a public/protected/whatever accessor 
 method, you can encapsulate this assignment a little and make it more 
 sane to the outside world.


Thanks a lot!! I remember having tried casting shared away, and ending 
up with a duplicate, but I have just tried it now and indeed it seems to 
work, will have to try with more complex use cases (comparing, adding 
dates and intervals, etc.), but it looks promising.

The problem might have been that I think I tried:

shared SysTime s_;
SysTime s = cast () s_; // Now I've got a duplicate! Ugh!

Because that works with classes... but (in hindsight) obviously not with 
value types.

I still think that it would be useful:

1) Allow __gshared for non-static fields to have this meaning, it would 
make it much more intuitive. A library solution is perhaps possible, but 
cumbersome.

2) Make it (sometimes) automatic as the original proposal.

Of course 1) is the most important part.

A.
Sep 13 2018
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
struct Unshared(T)
{
     private T value;
     T get() shared { return cast(T)value; }
     alias get this;
     void opAssign(T v) shared { value=cast(shared)v; }
}

shared synchronized class A
{
     private Unshared!(int[]) a;
     int[] f()
     {
         return a;
     }
}
Sep 13 2018
next sibling parent reply Arafel <er.krali gmail.com> writes:
On 09/13/2018 05:16 PM, Kagamin wrote:

 struct Unshared(T)
 {
      private T value;
      T get() shared { return cast(T)value; }
      alias get this;
      void opAssign(T v) shared { value=cast(shared)v; }
 }
 
 shared synchronized class A
 {
      private Unshared!(int[]) a;
      int[] f()
      {
          return a;
      }
 }
 


Doesn't work:

```
import std.datetime.systime;

struct Unshared(T)
{
     private T value;
     T get() shared { return cast(T)value; }
     alias get this;
     void opAssign(T v) shared { value=cast(shared)v; }
}

shared synchronized class A {
     private Unshared!SysTime t;

     this() {
         t = Clock.currTime;
     }
}

void main() {
     shared A a = new shared A;
}
```

Gives you:

onlineapp.d(6): Error: non-shared const method 
std.datetime.systime.SysTime.opCast!(SysTime).opCast is not callable 
using a shared mutable object
onlineapp.d(6):        Consider adding shared to 
std.datetime.systime.SysTime.opCast!(SysTime).opCast
onlineapp.d(8): Error: template std.datetime.systime.SysTime.opAssign 
cannot deduce function from argument types !()(shared(SysTime)) shared, 
candidates are:
/dlang/dmd/linux/bin64/../../src/phobos/std/datetime/systime.d(612): 
    std.datetime.systime.SysTime.opAssign()(auto ref const(SysTime) rhs)
onlineapp.d(12): Error: template instance `onlineapp.Unshared!(SysTime)` 
error instantiating
Sep 13 2018
parent reply Kagamin <spam here.lot> writes:
struct Unshared(T)
{
     private T value;
     this(T v) shared { opAssign(v); }
     T get() shared { return *cast(T*)&value; }
     alias get this;
     void opAssign(T v) shared { *cast(T*)&value=v; }
}

shared synchronized class A
{
     private Unshared!(int[]) a;
     private Unshared!SysTime t;
     this(){ t=Clock.currTime; }
     int[] f()
     {
         return a;
     }
}
Sep 14 2018
parent reply Arafel <er.krali gmail.com> writes:
On 09/14/2018 09:32 AM, Kagamin wrote:

 struct Unshared(T)
 {
      private T value;
      this(T v) shared { opAssign(v); }
      T get() shared { return *cast(T*)&value; }
      alias get this;
      void opAssign(T v) shared { *cast(T*)&value=v; }
 }
 
 shared synchronized class A
 {
      private Unshared!(int[]) a;
      private Unshared!SysTime t;
      this(){ t=Clock.currTime; }
      int[] f()
      {
          return a;
      }
 }


Almost there, you have to make "get" ref for it to work when calling 
methods that mutate the instance, though.


```
import std.datetime.systime;
import std.stdio;
import core.time;

shared struct Unshared(T)
{
     private T value;
     this(T v) shared { opAssign(v); }
     ref T get() shared { return *cast(T*)&value; }
     alias get this;
     void opAssign(T v) shared { *cast(T*)&value=v; }
}

shared synchronized class A
{
     private Unshared!(int[]) a;
     private Unshared!SysTime t;
     this(){ t=Clock.currTime; }
     int[] f()
     {
         return a;
     }
     void g() {
         writeln(t);
         t += 1.dur!"minutes"; // Doesn't work without "ref"
         writeln(t);
         t = t + 1.dur!"minutes";
         writeln(t);
     }
}

void main() {
     shared A a = new shared A;
     a.g;
}
```

Still, I don't know how to deal with structs with  disabled this() 
and/or specific constructors and other corner cases, it currently 
doesn't work 100% as it should:

```
import std.stdio;

struct S {
      disable this();
     int i;
     this(int i_) {
         i = 2 * i_;
     }

     void opAssign(int i_) {
         i = 2 * i_;
     }

     void f() {
         i *= 2;
     }
}

shared struct Unshared(T)
{
     private T value;
     this(T v) shared { *cast(T*)&value=v; }
     ref T get() shared { return *cast(T*)&value; }
     alias get this;
     void opAssign(T v) shared { *cast(T*)&value=v; }
}

shared synchronized class A
{
     private Unshared!S s; // Should this even be possible? What about 
the  disable this??
	// I would expect at least one, if not both of the following, to work
     //private Unshared!S s2 = S(1);
     //private Unshared!S s3 = 1;

     this(){
         s = S(1);
         //s = 1;
         s.f;
     }
     void f() {
         writeln(s.i);
     }
}

void main() {
     // This is rightly not possible:  disable this()
     // S s1;
     S s = 2; // This doesn't work in Unshared
     s = 1; // Neither does this
     s.f;
     writeln(s.i); // 4 as it should

     shared A a = new shared A;
     a.f; // 4 as it should
}
```

That's why I think it should be in the language itself, or at a minimum 
in Phobos once all the bugs are found and ironed out, if possible.

A.
Sep 14 2018
parent reply Kagamin <spam here.lot> writes:
On Friday, 14 September 2018 at 08:18:25 UTC, Arafel wrote:

 shared synchronized class A
 {
     private Unshared!S s; // Should this even be possible? What 
 about the  disable this??
 	// I would expect at least one, if not both of the following, 
 to work
     //private Unshared!S s2 = S(1);
     //private Unshared!S s3 = 1;

     this(){
         s = S(1);
         //s = 1;
         s.f;
     }
     void f() {
         writeln(s.i);
     }
 }


You can do
     private Unshared!(S*) s;
     this(){ s = new S(1); }
Sep 14 2018
parent reply Arafel <er.krali gmail.com> writes:
 
 You can do
      private Unshared!(S*) s;
      this(){ s = new S(1); }


Yeah, there are workarounds, also some other minor issues. For example, 
I wanted to use it with a pointer type, and then take the pointer of 
that (don't ask me: C library), and I had to find a workaround for this 
as well.

My point is that:

1) It's not transparent, not even remotely clear how to get this working.
2) It should be if shared is to be used. If shared is to be disowned / 
left as it is, then there needs to be an alternative to casting voodoo 
because right now doing "manual" multithreading is hell.

A.
Sep 14 2018
parent reply Kagamin <spam here.lot> writes:
On Friday, 14 September 2018 at 09:36:44 UTC, Arafel wrote:

 1) It's not transparent, not even remotely clear how to get 
 this working.
 2) It should be if shared is to be used. If shared is to be 
 disowned / left as it is, then there needs to be an alternative 
 to casting voodoo because right now doing "manual" 
 multithreading is hell.


If you prefer C-style multithreading, D supports it with 
__gshared. There's std.concurrency too.
Sep 14 2018
parent reply Arafel <er.krali gmail.com> writes:
 
 If you prefer C-style multithreading, D supports it with __gshared. 


As I recently discovered, "__gshared" means "static", so not so easy to 
use for class instance members. In fact, that's exactly what I'd like to 
have.
Sep 14 2018
parent reply Kagamin <spam here.lot> writes:
On Friday, 14 September 2018 at 11:13:00 UTC, Arafel wrote:

 As I recently discovered, "__gshared" means "static", so not so 
 easy to use for class instance members. In fact, that's exactly 
 what I'd like to have.


__gshared is for global storage. If you don't use global storage, 
you can simply not qualify anything shared, and you won't have to 
deal with it.
Sep 14 2018
parent reply Arafel <er.krali gmail.com> writes:
On 09/14/2018 01:37 PM, Kagamin wrote:

 On Friday, 14 September 2018 at 11:13:00 UTC, Arafel wrote:

 As I recently discovered, "__gshared" means "static", so not so easy 
 to use for class instance members. In fact, that's exactly what I'd 
 like to have.


 
 __gshared is for global storage. If you don't use global storage, you 
 can simply not qualify anything shared, and you won't have to deal with it.


Sure, then let's remove the "shared" keyword altogether.

Now, seriously, I understand that manually managed shared classes are 
not the preferred paradigm for many (most?) D programmers, and I'm not 
even against removing it altogether and then making it clear that it's 
not supported.

But in my view this situation where there *seems* to be support for it 
in the language, but it's just a minefield once you try, just gives a 
bad impression of the language.

Since I think this is commonly agreed, I was only trying to suggest a 
possible way to improve it (see my other messages in the thread), that's 
it.

I'll anyway keep working with (against) "shared" and finding workarounds 
because for me the benefits in other areas of the language compensate, 
but I'm sure many people won't, and it's a pity because I think it has 
the potential to become something useful.

A.
Sep 14 2018
parent Kagamin <spam here.lot> writes:
On Friday, 14 September 2018 at 12:00:19 UTC, Arafel wrote:

 Since I think this is commonly agreed, I was only trying to 
 suggest a possible way to improve it (see my other messages in 
 the thread), that's it.


Well, indeed synchronized classes are already treated a little 
special, e.g. they don't allow public fields. As they are 
unlikely to be used in low-level multithreading, I think their 
behavior can be changed to provide a middle ground between C and 
D concurrency - java style: 
https://docs.oracle.com/javase/tutorial/essential/concurrency/syncmeth.html
If `synchronized` class is changed to not imply `shared`, then it 
all can work.
1. `synchronized` class won't imply `shared` (java disallows 
synchronized constructors because it doesn't make sense)
2. methods of a synchronized class are callable on both shared 
and unshared instances
3. maybe even make it applicable only to class, not individual 
methods

AFAIK Andrei wanted some sort of compiler-assisted concurrency, 
maybe he will like such proposal.
Sep 14 2018
prev sibling parent Arafel <er.krali gmail.com> writes:
On 09/13/2018 05:16 PM, Kagamin wrote:

 struct Unshared(T)
 {
      private T value;
      T get() shared { return cast(T)value; }
      alias get this;
      void opAssign(T v) shared { value=cast(shared)v; }
 }
 
 shared synchronized class A
 {
      private Unshared!(int[]) a;
      int[] f()
      {
          return a;
      }
 }
 


My current attempt, still work in progress:


```
import std.stdio;
import std.datetime.systime;

shared struct GShared(T) {
	ubyte[T.sizeof] payload;

     this(T t) {
		*(cast(T*) &payload) = t;
	}
     this(shared T t) {
		*(cast(T*) &payload) = cast() t;
	}
	void opAssign(T t) {
  		*(cast(T*) &payload) = t;
	}
	void opAssign(shared T t) {
		*(cast(T*) &payload) = cast() t;
	}
     ref T data() {
         return *(cast(T*) &payload);
     }
     alias data this;
}

shared synchronized class A {
     this() {
	    t = Clock.currTime;
     }

     void printIt() {
         writeln(t);
     }

     private:
     GShared!SysTime t;
}

void main() {
     shared A a = new shared A;
     a.printIt;
}
```
Sep 13 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, September 13, 2018 7:53:49 AM MDT Arafel via Digitalmars-d 
wrote:

 Hi all,

 I know that many (most?) D users don't like using classes or old,
 manually controlled, concurrency using "shared" & co., but still, since
 they *are* in the language, I think they should at least be usable.

 After having had my share (no pun intended) of problems using shared,
 I've finally settled for the following:

 * Encapsulate all the shared stuff in classes (personal preference,
 easier to pass around).
 * When possible, try to use "shared synchronized" classes, because even
 if there are potential losses of performance, the simplicity is often
 worth it. This mean that the classed is declared:

 ```
 shared synchronized class A { }
 ```

 and now, the important point:

 * Make all _private non-reference fields_ of shared, synchronized
 classes __gshared.

 AIUI the access of those fields is already guaranteed to be safe by the
 fact that *all* the methods of the class are already synchronized on
 "this", and nothing else can access them.

 Of course, assuming you then don't escape references to them, but I
 think that would be a *really* silly thing to do, at least in the most
 common case... why on earth are they then private in the first place?.

 Now, the question is, would it make sense to have the compiler do this
 for me in a transparent way? i.e. the compiler would automatically store
 private fields of shared *and* synchronized classes in the global storage.

 Bonus points if it detects and forbids escaping references to them,
 although it could also be enough to warn the user.


Have you read the concurrency chapter in The D Programming Language by
Andrei? It sounds like you're trying to describe something vere similar to
the synchronized classes from TDPL (which have never been fully implemented
in the language). They would make it so that you had a class with shared
members but where the outer layer of shared was stripped away inside member
functions, because the compiler is able to guarantee that they don't escape
(though it can only guarantee that for the outer layer). Every member
function is synchronized and no direct access to the member variables
outside of the class (even in the same module) is allowed. It would make
shared easier to use in those cases where it makes sense to wrapped
everything protected by a mutex in a class (though since it can only safely
strip away the outer layer of shared, it's more limited than would be nice,
and there are plenty of cases where it doesn't make sense to stuff something
in a class just to use it as shared).


 This way I think there would an easy and sane way of using shared,
 because many of its worst quirks (for one, try using a struct like
 SysTime that overrides OpAssign, but not for shared objects, as a field)
 would be transparently dealt with.


The fact that most operations are not allowed with shared is _on purpose_.
If anything, too many operations are currently legal. What's really supposed
to be happening is that every single operation on a shared object is either
guaranteed to be thread-safe, or it's illegal. And if it's illegal, that
means that you either need to use atomics to do an operation (since they're
thread-safe), or you need to protect the shared object with a mutex and
temporarily cast away shared while the mutex is locked so that you can
actually do something with the object - and then make sure that no
thread-local references exist when the mutex is released.

Something like copying a shared object shouldn't even be legal in general.
An object that defines opAssign prevents it now, but the fact that it's
legal on any type where copying is not guaranteed to be thread-safe is a
bug. It's one of those details of shared that has never been fully fleshed
out like it should be. Walter and Andrei have been discussing finishing
shared, but it hasn't been a high enough priority for it actually get fully
sorted out yet. Once it is, unless you're dealing with a type that isn't
guaranteed to be thread-safe when copying it, it won't be legal copy it
without first casting away shared. Anything less than that would violate
what shared is supposed to do.

What you should be thinking when dealing with any shared object and whether
a particular operation should be allowed is whether that operation is
guaranteed to be thread-safe. If the compiler can't guarantee that the
operation is thread-safe, then it's not supposed to be legal. The main area
that Walter and Andrei haven't agreed upon yet is how much the compiler can
or should do to ensure that something is thread-safe rather than just making
an operation illegal (e.g. whether memory barriers should be involved). So,
_maybe_ some operations will end up as legal thanks to the compiler adding
extra code to do something to ensure thread-safety, but in most situations,
it's just going to be illegal.

So, ultimately, every type is either going to need to be designed such that
it simply does not work as shared, or it manages the thread-safety stuff for
you. If the object is not designed to be used as shared, then that means
that if you want to, you need to protect it with a mutex (be that with
synchronized or directly using mutexes) and cast away shared correctly when
the object is protected by the mutex. It's annoying, but it prevents
thread-safety bugs, and it allows the compiler (and the programmer) to treat
the rest of the program as thread-local. On the other hand, if the type is
designed to be used as shared (so it actually has shared member functions),
then that means that the type itself is going to need to deal with all of
the thread-safety stuff internally (be that by using mutexes and casting or
using atomics or whatever).

If we're going to find ways to make shared require less manual work, it
means finding a way to protect a shared object (or group of shared objects)
with a mutex in a way that is able to guarantee that when you operate on the
data, it's protected by that mutex and that no reference to that data has
escaped. TDPL's synchronized classes are one attempt to do that, but the
requirement that no references escape (so that shared can safely be cast
away) makes it so that only the outer layer of shared can be cast away, and
it's extremely difficult to do better than that with having holes such that
it isn't actually guaranteed to be thread-safe when shared is cast away.
Maybe someone will come up with something that will work, but I wouldn't bet
on it. Either way, I don't see how any solution is going to be acceptable
which does not actually guarantee thread-safety, because it would be
violating the guarantees of shared otherwise. A programmer can choose to
cast away shared in an unsafe manner (or use __gshared) and rely on their
ability to ensure that the code is thread-safe rather than letting shared do
its job, but that's not the sort of thing that we're going to do with a
language construct, and given that the compiler assumes that anything that
isn't shared or immutable is thread-local, it's very much a risky thing to
do.

As for __gshared, it's intended specifically for C globals, and using it for
anything else is just begging for bugs. Because the compiler assumes that
anything which is not marked as shared or immutable is thread-local, having
such an object actually be able to be mutated by another thread risks subtle
bugs of the sort that shared was supposed to prevent in the first place.
Unfortunately, due to some of the difficulties in using shared and some of
the misunderstandings about it, a number of folks have just used __gshared
instead of shared, but once you do that, you're risking subtle bugs, because
that's not at all what __gshared is intended for. If you're using __gshared
for anything other than a C global, it's arguably a bug. Certainly, it's a
risky proposition.

- Jonathan M Davis
Sep 13 2018
parent Arafel <er.krali gmail.com> writes:
On 09/13/2018 09:49 PM, Jonathan M Davis wrote:

 
 Have you read the concurrency chapter in The D Programming Language by
 Andrei? It sounds like you're trying to describe something vere similar to
 the synchronized classes from TDPL (which have never been fully implemented
 in the language). They would make it so that you had a class with shared
 members but where the outer layer of shared was stripped away inside member
 functions, because the compiler is able to guarantee that they don't escape
 (though it can only guarantee that for the outer layer). Every member
 function is synchronized and no direct access to the member variables
 outside of the class (even in the same module) is allowed. It would make
 shared easier to use in those cases where it makes sense to wrapped
 everything protected by a mutex in a class (though since it can only safely
 strip away the outer layer of shared, it's more limited than would be nice,
 and there are plenty of cases where it doesn't make sense to stuff something
 in a class just to use it as shared).
 


I hadn't read the book, but that's indeed the gist of what I'm 
proposing. I think it could be enough to restrict it to value types, 
where it's easier to assume (and even check) that there are no external 
references.


 
 [snip]
 
 If we're going to find ways to make shared require less manual work, it
 means finding a way to protect a shared object (or group of shared objects)
 with a mutex in a way that is able to guarantee that when you operate on the
 data, it's protected by that mutex and that no reference to that data has
 escaped. TDPL's synchronized classes are one attempt to do that, but the
 requirement that no references escape (so that shared can safely be cast
 away) makes it so that only the outer layer of shared can be cast away, and
 it's extremely difficult to do better than that with having holes such that
 it isn't actually guaranteed to be thread-safe when shared is cast away.
 Maybe someone will come up with something that will work, but I wouldn't bet
 on it. Either way, I don't see how any solution is going to be acceptable
 which does not actually guarantee thread-safety, because it would be
 violating the guarantees of shared otherwise. A programmer can choose to
 cast away shared in an unsafe manner (or use __gshared) and rely on their
 ability to ensure that the code is thread-safe rather than letting shared do
 its job, but that's not the sort of thing that we're going to do with a
 language construct, and given that the compiler assumes that anything that
 isn't shared or immutable is thread-local, it's very much a risky thing to
 do.
 


I completely agree with this argument, however please note that there 
must be a sensible way to work with shared, otherwise we enter in the 
"the perfect is the enemy of the good" area.

For reference types it's somehow workable, because you can just cast 
away and store it in a new variable:

```
class A {
	this() { }
}

shared synchronized class B {
	this(A a) {
		a_ = cast (shared) new A; // no shared this()
	}
	void foo() {
		A a = cast () a_;
		// Work with it
	}
	private:
	A a_;
}
```

It's still somewhat cumbersome, specially if you have many such members, 
but still doable.

However, this is not possible for value types, and it makes it nigh on 
impossible to work with them in a sensible way. You have either to use 
pointers, or cast away every type you want to use it. None of them are 
what I would call "practical".

While not the biggest problem (see the later point), I still think that 
synchronized classes are a good compromise, specially with the 
restriction of only applying to full value types (no internal references 
allowed). Of course it is still perhaps possible to bypass that 
mechanism, but so is the case with many other ones (assumeUnique?).

If it's hard enough to do by mistake, it can be assumed that the people 
messing with it should know what they are doing.

Finally, you suggest using __gshared, and I'm not sure you're not having 
the same misunderstanding I had: __gshared implies "static", so it's not 
a valid solution for class fields in most cases.


 As for __gshared, it's intended specifically for C globals, and using it for
 anything else is just begging for bugs. Because the compiler assumes that
 anything which is not marked as shared or immutable is thread-local, having
 such an object actually be able to be mutated by another thread risks subtle
 bugs of the sort that shared was supposed to prevent in the first place.
 Unfortunately, due to some of the difficulties in using shared and some of
 the misunderstandings about it, a number of folks have just used __gshared
 instead of shared, but once you do that, you're risking subtle bugs, because
 that's not at all what __gshared is intended for. If you're using __gshared
 for anything other than a C global, it's arguably a bug. Certainly, it's a
 risky proposition.
 


As I said, the current semantics of __gshared doesn't allow it to be a 
"drop-in" replacement of "shared". I also agree that it's not what it 
was meant for, and that changing that right now would risk breaking a 
lot of code.

However, I think that there should be *some* way in the language itself 
to express that without having to cast all over the place: access this 
member of the shared class as if it were local, I'll take care of 
controlling the access to it.

You can use a wrapper type, and that's what I'm trying to do right now, 
but I'm pretty sure there will be a ton of corner cases and interactions 
that will make it really hard to work reliably in a generic way.

Then, implementing a kind of "synchronized classes" that would automate 
this when possible would of course be a further and welcome improvement.

A.
Sep 14 2018