www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Shared Hell

reply "Denis Koroskin" <2korden gmail.com> writes:
I've recently updated to DMD2.035 (from DMD2.031 because all the later  
versions had issues with imports) and for the first time faced problems  
with shared modifier.

I don't need shared and all my globals are __gshared (they are globally  
unique instances that don't need per-thread copies).

Yet some of methods of the class hierarchy (a root singleton class and  
everything which is accessible through it) are synchronized (well, you  
know why). That's where the problems begin.

Marking a method as synchronized automatically makes it shared (more or  
less obvious). And marking the method shared makes it unable to invoke  
with non-shared instance (and __gshared != shared), meaning that I'm  
unable to use my __gshared variables anymore, making this attribute  
useless for any serious safe programming.

So I started with replacing __gshared with shared and quickly understood  
how viral it is. Not only you mast mark all the members shared (methods  
and field), instantiate classes with shared attribute, you also have to  
create a duplicate all the methods to make them accessible with both  
shared and non-shared (thread-local) instances:

class Array(T)
{
     const(T) opIndex(uint index) const
     {
         return data[index];
     }

     T opIndex(uint index)
     {
         return data[index];
     }

     const(T) opIndex(uint index) shared const
     {
         return data[index];
     }

     shared(T) opIndex(uint index) shared
     {
         return data[index];
     }

     private T[] data;
}

And that's just opIndex. Ooops...

But not only that, every interface now have to specify the same method  
twice, too:

interface Target
{
     bool build();
     bool build() shared;
     void clean();
     void clean() shared;
     bool ready();
     bool ready() shared;
     void setBuildListener(BuildListener buildListener);
     void setBuildListener(shared BuildListener buildListener) shared;
}

That's a bit frustrating. Most importantly, I don't even need shared  
(__gshared was more than enough for me), yet I'm imposed on using it.

Oh, I can't use any of the druntime/Phobos classes (e.g. create an  
instance of shared(Thread)), because none of them are shared-aware.

I think the design needs to be given a second thought before the plane  
takes off because I'm afraid it may painfully crash.
Oct 27 2009
next sibling parent #ponce <aliloko gmail.com> writes:
Denis Koroskin Wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later  
 versions had issues with imports) and for the first time faced problems  
 with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are globally  
 unique instances that don't need per-thread copies).
 
 Yet some of methods of the class hierarchy (a root singleton class and  
 everything which is accessible through it) are synchronized (well, you  
 know why). That's where the problems begin.
 
 Marking a method as synchronized automatically makes it shared (more or  
 less obvious). And marking the method shared makes it unable to invoke  
 with non-shared instance (and __gshared != shared), meaning that I'm  
 unable to use my __gshared variables anymore, making this attribute  
 useless for any serious safe programming.
 
 So I started with replacing __gshared with shared and quickly understood  
 how viral it is. Not only you mast mark all the members shared (methods  
 and field), instantiate classes with shared attribute, you also have to  
 create a duplicate all the methods to make them accessible with both  
 shared and non-shared (thread-local) instances:
 
 class Array(T)
 {
      const(T) opIndex(uint index) const
      {
          return data[index];
      }
 
      T opIndex(uint index)
      {
          return data[index];
      }
 
      const(T) opIndex(uint index) shared const
      {
          return data[index];
      }
 
      shared(T) opIndex(uint index) shared
      {
          return data[index];
      }
 
      private T[] data;
 }
 
 And that's just opIndex. Ooops...
 
 But not only that, every interface now have to specify the same method  
 twice, too:
 
 interface Target
 {
      bool build();
      bool build() shared;
      void clean();
      void clean() shared;
      bool ready();
      bool ready() shared;
      void setBuildListener(BuildListener buildListener);
      void setBuildListener(shared BuildListener buildListener) shared;
 }
 
 That's a bit frustrating. Most importantly, I don't even need shared  
 (__gshared was more than enough for me), yet I'm imposed on using it.
 
 Oh, I can't use any of the druntime/Phobos classes (e.g. create an  
 instance of shared(Thread)), because none of them are shared-aware.
 
 I think the design needs to be given a second thought before the plane  
 takes off because I'm afraid it may painfully crash.
Wow. I certainly won't switch to D2 if every getter must be written 4 times (and 4 times more with immutable(T) ?). I don't even understand the fundamental difference between __gshared and shared : is this only transitivity ?
Oct 28 2009
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Denis Koroskin wrote:
 I've recently updated to DMD2.035 (from DMD2.031 because all the later 
 versions had issues with imports) and for the first time faced problems 
 with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are globally 
 unique instances that don't need per-thread copies).
I don't understand. Are you running multiple threads? Are those threads accessing globals? A function that accesses shared data has to put in fences. There's no way to have the same code deal with shared and unshared code. As an escape from the type system, you can always cast away the shared-ness. But I wonder about code that both uses global variables shared across threads that don't need synchronization?
Oct 28 2009
next sibling parent Christopher Wright <dhasenan gmail.com> writes:
Walter Bright wrote:
 Denis Koroskin wrote:
 I've recently updated to DMD2.035 (from DMD2.031 because all the later 
 versions had issues with imports) and for the first time faced 
 problems with shared modifier.

 I don't need shared and all my globals are __gshared (they are 
 globally unique instances that don't need per-thread copies).
I don't understand. Are you running multiple threads? Are those threads accessing globals? A function that accesses shared data has to put in fences. There's no way to have the same code deal with shared and unshared code.
Acquiring a lock on a non-shared instance is safe, just an unnecessary expense. I would have looked into optimizing this expense away rather than punting the problem to the programmer.
 As an escape from the type system, you can always cast away the 
 shared-ness. But I wonder about code that both uses global variables 
 shared across threads that don't need synchronization?
Maybe the methods are mostly inherently threadsafe. Only a small portion requires locking, so it's more efficient to handle it manually.
Oct 28 2009
prev sibling next sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 28 Oct 2009 13:17:43 +0300, Walter Bright  
<newshound1 digitalmars.com> wrote:

 Denis Koroskin wrote:
 I've recently updated to DMD2.035 (from DMD2.031 because all the later  
 versions had issues with imports) and for the first time faced problems  
 with shared modifier.
  I don't need shared and all my globals are __gshared (they are  
 globally unique instances that don't need per-thread copies).
I don't understand. Are you running multiple threads? Are those threads accessing globals?
Yes.
 A function that accesses shared data has to put in fences. There's no  
 way to have the same code deal with shared and unshared code.
That's frustrating. I'd like to use the same class for both cases.
 But I wonder about code that both uses global variables shared across  
 threads that don't need synchronization?
You missed the point. I do the synchronization myself and I'm fine with switching to shared (I do believe it is a nice concept). The reason I use __gshared is because shared object were garbage-collected while still being in use a few versions of DMD back and I had no choice but to switch to __gshared. I hope it is fixed by now. But I still can't make my data shared, since shared is transitive (viral). After a few hours or work I still can't even compile my code.
 As an escape from the type system, you can always cast away the  
 shared-ness.
That's the only way I have now. Casts from shared to unshared *everywhere*: class BuildManager : BuildListener { synchronized void build(shared Target target) { // ... _buildingThread = new shared(Thread)(&_startBuild); // creating a new shared Thread. Yes, shared Thread, because BuildManager is global. //_buildingThread.start(); // Error: function core.thread.Thread.start () is not callable using argument types () shared (cast(Thread)_buildingThread).start(); // works, but ugly, and I don't have a reason to hijack the type system in this case // ... } } Andrei would suggest a Shared!(T) template that would wrap an unshared type and make all methods shared. This would work, but requires full AST manipulation capabilities (it's clearly not enough to just mark all the members shared). What should we do until then?
Oct 28 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Denis Koroskin wrote:
 On Wed, 28 Oct 2009 13:17:43 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 
 Denis Koroskin wrote:
 I've recently updated to DMD2.035 (from DMD2.031 because all the 
 later versions had issues with imports) and for the first time faced 
 problems with shared modifier.
  I don't need shared and all my globals are __gshared (they are 
 globally unique instances that don't need per-thread copies).
I don't understand. Are you running multiple threads? Are those threads accessing globals?
Yes.
 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.
That's frustrating. I'd like to use the same class for both cases.
 But I wonder about code that both uses global variables shared across 
 threads that don't need synchronization?
You missed the point. I do the synchronization myself and I'm fine with switching to shared (I do believe it is a nice concept). The reason I use __gshared is because shared object were garbage-collected while still being in use a few versions of DMD back and I had no choice but to switch to __gshared. I hope it is fixed by now.
Which OS are you using? This is definitely a bug. If it's still there, you can work around by adding the tls data as a "root" to the gc.
 But I still can't make my data shared, since shared is transitive 
 (viral). After a few hours or work I still can't even compile my code.
 
 As an escape from the type system, you can always cast away the 
 shared-ness.
That's the only way I have now. Casts from shared to unshared *everywhere*: class BuildManager : BuildListener { synchronized void build(shared Target target) { // ... _buildingThread = new shared(Thread)(&_startBuild); // creating a new shared Thread. Yes, shared Thread, because BuildManager is global. //_buildingThread.start(); // Error: function core.thread.Thread.start () is not callable using argument types () shared (cast(Thread)_buildingThread).start(); // works, but ugly, and I don't have a reason to hijack the type system in this case // ... } } Andrei would suggest a Shared!(T) template that would wrap an unshared type and make all methods shared. This would work, but requires full AST manipulation capabilities (it's clearly not enough to just mark all the members shared). What should we do until then?
shared(T) should transitively make a new type where it's all shared.
Oct 28 2009
parent "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 28 Oct 2009 20:30:45 +0300, Walter Bright  
<newshound1 digitalmars.com> wrote:

  Andrei would suggest a Shared!(T) template that would wrap an unshared  
 type and make all methods shared. This would work, but requires full  
 AST manipulation capabilities (it's clearly not enough to just mark all  
 the members shared). What should we do until then?
shared(T) should transitively make a new type where it's all shared.
Type, yes, but not the methods. It will make a type with *no* methods usable (because they still accept and operate on thread-local variables). I was hinting about template that would create a separate fully shared-aware type so that there would be no need for code duplication. I.e. it would transform the following class: class Float { this(float value) { this.value = value; } Float opAdd(Float other) { return new Vector(this.value + other.value); } private float value; } into the following: class SharedFloat { this(float value) shared { this.value = value; } shared Float opAdd(shared Float other) shared { return new shared Vector(this.value + other.value); } private shared float value; } This obviously requires techniques not available in D currently (AST manipulation).
Oct 28 2009
prev sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Walter Bright (newshound1 digitalmars.com)'s article
 Denis Koroskin wrote:
 I've recently updated to DMD2.035 (from DMD2.031 because all the later
 versions had issues with imports) and for the first time faced problems
 with shared modifier.

 I don't need shared and all my globals are __gshared (they are globally
 unique instances that don't need per-thread copies).
I don't understand. Are you running multiple threads? Are those threads accessing globals? A function that accesses shared data has to put in fences. There's no way to have the same code deal with shared and unshared code. As an escape from the type system, you can always cast away the shared-ness. But I wonder about code that both uses global variables shared across threads that don't need synchronization?
I have at least one use case for __gshareds in multithreaded code. I often use __gshared variables to hold program parameters that are only set using getopt at program startup and never modified after the program becomes multithreaded. That said, although I use D2 regularly, I basically have ignored shared's existence up to this point. The semantics aren't fully implemented, so right now you get all the bondage and discipline of it without any of the benefits. As far as the problem of synchronized methods automatically being shared, here's an easy workaround until the rough edges of shared are worked out: //Instead of this: synchronized SomeType someMethod(Foo args) { // Do stuff. } // Use this: SomeType someMethod(Foo args) { synchronized(this) { // Do stuff. } }
Oct 28 2009
parent "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 28 Oct 2009 16:19:11 +0300, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Walter Bright (newshound1 digitalmars.com)'s article
 Denis Koroskin wrote:
 I've recently updated to DMD2.035 (from DMD2.031 because all the later
 versions had issues with imports) and for the first time faced  
problems
 with shared modifier.

 I don't need shared and all my globals are __gshared (they are  
globally
 unique instances that don't need per-thread copies).
I don't understand. Are you running multiple threads? Are those threads accessing globals? A function that accesses shared data has to put in fences. There's no way to have the same code deal with shared and unshared code. As an escape from the type system, you can always cast away the shared-ness. But I wonder about code that both uses global variables shared across threads that don't need synchronization?
I have at least one use case for __gshareds in multithreaded code. I often use __gshared variables to hold program parameters that are only set using getopt at program startup and never modified after the program becomes multithreaded. That said, although I use D2 regularly, I basically have ignored shared's existence up to this point. The semantics aren't fully implemented, so right now you get all the bondage and discipline of it without any of the benefits. As far as the problem of synchronized methods automatically being shared, here's an easy workaround until the rough edges of shared are worked out: //Instead of this: synchronized SomeType someMethod(Foo args) { // Do stuff. } // Use this: SomeType someMethod(Foo args) { synchronized(this) { // Do stuff. } }
Yes, I've though about it. That's probably the only workaround for now and I'll give it. Thanks.
Oct 28 2009
prev sibling next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Denis Koroskin wrote:
 I've recently updated to DMD2.035 (from DMD2.031 because all the later 
 versions had issues with imports) and for the first time faced problems 
 with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are globally 
 unique instances that don't need per-thread copies).
 
 Yet some of methods of the class hierarchy (a root singleton class and 
 everything which is accessible through it) are synchronized (well, you 
 know why). That's where the problems begin.
 
 Marking a method as synchronized automatically makes it shared (more or 
 less obvious). And marking the method shared makes it unable to invoke 
 with non-shared instance (and __gshared != shared), meaning that I'm 
 unable to use my __gshared variables anymore, making this attribute 
 useless for any serious safe programming.
 
 So I started with replacing __gshared with shared and quickly understood 
 how viral it is. Not only you mast mark all the members shared (methods 
 and field), instantiate classes with shared attribute, you also have to 
 create a duplicate all the methods to make them accessible with both 
 shared and non-shared (thread-local) instances:
Why can't you use a non-shared method on a shared object? The compiler could insert locking on the caller side. Why can't you use a shared method on a non-shared object? The compiler could, as an optimization, duplicate the method, minus the synchronization. Or it could leave in the locking, which is expensive but correct.
Oct 28 2009
parent Jason House <jason.james.house gmail.com> writes:
Christopher Wright Wrote:

 Denis Koroskin wrote:
 I've recently updated to DMD2.035 (from DMD2.031 because all the later 
 versions had issues with imports) and for the first time faced problems 
 with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are globally 
 unique instances that don't need per-thread copies).
 
 Yet some of methods of the class hierarchy (a root singleton class and 
 everything which is accessible through it) are synchronized (well, you 
 know why). That's where the problems begin.
 
 Marking a method as synchronized automatically makes it shared (more or 
 less obvious). And marking the method shared makes it unable to invoke 
 with non-shared instance (and __gshared != shared), meaning that I'm 
 unable to use my __gshared variables anymore, making this attribute 
 useless for any serious safe programming.
 
 So I started with replacing __gshared with shared and quickly understood 
 how viral it is. Not only you mast mark all the members shared (methods 
 and field), instantiate classes with shared attribute, you also have to 
 create a duplicate all the methods to make them accessible with both 
 shared and non-shared (thread-local) instances:
Why can't you use a non-shared method on a shared object? The compiler could insert locking on the caller side. Why can't you use a shared method on a non-shared object? The compiler could, as an optimization, duplicate the method, minus the synchronization. Or it could leave in the locking, which is expensive but correct.
The caller would have to acquire locks for all the data accessed by the non-shared method and all non-shared methods it calls. Additionally, non-shared functions can access thread-local data. Neither of those issues are easily solved. Bartosz's scheme would solve the first one due to implied ownership.
Oct 28 2009
prev sibling next sibling parent Jason House <jason.james.house gmail.com> writes:
Denis Koroskin Wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later  
 versions had issues with imports) and for the first time faced problems  
 with shared modifier.
A quick trip over to bugzilla is all you need to see that shared is completely broken. Here's what I see as basic functionality that is broken: 3035, 3089, 3090, 3091, 3102, 3349. Half of those are bugs I created within an hour of trying to use shared for real. I also have a bug in druntime as well, but Batosz's rewrite should address that issue. Interestingly, even though I reported that bug long ago, it still bit me. It was the source of a segfault that took me 3 months to track down (mostly due to an inability to use gdb until recently). My code is littered with places that cast away shared since it was so utterly unusable when I tried. I'm still waiting for basic bugs to be closed before I try again.
Oct 28 2009
prev sibling next sibling parent Leandro Lucarella <llucax gmail.com> writes:
Denis Koroskin, el 28 de octubre a las 08:05 me escribiste:
 I've recently updated to DMD2.035 (from DMD2.031 because all the
 later versions had issues with imports) and for the first time faced
 problems with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are
 globally unique instances that don't need per-thread copies).
BTW, will __gshared and __traits be renamed to something that doesn't hurt my eyes before DMD2 is finalized or we will have to live with that until the end of the ages? I don't remember if I'm missing any other ugly identifier. -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- - Mire, don Inodoro! Una paloma con un anillo en la pata! Debe ser mensajera y cayó aquí! - Y... si no es mensajera es coqueta... o casada. -- Mendieta e Inodoro Pereyra
Oct 28 2009
prev sibling next sibling parent Kagamin <spam here.lot> writes:
Denis Koroskin Wrote:

 As an escape from the type system, you can always cast away the  
 shared-ness.
That's the only way I have now. Casts from shared to unshared *everywhere*: class BuildManager : BuildListener { synchronized void build(shared Target target) { // ... _buildingThread = new shared(Thread)(&_startBuild); // creating a new shared Thread. Yes, shared Thread, because BuildManager is global. //_buildingThread.start(); // Error: function core.thread.Thread.start () is not callable using argument types () shared (cast(Thread)_buildingThread).start(); // works, but ugly, and I don't have a reason to hijack the type system in this case // ... } }
you can use local variables to not have to cast in every statement. class BuildManager : BuildListener { synchronized void build(shared Target target) { // ... bt = new Thread(&_startBuild); _buildingThread = cast(shared Thread)bt; //store as shared bt.start(); //work with non-shared variable normally // ... } }
Oct 28 2009
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
Walter Bright Wrote:

 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.
Shared code perfectly deals with unshared data (it's not guaranteed that shared data is accessed by more than one thread). In other words unshared data is implicitly castable to shared.
Oct 29 2009
next sibling parent Kagamin <spam here.lot> writes:
Kagamin Wrote:

 Walter Bright Wrote:
 
 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.
Shared code perfectly deals with unshared data (it's not guaranteed that shared data is accessed by more than one thread). In other words unshared data is implicitly castable to shared.
Uh... this post was a little fast, ignore it, please.
Oct 29 2009
prev sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Kagamin wrote:
 Walter Bright Wrote:
 A function that accesses shared data has to put in fences. There's
 no way to have the same code deal with shared and unshared code.
Shared code perfectly deals with unshared data (it's not guaranteed that shared data is accessed by more than one thread). In other words unshared data is implicitly castable to shared.
Unfortunately, there is a subtle problem with that that makes such implicit casts unsafe. Shared data can be handed off to other threads. So if thread local data is implicitly cast to shared, then it can be handed off to other threads. Meaning it is no longer guaranteed to be thread local.
Oct 29 2009
prev sibling parent reply Kagamin <spam here.lot> writes:
Christopher Wright Wrote:

 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.
Acquiring a lock on a non-shared instance is safe, just an unnecessary expense. I would have looked into optimizing this expense away rather than punting the problem to the programmer.
Shared code can put data into *really* shared environment, which must not happen for unshared data.
Oct 29 2009
parent reply Christopher Wright <dhasenan gmail.com> writes:
Kagamin wrote:
 Christopher Wright Wrote:
 
 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.
Acquiring a lock on a non-shared instance is safe, just an unnecessary expense. I would have looked into optimizing this expense away rather than punting the problem to the programmer.
Shared code can put data into *really* shared environment, which must not happen for unshared data.
Okay, but the compiler can try copying the method, removing the 'shared' storage class, and running semantic on the result. If that succeeds, you're guaranteed not to be doing anything that would be unsafe unless you're casting, and you can call the shared method on a local instance. This lets you avoid whole program analysis and blacklisting unsafe operations. In this case, the compiler can also point to the line of code that prevents the method from being marked not shared.
Oct 29 2009
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 29 Oct 2009 07:25:20 -0400, Christopher Wright  
<dhasenan gmail.com> wrote:

 Kagamin wrote:
 Christopher Wright Wrote:

 A function that accesses shared data has to put in fences. There's no  
 way to have the same code deal with shared and unshared code.
Acquiring a lock on a non-shared instance is safe, just an unnecessary expense. I would have looked into optimizing this expense away rather than punting the problem to the programmer.
Shared code can put data into *really* shared environment, which must not happen for unshared data.
Okay, but the compiler can try copying the method, removing the 'shared' storage class, and running semantic on the result. If that succeeds, you're guaranteed not to be doing anything that would be unsafe unless you're casting, and you can call the shared method on a local instance. This lets you avoid whole program analysis and blacklisting unsafe operations. In this case, the compiler can also point to the line of code that prevents the method from being marked not shared.
Let's say your shared method your attempting to analyze calls an obscure shared method: int foo(shared C c); How do you determine if it can be called without the 'shared' qualifier? This is one of the problems with not importing from compiled interface files. -Steve
Oct 29 2009