www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Shared Hell

reply "Denis Koroskin" <2korden gmail.com> writes:
I've recently updated to DMD2.035 (from DMD2.031 because all the later  
versions had issues with imports) and for the first time faced problems  
with shared modifier.

I don't need shared and all my globals are __gshared (they are globally  
unique instances that don't need per-thread copies).

Yet some of methods of the class hierarchy (a root singleton class and  
everything which is accessible through it) are synchronized (well, you  
know why). That's where the problems begin.

Marking a method as synchronized automatically makes it shared (more or  
less obvious). And marking the method shared makes it unable to invoke  
with non-shared instance (and __gshared != shared), meaning that I'm  
unable to use my __gshared variables anymore, making this attribute  
useless for any serious safe programming.

So I started with replacing __gshared with shared and quickly understood  
how viral it is. Not only you mast mark all the members shared (methods  
and field), instantiate classes with shared attribute, you also have to  
create a duplicate all the methods to make them accessible with both  
shared and non-shared (thread-local) instances:

class Array(T)
{
     const(T) opIndex(uint index) const
     {
         return data[index];
     }

     T opIndex(uint index)
     {
         return data[index];
     }

     const(T) opIndex(uint index) shared const
     {
         return data[index];
     }

     shared(T) opIndex(uint index) shared
     {
         return data[index];
     }

     private T[] data;
}

And that's just opIndex. Ooops...

But not only that, every interface now have to specify the same method  
twice, too:

interface Target
{
     bool build();
     bool build() shared;
     void clean();
     void clean() shared;
     bool ready();
     bool ready() shared;
     void setBuildListener(BuildListener buildListener);
     void setBuildListener(shared BuildListener buildListener) shared;
}

That's a bit frustrating. Most importantly, I don't even need shared  
(__gshared was more than enough for me), yet I'm imposed on using it.

Oh, I can't use any of the druntime/Phobos classes (e.g. create an  
instance of shared(Thread)), because none of them are shared-aware.

I think the design needs to be given a second thought before the plane  
takes off because I'm afraid it may painfully crash.
Oct 27 2009
next sibling parent #ponce <aliloko gmail.com> writes:
Denis Koroskin Wrote:


 I've recently updated to DMD2.035 (from DMD2.031 because all the later  
 versions had issues with imports) and for the first time faced problems  
 with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are globally  
 unique instances that don't need per-thread copies).
 
 Yet some of methods of the class hierarchy (a root singleton class and  
 everything which is accessible through it) are synchronized (well, you  
 know why). That's where the problems begin.
 
 Marking a method as synchronized automatically makes it shared (more or  
 less obvious). And marking the method shared makes it unable to invoke  
 with non-shared instance (and __gshared != shared), meaning that I'm  
 unable to use my __gshared variables anymore, making this attribute  
 useless for any serious safe programming.
 
 So I started with replacing __gshared with shared and quickly understood  
 how viral it is. Not only you mast mark all the members shared (methods  
 and field), instantiate classes with shared attribute, you also have to  
 create a duplicate all the methods to make them accessible with both  
 shared and non-shared (thread-local) instances:
 
 class Array(T)
 {
      const(T) opIndex(uint index) const
      {
          return data[index];
      }
 
      T opIndex(uint index)
      {
          return data[index];
      }
 
      const(T) opIndex(uint index) shared const
      {
          return data[index];
      }
 
      shared(T) opIndex(uint index) shared
      {
          return data[index];
      }
 
      private T[] data;
 }
 
 And that's just opIndex. Ooops...
 
 But not only that, every interface now have to specify the same method  
 twice, too:
 
 interface Target
 {
      bool build();
      bool build() shared;
      void clean();
      void clean() shared;
      bool ready();
      bool ready() shared;
      void setBuildListener(BuildListener buildListener);
      void setBuildListener(shared BuildListener buildListener) shared;
 }
 
 That's a bit frustrating. Most importantly, I don't even need shared  
 (__gshared was more than enough for me), yet I'm imposed on using it.
 
 Oh, I can't use any of the druntime/Phobos classes (e.g. create an  
 instance of shared(Thread)), because none of them are shared-aware.
 
 I think the design needs to be given a second thought before the plane  
 takes off because I'm afraid it may painfully crash.


Wow.
I certainly won't switch to D2 if every getter must be written 4 times (and 4
times more with immutable(T) ?).

I don't even understand the fundamental difference between __gshared 
and shared : is this only transitivity ?
Oct 28 2009
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Denis Koroskin wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later 
 versions had issues with imports) and for the first time faced problems 
 with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are globally 
 unique instances that don't need per-thread copies).


I don't understand. Are you running multiple threads? Are those threads 
accessing globals?

A function that accesses shared data has to put in fences. There's no 
way to have the same code deal with shared and unshared code.

As an escape from the type system, you can always cast away the 
shared-ness. But I wonder about code that both uses global variables 
shared across threads that don't need synchronization?
Oct 28 2009
next sibling parent Christopher Wright <dhasenan gmail.com> writes:
Walter Bright wrote:

 Denis Koroskin wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later 
 versions had issues with imports) and for the first time faced 
 problems with shared modifier.

 I don't need shared and all my globals are __gshared (they are 
 globally unique instances that don't need per-thread copies).


 
 I don't understand. Are you running multiple threads? Are those threads 
 accessing globals?
 
 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.


Acquiring a lock on a non-shared instance is safe, just an unnecessary 
expense. I would have looked into optimizing this expense away rather 
than punting the problem to the programmer.


 As an escape from the type system, you can always cast away the 
 shared-ness. But I wonder about code that both uses global variables 
 shared across threads that don't need synchronization?


Maybe the methods are mostly inherently threadsafe. Only a small portion 
requires locking, so it's more efficient to handle it manually.
Oct 28 2009
prev sibling next sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 28 Oct 2009 13:17:43 +0300, Walter Bright  
<newshound1 digitalmars.com> wrote:


 Denis Koroskin wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later  
 versions had issues with imports) and for the first time faced problems  
 with shared modifier.
  I don't need shared and all my globals are __gshared (they are  
 globally unique instances that don't need per-thread copies).


 I don't understand. Are you running multiple threads? Are those threads  
 accessing globals?


Yes.


 A function that accesses shared data has to put in fences. There's no  
 way to have the same code deal with shared and unshared code.


That's frustrating. I'd like to use the same class for both cases.


 But I wonder about code that both uses global variables shared across  
 threads that don't need synchronization?


You missed the point. I do the synchronization myself and I'm fine with  
switching to shared (I do believe it is a nice concept). The reason I use  
__gshared is because shared object were garbage-collected while still  
being in use a few versions of DMD back and I had no choice but to switch  
to __gshared. I hope it is fixed by now.

But I still can't make my data shared, since shared is transitive (viral).  
After a few hours or work I still can't even compile my code.


 As an escape from the type system, you can always cast away the  
 shared-ness.


That's the only way I have now. Casts from shared to unshared *everywhere*:

class BuildManager : BuildListener
{
     synchronized void build(shared Target target)
     {
         // ...

         _buildingThread = new shared(Thread)(&_startBuild); // creating a  
new shared Thread. Yes, shared Thread, because BuildManager is global.

         //_buildingThread.start(); // Error: function  
core.thread.Thread.start () is not callable using argument types () shared
         (cast(Thread)_buildingThread).start(); // works, but ugly, and I  
don't have a reason to hijack the type system in this case

         // ...
     }
}

Andrei would suggest a Shared!(T) template that would wrap an unshared  
type and make all methods shared. This would work, but requires full AST  
manipulation capabilities (it's clearly not enough to just mark all the  
members shared). What should we do until then?
Oct 28 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Denis Koroskin wrote:

 On Wed, 28 Oct 2009 13:17:43 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 

 Denis Koroskin wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the 
 later versions had issues with imports) and for the first time faced 
 problems with shared modifier.
  I don't need shared and all my globals are __gshared (they are 
 globally unique instances that don't need per-thread copies).


 I don't understand. Are you running multiple threads? Are those 
 threads accessing globals?


 
 Yes.
 

 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.


 
 That's frustrating. I'd like to use the same class for both cases.
 

 But I wonder about code that both uses global variables shared across 
 threads that don't need synchronization?


 
 You missed the point. I do the synchronization myself and I'm fine with 
 switching to shared (I do believe it is a nice concept). The reason I 
 use __gshared is because shared object were garbage-collected while 
 still being in use a few versions of DMD back and I had no choice but to 
 switch to __gshared. I hope it is fixed by now.


Which OS are you using? This is definitely a bug. If it's still there, 
you can work around by adding the tls data as a "root" to the gc.



 But I still can't make my data shared, since shared is transitive 
 (viral). After a few hours or work I still can't even compile my code.
 

 As an escape from the type system, you can always cast away the 
 shared-ness.


 
 That's the only way I have now. Casts from shared to unshared *everywhere*:
 
 class BuildManager : BuildListener
 {
     synchronized void build(shared Target target)
     {
         // ...
 
         _buildingThread = new shared(Thread)(&_startBuild); // creating 
 a new shared Thread. Yes, shared Thread, because BuildManager is global.
 
         //_buildingThread.start(); // Error: function 
 core.thread.Thread.start () is not callable using argument types () shared
         (cast(Thread)_buildingThread).start(); // works, but ugly, and I 
 don't have a reason to hijack the type system in this case
 
         // ...
     }
 }
 
 Andrei would suggest a Shared!(T) template that would wrap an unshared 
 type and make all methods shared. This would work, but requires full AST 
 manipulation capabilities (it's clearly not enough to just mark all the 
 members shared). What should we do until then?


shared(T) should transitively make a new type where it's all shared.
Oct 28 2009
parent "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 28 Oct 2009 20:30:45 +0300, Walter Bright  
<newshound1 digitalmars.com> wrote:


  Andrei would suggest a Shared!(T) template that would wrap an unshared  
 type and make all methods shared. This would work, but requires full  
 AST manipulation capabilities (it's clearly not enough to just mark all  
 the members shared). What should we do until then?


 shared(T) should transitively make a new type where it's all shared.


Type, yes, but not the methods. It will make a type with *no* methods  
usable (because they still accept and operate on thread-local variables).

I was hinting about template that would create a separate fully  
shared-aware type so that there would be no need for code duplication.  
I.e. it would transform the following class:

class Float
{
     this(float value) { this.value = value; }

     Float opAdd(Float other)
     {
         return new Vector(this.value + other.value);
     }

     private float value;
}

into the following:

class SharedFloat
{
     this(float value) shared { this.value = value; }

     shared Float opAdd(shared Float other) shared
     {
         return new shared Vector(this.value + other.value);
     }

     private shared float value;
}

This obviously requires techniques not available in D currently (AST  
manipulation).
Oct 28 2009
prev sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Walter Bright (newshound1 digitalmars.com)'s article

 Denis Koroskin wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later
 versions had issues with imports) and for the first time faced problems
 with shared modifier.

 I don't need shared and all my globals are __gshared (they are globally
 unique instances that don't need per-thread copies).


 I don't understand. Are you running multiple threads? Are those threads
 accessing globals?
 A function that accesses shared data has to put in fences. There's no
 way to have the same code deal with shared and unshared code.
 As an escape from the type system, you can always cast away the
 shared-ness. But I wonder about code that both uses global variables
 shared across threads that don't need synchronization?


I have at least one use case for __gshareds in multithreaded code.  I often use
__gshared variables to hold program parameters that are only set using getopt at
program startup and never modified after the program becomes multithreaded.

That said, although I use D2 regularly, I basically have ignored shared's
existence up to this point.  The semantics aren't fully implemented, so right
now
you get all the bondage and discipline of it without any of the benefits.  As
far
as the problem of synchronized methods automatically being shared, here's an
easy
workaround until the rough edges of shared are worked out:

//Instead of this:
synchronized SomeType someMethod(Foo args) {
    // Do stuff.
}

// Use this:
SomeType someMethod(Foo args) {
    synchronized(this) {
        // Do stuff.
    }
}
Oct 28 2009
parent "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 28 Oct 2009 16:19:11 +0300, dsimcha <dsimcha yahoo.com> wrote:


 == Quote from Walter Bright (newshound1 digitalmars.com)'s article

 Denis Koroskin wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later
 versions had issues with imports) and for the first time faced  


 problems

 with shared modifier.

 I don't need shared and all my globals are __gshared (they are  


 globally

 unique instances that don't need per-thread copies).


 I don't understand. Are you running multiple threads? Are those threads
 accessing globals?
 A function that accesses shared data has to put in fences. There's no
 way to have the same code deal with shared and unshared code.
 As an escape from the type system, you can always cast away the
 shared-ness. But I wonder about code that both uses global variables
 shared across threads that don't need synchronization?


 I have at least one use case for __gshareds in multithreaded code.  I  
 often use
 __gshared variables to hold program parameters that are only set using  
 getopt at
 program startup and never modified after the program becomes  
 multithreaded.

 That said, although I use D2 regularly, I basically have ignored shared's
 existence up to this point.  The semantics aren't fully implemented, so  
 right now
 you get all the bondage and discipline of it without any of the  
 benefits.  As far
 as the problem of synchronized methods automatically being shared,  
 here's an easy
 workaround until the rough edges of shared are worked out:

 //Instead of this:
 synchronized SomeType someMethod(Foo args) {
     // Do stuff.
 }

 // Use this:
 SomeType someMethod(Foo args) {
     synchronized(this) {
         // Do stuff.
     }
 }


Yes, I've though about it. That's probably the only workaround for now and  
I'll give it. Thanks.
Oct 28 2009
prev sibling next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Denis Koroskin wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later 
 versions had issues with imports) and for the first time faced problems 
 with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are globally 
 unique instances that don't need per-thread copies).
 
 Yet some of methods of the class hierarchy (a root singleton class and 
 everything which is accessible through it) are synchronized (well, you 
 know why). That's where the problems begin.
 
 Marking a method as synchronized automatically makes it shared (more or 
 less obvious). And marking the method shared makes it unable to invoke 
 with non-shared instance (and __gshared != shared), meaning that I'm 
 unable to use my __gshared variables anymore, making this attribute 
 useless for any serious safe programming.
 
 So I started with replacing __gshared with shared and quickly understood 
 how viral it is. Not only you mast mark all the members shared (methods 
 and field), instantiate classes with shared attribute, you also have to 
 create a duplicate all the methods to make them accessible with both 
 shared and non-shared (thread-local) instances:


Why can't you use a non-shared method on a shared object? The compiler 
could insert locking on the caller side.

Why can't you use a shared method on a non-shared object? The compiler 
could, as an optimization, duplicate the method, minus the 
synchronization. Or it could leave in the locking, which is expensive 
but correct.
Oct 28 2009
parent Jason House <jason.james.house gmail.com> writes:
Christopher Wright Wrote:


 Denis Koroskin wrote:

 I've recently updated to DMD2.035 (from DMD2.031 because all the later 
 versions had issues with imports) and for the first time faced problems 
 with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are globally 
 unique instances that don't need per-thread copies).
 
 Yet some of methods of the class hierarchy (a root singleton class and 
 everything which is accessible through it) are synchronized (well, you 
 know why). That's where the problems begin.
 
 Marking a method as synchronized automatically makes it shared (more or 
 less obvious). And marking the method shared makes it unable to invoke 
 with non-shared instance (and __gshared != shared), meaning that I'm 
 unable to use my __gshared variables anymore, making this attribute 
 useless for any serious safe programming.
 
 So I started with replacing __gshared with shared and quickly understood 
 how viral it is. Not only you mast mark all the members shared (methods 
 and field), instantiate classes with shared attribute, you also have to 
 create a duplicate all the methods to make them accessible with both 
 shared and non-shared (thread-local) instances:


 
 Why can't you use a non-shared method on a shared object? The compiler 
 could insert locking on the caller side.
 
 Why can't you use a shared method on a non-shared object? The compiler 
 could, as an optimization, duplicate the method, minus the 
 synchronization. Or it could leave in the locking, which is expensive 
 but correct.


The caller would have to acquire locks for all the data accessed by the
non-shared method and all non-shared methods it calls. Additionally, non-shared
functions can access thread-local data. Neither of those issues are easily
solved. Bartosz's scheme would solve the first one due to implied ownership.
Oct 28 2009
prev sibling next sibling parent Jason House <jason.james.house gmail.com> writes:
Denis Koroskin Wrote:


 I've recently updated to DMD2.035 (from DMD2.031 because all the later  
 versions had issues with imports) and for the first time faced problems  
 with shared modifier.


A quick trip over to bugzilla is all you need to see that shared is completely
broken. Here's what I see as basic functionality that is broken: 3035, 3089,
3090, 3091, 3102, 3349.

Half of those are bugs I created within an hour of trying to use shared for
real. I also have a bug in druntime as well, but Batosz's rewrite should
address that issue. Interestingly, even though I reported that bug long ago, it
still bit me. It was the source of a segfault that took me 3 months to track
down (mostly due to an inability to use gdb until recently).

My code is littered with places that cast away shared since it was so utterly
unusable when I tried. I'm still waiting for basic bugs to be closed before I
try again.
Oct 28 2009
prev sibling next sibling parent Leandro Lucarella <llucax gmail.com> writes:
Denis Koroskin, el 28 de octubre a las 08:05 me escribiste:

 I've recently updated to DMD2.035 (from DMD2.031 because all the
 later versions had issues with imports) and for the first time faced
 problems with shared modifier.
 
 I don't need shared and all my globals are __gshared (they are
 globally unique instances that don't need per-thread copies).


BTW, will __gshared and __traits be renamed to something that doesn't hurt
my eyes before DMD2 is finalized or we will have to live with that until
the end of the ages? I don't remember if I'm missing any other ugly
identifier.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
- Mire, don Inodoro! Una paloma con un anillo en la pata! Debe ser
  mensajera y cayó aquí!
- Y... si no es mensajera es coqueta... o casada.
	-- Mendieta e Inodoro Pereyra
Oct 28 2009
prev sibling next sibling parent Kagamin <spam here.lot> writes:
Denis Koroskin Wrote:


 As an escape from the type system, you can always cast away the  
 shared-ness.


 
 That's the only way I have now. Casts from shared to unshared *everywhere*:
 
 class BuildManager : BuildListener
 {
      synchronized void build(shared Target target)
      {
          // ...
 
          _buildingThread = new shared(Thread)(&_startBuild); // creating a  
 new shared Thread. Yes, shared Thread, because BuildManager is global.
 
          //_buildingThread.start(); // Error: function  
 core.thread.Thread.start () is not callable using argument types () shared
          (cast(Thread)_buildingThread).start(); // works, but ugly, and I  
 don't have a reason to hijack the type system in this case
 
          // ...
      }
 }


you can use local variables to not have to cast in every statement.

class BuildManager : BuildListener
{
     synchronized void build(shared Target target)
     {
         // ...
         bt = new Thread(&_startBuild);
         _buildingThread = cast(shared Thread)bt; //store as shared
         bt.start(); //work with non-shared variable normally
         // ...
     }
}
Oct 28 2009
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
Walter Bright Wrote:


 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.


Shared code perfectly deals with unshared data (it's not guaranteed that shared
data is accessed by more than one thread). In other words unshared data is
implicitly castable to shared.
Oct 29 2009
next sibling parent Kagamin <spam here.lot> writes:
Kagamin Wrote:


 Walter Bright Wrote:
 

 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.


 
 Shared code perfectly deals with unshared data (it's not guaranteed that
shared data is accessed by more than one thread). In other words unshared data
is implicitly castable to shared.


Uh... this post was a little fast, ignore it, please.
Oct 29 2009
prev sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Kagamin wrote:

 Walter Bright Wrote:

 A function that accesses shared data has to put in fences. There's
 no way to have the same code deal with shared and unshared code.


 
 Shared code perfectly deals with unshared data (it's not guaranteed
 that shared data is accessed by more than one thread). In other words
 unshared data is implicitly castable to shared.


Unfortunately, there is a subtle problem with that that makes such 
implicit casts unsafe. Shared data can be handed off to other threads. 
So if thread local data is implicitly cast to shared, then it can be 
handed off to other threads. Meaning it is no longer guaranteed to be 
thread local.
Oct 29 2009
prev sibling parent reply Kagamin <spam here.lot> writes:
Christopher Wright Wrote:


 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.


 
 Acquiring a lock on a non-shared instance is safe, just an unnecessary 
 expense. I would have looked into optimizing this expense away rather 
 than punting the problem to the programmer.


Shared code can put data into *really* shared environment, which must not
happen for unshared data.
Oct 29 2009
parent reply Christopher Wright <dhasenan gmail.com> writes:
Kagamin wrote:

 Christopher Wright Wrote:
 

 A function that accesses shared data has to put in fences. There's no 
 way to have the same code deal with shared and unshared code.


 Acquiring a lock on a non-shared instance is safe, just an unnecessary 
 expense. I would have looked into optimizing this expense away rather 
 than punting the problem to the programmer.


 
 Shared code can put data into *really* shared environment, which must not
happen for unshared data.


Okay, but the compiler can try copying the method, removing the 'shared' 
storage class, and running semantic on the result. If that succeeds, 
you're guaranteed not to be doing anything that would be unsafe unless 
you're casting, and you can call the shared method on a local instance.

This lets you avoid whole program analysis and blacklisting unsafe 
operations.

In this case, the compiler can also point to the line of code that 
prevents the method from being marked not shared.
Oct 29 2009
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 29 Oct 2009 07:25:20 -0400, Christopher Wright  
<dhasenan gmail.com> wrote:


 Kagamin wrote:

 Christopher Wright Wrote:


 A function that accesses shared data has to put in fences. There's no  
 way to have the same code deal with shared and unshared code.


 Acquiring a lock on a non-shared instance is safe, just an unnecessary  
 expense. I would have looked into optimizing this expense away rather  
 than punting the problem to the programmer.


  Shared code can put data into *really* shared environment, which must  
 not happen for unshared data.


 Okay, but the compiler can try copying the method, removing the 'shared'  
 storage class, and running semantic on the result. If that succeeds,  
 you're guaranteed not to be doing anything that would be unsafe unless  
 you're casting, and you can call the shared method on a local instance.

 This lets you avoid whole program analysis and blacklisting unsafe  
 operations.

 In this case, the compiler can also point to the line of code that  
 prevents the method from being marked not shared.


Let's say your shared method your attempting to analyze calls an obscure  
shared method:

int foo(shared C c);

How do you determine if it can be called without the 'shared' qualifier?   
This is one of the problems with not importing from compiled interface  
files.

-Steve
Oct 29 2009