Jeremie Pelletier <jeremiep gmail.com> writes:

I decided to play once again with shared and see what 2.032 is capable of.
Turns out a lot of the previous issues I was having last time are gone,
however, there are still a few things left which prevent me from rewriting my
code.

The first issue that jumped to my face straight away was how 'shared const'
methods are not callable from 'shared' objects.

shared class Foo {
	void bar() const;
}
auto foo = new Foo; // foo is of type shared(Foo)
foo.bar; //  Error: function Foo.bar () shared const is not callable using
argument types () shared

Considering how 'const' methods can be called from mutable objects, this looks
like either a bug or a really awkward feature to me. Sending a shared(Foo) to a
method expecting a shared(const(Foo)) also triggers a similar error from the
compiler.

The other issue may be an intended feature, but it doesn't sound practical to
me. Marking a method as shared assumes all used properties in the method's
scope are also shared. Here is an example to illustrate my point:

class SimpleReader {
   this(LocalFile file) { _stream = new FileInputStream(file); }
   ...
private:
    synchronized void read(ubyte[] buf, long offset) {
        _stream.seek(offset);
        _stream.read(buf);
    }
    FileInputStream _stream;
}

The FileInputStream here is a generic blocking binary stream which is not
thread-safe by design. The reader is a composite class where every instance has
its own unique stream instance and use it to implement asynchronous reads over
the file format it abstracts, which in my case is a specialized read-only
archive using a lot of random accesses from different threads.

This is where the issue shows its ugly head. The 'synchronized' keyword tags
the read method as shared, which in itself is quite neat, what is annoying
however is that it also changes the type of _stream in the method's scope to
shared(FileInputStream) and therefore triggers compiler errors because
_stream.seek and _stream.read are not shared:

Error: function FileInputStream.read (ubyte[]) is not callable using argument
types (ubyte[]) shared

While it may be an attempt to keep shared usage safe, it isn't very practical.
The stream object here is not shared because it is not thread-safe. While it
may be used by different threads, it is unique to the reader's context and its
accesses are synchronized by the reader, the stream should therefore be
completely oblivious to the fact it is being used by different threads.

Maybe this could be the time to implement an unique qualifier; this is a
context where having _stream be of type unique(FileInputStream) would solve the
problem and allow further compiler optimizations. I don't know if it can be
done with templates, and without any overhead whatsoever. I know I would much
rather see unique(Foo) than Unique!Foo, and it would allow the use of 'is(foo :
unique)'.

Furthermore, tagging a method with shared does not make it thread-safe, it may
however use synchronized within its scope to protect its shared or unique data.
This may be confusing when calling shared methods vs calling synchronized
methods; one may think the shared one is not thread-safe and optionally
synchronize the call, resulting in another monitor being used for nothing, or
no monitor being used at all:

class Foo {
    shared void bar() {
        // Do stuff with local or immutable data
        synchronized(this) { /* do stuff with shared data */ }
    }
    shared void bar2() {
        // Do stuff on shared data
    }
}

Someone seeing only the prototype of Foo.bar may assume the method is not
thread-safe and call it as 'synchronized(foo) foo.bar()'. Just like they could
see the prototype of bar2 and assume it is thread-safe, calling it as
'foo.bar2()'.

What could be a good design against this sort of misleading behavior?

Phew, that's about enough issues and questions for now :)

Jason House <jason.james.house gmail.com> writes:

I'm glad to see I'm not the only one trying to use shared. I tried to use it
with 2.031 and rapidly hit bug after bug... I submitted several bug reports for
basic functionality, and none of it appeared in the changelog.

http://d.puremagic.com/issues/show_bug.cgi?id=3089
http://d.puremagic.com/issues/show_bug.cgi?id=3090
http://d.puremagic.com/issues/show_bug.cgi?id=3091



Jeremie Pelletier Wrote:

 I decided to play once again with shared and see what 2.032 is capable of.
Turns out a lot of the previous issues I was having last time are gone,
however, there are still a few things left which prevent me from rewriting my
code.
 
 The first issue that jumped to my face straight away was how 'shared const'
methods are not callable from 'shared' objects.
 
 shared class Foo {
 	void bar() const;
 }
 auto foo = new Foo; // foo is of type shared(Foo)
 foo.bar; //  Error: function Foo.bar () shared const is not callable using
argument types () shared
 
 Considering how 'const' methods can be called from mutable objects, this looks
like either a bug or a really awkward feature to me. Sending a shared(Foo) to a
method expecting a shared(const(Foo)) also triggers a similar error from the
compiler.
 
 The other issue may be an intended feature, but it doesn't sound practical to
me. Marking a method as shared assumes all used properties in the method's
scope are also shared. Here is an example to illustrate my point:
 
 class SimpleReader {
    this(LocalFile file) { _stream = new FileInputStream(file); }
    ...
 private:
     synchronized void read(ubyte[] buf, long offset) {
         _stream.seek(offset);
         _stream.read(buf);
     }
     FileInputStream _stream;
 }
 
 The FileInputStream here is a generic blocking binary stream which is not
thread-safe by design. The reader is a composite class where every instance has
its own unique stream instance and use it to implement asynchronous reads over
the file format it abstracts, which in my case is a specialized read-only
archive using a lot of random accesses from different threads.
 
 This is where the issue shows its ugly head. The 'synchronized' keyword tags
the read method as shared, which in itself is quite neat, what is annoying
however is that it also changes the type of _stream in the method's scope to
shared(FileInputStream) and therefore triggers compiler errors because
_stream.seek and _stream.read are not shared:
 
 Error: function FileInputStream.read (ubyte[]) is not callable using argument
types (ubyte[]) shared
 
 While it may be an attempt to keep shared usage safe, it isn't very practical.
The stream object here is not shared because it is not thread-safe. While it
may be used by different threads, it is unique to the reader's context and its
accesses are synchronized by the reader, the stream should therefore be
completely oblivious to the fact it is being used by different threads.
 
 Maybe this could be the time to implement an unique qualifier; this is a
context where having _stream be of type unique(FileInputStream) would solve the
problem and allow further compiler optimizations. I don't know if it can be
done with templates, and without any overhead whatsoever. I know I would much
rather see unique(Foo) than Unique!Foo, and it would allow the use of 'is(foo :
unique)'.
 
 Furthermore, tagging a method with shared does not make it thread-safe, it may
however use synchronized within its scope to protect its shared or unique data.
This may be confusing when calling shared methods vs calling synchronized
methods; one may think the shared one is not thread-safe and optionally
synchronize the call, resulting in another monitor being used for nothing, or
no monitor being used at all:
 
 class Foo {
     shared void bar() {
         // Do stuff with local or immutable data
         synchronized(this) { /* do stuff with shared data */ }
     }
     shared void bar2() {
         // Do stuff on shared data
     }
 }
 
 Someone seeing only the prototype of Foo.bar may assume the method is not
thread-safe and call it as 'synchronized(foo) foo.bar()'. Just like they could
see the prototype of bar2 and assume it is thread-safe, calling it as
'foo.bar2()'.
 
 What could be a good design against this sort of misleading behavior?
 
 Phew, that's about enough issues and questions for now :)

Graham St Jack <graham.stjack internode.on.net> writes:

I'm also having the same problems.

As Jeremie said, as soon as you start introducing shared methods (via 
synchronized for example), you rapidly get into trouble that can only be 
overcome by excessive casting.

It may be possible to contain the problem by refactoring multi-threaded 
code so that the shared objects are very small and simple, but even then 
the casting required is too much. This approach might be ok if you could 
define classes as being shared or immutable, and ALL instance of them 
were then implicitly shared or immutable. Also, immutable objects should 
be implicitly shareable.


On Sat, 12 Sep 2009 15:32:05 -0400, Jason House wrote:

 I'm glad to see I'm not the only one trying to use shared. I tried to
 use it with 2.031 and rapidly hit bug after bug... I submitted several
 bug reports for basic functionality, and none of it appeared in the
 changelog.
 
 http://d.puremagic.com/issues/show_bug.cgi?id=3089
 http://d.puremagic.com/issues/show_bug.cgi?id=3090
 http://d.puremagic.com/issues/show_bug.cgi?id=3091
 
 
 
 Jeremie Pelletier Wrote:
 
 I decided to play once again with shared and see what 2.032 is capable
 of. Turns out a lot of the previous issues I was having last time are
 gone, however, there are still a few things left which prevent me from
 rewriting my code.
 
 The first issue that jumped to my face straight away was how 'shared
 const' methods are not callable from 'shared' objects.
 
 shared class Foo {
 	void bar() const;
 }
 auto foo = new Foo; // foo is of type shared(Foo) foo.bar; //  Error:
 function Foo.bar () shared const is not callable using argument types
 () shared
 
 Considering how 'const' methods can be called from mutable objects,
 this looks like either a bug or a really awkward feature to me. Sending
 a shared(Foo) to a method expecting a shared(const(Foo)) also triggers
 a similar error from the compiler.
 
 The other issue may be an intended feature, but it doesn't sound
 practical to me. Marking a method as shared assumes all used properties
 in the method's scope are also shared. Here is an example to illustrate
 my point:
 
 class SimpleReader {
    this(LocalFile file) { _stream = new FileInputStream(file); } ...
 private:
     synchronized void read(ubyte[] buf, long offset) {
         _stream.seek(offset);
         _stream.read(buf);
     }
     FileInputStream _stream;
 }
 
 The FileInputStream here is a generic blocking binary stream which is
 not thread-safe by design. The reader is a composite class where every
 instance has its own unique stream instance and use it to implement
 asynchronous reads over the file format it abstracts, which in my case
 is a specialized read-only archive using a lot of random accesses from
 different threads.
 
 This is where the issue shows its ugly head. The 'synchronized' keyword
 tags the read method as shared, which in itself is quite neat, what is
 annoying however is that it also changes the type of _stream in the
 method's scope to shared(FileInputStream) and therefore triggers
 compiler errors because _stream.seek and _stream.read are not shared:
 
 Error: function FileInputStream.read (ubyte[]) is not callable using
 argument types (ubyte[]) shared
 
 While it may be an attempt to keep shared usage safe, it isn't very
 practical. The stream object here is not shared because it is not
 thread-safe. While it may be used by different threads, it is unique to
 the reader's context and its accesses are synchronized by the reader,
 the stream should therefore be completely oblivious to the fact it is
 being used by different threads.
 
 Maybe this could be the time to implement an unique qualifier; this is
 a context where having _stream be of type unique(FileInputStream) would
 solve the problem and allow further compiler optimizations. I don't
 know if it can be done with templates, and without any overhead
 whatsoever. I know I would much rather see unique(Foo) than Unique!Foo,
 and it would allow the use of 'is(foo : unique)'.
 
 Furthermore, tagging a method with shared does not make it thread-safe,
 it may however use synchronized within its scope to protect its shared
 or unique data. This may be confusing when calling shared methods vs
 calling synchronized methods; one may think the shared one is not
 thread-safe and optionally synchronize the call, resulting in another
 monitor being used for nothing, or no monitor being used at all:
 
 class Foo {
     shared void bar() {
         // Do stuff with local or immutable data synchronized(this) {
         /* do stuff with shared data */ }
     }
     shared void bar2() {
         // Do stuff on shared data
     }
 }
 
 Someone seeing only the prototype of Foo.bar may assume the method is
 not thread-safe and call it as 'synchronized(foo) foo.bar()'. Just like
 they could see the prototype of bar2 and assume it is thread-safe,
 calling it as 'foo.bar2()'.
 
 What could be a good design against this sort of misleading behavior?
 
 Phew, that's about enough issues and questions for now :)

Jeremie Pelletier <jeremiep gmail.com> writes:

Graham St Jack Wrote:

 I'm also having the same problems.
 
 As Jeremie said, as soon as you start introducing shared methods (via 
 synchronized for example), you rapidly get into trouble that can only be 
 overcome by excessive casting.
 
 It may be possible to contain the problem by refactoring multi-threaded 
 code so that the shared objects are very small and simple, but even then 
 the casting required is too much. This approach might be ok if you could 
 define classes as being shared or immutable, and ALL instance of them 
 were then implicitly shared or immutable. Also, immutable objects should 
 be implicitly shareable.

I agree, this is also one of my main concerns with shared in its current state.
It's an amazing and powerful concept and has the potential to make multi-thread
code much easier and safer to write. But all the required casting is killing
the safety, and makes it harder to use than it would be not having shared at
all. The lack of an unique qualifier certainly doesn't help either.

Unique data could only be used for aggregate properties, const/immutable data
would also be implicitly unique. This qualifier alone would simplify shared
quite a lot, allowing the use of unshared objects in shared contexts safely.

The compiler should make the distinction between shared code and shared data
and allow both shared and unshared instances to use shared methods, just like
both const and mutable instances may call const methods. An error should also
be triggered when calling a shared method of a shared object without
synchronization, and maybe have a __sync keyword to override this. If a
synchronized method is called from a non-shared object, no synchronization
takes place.

Allow me to illustrate my point with some code:

class Foo {
    int bar() shared { return a; }
    __sync bar2() { synchronized(this) return a; }
    synchronized void foo() { a = 1; }
    int a;
}
auto foo1 = new shared(Foo)();
auto foo2 = new Foo;

foo1.foo(); // ok, synchronized call
synchronized(foo1) foo1.foo(); // warning: recursive synchronization
foo2.foo(); // ok, unsynchronized call
synchronized(foo2) foo2.foo(); // ok synchronized call

foo1.bar(); // error, unsynchronized call to bar() shared
synchronized(foo1) foo1.bar(); // ok, synchronized call
foo2.bar(); // ok, unsynchronized call
synchronized(foo1) foo1.bar(); // ok, synchronized call

foo1.bar2(); // ok, method handles synchronized
synchronized(foo1) foo1.bar2(); // warning, recursive synchronization
foo2.bar2(); // ok, method handles synchronized, even on unshared object
synchronized(foo2) foo2.bar2(); // warning, recursive synchronization, even on
unshared object

That's about it, I believe this behavior would allow quite a number of
multi-threaded techniques to be easily implemented and documented. It would
only be the most natural thing since its quite similar to how const works.

"Robert Jacques" <sandford jhu.edu> writes:

On Sun, 13 Sep 2009 15:04:57 -0400, Jeremie Pelletier <jeremiep gmail.com>  
wrote:
[snip]
 Unique data could only be used for aggregate properties, const/immutable  
 data would also be implicitly unique. This qualifier alone would  
 simplify shared quite a lot, allowing the use of unshared objects in  
 shared contexts safely.

Neither const nor immutable data can be considered unique. First, any  
const data may be being mutated by another routine, so it can't be safely  
accessed without synchronization. Second, unique data is mutable while  
const/immutable data is not. Third, most implementations of unique allow  
for deterministic memory reclamation, which isn't possible if the unique  
data might actually be const/immutable.

 The compiler should make the distinction between shared code and shared  
 data and allow both shared and unshared instances to use shared methods,  
 just like both const and mutable instances may call const methods. An  
 error should also be triggered when calling a shared method of a shared  
 object without synchronization, and maybe have a __sync keyword to  
 override this. If a synchronized method is called from a non-shared  
 object, no synchronization takes place.

I think you have the wrong paradigm in mind. Shared and non-shared aren't  
mutable and const. They're mutable and immutable. From a technical  
perspective, synchronization of shared methods are handled by the callee,  
so there is no way not to call them and non-shared objects don't have a  
monitor that can be synchronized. Now you can have the compiler use the  
same code to generate two different object types (vtables, object layouts,  
etc) with have the same interface, but that doesn't sound like what you're  
suggesting.

 Allow me to illustrate my point with some code:

 class Foo {
     int bar() shared { return a; }
     __sync bar2() { synchronized(this) return a; }
     synchronized void foo() { a = 1; }
     int a;
 }
 auto foo1 = new shared(Foo)();
 auto foo2 = new Foo;

 foo1.foo(); // ok, synchronized call
 synchronized(foo1) foo1.foo(); // warning: recursive synchronization

Why a warning? Monitors are designed to handle recursive synchronization.

 foo2.foo(); // ok, unsynchronized call
 synchronized(foo2) foo2.foo(); // ok synchronized call

 foo1.bar(); // error, unsynchronized call to bar() shared
 synchronized(foo1) foo1.bar(); // ok, synchronized call
 foo2.bar(); // ok, unsynchronized call
 synchronized(foo1) foo1.bar(); // ok, synchronized call

 foo1.bar2(); // ok, method handles synchronized
 synchronized(foo1) foo1.bar2(); // warning, recursive synchronization
 foo2.bar2(); // ok, method handles synchronized, even on unshared object
 synchronized(foo2) foo2.bar2(); // warning, recursive synchronization,  
 even on unshared object

 That's about it, I believe this behavior would allow quite a number of  
 multi-threaded techniques to be easily implemented and documented. It  
 would only be the most natural thing since its quite similar to how  
 const works.

The major benefit of const isn't method declaration, but object use: i.e.  
only having to declare func(const T var) and not func(immutable T var) and  
func(T var). Currently, there's no planned type to fill this role though  
there have been some proposals.

P.S. Shouldn't 'a' be either private or protected?
P.S.S. Bartosz Milewski has a good series of blogs on multi-threading  
(with an eye on how to do it well in D).

Bike-shed: I've always preferred the CSP/pi-calculas term 'mobile' for the  
concept of 'unique'. I think mobile better expresses the concept with  
regard to multi-threading, where mobile is used to cheaply transfer data  
between threads (i.e. it moves around/can move between threads, but isn't  
shared between them). I find 'unique' to mainly convey the memory storage  
aspect of the concept, which is less important outside of C/C++.

Jeremie Pelletier <jeremiep gmail.com> writes:

Robert Jacques Wrote:

 On Sun, 13 Sep 2009 15:04:57 -0400, Jeremie Pelletier <jeremiep gmail.com>  
 wrote:
 [snip]
 Unique data could only be used for aggregate properties, const/immutable  
 data would also be implicitly unique. This qualifier alone would  
 simplify shared quite a lot, allowing the use of unshared objects in  
 shared contexts safely.

 
 Neither const nor immutable data can be considered unique. First, any  
 const data may be being mutated by another routine, so it can't be safely  
 accessed without synchronization. Second, unique data is mutable while  
 const/immutable data is not. Third, most implementations of unique allow  
 for deterministic memory reclamation, which isn't possible if the unique  
 data might actually be const/immutable.

Good points, I can only agree with you here. However I still believe immutable
data should be able to be used in shared contexts without being 'shared' or
protected by a monitor.

 The compiler should make the distinction between shared code and shared  
 data and allow both shared and unshared instances to use shared methods,  
 just like both const and mutable instances may call const methods. An  
 error should also be triggered when calling a shared method of a shared  
 object without synchronization, and maybe have a __sync keyword to  
 override this. If a synchronized method is called from a non-shared  
 object, no synchronization takes place.

 
 I think you have the wrong paradigm in mind. Shared and non-shared aren't  
 mutable and const. They're mutable and immutable. From a technical  
 perspective, synchronization of shared methods are handled by the callee,  
 so there is no way not to call them and non-shared objects don't have a  
 monitor that can be synchronized. Now you can have the compiler use the  
 same code to generate two different object types (vtables, object layouts,  
 etc) with have the same interface, but that doesn't sound like what you're  
 suggesting.

I know that shared/unshared is not const/mutable. What I meant is that right
now in D if a method is 'shared' it cannot be called from a non-shared object,
which makes unshared instance of the class unusable without plenty of dirty
casts. Take the following objects:

class Foo { void foo() const; }
class Bar { void bar() shared; }

Foo foo; foo.foo(); // ok, mutable object can call const method
Bar bar; bar.bar(); // error, unshared object may not call shared method

I had only presented the concept, your idea of using two virtual tables for
shared/unshared instances is also what I had in mind for the implementation,
and it would give exactly the behavior I had in mind. 

 Allow me to illustrate my point with some code:

 class Foo {
     int bar() shared { return a; }
     __sync bar2() { synchronized(this) return a; }
     synchronized void foo() { a = 1; }
     int a;
 }
 auto foo1 = new shared(Foo)();
 auto foo2 = new Foo;

 foo1.foo(); // ok, synchronized call
 synchronized(foo1) foo1.foo(); // warning: recursive synchronization

 
 Why a warning? Monitors are designed to handle recursive synchronization.

Its a performance issue that can easily be avoided, but still generates valid
code.
 
 foo2.foo(); // ok, unsynchronized call
 synchronized(foo2) foo2.foo(); // ok synchronized call

 foo1.bar(); // error, unsynchronized call to bar() shared
 synchronized(foo1) foo1.bar(); // ok, synchronized call
 foo2.bar(); // ok, unsynchronized call
 synchronized(foo1) foo1.bar(); // ok, synchronized call

 foo1.bar2(); // ok, method handles synchronized
 synchronized(foo1) foo1.bar2(); // warning, recursive synchronization
 foo2.bar2(); // ok, method handles synchronized, even on unshared object
 synchronized(foo2) foo2.bar2(); // warning, recursive synchronization,  
 even on unshared object

 That's about it, I believe this behavior would allow quite a number of  
 multi-threaded techniques to be easily implemented and documented. It  
 would only be the most natural thing since its quite similar to how  
 const works.

 
 The major benefit of const isn't method declaration, but object use: i.e.  
 only having to declare func(const T var) and not func(immutable T var) and  
 func(T var). Currently, there's no planned type to fill this role though  
 there have been some proposals.

I disagree, I think const methods are just as useful as const objects, since
they are the only methods that can be called on such objects. They do not
however prevent you from calling them on a mutable object. This is the behavior
I want with shared too; unshared objects should be able to call shared methods,
but shared objects should only be able to call shared methods.

 P.S. Shouldn't 'a' be either private or protected?

It should, but this was just an example ;)

 P.S.S. Bartosz Milewski has a good series of blogs on multi-threading  
 (with an eye on how to do it well in D).

I know, this is what sparked my interest for shared in the first place, I
really look forward to implement most of his ideas in my runtime, but I am
waiting until shared gets better semantics.

 Bike-shed: I've always preferred the CSP/pi-calculas term 'mobile' for the  
 concept of 'unique'. I think mobile better expresses the concept with  
 regard to multi-threading, where mobile is used to cheaply transfer data  
 between threads (i.e. it moves around/can move between threads, but isn't  
 shared between them). I find 'unique' to mainly convey the memory storage  
 aspect of the concept, which is less important outside of C/C++.

Maybe this is where 'volatile' could come back, from what I know it's still a
reserved keyword in D and would fit nicely this purpose.

Michel Fortin <michel.fortin michelf.com> writes:

On 2009-09-13 18:08:57 -0400, Jeremie Pelletier <jeremiep gmail.com> said:

 foo1.foo(); // ok, synchronized call
 synchronized(foo1) foo1.foo(); // warning: recursive synchronization

 
 Why a warning? Monitors are designed to handle recursive synchronization.

 
 Its a performance issue that can easily be avoided, but still generates 
 valid code.

Also, some people consider recursive locks as potential design flaws.
<http://landheer-cieslak.com/wordpress/?p=57>

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/

"Robert Jacques" <sandford jhu.edu> writes:

On Sun, 13 Sep 2009 18:08:57 -0400, Jeremie Pelletier <jeremiep gmail.com>  
wrote:

 Robert Jacques Wrote:

 On Sun, 13 Sep 2009 15:04:57 -0400, Jeremie Pelletier  
 <jeremiep gmail.com>
 wrote:
 [snip]
 Unique data could only be used for aggregate properties,  

 const/immutable
 data would also be implicitly unique. This qualifier alone would
 simplify shared quite a lot, allowing the use of unshared objects in
 shared contexts safely.

 Neither const nor immutable data can be considered unique. First, any
 const data may be being mutated by another routine, so it can't be  
 safely
 accessed without synchronization. Second, unique data is mutable while
 const/immutable data is not. Third, most implementations of unique allow
 for deterministic memory reclamation, which isn't possible if the unique
 data might actually be const/immutable.

 Good points, I can only agree with you here. However I still believe  
 immutable data should be able to be used in shared contexts without  
 being 'shared' or protected by a monitor.

One of the purposes behind immutable was lock-free access. As far as I  
know you can use immutable data in shared contexts today without any other  
modifiers. A quick test seems to indicate this works today, but if you've  
got a test case where it doesn't, I'd recommend filing it as a bug.

 The compiler should make the distinction between shared code and  

 shared
 data and allow both shared and unshared instances to use shared  

 methods,
 just like both const and mutable instances may call const methods. An
 error should also be triggered when calling a shared method of a  

 shared
 object without synchronization, and maybe have a __sync keyword to
 override this. If a synchronized method is called from a non-shared
 object, no synchronization takes place.

 I think you have the wrong paradigm in mind. Shared and non-shared  
 aren't
 mutable and const. They're mutable and immutable. From a technical
 perspective, synchronization of shared methods are handled by the  
 callee,
 so there is no way not to call them and non-shared objects don't have a
 monitor that can be synchronized. Now you can have the compiler use the
 same code to generate two different object types (vtables, object  
 layouts,
 etc) with have the same interface, but that doesn't sound like what  
 you're
 suggesting.

 I know that shared/unshared is not const/mutable. What I meant is that  
 right now in D if a method is 'shared' it cannot be called from a  
 non-shared object, which makes unshared instance of the class unusable  
 without plenty of dirty casts. Take the following objects:

 class Foo { void foo() const; }
 class Bar { void bar() shared; }

 Foo foo; foo.foo(); // ok, mutable object can call const method
 Bar bar; bar.bar(); // error, unshared object may not call shared method

 I had only presented the concept, your idea of using two virtual tables  
 for shared/unshared instances is also what I had in mind for the  
 implementation, and it would give exactly the behavior I had in mind.

Bartosz took the concept one step further: when declared as shared, all  
methods are implicitly wrapped in synchronize blocks. He then added a  
keyword for more manual, lock-free style programming. But this syntactic  
sugar isn't implemented yet.

 Allow me to illustrate my point with some code:

 class Foo {
     int bar() shared { return a; }
     __sync bar2() { synchronized(this) return a; }
     synchronized void foo() { a = 1; }
     int a;
 }
 auto foo1 = new shared(Foo)();
 auto foo2 = new Foo;

 foo1.foo(); // ok, synchronized call
 synchronized(foo1) foo1.foo(); // warning: recursive synchronization

 Why a warning? Monitors are designed to handle recursive  
 synchronization.

 Its a performance issue that can easily be avoided, but still generates  
 valid code.

Really? Every public method that calls another public method (of the same  
object) results in recursive synchronization. And if your example was  
longer than a one liner, you'd also have to have recursive  
synchronization. There are ways to reduce recursive synchronization, like  
public wrappers of protected/private methods, but they are not always  
appropriate or feasible for the use case. BTW, in general the threshold  
for what's a warning in DMD is generally a lot higher than other compilers  
(on the theory that if warnings are generated for every build you'll never  
read them)

[snip]

 Bike-shed: I've always preferred the CSP/pi-calculas term 'mobile' for  
 the
 concept of 'unique'. I think mobile better expresses the concept with
 regard to multi-threading, where mobile is used to cheaply transfer data
 between threads (i.e. it moves around/can move between threads, but  
 isn't
 shared between them). I find 'unique' to mainly convey the memory  
 storage
 aspect of the concept, which is less important outside of C/C++.

 Maybe this is where 'volatile' could come back, from what I know it's  
 still a reserved keyword in D and would fit nicely this purpose.

The volatile keyword has a very precise meaning in C/C++, which D altered  
and then abandoned. I think using it for the concept of mobile/unique  
would confusing. It also lacks any connotations related to a mobile/unique  
type. (i.e. I don't see the logic behind the choice, besides the keyword  
being unused)

Jeremie Pelletier <jeremiep gmail.com> writes:

Robert Jacques Wrote:

 On Sun, 13 Sep 2009 18:08:57 -0400, Jeremie Pelletier <jeremiep gmail.com>  
 wrote:
 
 Robert Jacques Wrote:

 On Sun, 13 Sep 2009 15:04:57 -0400, Jeremie Pelletier  
 <jeremiep gmail.com>
 wrote:
 [snip]
 Unique data could only be used for aggregate properties,  

 const/immutable
 data would also be implicitly unique. This qualifier alone would
 simplify shared quite a lot, allowing the use of unshared objects in
 shared contexts safely.

 Neither const nor immutable data can be considered unique. First, any
 const data may be being mutated by another routine, so it can't be  
 safely
 accessed without synchronization. Second, unique data is mutable while
 const/immutable data is not. Third, most implementations of unique allow
 for deterministic memory reclamation, which isn't possible if the unique
 data might actually be const/immutable.

 Good points, I can only agree with you here. However I still believe  
 immutable data should be able to be used in shared contexts without  
 being 'shared' or protected by a monitor.

 
 One of the purposes behind immutable was lock-free access. As far as I  
 know you can use immutable data in shared contexts today without any other  
 modifiers. A quick test seems to indicate this works today, but if you've  
 got a test case where it doesn't, I'd recommend filing it as a bug.

Oh yeah, I'm confusing it with 'shared' methods not able to call 'const shared'
methods, which is a pain in the ass :(

 The compiler should make the distinction between shared code and  

 shared
 data and allow both shared and unshared instances to use shared  

 methods,
 just like both const and mutable instances may call const methods. An
 error should also be triggered when calling a shared method of a  

 shared
 object without synchronization, and maybe have a __sync keyword to
 override this. If a synchronized method is called from a non-shared
 object, no synchronization takes place.

 I think you have the wrong paradigm in mind. Shared and non-shared  
 aren't
 mutable and const. They're mutable and immutable. From a technical
 perspective, synchronization of shared methods are handled by the  
 callee,
 so there is no way not to call them and non-shared objects don't have a
 monitor that can be synchronized. Now you can have the compiler use the
 same code to generate two different object types (vtables, object  
 layouts,
 etc) with have the same interface, but that doesn't sound like what  
 you're
 suggesting.

 I know that shared/unshared is not const/mutable. What I meant is that  
 right now in D if a method is 'shared' it cannot be called from a  
 non-shared object, which makes unshared instance of the class unusable  
 without plenty of dirty casts. Take the following objects:

 class Foo { void foo() const; }
 class Bar { void bar() shared; }

 Foo foo; foo.foo(); // ok, mutable object can call const method
 Bar bar; bar.bar(); // error, unshared object may not call shared method

 I had only presented the concept, your idea of using two virtual tables  
 for shared/unshared instances is also what I had in mind for the  
 implementation, and it would give exactly the behavior I had in mind.

 
 Bartosz took the concept one step further: when declared as shared, all  
 methods are implicitly wrapped in synchronize blocks. He then added a  
 keyword for more manual, lock-free style programming. But this syntactic  
 sugar isn't implemented yet.

The current D keywords (synchronized and shared) are already designed for that,
since synchronized implies shared. I don't want implicit synchronization, I'd
much rather have a shared class marking all its members/properties as shared
and letting me explicitely decide where the synchronization takes place.

 Allow me to illustrate my point with some code:

 class Foo {
     int bar() shared { return a; }
     __sync bar2() { synchronized(this) return a; }
     synchronized void foo() { a = 1; }
     int a;
 }
 auto foo1 = new shared(Foo)();
 auto foo2 = new Foo;

 foo1.foo(); // ok, synchronized call
 synchronized(foo1) foo1.foo(); // warning: recursive synchronization

 Why a warning? Monitors are designed to handle recursive  
 synchronization.

 Its a performance issue that can easily be avoided, but still generates  
 valid code.

 
 Really? Every public method that calls another public method (of the same  
 object) results in recursive synchronization. And if your example was  
 longer than a one liner, you'd also have to have recursive  
 synchronization. There are ways to reduce recursive synchronization, like  
 public wrappers of protected/private methods, but they are not always  
 appropriate or feasible for the use case. BTW, in general the threshold  
 for what's a warning in DMD is generally a lot higher than other compilers  
 (on the theory that if warnings are generated for every build you'll never  
 read them)

Its a behavior that is really easy to avoid, and therefore the overhead easily
avoided. The custom runtime I use doesn't use the reentrant attribute on
pthread's mutexes and recursing into a monitor triggers a runtime exception,
this is by design to better optimize the code. On Windows critical sections are
sadly always reentrant.

I haven't come across any case I wasn't able to easily design to avoid
recursive mutexes yet.

 [snip]
 
 Bike-shed: I've always preferred the CSP/pi-calculas term 'mobile' for  
 the
 concept of 'unique'. I think mobile better expresses the concept with
 regard to multi-threading, where mobile is used to cheaply transfer data
 between threads (i.e. it moves around/can move between threads, but  
 isn't
 shared between them). I find 'unique' to mainly convey the memory  
 storage
 aspect of the concept, which is less important outside of C/C++.

 Maybe this is where 'volatile' could come back, from what I know it's  
 still a reserved keyword in D and would fit nicely this purpose.

 
 The volatile keyword has a very precise meaning in C/C++, which D altered  
 and then abandoned. I think using it for the concept of mobile/unique  
 would confusing. It also lacks any connotations related to a mobile/unique  
 type. (i.e. I don't see the logic behind the choice, besides the keyword  
 being unused)

Ok, maybe not the greatest idea of all times, I agree :)

Jason House <jason.james.house gmail.com> writes:

Robert Jacques Wrote:

 On Sun, 13 Sep 2009 18:08:57 -0400, Jeremie Pelletier <jeremiep gmail.com>  
 wrote:
 .
 
 Bartosz took the concept one step further: when declared as shared, all  
 methods are implicitly wrapped in synchronize blocks. He then added a  
 keyword for more manual, lock-free style programming. But this syntactic  
 sugar isn't implemented yet.
 

That is not the design for D2. shared means shared. It is neither meant to mean
synchronized nor lockfree. I worry about optimization opportunities for shared
in D2. There may be way too many memory fences in synchronized code. Without a
mapping of a monitor to what's protected under a monitor, the
compiler/optimizer's hands are tied. At best, every object will be its own
monitor, but that hardly makes any sense...

"Robert Jacques" <sandford jhu.edu> writes:

On Mon, 14 Sep 2009 07:44:44 -0400, Jason House  
<jason.james.house gmail.com> wrote:

 Robert Jacques Wrote:

 On Sun, 13 Sep 2009 18:08:57 -0400, Jeremie Pelletier  
 <jeremiep gmail.com>
 wrote:
 .

 Bartosz took the concept one step further: when declared as shared, all
 methods are implicitly wrapped in synchronize blocks. He then added a
 keyword for more manual, lock-free style programming. But this syntactic
 sugar isn't implemented yet.

 That is not the design for D2. shared means shared. It is neither meant  
 to mean synchronized nor lockfree. I worry about optimization  
 opportunities for shared in D2. There may be way too many memory fences  
 in synchronized code. Without a mapping of a monitor to what's protected  
 under a monitor, the compiler/optimizer's hands are tied. At best, every  
 object will be its own monitor, but that hardly makes any sense...

That is the Java model by the way. And really, no one knows what the  
shared design is for D2. The keyword has been added, but it has yet to be  
fleshed out. And once you start talking about how to flesh it out and what  
kinds of syntactic sugar are wanted/needed you need to start looking at  
previous solutions, which is what Bartosz has done in his blog posts. The  
specific issue you raise, that of excessive monitors, was addressed using  
the concept of unique/mobile objects which are both thread-safe and don't  
require locking. However, it appears that this won't make it into D2,  
which I feel is a shame.

Graham St Jack <graham.stjack internode.on.net> writes:

So, what is the design of shared supposed to be then? Its time for Walter 
to buy in and tell us where this is all going - I for one am very 
confused right now.

Currently I am working around it by not using synchronized methods (I put 
synchronized blocks inside the methods), which is very bad form, but what 
else can I do?

Jeremie Pelletier <jeremiep gmail.com> writes:

Graham St Jack Wrote:

 So, what is the design of shared supposed to be then? Its time for Walter 
 to buy in and tell us where this is all going - I for one am very 
 confused right now.
 
 Currently I am working around it by not using synchronized methods (I put 
 synchronized blocks inside the methods), which is very bad form, but what 
 else can I do?

You're already more adventurous than I am, my current workaround has been to
drop all threading support in my runtime, I'm focusing on different parts of my
project that do not require threading for now and I'll piece it together when
shared gets closer to a final concept.

Jeremie Pelletier <jeremiep gmail.com> writes:

Jason House Wrote:

 I'm glad to see I'm not the only one trying to use shared. I tried to use it
with 2.031 and rapidly hit bug after bug... I submitted several bug reports for
basic functionality, and none of it appeared in the changelog.
 
 http://d.puremagic.com/issues/show_bug.cgi?id=3089

shared methods no longer trigger errors in 2.032, this issue should be marked
as fixed, it must be a side effect from fixing another issue.

 http://d.puremagic.com/issues/show_bug.cgi?id=3090

I just made a quick template this seems to work in 2.032:

immutable template isShared(T) {
	static if(is(T U : shared U))
		bool isShared = true;
	else
		bool isShared = false;
}

 http://d.puremagic.com/issues/show_bug.cgi?id=3091

This one still isn't solved, I too found it annoying that you cant use 'new
shared Foo()'. You can however declare Foo as 'shared class Foo', this works
just like 'immutable class' or 'const class' by marking all properties and
members with the qualifier. As a side note, I would like the same behavior for
'static class'.

However using 'shared class' is not always the wanted behavior, you may only
want a subset of the members and properties to be shared.

We also miss a unique qualifier to allow unshared objects to be used in shared
contexts without the need for shared methods.

It's also awkward to use, const members may be called from either const or
mutable objects. Shared members must be called from shared objects, so why
allow a class to have shared and unshared members, if all instances are going
to be shared anyways? It makes it much harder to draw the line between shared
and unshared; you may have only a few objects actually shared, but you are
required to make all the other objects they may use shared, even if they are
synchronized or unique. This makes all the other contexts these objects are
used in as shared, and soon your entire program is shared.

Jason House <jason.james.house gmail.com> writes:

Jeremie Pelletier Wrote:

 Jason House Wrote:
 
 I'm glad to see I'm not the only one trying to use shared. I tried to use it
with 2.031 and rapidly hit bug after bug... I submitted several bug reports for
basic functionality, and none of it appeared in the changelog.
 
 http://d.puremagic.com/issues/show_bug.cgi?id=3089

 
 shared methods no longer trigger errors in 2.032, this issue should be marked
as fixed, it must be a side effect from fixing another issue.
 

*sigh* 

Jason House <jason.james.house gmail.com> writes:

Graham St Jack Wrote:

 So, what is the design of shared supposed to be then? Its time for Walter 
 to buy in and tell us where this is all going - I for one am very 
 confused right now.

Here's what I know:
• Bartosz's ownership scheme is delayed until at least D3
• Shared code will be sequentially consistent
• Walter likes the idea of optimizing away memory barriers that the compiler
can prove are unneeded (some barriers in synchronized sections)
• Bartosz is rewriting how threads are done similar to what his blogs hint at
• Issues that Bartosz hits with shared are fixed immediately

Here's what I suspect from a number of emails:
• Because every class contains a monitor, Walter/dmd will treat every class as
its own monitor for the purposes of optimization.

I too wish Walter would advertise the design, but I think the simple fact is
that he doesn't know what the design is!

Graham St Jack <Graham.StJack internode.on.net> writes:

On Wed, 16 Sep 2009 08:00:40 -0400, Jason House wrote:

 Graham St Jack Wrote:
 
 So, what is the design of shared supposed to be then? Its time for
 Walter to buy in and tell us where this is all going - I for one am
 very confused right now.

 

Thanks for that. Its good to know that there is a plan in there 
somewhere, even if the details are still very fuzzy. I agree that the 
lofty goal of improving thread-safety for mere mortals is worthwhile, and 
that it won't be easy to pull off.

What I was really after though is what the plan is for D2 right now. The 
whole shared situation in D2 looks like a mess to me, and I would like 
some reassurance that something simple and tidy will be happening soon.



 Here's what I know:
 • Bartosz's ownership scheme is delayed until at least D3 • Shared 

code
 will be sequentially consistent • Walter likes the idea of optimizing
 away memory barriers that the compiler can prove are unneeded (some
 barriers in synchronized sections) • Bartosz is rewriting how threads
 are done similar to what his blogs hint at • Issues that Bartosz hits
 with shared are fixed immediately
 
 Here's what I suspect from a number of emails: • Because every class
 contains a monitor, Walter/dmd will treat every class as its own monitor
 for the purposes of optimization.
 
 I too wish Walter would advertise the design, but I think the simple
 fact is that he doesn't know what the design is!