www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - std.concurrency and immutable

reply "Antoche" <a a.a> writes:
The following code compiles but doesn't work as expected:

      import std.stdio;
      import std.concurrency;

      class A
      {
          this() immutable {}
      }

      void main()
      {
          auto tid = spawn( &fooBar, thisTid );
          while(true)
          {
              receive(
                  (Variant any) {
                      writeln( "Received a variant" );
                      writeln( "Received ", any );
                  }
              );
          }
      }

      void fooBar( Tid masterTid )
      {
          scope(failure) writeln( "fooBar failed" );
          scope(success) writeln( "fooBar succeeded" );
          scope(exit) writeln( "fooBar exiting" );
          try
          {
              immutable A b = new immutable A();
              masterTid.send( 42 ); // This works
              masterTid.send( b );  // This doesn't
          }
          catch( Exception e )
          {
              writeln( "Exception received" );
          }
      }


I see this in the console:

      fooBar exiting
      fooBar failed
      Received a variant
      Received 42

(then it just hangs)

I'm especially puzzled by:
   * Sending an int as a message works but not an immutable object.
Wasn't this  (safely sharing objects across threads) one of the
basic use cases for the immutable type qualifier?
   * scope(failure) failed but my exception handler didn't catch
anything. How is this possible? What could cause that?
Assertions/abort?

There seemed to be a 3-year-old ticket on this issue
(http://d.puremagic.com/issues/show_bug.cgi?id=5538) with very
little activity, which is a bit surprising given how much
emphasis is given to this feature (the D homepage mentions "D
offers an innovative approach to concurrency, featuring true
immutable data, message passing, no sharing by default, and
controlled mutable sharing across threads", and TDPL devotes a
whole chapter on it). This thread
(http://forum.dlang.org/thread/kgk8hc$12fa$1 digitalmars.com)
also says std.concurrency is "very buggy".

If I can't use std.concurrency, is there any other safe
alternative for multithreaded programming with D?

Thanks,
A.
Nov 24 2013
next sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-25 06:03:27 +0000, Antoche said:


 The following code compiles but doesn't work as expected:
 
       import std.stdio;
       import std.concurrency;
 
       class A
       {
           this() immutable {}
       }
 
       void main()
       {
           auto tid = spawn( &fooBar, thisTid );
           while(true)
           {
               receive(
                   (Variant any) {
                       writeln( "Received a variant" );
                       writeln( "Received ", any );
                   }
               );
           }
       }
 
       void fooBar( Tid masterTid )
       {
           scope(failure) writeln( "fooBar failed" );
           scope(success) writeln( "fooBar succeeded" );
           scope(exit) writeln( "fooBar exiting" );
           try
           {
               immutable A b = new immutable A();
               masterTid.send( 42 ); // This works
               masterTid.send( b );  // This doesn't
           }
           catch( Exception e )
           {
               writeln( "Exception received" );
           }
       }
 
 
 I see this in the console:
 
       fooBar exiting
       fooBar failed
       Received a variant
       Received 42
 
 (then it just hangs)
 
 I'm especially puzzled by:
    * Sending an int as a message works but not an immutable object.
 Wasn't this  (safely sharing objects across threads) one of the
 basic use cases for the immutable type qualifier?
    * scope(failure) failed but my exception handler didn't catch
 anything. How is this possible? What could cause that?
 Assertions/abort?
 
 There seemed to be a 3-year-old ticket on this issue
 (http://d.puremagic.com/issues/show_bug.cgi?id=5538) with very
 little activity, which is a bit surprising given how much
 emphasis is given to this feature (the D homepage mentions "D
 offers an innovative approach to concurrency, featuring true
 immutable data, message passing, no sharing by default, and
 controlled mutable sharing across threads", and TDPL devotes a
 whole chapter on it). This thread
 (http://forum.dlang.org/thread/kgk8hc$12fa$1 digitalmars.com)
 also says std.concurrency is "very buggy".
 
 If I can't use std.concurrency, is there any other safe
 alternative for multithreaded programming with D?
 
 Thanks,
 A.



There is a bug with the internals of send/receive.   It has to be able 
to copy the reference to a another place while sending --  and it seems 
it can't do that because it's immutable.     I don't think many people 
are using it.    I personally did not realize the bug was from 2011.  I 
think DMD has been fixed enough to make a patch to phobos now.  I'm 
going to ping this bug report.

However,  even CONSTRUCTING immutable objects right now is very 
difficult.   It's arguably "correct" but, seems impossible to actually 
be able to make something of use without many many idups.  It's kind of 
in the same boat as shared.

What I am currently doing, is casting to shared, and then back.  I 
tried making my classes shared, and also immutable as you have.   
Nothing else seems to work:


 class A
       {
           this() {}
       }
 
       void main()
       {
           auto tid = spawn( &fooBar, thisTid );
           while(true)
           {
               receive(


		   (shared A _m)
	     	   {
			auto m = cast(A)_m;
			//Do stuff.
		   },

                   (Variant any) {
                       writeln( "Received a variant" );
                       writeln( "Received ", any );
                   }
               );
           }
       }
 
       void fooBar( Tid masterTid )
       {
           scope(failure) writeln( "fooBar failed" );
           scope(success) writeln( "fooBar succeeded" );
           scope(exit) writeln( "fooBar exiting" );
           try
           {
               immutable A b = new A();
               masterTid.send( 42 ); // This works
               masterTid.send( cast(shared) b );  // Should work
           }
           catch( Exception e )
           {
               writeln( "Exception received" );
           }
       }


-Shammah
Nov 25 2013
parent reply "Antoche" <N A.c> writes:
On Monday, 25 November 2013 at 11:48:06 UTC, Shammah Chancellor 
wrote:

 On 2013-11-25 06:03:27 +0000, Antoche said:


 The following code compiles but doesn't work as expected:
 
      import std.stdio;
      import std.concurrency;
 
      class A
      {
          this() immutable {}
      }
 
      void main()
      {
          auto tid = spawn( &fooBar, thisTid );
          while(true)
          {
              receive(
                  (Variant any) {
                      writeln( "Received a variant" );
                      writeln( "Received ", any );
                  }
              );
          }
      }
 
      void fooBar( Tid masterTid )
      {
          scope(failure) writeln( "fooBar failed" );
          scope(success) writeln( "fooBar succeeded" );
          scope(exit) writeln( "fooBar exiting" );
          try
          {
              immutable A b = new immutable A();
              masterTid.send( 42 ); // This works
              masterTid.send( b );  // This doesn't
          }
          catch( Exception e )
          {
              writeln( "Exception received" );
          }
      }
 
 
 I see this in the console:
 
      fooBar exiting
      fooBar failed
      Received a variant
      Received 42
 
 (then it just hangs)
 
 I'm especially puzzled by:
   * Sending an int as a message works but not an immutable 
 object.
 Wasn't this  (safely sharing objects across threads) one of the
 basic use cases for the immutable type qualifier?
   * scope(failure) failed but my exception handler didn't catch
 anything. How is this possible? What could cause that?
 Assertions/abort?
 
 There seemed to be a 3-year-old ticket on this issue
 (http://d.puremagic.com/issues/show_bug.cgi?id=5538) with very
 little activity, which is a bit surprising given how much
 emphasis is given to this feature (the D homepage mentions "D
 offers an innovative approach to concurrency, featuring true
 immutable data, message passing, no sharing by default, and
 controlled mutable sharing across threads", and TDPL devotes a
 whole chapter on it). This thread
 (http://forum.dlang.org/thread/kgk8hc$12fa$1 digitalmars.com)
 also says std.concurrency is "very buggy".
 
 If I can't use std.concurrency, is there any other safe
 alternative for multithreaded programming with D?
 
 Thanks,
 A.



 There is a bug with the internals of send/receive.   It has to 
 be able to copy the reference to a another place while sending 
 --  and it seems it can't do that because it's immutable.     I 
 don't think many people are using it.    I personally did not 
 realize the bug was from 2011.  I think DMD has been fixed 
 enough to make a patch to phobos now.  I'm going to ping this 
 bug report.

 However,  even CONSTRUCTING immutable objects right now is very 
 difficult.   It's arguably "correct" but, seems impossible to 
 actually be able to make something of use without many many 
 idups.  It's kind of in the same boat as shared.

 What I am currently doing, is casting to shared, and then back.
  I tried making my classes shared, and also immutable as you 
 have.   Nothing else seems to work:


 class A
      {
          this() {}
      }
 
      void main()
      {
          auto tid = spawn( &fooBar, thisTid );
          while(true)
          {
              receive(


 		   (shared A _m)
 	     	   {
 			auto m = cast(A)_m;
 			//Do stuff.
 		   },

                  (Variant any) {
                      writeln( "Received a variant" );
                      writeln( "Received ", any );
                  }
              );
          }
      }
 
      void fooBar( Tid masterTid )
      {
          scope(failure) writeln( "fooBar failed" );
          scope(success) writeln( "fooBar succeeded" );
          scope(exit) writeln( "fooBar exiting" );
          try
          {
              immutable A b = new A();
              masterTid.send( 42 ); // This works
              masterTid.send( cast(shared) b );  // Should work
          }
          catch( Exception e )
          {
              writeln( "Exception received" );
          }
      }


 -Shammah


Thanks for the suggestion. My problem with casting is that it 
results in undefined behaviour, so I have no guarantee whatsoever 
what the program is actually going to do. I could simply not use 
immutable at all, and use shared instead, but then I might as 
well stick to C++ and volatile. The whole point of using D was 
trying to write safer code with equivalent performance.

I see std.parallelism seems to offer some features that might 
compensate for the broken std.concurrency. Is it working better 
or is it as broken as std.concurrency? Is there any other way to 
write safe multithreaded programs?

Finally, can anyone explained why the scope(failure) is reached? 
Looking at the stacktrace in gdb at this point gives me no 
indication of what's happening.
Nov 25 2013
next sibling parent Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-25 20:55:15 +0000, Antoche said:


 On Monday, 25 November 2013 at 11:48:06 UTC, Shammah Chancellor wrote:

 On 2013-11-25 06:03:27 +0000, Antoche said:
 

 The following code compiles but doesn't work as expected:
 
 import std.stdio;
 import std.concurrency;
 
 class A
 {
 this() immutable {}
 }
 
 void main()
 {
 auto tid = spawn( &fooBar, thisTid );
 while(true)
 {
 receive(
 (Variant any) {
 writeln( "Received a variant" );
 writeln( "Received ", any );
 }
 );
 }
 }
 
 void fooBar( Tid masterTid )
 {
 scope(failure) writeln( "fooBar failed" );
 scope(success) writeln( "fooBar succeeded" );
 scope(exit) writeln( "fooBar exiting" );
 try
 {
 immutable A b = new immutable A();
 masterTid.send( 42 ); // This works
 masterTid.send( b );  // This doesn't
 }
 catch( Exception e )
 {
 writeln( "Exception received" );
 }
 }
 
 
 I see this in the console:
 
 fooBar exiting
 fooBar failed
 Received a variant
 Received 42
 
 (then it just hangs)
 
 I'm especially puzzled by:
 * Sending an int as a message works but not an immutable object.
 Wasn't this  (safely sharing objects across threads) one of the
 basic use cases for the immutable type qualifier?
 * scope(failure) failed but my exception handler didn't catch
 anything. How is this possible? What could cause that?
 Assertions/abort?
 
 There seemed to be a 3-year-old ticket on this issue
 (http://d.puremagic.com/issues/show_bug.cgi?id=5538) with very
 little activity, which is a bit surprising given how much
 emphasis is given to this feature (the D homepage mentions "D
 offers an innovative approach to concurrency, featuring true
 immutable data, message passing, no sharing by default, and
 controlled mutable sharing across threads", and TDPL devotes a
 whole chapter on it). This thread
 (http://forum.dlang.org/thread/kgk8hc$12fa$1 digitalmars.com)
 also says std.concurrency is "very buggy".
 
 If I can't use std.concurrency, is there any other safe
 alternative for multithreaded programming with D?
 
 Thanks,
 A.


 
 
 There is a bug with the internals of send/receive.   It has to be able 
 to copy the reference to a another place while sending --  and it seems 
 it can't do that because it's immutable.     I don't think many people 
 are using it.    I personally did not realize the bug was from 2011.  I 
 think DMD has been fixed enough to make a patch to phobos now.  I'm 
 going to ping this bug report.
 
 However,  even CONSTRUCTING immutable objects right now is very 
 difficult.   It's arguably "correct" but, seems impossible to actually 
 be able to make something of use without many many idups.  It's kind of 
 in the same boat as shared.
 
 What I am currently doing, is casting to shared, and then back.
 I tried making my classes shared, and also immutable as you have.   
 Nothing else seems to work:
 

 class A
 {
 this() {}
 }
 
 void main()
 {
 auto tid = spawn( &fooBar, thisTid );
 while(true)
 {
 receive(


 		   (shared A _m)
 	     	   {
 			auto m = cast(A)_m;
 			//Do stuff.
 		   },

 (Variant any) {
 writeln( "Received a variant" );
 writeln( "Received ", any );
 }
 );
 }
 }
 
 void fooBar( Tid masterTid )
 {
 scope(failure) writeln( "fooBar failed" );
 scope(success) writeln( "fooBar succeeded" );
 scope(exit) writeln( "fooBar exiting" );
 try
 {
 immutable A b = new A();
 masterTid.send( 42 ); // This works
 masterTid.send( cast(shared) b );  // Should work
 }
 catch( Exception e )
 {
 writeln( "Exception received" );
 }
 }


 
 -Shammah


 
 Thanks for the suggestion. My problem with casting is that it results 
 in undefined behaviour, so I have no guarantee whatsoever what the 
 program is actually going to do. I could simply not use immutable at 
 all, and use shared instead, but then I might as well stick to C++ and 
 volatile. The whole point of using D was trying to write safer code 
 with equivalent performance.
 
 I see std.parallelism seems to offer some features that might 
 compensate for the broken std.concurrency. Is it working better or is 
 it as broken as std.concurrency? Is there any other way to write safe 
 multithreaded programs?
 
 Finally, can anyone explained why the scope(failure) is reached? 
 Looking at the stacktrace in gdb at this point gives me no indication 
 of what's happening.



Try wrapping it in a Try/catch and see.    Sending non-value types 
right now seems to be a bit frustrating.   Unless you *need* 
polymorphism for your messages I would suggest sending immutable struct 
references or something.   Or you can also use the idea I suggested and 
cast(const) to ensure you are not messing with the data again 
afterwards.    I am going to ping this issue very heavily though, and 
possibly start working on std.concurrency myself.   The current state 
of the library is unacceptable.

-Shammah
Nov 26 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
By the way this works:

import std.stdio;
import std.concurrency;

class A
{
     this() immutable {}

     // this line
     string toString() const { return "aaa"; }
}

But it will still fail if you try to receive by class type 
explicitly instead of Variant.


std.parallelism is much more mature in my opinion, but 
std.concurrency is mostly broken when it comes to qualifiers and 
needs some serious bug-fixing rampage.
Nov 26 2013
prev sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 25 November 2013 at 20:55:18 UTC, Antoche wrote:

 Thanks for the suggestion. My problem with casting is that it 
 results in undefined behaviour,


Casting is defined, modifications afterwards are not.


 Finally, can anyone explained why the scope(failure) is 
 reached? Looking at the stacktrace in gdb at this point gives 
 me no indication of what's happening.


An assert error wouldn't get picked up by your catch while still 
triggering scope(failure)

catch(Throwable) to grab anything.
Nov 26 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
Sending immutable classes currently does not work because of 
https://d.puremagic.com/issues/show_bug.cgi?id=7069 (and has 
never worked despite being intended).
Nov 25 2013
parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2013-11-25 14:08:53 +0000, Dicebot said:


 Sending immutable classes currently does not work because of 
 https://d.puremagic.com/issues/show_bug.cgi?id=7069 (and has never 
 worked despite being intended).


Can you send immutable struct references?
Nov 25 2013
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 25 November 2013 at 23:00:41 UTC, Shammah Chancellor 
wrote:

 On 2013-11-25 14:08:53 +0000, Dicebot said:


 Sending immutable classes currently does not work because of 
 https://d.puremagic.com/issues/show_bug.cgi?id=7069 (and has 
 never worked despite being intended).


 Can you send immutable struct references?


If you mean mutable pointer to immutable struct - yes, this 
should work (Did not try that). But any fully immutable type will 
fail because of Variant implementation issue.
Nov 26 2013