www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - shared methods

reply "unknown soldier" <unknownsoldier gmail.com> writes:
I'm confused by shared and how to use it.

import std.stdio;
class Foo {
     File logFile;
     void log(in string line) shared {
       synchronized(this){
         logFile.writeln(line);
       }
     }
}

This (or the equivalent code in my full size program) won't
compile:

source/log.d(256): Error: template std.stdio.File.writeln does
not match any function template declaration. Candidates are:
/usr/include/dmd/phobos/std/stdio.d(781):
std.stdio.File.writeln(S...)(S args)
source/log.d(256): Error: template std.stdio.File.writeln(S...)(S
args) cannot deduce template function from argument types
!()(const(immutable(char)[])) shared

Why is logFile suddenly being treated as shared? This does not
make sense to me. As a programmer I have acknowledged that log()
is a shared function in the declaration 'void log(in string line)
shared', so isn't it up to me to ensure that I do proper
synchronization within the function? Why is the compiler trying
to ensure that all functions called within log() are also marked
shared?
What is the right thing to do here?

Note that I can't just mark log() as not shared; log() must be
shared because elsewhere I have:

shared Foo sfoo = ...
sfoo.log(...)
Jan 21 2014
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 22.01.2014 06:16, schrieb unknown soldier:

 I'm confused by shared and how to use it.

 import std.stdio;
 class Foo {
      File logFile;
      void log(in string line) shared {
        synchronized(this){
          logFile.writeln(line);
        }
      }
 }

 This (or the equivalent code in my full size program) won't
 compile:

 source/log.d(256): Error: template std.stdio.File.writeln does
 not match any function template declaration. Candidates are:
 /usr/include/dmd/phobos/std/stdio.d(781):
 std.stdio.File.writeln(S...)(S args)
 source/log.d(256): Error: template std.stdio.File.writeln(S...)(S
 args) cannot deduce template function from argument types
 !()(const(immutable(char)[])) shared

 Why is logFile suddenly being treated as shared? This does not
 make sense to me. As a programmer I have acknowledged that log()
 is a shared function in the declaration 'void log(in string line)
 shared', so isn't it up to me to ensure that I do proper
 synchronization within the function? Why is the compiler trying
 to ensure that all functions called within log() are also marked
 shared?
 What is the right thing to do here?

 Note that I can't just mark log() as not shared; log() must be
 shared because elsewhere I have:

 shared Foo sfoo = ...
 sfoo.log(...)


For a shared method the this pointer is also shared. That means that you 
have to tell the compiler (manually) that it is now safe to use 
non-shared code. You usually do this by casting the this pointer to a 
unshared type.

class Foo {
      File logFile;
      void log(in string line) shared {
        synchronized(this){
          // safe from here on because synchronized
          auto self = cast(Foo)this;
          self.logFile.writeln(line);
        }
      }
}

The compiler does not do this for you, because it can not know if 
accessing any member is truly thread-safe. You might share the logFile 
with other instances of Foo. This would be a case where the compiler 
would wrongly remove the shared from this, if it would do so 
automatically inside synchronized blocks.
Jan 22 2014
parent reply "unknown soldier" <unknownsoldier gmail.com> writes:
 The compiler does not do this for you, because it can not know 
 if accessing any member is truly thread-safe. You might share 
 the logFile with other instances of Foo. This would be a case 
 where the compiler would wrongly remove the shared from this, 
 if it would do so automatically inside synchronized blocks.


Makes sense; thank you!
Jan 22 2014
parent reply "Kagamin" <spam here.lot> writes:
A minor caveat is GDC treats shared as volatile, which can be not 
optimal. Even worse, on some platforms like itanium volatile 
generates fences.
Jan 23 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 23 Jan 2014 19:20:34 +0000
schrieb "Kagamin" <spam here.lot>:


 A minor caveat is GDC treats shared as volatile, which can be not 
 optimal. Even worse, on some platforms like itanium volatile 
 generates fences.


Do you have more information on that issue?
There are claims that Itanium "automatically generates fences", however
that statement is not precise and I can't find a detailed description.

Itanium has special instructions which refresh the caches before
reading/writing. But the interesting question is this: Does GCC use
these instructions if you access a variable marked as volatile?

If it does, that's really a performance problem. AFAICS we need a
'volatile' in D for embedded programming which just does the following:

* Avoid removing any stores / or loads to that variable (includes
  moving accesses out of loops)
* Avoid instruction reordering between accesses to volatile variables:
  volatile int a, c;
  int b;
  a = 0;
  b = 0;
  c = 0;
  b can be moved anywhere, but c must always come after a

Here volatile should only affect the ASM generated by the compiler, it
shouldn't care at all what the final CPU does, that's the programmers
task.

In some cases the compiler can't even know: For example some
memory areas in ARM are special and the processor won't reorder
accesses to those areas anyway, so a 'smart compiler' adding memory
barriers would actually only harm performance.
http://infocenter.arm.com/help/topic/com.arm.doc.dui0552a/CIHGEIID.html

Such a 'volatile' could be integrated with shared as it should not
cause any huger performance problems. However, if you think of this
code:

shared int X;
synchronized(x)
{
    for(int i=0; i< 100; i++)
        X = i;
}

it's actually valid to optimize the loop, so 'volatile' behavior is not
wanted. But what if we used atomic ops?

shared int X;

for(int i=0; i< 100; i++)
    atomicSet(X, i);

Now the loop can't be optimized away. Is the compiler smart enough to
figure out that an atomic op is used and it can't optimize this? Or
would we have to mark this volatile?
Jan 24 2014
next sibling parent "Kagamin" <spam here.lot> writes:
for example 
http://software.intel.com/en-us/blogs/2007/11/30/volatile-almost-useless-for-multi-threaded-programming
Jan 24 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
http://igoro.com/archive/volatile-keyword-in-c-memory-model-explained/
As I understand, because Itanium doesn't have cache coherency, a 
memory fence is needed to implement volatile load and store. On 
x86 load and store are already volatile because of cache 
coherency, so fences are not needed.
Jan 24 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 24 Jan 2014 22:30:13 +0000
schrieb "Kagamin" <spam here.lot>:


 http://igoro.com/archive/volatile-keyword-in-c-memory-model-explained/
 As I understand, because Itanium doesn't have cache coherency, a 
 memory fence is needed to implement volatile load and store. On 
 x86 load and store are already volatile because of cache 
 coherency, so fences are not needed.


Seems like C#s volatile cares about "compiler optimizations and
processor optimizations.". I'd rather want a volatile for D (or as a
base for shared) which only cares about compiler optimizations, leaving
the processor part to the programmer.

(For example it isn't valid in D to access a shared variable with
normal operations anyway, AFAIK. You need to use the atomicOp things
and these will the worry about the hardware part, using the correct
instructions and so on)

See also:
http://www.drdobbs.com/parallel/volatile-vs-volatile/212701484


is why I'm confused, C compilers are not supposed to guarantee that
another thread can see all changes to a volatile variable, volatile must
just prevent compiler optimizations. So if Itanium C compilers
automatically use the barrier/fence instructions for volatile that's
IMHO a compiler performance bug.

I also can't find any information the GCC adds barriers on Itanium.
Maybe this is just true for the Intel compiler, as the main source for
these claims are Intel pages.
Jan 25 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 25 January 2014 at 10:00:58 UTC, Johannes Pfau wrote:

 (For example it isn't valid in D to access a shared variable 
 with
 normal operations anyway, AFAIK. You need to use the atomicOp 
 things
 and these will the worry about the hardware part, using the 
 correct
 instructions and so on)


Is it invalid to access shared data on stack too? How about 
closures?


 See also:
 http://www.drdobbs.com/parallel/volatile-vs-volatile/212701484


 C/C++. This
 is why I'm confused, C compilers are not supposed to guarantee 
 that
 another thread can see all changes to a volatile variable, 
 volatile must
 just prevent compiler optimizations. So if Itanium C compilers
 automatically use the barrier/fence instructions for volatile 
 that's
 IMHO a compiler performance bug.


The standard says "memory" and doesn't address cache and 
processor optimizations, so it probably allows different 
interpretations. Even if there's no fence, disallowed compiler 
optimizations are still a performance hit - on all platforms.
Jan 25 2014
parent Johannes Pfau <nospam example.com> writes:
Am Sat, 25 Jan 2014 21:41:20 +0000
schrieb "Kagamin" <spam here.lot>:


 On Saturday, 25 January 2014 at 10:00:58 UTC, Johannes Pfau wrote:

 (For example it isn't valid in D to access a shared variable 
 with
 normal operations anyway, AFAIK. You need to use the atomicOp 
 things
 and these will the worry about the hardware part, using the 
 correct
 instructions and so on)


 
 Is it invalid to access shared data on stack too? How about 
 closures?
 


I'm not sure about the details - shared hasn't been completely
implemented or even specified yet. AFAIK the most recent idea was that
shared does basically nothing but prevents direct access to variables.
Synchronized statements then remove the shared qualifier and you can
access the variables as usual. Atomic OPs also work with shared
variables. Using locks manually then probably requires casting away
shared.

However, this part of the language is really unfinished. I hoped we
could at least use shared as a replacement for volatile, but as you
said in your other reply we probably can't.
Jan 25 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
Also if you read a shared value with atomicLoad every time, this 
disallows caching in registers or on stack, which is also 
performance hit. The shared value should be read once and cached 
if possible.
Jan 25 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 25 Jan 2014 21:48:39 +0000
schrieb "Kagamin" <spam here.lot>:


 Also if you read a shared value with atomicLoad every time, this 
 disallows caching in registers or on stack, which is also 
 performance hit. The shared value should be read once and cached 
 if possible.


Yes, I came to the same conclusion. If we combine volatile
and shared into one qualifier we'll always have a certain performance
hit.

Great, now we have to convince Walter that we have to undeprecate
volatile for embedded programming...
Jan 25 2014
parent "Kagamin" <spam here.lot> writes:
On Saturday, 25 January 2014 at 23:20:47 UTC, Johannes Pfau wrote:

 Yes, I came to the same conclusion. If we combine volatile
 and shared into one qualifier we'll always have a certain 
 performance
 hit.


This is what GDC does now.
Jan 26 2014