www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Overriden method not detected ?

reply chmike <christophe meessen.net> writes:
When trying to compile the following code I get a compilation 
error

----
import std.stdio;

class Info
{
     final string name() { return nameImpl(); }
     protected abstract string nameImpl();
}

final class MyInfo : Info
{
     this() { assert(__ctfe); }
     private __gshared info_ = new MyInfo; // Line 12

     static string name() { return "MyInfo"; }
     protected override string nameImpl() { return name(); }
}

void main()
{
     writeln("Hello world!");
}
----
source/app.d(12,31): Error: cannot create instance of abstract 
class MyInfo
source/app.d(12,31):        function 'string nameImpl()' is not 
implemented

If I move the info_ static variable declaration after the 
nameImpl method declaration, there is no error anymore.

Is this normal ? What rule is in play here ? Or is this a 
compiler bug ?
Jun 03 2016
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/3/16 8:03 AM, chmike wrote:

 import std.stdio;

 class Info
 {
     final string name() { return nameImpl(); }
     protected abstract string nameImpl();
 }

 final class MyInfo : Info
 {
     this() { assert(__ctfe); }
     private __gshared info_ = new MyInfo; // Line 12

     static string name() { return "MyInfo"; }
     protected override string nameImpl() { return name(); }
 }

 void main()
 {
     writeln("Hello world!");
 }


Yes, this is a forward reference bug.

Move the info_ member to the end of the class and it compiles.

Please file https://issues.dlang.org

-Steve
Jun 03 2016
prev sibling parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Friday, June 03, 2016 12:03:29 chmike via Digitalmars-d-learn wrote:

 When trying to compile the following code I get a compilation
 error

 ----
 import std.stdio;

 class Info
 {
      final string name() { return nameImpl(); }
      protected abstract string nameImpl();
 }

 final class MyInfo : Info
 {
      this() { assert(__ctfe); }
      private __gshared info_ = new MyInfo; // Line 12

      static string name() { return "MyInfo"; }
      protected override string nameImpl() { return name(); }
 }

 void main()
 {
      writeln("Hello world!");
 }
 ----
 source/app.d(12,31): Error: cannot create instance of abstract
 class MyInfo
 source/app.d(12,31):        function 'string nameImpl()' is not
 implemented

 If I move the info_ static variable declaration after the
 nameImpl method declaration, there is no error anymore.

 Is this normal ? What rule is in play here ? Or is this a
 compiler bug ?


It definitely looks like a compiler bug, though it wouldn't have surprised
me if the compiler considered it illegal to declare a static function in a
derived class where the base class has a final function with the same name.
Still, regardless of the legality of reusing the name like that, the error
message is pretty clearly wrong, and the order of declarations shouldn't
matter within a class unless you're doing something with static if.

On a side note, be warned that you almost certainly shouldn't be using
__gshared like this. It's intended for interacting with C code not for D
objects to be marked with D. As far as the type system is concerned,
__gshared isn't part of the type, and the variable will be treated as
thread-local by all of the code that uses it, which can result in really
nasty, subtle bugs when the compiler starts doing stuff like optimizations.
If you want to be sharing D objects across threads, you really should be
using shared so that the compiler knows that it's shared across threads and
will treat it that way.

- Jonathan M Davis
Jun 03 2016
parent reply chmike <christophe meessen.net> writes:
On Friday, 3 June 2016 at 12:41:39 UTC, Jonathan M Davis wrote:
...

 On a side note, be warned that you almost certainly shouldn't 
 be using __gshared like this. It's intended for interacting 
 with C code not for D objects to be marked with D. As far as 
 the type system is concerned, __gshared isn't part of the type, 
 and the variable will be treated as thread-local by all of the 
 code that uses it, which can result in really nasty, subtle 
 bugs when the compiler starts doing stuff like optimizations. 
 If you want to be sharing D objects across threads, you really 
 should be using shared so that the compiler knows that it's 
 shared across threads and will treat it that way.


Thanks to point this out. What does shared really do behind the 
scene ?
Does it add synchronization instructions ?

In my case I really don't want the compiler to add 
synchronization instructions because the objects are immutable 
from the user perspective. This is enforced by the interface. Tho 
objects are fully instantiated in a private static this() {} 
function which shouldn't be affected by multi-threading since it 
is executed at startup.

The unpleasant side effect of shared is that I then have to use 
shared(Info) instead of the shorter type name Info.

What are the subtle and nasty bugs you are referring to ?
Jun 03 2016
parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Friday, June 03, 2016 13:27:41 chmike via Digitalmars-d-learn wrote:

 On Friday, 3 June 2016 at 12:41:39 UTC, Jonathan M Davis wrote:
 ...


 On a side note, be warned that you almost certainly shouldn't
 be using __gshared like this. It's intended for interacting
 with C code not for D objects to be marked with D. As far as
 the type system is concerned, __gshared isn't part of the type,
 and the variable will be treated as thread-local by all of the
 code that uses it, which can result in really nasty, subtle
 bugs when the compiler starts doing stuff like optimizations.
 If you want to be sharing D objects across threads, you really
 should be using shared so that the compiler knows that it's
 shared across threads and will treat it that way.


 Thanks to point this out. What does shared really do behind the
 scene ?
 Does it add synchronization instructions ?


shared does not add any synchronization instructions. It makes it so that
the object is on the shared heap instead of the thread-local heap; it makes
it so that that the compiler treats it as shared across threads (whereas if
it's not shared, the compiler assumes that it's on the thread-local heap and
that no other thread has access to it); and it makes it so that certain
things are illegal without using core.atomic (e.g. directly incrementing a
shared int isn't legal, because it's not thread-safe). If a variable is not
shared, then the compiler is free to assume that it's thread local and make
optimizations based on that fact.

So, if you have a shared object, you have to either make it use core.atomic
to do anything like incrementing values, or you have to cast away shared to
operate on the object (in which case, you need to have protected all access
to the object with a mutex or synchronized block just like you'd do with
C++). So, you end up with code like

shared MyObj myObj = getObjFromSomewhere();
synchronized(lockObj)
{
    // myObj is now protected by a lock; all other accesses to it
    // should be as well, or this isn't thread-safe.

    auto threadLocal = cast(MyObj)myObj;
    // do stuff with the thread-local reference

    // don't let any thread-local references to myObj escape
}
// the lock has been released, and the only references to myObj are
// shared.

It's basically the same thing that you'd do in languages like C++ or Java
except that you're forced to cast away shared, whereas in those languages,
everything is shared, and you can potentially have thread-related problems
pretty much anywhere in your code.

You can have also all of the functions on your struct/class be shared and
call them without locking, but you risk all of the normal race conditions if
you're not either protecting it via locks or using the atomic operations
from core.atomic, otherwise a number of operations on built-in types will be
considered illegal and result in compilation errors, since if operations
aren't atomic, they need to be done with the object being properly locked,
or they're not thread-safe.

In theory, we're supposed to get synchronized classes, which protect their
member variables via a lock for the whole class such that the outer layer of
shared can be stripped away within its member functions, but it's never been
fully implemented. Rather, right now, the closest we have is synchronized
functions, which is basically the same as using synchronized blocks, so it
doesn't have any of the properties that would allow the compiler to strip
away the outer layer of shared (since it can't guaranteed that there are no
other, unprotected references to the same data). So, if we had synchronized
classes, you could potentially avoid the ugliness of casting away shared,
but we don't have them yet.

The concurrency chapter from Andrei's book, "The D Programming Language" is
available for free online:

http://www.informit.com/articles/article.aspx?p=1609144

So, you can read that for more details, but it does talk like synchronized
classes are implemented, and we don't have synchronized functions (since
that was the plan when it was written but hasn't happened yet).

Another resource to look at would be Ali's book:

http://ddili.org/ders/d.en/concurrency_shared.html

shared does feel a bit clunky (particularly when you need to cast it away),
but it does allow you to segregate the code that involves threads, and it
helps protect you against race conditions, whereas if you use __gshared,
you're risking serious problems.


 In my case I really don't want the compiler to add
 synchronization instructions because the objects are immutable
 from the user perspective. This is enforced by the interface. Tho
 objects are fully instantiated in a private static this() {}
 function which shouldn't be affected by multi-threading since it
 is executed at startup.


If the object is really immutable, then just make it immutable. immutable is
implicitly shared across threads, but it doesn't have any of the locking
issues, because it can't ever be mutated.

Now, if what you mean is that the object is const such that a mutable
reference to that data could still change it, then that's different, and
you'll need to use shared to share across threads and do all of the
appropriate locking.

Also, be warned that if you're using static this with either shared or
immutable (or __gshared), then that static constructor needs to be marked
with shared as well, otherwise it will be run in every thread (the compiler
should prevent you from initializing static or module-level variables which
are shared or immutable in a non-shared, static constructor, but it doesn't
currently).


 The unpleasant side effect of shared is that I then have to use
 shared(Info) instead of the shorter type name Info.


Well, to some extent, that's on purpose in that the portion of your code
that deals with objects that are shared across threads should be very small
and segregated so that it's easy to manage - similar to how using  safe and
 trusted makes it so that you only have to look at  trusted or  system code
to figure out how something like a memory corruption happened, because as
long as  trusted is used properly,  safe code is guaranteed to never be able
to corrupt memory. And if shared is used correctly, then you're guaranteed
to never have threading issues with your thread-local objects.


 What are the subtle and nasty bugs you are referring to ?


The compiler is free to optimize based on the fact that an object is
thread-local if it's not shared. So, any optimizations that it ends up
making based on that will cause weird, subtle problems if the object is
actually shared across threads. What those problems are depends on what
optimizations the compiler does and how the object is being used across
threads.

- Jonathan M Davis
Jun 03 2016
parent reply chmike <christophe meessen.net> writes:
On Friday, 3 June 2016 at 15:23:16 UTC, Jonathan M Davis wrote:
Thank you for your detailed explanation.

If I have a static immutable object, I don't have to declare it 
as shared because it is implicit. Right ?

Changing my __gshared into static shared raises some errors which 
I don't know how to address. Ali's chapter doesn't address shared 
classes. May I ask for your help ?

1. question
-----------
I had a __gshared Info[N] infos_; that I fill in a static this() 
method.

How should I declare this with shared ? Is it enough to write 
static shared Info[N] infos; ? Since shared is transitive, will 
the info objects be implicitly shared ? 

2. question
-----------
As I said above, I instantiate my Info objects in a private 
static this() method. The objects are emplaced in a static shared 
void array. Since emplace only accept a void[] as argument I had 
to cast away the shared as you did in your example. The compiler 
seams happy.

However I didn't synchronized that code because I assume it is 
executed by the main thread before main starts. There is thus no 
risk of race conditions. Is this assumption correct ?


3. error
--------
A weird error is that there is apparently no overload for 
opEquals. Do I have to define one my self ? Is this so that users 
can synchronize themselves if needed ?
source/app.d(9,13): Error: none of the overloads of 'opEquals' 
are callable using argument types (shared(Info), shared(Info)), 
candidates are:
/usr/include/dmd/druntime/import/object.d(143,6):        
object.opEquals(Object lhs, Object rhs)
/usr/include/dmd/druntime/import/object.d(168,6):        
object.opEquals(const(Object) lhs, const(Object) rhs)
I guess I have to define my own opEqual.


4. error
--------
Another error is with writeln(info); where info is now a 
shared(Info).

/usr/include/dmd/phobos/std/format.d(2904,5): Error: static 
assert  "unable to format shared objects"
/usr/include/dmd/phobos/std/format.d(3477,16):        
instantiated from here: formatValue!(LockingTextWriter, 
shared(Info), char)
/usr/include/dmd/phobos/std/format.d(467,54):        instantiated 
from here: formatGeneric!(LockingTextWriter, shared(Info), char)
/usr/include/dmd/phobos/std/stdio.d(1316,31):        instantiated 
from here: formattedWrite!(LockingTextWriter, char, shared(Info))
/usr/include/dmd/phobos/std/stdio.d(3114,28):        instantiated 
from here: write!(shared(Info), char)
source/app.d(12,12):        instantiated from here: 
writeln!(shared(Info))

How can I solve that error ?
Jun 03 2016
parent reply ag0aep6g <anonymous example.com> writes:
On 06/03/2016 10:06 PM, chmike wrote:

 If I have a static immutable object, I don't have to declare it as
 shared because it is implicit. Right ?


Right.

[...]

 1. question
 -----------
 I had a __gshared Info[N] infos_; that I fill in a static this() method.


Should be filling it in a `shared static this`, I think. With just 
`static this`, the fill code is run once per thread.


 How should I declare this with shared ? Is it enough to write static
 shared Info[N] infos; ? Since shared is transitive, will the info
 objects be implicitly shared ?


The variable `infos` and its elements are shared then, yes.

When you're interested in the type of some expression, you can use 
pragma(msg, ...) to print it during compilation. For example:

     pragma(msg, typeof(infos[0])); /* Should print "shared(Info)". */


 2. question
 -----------
 As I said above, I instantiate my Info objects in a private static
 this() method. The objects are emplaced in a static shared void array.
 Since emplace only accept a void[] as argument I had to cast away the
 shared as you did in your example. The compiler seams happy.


As far as I know, that's often the way to treat shared. Cast it away 
when you know it's safe.


 However I didn't synchronized that code because I assume it is executed
 by the main thread before main starts. There is thus no risk of race
 conditions. Is this assumption correct ?


Use `shared static this` as mentioned.


 3. error
 --------
 A weird error is that there is apparently no overload for opEquals. Do I
 have to define one my self ? Is this so that users can synchronize
 themselves if needed ?
 source/app.d(9,13): Error: none of the overloads of 'opEquals' are
 callable using argument types (shared(Info), shared(Info)), candidates are:
 /usr/include/dmd/druntime/import/object.d(143,6): object.opEquals(Object
 lhs, Object rhs)
 /usr/include/dmd/druntime/import/object.d(168,6):
 object.opEquals(const(Object) lhs, const(Object) rhs)
 I guess I have to define my own opEqual.


Well, there doesn't seem to be an equality function for shared class 
objects. I'm not sure what the difficulties are in providing one.

In the meantime, you can either do your own equality thing without the 
`==` operator, or assure thread-safety yourself and cast shared away.


 4. error
 --------
 Another error is with writeln(info); where info is now a shared(Info).

 /usr/include/dmd/phobos/std/format.d(2904,5): Error: static assert
 "unable to format shared objects"
 /usr/include/dmd/phobos/std/format.d(3477,16): instantiated from here:
 formatValue!(LockingTextWriter, shared(Info), char)
 /usr/include/dmd/phobos/std/format.d(467,54):        instantiated from
 here: formatGeneric!(LockingTextWriter, shared(Info), char)
 /usr/include/dmd/phobos/std/stdio.d(1316,31):        instantiated from
 here: formattedWrite!(LockingTextWriter, char, shared(Info))
 /usr/include/dmd/phobos/std/stdio.d(3114,28):        instantiated from
 here: write!(shared(Info), char)
 source/app.d(12,12):        instantiated from here: writeln!(shared(Info))

 How can I solve that error ?


writeln isn't thread-safe, I guess. Synchronize and cast shared away, I 
suppose.
Jun 03 2016
parent reply chmike <christophe meessen.net> writes:
On Friday, 3 June 2016 at 21:04:41 UTC, ag0aep6g wrote:
Thank you ag0aep6g, especially for the missing shared in my 
static this !

Since I'm implementing a (hopefully useful) library, it would be 
unpleasant for users to have to cast away shared to print the 
info.

It works with immutable at the condition that I declare the 
toString() method like this

     string toString() immutable {...}

There is also no missing opEqual.


Is it possible to instantiate immutable objects by using emplace 
and modify the object in the toString() call ? This is to cache 
the resulting string to avoid creating a new string at each call.

I would have to cast away the immutable attribute of 'this'. Is 
this possible ?
Note that the objects would be instantiated by emplace in a void 
array at start up. So the memory is writable.
Jun 04 2016
parent ag0aep6g <anonymous example.com> writes:
On 06/04/2016 05:02 PM, chmike wrote:

 Is it possible to instantiate immutable objects by using emplace


Yes. I'm not sure, but the memory may have to be untyped for the emplace 
call to avoid mutating immutable data. I.e., call emplace with void[], 
not with a pointer whose target is already type as immutable.


 and
 modify the object in the toString() call ?


No. immutable means the object cannot, will not, must not ever change 
after construction.


 This is to cache the
 resulting string to avoid creating a new string at each call.

 I would have to cast away the immutable attribute of 'this'. Is this
 possible ?


You can cast immutable away, but as soon as you mutate the object, you 
have an invalid program. You'd rely on undefined behavior.


 Note that the objects would be instantiated by emplace in a void array
 at start up. So the memory is writable.


Doesn't matter. The compiler is free to assume that the data doesn't 
change after construction.

The compiler will let you cast away immutable and mutate then. It may 
work out as you expect. It's still undefined behavior, and I advise 
against doing it.
Jun 04 2016