www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Slow performance compared to C++, ideas?

reply "finalpatch" <fengli gmail.com> writes:
Recently I ported a simple ray tracer I wrote in C++11 to D. 
Thanks to the similarity between D and C++ it was almost a line 
by line translation, in other words, very very close. However, 
the D verson runs much slower than the C++11 version. On Windows, 
with MinGW GCC and GDC, the C++ version is twice as fast as the D 
version. On OSX, I used Clang++ and LDC, and the C++11 version 
was 4x faster than D verson.  Since the comparison were between 
compilers that share the same codegen backends I suppose that's a 
relatively fair comparison.  (flags used for GDC: -O3 
-fno-bounds-check -frelease,  flags used for LDC: -O3 -release)

I really like the features offered by D but it's the raw 
performance that's worrying me. From what I read D should offer 
similar performance when doing similar things but my own test 
results is not consistent with this claim. I want to know whether 
this slowness is inherent to the language or it's something I was 
not doing right (very possible because I have only a few days of 
experience with D).

Below is the link to the D and C++ code, in case anyone is 
interested to have a look.

https://dl.dropboxusercontent.com/u/974356/raytracer.d
https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
May 30 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
finalpatch:

 I really like the features offered by D but it's the raw 
 performance that's worrying me.
From my experience if you know what you are doing, you are able to write that kind of numerical D code that LDC compiles with a performance very close to C++, and sometimes higher. But you need to be careful about some things. Don't do this: foreach (y; (iota(height))) Use this, because those abstractions are not for free: foreach (y; 0 .. height) Be careful with foreach on arrays of structs, because it perform copies that are slow if the structs aren't very small. Be careful with classes, because on default their methods are virtual. Sometimes in D you want to use structs for performance reasons. Sometimes in inner loops it's better to use a classic for instead of a foreach. LDC needs far more flags to compile a raytracer well. LDC even support link time optimization, but you need even more obscure flags. Also the ending brace of classes and structs doesn't need a semicolon in D. Bye, bearophile
May 30 2013
parent reply "finalpatch" <fengli gmail.com> writes:
Hi bearophile,

Thanks for the reply. I changed it to 0..height and it has no 
measurable effect to the runtime.

The reason I used iota(height) was to test 
std.parallelism.parallel. On Windows if I do foreach (y; 
parallel(iota(height))) I do get almost 4x speed up on a quadcore 
computer. However, on OSX, parallel() either does nothing (LDC) 
or makes it slower than single threaded(DMD).

On Friday, 31 May 2013 at 01:42:53 UTC, bearophile wrote:
 Don't do this:
 foreach (y; (iota(height)))

 Use this, because those abstractions are not for free:
 foreach (y;  0 .. height)
May 30 2013
parent "bearophile" <bearophileHUGS lycos.com> writes:
finalpatch:

 Thanks for the reply. I changed it to 0..height and it has no 
 measurable effect to the runtime.
Have you also fixed all the other things? :-) Probably you have to keep fixing potentially slow spots until you find the truly slow ones. Bye, bearophile
May 30 2013
prev sibling next sibling parent reply "Rob T" <alanb ucora.com> writes:
I don't know if this is the case with the code in question (I 
have not looked at it), but sometimes there will be a significant 
effect on performance caused by the use of the garbage collector. 
This is an area in need of radical improvements.

You have to minimize situations where there's a lot of 
allocations going on while the GC is enabled because that will 
fire up the GC more often than is required and it can slow down 
your app significantly; A 2x or more performance penalty is 
certainly possible. It can also make performance unpredictable 
with large delays at inappropriate points in the execution.

BTW, you should post questions like this into d.learn rather than 
in the general discussion area.

--rt
May 30 2013
parent "finalpatch" <fengli gmail.com> writes:
Hi Rob,

I have tried put GC.disable() and GC.enable() around the 
rendering call and it made no difference.

On Friday, 31 May 2013 at 02:13:36 UTC, Rob T wrote:
 I don't know if this is the case with the code in question (I 
 have not looked at it), but sometimes there will be a 
 significant effect on performance caused by the use of the 
 garbage collector. This is an area in need of radical 
 improvements.

 You have to minimize situations where there's a lot of 
 allocations going on while the GC is enabled because that will 
 fire up the GC more often than is required and it can slow down 
 your app significantly; A 2x or more performance penalty is 
 certainly possible. It can also make performance unpredictable 
 with large delays at inappropriate points in the execution.

 BTW, you should post questions like this into d.learn rather 
 than in the general discussion area.

 --rt
May 30 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/30/2013 6:26 PM, finalpatch wrote:
 Recently I ported a simple ray tracer I wrote in C++11 to D. Thanks to the
 similarity between D and C++ it was almost a line by line translation, in other
 words, very very close. However, the D verson runs much slower than the C++11
 version. On Windows, with MinGW GCC and GDC, the C++ version is twice as fast
as
 the D version. On OSX, I used Clang++ and LDC, and the C++11 version was 4x
 faster than D verson.  Since the comparison were between compilers that share
 the same codegen backends I suppose that's a relatively fair comparison. 
(flags
 used for GDC: -O3 -fno-bounds-check -frelease,  flags used for LDC: -O3
-release)
For max speed using dmd, use the flags: -O -release -inline -noboundscheck The -inline is especially important.
 I really like the features offered by D but it's the raw performance that's
 worrying me. From what I read D should offer similar performance when doing
 similar things but my own test results is not consistent with this claim. I
want
 to know whether this slowness is inherent to the language or it's something I
 was not doing right (very possible because I have only a few days of experience
 with D).

 Below is the link to the D and C++ code, in case anyone is interested to have a
 look.

 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
May 30 2013
parent reply "finalpatch" <fengli gmail.com> writes:
Hi Walter,

Thanks for the reply. I have already tried these flags. However, 
DMD's codegen is lagging behind GCC and LLVM at the moment, so 
even with these flags, the runtime is ~10x longer than the C++ 
version compiled with clang++ (2sec with DMD, 200ms with clang++ 
on a Core2 Mac Pro). I know this is comparing apples to oranges 
though, that's why I was comparing GDC vs G++ and LDC vs Clang++.

On Friday, 31 May 2013 at 02:19:40 UTC, Walter Bright wrote:
 For max speed using dmd, use the flags:

    -O -release -inline -noboundscheck

 The -inline is especially important.
May 30 2013
next sibling parent Juan Manuel Cabo <juanmanuel.cabo gmail.com> writes:
On 05/30/2013 11:31 PM, finalpatch wrote:
 Hi Walter,
 
 Thanks for the reply. I have already tried these flags. However, DMD's codegen
is lagging behind GCC and LLVM at the
 moment, so even with these flags, the runtime is ~10x longer than the C++
version compiled with clang++ (2sec with DMD,
 200ms with clang++ on a Core2 Mac Pro). I know this is comparing apples to
oranges though, that's why I was comparing
 GDC vs G++ and LDC vs Clang++.
 
 On Friday, 31 May 2013 at 02:19:40 UTC, Walter Bright wrote:
 For max speed using dmd, use the flags:

    -O -release -inline -noboundscheck

 The -inline is especially important.
Have you tried: dmd -profile it compiles in trace generation, so that when you run the program you get a .log file which tells you the slowest functions and other info. Please not that the resulting code compiled with -profile is slower because it is instrumented. --jm
May 30 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/30/2013 7:31 PM, finalpatch wrote:
 Thanks for the reply. I have already tried these flags. However, DMD's codegen
 is lagging behind GCC and LLVM at the moment, so even with these flags, the
 runtime is ~10x longer than the C++ version compiled with clang++ (2sec with
 DMD, 200ms with clang++ on a Core2 Mac Pro). I know this is comparing apples to
 oranges though, that's why I was comparing GDC vs G++ and LDC vs Clang++.
Yes, you're comparing things the right way.
May 30 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Manu's gonna love this one: make all methods final. Andrei
May 30 2013
next sibling parent reply "finalpatch" <fengli gmail.com> writes:
Hi Andrei,

Making all methods final helps a little bit, but not by much (4x 
slower to 3.8x slower). It is really strange because I didn't 
spent any time optimizing either the D version or the C++ 
version, but it appears C++ is just fast by default.

On Friday, 31 May 2013 at 02:56:25 UTC, Andrei Alexandrescu wrote:
 On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Manu's gonna love this one: make all methods final. Andrei
May 30 2013
parent "estew" <estewh gmail.com> writes:
Two quick notes:

a) Profile first the optimise.


b) This probably wouldn't make 4x difference but in the C++ code 
you're passing most objects around by ref. In the D version 
you're passing structs by value.

They are only small but there's a tight loop of recursion to 
consider...

That said, I don't know the details of D optimisation all that 
well and it may be a non-issue in release builds.

Stewart
May 30 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 31 May 2013 at 02:56:25 UTC, Andrei Alexandrescu wrote:
 On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Manu's gonna love this one: make all methods final.
I don't think going as far as making thing final by default make sense at this point. But we sure need a way to be able to finalize methods. We had an extensive discussion with Don and Manu at DConf, here are some idea that came out : - Final by default This one is really a plus when it come to performance code. However, virtual by default have proven itself very useful when performance isn't that big of a deal (and it is the case for 90% of a program's code usually) and limiting the usage of some pattern like decorator (that also have been proven to be useful). This is also huge breakage. - Introduce a virtual keyword. Virtual by default isn't such a big deal if you can do final: and reverse the default behavior. However, once you key in the final land, you are trapped here, you can't get out. Introducing a virtual keyword would allow for aggressive final: declarations. - Require explicitly export when you want to create shared objects. This one is an enabler for an optimizer to finalize virtual method. With this mechanism in place, the compile knows all the override and can finalize many calls during LTO. I especially like that one as it allow for stripping many symbols at link time and allow for other LTO in general (for instance, the compiler can choose custom calling conventions for some methods, knowing all call sites). The explicit export one have my preference, however, ti require that symbol used in shared lib are explicitly declared export. I think we shouldn't break the virtual by default behavior, but we still need to figure out a way to make thing more performant on this point.
May 30 2013
next sibling parent "Kiith-Sa" <kiithsacmp gmail.com> writes:
Profile.

Don't even think of asking for help before profiling. Those bugs 
you fixed here would be trivial to detect with a profiler. 
GC-dependent stuff usually is (like array literals mentioned 
here) usually are.

As for profiling of something like this, both gprof and DMD's 
builtin profiler are probably going to be useless. Use perf (if 
on Linux), or AMD CodeAnalyst, or some other sampling profiler 
(these two are free).
May 30 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-05-31 06:06, deadalnix wrote:

   - Introduce a virtual keyword.

 Virtual by default isn't such a big deal if you can do final: and
 reverse the default behavior. However, once you key in the final land,
 you are trapped here, you can't get out. Introducing a virtual keyword
 would allow for aggressive final: declarations.
I guess this would be least intrusive change. -- /Jacob Carlborg
May 31 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 31 May 2013 14:06, deadalnix <deadalnix gmail.com> wrote:

 On Friday, 31 May 2013 at 02:56:25 UTC, Andrei Alexandrescu wrote:

 On 5/30/13 9:26 PM, finalpatch wrote:

 https://dl.dropboxusercontent.**com/u/974356/raytracer.d<https://dl.dropboxusercontent.com/u/974356/raytracer.d>
 https://dl.dropboxusercontent.**com/u/974356/raytracer.cpp<https://dl.dropboxusercontent.com/u/974356/raytracer.cpp>
Manu's gonna love this one: make all methods final.
I don't think going as far as making thing final by default make sense at this point. But we sure need a way to be able to finalize methods. We had an extensive discussion with Don and Manu at DConf, here are some idea that came out : - Final by default This one is really a plus when it come to performance code. However, virtual by default have proven itself very useful when performance isn't that big of a deal (and it is the case for 90% of a program's code usually) and limiting the usage of some pattern like decorator (that also have been proven to be useful). This is also huge breakage.
The correct choice :)
 virtual by default have proven itself very useful when performance isn't
that big of a deal Has it? I'm not sure it's ever proven its self useful, can you point to any such proof, or evidence? I think it's only *proven* its self to be a massive mistake; you've all heard my arguments a million times, Don will also give a massive rant, and others. People already have to type 'override' in every derived class, and they're happy to do that. Requiring to type 'virtual' in the base is hardly an inconvenience by contrast. Actually, it's quite orthogonal. D tends to prefer being explicit. Why bend the rules in this case, especially considering the counterpart (override) is expected to be explicit? Surely both being explicit is what people would expect? There's another detail that came up late at night at dconf; I was talking to Daniel Murphy about the extern(C++) class work he's working on to write the D-DMD front end, and apparently it would make the problem a lot easier if virtual was explicit there too. So perhaps it also offers improved interoperability with C++, which I think most of us have agreed is of heightened importance recently.
 and it is the case for 90% of a program's code usually
Can you justify this? I believe it's more like 5% of methods, or less. In my experience, virtuals account for 1-2 functions, and trivial properties (which should avoid being virtual at all costs) account for almost all of them. int getPrivateMember() { return privateMember; } // <- most methods are like this; should NEVER be virtual
 This is also  huge breakage.
I think this is another fallacy. The magnitude of the breakage is already known. It's comparable to the recent change where 'override' became an explicit requirement, which was hardly catastrophic. In that case, the breakage occurred in *every derived class*, which is a many:1 relationship to base classes. Explicit virtual would only affect _base_ classes. The magnitude of which is far smaller than what was recently considered acceptable, precisely: magnitudeOfPriorBreakage / numberOfDerivedClasses. So considering explicit override was an argument won, I'd like to think that explicit virtual deserves the same treatment for even more compelling reasons, and at a significantly lesser cost in breakage. - Introduce a virtual keyword.
 Virtual by default isn't such a big deal if you can do final: and reverse
 the default behavior. However, once you key in the final land, you are
 trapped here, you can't get out. Introducing a virtual keyword would allow
 for aggressive final: declarations.
This needs to be done either way. - Require explicitly export when you want to create shared objects.
 This one is an enabler for an optimizer to finalize virtual method. With
 this mechanism in place, the compile knows all the override and can
 finalize many calls during LTO. I especially like that one as it allow for
 stripping many symbols at link time and allow for other LTO in general (for
 instance, the compiler can choose custom calling conventions for some
 methods, knowing all call sites).

 The explicit export one have my preference, however, ti require that
 symbol used in shared lib are explicitly declared export. I think we
 shouldn't break the virtual by default behavior, but we still need to
 figure out a way to make thing more performant on this point.
You're concerned about the magnitude of breakage introducing an explicit virtual requirement. This would seem to be a much much bigger breakage to me, and I don't think it's intuitive.
May 31 2013
next sibling parent reply Sean Cavanaugh <WorksOnMyMachine gmail.com> writes:
On 5/31/2013 4:42 AM, Manu wrote:
 People already have to type 'override' in every derived class, and
 they're happy to do that. Requiring to type 'virtual' in the base is
 hardly an inconvenience by contrast. Actually, it's quite orthogonal.
 D tends to prefer being explicit. Why bend the rules in this case,
 especially considering the counterpart (override) is expected to be
 explicit? Surely both being explicit is what people would expect?
Maybe the solution is to make everyone equally unhappy: all (non constructor) class methods must be either final, override, or virtual, if you leave one of these off, you get an error :)
May 31 2013
next sibling parent "Nicolas Guillemot" <nlguillemot gmail.com> writes:
On Friday, 31 May 2013 at 19:17:05 UTC, Sean Cavanaugh wrote:
 On 5/31/2013 4:42 AM, Manu wrote:
 People already have to type 'override' in every derived class, 
 and
 they're happy to do that. Requiring to type 'virtual' in the 
 base is
 hardly an inconvenience by contrast. Actually, it's quite 
 orthogonal.
 D tends to prefer being explicit. Why bend the rules in this 
 case,
 especially considering the counterpart (override) is expected 
 to be
 explicit? Surely both being explicit is what people would 
 expect?
Maybe the solution is to make everyone equally unhappy: all (non constructor) class methods must be either final, override, or virtual, if you leave one of these off, you get an error :)
Might not be so bad class Foo { final: foo(); boo(); override: fun(); kun(); virtual: baz(); bar(); }
May 31 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 31 May 2013 at 19:17:05 UTC, Sean Cavanaugh wrote:
 On 5/31/2013 4:42 AM, Manu wrote:
 People already have to type 'override' in every derived class, 
 and
 they're happy to do that. Requiring to type 'virtual' in the 
 base is
 hardly an inconvenience by contrast. Actually, it's quite 
 orthogonal.
 D tends to prefer being explicit. Why bend the rules in this 
 case,
 especially considering the counterpart (override) is expected 
 to be
 explicit? Surely both being explicit is what people would 
 expect?
Maybe the solution is to make everyone equally unhappy: all (non constructor) class methods must be either final, override, or virtual, if you leave one of these off, you get an error :)
A method can be override AND final.
May 31 2013
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 31 May 2013 14:17:04 -0500
schrieb Sean Cavanaugh <WorksOnMyMachine gmail.com>:

 On 5/31/2013 4:42 AM, Manu wrote:
 People already have to type 'override' in every derived class, and
 they're happy to do that. Requiring to type 'virtual' in the base is
 hardly an inconvenience by contrast. Actually, it's quite orthogonal.
 D tends to prefer being explicit. Why bend the rules in this case,
 especially considering the counterpart (override) is expected to be
 explicit? Surely both being explicit is what people would expect?
Maybe the solution is to make everyone equally unhappy: all (non constructor) class methods must be either final, override, or virtual, if you leave one of these off, you get an error :)
A method can be private, which means inherently final. -- Marco
Jun 01 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 31 May 2013 at 09:43:05 UTC, Manu wrote:
  - Require explicitly export when you want to create shared 
 objects.
 This one is an enabler for an optimizer to finalize virtual 
 method. With
 this mechanism in place, the compile knows all the override 
 and can
 finalize many calls during LTO. I especially like that one as 
 it allow for
 stripping many symbols at link time and allow for other LTO in 
 general (for
 instance, the compiler can choose custom calling conventions 
 for some
 methods, knowing all call sites).

 The explicit export one have my preference, however, ti 
 require that
 symbol used in shared lib are explicitly declared export. I 
 think we
 shouldn't break the virtual by default behavior, but we still 
 need to
 figure out a way to make thing more performant on this point.
You're concerned about the magnitude of breakage introducing an explicit virtual requirement. This would seem to be a much much bigger breakage to me, and I don't think it's intuitive.
Officially, we are just starting to support shared object. I know you use them, and that this is a breakage to you. However, I think it is reasonable to expect that most people do not. Still, the migration should be carefully handled. This change is actually a fairly big deal as it empower the optimizer quite a lot, and not only for virtual functions. It can do way more than the dev could manually annotating any function. This is proven technology, not the sufficiently smart compiler here. Ti be fair, I'd push for this change even if final was the default, as it enable for many other optimizations. As the example shown, manually annotating everything isn't going to provide 5% to 7%, which irrelevant in most code path, and when a compiler could improve the situation significantly automatically.
Jun 02 2013
prev sibling next sibling parent "Michal Minich" <michal.minich gmail.com> writes:
On Friday, 31 May 2013 at 04:06:58 UTC, deadalnix wrote:

 I don't think going as far as making thing final by default 
 make sense at this point. But we sure need a way to be able to 
 finalize methods. We had an extensive discussion with Don and 
 Manu at DConf, here are some idea that came out :

  - Final by default
[..]
  - Introduce a virtual keyword.
[..] Barely anyone complained. Mostly is considered a good thing to be explicit. As for the D, since the override keyword is mandatory, it would be a compiler error to omit a virtual keyword on base method, and thus easily fixable. As for the methods that are supposed to be virtual but never actual overridden - there you must know which ones that should be, and make that explicit. This applies only for libraries, and only the ones that use lot of methods intend to be overridden (generally questionable design) have some work to do on being explicit about virtual.
May 31 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, May 31, 2013 19:42:55 Manu wrote:
 On 31 May 2013 14:06, deadalnix <deadalnix gmail.com> wrote:
 On Friday, 31 May 2013 at 02:56:25 UTC, Andrei Alexandrescu wrote:
 On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.**com/u/974356/raytracer.d<https://dl.drop
 boxusercontent.com/u/974356/raytracer.d>
 https://dl.dropboxusercontent.**com/u/974356/raytracer.cpp<https://dl.d
 ropboxusercontent.com/u/974356/raytracer.cpp>>> 
Manu's gonna love this one: make all methods final.
I don't think going as far as making thing final by default make sense at this point. But we sure need a way to be able to finalize methods. We had an extensive discussion with Don and Manu at DConf, here are some idea that came out : - Final by default This one is really a plus when it come to performance code. However, virtual by default have proven itself very useful when performance isn't that big of a deal (and it is the case for 90% of a program's code usually) and limiting the usage of some pattern like decorator (that also have been proven to be useful). This is also huge breakage.
The correct choice :)
 virtual by default have proven itself very useful when performance isn't
that big of a deal Has it? I'm not sure it's ever proven its self useful, can you point to any such proof, or evidence?
It's useful when you don't have override, because you avoid the problem of overriding non-virtual functions. So, going with virtual by default solves that nasty bug, but enforcing override _also_ solves it. So IMHO, requiring override (which we already do) solves the problem, making it pointless to make member functions virtual by default. At that point, it just makes more sense to make them non-virtual by default and avoid the performance issues caused by virtual by default. The _only_ thing that it costs you is that it forces you to put virtual on all functions which are going to be overridden, which is even fewer functions than we have to mark with override (because multiple classes can override whereas only the base class function would be marked with virtual).
 There's another detail that came up late at night at dconf; I was talking
 to Daniel Murphy about the extern(C++) class work he's working on to write
 the D-DMD front end, and apparently it would make the problem a lot easier
 if virtual was explicit there too. So perhaps it also offers improved
 interoperability with C++, which I think most of us have agreed is of
 heightened importance recently.
Yes. I intend to create a DIP for non-virtual by default (assuming that no one else does first), and I think that this issue gives even more weight to why we need to make the change.
 This is also huge breakage.
I think this is another fallacy. The magnitude of the breakage is already known. It's comparable to the recent change where 'override' became an explicit requirement, which was hardly catastrophic. In that case, the breakage occurred in *every derived class*, which is a many:1 relationship to base classes. Explicit virtual would only affect _base_ classes. The magnitude of which is far smaller than what was recently considered acceptable, precisely: magnitudeOfPriorBreakage / numberOfDerivedClasses. So considering explicit override was an argument won, I'd like to think that explicit virtual deserves the same treatment for even more compelling reasons, and at a significantly lesser cost in breakage.
Yes. The breakage is surprisingly small, and the change is arguably critical for both performance-critical code and for moving the compiler frontend to D. - Jonathan M Davis
May 31 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 1 June 2013 02:32, Jonathan M Davis <jmdavisProg gmx.com> wrote:

 On Friday, May 31, 2013 19:42:55 Manu wrote:
 On 31 May 2013 14:06, deadalnix <deadalnix gmail.com> wrote:
 On Friday, 31 May 2013 at 02:56:25 UTC, Andrei Alexandrescu wrote:
 On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.**com/u/974356/raytracer.d<
https://dl.drop
 boxusercontent.com/u/974356/raytracer.d>
 https://dl.dropboxusercontent.**com/u/974356/raytracer.cpp<
https://dl.d
 ropboxusercontent.com/u/974356/raytracer.cpp>>>
Manu's gonna love this one: make all methods final.
I don't think going as far as making thing final by default make sense
at
 this point. But we sure need a way to be able to finalize methods. We
had
 an extensive discussion with Don and Manu at DConf, here are some idea
 that

 came out :
 - Final by default

 This one is really a plus when it come to performance code. However,
 virtual by default have proven itself very useful when performance
isn't
 that big of a deal (and it is the case for 90% of a program's code
 usually)
 and limiting the usage of some pattern like decorator (that also have
been
 proven to be useful). This is also huge breakage.
The correct choice :)
 virtual by default have proven itself very useful when performance
isn't
 that big of a deal

 Has it? I'm not sure it's ever proven its self useful, can you point to
any
 such proof, or evidence?
It's useful when you don't have override, because you avoid the problem of overriding non-virtual functions. So, going with virtual by default solves that nasty bug, but enforcing override _also_ solves it. So IMHO, requiring override (which we already do) solves the problem, making it pointless to make member functions virtual by default. At that point, it just makes more sense to make them non-virtual by default and avoid the performance issues caused by virtual by default. The _only_ thing that it costs you is that it forces you to put virtual on all functions which are going to be overridden, which is even fewer functions than we have to mark with override (because multiple classes can override whereas only the base class function would be marked with virtual).
 There's another detail that came up late at night at dconf; I was talking
 to Daniel Murphy about the extern(C++) class work he's working on to
write
 the D-DMD front end, and apparently it would make the problem a lot
easier
 if virtual was explicit there too. So perhaps it also offers improved
 interoperability with C++, which I think most of us have agreed is of
 heightened importance recently.
Yes. I intend to create a DIP for non-virtual by default (assuming that no one else does first), and I think that this issue gives even more weight to why we need to make the change.
 This is also huge breakage.
I think this is another fallacy. The magnitude of the breakage is already known. It's comparable to the recent change where 'override' became an explicit requirement, which was hardly catastrophic. In that case, the breakage occurred in *every derived class*, which is a many:1 relationship to base classes. Explicit virtual would only affect _base_ classes. The magnitude of which is far smaller than what was recently considered acceptable, precisely: magnitudeOfPriorBreakage / numberOfDerivedClasses. So considering explicit override was an argument won, I'd like to think that explicit virtual deserves the same treatment for even more
compelling
 reasons, and at a significantly lesser cost in breakage.
Yes. The breakage is surprisingly small, and the change is arguably critical for both performance-critical code and for moving the compiler frontend to D.
**applause** Someone other than me said it, out loud! This is a magnificent day! :)
May 31 2013
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On 31 May 2013 12:56, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 5/30/13 9:26 PM, finalpatch wrote:

 https://dl.dropboxusercontent.**com/u/974356/raytracer.d<https://dl.dropboxusercontent.com/u/974356/raytracer.d>
 https://dl.dropboxusercontent.**com/u/974356/raytracer.cpp<https://dl.dropboxusercontent.com/u/974356/raytracer.cpp>
Manu's gonna love this one: make all methods final.
Hehe, oh yeah! Coming from you, that's like music to my ears ;) His observed 5% is a massive speed up, almost 1ms across a 16ms frame!
May 30 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/30/2013 7:56 PM, Andrei Alexandrescu wrote:
 On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Manu's gonna love this one: make all methods final.
I have another suggestion. class Sphere and class Ray should be structs. Neither class uses polymorphism in any way, so there's no reason to make them classes with virtual functions.
Jun 01 2013
next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 01 Jun 2013 11:22:41 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 5/30/2013 7:56 PM, Andrei Alexandrescu wrote:
 On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Manu's gonna love this one: make all methods final.
I have another suggestion. class Sphere and class Ray should be structs. Neither class uses polymorphism in any way, so there's no reason to make them classes with virtual functions.
Yes, that's the easiest (and for Ray the right(tm)) way to get rid of virtual functions. But, I think it is safe to assume that there will be different object classes in the big picture. -- Marco
Jun 01 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, June 01, 2013 22:43:32 Marco Leise wrote:
 Am Sat, 01 Jun 2013 11:22:41 -0700
 
 schrieb Walter Bright <newshound2 digitalmars.com>:
 On 5/30/2013 7:56 PM, Andrei Alexandrescu wrote:
 On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Manu's gonna love this one: make all methods final.
I have another suggestion. class Sphere and class Ray should be structs. Neither class uses polymorphism in any way, so there's no reason to make them classes with virtual functions.
Yes, that's the easiest (and for Ray the right(tm)) way to get rid of virtual functions. But, I think it is safe to assume that there will be different object classes in the big picture.
If you don't need polymorphism, then in general, you shouldn't use a class (though sometimes it might make sense simply because it's an easy way to get a reference type). Where it becomes more of a problem is when you need a few polymorphic functions and a lot of non-polymorphic functions (e.g. when a class has a few methods which get overridden and then a lot of properties which it makes no sense to override). In that case, you have to use a class, and then you have to mark a lot of functions as final. This is what folks like Manu and Don really don't like, particularly when they're in environments where the extra cost of the virtual function calls actually matters. - Jonathan M Davis
Jun 01 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-06-01 23:08, Jonathan M Davis wrote:

 If you don't need polymorphism, then in general, you shouldn't use a class
 (though sometimes it might make sense simply because it's an easy way to get a
 reference type). Where it becomes more of a problem is when you need a few
 polymorphic functions and a lot of non-polymorphic functions (e.g. when a
 class has a few methods which get overridden and then a lot of properties
 which it makes no sense to override). In that case, you have to use a class,
 and then you have to mark a lot of functions as final. This is what folks like
 Manu and Don really don't like, particularly when they're in environments
 where the extra cost of the virtual function calls actually matters.
If a reference type is needed but not a polymorphic type, then a final class can be used. -- /Jacob Carlborg
Jun 02 2013
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, June 02, 2013 11:53:26 Jacob Carlborg wrote:
 On 2013-06-01 23:08, Jonathan M Davis wrote:
 If you don't need polymorphism, then in general, you shouldn't use a class
 (though sometimes it might make sense simply because it's an easy way to
 get a reference type). Where it becomes more of a problem is when you
 need a few polymorphic functions and a lot of non-polymorphic functions
 (e.g. when a class has a few methods which get overridden and then a lot
 of properties which it makes no sense to override). In that case, you
 have to use a class, and then you have to mark a lot of functions as
 final. This is what folks like Manu and Don really don't like,
 particularly when they're in environments where the extra cost of the
 virtual function calls actually matters.
If a reference type is needed but not a polymorphic type, then a final class can be used.
Yes. The main problem is when you have a class with a few methods which should be virtual and a lot that don't. You're forced to mark a large number of functions as final. That burden can be lessened by using final with a colon rather than marking them individually, but rather what seems to inevitably happen is that programmers forget to mark any of them as final (Manu can rant quite a bit about that, as he's had to deal with it at work, and it's cost him quite a bit of time, as he has to go through every function which wasn't marked as final and determine whether it's actuallly supposed to be virtual or not). Having non-virtual be the default makes functions efficient by default. - Jonathan M Davis
Jun 02 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 2 June 2013 19:53, Jacob Carlborg <doob me.com> wrote:

 On 2013-06-01 23:08, Jonathan M Davis wrote:

  If you don't need polymorphism, then in general, you shouldn't use a class
 (though sometimes it might make sense simply because it's an easy way to
 get a
 reference type). Where it becomes more of a problem is when you need a few
 polymorphic functions and a lot of non-polymorphic functions (e.g. when a
 class has a few methods which get overridden and then a lot of properties
 which it makes no sense to override). In that case, you have to use a
 class,
 and then you have to mark a lot of functions as final. This is what folks
 like
 Manu and Don really don't like, particularly when they're in environments
 where the extra cost of the virtual function calls actually matters.
If a reference type is needed but not a polymorphic type, then a final class can be used.
I've never said that virtuals are bad. The key function of a class is polymorphism. But the reality is that in non-tool or container/foundational classes (which are typically write-once, use-lots; you don't tend to write these daily), a typical class will have a couple of virtuals, and a whole bunch of properties. The majority of functions in OO code (in the sorts of classes that you tend to write daily, ie, the common case) are trivial accessors or properties. The cost of forgetting to type 'final' is severe, particularly so on a property, and there is absolutely nothing the compiler can do to help you. There's no reasonable warning it can offer either, it must presume you intended to do that. explicitly, so they will forget to write 'final' all the time. I can tell from hard experience, that despite training programmers that they need to write 'final', they have actually done so precisely ZERO TIMES EVER. People don't just forget the odd final here or there, in practise, they've never written it yet. The problem really goes pear shaped when faced with the task of opportunistic de-virtualisation - that is, de-virtualising functions that should never have been virtual in the first place, but are; perhaps because code has changed/evolved, but more likely, because uni tends to output programmers that are obsessed with the possibility of overriding everything ;) It becomes even worse than what we already have with C++, because now in D, I have to consider every single method and manually attempt to determine whether it should actually be virtual or not. A time consuming and certainly dangerous/error-prone task when I didn't author the code! In C++ I can at least identify the methods I need to consider by searching for 'virtual', saving maybe 80% of that horrible task by contrast. But there are other compelling reasons too, for instance during conversations with Daniel Murphy and others, it was noted that it will enhance interoperation with C++ (a key target for improvement being flagged recently), and further enabled the D DFE. I also think with explicit 'override' a requirement, it's rather non-orthogonal to not require explicit 'virtual' too. So, consider this reasonably. At least Myself and Don have both made strong claims to this end... and we're keen to pay for it with fixing broken base classes. Is it REALLY that much of an inconvenience to people to be explicit with 'virtual' (as they already are with 'override')? Is catering to that inconvenience worth the demonstrable cost? I'm not talking about minor nuisance, I'm talking about time and money.
Jun 02 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/2/13 9:59 AM, Manu wrote:
 I've never said that virtuals are bad. The key function of a class is
 polymorphism.
 But the reality is that in non-tool or container/foundational classes
 (which are typically write-once, use-lots; you don't tend to write these
 daily), a typical class will have a couple of virtuals, and a whole
 bunch of properties.
I've argued if no dispatch is needed just make those free functions. _Everything_ in a class is supposed to be overridable, unless inherited and explicitly "final"ized. It's sort of a historical accident that things got the way they are. But in D we know better because we have the module-level privacy model and UFCS. So we should break clean from history. Andrei
Jun 02 2013
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, June 02, 2013 12:37:38 Andrei Alexandrescu wrote:
 On 6/2/13 9:59 AM, Manu wrote:
 I've never said that virtuals are bad. The key function of a class is
 polymorphism.
 But the reality is that in non-tool or container/foundational classes
 (which are typically write-once, use-lots; you don't tend to write these
 daily), a typical class will have a couple of virtuals, and a whole
 bunch of properties.
I've argued if no dispatch is needed just make those free functions. _Everything_ in a class is supposed to be overridable, unless inherited and explicitly "final"ized. It's sort of a historical accident that things got the way they are. But in D we know better because we have the module-level privacy model and UFCS. So we should break clean from history.
There are four problems with that: 1. Very few programmers think that way. The normal thing in most every OO language is to put all of the functions on the class, so pretty much no one is going to make them free functions. Do you really expect people to put properties (what would have been getters and setters in other languages) outside the class? It's not going to happen. And by default, everyone takes a performance hit as a result (folks like Manu and Don wcare about that performance hit more than many of us, but it's still there). 2. The class' functions are no longer encapsulated inside the class. For some programmers, this is a big deal. They want all of the class' functionality on the class where it's easy to find. Having UFCS makes using free functions less of a problem, but many programmers will absolutely hate the idea of putting their non-virtual functions outside of the class, so they won't do it, and they (and everyone using their code) will end up with virtual functions when the functions shouldn't be virtual. 3. In many cases, putting a function outside of the class is a royal pain. This is particularly true with templated classes. If you have class C(T) { private T _var; } and you want a property to give you var, you end up with something ugly like property auto var(U)(U this_) if(is(U V == C!W, W)) { return this_._var; } Do you really expect many programmers to be able to pull off complicated is expressions like that? _I_ have to look it up every time I use it. Sure, using free functions might work in simple cases, but it starts falling apart when you have to deal with stuff like templated types. 4. It causes more ambiguities and compilation errors. If a function is on the class, it always wins. If it's a free function, then you potentially have ambiguities due to overload sets. Making sure that the function takes the exact type significantly reduces the problem, but someone else could easily create a function with the same signature which now conflicts with the one which is effectively supposed to be a member function. I also don't see what trying to turn all of these member functions into free functions to use with UFCS buys us. Given that we have virtual by default, it's one way to avoid the function being virtual, but it's just simpler to mark it final if that's what you're trying to do. I think that the view point that every function in a class is supposed to be overridable is demonstrably false. How many functions in your average class get overriden? How often does final get used (or virtual not get used in the case of C++)? I think that it's quite clear that a lot of programmers don't want all of their class functions to be virtual. It seems to me that your argument is based purely on the idea that the main reason to use a class is for polymorphism, but that's not the only reason to use a class, and just because you have a class doesn't mean that you want everything it does to be polymorphic. And you're saying that everything that isn't polymorphic doesn't belong in a class just because the main purpose of a class is polymorphism. I don't buy that at all, and the average programmer agreed with you, they would have been using free functions all along, but the vast majority of them put all of the functions on the class, and I don't think that UFCS is going to change that at all. - Jonathan M Davis
Jun 02 2013
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06/02/2013 10:44 PM, Jonathan M Davis wrote:
 On Sunday, June 02, 2013 12:37:38 Andrei Alexandrescu wrote:
 On 6/2/13 9:59 AM, Manu wrote:
 I've never said that virtuals are bad. The key function of a class is
 polymorphism.
 But the reality is that in non-tool or container/foundational classes
 (which are typically write-once, use-lots; you don't tend to write these
 daily), a typical class will have a couple of virtuals, and a whole
 bunch of properties.
I've argued if no dispatch is needed just make those free functions. _Everything_ in a class is supposed to be overridable, unless inherited and explicitly "final"ized. It's sort of a historical accident that things got the way they are. But in D we know better because we have the module-level privacy model and UFCS. So we should break clean from history.
There are four problems with that: 1. Very few programmers think that way. The normal thing in most every OO language is to put all of the functions on the class, so pretty much no one is going to make them free functions. Do you really expect people to put properties (what would have been getters and setters in other languages) outside the class? It's not going to happen. And by default, everyone takes a performance hit as a result (folks like Manu and Don wcare about that performance hit more than many of us, but it's still there). ...
Yup.
 2. The class' functions are no longer encapsulated inside the class. For some
 programmers, this is a big deal. They want all of the class' functionality on
 the class where it's easy to find. Having UFCS makes using free functions less
 of a problem, but many programmers will absolutely hate the idea of putting
 their non-virtual functions outside of the class, so they won't do it, and
 they (and everyone using their code) will end up with virtual functions when
 the functions shouldn't be virtual.
This is 1. again. (The unit of encapsulation in D is a module.)
 3. In many cases, putting a function outside of the class is a royal pain.
 This is particularly true with templated classes. If you have

 class C(T)
 {
     private T _var;
 }

 and you want a property to give you var, you end up with something ugly like

  property auto var(U)(U this_)
      if(is(U V == C!W, W))
 {
      return this_._var;
 }

 Do you really expect many programmers to be able to pull off complicated is
 expressions like that? _I_ have to look it up every time I use it. Sure, using
 free functions might work in simple cases, but it starts falling apart when
 you have to deal with stuff like templated types.
1. Just do property auto var(T)(C!T this_){ return this_._var; } 2. The is-expression is not that complicated. The latest release finally fixes the issue that you needed to specify a redundant identifier to make it work: property auto var(U)(U this_)if(is(U==C!T,T)){ return this_._var; }
 4. It causes more ambiguities and compilation errors. If a function is on the
 class, it always wins. If it's a free function, then you potentially have
 ambiguities due to overload sets. Making sure that the function takes the
 exact type significantly reduces the problem, but someone else could easily
 create a function with the same signature which now conflicts with the one
 which is effectively supposed to be a member function.
 ...
Well, yes, but I think this will not be a huge problem in practise. (The private symbol clash stupidity may exacerbate the problem when it occurs though.) I wouldn't mind if the default changed. (But I currently cannot update my compiler anyway, because of hard-to-reduce forward reference bugs.)
Jun 02 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, June 02, 2013 13:44:33 Jonathan M Davis wrote:
 On Sunday, June 02, 2013 12:37:38 Andrei Alexandrescu wrote:
 On 6/2/13 9:59 AM, Manu wrote:
 I've never said that virtuals are bad. The key function of a class is
 polymorphism.
 But the reality is that in non-tool or container/foundational classes
 (which are typically write-once, use-lots; you don't tend to write these
 daily), a typical class will have a couple of virtuals, and a whole
 bunch of properties.
I've argued if no dispatch is needed just make those free functions. _Everything_ in a class is supposed to be overridable, unless inherited and explicitly "final"ized. It's sort of a historical accident that things got the way they are. But in D we know better because we have the module-level privacy model and UFCS. So we should break clean from history.
There are four problems with that:
[snip] Another issue is extern(C++). In discussions at dconf, it became clear that we need to support non-virtual member functions with extern(C++) in order to move the compiler to D. Daniel Murphy has been working on this and could give better details, but the UFCS trick isn't going to work for that, and the fact that functions are virtual by default makes it problematic at best and possibly prevents it outright. We actually need extern(C++) classes. - Jonathan M Davis
Jun 02 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 3 June 2013 02:37, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/2/13 9:59 AM, Manu wrote:

 I've never said that virtuals are bad. The key function of a class is
 polymorphism.
 But the reality is that in non-tool or container/foundational classes
 (which are typically write-once, use-lots; you don't tend to write these
 daily), a typical class will have a couple of virtuals, and a whole
 bunch of properties.
I've argued if no dispatch is needed just make those free functions.
You're not going to win many friends, and probably not many potential D users by insisting people completely change their coding patterns that they've probably held for decades on a trivial matter like this. And if that's to be realistic, there are some problems that need to be addressed with that. The template issue for a start. _Everything_ in a class is supposed to be overridable, unless inherited and
 explicitly "final"ized.
Who says? I certainly wouldn't say that. Sounds like a horrible idea to me. It sounds like a disaster waiting to happen to me. There are functions that the author intended to be overridden, and functions that have no business being overridden, that the author probably never imagined anyone would override. What if someone does come along and override one of these, and it was never designed to work under that circumstance in the first place? At very least, it will have never been tested. That's not a very robust API offering if you ask me. It's sort of a historical accident that things got the way they are. But in
 D we know better because we have the module-level privacy model and UFCS.
 So we should break clean from history.
Why? Can you actually justify what's bad about a class containing it's properties/accessors? (I've read the article you pointed me at once, it was interesting, but didn't convince me) Interestingly, you didn't actually disagree with my point about the common case here, and I don't buy the Java doctrine.
Jun 03 2013
next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Monday, 3 June 2013 at 07:06:05 UTC, Manu wrote:
 There are functions that
 the author intended to be overridden, and functions that have 
 no business
 being overridden, that the author probably never imagined 
 anyone would
 override.
 What if someone does come along and override one of these, and 
 it was never
 designed to work under that circumstance in the first place?
 At very least, it will have never been tested. That's not a 
 very robust API
 offering if you ask me.
This is something just as important as the performance issues. Most of the time people will leave functions to simply use whatever the default is for virtual/final. If it's final, this works fairly well. The author hasn't put in the effort to decide how to handle people overriding their function. But with virtual by default, you don't know if the author actually considered that people will be overriding the function or if it's simply that they didn't bother specifying. I know the vast majority of my code is virtual, simply because I didn't specify the 'final' keyword 500 times, and didn't think about that I'd need to do it. The resulting code is unsafe because I didn't consider those functions actually being overridden. A standard example is the addRange vs add functions. If these are left as default, you have no clue if the author considered that someone would be overriding it. Will addRange call add, or do you need to override both? Is this an implementation detail that may change at any moment? Or when a different class overrides addRange and makes it call add? By forcing the author to write 'virtual', you know that they at least *considered* that someone may override it, and hopefully thought about the consequences or documented whether addRange calls add. Or they may think about it later and realize that it's a mistake to have left it default and make addRange final. Going from final to virtual is fine; going from virtual to final breaks code. And of course, the vast majority of functions should *not* be virtual. So why is it the default to have to specify 'final' for all of these functions? I think that there are substantial benefits to final being default, but breaking every single program that uses inheritance is a pretty big issue. Still, fixing the code would be quite trivial if you can see a list of the functions that need to be made virtual (either by specifying --transition=virtual or just looking at the compiler errors that pop up when you build). It would even be possible to write a tool to automatically update code using the results of --transition if it was implemented in a way that gave enough details. Unfortunately this would only solve the issue of making your code work again, you would still have to go through everything and decide if it should be virtual.
Jun 03 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 3 June 2013 at 07:30:56 UTC, Kapps wrote:
 On Monday, 3 June 2013 at 07:06:05 UTC, Manu wrote:
 There are functions that
 the author intended to be overridden, and functions that have 
 no business
 being overridden, that the author probably never imagined 
 anyone would
 override.
 What if someone does come along and override one of these, and 
 it was never
 designed to work under that circumstance in the first place?
 At very least, it will have never been tested. That's not a 
 very robust API
 offering if you ask me.
This is something just as important as the performance issues. Most of the time people will leave functions to simply use whatever the default is for virtual/final. If it's final, this works fairly well. The author hasn't put in the effort to decide how to handle people overriding their function. But with virtual by default, you don't know if the author actually considered that people will be overriding the function or if it's simply that they didn't bother specifying. I know the vast majority of my code is virtual, simply because I didn't specify the 'final' keyword 500 times, and didn't think about that I'd need to do it. The resulting code is unsafe because I didn't consider those functions actually being overridden.
The whole concept of OOP revolve around the fact that a given class and users of the given class don't need to know about its subclasses (Liskov's substitution principle). It is subclass's responsibility to decide what it override or not, not the upper class to decide what is overriden by subclasses. If you want to create a class with customizable parts, pass parameters to the constructor. This isn't OOP what OOP is about. The performance concern is only here because things has been smashed together in a inconsequent way (as it is often done in D). In Java for instance, only overriden function are actually virtual. Everything else is finalized at link time. Which is great because you are able to override everything when testing to create mock for instance, while keeping good performance when actually running the application.
Jun 03 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-06-03 10:11, deadalnix wrote:

 The whole concept of OOP revolve around the fact that a given class and
 users of the given class don't need to know about its subclasses
 (Liskov's substitution principle). It is subclass's responsibility to
 decide what it override or not, not the upper class to decide what is
 overriden by subclasses.

 If you want to create a class with customizable parts, pass parameters
 to the constructor. This isn't OOP what OOP is about.

 The performance concern is only here because things has been smashed
 together in a inconsequent way (as it is often done in D). In Java for
 instance, only overriden function are actually virtual. Everything else
 is finalized at link time. Which is great because you are able to
 override everything when testing to create mock for instance, while
 keeping good performance when actually running the application.
I've read a book, Effective Java, where it says, something like: If you don't intend your class to be subclassed make it final, otherwise document how to subclass and which methods to override. -- /Jacob Carlborg
Jun 03 2013
parent reply Manu <turkeyman gmail.com> writes:
On 3 June 2013 18:20, Jacob Carlborg <doob me.com> wrote:

 On 2013-06-03 10:11, deadalnix wrote:

  The whole concept of OOP revolve around the fact that a given class and
 users of the given class don't need to know about its subclasses
 (Liskov's substitution principle). It is subclass's responsibility to
 decide what it override or not, not the upper class to decide what is
 overriden by subclasses.

 If you want to create a class with customizable parts, pass parameters
 to the constructor. This isn't OOP what OOP is about.

 The performance concern is only here because things has been smashed
 together in a inconsequent way (as it is often done in D). In Java for
 instance, only overriden function are actually virtual. Everything else
 is finalized at link time. Which is great because you are able to
 override everything when testing to create mock for instance, while
 keeping good performance when actually running the application.
I've read a book, Effective Java, where it says, something like: If you don't intend your class to be subclassed make it final, otherwise document how to subclass and which methods to override.
Sounds like even they know the truth I speak, but they must enforce this by convention/documentation rather than offering strict guarantees ;) modern decided to go the C++ way rather than the Java way.
Jun 03 2013
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 03.06.2013 18:19, schrieb Manu:
 On 3 June 2013 18:20, Jacob Carlborg <doob me.com <mailto:doob me.com>>
 wrote:

     On 2013-06-03 10:11, deadalnix wrote:

         The whole concept of OOP revolve around the fact that a given
         class and
         users of the given class don't need to know about its subclasses
         (Liskov's substitution principle). It is subclass's
         responsibility to
         decide what it override or not, not the upper class to decide
         what is
         overriden by subclasses.

         If you want to create a class with customizable parts, pass
         parameters
         to the constructor. This isn't OOP what OOP is about.

         The performance concern is only here because things has been smashed
         together in a inconsequent way (as it is often done in D). In
         Java for
         instance, only overriden function are actually virtual.
         Everything else
         is finalized at link time. Which is great because you are able to
         override everything when testing to create mock for instance, while
         keeping good performance when actually running the application.


     I've read a book, Effective Java, where it says, something like:

     If you don't intend your class to be subclassed make it final,
     otherwise document how to subclass and which methods to override.


 Sounds like even they know the truth I speak, but they must enforce this
 by convention/documentation rather than offering strict guarantees ;)

 modern decided to go the C++ way rather than the Java way.
That's why. You have to thank Anders for it. -- Paulo
Jun 03 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, June 03, 2013 10:11:26 deadalnix wrote:
 The whole concept of OOP revolve around the fact that a given
 class and users of the given class don't need to know about its
 subclasses (Liskov's substitution principle). It is subclass's
 responsibility to decide what it override or not, not the upper
 class to decide what is overriden by subclasses.
It's the base class' job to define the API that derived classes will be overriding and the derived classes' choice as to exactly which functions they override (assuming that they're not abstract and therefore have to be overridden). That doesn't mean that the base class can't or shouldn't have other functions which are not intended to be overridden. Nothing about Liskov's substitution principle requires that the entire API of the base class be polymorphic. - Jonathan M Davis
Jun 03 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 3 June 2013 18:11, deadalnix <deadalnix gmail.com> wrote:

 On Monday, 3 June 2013 at 07:30:56 UTC, Kapps wrote:

 On Monday, 3 June 2013 at 07:06:05 UTC, Manu wrote:

 There are functions that
 the author intended to be overridden, and functions that have no business
 being overridden, that the author probably never imagined anyone would
 override.
 What if someone does come along and override one of these, and it was
 never
 designed to work under that circumstance in the first place?
 At very least, it will have never been tested. That's not a very robust
 API
 offering if you ask me.
This is something just as important as the performance issues. Most of the time people will leave functions to simply use whatever the default is for virtual/final. If it's final, this works fairly well. The author hasn't put in the effort to decide how to handle people overriding their function. But with virtual by default, you don't know if the author actually considered that people will be overriding the function or if it's simply that they didn't bother specifying. I know the vast majority of my code is virtual, simply because I didn't specify the 'final' keyword 500 times, and didn't think about that I'd need to do it. The resulting code is unsafe because I didn't consider those functions actually being overridden.
The whole concept of OOP revolve around the fact that a given class and users of the given class don't need to know about its subclasses (Liskov's substitution principle). It is subclass's responsibility to decide what it override or not, not the upper class to decide what is overriden by subclasses.
Then OOP is fundamentally unsafe, because the author will never consider all the possibilities! If you want to create a class with customizable parts, pass parameters to
 the constructor. This isn't OOP what OOP is about.
Eh? The performance concern is only here because things has been smashed
 together in a inconsequent way (as it is often done in D). In Java for
 instance, only overriden function are actually virtual. Everything else is
 finalized at link time.
Java is not compiled. If you compile Java code, all functions are virtual always. It's impossible in D with separate compilation, and dynamic libraries seal the deal.
 Which is great because you are able to override everything when testing to
 create mock for instance, while keeping good performance when actually
 running the application.
I'm not taking away your ability to make everything virtual, you can type 'virtual:' as much as you like.
Jun 03 2013
next sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 03.06.2013 18:16, schrieb Manu:
 On 3 June 2013 18:11, deadalnix <deadalnix gmail.com
 <mailto:deadalnix gmail.com>> wrote:

     On Monday, 3 June 2013 at 07:30:56 UTC, Kapps wrote:

         On Monday, 3 June 2013 at 07:06:05 UTC, Manu wrote:

             There are functions that
             the author intended to be overridden, and functions that
             have no business
             being overridden, that the author probably never imagined
             anyone would
             override.
             What if someone does come along and override one of these,
             and it was never
             designed to work under that circumstance in the first place?
             At very least, it will have never been tested. That's not a
             very robust API
             offering if you ask me.


         This is something just as important as the performance issues.
         Most of the time people will leave functions to simply use
         whatever the default is for virtual/final. If it's final, this
         works fairly well. The author hasn't put in the effort to decide
         how to handle people overriding their function. But with virtual
         by default, you don't know if the author actually considered
         that people will be overriding the function or if it's simply
         that they didn't bother specifying. I know the vast majority of
         my code is virtual, simply because I didn't specify the 'final'
         keyword 500 times, and didn't think about that I'd need to do
         it. The resulting code is unsafe because I didn't consider those
         functions actually being overridden.


     The whole concept of OOP revolve around the fact that a given class
     and users of the given class don't need to know about its subclasses
     (Liskov's substitution principle). It is subclass's responsibility
     to decide what it override or not, not the upper class to decide
     what is overriden by subclasses.


 Then OOP is fundamentally unsafe, because the author will never consider
 all the possibilities!

     If you want to create a class with customizable parts, pass
     parameters to the constructor. This isn't OOP what OOP is about.


 Eh?

     The performance concern is only here because things has been smashed
     together in a inconsequent way (as it is often done in D). In Java
     for instance, only overriden function are actually virtual.
     Everything else is finalized at link time.


 Java is not compiled. If you compile Java code, all functions are
 virtual always.
That depends, http://www.excelsior-usa.com/jet.html
Jun 03 2013
parent reply "w0rp" <devw0rp gmail.com> writes:
The virtual vs. non-virtual thing doesn't really matter that 
much. We're talking about a 5% performance difference here, at 
most. You just type 'final' a little bit here and there when 
you're near the end of writing our your class hierarchy and get a 
little performance boost.
Jun 03 2013
next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On 2013-06-03, 20:00, w0rp wrote:

 The virtual vs. non-virtual thing doesn't really matter that much. We're  
 talking about a 5% performance difference here, at most. You just type  
 'final' a little bit here and there when you're near the end of writing  
 our your class hierarchy and get a little performance boost.
5%? Not when the function you call in an inner loop is virtual. I see up to a 100% time increase for simple functions. -- Simen
Jun 03 2013
prev sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 03.06.2013 20:00, schrieb w0rp:
 The virtual vs. non-virtual thing doesn't really matter that much. We're
 talking about a 5% performance difference here, at most. You just type
 'final' a little bit here and there when you're near the end of writing
 our your class hierarchy and get a little performance boost.
Personally I don't care. I learned OOP with Turbo Pascal and C++, both use static by default. After that I lost count how many OO languages I have used since those days, each with its pluses and minus. Since I only use D for playing around outside work, I'll just go with what the real D users decide for the language. -- Paulo
Jun 03 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 3 June 2013 at 16:16:48 UTC, Manu wrote:
 Java is not compiled. If you compile Java code, all functions 
 are virtual
 always.
Java is JIT compiled and function that aren't overriden are finalized automatically by the JIT compiler. It's even able to virtualize at runtime if an override is finally linked, or the other way around, refinalize. We could do that in D as LTO, as long as we have a way to tell the compiler if a function can be overriden in a shared object.
Jun 03 2013
prev sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
On Monday, 3 June 2013 at 16:16:48 UTC, Manu wrote:
 It's impossible in D with separate compilation, and dynamic 
 libraries seal
 the deal.
Not really. It will be possible the very moment (if) "export" gets strict so that everything not marked with "export" won't be available for outer world.
Jun 03 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, June 03, 2013 22:15:15 Dicebot wrote:
 On Monday, 3 June 2013 at 16:16:48 UTC, Manu wrote:
 It's impossible in D with separate compilation, and dynamic
 libraries seal
 the deal.
Not really. It will be possible the very moment (if) "export" gets strict so that everything not marked with "export" won't be available for outer world.
1. That'll only affect Windows unless we change the linking model on *nix systems. 2. That'll only affect stuff that isn't exported from a shared library. There are plenty of cases where a class is exported from a shared library, and it has lots of functions on it which are supposed to be non-virtual. 3. Doesn't doing this require that the _linker_ optimize out the virtuality of the functions for you? If that's the case, it won't work any time soon (if ever), because we use the C linker, not our own. So, I really don't think that it makes sense to say that export is going to fix much of anything with regards to virtuality. - Jonathan M Davis
Jun 03 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 3 June 2013 at 23:47:33 UTC, Jonathan M Davis wrote:
 1. That'll only affect Windows unless we change the linking 
 model on *nix
 systems.
It is evolving on the C/C++ side, so I see no point in being more conservative than theses.
 2. That'll only affect stuff that isn't exported from a shared 
 library. There
 are plenty of cases where a class is exported from a shared 
 library, and it
 has lots of functions on it which are supposed to be 
 non-virtual.
Calling into/from a shared lib is doomed to be a performance hit as the called code is opaque to the compiler anyway. Which mean assuming the worse on the caller side and disabling most optimizations.
 3. Doesn't doing this require that the _linker_ optimize out 
 the virtuality of
 the functions for you? If that's the case, it won't work any 
 time soon (if
 ever), because we use the C linker, not our own.
I'm not sure what it imply for GCC, but this is actually not hard to implement in LLVM.
Jun 03 2013
prev sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 03.06.2013 10:11, schrieb deadalnix:
 On Monday, 3 June 2013 at 07:30:56 UTC, Kapps wrote:
 On Monday, 3 June 2013 at 07:06:05 UTC, Manu wrote:
 There are functions that
 the author intended to be overridden, and functions that have no
 business
 being overridden, that the author probably never imagined anyone would
 override.
 What if someone does come along and override one of these, and it was
 never
 designed to work under that circumstance in the first place?
 At very least, it will have never been tested. That's not a very
 robust API
 offering if you ask me.
This is something just as important as the performance issues. Most of the time people will leave functions to simply use whatever the default is for virtual/final. If it's final, this works fairly well. The author hasn't put in the effort to decide how to handle people overriding their function. But with virtual by default, you don't know if the author actually considered that people will be overriding the function or if it's simply that they didn't bother specifying. I know the vast majority of my code is virtual, simply because I didn't specify the 'final' keyword 500 times, and didn't think about that I'd need to do it. The resulting code is unsafe because I didn't consider those functions actually being overridden.
The whole concept of OOP revolve around the fact that a given class and users of the given class don't need to know about its subclasses (Liskov's substitution principle). It is subclass's responsibility to decide what it override or not, not the upper class to decide what is overriden by subclasses. If you want to create a class with customizable parts, pass parameters to the constructor. This isn't OOP what OOP is about. The performance concern is only here because things has been smashed together in a inconsequent way (as it is often done in D). In Java for instance, only overriden function are actually virtual. Everything else is finalized at link time. Which is great because you are able to override everything when testing to create mock for instance, while keeping good performance when actually running the application.
While this is true for most OO languages, you should not forget the "Fragile base class" principle. But this applies to both cases, regardless what is defined by default. -- Paulo
Jun 03 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/13 3:05 AM, Manu wrote:
 On 3 June 2013 02:37, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org
 <mailto:SeeWebsiteForEmail erdani.org>> wrote:

     On 6/2/13 9:59 AM, Manu wrote:

         I've never said that virtuals are bad. The key function of a
         class is
         polymorphism.
         But the reality is that in non-tool or container/foundational
         classes
         (which are typically write-once, use-lots; you don't tend to
         write these
         daily), a typical class will have a couple of virtuals, and a whole
         bunch of properties.


     I've argued if no dispatch is needed just make those free functions.


 You're not going to win many friends, and probably not many potential D
 users by insisting people completely change their coding patterns that
 they've probably held for decades on a trivial matter like this.
This is actually part of the point. You keep on discussing as if we design the language now, when in fact there's a lot of code out there that relies on the current behavior. We won't win many friends if we break every single method that has ever been overridden in D, over a trivial matter. Andrei
Jun 03 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 3 June 2013 at 15:27:58 UTC, Andrei Alexandrescu wrote:
 This is actually part of the point. You keep on discussing as 
 if we design the language now, when in fact there's a lot of 
 code out there that relies on the current behavior. We won't 
 win many friends if we break every single method that has ever 
 been overridden in D, over a trivial matter.
Agreed. But can you please consider the export proposal ? This would allow for finalization using LTO. That we can avoid virtual dispatch when it is unneeded.
Jun 03 2013
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 01:28, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/3/13 3:05 AM, Manu wrote:

 On 3 June 2013 02:37, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org
 <mailto:SeeWebsiteForEmail **erdani.org <SeeWebsiteForEmail erdani.org>>>
 wrote:

     On 6/2/13 9:59 AM, Manu wrote:

         I've never said that virtuals are bad. The key function of a
         class is
         polymorphism.
         But the reality is that in non-tool or container/foundational
         classes
         (which are typically write-once, use-lots; you don't tend to
         write these
         daily), a typical class will have a couple of virtuals, and a
 whole
         bunch of properties.


     I've argued if no dispatch is needed just make those free functions.


 You're not going to win many friends, and probably not many potential D
 users by insisting people completely change their coding patterns that
 they've probably held for decades on a trivial matter like this.
This is actually part of the point. You keep on discussing as if we design the language now, when in fact there's a lot of code out there that relies on the current behavior. We won't win many friends if we break every single method that has ever been overridden in D, over a trivial matter.
You won't break every single method, they already went through that recently when override was made a requirement. It will only break the base declarations, which are far less numerous. How can you justify the change to 'override' with a position like that? We have already discussed that we know PRECISELY the magnitude of breakage that will occur. It is: magnitude_of_breakage_from_override / total_number_of_derived_classes. A much smaller number than the breakage which was gladly accepted recently. And the matter is far from trivial. In fact, if you think this is trivial, then how did the override change ever get accepted? That is most certainly trivial by contrast, and far more catastrophic in terms of breakage.
Jun 03 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/13 12:25 PM, Manu wrote:
 You won't break every single method, they already went through that
 recently when override was made a requirement.
 It will only break the base declarations, which are far less numerous.
That's what I meant.
 How can you justify the change to 'override' with a position like that?
 We have already discussed that we know PRECISELY the magnitude of
 breakage that will occur.
 It is: magnitude_of_breakage_from_override /
 total_number_of_derived_classes. A much smaller number than the breakage
 which was gladly accepted recently.
Well it's kinda too much relativism that the number of breakages is considered small because it's smaller than another number.
 And the matter is far from trivial.
It is trivial. To paraphrase a classic: "I'm not taking away your ability to make everything final, you can type 'final:' as much as you like."
 In fact, if you think this is
 trivial, then how did the override change ever get accepted? That is
 most certainly trivial by contrast, and far more catastrophic in terms
 of breakage.
That's a completely different issue, so this part of the argument can be considered destroyed. Andrei
Jun 03 2013
next sibling parent "Byron Heads" <byron.heads gmail.com> writes:
On Monday, 3 June 2013 at 16:59:43 UTC, Andrei Alexandrescu wrote:
 It is trivial. To paraphrase a classic: "I'm not taking away 
 your ability to make everything final, you can type 'final:' as 
 much as you like."
Wont we at least need 'virtual:' for the inverse of 'final:'?
Jun 03 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 4 June 2013 02:59, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/3/13 12:25 PM, Manu wrote:

 You won't break every single method, they already went through that
 recently when override was made a requirement.
 It will only break the base declarations, which are far less numerous.
That's what I meant. How can you justify the change to 'override' with a position like that?
 We have already discussed that we know PRECISELY the magnitude of
 breakage that will occur.
 It is: magnitude_of_breakage_from_**override /
 total_number_of_derived_**classes. A much smaller number than the
 breakage
 which was gladly accepted recently.
Well it's kinda too much relativism that the number of breakages is considered small because it's smaller than another number. And the matter is far from trivial.

 It is trivial. To paraphrase a classic: "I'm not taking away your ability
 to make everything final, you can type 'final:' as much as you like."
You read my posts where I illustrate how going virtual is a one-way ride right? You can safely change to virtual, but you can't go from virtual back to final. Aside from the performance hazards, and the empirical evidence from watching C++ programmers use D, virtual-by-default has the dangerous potential to lock you in to an awkward situation where you can never safely make a change to fix something that should never have been that way in the first place. In fact, if you think this is
 trivial, then how did the override change ever get accepted? That is
 most certainly trivial by contrast, and far more catastrophic in terms
 of breakage.
That's a completely different issue, so this part of the argument can be considered destroyed.
Rubbish. A similar set of considerations apply, and there's an awful lot more supporting this one in addition.
Jun 03 2013
prev sibling next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Monday, 3 June 2013 at 16:25:24 UTC, Manu wrote:
 You won't break every single method, they already went through 
 that recently when override was made a requirement. […] A much
 smaller number than the breakage
 which was gladly accepted recently. […] how did the override
 change ever get accepted […]
It appears as if either you have a interesting definition of "recently", or you are deliberately misleading people by bringing up that point over and over again. According to http://dlang.org/changelog.html, omitting "override" produced a warning since D 2.004, which was released back in September 2007! Granted, it was only actually turned from a deprecation warning into an actual deprecation in 2.061 (if my memory serves me right), but it's mostly a flaw in the handling of that particular deprecation that it stayed at the first level for so long. The actual language change was made – and user-visible – almost six (!) years ago, which is a lot on the D time scale. You are also ignoring the fact that in contrast to requiring "override", there is no clean deprecation path for your proposal, at least as far as I can see: Omitting the keyword started out as a warning, and IIRC still is allowed when you enable deprecated features via the compiler switch. How would a similar process look for virtual-by-default? As far as am isolated module with only a base class is concerned, this is not question of valid vs. invalid code, but a silent change in language semantics. From DConf I know that you are actually are a friendly, reasonable person, but in this discussion, you really come across as a narrow-minded zealot to me. So, please, let's focus on finding an actually practical solution! For example, if we had !pure/!nothrow/!final or something along the lines, just mandate that "final:" is put at the top of everything in your style guide (easily machine-enforceable too) – problem solved? And maybe it would even catch on in the whole D community and lead to a language change in D3 or a future iteration of the language. David
Jun 03 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/13 1:06 PM, David Nadlinger wrote:
 On Monday, 3 June 2013 at 16:25:24 UTC, Manu wrote:
 You won't break every single method, they already went through that
 recently when override was made a requirement. […] A much
 smaller number than the breakage
 which was gladly accepted recently. […] how did the override
 change ever get accepted […]
It appears as if either you have a interesting definition of "recently", or you are deliberately misleading people by bringing up that point over and over again. According to http://dlang.org/changelog.html, omitting "override" produced a warning since D 2.004, which was released back in September 2007! Granted, it was only actually turned from a deprecation warning into an actual deprecation in 2.061 (if my memory serves me right), but it's mostly a flaw in the handling of that particular deprecation that it stayed at the first level for so long. The actual language change was made – and user-visible – almost six (!) years ago, which is a lot on the D time scale. You are also ignoring the fact that in contrast to requiring "override", there is no clean deprecation path for your proposal, at least as far as I can see: Omitting the keyword started out as a warning, and IIRC still is allowed when you enable deprecated features via the compiler switch. How would a similar process look for virtual-by-default? As far as am isolated module with only a base class is concerned, this is not question of valid vs. invalid code, but a silent change in language semantics.
[snip] There's one more issue with the comparison that must be clarified (I thought it was fairly obvious so I didn't make it explicit in my previous note): override is not comparable because it improves code correctness and maintainability, for which there is ample prior evidence. It's also a matter for which, unlike virtual/final, there is no reasonable recourse. So invoking the cost of imposing explicit override vs imposing virtual as an argument for the latter is fallacious. Andrei
Jun 03 2013
parent reply "Michal Minich" <michal.minich gmail.com> writes:
On Monday, 3 June 2013 at 17:18:55 UTC, Andrei Alexandrescu wrote:
  override is not comparable
 because it improves code correctness and maintainability, for 
 which there is ample prior evidence. It's also a matter for 
 which, unlike virtual/final, there is no reasonable recourse.
Virtual by default makes it simpler to call method on object that is not initialized yet (constructor not called yet). This situation is possible regardless if virtual is default or not (it can just happen more easily). I think this calling virtual function in constructor should generate a warning. (I wouldn't be surprised if there is enhancement request filed for this already) module test; import std.stdio; class Base { this () { writeln("Base.this"); foo(); } void foo () { writeln("Base.foo"); } } class Derived : Base { this () { writeln("Derived.this"); } override void foo () { writeln("Derifed.foo"); } } void main () { auto d = new Derived; } Program output: Base.this Derifed.foo // Derived.foo is called before object constructor Derived.this
Jun 05 2013
parent reply "Regan Heath" <regan netmail.co.nz> writes:
On Wed, 05 Jun 2013 13:53:58 +0100, Michal Minich  
<michal.minich gmail.com> wrote:

 On Monday, 3 June 2013 at 17:18:55 UTC, Andrei Alexandrescu wrote:
   override is not comparable
 because it improves code correctness and maintainability, for which  
 there is ample prior evidence. It's also a matter for which, unlike  
 virtual/final, there is no reasonable recourse.
Virtual by default makes it simpler to call method on object that is not initialized yet (constructor not called yet). This situation is possible regardless if virtual is default or not (it can just happen more easily).
Yeah, it happened to me in C++ .. same with virtuals in the destructor. Lesson learned first time tho :p
 I think this calling virtual function in constructor should generate a  
 warning. (I wouldn't be surprised if there is enhancement request filed  
 for this already)
With virtual by default, could D statically verify/deny these? What about with static by default? Does it get easier or harder to detect/deny these in either case? R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Jun 05 2013
next sibling parent "Michal Minich" <michal.minich gmail.com> writes:
On Wednesday, 5 June 2013 at 14:08:46 UTC, Regan Heath wrote:

 With virtual by default, could D statically verify/deny these?  
 What about with static by default?  Does it get easier or 
 harder to detect/deny these in either case?
Without understanding dmd internals, the implementation effort should be exactly the same regardless of default. And it should be easy too, like checking a flag on function call expression: walk each ast tree item in constructor if funcApplicattion.isVirtual then emit warning ... The bug reports regarding this issue are http://d.puremagic.com/issues/show_bug.cgi?id=5056 http://d.puremagic.com/issues/show_bug.cgi?id=3393
Jun 05 2013
prev sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 05.06.2013 16:08, schrieb Regan Heath:
 On Wed, 05 Jun 2013 13:53:58 +0100, Michal Minich
 <michal.minich gmail.com> wrote:

 On Monday, 3 June 2013 at 17:18:55 UTC, Andrei Alexandrescu wrote:
   override is not comparable
 because it improves code correctness and maintainability, for which
 there is ample prior evidence. It's also a matter for which, unlike
 virtual/final, there is no reasonable recourse.
Virtual by default makes it simpler to call method on object that is not initialized yet (constructor not called yet). This situation is possible regardless if virtual is default or not (it can just happen more easily).
Yeah, it happened to me in C++ .. same with virtuals in the destructor. Lesson learned first time tho :p
Me as well, I was used to being able to call virtual methods on the constructor in Object Pascal and it took me a while to find out why my C++ program was crashing, back in the day.
Jun 05 2013
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, June 03, 2013 19:06:53 David Nadlinger wrote:
 On Monday, 3 June 2013 at 16:25:24 UTC, Manu wrote:
 You won't break every single method, they already went through
 that recently when override was made a requirement. [=E2=80=A6] A m=
uch
 smaller number than the breakage
 which was gladly accepted recently. [=E2=80=A6] how did the overrid=
e
 change ever get accepted [=E2=80=A6]
=20 It appears as if either you have a interesting definition of "recently", or you are deliberately misleading people by bringing up that point over and over again. =20 According to http://dlang.org/changelog.html, omitting "override" produced a warning since D 2.004, which was released back in September 2007! Granted, it was only actually turned from a deprecation warning into an actual deprecation in 2.061 (if my memory serves me right), but it's mostly a flaw in the handling of that particular deprecation that it stayed at the first level for so long. The actual language change was made =E2=80=93 and user-visible =E2=80=93 almost six (!) years ago, which is a lot on th=
e D
 time scale.
Except that because it was only treated as a warning until very recentl= y and=20 because dmd has the abnormal behavior of not printing warnings by defau= lt,=20 plenty of folks weren't using override until quite recently.
 You are also ignoring the fact that in contrast to requiring
 "override", there is no clean deprecation path for your proposal,
 at least as far as I can see: Omitting the keyword started out as
 a warning, and IIRC still is allowed when you enable deprecated
 features via the compiler switch. How would a similar process
 look for virtual-by-default? As far as am isolated module with
 only a base class is concerned, this is not question of valid vs.
 invalid code, but a silent change in language semantics.
The change in semantics isn't very silent at all thanks to override. It= =20 actually breaks quite loudly. Historically, the steps to do it would ha= ve been=20 something like 1. Print a message or warning that a function which is being overridden= isn't=20 marked as virtual. 2. Deprecate overriding a function which is not marked as virtual. 3. Make it an error to override a function which isn't marked as virtua= l and=20 then actually make it so that functions which are not marked as virtual= are=20 non-virtual. But now that deprecations don't stop compilation by default anymore, we= might=20 even be able to skip the first step and do 1. Deprecate overriding a function which is not marked as virtual. 2. Make it an error to override a function which isn't marked as virtua= l and=20 then actually make it so that functions which are not marked as virtual= are=20 non-virtual. And the only functions which have to be changed are those which are act= ually=20 being overridden, and the compiler will already give an error when you=20= override a function without marking the overriding function with overri= de, so=20 programmers are already forced to mark all of the functions that care w= hether=20 the base class function is virtual or not. So, you won't be getting sil= ent=20 breakage. It's more of a problem when a library writer puts an API in t= heir=20 library and does not actually override it themselves (even if it's inte= nded to=20 be overridden), as it's the users of that library which will see the=20= deprecation warnings and eventual breakage, but the library writer shou= ld be=20 testing their own stuff, and the users of the library can yell at them = about=20 it. So, the breakage is minimal and noisy, but it _is_ still breakage. If w= e were=20 starting from scratch, I really don't think that there would be much ex= cuse=20 for not making functions non-virtual by default given that we require=20= overriding functions to be marked with override, but it is more of an o= pen=20 question when we're this far into the game. I still think that it's wor= th=20 making the change though. - Jonathan M Davis
Jun 03 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/13 8:18 PM, Jonathan M Davis wrote:
 So, the breakage is minimal and noisy, but it _is_ still breakage. If we were
 starting from scratch, I really don't think that there would be much excuse
 for not making functions non-virtual by default given that we require
 overriding functions to be marked with override, but it is more of an open
 question when we're this far into the game. I still think that it's worth
 making the change though.
Absolutely it is breakage. The entire discussion is aimless though because it compares poorly costs and benefits of very different design choices, and as you mention at completely different historical times. It's useless to focus on the breakage override has caused. Yes it did cause breakage. There is no conclusion to draw from that without considering the considerably complex dynamics surrounding the whole matter (experience, benefits, number of users affected positively and negatively). To use that breakage as an argument linked to absorbing breakage caused by switching to final-by-default does not make sense. I'll try to abstain replying to this particular point in the future, it just instantly lowers the quality of the dialog. Andrei
Jun 03 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, June 03, 2013 22:25:13 Andrei Alexandrescu wrote:
 It's useless to focus on the breakage override has caused. Yes it did
 cause breakage. There is no conclusion to draw from that without
 considering the considerably complex dynamics surrounding the whole
 matter (experience, benefits, number of users affected positively and
 negatively).
 
 To use that breakage as an argument linked to absorbing breakage caused
 by switching to final-by-default does not make sense. I'll try to
 abstain replying to this particular point in the future, it just
 instantly lowers the quality of the dialog.
The comparison is made because we're talking about a related change, and the actual breakage caused by override was fairly recent, so while the decision to cause that breakage was made quite some time ago, we were still willing to cause that breakage fairly recently, so the implication then is that it would be acceptable to do something related which causes less breakage. Now, that being said, I do think that we need to look at this change in its own right, and it needs to justify itself, but clearly folks like Manu think that it does justify itself and feel that there's something off if we're willing to make the change with override and not this one, when they're related, and this one causes even less breakage. I do think that virtual-by-default was a mistake and would like to see it fixed, but I'm also not as passionate about it as Manu or Don. Manu in particular seems to be sick of having to fix performance bugs at Remedy Games caused by this issue and so would really like to see non-virtual be the default. The folks using D in companies in real-world code seem to think that the ROI on this change is well worth it. And a technical issue which affects us all is how this interacts with extern(C++). Daniel Murphy is having to improve extern(C++) in order to be able to port the dmd frontend to D (so that it can properly interact with the backends), and the fact that member functions are virtual by default definitely causes problems there. He would know the details about that better than I would, but IIRC, it had to do with the fact that we needed to be able to interface with non-virtual member C++ functions. So, depending on the details with that, that alone could make it worth switching to non-virtual by default, particularly when the breakage is actually quite loud and easy to fix. - Jonathan M Davis
Jun 03 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/13 10:51 PM, Jonathan M Davis wrote:
 On Monday, June 03, 2013 22:25:13 Andrei Alexandrescu wrote:
 It's useless to focus on the breakage override has caused. Yes it did
 cause breakage. There is no conclusion to draw from that without
 considering the considerably complex dynamics surrounding the whole
 matter (experience, benefits, number of users affected positively and
 negatively).

 To use that breakage as an argument linked to absorbing breakage caused
 by switching to final-by-default does not make sense. I'll try to
 abstain replying to this particular point in the future, it just
 instantly lowers the quality of the dialog.
The comparison is made because we're talking about a related change, and the actual breakage caused by override was fairly recent, so while the decision to cause that breakage was made quite some time ago, we were still willing to cause that breakage fairly recently, so the implication then is that it would be acceptable to do something related which causes less breakage.
This nice one-long-sentence paragraph does little in the way of helping because it just restates the same well-understood matter that I disagree with without adding information. My argument is that the relatedness of the change is tenuous, and explains why I think so. The paragraph above presupposes again relatedness, and proceeds with a tedious re-explanation of the consequences of that assumption. We don't make progress like this: Speaker A: "XYZ, therefore ABC". Speaker B: "I disagree with XYZ because TUV." Speaker A: "But since XYZ then ABC."
 Now, that being said, I do think that we need to look at this change in its
 own right, and it needs to justify itself, but clearly folks like Manu think
 that it does justify itself and feel that there's something off if we're
 willing to make the change with override and not this one, when they're
 related, and this one causes even less breakage.
The matters of override vs virtual/final are not related so analyzing the consequences of said relatedness is not very productive. That doesn't make one necessarily more important than the other, but it must be understood that they are distinct matters, and we can't compare one comma in one with one comma in the other. Requiring "override" in overriding methods: 1. Protects against categories of bugs that otherwise would be impossible to protect against: accidental overriding and accidental non-overriding. 2. Provides an important maintenance tool for code evolution, statically breaking code that would otherwise change semantics silently. It's important that without "override" there is virtually no protection against these issues. We're talking about an "all goodness" feature, and this kind of stuff is in very low supply in this world. Choosing "virtual" by default: 1. Fosters flexibility by allowing derived classes to override unannotated methods in base classes. 2. Is suboptimal in speed because users pay for the potential flexibility, even when that flexibility is not actually realized (barring a static analysis called class hierarchy analysis). 3. Ultimately lets the programmer choose the right design by using annotations appropriately. Choosing "final" by default: 1. Fosters speed by statically binding calls to unannotated methods. 2. Is suboptimal in flexibility because users pay for the speed with loss of flexibility, even when speed is not a concern but flexibility is. 3. Ultimately lets the programmer choose the right design by using annotations appropriately. The introduction of "override" allows a language to choose either final or virtual by default, without being exposed to potential bugs. This is pretty much the entire extent to which "override" is related to the choice of virtual vs. final by default. Today, D makes it remarkably easy to choose the right design without significant boilerplate, regardless of the default choice: - "struct" introduces a monomorphic type with a limited form of subtyping (via alias this) and no dynamic binding of methods. - "final class" introduces a leaf class that statically disallows inheritance and consequently forces static calls to all methods. (BTW I recall there were some unnecessary virtual calls for final classes, has that been fixed?) - "final { ... }" introduces a pseudo-scope in which all declared methods are final - "final:" introduces a pseudo-label after which all declared methods are final (Granted, there's an asymmetry - there's no "~final:" label to end final, which makes it marginally more tedious to arrange final and non-final methods in the class.) This leaves an arguably small subset of designs, scenarios, projects, and teams that would be affected by the choice of default. When that point has been made, it has been glibly neglected with an argument along the lines of "yeah, well programmers will take the path of least resistance, not really think things through, come from C++ and assume the wrong default", which may as well be true for a subset of situations, but further reduces the persona typically affected by the choice of default. I personally have a hard time picturing someone who is at the same time obsessed with performance, disinclined to assess it, unwilling to learn how to improve it, and incapable of using simple tools to control it. Yet this persona is put at the center of the argument that we must change the default right now seeing as it is a huge problem. To top it off, the entire fallacy about override causing more breakage is brought about. Yes, smoking kills, and the fact that cars kill more people doesn't quite have a bearing on that.
 I do think that virtual-by-default was a mistake and would like to see it
 fixed, but I'm also not as passionate about it as Manu or Don.
Choosing virtual (or not) by default may be dubbed a mistake only in a flexibility and then do their best to obtain performance. In the context of D in particular, there are arguments for the default going either way. If I were designing D from scratch it may even make sense to e.g. force a choice while offering no default whatsoever. But bottom line is, choosing the default is not a big deal for D because this wonderful language offers so many great building blocks for any design one might imagine.
 Manu in
 particular seems to be sick of having to fix performance bugs at Remedy Games
 caused by this issue and so would really like to see non-virtual be the
 default. The folks using D in companies in real-world code seem to think that
 the ROI on this change is well worth it.
I'm wary/weary of polls with a small number of participants. Let's also not forget that these people do use D successfully, and if sticking "final" here and there is the most difficult endeavor that has helped performance of their programs, I'd say both them and D are in great shape.
 And a technical issue which affects us all is how this interacts with
 extern(C++). Daniel Murphy is having to improve extern(C++) in order to be
 able to port the dmd frontend to D (so that it can properly interact with the
 backends), and the fact that member functions are virtual by default definitely
 causes problems there. He would know the details about that better than I
 would, but IIRC, it had to do with the fact that we needed to be able to
 interface with non-virtual member C++ functions. So, depending on the details
 with that, that alone could make it worth switching to non-virtual by default,
 particularly when the breakage is actually quite loud and easy to fix.
I don't know much about that matter, as I don't know about the argument related to mock injection and such, so I won't comment on this. Finally, I'll note that I'd started a reply to this remark by Manu (who in turn replied to David):
 Is there a reason this change offends you enough to call me names? Or
 can you at least tell how I'm being narrow-minded?
I deleted that reply, but let me say this. In a good argument: 1. Participants have opinions and beliefs derived from evidence they have accumulated. 2. The very ongoing discourse offers additional evidence to all participants by means of exchange of information. This is to be expected because participants have varied backgrounds and often it's possible to assess how competent they are. 3. The merits of various arguments are discussed, appreciated, and integrated within the opinions and beliefs of the participants. 4. A conclusion is reached in light of everything discussed and everybody is richer that way. In a not-so good argument: 1. Participants start each from an immutable belief. 2. Their preoccupation is to amass, bend, or fabricate any argument that would make that belief prevail, and to neglect any argument to the contrary. 3. The entire discussion has a foregone conclusion for everyone involved, i.e. nobody changes opinions and nobody is gained. The attitude "I know what's right, the only problem is to make you understand" doesn't serve anyone, because it locks "me" in a trench with no horizon and no mobility, and elicits an emotional response in "you". Here we don't want to keep "virtual" default and we don't want to make "final" default. We want to do what's right. So the discussion should progress toward finding what's right, not starting from knowing what's right and working arguments from there. Andrei
Jun 03 2013
parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Tuesday, 4 June 2013 at 04:07:10 UTC, Andrei Alexandrescu 
wrote:
 Choosing virtual (or not) by default may be dubbed a mistake 

 languages aim for flexibility and then do their best to obtain 
 performance. In the context of D in particular, there are 
 arguments for the default going either way. If I were designing 
 D from scratch it may even make sense to e.g. force a choice 
 while offering no default whatsoever.
issues but because of issues with incorrect code. While work around that using their JIT compiler, just like HotSpot does for Java. The real issue is that overriding methods that the author did not think about being overridden is *wrong*. It leads to incorrect code. Making methods virtual relies on some implementation details being fixed, such as whether to call fields or properties. Many people who wrap fields in properties use the fields still in various places in the class. Now someone overrides the property, and finds that it either makes no change or in one or it works everywhere except in one or two fields where the author refers to the field. By forcing the author to specifically make things virtual you force them to recognize that they should or shouldn't be using the property vs the field. There are many other examples for similar issues, but I feel properties are one of the biggest issues with virtual-by-default, both from a correctness standpoint and a performance standpoint. Having property imply final seems rather hackish though perhaps. Anders Hejlsberg talks about why they decided to use final by the Non-Virtual is the Default section. They do this *because* they saw the drawbacks of Java's virtual by default and were able to learn from it. Switching to final-by-default doesn't have to be an immediate breaking change. If we had a virtual keyword it could go through a deprecation process where overriding a method not declared as virtual results in a warning, exactly like the override enforcement went through.
Jun 05 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/5/13 4:01 PM, Kapps wrote:
 Anders Hejlsberg talks about why they decided to use final by default in

 is the Default section. They do this *because* they saw the drawbacks of
 Java's virtual by default and were able to learn from it.
This is a solid piece of evidence. Andrei
Jun 05 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 1:13 PM, Andrei Alexandrescu wrote:
 On 6/5/13 4:01 PM, Kapps wrote:
 Anders Hejlsberg talks about why they decided to use final by default in

 is the Default section. They do this *because* they saw the drawbacks of
 Java's virtual by default and were able to learn from it.
This is a solid piece of evidence.
Yup.
Jun 05 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 2:55 PM, Walter Bright wrote:
 On 6/5/2013 1:13 PM, Andrei Alexandrescu wrote:
 On 6/5/13 4:01 PM, Kapps wrote:
 Anders Hejlsberg talks about why they decided to use final by default in

 is the Default section. They do this *because* they saw the drawbacks of
 Java's virtual by default and were able to learn from it.
This is a solid piece of evidence.
Yup.
We can do an upgrade path as follows: 1. Introduce 'virtual' storage class. 'virtual' not only means a method is virtual, but it is an *introducing* virtual, i.e. it starts a new vtbl[] entry even if there's a virtual of the same name in the base classes. This means that functions marked 'virtual' do not override functions marked 'virtual'. 2. Issue a warning if a function overrides a function that is not marked 'virtual'. 3. Deprecate (2). 4. Error (2), and make non-virtual the default.
Jun 05 2013
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 June 2013 at 22:03:05 UTC, Walter Bright wrote:
 1. Introduce 'virtual' storage class. 'virtual' not only means 
 a method is virtual, but it is an *introducing* virtual, i.e. 
 it starts a new vtbl[] entry even if there's a virtual of the 
 same name in the base classes. This means that functions marked 
 'virtual' do not override functions marked 'virtual'.
Your upgrade path sounds generally good to me, I can live with that. class A { virtual void foo(); } class B : A { virtual void foo(); } Error, yes? It should be "override void foo();" or "override final void foo();". (override and virtual together would always be an error, correct?) Whereas: class A { virtual void foo(); } class B : A { virtual void foo(int); } is OK because foo(int) is a new overload, right? If I have these right, then yeah, I think your plan is good and should happen.
Jun 05 2013
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 05 Jun 2013 18:32:58 -0400, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 On Wednesday, 5 June 2013 at 22:03:05 UTC, Walter Bright wrote:
 1. Introduce 'virtual' storage class. 'virtual' not only means a method  
 is virtual, but it is an *introducing* virtual, i.e. it starts a new  
 vtbl[] entry even if there's a virtual of the same name in the base  
 classes. This means that functions marked 'virtual' do not override  
 functions marked 'virtual'.
Your upgrade path sounds generally good to me, I can live with that. class A { virtual void foo(); } class B : A { virtual void foo(); } Error, yes? It should be "override void foo();" or "override final void foo();". (override and virtual together would always be an error, correct?) Whereas: class A { virtual void foo(); } class B : A { virtual void foo(int); } is OK because foo(int) is a new overload, right?
No, I think it introduces a new foo. Calling A.foo does not call B.foo. In other words, it hides the original implementation, there are two vtable entries for foo. it seems Walter is trying to specify that. The idea is that B probably defined foo before A did, and A adding foo should not break B, B didn't even know about A's foo. -Steve
Jun 05 2013
next sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 06.06.2013 00:37, schrieb Steven Schveighoffer:
 On Wed, 05 Jun 2013 18:32:58 -0400, Adam D. Ruppe
 <destructionator gmail.com> wrote:

 On Wednesday, 5 June 2013 at 22:03:05 UTC, Walter Bright wrote:
 1. Introduce 'virtual' storage class. 'virtual' not only means a
 method is virtual, but it is an *introducing* virtual, i.e. it starts
 a new vtbl[] entry even if there's a virtual of the same name in the
 base classes. This means that functions marked 'virtual' do not
 override functions marked 'virtual'.
Your upgrade path sounds generally good to me, I can live with that. class A { virtual void foo(); } class B : A { virtual void foo(); } Error, yes? It should be "override void foo();" or "override final void foo();". (override and virtual together would always be an error, correct?) Whereas: class A { virtual void foo(); } class B : A { virtual void foo(int); } is OK because foo(int) is a new overload, right?
No, I think it introduces a new foo. Calling A.foo does not call B.foo. In other words, it hides the original implementation, there are two vtable entries for foo. and it seems Walter is trying to specify that. The idea is that B probably defined foo before A did, and A adding foo should not break B, B didn't even know about A's foo. -Steve
http://msdn.microsoft.com/en-us/library/6fawty39.aspx
Jun 05 2013
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 June 2013 at 22:53:36 UTC, Paulo Pinto wrote:


 http://msdn.microsoft.com/en-us/library/6fawty39.aspx
me. This seems ok too, and actually now that I think about it, the override already takes care of my biggest concern, that we'd accidentally hide something. Given: class A { void foo(); } class B:A { override void foo(); } class C:B { override void foo(); } The error can pretty easily be "B.foo overrides non-virtual function A.foo" as well as "C.foo overrides non-virtual function A.foo" - the warning on C doesn't need to mention B, since it is already marked override. Thus newbies like me won't incorrectly put virtual on B and accidentally hide A. So this deprecation will easily point us to put virtual in all the right places and none of the wrong places. I like it.
Jun 05 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 3:37 PM, Steven Schveighoffer wrote:
 No, I think it introduces a new foo.  Calling A.foo does not call B.foo.  In
 other words, it hides the original implementation, there are two vtable entries
 for foo.


 seems Walter is trying to specify that.  The idea is that B probably defined
foo
 before A did, and A adding foo should not break B, B didn't even know about A's
 foo.
That's right.
Jun 05 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 05 Jun 2013 18:56:28 -0400, Walter Bright  =

<newshound2 digitalmars.com> wrote:

 On 6/5/2013 3:37 PM, Steven Schveighoffer wrote:
 No, I think it introduces a new foo.  Calling A.foo does not call  =
 B.foo.  In
 other words, it hides the original implementation, there are two vtab=
le =
 entries
 for foo.


=
 and it
 seems Walter is trying to specify that.  The idea is that B probably =
=
 defined foo
 before A did, and A adding foo should not break B, B didn't even know=
=
 about A's
 foo.
That's right.
Prompted by Paulo's reference, I see there is another twist not clear fr= om = the interview article. = judge which parts to have. k = is very informative (bear with the unicode errors, this is copy pasted = from this document: = http://www.ecma-international.org/publications/files/ECMA-ST-WITHDRAWN/E= CMA-334,%202nd%20edition,%20December%202002.pdf = ): 8.13 Versioning Versioning is the process of evolving a component over time in a = compatible manner. A new version of a component is source compatible wit= h = a previous version if code that depends on the previous version can, whe= n = recompiled, work with the new version. In contrast, a new version of a = component is binary compatible if an application that depended on the ol= d = version can, without recompilation, work with the new version. Most languages do not support binary compatibility at all, and many do = little to facilitate source compatibility. In fact, some languages conta= in = flaws that make it impossible, in general, to evolve a class over time = without breaking at least some client code. As an example, consider the situation of a base class author who ships a= = class named Base. In the first version, Base contains no method F. A = component named Derived derives from Base, and introduces an F. This = Derived class, along with the class Base on which it depends, is release= d = to customers, who deploy to numerous clients and servers. // Author A namespace A { public class Base // version 1 { } } // Author B namespace B { class Derived: A.Base { public virtual void F() { System.Console.WriteLine("Derived.F"); } } } So far, so good, but now the versioning trouble begins. The author of Ba= se = produces a new version, giving it its own method F. // Author A namespace A { public class Base // version 2 { public virtual void F() // added in version 2 { System.Console.WriteLine("Base.F"); } } } This new version of Base should be both source and binary compatible wit= h = the initial version. (If it wereni=CC=81t possible to simply add a metho= d then = a base class could never evolve.) Unfortunately, the new F in Base makes= = the meaning of Derivedi=CC=81s F unclear. Did Derived mean to override B= asei=CC=81s = F? This seems unlikely, since when Derived was compiled, Base did not ev= en = have an F! Further, if Derivedi=CC=81s F does override Basei=CC=81s F, t= hen it must = adhere to the contract specified by Baseo=CC=81a contract that was unspe= cified = when Derived was written. In some cases, this is impossible. For example= , = Basei=CC=81s F might require that overrides of it always call the base. = = Derivedi=CC=81s F could not possibly adhere to such a contract. their intent clearly. In the original code example, the code was clear, = = since Base did not even have an F. Clearly, Derivedi=CC=81s F is intende= d as a = new method rather than an override of a base method, since no base metho= d = named F exists. If Base adds an F and ships a new version, then the intent of a binary = version of Derived is still clearo=CC=81 Derivedi=CC=81s F is semantical= ly = unrelated, and should not be treated as an override. However, when Derived is recompiled, the meaning is unclearo=CC=81the au= thor of = Derived may intend its F to override Basei=CC=81s F, or to hide it. Sinc= e the = intent is unclear, the compiler produces a warning, and by default makes= = Derivedi=CC=81s F hide Basei=CC=81s F. This course of action duplicates = the = semantics for the case in which Derived is not recompiled. The warning = that is generated alerts Derivedi=CC=81s author to the presence of the F method in Base. If Derived's F is semantically unrelated to Basei=CC=81s F, then Derived= i=CC=81s = author can express this intento=CC=81and, in effect, turn off the warningo=CC=81by using the new keyword in the = declaration of F. // Author A namespace A { public class Base // version 2 { public virtual void F() // added in version 2 { System.Console.WriteLine("Base.F"); } } } // Author B namespace B { class Derived: A.Base // version 2a: new { new public virtual void F() { System.Console.WriteLine("Derived.F"); } } } On the other hand, Derivedi=CC=81s author might investigate further, and= decide = that Derivedi=CC=81s F should override Basei=CC=81s F. This intent can b= e specified = by using the override keyword, as shown below. // Author A namespace A { public class Base // version 2 { public virtual void F() // added in version 2 { System.Console.WriteLine("Base.F"); } } } // Author B namespace B { class Derived: A.Base // version 2b: override { public override virtual void F() { base.F(); System.Console.WriteLine("Derived.F"); } } } The author of Derived has one other option, and that is to change the na= me = of F, thus completely avoiding the name collision. Although this change = = would break source and binary compatibility for Derived, the importance = of = this compatibility varies depending on the scenario. If Derived is not = exposed to other programs, then changing the name of F is likely a good = = idea, as it would improve the readability of the programo=CC=81there wou= ld no = longer be any confusion about the meaning of F.
Jun 05 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 4:29 PM, Steven Schveighoffer wrote:
 On Wed, 05 Jun 2013 18:56:28 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 6/5/2013 3:37 PM, Steven Schveighoffer wrote:
 No, I think it introduces a new foo.  Calling A.foo does not call B.foo.  In
 other words, it hides the original implementation, there are two vtable entries
 for foo.


 seems Walter is trying to specify that.  The idea is that B probably defined
foo
 before A did, and A adding foo should not break B, B didn't even know about A's
 foo.
That's right.
Prompted by Paulo's reference, I see there is another twist not clear from the interview article. which parts to have.
I think we accomplish this in a simpler way: 1. 'virtual' means a method is an "introducing" one. 2. 'override' means a method overrides a base virtual function with a final function. 3. 'override virtual' means override with a non-final function. 4. none means final and non-overriding.
Jun 05 2013
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 6 June 2013 at 00:49:55 UTC, Walter Bright wrote:
 4. none means final and non-overriding.
BTW be sure not to remove the final keyword, since having it to undo a "virtual:" will still be useful to at least of us.
Jun 05 2013
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, June 05, 2013 17:49:17 Walter Bright wrote:
 I think we accomplish this in a simpler way:
 
 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means a method overrides a base virtual function with a final
 function.
 3. 'override virtual' means override with a non-final function.
 4. none means final and non-overriding.
I would have expected something more like 1. 'virtual' means a method is an "introducing" one. 2. 'override' means override with a non-final function. 3. 'final override' means a method overrides a base virtual function with a final function. 4. 'final' by itself both mean final and non-overriding. I think that it would be confusing for final to be implied if virtual is not used with override, and I'd argue that virtual should only go on the "introducing" one. Doing so will mean less anotation, will be less confusing IMHO, and will break less code. If override by itself means override non-final, then every case of override that we currently have is valid, and all that needs to be changed is that introducing functions will need to be marked as virtual. Then the only real changes are 1. Functions not marked with virtual or override are final. 2. Introducing functions must be marked with virtual. What you're suggesting would be a larger change, and I don't see how it would be better. - Jonathan M Davis
Jun 05 2013
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-06-06 01:14:08 +0000, "Jonathan M Davis" <jmdavisProg gmx.com> said:

 On Wednesday, June 05, 2013 17:49:17 Walter Bright wrote:
 I think we accomplish this in a simpler way:
 
 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means a method overrides a base virtual function with a final
 function.
 3. 'override virtual' means override with a non-final function.
 4. none means final and non-overriding.
I would have expected something more like 1. 'virtual' means a method is an "introducing" one. 2. 'override' means override with a non-final function. 3. 'final override' means a method overrides a base virtual function with a final function. 4. 'final' by itself both mean final and non-overriding. I think that it would be confusing for final to be implied if virtual is not used with override, and I'd argue that virtual should only go on the "introducing" one.
I concur. I don't think "override" and "virtual" should be allowed together on the same function. Here's a few reasons. Let "virtual" mean "create a new slot in the vtable". With that in mind, "virtual override" makes no sense: you can't make a new vtable slot and override it in the same class. If the function is already virtual, there's much less to gain by making it automatically final at the first override. It's very likely that it can be overridden again safely, and it probably can't be optimized into calling it statically most of the time since otherwise it wouldn't have been made virtual in the first place. There's also the consideration that adding/removing/reordering virtual functions change the ABI, while doing the same for overridden function does not. By only using "virtual" when allocating vtable slots, you're making it easier to recognize changes which are likely to break already compiled code. P.S.: while implementing this change, please make sure private and package functions can be virtual. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca/
Jun 05 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, June 05, 2013 22:50:14 Michel Fortin wrote:
 P.S.: while implementing this change, please make sure private and
 package functions can be virtual.
I'd agree with package on that, but I don't think that private should be virtualizable, because that conflicts with actually hiding non-accessible functions, which is what a number of us are pushing for. Otherwise, private implementation details of classes and modules risk causing code breakage (e.g. the fact that overload resolution occurs before checking the access level is atrocious). - Jonathan M Davis
Jun 06 2013
parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.894.1370540538.13711.digitalmars-d puremagic.com...
 On Wednesday, June 05, 2013 22:50:14 Michel Fortin wrote:
 P.S.: while implementing this change, please make sure private and
 package functions can be virtual.
I'd agree with package on that, but I don't think that private should be virtualizable, because that conflicts with actually hiding non-accessible functions, which is what a number of us are pushing for. Otherwise, private implementation details of classes and modules risk causing code breakage (e.g. the fact that overload resolution occurs before checking the access level is atrocious). - Jonathan M Davis
Private functions can be overridden only within the same module. This does not conflict with hiding private members from _outside_ the module. You will always be able to _see_ private members from inside the same module, virtual or not.
Jun 11 2013
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, June 12, 2013 13:26:44 Daniel Murphy wrote:
 "Jonathan M Davis" <jmdavisProg gmx.com> wrote in message
 news:mailman.894.1370540538.13711.digitalmars-d puremagic.com...
 
 On Wednesday, June 05, 2013 22:50:14 Michel Fortin wrote:
 P.S.: while implementing this change, please make sure private and
 package functions can be virtual.
I'd agree with package on that, but I don't think that private should be virtualizable, because that conflicts with actually hiding non-accessible functions, which is what a number of us are pushing for. Otherwise, private implementation details of classes and modules risk causing code breakage (e.g. the fact that overload resolution occurs before checking the access level is atrocious). - Jonathan M Davis
Private functions can be overridden only within the same module. This does not conflict with hiding private members from _outside_ the module. You will always be able to _see_ private members from inside the same module, virtual or not.
Good point, but I fully expect that many of the folks who want to do NVI will want to use private functions for that as they do in C++, and I think that that would be a mistake. private is accessible within a module, so it should be visible to everything in that module, but IMHO, it shouldn't be visible or overridable outside of the module, as nothing outside of the module would have access to it, and it would be too detrimental with regards to name clashes to permit it. - Jonathan M Davis
Jun 11 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 12 June 2013 at 03:26:43 UTC, Daniel Murphy wrote:
 "Jonathan M Davis" <jmdavisProg gmx.com> wrote in message
 news:mailman.894.1370540538.13711.digitalmars-d puremagic.com...
 On Wednesday, June 05, 2013 22:50:14 Michel Fortin wrote:
 P.S.: while implementing this change, please make sure 
 private and
 package functions can be virtual.
I'd agree with package on that, but I don't think that private should be virtualizable, because that conflicts with actually hiding non-accessible functions, which is what a number of us are pushing for. Otherwise, private implementation details of classes and modules risk causing code breakage (e.g. the fact that overload resolution occurs before checking the access level is atrocious). - Jonathan M Davis
Private functions can be overridden only within the same module. This does not conflict with hiding private members from _outside_ the module. You will always be able to _see_ private members from inside the same module, virtual or not.
Which also make them finalizable automagically.
Jun 11 2013
prev sibling next sibling parent reply "Rob T" <alanb ucora.com> writes:
On Thursday, 6 June 2013 at 01:14:23 UTC, Jonathan M Davis wrote:
 I would have expected something more like

 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means override with a non-final function.
 3. 'final override' means a method overrides a base virtual 
 function with a
 final function.
 4. 'final' by itself both mean final and non-overriding.
[...] Yes I agree that's much more intuitive. Also having ability for methods to individually opt out of a virtual: or final: block will be nice to have. eg virtual: void a(); final void b(); ... This should satisfy everyone since you'll easily be able to make a class virtual by default or final by default (more or less). --rt
Jun 05 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 6 June 2013 at 05:31:21 UTC, Rob T wrote:
 Yes I agree that's much more intuitive.

 Also having ability for methods to individually opt out of a 
 virtual: or final: block will be nice to have.

 eg

 virtual:

    void a();
    final void b();

 ...

 This should satisfy everyone since you'll easily be able to 
 make a class virtual by default or final by default (more or 
 less).
The other way around break less code and achieve the same result. discussion that much. Only 3 argument are provided, one that have been made obsolete by compiler technology, one that have been could be fixed by providing tooling that patch the bytecode, which isn't an option in D) and one that do not apply to D.
Jun 05 2013
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 05 Jun 2013 21:14:08 -0400, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:

 On Wednesday, June 05, 2013 17:49:17 Walter Bright wrote:
 I think we accomplish this in a simpler way:

 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means a method overrides a base virtual function with a  
 final
 function.
 3. 'override virtual' means override with a non-final function.
 4. none means final and non-overriding.
I would have expected something more like 1. 'virtual' means a method is an "introducing" one. 2. 'override' means override with a non-final function. 3. 'final override' means a method overrides a base virtual function with a final function. 4. 'final' by itself both mean final and non-overriding.
I agree, I think it can be expressed by answering two questions: 1. Do you want to participate in a base class' virtual call 2. Do you want to allow derived classes to participate in the virtual call. If you answer yes to 1, add override, otherwise (or if there is no base an existing base function) If you answer yes to 2, add virtual, otherwise, add final. But there are a couple of defaults I think are important to establish: - With no specific storage classes, the default is non-overriding, final. - When override is specified, the default for allowing participation up the chain switches to virtual. So explicitly stating virtual is OK but redundant. This is where Walter's idea differs. And I think it makes sense, when a base class has a virtual call, the overwhelming default is to continue the virtual chain. So when answering the original questions [optional]: 1 = yes, 2 = yes, use override [virtual] 1 = no*, 2 = yes, use virtual 1 = yes 2 = no, use override final 1 = no*, 2 = no, use [final] * note: if no base class virtual function exists, then the answer to question 1 is always no, and override is an error. The one deviation from the current "require override" behavior of the compiler exposes the following sequence: 1. Base class does not define foo, derived class defines foo, virtual or not. 2. Base class adds foo as virtual Currently, this will cause an error in the compiler. With this new scheme, the compiler silently accepts this as two unrelated functions, because override is not used. Arguably this is the correct choice, it is compiling, but it does give a warning. Since we would have no keyword to indicate "non-overriding", a warning is not possible, since there's no explicit way to say "I'm not overriding". I would argue that putting 'new' there doesn't add much, you are unlikely to change your method's semantics to match that of the base class' new the author of the derived class who is affected, it's the USER of the derived class. The confusion is for usage, not semantics, and that can be a problem. Admittedly, the case above should be rather rare. Perhaps the compiler can have a switch which identifies all the places where functions are masked? I would like to see the switch be configurable as to which packages to do this for. -Steve
Jun 06 2013
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/06/2013 05:00 PM, Steven Schveighoffer wrote:
 On Wed, 05 Jun 2013 21:14:08 -0400, Jonathan M Davis <jmdavisProg gmx.com>
wrote:
 I would have expected something more like

 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means override with a non-final function.
 3. 'final override' means a method overrides a base virtual function with a
 final function.
 4. 'final' by itself both mean final and non-overriding.
I agree, I think it can be expressed by answering two questions: 1. Do you want to participate in a base class' virtual call 2. Do you want to allow derived classes to participate in the virtual call. If you answer yes to 1, add override, otherwise (or if there is no base method), base function) If you answer yes to 2, add virtual, otherwise, add final.
I see a potential problem with allowing 'final' on its own to mean 'final and non-overriding', which is that if you _mean_ to override a function in the base class, but put simply 'final' and not 'override', it will still be accepted by the compiler -- and it may not be obvious that the override is not taking place. So, I think 'new' could have a place here after all.
Jun 06 2013
next sibling parent reply "Rob T" <alanb ucora.com> writes:
On Thursday, 6 June 2013 at 15:40:26 UTC, Joseph Rushton Wakeling 
wrote:

 I see a potential problem with allowing 'final' on its own to 
 mean 'final and
 non-overriding', which is that if you _mean_ to override a 
 function in the base
 class, but put simply 'final' and not 'override', it will still 
 be accepted by
 the compiler -- and it may not be obvious that the override is 
 not taking place.

 So, I think 'new' could have a place here after all.
My understanding is that final on it's own would be an error if the same named function was virtual in the base class, otherwise you would have to specify "final override". If that understanding is not correct, then I agree that would be a source of hidden errors. --rt
Jun 06 2013
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/06/2013 07:10 PM, Rob T wrote:
 My understanding is that final on it's own would be an error if the same named
 function was virtual in the base class, otherwise you would have to specify
 "final override". If that understanding is not correct, then I agree that would
 be a source of hidden errors.
The problem as I understand it is as follows. First, consider this scenario: there's a base class. You've created a subclass in it with a method, foo(), which is not in the base class. Now, suppose that the base class adds its own foo(), which is virtual. Now, your subclass will fail to compile [*] unless you tweak its own foo(), and apart from renaming foo() in your subclass, you have only two options: (i) to override the base class foo() or (ii) to "hide" or replace the base class foo(). So, the language should require you to be unambiguous and indicate explicitly which. Obviously "override" or "final override" will do for the first case. But what if you allow "final" to be the keyword for the second, "hide the base method" case? (This is my understanding of the proposal I replied to.) If you do, then consider the alternative scenario. You have a base class with a virtual method bar(). You _mean_ to override it with a final method, but accidentally type just "final" and not "final override". The compiler will accept this without warning, because it takes "final" to mean your explicit indication to "hide the base class method", and it may take you some time to realize that the base class method is incorrectly being hidden instead of overridden. So, that's why I suggest an alternative keyword to indicate the programmer's "new" seems as good as any. That's assuming that we want to allow base class methods to be hidden, but the http://forum.dlang.org/thread/yzsqwejxqlnzryhrkfuq forum.dlang.org?page=24#post-op.wx8biyx7eav7ka:40stevens-macbook-pro.local [* Technically it need not fail to compile but the compiler would have to assume a default behaviour and warn you in the absence of an explicit indicator of your method, and it warns you if you don't explicitly indicate "override" or "new".]
Jun 07 2013
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 06 Jun 2013 11:40:18 -0400, Joseph Rushton Wakeling  
<joseph.wakeling webdrake.net> wrote:

 On 06/06/2013 05:00 PM, Steven Schveighoffer wrote:
 On Wed, 05 Jun 2013 21:14:08 -0400, Jonathan M Davis  
 <jmdavisProg gmx.com> wrote:
 I would have expected something more like

 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means override with a non-final function.
 3. 'final override' means a method overrides a base virtual function  
 with a
 final function.
 4. 'final' by itself both mean final and non-overriding.
I agree, I think it can be expressed by answering two questions: 1. Do you want to participate in a base class' virtual call 2. Do you want to allow derived classes to participate in the virtual call. If you answer yes to 1, add override, otherwise (or if there is no base method), existing base function) If you answer yes to 2, add virtual, otherwise, add final.
I see a potential problem with allowing 'final' on its own to mean 'final and non-overriding', which is that if you _mean_ to override a function in the base class, but put simply 'final' and not 'override', it will still be accepted by the compiler -- and it may not be obvious that the override is not taking place.
Yes, this is a departure from the current code, which doesn't allow this. But it is a generalized problem, not specific to final. It's basically the lack of a storage class for meaning "no, I don't want to override the function". Note that final does NOT do this. In fact, we don't have a way to do this now. new could be that storage class, but I am cautious about it. new has a position close to that context, for custom allocators. Another storage class would complete the solution, and make intentions very easy to make obvious. At that point, we could have a warning/error for under-specification. And another note to make is that the case where you are 'hiding' a base final function isn't really a problem. It's more when you are hiding a virtual function, that this becomes an issue. More likely, when you are hiding a *new* virtual function, because it didn't exist when you wrote your code. -Steve
Jun 06 2013
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jun 05, 2013 at 09:14:08PM -0400, Jonathan M Davis wrote:
 On Wednesday, June 05, 2013 17:49:17 Walter Bright wrote:
 I think we accomplish this in a simpler way:
 
 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means a method overrides a base virtual function with a final
 function.
 3. 'override virtual' means override with a non-final function.
 4. none means final and non-overriding.
I would have expected something more like 1. 'virtual' means a method is an "introducing" one. 2. 'override' means override with a non-final function. 3. 'final override' means a method overrides a base virtual function with a final function. 4. 'final' by itself both mean final and non-overriding.
[...] Yeah, I think requiring 'override virtual' is a bit counterintuitive. Generally speaking, if I'm overriding a virtual method, then I'm also planning to expose the same API (i.e. a virtual method) to my derived classes, so it should also be virtual in turn. Overriding a virtual method with a final method is something exceptional (i.e. at the leaf nodes of the tree of overridden methods), so it should be the one that requires explicit specification. As for no specification, I thought the whole point was to have it default to final? So 'final' should be optional in this case. T -- It won't be covered in the book. The source code has to be useful for something, after all. -- Larry Wall
Jun 05 2013
parent "Rob T" <alanb ucora.com> writes:
On Thursday, 6 June 2013 at 05:19:32 UTC, H. S. Teoh wrote:
 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means override with a non-final function.
 3. 'final override' means a method overrides a base virtual 
 function with a final function.
 4. 'final' by itself both mean final and non-overriding.
 As for no specification, I thought the whole point was to have 
 it
 default to final? So 'final' should be optional in this case.
I see your point, but when I override a virtual it will usually always be with another virtual unless for some reason I wanted to explicitly state final. That's how it's always been done in other languages that I'm aware of, and switching to final by default on overrides will likely be a source of frustration. --rt
Jun 05 2013
prev sibling parent reply "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On 2013-06-06, 00:32, Adam D. Ruppe wrote:



 class A { virtual void foo(); }
 class B : A { virtual void foo(); }
class A { virtual void foo() { writeln("A.foo"); } } class B : A { virtual void foo() { writeln("B.foo"); } } void bar() { B b = new B(); A a = b; a.foo(); // Prints "A.foo" b.foo(); // Prints "B.foo" } -- Simen
Jun 05 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 5 June 2013 at 22:50:27 UTC, Simen Kjaeraas wrote:
 On 2013-06-06, 00:32, Adam D. Ruppe wrote:



 class A { virtual void foo(); }
 class B : A { virtual void foo(); }
class A { virtual void foo() { writeln("A.foo"); } } class B : A { virtual void foo() { writeln("B.foo"); } } void bar() { B b = new B(); A a = b; a.foo(); // Prints "A.foo" b.foo(); // Prints "B.foo" }
completely miss the point of OOP. On the same path, in the previously linked document : Every time you say virtual in an API, you are creating a call back hook. Which seems that OOP is limited to the observer pattern according to Anders Hejlsberg. VM, but won't be possible in D.
Jun 05 2013
prev sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Wednesday, 5 June 2013 at 22:03:05 UTC, Walter Bright wrote:

 We can do an upgrade path as follows:

 1. Introduce 'virtual' storage class. 'virtual' not only means 
 a method is virtual, but it is an *introducing* virtual, i.e. 
 it starts a new vtbl[] entry even if there's a virtual of the 
 same name in the base classes. This means that functions marked 
 'virtual' do not override functions marked 'virtual'.
methods (MS calls it "name hiding"). Not using 'new' when hiding a public member is a warning. References: http://msdn.microsoft.com/en-us/library/vstudio/435f1dw2.aspx http://msdn.microsoft.com/en-us/library/aa691135(v=vs.71).aspx http://msdn.microsoft.com/en-us/library/aa645767(v=vs.71).aspx Some points from limited personal experience: 1. Private final methods in D can hide base methods. That's *useful*. Don't try to change that. 2. Public final methods in D cannot be hidden. That's been a problem for me and other people. myself. I've never seen virtual methods being hidden and never needed that myself.
 2. Issue a warning if a function overrides a function that is 
 not marked 'virtual'.

 3. Deprecate (2).

 4. Error (2), and make non-virtual the default.
Jun 05 2013
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 05.06.2013 22:01, schrieb Kapps:
 On Tuesday, 4 June 2013 at 04:07:10 UTC, Andrei Alexandrescu wrote:
 Choosing virtual (or not) by default may be dubbed a mistake only in a

 flexibility and then do their best to obtain performance. In the
 context of D in particular, there are arguments for the default going
 either way. If I were designing D from scratch it may even make sense
 to e.g. force a choice while offering no default whatsoever.
because of issues with incorrect code. While performance is an issue JIT compiler, just like HotSpot does for Java. The real issue is that overriding methods that the author did not think about being overridden is *wrong*. It leads to incorrect code. Making methods virtual relies on some implementation details being fixed, such as whether to call fields or properties. Many people who wrap fields in properties use the fields still in various places in the class. Now someone overrides the property, and finds that it either makes no change or in one or it works everywhere except in one or two fields where the author refers to the field. By forcing the author to specifically make things virtual you force them to recognize that they should or shouldn't be using the property vs the field. There are many other examples for similar issues, but I feel properties are one of the biggest issues with virtual-by-default, both from a correctness standpoint and a performance standpoint. Having property imply final seems rather hackish though perhaps. Anders Hejlsberg talks about why they decided to use final by default in is the Default section. They do this *because* they saw the drawbacks of Java's virtual by default and were able to learn from it.
Oh, I though it was based on his experience with Object Pascal and Delphi, that also do final by default. Thanks for sharing. -- Paulo
Jun 05 2013
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Kapps:


 issues but because of [...]
One of the best posts of this thread, Kapps :-) design. Bye, bearophile
Jun 05 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
Thank you!
A hyperlink is always so much more substantial than a reasoned claim ;)
On 6 Jun 2013 06:05, "Kapps" <opantm2+spam gmail.com> wrote:

 On Tuesday, 4 June 2013 at 04:07:10 UTC, Andrei Alexandrescu wrote:

 Choosing virtual (or not) by default may be dubbed a mistake only in a

 flexibility and then do their best to obtain performance. In the context of
 D in particular, there are arguments for the default going either way. If I
 were designing D from scratch it may even make sense to e.g. force a choice
 while offering no default whatsoever.
because of issues with incorrect code. While performance is an issue (much compiler, just like HotSpot does for Java. The real issue is that overriding methods that the author did not think about being overridden is *wrong*. It leads to incorrect code. Making methods virtual relies on some implementation details being fixed, such as whether to call fields or properties. Many people who wrap fields in properties use the fields still in various places in the class. Now someone overrides the property, and finds that it either makes no change or in one or it works everywhere except in one or two fields where the author refers to the field. By forcing the author to specifically make things virtual you force them to recognize that they should or shouldn't be using the property vs the field. There are many other examples for similar issues, but I feel properties are one of the biggest issues with virtual-by-default, both from a correctness standpoint and a performance standpoint. Having property imply final seems rather hackish though perhaps. Anders Hejlsberg talks about why they decided to use final by default in See the Non-Virtual is the Default section. They do this *because* they saw the drawbacks of Java's virtual by default and were able to learn from it. Switching to final-by-default doesn't have to be an immediate breaking change. If we had a virtual keyword it could go through a deprecation process where overriding a method not declared as virtual results in a warning, exactly like the override enforcement went through.
Jun 05 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/5/13 7:39 PM, Manu wrote:
 Thank you!
 A hyperlink is always so much more substantial than a reasoned claim ;)
It's a hyperlink to an extensive argument. And assuming you refer to yours as the reasoned claim, I'd have to raise a finger to parts of that :o). That being said, a gentleman must be a gentleman. You destroyed, and I got destroyed. If Walter is on board with the change, I won't oppose. Congratulations. Andrei
Jun 05 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 5 June 2013 at 20:01:06 UTC, Kapps wrote:
 Anders Hejlsberg talks about why they decided to use final by 

 See the Non-Virtual is the Default section. They do this 
 *because* they saw the drawbacks of Java's virtual by default 
 and were able to learn from it.
The first point : Anders Hejlsberg: There are several reasons. One is performance. We can observe that as people write code in Java, they forget to mark their methods final. Therefore, those methods are virtual. Because they're virtual, they don't perform as well. There's just performance overhead associated with being a virtual method. That's one issue. It is blatantly false. Maybe it was true at the time, I don't know, but I find quite disturbing that the first argument is 100% moot.
Jun 05 2013
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 05 Jun 2013 20:49:00 -0400, deadalnix <deadalnix gmail.com> wrote:

 On Wednesday, 5 June 2013 at 20:01:06 UTC, Kapps wrote:
 Anders Hejlsberg talks about why they decided to use final by default  

 Non-Virtual is the Default section. They do this *because* they saw the  
 drawbacks of Java's virtual by default and were able to learn from it.
The first point : Anders Hejlsberg: There are several reasons. One is performance. We can observe that as people write code in Java, they forget to mark their methods final. Therefore, those methods are virtual. Because they're virtual, they don't perform as well. There's just performance overhead associated with being a virtual method. That's one issue. It is blatantly false. Maybe it was true at the time, I don't know, but I find quite disturbing that the first argument is 100% moot.
This was circa 2003. Look at the state of Java from then. And also consider that when the *decision* was made to make non-virtual the default, was considerably before then. -Steve
Jun 05 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 6 June 2013 at 01:00:36 UTC, Steven Schveighoffer 
wrote:
 This was circa 2003.  Look at the state of Java from then.  And 
 also consider that when the *decision* was made to make 
 non-virtual the default, was considerably before then.

 -Steve
This is why I wrote that this may have been true in the past. Nevertheless, it is completely false today. method, for several purposes like mock. We can't simple take this argument and don't look at it with the light of history. The history shows that out of 3 point, only one remains valid, and this is the one about properties. Ironically, this is the one that do not apply to D (in its current shape) as we don't have an proper property.
Jun 05 2013
next sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 6 June 2013 at 01:08:36 UTC, deadalnix wrote:
 On Thursday, 6 June 2013 at 01:00:36 UTC, Steven Schveighoffer 
 wrote:
 This was circa 2003.  Look at the state of Java from then.  
 And also consider that when the *decision* was made to make 
 non-virtual the default, was considerably before then.

 -Steve
This is why I wrote that this may have been true in the past. Nevertheless, it is completely false today. method, for several purposes like mock. We can't simple take this argument and don't look at it with the light of history. The history shows that out of 3 point, only one remains valid, and this is the one about properties. Ironically, this is the one that do not apply to D (in its current shape) as we don't have an proper property.
As I mentioned before, given that I speak more than really using D for anything, my opinion should not count. also ahead of time compilation. Virtual by default wins when you have a VM that can do devirtualization. When doing static compilation, even with LTO, you are limited in what you can do, specially if generating dynamic libraries. I complain a lot about limited choice in Java native compilers, but given the language semantics, a JVM with JIT is actually faster, hence why no one bothers except for a few niche cases, other being the price for such compilers. So a good question would be in the languages that have native compilers as canonical implementation and use virtual by default, how fast can method invocations be done. Ada, Eiffel, Dylan, Lisp, Go, ... -- Paulo
Jun 06 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 6 June 2013 at 07:43:30 UTC, Paulo Pinto wrote:
 As I mentioned before, given that I speak more than really 
 using D for anything, my opinion should not count.


 but also ahead of time compilation.

 Virtual by default wins when you have a VM that can do 
 devirtualization.
devirtualization is a link time problem, not a runtime one. We can do that in D (except when it come to shared objects) and I even proposed solutions to do so. revirtualization, however, can only be done with a JIT compiler.
 When doing static compilation, even with LTO, you are limited 
 in what you can do, specially if generating dynamic libraries.
Exact, but this is nothing specific to virtual. shared library come at a cost, as the call is opaque to the compiler. It has to assume the worse and must disable most optimizations anyway. You are trying to pose a bandage on a wooden leg.
Jun 06 2013
prev sibling next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Thursday, 6 June 2013 at 01:08:36 UTC, deadalnix wrote:
 This is why I wrote that this may have been true in the past. 
 Nevertheless, it is completely false today.
inline them there's still an overhead. This (2008) article goes into depth about how it handles it: www.codeproject.com/Articles/25801/JIT-Optimizations - Essentially uses frequency analysis to determine if the virtual method call is still going to call the same method as it would previously. Regardless, we can not perform such optimizations, so

 method, for several purposes like mock. We can't simple take 
 this argument and don't look at it with the light of history.
mocking frameworks can use to replace methods with their own implementations (ie: Microsoft's Moles Framework). Simply making a method virtual is not sufficient as static methods could then not be mocked. Besides that, making your classes / methods virtual for the sole purpose of mocking seems like a bad idea to me. Even if virtual was the default though, you still have to worry about final methods. Java for example uses frameworks such as Mockito to allow you to mock final methods (and static methods as well); the virtual by default doesn't change that.
Jun 06 2013
next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Thursday, 6 June 2013 at 15:06:38 UTC, Kapps wrote:
 Java for example uses frameworks such as Mockito to allow you 
 to mock final methods (and static methods as well); the virtual 
 by default doesn't change that.
Sorry, that should be PowerMock not Mockito.
Jun 06 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 6 June 2013 at 15:06:38 UTC, Kapps wrote:
 On Thursday, 6 June 2013 at 01:08:36 UTC, deadalnix wrote:
 This is why I wrote that this may have been true in the past. 
 Nevertheless, it is completely false today.
inline them there's still an overhead. This (2008) article goes into depth about how it handles it: www.codeproject.com/Articles/25801/JIT-Optimizations - Essentially uses frequency analysis to determine if the virtual method call is still going to call the same method as it would previously. Regardless, we can not perform such optimizations,
time, and we can do it at link time the same way, that is all that matter for this discussion. implementation detail.
Jun 06 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 06 Jun 2013 13:50:11 -0400, deadalnix <deadalnix gmail.com> wrote:

 On Thursday, 6 June 2013 at 15:06:38 UTC, Kapps wrote:
 On Thursday, 6 June 2013 at 01:08:36 UTC, deadalnix wrote:
 This is why I wrote that this may have been true in the past.  
 Nevertheless, it is completely false today.
them there's still an overhead. This (2008) article goes into depth about how it handles it: www.codeproject.com/Articles/25801/JIT-Optimizations - Essentially uses frequency analysis to determine if the virtual method call is still going to call the same method as it would previously. Regardless, we it does apply to D.
we can do it at link time the same way, that is all that matter for this discussion.
How do you finalize a method with the possibility that a dynamic library will come along and extend that type? Not a rhetorical question, I really there, since they are run on a VM. -Steve
Jun 06 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 6 June 2013 at 17:56:05 UTC, Steven Schveighoffer 
wrote:
 How do you finalize a method with the possibility that a 
 dynamic library will come along and extend that type?  Not a 
 rhetorical question, I really want to know if there is a way.  

 run on a VM.
We are turning around here. You can't. But it doesn't really matter as the call is already opaque and so it is slow. The impact isn't that big relatively (consider in the code above it was 5% on a statically compiled code). It is also possible to automatically generate code like : if (virtualMethod == givenMethod) { givenMethod(); } else { virtualMethod(); } It sound like it is completely stupid, but in fact, as the compiler know the call you'll do, it can run optimizations. But I want to repeat myself once again : trying to fix virtual call to shared object isn't gone change much in regard of performances.
Jun 06 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 11:11 AM, deadalnix wrote:
 It is also possible to automatically generate code like :
 if (virtualMethod == givenMethod) {
      givenMethod();
 } else {
      virtualMethod();
 }

 It sound like it is completely stupid, but in fact, as the compiler know the
 call you'll do, it can run optimizations.
You're right, that is a valid optimization for virtual methods. But, it is not as good as final, because the call to virtualMethod() negatively affects the code generation even if it is never called. (This is because of things like function calls destroy the scratch registers, meaning that variables can't easily be enregistered into them.)
Jun 06 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 11:27 AM, Walter Bright wrote:
 On 6/6/2013 11:11 AM, deadalnix wrote:
 It is also possible to automatically generate code like :
 if (virtualMethod == givenMethod) {
      givenMethod();
 } else {
      virtualMethod();
 }

 It sound like it is completely stupid, but in fact, as the compiler know the
 call you'll do, it can run optimizations.
You're right, that is a valid optimization for virtual methods.
(Also, this is often done as a profile guided optimization, with givenMethod being the most often case.)
Jun 06 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 6 June 2013 at 18:27:10 UTC, Walter Bright wrote:
 On 6/6/2013 11:11 AM, deadalnix wrote:
 It is also possible to automatically generate code like :
 if (virtualMethod == givenMethod) {
     givenMethod();
 } else {
     virtualMethod();
 }

 It sound like it is completely stupid, but in fact, as the 
 compiler know the
 call you'll do, it can run optimizations.
You're right, that is a valid optimization for virtual methods. But, it is not as good as final, because the call to virtualMethod() negatively affects the code generation even if it is never called. (This is because of things like function calls destroy the scratch registers, meaning that variables can't easily be enregistered into them.)
Yes, which happen anyway for shared objects !
Jun 06 2013
parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Friday, 7 June 2013 at 00:34:45 UTC, deadalnix wrote:
 Yes, which happen anyway for shared objects !
This. We can do whole-program-optimization to finalize non-shared-library functions - and for shared-library functions it doesn't make sense to finalize for performance - you're trading one kind of indirection function call for another. Can we all please address or recognize this?
Jun 06 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 5:43 PM, Jakob Ovrum wrote:
 On Friday, 7 June 2013 at 00:34:45 UTC, deadalnix wrote:
 Yes, which happen anyway for shared objects !
This. We can do whole-program-optimization to finalize non-shared-library functions - and for shared-library functions it doesn't make sense to finalize for performance - you're trading one kind of indirection function call for another. Can we all please address or recognize this?
Sure, but the trouble is determining which classes are going to be shared.
Jun 06 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 7 June 2013 at 01:22:25 UTC, Walter Bright wrote:
 On 6/6/2013 5:43 PM, Jakob Ovrum wrote:
 On Friday, 7 June 2013 at 00:34:45 UTC, deadalnix wrote:
 Yes, which happen anyway for shared objects !
This. We can do whole-program-optimization to finalize non-shared-library functions - and for shared-library functions it doesn't make sense to finalize for performance - you're trading one kind of indirection function call for another. Can we all please address or recognize this?
Sure, but the trouble is determining which classes are going to be shared.
By requiring export for what can be shared. Can be enforced by setting the visibility of the class's typeid.
Jun 06 2013
parent Manu <turkeyman gmail.com> writes:
On 7 June 2013 11:27, deadalnix <deadalnix gmail.com> wrote:

 On Friday, 7 June 2013 at 01:22:25 UTC, Walter Bright wrote:

 On 6/6/2013 5:43 PM, Jakob Ovrum wrote:

 On Friday, 7 June 2013 at 00:34:45 UTC, deadalnix wrote:

 Yes, which happen anyway for shared objects !
This. We can do whole-program-optimization to finalize non-shared-library functions - and for shared-library functions it doesn't make sense to finalize for performance - you're trading one kind of indirection function call for another. Can we all please address or recognize this?
Sure, but the trouble is determining which classes are going to be shared.
By requiring export for what can be shared. Can be enforced by setting the visibility of the class's typeid.
But you realise that a shared library isn't EXCLUSIVELY shared, it is still used internally? And the act of sharing it under your proposal means it can't be optimised ANYWHERE, even for internal usage.
Jun 06 2013
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06/06/2013 07:55 PM, Steven Schveighoffer wrote:
 On Thu, 06 Jun 2013 13:50:11 -0400, deadalnix <deadalnix gmail.com> wrote:

 On Thursday, 6 June 2013 at 15:06:38 UTC, Kapps wrote:
 On Thursday, 6 June 2013 at 01:08:36 UTC, deadalnix wrote:
 This is why I wrote that this may have been true in the past.
 Nevertheless, it is completely false today.
them there's still an overhead. This (2008) article goes into depth about how it handles it: www.codeproject.com/Articles/25801/JIT-Optimizations - Essentially uses frequency analysis to determine if the virtual method call is still going to call the same method as it would previously. Regardless, we can not perform such optimizations, so whether or not
and we can do it at link time the same way, that is all that matter for this discussion.
How do you finalize a method with the possibility that a dynamic library will come along and extend that type? Not a rhetorical question, I flexibility there, since they are run on a VM. -Steve
The more advanced JVM's assume a closed world for JIT optimization and then perform deoptimization whenever a dynamic library is loaded that invalidates some assumptions made for optimization.
Jun 06 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 6:08 PM, deadalnix wrote:

 several purposes like mock. We can't simple take this argument and don't look
at
 it with the light of history.
This is an interesting point that I didn't know about and didn't think of. Is there an article about it that goes into some depth about the rationale?
Jun 06 2013
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 11:19:52 Walter Bright wrote:
 On 6/5/2013 6:08 PM, deadalnix wrote:

 several purposes like mock. We can't simple take this argument and don't
 look at it with the light of history.
This is an interesting point that I didn't know about and didn't think of.

enable mock objects, but it also sounds like they didn't really revirtualize methods so much as provide was of replacing them (since revirtualizing them wouldn't help with stuff like static methods).
 Is there an article about it that goes into some depth about the rationale?
such an article - especially for those of us who don't know all that much - Jonathan M Davis
Jun 06 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 5:49 PM, deadalnix wrote:
 The first point : Anders Hejlsberg: There are several reasons. One is
 performance. We can observe that as people write code in Java, they forget to
 mark their methods final. Therefore, those methods are virtual. Because they're
 virtual, they don't perform as well. There's just performance overhead
 associated with being a virtual method. That's one issue.

 It is blatantly false. Maybe it was true at the time, I don't know, but I find
 quite disturbing that the first argument is 100% moot.
It may very well be false for JIT systems, but for native code, we already discussed that auto-finalization is unlikely to be practical for D.
Jun 06 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 2:20 PM, Walter Bright wrote:
 On 6/5/2013 5:49 PM, deadalnix wrote:
 The first point : Anders Hejlsberg: There are several reasons. One is
 performance. We can observe that as people write code in Java, they
 forget to
 mark their methods final. Therefore, those methods are virtual.
 Because they're
 virtual, they don't perform as well. There's just performance overhead
 associated with being a virtual method. That's one issue.

 It is blatantly false. Maybe it was true at the time, I don't know,
 but I find
 quite disturbing that the first argument is 100% moot.
It may very well be false for JIT systems, but for native code, we already discussed that auto-finalization is unlikely to be practical for D.
I think class hierarchy analysis is very doable for whole D projects. You just pass the tool all files in the project and it does its thing. http://www.cs.ucla.edu/~palsberg/tba/papers/dean-grove-chambers-ecoop95.pdf This would actually be a great GSoC-style project, distributable via tools/. Andrei
Jun 06 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 12:31 PM, Andrei Alexandrescu wrote:
 I think class hierarchy analysis is very doable for whole D projects. You just
 pass the tool all files in the project and it does its thing.

 http://www.cs.ucla.edu/~palsberg/tba/papers/dean-grove-chambers-ecoop95.pdf

 This would actually be a great GSoC-style project, distributable via tools/.
The trouble, as has been pointed out before, is shared libraries.
Jun 06 2013
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 12:37:12 Walter Bright wrote:
 On 6/6/2013 12:31 PM, Andrei Alexandrescu wrote:
 I think class hierarchy analysis is very doable for whole D projects. You
 just pass the tool all files in the project and it does its thing.
 
 http://www.cs.ucla.edu/~palsberg/tba/papers/dean-grove-chambers-ecoop95.pd
 f
 
 This would actually be a great GSoC-style project, distributable via
 tools/.
The trouble, as has been pointed out before, is shared libraries.
Yes. Especially those which are dynamically loaded while the program is running. What you _could_ do is have a program which looked over your code and pointed out which functions which probably didn't need to be virtual, but it would still need to be up to the programmer to actually make the change, since the program could easily be wrong. - Jonathan M Davis
Jun 06 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 3:37 PM, Walter Bright wrote:
 On 6/6/2013 12:31 PM, Andrei Alexandrescu wrote:
 I think class hierarchy analysis is very doable for whole D projects.
 You just
 pass the tool all files in the project and it does its thing.

 http://www.cs.ucla.edu/~palsberg/tba/papers/dean-grove-chambers-ecoop95.pdf


 This would actually be a great GSoC-style project, distributable via
 tools/.
The trouble, as has been pointed out before, is shared libraries.
I wrote: "I think class hierarchy analysis is very doable for whole D projects." Whole. Projects. WHOLE PROJECTS. WHOLE. PROJECTS. Worked on by Top. Men. Andrei
Jun 06 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 15:56:24 Andrei Alexandrescu wrote:
 On 6/6/13 3:37 PM, Walter Bright wrote:
 On 6/6/2013 12:31 PM, Andrei Alexandrescu wrote:
 I think class hierarchy analysis is very doable for whole D projects.
 You just
 pass the tool all files in the project and it does its thing.
 
 http://www.cs.ucla.edu/~palsberg/tba/papers/dean-grove-chambers-ecoop95.p
 df
 
 
 This would actually be a great GSoC-style project, distributable via
 tools/.
The trouble, as has been pointed out before, is shared libraries.
I wrote: "I think class hierarchy analysis is very doable for whole D projects." Whole. Projects. WHOLE PROJECTS. WHOLE. PROJECTS. Worked on by Top. Men.
LOL. Yes, well. That's certainly much closer, but if the shared library changes and later derives more types than it did before. And since shared libraries can be swapped out, that could break the optimizations that the tool did, so you'd have to run the tool over the whole thing again. So, it's definitely an idea with potential, but I don't think that it could be guaranteed to work in all cases, and the programmer likely have to be aware of when it didn't work in order to avoid some nasty bugs. - Jonathan M Davis
Jun 06 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 4:30 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 15:56:24 Andrei Alexandrescu wrote:
 On 6/6/13 3:37 PM, Walter Bright wrote:
 On 6/6/2013 12:31 PM, Andrei Alexandrescu wrote:
 I think class hierarchy analysis is very doable for whole D projects.
 You just
 pass the tool all files in the project and it does its thing.

 http://www.cs.ucla.edu/~palsberg/tba/papers/dean-grove-chambers-ecoop95.p
 df


 This would actually be a great GSoC-style project, distributable via
 tools/.
The trouble, as has been pointed out before, is shared libraries.
I wrote: "I think class hierarchy analysis is very doable for whole D projects." Whole. Projects. WHOLE PROJECTS. WHOLE. PROJECTS. Worked on by Top. Men.
LOL. Yes, well. That's certainly much closer, but if the shared library changes and later derives more types than it did before. And since shared libraries can be swapped out, that could break the optimizations that the tool did, so you'd have to run the tool over the whole thing again. So, it's definitely an idea with potential, but I don't think that it could be guaranteed to work in all cases, and the programmer likely have to be aware of when it didn't work in order to avoid some nasty bugs.
There would be no bugs, worst case compilation errors. (The tool I'm envisioning would add final annotations or prompt the user to add them.) Andrei
Jun 06 2013
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 17:23:03 Andrei Alexandrescu wrote:
 LOL. Yes, well. That's certainly much closer, but if the shared library
 changes and later derives more types than it did before. And since shared
 libraries can be swapped out, that could break the optimizations that the
 tool did, so you'd have to run the tool over the whole thing again. So,
 it's definitely an idea with potential, but I don't think that it could
 be guaranteed to work in all cases, and the programmer likely have to be
 aware of when it didn't work in order to avoid some nasty bugs.
There would be no bugs, worst case compilation errors. (The tool I'm envisioning would add final annotations or prompt the user to add them.)
That would definitely work, but it would probably work better if the user were told to add them rather than them being add automatically, or you risk some functions being devrtualized when it's known a shared library may need them to be virtual in a later version. Regardless, at minimum, it would provide a way to track down all of the virtual functions which may not need to be virtual, which could be quite valuable regardless of whether the programmer decides to make them non-virtual or not - Jonathan M Davis
Jun 06 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 5:45 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 17:23:03 Andrei Alexandrescu wrote:
 LOL. Yes, well. That's certainly much closer, but if the shared library
 changes and later derives more types than it did before. And since shared
 libraries can be swapped out, that could break the optimizations that the
 tool did, so you'd have to run the tool over the whole thing again. So,
 it's definitely an idea with potential, but I don't think that it could
 be guaranteed to work in all cases, and the programmer likely have to be
 aware of when it didn't work in order to avoid some nasty bugs.
There would be no bugs, worst case compilation errors. (The tool I'm envisioning would add final annotations or prompt the user to add them.)
That would definitely work, but it would probably work better if the user were told to add them rather than them being add automatically, or you risk some functions being devrtualized when it's known a shared library may need them to be virtual in a later version. Regardless, at minimum, it would provide a way to track down all of the virtual functions which may not need to be virtual, which could be quite valuable regardless of whether the programmer decides to make them non-virtual or not - Jonathan M Davis
--in-place Andrei
Jun 06 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
Jun 06 2013
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 14:57:00 Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
It could tell the programmer which functions it _thinks_ don't need to be virtual, but it can't be 100% correct. So, it would effectively be a lint-like tool targeting possible devirtualization opportunities. It would actually be potentially useful regardless of whether virtual or non-virtual is the default, since programmers may have needlessly marked functions as virtual. But if it's a question of whether it's a good solution for optimizing away virtuality instead of making functions non-virtual, then I don't think that it would fly - not if optimization is a prime concern. It would just be a nice helper tool for static analysis which could give you suggestions on things you might be able to improve in your program. But as it sounds like the primary argument which has swayed you towards making non-virtual the default is tied to cleaner code evolution and maintenance rather than performance, the suggestion obviously wouldn't be a viable counterargument for going with virtual-by-default. - Jonathan M Davis
Jun 06 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 3:12 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 14:57:00 Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
It could tell the programmer which functions it _thinks_ don't need to be virtual, but it can't be 100% correct. So, it would effectively be a lint-like tool targeting possible devirtualization opportunities. It would actually be potentially useful regardless of whether virtual or non-virtual is the default, since programmers may have needlessly marked functions as virtual. But if it's a question of whether it's a good solution for optimizing away virtuality instead of making functions non-virtual, then I don't think that it would fly - not if optimization is a prime concern. It would just be a nice helper tool for static analysis which could give you suggestions on things you might be able to improve in your program.
I know. But people are never going to use that tool.
 But as it sounds like the primary argument which has swayed you towards making
 non-virtual the default is tied to cleaner code evolution and maintenance
 rather than performance, the suggestion obviously wouldn't be a viable
 counterargument for going with virtual-by-default.
The thing is, when code 'works' there is rarely sufficient motivation to go back and annotate things for safety and performance (hence why the tool above will be a failure). Code that works is left alone, and we see the situation Manu is talking about. But if it's final by default, if the user needs it to be virtual, then he has to go back and add the annotation - it's not going to work, and the compiler will tell him it doesn't work. I wouldn't have changed my mind if it were possible for the compiler to auto-finalize methods. BTW, this is also why D hasn't opted for the pointer tagging system Rust has. It all looks great on paper, but I suspect that in practice not much use will be made of it - people will just default to using the most widely usable pointer type and their code will work and they'll forget about the rest of the annotations. I have a lot of experience with this with DOS 16 bit code. There we had all kinds of pointer types - near, far, SS relative, CS relative, etc. You know what people did? The default pointer type. Almost nobody used those optimized pointer types, even though they got big speed boosts when used appropriately. What does work is throw -O and have the compiler go to town and optimize the hell out of it. As much as possible we should be selecting default semantics that enable -O to kick ass.
Jun 06 2013
next sibling parent reply "Flamaros" <flamaros.xavier gmail.com> writes:
On Thursday, 6 June 2013 at 23:48:33 UTC, Walter Bright wrote:
 On 6/6/2013 3:12 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 14:57:00 Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
It could tell the programmer which functions it _thinks_ don't need to be virtual, but it can't be 100% correct. So, it would effectively be a lint-like tool targeting possible devirtualization opportunities. It would actually be potentially useful regardless of whether virtual or non-virtual is the default, since programmers may have needlessly marked functions as virtual. But if it's a question of whether it's a good solution for optimizing away virtuality instead of making functions non-virtual, then I don't think that it would fly - not if optimization is a prime concern. It would just be a nice helper tool for static analysis which could give you suggestions on things you might be able to improve in your program.
I know. But people are never going to use that tool.
I think it depend of his simplicity and integration in the common D process development. Maybe because D build fast we can add some extra steps during build of the release? And developers of companies that develop the biggest application will be aware of this tool and certainly have script or advanced tools to build their software release, adding a line during the building process seems acceptable.
Jun 06 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 4:56 PM, Flamaros wrote:
 I think it depend of his simplicity and integration in the common D process
 development. Maybe because D build fast we can add some extra steps during
build
 of the release?
 And developers of companies that develop the biggest application will be aware
 of this tool and certainly have script or advanced tools to build their
software
 release, adding a line during the building process seems acceptable.
Consider a trivial and effective DMD tool for debugging and improving efficiency: -cov and I'm the only one I know of that uses it. Usage really doesn't get any simpler than that, but it fails because it requires the user to look at the feedback and then go edit his source code. Bzzzt! What does work is: -O and voila! Program runs faster! When we're talking about payoff for D users, bang for the buck, whatever, improving -O is where it's at, and making the semantics of D amenable to automated optimization is a big win.
Jun 06 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 7 June 2013 at 01:12:07 UTC, Walter Bright wrote:
 On 6/6/2013 4:56 PM, Flamaros wrote:
 I think it depend of his simplicity and integration in the 
 common D process
 development. Maybe because D build fast we can add some extra 
 steps during build
 of the release?
 And developers of companies that develop the biggest 
 application will be aware
 of this tool and certainly have script or advanced tools to 
 build their software
 release, adding a line during the building process seems 
 acceptable.
Consider a trivial and effective DMD tool for debugging and improving efficiency: -cov and I'm the only one I know of that uses it.
Integration to IDE is the key. I did use similar tool in many languages but only rarely in D.
Jun 06 2013
prev sibling next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Friday, 7 June 2013 at 01:12:07 UTC, Walter Bright wrote:
 and I'm the only one I know of that uses it.
I've used it in LuaD since a couple of years ago to check what code isn't covered by tests. It's much more useful now that it has VisualD integration and the -cov=nn form, though.
Jun 06 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 7:01 PM, Jakob Ovrum wrote:
 On Friday, 7 June 2013 at 01:12:07 UTC, Walter Bright wrote:
 and I'm the only one I know of that uses it.
I've used it in LuaD since a couple of years ago to check what code isn't covered by tests. It's much more useful now that it has VisualD integration and the -cov=nn form, though.
There's 2! :-)
Jun 06 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 19:22:56 Walter Bright wrote:
 On 6/6/2013 7:01 PM, Jakob Ovrum wrote:
 On Friday, 7 June 2013 at 01:12:07 UTC, Walter Bright wrote:
 and I'm the only one I know of that uses it.
I've used it in LuaD since a couple of years ago to check what code isn't covered by tests. It's much more useful now that it has VisualD integration and the -cov=nn form, though.
There's 2! :-)
I've definitely used it before (particularly when originally writing std.datetime), but it's not something that I remember to use often enough. I'm thorough enough with my unit tests that I tend to assume that I have fully coverage (which I probably do in most cases), but it would be much safer to actually check. Having at least the percentage spit out as part of the build step would certainly be useful though, since then I could see when it changes.. I wonder if I should set it up so that my unit test builds do that. - Jonathan M Davis
Jun 06 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 7:40 PM, Jonathan M Davis wrote:
 I've definitely used it before (particularly when originally writing
 std.datetime), but it's not something that I remember to use often enough. I'm
 thorough enough with my unit tests that I tend to assume that I have fully
 coverage (which I probably do in most cases), but it would be much safer to
 actually check. Having at least the percentage spit out as part of the build
 step would certainly be useful though, since then I could see when it
 changes.. I wonder if I should set it up so that my unit test builds do that.
If you look in phobos' win32.mak, you'll see some rules that give a minimum cov percentage that each module must pass. It's meant to detect if the coverage is backsliding without requiring any ongoing effort.
Jun 07 2013
prev sibling next sibling parent reply Brad Roberts <braddr puremagic.com> writes:
On 6/6/13 6:12 PM, Walter Bright wrote:

 and I'm the only one I know of that uses it.
You really need to get out of the habit of associating "I can't remember people telling me they use something" and "no one uses it". 1) your memory isn't perfect. 2) just because you're not aware it's happening doesn't mean it isn't. 3) likely lots of other reasons, but come on, 1 and 2 are more than sufficient to kill the phrase. Yeah, I know you couched it with "I know of" this time, but the pattern is pretty silly regardless. This community is awfully myopic when it comes to use of personal history and mapping it onto everyone else's behavior.
Jun 06 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/7/13 12:34 AM, Brad Roberts wrote:
 On 6/6/13 6:12 PM, Walter Bright wrote:

 and I'm the only one I know of that uses it.
You really need to get out of the habit of associating "I can't remember people telling me they use something" and "no one uses it".
I concur. That, and "When I was at Boeing" :o). Andrei
Jun 06 2013
parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Fri, 07 Jun 2013 07:20:12 +0200, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/7/13 12:34 AM, Brad Roberts wrote:
 On 6/6/13 6:12 PM, Walter Bright wrote:

 and I'm the only one I know of that uses it.
You really need to get out of the habit of associating "I can't remember people telling me they use something" and "no one uses it".
I concur. That, and "When I was at Boeing" :o).
Aw, I like those. I tend to think of it as Walter sitting with a beard, pipe in hand, telling stories to his grandchildren. -- Simen
Jun 07 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 9:34 PM, Brad Roberts wrote:
 On 6/6/13 6:12 PM, Walter Bright wrote:

 and I'm the only one I know of that uses it.
You really need to get out of the habit of associating "I can't remember people telling me they use something" and "no one uses it". 1) your memory isn't perfect. 2) just because you're not aware it's happening doesn't mean it isn't. 3) likely lots of other reasons, but come on, 1 and 2 are more than sufficient to kill the phrase. Yeah, I know you couched it with "I know of" this time, but the pattern is pretty silly regardless. This community is awfully myopic when it comes to use of personal history and mapping it onto everyone else's behavior.
A pretty good metric of some feature being used is the frequency it comes up in discussion here and the action it sees in bugzilla. There are two obvious reasons why a feature would not have much buzz: 1. it works perfectly 2. it isn't being used Value Range Propagation is a pretty good example of (1). Sadly, I suspect -cov is (2). I'd be happy to be wrong about that.
Jun 07 2013
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 06/07/2013 09:22 AM, Walter Bright wrote:
 ...
 There are two obvious reasons why a feature would not have much buzz:

 1. it works perfectly
 2. it isn't being used

 Value Range Propagation is a pretty good example of (1). Sadly, I
 suspect -cov is (2).

 I'd be happy to be wrong about that.
You are certainly wrong about the value range propagation part. The transformers for the bitwise operators are not the best possible. ubyte x = ((y&252)^2)+1; The above term can be easily proven to fit into ubyte by just using an analysis of the ranges of its subterms, yet DMD rejects it.
Jun 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/7/2013 2:52 AM, Timon Gehr wrote:
 You are certainly wrong about the value range propagation part. The
transformers
 for the bitwise operators are not the best possible.

 ubyte x = ((y&252)^2)+1;

 The above term can be easily proven to fit into ubyte by just using an analysis
 of the ranges of its subterms, yet DMD rejects it.
Since it's easy, I expect a pull request from you, or at least a bugzilla entry with a description of the algorithm to use!
Jun 07 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 06/08/2013 12:43 AM, Walter Bright wrote:
 On 6/7/2013 2:52 AM, Timon Gehr wrote:
 You are certainly wrong about the value range propagation part. The
 transformers
 for the bitwise operators are not the best possible.

 ubyte x = ((y&252)^2)+1;

 The above term can be easily proven to fit into ubyte by just using an
 analysis
 of the ranges of its subterms, yet DMD rejects it.
Since it's easy, I expect a pull request from you,
Well, 1. DMD is written in C++. 2. My last pull request has rotten for 6 months now.
 or at least a bugzilla entry with a description of the algorithm to use!
http://d.puremagic.com/issues/show_bug.cgi?id=10310
Jun 08 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/8/2013 4:35 PM, Timon Gehr wrote:
 2. My last pull request has rotten for 6 months now.
Sorry about that.
Jun 08 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 06/09/2013 04:57 AM, Walter Bright wrote:
 On 6/8/2013 4:35 PM, Timon Gehr wrote:
 2. My last pull request has rotten for 6 months now.
Sorry about that.
No problem. Thanks for merging!
Jun 09 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/13 6:58 AM, Timon Gehr wrote:
 On 06/09/2013 04:57 AM, Walter Bright wrote:
 On 6/8/2013 4:35 PM, Timon Gehr wrote:
 2. My last pull request has rotten for 6 months now.
Sorry about that.
No problem. Thanks for merging!
FWIW I look forward to https://github.com/D-Programming-Language/dmd/pull/1839 being pulled. I've recently done some manual work of removing unnecessary dependencies, and it's very tedious. Any help would be great. For example, I'd love it if the compiler with -deps generated info such as: importUseCount modulename count meaning the imported modulename was used inside the current module count times. A simple script can grep for instances of count=0 so that users know what imports can be safely removed. A low use count such as 1-2 suggests it's possible to push that import down to places using it. One more important use case is: importUseCountInUnittests modulename count which only counts use inside unittests. Modules that are only used while unittesting should not be imported for normal use. Andrei
Jun 09 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/13 6:58 AM, Timon Gehr wrote:
 On 06/09/2013 04:57 AM, Walter Bright wrote:
 On 6/8/2013 4:35 PM, Timon Gehr wrote:
 2. My last pull request has rotten for 6 months now.
Sorry about that.
No problem. Thanks for merging!
Speaking of imports and dependencies, there was a dependency graph for Phobos that was circulating at some point. It used a script in conjunction with a graph printing program (such as dot or viz). Where is it? Thanks, Andrei
Jun 09 2013
next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Sun, 09 Jun 2013 16:19:32 +0200, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/9/13 6:58 AM, Timon Gehr wrote:
 On 06/09/2013 04:57 AM, Walter Bright wrote:
 On 6/8/2013 4:35 PM, Timon Gehr wrote:
 2. My last pull request has rotten for 6 months now.
Sorry about that.
No problem. Thanks for merging!
Speaking of imports and dependencies, there was a dependency graph for Phobos that was circulating at some point. It used a script in conjunction with a graph printing program (such as dot or viz). Where is it? Thanks, Andrei
Here: http://forum.dlang.org/thread/humcf2$2ufd$1 digitalmars.com -- Simen
Jun 09 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/13 10:19 AM, Andrei Alexandrescu wrote:
 On 6/9/13 6:58 AM, Timon Gehr wrote:
 On 06/09/2013 04:57 AM, Walter Bright wrote:
 On 6/8/2013 4:35 PM, Timon Gehr wrote:
 2. My last pull request has rotten for 6 months now.
Sorry about that.
No problem. Thanks for merging!
Speaking of imports and dependencies, there was a dependency graph for Phobos that was circulating at some point. It used a script in conjunction with a graph printing program (such as dot or viz). Where is it? Thanks, Andrei
Philippe sent this to me via email. Getting it to work again doesn't seem like a slam dunk, so I'm posting this in case someone has the time. I think it would be a nice tool to have.
 The relevant graph algorithms are there:

 https://github.com/PhilippeSigaud/dranges/blob/master/graphalgorithm.d

 Lines 767-873. I see they are commented out, it seems there was a problem in
2.050 version of std.algo.startswith.
 Seeing how today Phobos and DMD are quite better, there is probably a much
more elegant way to do the extraction now.

 The module also needs:

 https://github.com/PhilippeSigaud/dranges/blob/master/graph.d
 and
 https://github.com/PhilippeSigaud/dranges/blob/master/graphrange.d
Andrei
Jun 09 2013
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 6/10/13, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 Philippe sent this to me via email
 The relevant graph algorithms are there:

 https://github.com/PhilippeSigaud/dranges/blob/master/graphalgorithm.d
This is very, very hacky. It fails as soon as you have a comment line inbetween your imports. Anyway it works if you're very careful where you put the running script. It took just a few small fixes, clone this repo: https://github.com/AndrejMitrovic/dranges The FixGraph branch should be checked out. Then copy the "test.d" file one directory UP, and run "rdmd test.d"
Jun 09 2013
prev sibling next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Friday, 7 June 2013 at 07:22:18 UTC, Walter Bright wrote:
 A pretty good metric of some feature being used is the 
 frequency it comes up in discussion here and the action it sees 
 in bugzilla.

 There are two obvious reasons why a feature would not have much 
 buzz:

 1. it works perfectly
 2. it isn't being used

 Value Range Propagation is a pretty good example of (1). Sadly, 
 I suspect -cov is (2).

 I'd be happy to be wrong about that.
Or it just isn't used much compared to other things and works well enough when it is. For what it's worth, there are a few bugs on Bugzilla about -cov, so it is being used.
Jun 07 2013
parent "Dicebot" <m.strashun gmail.com> writes:
On Friday, 7 June 2013 at 10:35:12 UTC, Peter Alexander wrote:
 Or it just isn't used much compared to other things and works 
 well enough when it is.

 For what it's worth, there are a few bugs on Bugzilla about 
 -cov, so it is being used.
It is also worth saying that "-cov" is more of production tool, you start using it when you invest your efforts into long-term project. It will never get much attention while certain language bugs/issues persist that jump straight into your face when you start experimenting with small snippets. Nothing wrong with "-cov' itself here.
Jun 07 2013
prev sibling parent reply "Rob T" <alanb ucora.com> writes:
On Friday, 7 June 2013 at 07:22:18 UTC, Walter Bright wrote:
 A pretty good metric of some feature being used is the 
 frequency it comes up in discussion here and the action it sees 
 in bugzilla.

 There are two obvious reasons why a feature would not have much 
 buzz:

 1. it works perfectly
 2. it isn't being used

 Value Range Propagation is a pretty good example of (1). Sadly, 
 I suspect -cov is (2).

 I'd be happy to be wrong about that.
If D had a compiler option switch to collect statistics on feature usage, maybe you could get something a lot better than a guess for both 1 and 2. Such a thing may be useful for other reasons too. --rt
Jun 07 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/7/2013 9:51 AM, Rob T wrote:
 If D had a compiler option switch to collect statistics on feature usage, maybe
 you could get something a lot better than a guess for both 1 and 2.
And then you'd have to convince people to use it and send you the results!
Jun 07 2013
parent "Rob T" <alanb ucora.com> writes:
On Friday, 7 June 2013 at 22:57:01 UTC, Walter Bright wrote:
 On 6/7/2013 9:51 AM, Rob T wrote:
 If D had a compiler option switch to collect statistics on 
 feature usage, maybe
 you could get something a lot better than a guess for both 1 
 and 2.
And then you'd have to convince people to use it and send you the results!
The stats gathering idea may be food for further thought. For example a more-likely-to-work method may be to allow users to rate features through the on-line documentation. You could derive a reasonably accurate idea of success and failings from such a thing. Perhaps something like this has been successfully done before that can serve as a model to emulate. --rt
Jun 07 2013
prev sibling parent reply Peter Williams <pwil3058 bigpond.net.au> writes:
On 07/06/13 11:12, Walter Bright wrote:
 On 6/6/2013 4:56 PM, Flamaros wrote:
 I think it depend of his simplicity and integration in the common D
 process
 development. Maybe because D build fast we can add some extra steps
 during build
 of the release?
 And developers of companies that develop the biggest application will
 be aware
 of this tool and certainly have script or advanced tools to build
 their software
 release, adding a line during the building process seems acceptable.
Consider a trivial and effective DMD tool for debugging and improving efficiency: -cov and I'm the only one I know of that uses it.
I use it to make sure that my unittests are complete. Peter PS It would be nice if it printed the overview (i.e. % complete line) to stdout as well as sticking it in the coverage file report.
Jun 06 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 11:30 PM, Peter Williams wrote:
 I use it to make sure that my unittests are complete.
Awesome! -cov is also useful for: 1. debugging - if your unittests can't seem to cover certain lines, those lines might indicate a bug 2. performance - giving usage counts of lines tells you where the 'hot' paths are, and those can be hand-tuned It's also quite possible to have the compiler read the coverage report, use it to identify the hot paths, and rearrange the code while optimizing so that the hot paths have the fewest branches. Profile guided optimization, if you will.
 Peter
 PS It would be nice if it printed the overview (i.e. % complete line) to stdout
 as well as sticking it in the coverage file report.
Please file this as an enhancement request on bugzilla.
Jun 07 2013
parent Peter Williams <pwil3058 bigpond.net.au> writes:
On 07/06/13 17:28, Walter Bright wrote:
 On 6/6/2013 11:30 PM, Peter Williams wrote:
 I use it to make sure that my unittests are complete.
Awesome! -cov is also useful for: 1. debugging - if your unittests can't seem to cover certain lines, those lines might indicate a bug
That's what I meant by "unittests are complete" i.e. they cover all of the code.
 2. performance - giving usage counts of lines tells you where the 'hot'
 paths are, and those can be hand-tuned

 It's also quite possible to have the compiler read the coverage report,
 use it to identify the hot paths, and rearrange the code while
 optimizing so that the hot paths have the fewest branches. Profile
 guided optimization, if you will.

 Peter
 PS It would be nice if it printed the overview (i.e. % complete line)
 to stdout
 as well as sticking it in the coverage file report.
Please file this as an enhancement request on bugzilla.
Done. Peter
Jun 07 2013
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 16:48:32 Walter Bright wrote:
 On 6/6/2013 3:12 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 14:57:00 Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
It could tell the programmer which functions it _thinks_ don't need to be virtual, but it can't be 100% correct. So, it would effectively be a lint-like tool targeting possible devirtualization opportunities. It would actually be potentially useful regardless of whether virtual or non-virtual is the default, since programmers may have needlessly marked functions as virtual. But if it's a question of whether it's a good solution for optimizing away virtuality instead of making functions non-virtual, then I don't think that it would fly - not if optimization is a prime concern. It would just be a nice helper tool for static analysis which could give you suggestions on things you might be able to improve in your program.
I know. But people are never going to use that tool.
Some would, but I completely agree that it wouldn't be used enough to be viable as a real solution for making functions non-virtual.
 I wouldn't have changed my mind if it were possible for the compiler to
 auto-finalize methods.
Yeah, if the compiler could figure it all out, that would be great, and this might not be necessary, but unfortunately, it clearly can't -t hough even if it could, there's definitely some value in the programmer being explicit about what is and isn't overridden. Taken to the extreme, that would probably require that every single member function be marked with either virtual, override, or final so that the programmer has to explicitly choose in each case what a function is supposed to be doing in terms of virtuality, but that's likely overkill and would break too much code at this point even if it were ultimately a good idea. As it stands, we can simply say that introducing functions must be marked with virtual and that functions marked with override must override a function which is marked with either virtual or override, and all people will have to do is mark the introducing functions with virtual. That'll certainly break some code, but considering how big a change it is, it affects a suprisingly small portion of the code. - Jonathan M Davis
Jun 06 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 6:12 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 14:57:00 Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
It could tell the programmer which functions it _thinks_ don't need to be virtual, but it can't be 100% correct.
It would be 100% correct if given the entire application. You may want to take a look at the CHA paper. Andrei
Jun 06 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 20:01:09 Andrei Alexandrescu wrote:
 On 6/6/13 6:12 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 14:57:00 Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
It could tell the programmer which functions it _thinks_ don't need to be virtual, but it can't be 100% correct.
It would be 100% correct if given the entire application. You may want to take a look at the CHA paper.
It may be (I'd have to look at the paper), but it's my understanding that dynamically loading libraries while the program would kill that when those libraries can change. You could have functions which are not currently overridden but which a later version of a plugin could override without needing the program to be recompiled. The tool would mark such functions as final, making it so that the plugin couldn't add such an override later without requiring the main program to be rebuilt. - Jonathan M Davis
Jun 06 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 8:13 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 20:01:09 Andrei Alexandrescu wrote:
 On 6/6/13 6:12 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 14:57:00 Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
It could tell the programmer which functions it _thinks_ don't need to be virtual, but it can't be 100% correct.
It would be 100% correct if given the entire application. You may want to take a look at the CHA paper.
It may be (I'd have to look at the paper), but it's my understanding that dynamically loading libraries while the program would kill that when those libraries can change.
I'm carefully mentioning "entire applications" every time I got a chance. It's not working. Andrei
Jun 06 2013
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, June 07, 2013 01:04:40 Andrei Alexandrescu wrote:
 On 6/6/13 8:13 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 20:01:09 Andrei Alexandrescu wrote:
 On 6/6/13 6:12 PM, Jonathan M Davis wrote:
 On Thursday, June 06, 2013 14:57:00 Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
It could tell the programmer which functions it _thinks_ don't need to be virtual, but it can't be 100% correct.
It would be 100% correct if given the entire application. You may want to take a look at the CHA paper.
It may be (I'd have to look at the paper), but it's my understanding that dynamically loading libraries while the program would kill that when those libraries can change.
I'm carefully mentioning "entire applications" every time I got a chance. It's not working.
I know, but the problem is that part of the application (the external library or plugin) could change over time rendering some of the devirtualizations invalid. You'd have to have some sort of guarantee that that's not going to happen to avoid problems. - Jonathan M Davis
Jun 06 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 10:04 PM, Andrei Alexandrescu wrote:
 I'm carefully mentioning "entire applications" every time I got a chance. It's
 not working.
As Jonathan mentioned, I don't see how this can work with shared libraries (especially dynamically loaded ones), as you cannot know what the entire application is. deadalnix mentioned enforcing 'export' on classes exposed to shared libraries, but then aren't we back to expecting user annotation rather than doing things automatically?
Jun 07 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 7 June 2013 at 07:31:52 UTC, Walter Bright wrote:
 deadalnix mentioned enforcing 'export' on classes exposed to 
 shared libraries, but then aren't we back to expecting user 
 annotation rather than doing things automatically?
Yes, let me explain more. The export problem isn't new, and you'll find 2 major approach : the UNIX approach of export everything, and the windows approach of requiring explicit export. The first approach tend to show its limit. It limit what the optimizer can do, and create binary more and more costly to link (especially since the number of symbol tends to explode with metaprogramming) (linking a debug version of LLVM as shared object for instance is quite costly). Both GCC and clang propose language extensions to manage export explicitly on UNIXes. Additionally, it is beneficial to be very picky about what is exported. Once something is exported, you are tied to an API (as with public) but also to an ABI. Adding a function to a class for instance, can cause a relayout of the vtable. None of the existing interface is broken, but the shared object is broken anyway, because of symbols it don't use. Considering this, we need either a way to export explicitly, or the other way around. It is also worth noting that consistency accross plateforms is a plus, and this is an area where we can do better that C (and we already do). Regarding virtual in particular, it is known that calling a virtual method have a cost. It cause an indirect branch, but also prevent the compiler from optimizing, as the call is opaque. The same way, calling to a shared object is opaque to the compiler, and so, you can't get the full benefice of finalizing the exported function. Requiring classes to be exported provide the following benefit : - LTO can finalize methods that aren't overridden. It include function that you want virtual by design, but you don't use that capability is your specific situation. In this regard, this is superior to the explicit virtual solution as it do not require to annotate virtual and can finalize method that would have to be annotated virtual. - Shared object can be stripped of all non exported symbols, reducing their size. - Reduce the surface area of breakage for shared objects. It however require careful crafting of exported objects. I think this is mitigated by the fact that shared object interface require careful crafting anyway, as what is exported cannot be unexported (and cannot even change its ABI in way that aren't easily visible in the source). The solution can be completed later by a tool as Andrei proposed (great idea), but by itself provide a lot of opportunity to finalize automagically. As you argued, this is the best way to go. Executive summary : - We can provide a toll to finalize the whole program. - We can automatically finalize everything that isn't exported. - We don't break any code. - We get consistency between windows and UNIXes. - Manu will rant. I see only benefits :D
Jun 07 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/7/2013 8:23 AM, deadalnix wrote:
 Yes, let me explain more.
Thanks for the detailed explanation.
Jun 07 2013
parent "David Nadlinger" <code klickverbot.at> writes:
On Friday, 7 June 2013 at 23:52:59 UTC, Walter Bright wrote:
 On 6/7/2013 8:23 AM, deadalnix wrote:
 Yes, let me explain more.
Thanks for the detailed explanation.
If you are looking further material on the topic, you might also want to try a quick web search for '-fvisibility=hidden', which is the GCC flag for hiding symbols by default. David
Jun 07 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 8 June 2013 01:23, deadalnix <deadalnix gmail.com> wrote:

 Requiring classes to be exported provide the following benefit :
  - LTO can finalize methods that aren't overridden. It include function
 that you want virtual by design, but you don't use that capability is your
 specific situation. In this regard, this is superior to the explicit
 virtual solution as it do not require to annotate virtual and can finalize
 method that would have to be annotated virtual.
  - Shared object can be stripped of all non exported symbols, reducing
 their size.
  - Reduce the surface area of breakage for shared objects.

 It however require careful crafting of exported objects. I think this is
 mitigated by the fact that shared object interface require careful crafting
 anyway, as what is exported cannot be unexported (and cannot even change
 its ABI in way that aren't easily visible in the source).

 The solution can be completed later by a tool as Andrei proposed (great
 idea), but by itself provide a lot of opportunity to finalize
 automagically. As you argued, this is the best way to go.

 Executive summary :
  - We can provide a toll to finalize the whole program.
  - We can automatically finalize everything that isn't exported.
  - We don't break any code.
  - We get consistency between windows and UNIXes.
  - Manu will rant.

 I see only benefits :D
It's like you've missed most of my points though. So under this proposal, which *entails* said 'sufficiently advanced optimiser/linker/etc', which doesn't exist. And even if it did, what architectures are supported? Many architectures will most certainly never receive support, and these are usually the architectures that need it most. Put that aside for a second, it's still flawed in a basic sense. Anything 'exported' still falls under precisely the same set of problems as we have now. By exporting something, NOTHING can be de-virtualised, even when used internally. And exported things DO get used internally. So by exporting, you lose all benefit within your own code anyway. I can tell you that in my case, we export a lot(/most) things. Renderer api, sound api, etc are often all in their own libraries. So none of them are ever eligible for optimisation. Additionally, one of my central points is completely un-satisfied, that is, 3rd party libraries. To make use of them at all implies they are exported, so there is never any possibility for optimisation there. What am I left with? Basically nothing. And finally, even if none of these problems existed, you still don't want virtual methods across export boundaries. Just because you use a class implemented in a DLL, that doesn't prevent you from inlining the trivial accessors within your own code. Using a library is an ABI commitment anyway, and it's completely normal to inline foreign libraries trivial methods within your own code. So, basically none of my usages are satisfied by your proposal. Sorry.
Jun 07 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/7/2013 5:15 PM, Manu wrote:
 I can tell you that in my case, we export a lot(/most) things. Renderer api,
 sound api, etc are often all in their own libraries. So none of them are ever
 eligible for optimisation.
I'm curious why these exports all must be classes, and cannot be structs?
Jun 07 2013
parent reply Manu <turkeyman gmail.com> writes:
On 8 June 2013 13:25, Walter Bright <newshound2 digitalmars.com> wrote:

 On 6/7/2013 5:15 PM, Manu wrote:

 I can tell you that in my case, we export a lot(/most) things. Renderer
 api,
 sound api, etc are often all in their own libraries. So none of them are
 ever
 eligible for optimisation.
I'm curious why these exports all must be classes, and cannot be structs?
Because they embody functionality, not just data. That's just how many many programmers write code. Go to university for a couple of years, see what they tell you... ;) Some of these systems can effectively be considered plugins. Consider OpenGL/DirectX? DirectSound/XAudio? Linux has a million back-end API's to choose from. I can see why many people feel it's natural to design their API's/systems that way, right or wrong. I don't agree with it personally, I would write it differently, but I'll never win that argument. Tech/systems programmers are vastly outnumbered in most studios. And of course, most programmers are junior-mid experience, that's just commercial reality.
Jun 07 2013
next sibling parent reply "Flamaros" <flamaros.xavier gmail.com> writes:
On Saturday, 8 June 2013 at 03:59:25 UTC, Manu wrote:
 On 8 June 2013 13:25, Walter Bright 
 <newshound2 digitalmars.com> wrote:

 On 6/7/2013 5:15 PM, Manu wrote:

 I can tell you that in my case, we export a lot(/most) 
 things. Renderer
 api,
 sound api, etc are often all in their own libraries. So none 
 of them are
 ever
 eligible for optimisation.
I'm curious why these exports all must be classes, and cannot be structs?
Because they embody functionality, not just data. That's just how many many programmers write code. Go to university for a couple of years, see what they tell you... ;) Some of these systems can effectively be considered plugins. Consider OpenGL/DirectX? DirectSound/XAudio? Linux has a million back-end API's to choose from. I can see why many people feel it's natural to design their API's/systems that way, right or wrong. I don't agree with it personally, I would write it differently, but I'll never win that argument. Tech/systems programmers are vastly outnumbered in most studios. And of course, most programmers are junior-mid experience, that's just commercial reality.
Personally, I never understood why portability must pass by a plugin architecture. In our game engine we use macro to build the right implementation depending on the target platform and pure interfaces to be sure that API are respected. It's not an issue, because the user don't have to be able to choose the sound backend, and developer can do it for testing but it requires a full rebuild (a real issue due to the C++ slow compilation).
Jun 08 2013
parent reply Manu <turkeyman gmail.com> writes:
On 8 June 2013 21:27, Flamaros <flamaros.xavier gmail.com> wrote:

 On Saturday, 8 June 2013 at 03:59:25 UTC, Manu wrote:

 On 8 June 2013 13:25, Walter Bright <newshound2 digitalmars.com> wrote:

  On 6/7/2013 5:15 PM, Manu wrote:
  I can tell you that in my case, we export a lot(/most) things. Renderer
 api,
 sound api, etc are often all in their own libraries. So none of them are
 ever
 eligible for optimisation.
I'm curious why these exports all must be classes, and cannot be structs?
Because they embody functionality, not just data. That's just how many many programmers write code. Go to university for a couple of years, see what they tell you... ;) Some of these systems can effectively be considered plugins. Consider OpenGL/DirectX? DirectSound/XAudio? Linux has a million back-end API's to choose from. I can see why many people feel it's natural to design their API's/systems that way, right or wrong. I don't agree with it personally, I would write it differently, but I'll never win that argument. Tech/systems programmers are vastly outnumbered in most studios. And of course, most programmers are junior-mid experience, that's just commercial reality.
Personally, I never understood why portability must pass by a plugin architecture. In our game engine we use macro to build the right implementation depending on the target platform and pure interfaces to be sure that API are respected. It's not an issue, because the user don't have to be able to choose the sound backend, and developer can do it for testing but it requires a full rebuild (a real issue due to the C++ slow compilation).
But you are aware that people write code that way? I'm trying to represent the interests of an industry, not just myself. Anyone who rejects D because they perceive it doesn't meet their needs is a lost opportunity. Here's some example game-dev type API's. http://www.ogre3d.org/docs/api/html/namespaceOgre.html http://www.continuousphysics.com/Bullet/BulletFull/annotated.html These aren't particularly aggressive libs, but they are public, so you can see the sort of API that's typical as an example. Count the occurrences of 'virtual'... Note, the accessors almost never declare virtual, unless they are low-frequency classes, like 'Compositor', which is mostly virtual (gets hit once per frame). The ones I'm most familiar with are naturally closed-source or proprietary. For instance, Havok, FMod, Morpheme, Unreal, etc... All C++, with careful application of 'virtual'. Microsoft API's (DirectX, etc) are COM, which implies classes. People use classes. Classes are not optional. I'm not looking for workarounds or alternative solutions. If we can't supply classes that people can use in more-or-less the same way they have been doing so for decades, they probably won't choose D. If D does offer some superior alternative paradigms, given time, they might experiment and become comfortable with them eventually (prove to themselves it's solid and satisfies all their requirements). But they won't just take your word for it, and certainly not without a wealth of other successful examples in the industry. Programmers that make the switch to D have already taken on enough risk in that first step alone. If they have to risk changing a decade of coding habits and conventions too (which involves retraining all the staff), how can any business ever realistically make the change? I'm here, and interested in D due mainly to it's immediate familiarity, as opposed to others like Rust for instance. I can actually envision a migration. It seems like it's possible... one step at a time. ...it's also compiled and binary compatible, which is obviously critical ;) without question)
Jun 08 2013
parent "Rob T" <alanb ucora.com> writes:
On Sunday, 9 June 2013 at 06:03:14 UTC, Manu wrote:
 I'm here, and interested in D due mainly to it's immediate 
 familiarity, as
 opposed to others like Rust for instance. I can actually 
 envision a
 migration. It seems like it's possible... one step at a time.
 ...it's also compiled and binary compatible, which is obviously 
 critical ;)

 that wagon
 without question)
I see your point, but there's a contradiction since I think a lot of the reason to move away from C/C++ to D is because D is actually different and does things in a more sane way. If you want to retain familiarity and 100% compatibility with C/C++, then just stick with C/C++. Yes D has attempted to remain sort of compatible with C.C++, so as to allow for a migration and to leverage the momentum, but that means that D is not truly standing on its own, and because of this, it has limited itself and deeply integrated some of the same bad things about C/C++ into D. Rather than pollute D for sake of compatibility with all of the bad mistakes of the past, we should find better interfacing strategies. For instance, maybe a smart linker can help. --rt
Jun 09 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/7/2013 8:59 PM, Manu wrote:
 Because they embody functionality, not just data. That's just how many many
 programmers write code.
 Go to university for a couple of years, see what they tell you... ;)
 Some of these systems can effectively be considered plugins. Consider
 OpenGL/DirectX? DirectSound/XAudio? Linux has a million back-end API's to
choose
 from.
 I can see why many people feel it's natural to design their API's/systems that
 way, right or wrong.
So they don't need to be classes at all. It's not about embodying functionality vs data. It's about having a value type vs a polymorphic ref type.
Jun 08 2013
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/08/2013 06:39 PM, Walter Bright wrote:
 So they don't need to be classes at all. It's not about embodying functionality
 vs data. It's about having a value type vs a polymorphic ref type.
I've had quite good experiences using template mixins to generate struct polymorphism (in practice it's more of a policy class way of operating). The main problem is that this tends to slow down compilation quite a bit, though less so with DMD than with the other 2 compilers.
Jun 08 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 9 June 2013 02:39, Walter Bright <newshound2 digitalmars.com> wrote:

 On 6/7/2013 8:59 PM, Manu wrote:

 Because they embody functionality, not just data. That's just how many
 many
 programmers write code.
 Go to university for a couple of years, see what they tell you... ;)
 Some of these systems can effectively be considered plugins. Consider
 OpenGL/DirectX? DirectSound/XAudio? Linux has a million back-end API's to
 choose
 from.
 I can see why many people feel it's natural to design their API's/systems
 that
 way, right or wrong.
So they don't need to be classes at all. It's not about embodying functionality vs data. It's about having a value type vs a polymorphic ref type.
Hey? I'm not sure what you're saying.. I just said that people do use classes. And that implies a ref type with some amount of polymorphism. These API's are reference based, and there is always a few virtuals. And these are just trivial examples, a large game unifies a lot of code from basically all fields of computing. Trust me, I wouldn't bother spending all this time making a noise about the performance characteristics of classes if we didn't want/use classes.
Jun 08 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 8 June 2013 at 00:16:05 UTC, Manu wrote:
 It's like you've missed most of my points though.

 So under this proposal, which *entails* said 'sufficiently 
 advanced
 optimiser/linker/etc', which doesn't exist. And even if it did, 
 what
 architectures are supported?
 Many architectures will most certainly never receive support, 
 and these are
 usually the architectures that need it most.
It seems that you missed some of my points. LTO work by dumping the IR in object file. Linker with LTO enabled will read the ir, and redo the codegen. So the LTO is done before the codegen, and it shouldn't be a problem for most architectures. I also recognize that architectures are not equal on that regard. My reasoning here is that ARM and X86 are roughly 90% of the CPU you'll find and they handle indirect branch nicely. It is a bigger issue for other architectures, but it is desirable that architecture that are less used don't clutter the language.
 Put that aside for a second, it's still flawed in a basic sense.

 Anything 'exported' still falls under precisely the same set of 
 problems as
 we have now. By exporting something, NOTHING can be 
 de-virtualised, even
 when used internally. And exported things DO get used 
 internally. So by
 exporting, you lose all benefit within your own code anyway.
 I can tell you that in my case, we export a lot(/most) things. 
 Renderer
 api, sound api, etc are often all in their own libraries. So 
 none of them
 are ever eligible for optimisation.
My point is that exported symbol require very careful crafting anyway. So caring about what is final or not is a concern. Additionally, it is possible to use th following pattern : export class Exported { } // All intern subclass inherit that one. class NotExported : Exported { } And manipulated the NotExported class internally whenever it is possible. It allow for very precise crafting of the exported API, which is also a benefit.
 Additionally, one of my central points is completely 
 un-satisfied, that is,
 3rd party libraries. To make use of them at all implies they 
 are exported,
 so there is never any possibility for optimisation there.
That is true. However, this would result in an opaque call anyway, so opportunity for optimization isn't that big. If this is a huge problem, the final keyword is here.
 So, basically none of my usages are satisfied by your proposal.
 Sorry.
I said that you would rant, that one is quite nicela satisfied.
Jun 07 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Saturday, 8 June 2013 at 06:33:48 UTC, deadalnix wrote:
 ...
Btw, are there any reasons why "export" can't be applied on per-method basis instead of whole classes?
Jun 08 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 8 June 2013 at 10:22:42 UTC, Dicebot wrote:
 On Saturday, 8 June 2013 at 06:33:48 UTC, deadalnix wrote:
 ...
Btw, are there any reasons why "export" can't be applied on per-method basis instead of whole classes?
For final method, no problem, but for virtual method, it is either all or nothing, as they share the same virtual table.
Jun 08 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Saturday, 8 June 2013 at 10:34:17 UTC, deadalnix wrote:
 On Saturday, 8 June 2013 at 10:22:42 UTC, Dicebot wrote:
 On Saturday, 8 June 2013 at 06:33:48 UTC, deadalnix wrote:
 ...
Btw, are there any reasons why "export" can't be applied on per-method basis instead of whole classes?
For final method, no problem, but for virtual method, it is either all or nothing, as they share the same virtual table.
I don't see an issue. Non-export virtual methods can be implicitly final if compiler/linker finds so. It does not matter if they are in virtual table if no virtual table look-up happens. Any micro-example to clarify?
Jun 08 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 8 June 2013 at 10:55:37 UTC, Dicebot wrote:
 On Saturday, 8 June 2013 at 10:34:17 UTC, deadalnix wrote:
 On Saturday, 8 June 2013 at 10:22:42 UTC, Dicebot wrote:
 On Saturday, 8 June 2013 at 06:33:48 UTC, deadalnix wrote:
 ...
Btw, are there any reasons why "export" can't be applied on per-method basis instead of whole classes?
For final method, no problem, but for virtual method, it is either all or nothing, as they share the same virtual table.
I don't see an issue. Non-export virtual methods can be implicitly final if compiler/linker finds so. It does not matter if they are in virtual table if no virtual table look-up happens. Any micro-example to clarify?
It does matter that all parties agree on the layout of the virtual table. Virtual method are never linked, it don't makeany sens to export them or not. Even if no lookup for a given function is present, it is still important, because its presence change the layout of the virtual table, and so the lookup of other virtual methods.
Jun 08 2013
prev sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 7 June 2013 at 15:23:49 UTC, deadalnix wrote:
 On Friday, 7 June 2013 at 07:31:52 UTC, Walter Bright wrote:
 deadalnix mentioned enforcing 'export' on classes exposed to 
 shared libraries, but then aren't we back to expecting user 
 annotation rather than doing things automatically?
Yes, let me explain more. The export problem isn't new, and you'll find 2 major approach : the UNIX approach of export everything, and the windows approach of requiring explicit export.
Beware of generalizations, last time I used Aix (2002), it used the same model as Windows, including export files. Not sure how it works nowadays. -- Paulo
Jun 11 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 5:57 PM, Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
Like a dove. Andrei
Jun 06 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 5:00 PM, Andrei Alexandrescu wrote:
 On 6/6/13 5:57 PM, Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
Like a dove.
Few companies care about performance like Facebook does. What performance analysis tools does FB use routinely and pervasively? In my experience with major software companies, using anything other than -O is quite rare. Use of a profiler? nevah hoppin. Yes, a maverick here and there will use them, but use is not pervasive and routine. Will our canonical D newbie coming from C++ or Java use such a tool? Nope. They'll run the compiler, if we're lucky they'll throw -O -release -inline -noboundscheck, then they'll give the thumbs up or down on performance. This thread is a classic example.
Jun 06 2013
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 18:19:28 Walter Bright wrote:
 In my experience with major software companies, using anything other than -O
 is quite rare. Use of a profiler? nevah hoppin. Yes, a maverick here and
 there will use them, but use is not pervasive and routine.
We've profiled our code where I work from time to time to sort out specific performance issues, but it's still quite rare, and it's really only a couple of people who do it. It's definitely not pervasive. - Jonathan M Davis
Jun 06 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 9:19 PM, Walter Bright wrote:
 On 6/6/2013 5:00 PM, Andrei Alexandrescu wrote:
 On 6/6/13 5:57 PM, Walter Bright wrote:
 On 6/6/2013 2:23 PM, Andrei Alexandrescu wrote:
 (The tool I'm envisioning
 would add final annotations or prompt the user to add them.)
Sorry, that's never going to fly.
Like a dove.
Few companies care about performance like Facebook does. What performance analysis tools does FB use routinely and pervasively?
We built our own, an entire complex system, and an entire team is working on it. I've used it for four experiments today, and am about to launch one more.
 In my experience with major software companies, using anything other
 than -O is quite rare. Use of a profiler? nevah hoppin. Yes, a maverick
 here and there will use them, but use is not pervasive and routine.

 Will our canonical D newbie coming from C++ or Java use such a tool?
 Nope. They'll run the compiler, if we're lucky they'll throw -O -release
 -inline -noboundscheck, then they'll give the thumbs up or down on
 performance. This thread is a classic example.
I disapprove of this attitude, it's not constructive and kills creativity. When I wrote rdmd (not that that's an example of a creative idea!), I was its sole user for some six months, and whenever I'd mention it you were just as skeptical of its utility. Very slowly word got by and now it's a quite popular tool. You were just as skeptical about dustmite, and that has also become popular. If we have a tool that just class hierarchy analysis on a project by just saying cha --in-place *.d then people will notice. I don't buy all that "humans aren't rational" stuff. Andrei
Jun 06 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 10:14 PM, Andrei Alexandrescu wrote:
 I don't buy all that "humans aren't rational" stuff.
I don't either, but I don't think that was my point. My point is that a language where -O delivers the performance is preferable to a language that you have to do careful annotations and run extra tools on to get it. We discussed a while back an experimental Java compiler where annotations were used to signify uniqueness, etc. The compiler was a technical success, but the researcher was not very successful in getting users to use the annotations. This is why I am pushing for D doing attribute inference as much as possible, rather than expecting people to annotate carefully. I believe we are much more likely to be successful with the former approach. I'd very much prefer an auto-finalize mechanical solution. BTW, dustmite and rdmd are great tools. I'm happy to have been proven wrong about them.
Jun 07 2013
next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Friday, 7 June 2013 at 07:42:58 UTC, Walter Bright wrote:
 My point is that a language where -O delivers the performance 
 is preferable to a language that you have to do careful 
 annotations and run extra tools on to get it.
Depends on target audience. Some want tight control over resulting binary. At one of my jobs full scale performance test where run on every added feature and bug-fix block - mostly because even minor change to alignment to cache lines could have resulted in an observable performance drop. But that is not mainstream attitude for sure.
Jun 07 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/7/13 3:42 AM, Walter Bright wrote:
 On 6/6/2013 10:14 PM, Andrei Alexandrescu wrote:
 I don't buy all that "humans aren't rational" stuff.
I don't either, but I don't think that was my point. My point is that a language where -O delivers the performance is preferable to a language that you have to do careful annotations and run extra tools on to get it. We discussed a while back an experimental Java compiler where annotations were used to signify uniqueness, etc. The compiler was a technical success, but the researcher was not very successful in getting users to use the annotations. This is why I am pushing for D doing attribute inference as much as possible, rather than expecting people to annotate carefully. I believe we are much more likely to be successful with the former approach. I'd very much prefer an auto-finalize mechanical solution. BTW, dustmite and rdmd are great tools. I'm happy to have been proven wrong about them.
The more I think of this the more I regress toward my initial opinion: just let it be. I understand the arguments involved and it is my opinion that the problem is overstated (it's just a default, not something that prevents people from doing what they want to do), the benefits are exaggerated (we're not going to see whopping performance improvements), and the costs are underplayed (let's not forget that many breaking changes look great _before_ the breakage). We need to think holistically. If improving performance of generated code is a goal of ours, then there are so many better, nonbreaking vehicles for that: the backend interface, the inliner, the GC, the standard library, the SIMD builtins and associated library code, and many many more low-hanging fruit of lesser disruption and better returns. Andrei
Jun 07 2013
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 7 June 2013 05:37, Walter Bright <newshound2 digitalmars.com> wrote:

 On 6/6/2013 12:31 PM, Andrei Alexandrescu wrote:

 I think class hierarchy analysis is very doable for whole D projects. You
 just
 pass the tool all files in the project and it does its thing.

 http://www.cs.ucla.edu/~**palsberg/tba/papers/dean-**
 grove-chambers-ecoop95.pdf<http://www.cs.ucla.edu/~palsberg/tba/papers/dean-grove-chambers-ecoop95.pdf>

 This would actually be a great GSoC-style project, distributable via
 tools/.
The trouble, as has been pointed out before, is shared libraries.
And the existence of 'sufficiently smart linker', and the fact that the platforms that suffer from this stuff way harder than x86 almost always have less mature compilers/optimisers/linkers. I just wouldn't ever place my faith in the future arrival of some sufficiently-smart-[tool]. You couldn't make a business investment on that illusive possibility.
Jun 06 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 6 June 2013 at 23:54:49 UTC, Manu wrote:
 The trouble, as has been pointed out before, is shared 
 libraries.
And the existence of 'sufficiently smart linker', and the fact that the platforms that suffer from this stuff way harder than x86 almost always have less mature compilers/optimisers/linkers. I just wouldn't ever place my faith in the future arrival of some sufficiently-smart-[tool]. You couldn't make a business investment on that illusive possibility.
GCC and LLVM have what it take to implement this kind of stuff and can do codegen for a large variety of plateforms. I think it's never gonna work with dmd, and I think this is why Walter and yourself are pushing that hard to break everybody's code.
Jun 06 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 5:44 PM, deadalnix wrote:
 GCC and LLVM have what it take to implement this kind of stuff
 and can do codegen for a large variety of plateforms. I think
 it's never gonna work with dmd, and I think this is why Walter
 and yourself are pushing that hard to break everybody's code.
Please explain how they deal with the shared library issue.
Jun 06 2013
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 7 June 2013 10:44, deadalnix <deadalnix gmail.com> wrote:

 On Thursday, 6 June 2013 at 23:54:49 UTC, Manu wrote:

 The trouble, as has been pointed out before, is shared libraries.

 And the existence of 'sufficiently smart linker', and the fact that the
 platforms that suffer from this stuff way harder than x86 almost always
 have less mature compilers/optimisers/linkers.
 I just wouldn't ever place my faith in the future arrival of some
 sufficiently-smart-[tool]. You couldn't make a business investment on that
 illusive possibility.
GCC and LLVM have what it take to implement this kind of stuff and can do codegen for a large variety of plateforms. I think it's never gonna work with dmd, and I think this is why Walter and yourself are pushing that hard to break everybody's code.
IIRC, GCC requires explicit support for LTO in the backend, which means minority architectures will probably never get support, and these are the ones that need it the most. Don't know about LLVM, but I'll bet again, the minority architectures will not have good support. And there's still the DLL case. You can't simply compare the relative cost of a DLL call and a virtual call (although a DLL call is still slightly less work than a virtual call). The real issue is though, that code within your program which IS subject to LTO still can't have a de-virtualisation optimisation applied anyway, since it can't know if a DLL might introduce a new derived class. The possibility of a DLL simply existing means such an optimisation can't be performed, even if it is possible. If D were a JITed language, I wouldn't make a fuss. But it's not, it's a compiled systems language, and it's the only realistic competitor to C++ I know of in the same space. All other JIT languages have alternatives, they live in a crowded space, and while D might be a compelling option in those spaces, in the compiled systems space, D pretty much stands alone, and would do well not to inhibit the needs of those users.
Jun 06 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 7 June 2013 at 01:32:17 UTC, Manu wrote:
 IIRC, GCC requires explicit support for LTO in the backend, 
 which means
 minority architectures will probably never get support, and 
 these are the
 ones that need it the most.
 Don't know about LLVM, but I'll bet again, the minority 
 architectures will
 not have good support.
LTO is done y dumping the IR in the object file. The codegen occurs after LTO if they are enabled.
 And there's still the DLL case.
 You can't simply compare the relative cost of a DLL call and a 
 virtual call
 (although a DLL call is still slightly less work than a virtual 
 call).
The only difference is the indirect branch. In both cases the compiler is blind, so it will be slow. You are right, the indirect branch is an additional cost. In the case of a static call, you can expect the call to be several time faster, when you'll only get a marginal improvement on shared object.
 The real issue is though, that code within your program which 
 IS subject to
 LTO still can't have a de-virtualisation optimisation applied 
 anyway, since
 it can't know if a DLL might introduce a new derived class.
 The possibility of a DLL simply existing means such an 
 optimisation can't
 be performed, even if it is possible.
This is why I'm proposing for the nth time that virtual is enforced by the compiler. And yes, this can be done by setting the visibility of class's typeid (so overriding a class that isn't exported is a link time error).
Jun 06 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 6:45 PM, deadalnix wrote:
 This is why I'm proposing for the nth time that virtual is
 enforced by the compiler. And yes, this can be done by setting
 the visibility of class's typeid (so overriding a class that
 isn't exported is a link time error).
The horse may be out of the barn on that one. People just want to throw a switch and put their code in a DLL (like in libphobos2.so).
Jun 06 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 4:54 PM, Manu wrote:
 And the existence of 'sufficiently smart linker', and the fact that the
 platforms that suffer from this stuff way harder than x86 almost always have
 less mature compilers/optimisers/linkers.
 I just wouldn't ever place my faith in the future arrival of some
 sufficiently-smart-[tool]. You couldn't make a business investment on that
 illusive possibility.
Linkers can't know what a plugin shared library is going to do.
Jun 06 2013
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 03:06, David Nadlinger <code klickverbot.at> wrote:

 On Monday, 3 June 2013 at 16:25:24 UTC, Manu wrote:

 You won't break every single method, they already went through that
 recently when override was made a requirement. [=E2=80=A6] A much

 smaller number than the breakage
 which was gladly accepted recently. [=E2=80=A6] how did the override
 change ever get accepted [=E2=80=A6]
It appears as if either you have a interesting definition of "recently", or you are deliberately misleading people by bringing up that point over and over again. According to http://dlang.org/changelog.**html<http://dlang.org/changelog=
.html>,
 omitting "override" produced a warning since D 2.004, which was released
 back in September 2007! Granted, it was only actually turned from a
 deprecation warning into an actual deprecation in 2.061 (if my memory
 serves me right), but it's mostly a flaw in the handling of that particul=
ar
 deprecation that it stayed at the first level for so long. The actual
 language change was made =E2=80=93 and user-visible =E2=80=93 almost six =
(!) years ago,
 which is a lot on the D time scale.
Ah, sorry, I didn't realise that. I only recall conversations arising when it actually became deprecated, and people had to change their code. You are also ignoring the fact that in contrast to requiring "override",
 there is no clean deprecation path for your proposal, at least as far as =
I
 can see: Omitting the keyword started out as a warning, and IIRC still is
 allowed when you enable deprecated features via the compiler switch. How
 would a similar process look for virtual-by-default? As far as am isolate=
d
 module with only a base class is concerned, this is not question of valid
 vs. invalid code, but a silent change in language semantics.
The exact same path is available if you want to apply it: 1. Introduce 'virtual', create a warning when override is used on a function not marked. Functions remain virtual-by-default for the time being= . 2. At some later time, deprecate it (collecting the stragglers who haven't updated their code yet). Functions remain virtual-by-default. 3. Pull the pin. Functions become final by default, and virtuals should have already been marked during the previous periods. I'm not sure I follow your point about an isolated base class. In the event you have an isolated base class, where no classes are derived from it, then it's not actually polymorphic, so why should it's methods be virtual? In the event that it's a base-class-in-waiting, then it's true that you'll notice a compile error at a later time when you do eventually derive from it, but that's not really a breaking change, and it's almost in line with my points about explicit consideration; each method that receives the virtual keyword would take a moments consideration from the author as to whether it's actually correct/safe to be overriding that method or not. Ie, the momentary compile error where you write 'virtual' gives you the opportunity for that thought.
From DConf I know that you are actually are a friendly, reasonable person,
 but in this discussion, you really come across as a narrow-minded zealot =
to
 me. So, please, let's focus on finding an actually practical solution!
But this is a practical solution. The only counter-proposal I've heard is Andrei's 'all methods use ufcs' idea, but I think that one would be a much harder sell to the community. I'm certainly not convinced. It's really quite trivial, it's orthogonal with usage of override, it's more efficient, less error prone (which I have demonstrated happen all the time), enhances interoperation with C++, and it should even improve code fragility too (ie, people possibly overriding methods that are unsafe to override where the author never considered the possibility). Is there a reason this change offends you enough to call me names? Or can you at least tell how I'm being narrow-minded? For example, if we had !pure/!nothrow/!final or something along the lines,
 just mandate that "final:" is put at the top of everything in your style
 guide (easily machine-enforceable too) =E2=80=93 problem solved?
That's not quite the case though. Even if I could retrain internal staff to start doing that everywhere, you've potentially blocked access to a whole bunch of libraries because library authors don't follow our style guide. We suffer this in C++ all the time (see my many rant's about unnecessarily spending my life re-inventing wheels). Anything to make subtle pushes that improve usability/portability of libraries can only be a good thing. Especially when library authors generally don't specifically consider all usage environments. The language can assist some some extent. It's also precisely the same amount of work to type 'virtual:' (and it's the lesser common case to want to), except taking that angle enables the advantages I mention above, and also tends to force people to give a moments consideration to their API design/intent wrt virtual. Remember going virtual is a one-way trip. It can never be undone, which makes it a terribly dangerous default state. And maybe it would even catch on in the whole D community and lead to a
 language change in D3 or a future iteration of the language.

 David
Jun 03 2013
next sibling parent "Declan" <oyscal 163.com> writes:
On Tuesday, 4 June 2013 at 00:19:39 UTC, Manu wrote:
 On 4 June 2013 03:06, David Nadlinger <code klickverbot.at> 
 wrote:

 On Monday, 3 June 2013 at 16:25:24 UTC, Manu wrote:

 You won't break every single method, they already went 
 through that
 recently when override was made a requirement. […] A much

 smaller number than the breakage
 which was gladly accepted recently. […] how did the override
 change ever get accepted […]
It appears as if either you have a interesting definition of "recently", or you are deliberately misleading people by bringing up that point over and over again. According to http://dlang.org/changelog.**html<http://dlang.org/changelog.html>, omitting "override" produced a warning since D 2.004, which was released back in September 2007! Granted, it was only actually turned from a deprecation warning into an actual deprecation in 2.061 (if my memory serves me right), but it's mostly a flaw in the handling of that particular deprecation that it stayed at the first level for so long. The actual language change was made – and user-visible – almost six (!) years ago, which is a lot on the D time scale.
Ah, sorry, I didn't realise that. I only recall conversations arising when it actually became deprecated, and people had to change their code. You are also ignoring the fact that in contrast to requiring "override",
 there is no clean deprecation path for your proposal, at least 
 as far as I
 can see: Omitting the keyword started out as a warning, and 
 IIRC still is
 allowed when you enable deprecated features via the compiler 
 switch. How
 would a similar process look for virtual-by-default? As far as 
 am isolated
 module with only a base class is concerned, this is not 
 question of valid
 vs. invalid code, but a silent change in language semantics.
The exact same path is available if you want to apply it: 1. Introduce 'virtual', create a warning when override is used on a function not marked. Functions remain virtual-by-default for the time being. 2. At some later time, deprecate it (collecting the stragglers who haven't updated their code yet). Functions remain virtual-by-default. 3. Pull the pin. Functions become final by default, and virtuals should have already been marked during the previous periods. I'm not sure I follow your point about an isolated base class. In the event you have an isolated base class, where no classes are derived from it, then it's not actually polymorphic, so why should it's methods be virtual? In the event that it's a base-class-in-waiting, then it's true that you'll notice a compile error at a later time when you do eventually derive from it, but that's not really a breaking change, and it's almost in line with my points about explicit consideration; each method that receives the virtual keyword would take a moments consideration from the author as to whether it's actually correct/safe to be overriding that method or not. Ie, the momentary compile error where you write 'virtual' gives you the opportunity for that thought.
From DConf I know that you are actually are a friendly, 
reasonable person,
 but in this discussion, you really come across as a 
 narrow-minded zealot to
 me. So, please, let's focus on finding an actually practical 
 solution!
But this is a practical solution. The only counter-proposal I've heard is Andrei's 'all methods use ufcs' idea, but I think that one would be a much harder sell to the community. I'm certainly not convinced. It's really quite trivial, it's orthogonal with usage of override, it's more efficient, less error prone (which I have demonstrated happen all the time), enhances interoperation with C++, and it should even improve code fragility too (ie, people possibly overriding methods that are unsafe to override where the author never considered the possibility). Is there a reason this change offends you enough to call me names? Or can you at least tell how I'm being narrow-minded? For example, if we had !pure/!nothrow/!final or something along the lines,
 just mandate that "final:" is put at the top of everything in 
 your style
 guide (easily machine-enforceable too) – problem solved?
That's not quite the case though. Even if I could retrain internal staff to start doing that everywhere, you've potentially blocked access to a whole bunch of libraries because library authors don't follow our style guide. We suffer this in C++ all the time (see my many rant's about unnecessarily spending my life re-inventing wheels). Anything to make subtle pushes that improve usability/portability of libraries can only be a good thing. Especially when library authors generally don't specifically consider all usage environments. The language can assist some some extent. It's also precisely the same amount of work to type 'virtual:' (and it's the lesser common case to want to), except taking that angle enables the advantages I mention above, and also tends to force people to give a moments consideration to their API design/intent wrt virtual. Remember going virtual is a one-way trip. It can never be undone, which makes it a terribly dangerous default state. And maybe it would even catch on in the whole D community and lead to a
 language change in D3 or a future iteration of the language.

 David
+1
Jun 03 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/13 8:19 PM, Manu wrote:
 But this is a practical solution.
It's not a solution because there's no problem. We're talking about a default, not about the ability or lack thereof to do something. So the "solution" does not _solve_ anything. Andrei
Jun 03 2013
parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 12:26, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/3/13 8:19 PM, Manu wrote:

 But this is a practical solution.
It's not a solution because there's no problem. We're talking about a default, not about the ability or lack thereof to do something. So the "solution" does not _solve_ anything.
Virtual is a significant performance problem, and x86 is BY FAR the most tolerant architecture wrt virtual. The fact that virtual is a one way trip, and it can not safely be revoked later and therefore a very dangerous choice as the default is a maintenance problem. The fact that I'm yet to witness a single programmer ever declare their final methods at the time of authoring is a problem. The fact that many useful libraries might become inaccessible to what I'm sure is not an insignificant niche of potential D users is a problem. And I argue the subjective opinion, that code can't possibly be correct if the author never considered how the API may be used outside his design premise, and can never test it.
Jun 03 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/4/13 12:13 AM, Manu wrote:
 The fact that virtual is a one way trip, and it can not safely be
 revoked later and therefore a very dangerous choice as the default is a
 maintenance problem.
Certainly you're omitting a good part of the setup, which I assume has to do with binary compatibility and prebuilt binaries. In other setups, final is the one-way trip by definition - it restricts potential flexibility.
 The fact that I'm yet to witness a single programmer ever declare their
 final methods at the time of authoring is a problem.
Too narrow a social circle? :o)
 The fact that many useful libraries might become inaccessible to what
 I'm sure is not an insignificant niche of potential D users is a problem.
Not getting this. I dare believe that a competent library designer would be able to choose which functions ought to be overridden and which oughtn't. The moment the issue gets raised, the way the default goes is irrelevant. (But maybe I'm just not getting this.)
 And I argue the subjective opinion, that code can't possibly be correct
 if the author never considered how the API may be used outside his
 design premise, and can never test it.
I think you are wrong in thinking traditional procedural testing methods should apply to OOP designs. I can see how that fails indeed. Andrei
Jun 03 2013
parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 14:23, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/4/13 12:13 AM, Manu wrote:

 The fact that virtual is a one way trip, and it can not safely be
 revoked later and therefore a very dangerous choice as the default is a
 maintenance problem.
Certainly you're omitting a good part of the setup, which I assume has to do with binary compatibility and prebuilt binaries. In other setups, final is the one-way trip by definition - it restricts potential flexibility.
I don't buy the flexibility argument as a plus. I think that's a mistake, but I granted that's a value judgement. The fact that I'm yet to witness a single programmer ever declare their
 final methods at the time of authoring is a problem.
Too narrow a social circle? :o)
Well let's consider Steven's example from a short while ago. He didn't write final anywhere, and at some later time, retro-actively introduced it because he realised it was a performance burden. without any complaints from customers. But it's a breaking change to the API no matter which way you slice it, and I suspect this will be the prevalent pattern. So it basically commits to a future of endless breaking changes when someone wants to tighten up the performance of their library, and typically only after it has had time in the wild to identify the problem. The fact that many useful libraries might become inaccessible to what
 I'm sure is not an insignificant niche of potential D users is a problem.
Not getting this. I dare believe that a competent library designer would be able to choose which functions ought to be overridden and which oughtn't. The moment the issue gets raised, the way the default goes is irrelevant. (But maybe I'm just not getting this.)
Situation: I have a closed source library I want to use. I test and find that it doesn't meet our requirements for some trivial matter like performance (super common, I assure you). The author is not responsive, possibly because it would be a potentially breaking change to all the other customers of the library, I've now wasted a month of production time in discussions in an already tight schedule, and I begin the process of re-inventing the wheel. I've spent 10 years repeating this pattern. It will still be present with final-by-default, but it will be MUCH WORSE with virtual-by-default. I don't want to step backwards on this front. Even with C++ final-by-default, we've had to avoid libraries because C++ developers can be virtual-tastic sticking it on everything. D will magnify this issue immensely with virtual-by-default. At least in C++, nobody ever writes virtual on trivial accessors. virtual accessors/properties will likely eliminate many more libraries on the spot for being used in high frequency situations. Again, refer to Steven's pattern. Methods will almost always be virtual in D (because the author didn't care), until someone flags the issue years later... and then can it realistically be changed? Is it too late? Conversely, if virtual needs to be added at a later time, there are no such nasty side effects. It is always safe. And I argue the subjective opinion, that code can't possibly be correct
 if the author never considered how the API may be used outside his
 design premise, and can never test it.
I think you are wrong in thinking traditional procedural testing methods should apply to OOP designs. I can see how that fails indeed.
Can you elaborate? And can you convince me that an author of a class that can be transformed/abused in any way that he may have never even considered, can realistically reason about how to design his class well without being explicit about virtuals? I've made the point before that the sorts of super-polymorphic classes that might have mostly-virtuals are foundational classes, written once and used many times. These are not the classes that programmers sitting at their desk are banging out day after day. This are not the common case. Such a carefully designed and engineered base class can afford a moment to type 'virtual:' at the top.
Jun 03 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 04:53:48 UTC, Manu wrote:
 Situation: I have a closed source library I want to use. I test 
 and find
 that it doesn't meet our requirements for some trivial matter 
 like
 performance (super common, I assure you).
 The author is not responsive, possibly because it would be a 
 potentially
 breaking change to all the other customers of the library, I've 
 now wasted
 a month of production time in discussions in an already tight 
 schedule, and
 I begin the process of re-inventing the wheel.
 I've spent 10 years repeating this pattern. It will still be 
 present with
 final-by-default, but it will be MUCH WORSE with 
 virtual-by-default. I
 don't want to step backwards on this front.

 Even with C++ final-by-default, we've had to avoid libraries 
 because C++
 developers can be virtual-tastic sticking it on everything.
 D will magnify this issue immensely with virtual-by-default. At 
 least in
 C++, nobody ever writes virtual on trivial accessors.
 virtual accessors/properties will likely eliminate many more 
 libraries on
 the spot for being used in high frequency situations.
Paragraph 2 destroy paragraph one.
 Again, refer to Steven's pattern. Methods will almost always be 
 virtual in
 D (because the author didn't care), until someone flags the 
 issue years
 later... and then can it realistically be changed? Is it too 
 late?
 Conversely, if virtual needs to be added at a later time, there 
 are no such
 nasty side effects. It is always safe.
The solution is crystal clear to me from the beginning. You must pay the price when you actually override a method, not when you have the opportunity to do so. You simply don't want to consider that option as it break your way of doing something currently unsupported (shared object), provide higher benefice, and do not break everybody else's code.
Jun 03 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/4/13 12:53 AM, Manu wrote:
 I don't buy the flexibility argument as a plus. I think that's a
 mistake, but I granted that's a value judgement.
Great.
 But it's a breaking change to the API no matter which way you slice it,
 and I suspect this will be the prevalent pattern.
 So it basically commits to a future of endless breaking changes when
 someone wants to tighten up the performance of their library, and
 typically only after it has had time in the wild to identify the problem.
You're framing the matter all wrongly. Changing a method from virtual to final breaks the code of people who chose to override it - i.e. EXACTLY those folks who found it useful to TAP into the FLEXIBILITY of the design. Do you understand how you are wrong about this particular little thing?
 Situation: I have a closed source library I want to use. I test and find
 that it doesn't meet our requirements for some trivial matter like
 performance (super common, I assure you).
 The author is not responsive, possibly because it would be a potentially
 breaking change to all the other customers of the library, I've now
 wasted a month of production time in discussions in an already tight
 schedule, and I begin the process of re-inventing the wheel.
 I've spent 10 years repeating this pattern. It will still be present
 with final-by-default, but it will be MUCH WORSE with
 virtual-by-default. I don't want to step backwards on this front.
Situation: I have a closed source library I want to use. I test and find that it doesn't meet our requirements for some trivial matter like the behavior of a few methods (super common, I assure you). The author is not responsive, possibly because it would be a potentially breaking change to all the other customers of the library, I've now wasted a month of production time in discussions in an already tight schedule, and I begin the process of re-inventing the wheel. I've spent 10 years repeating this pattern. It will still be present with virtual-by-default, but it will be MUCH WORSE with final-by-default. I don't want to step backwards on this front. Destroyed?
 Even with C++ final-by-default, we've had to avoid libraries because C++
 developers can be virtual-tastic sticking it on everything.
Oh, so now the default doesn't matter. The amount of self-destruction is high in this post.
 D will magnify this issue immensely with virtual-by-default.
It will also magnify the flexibility benefits.
 At least in
 C++, nobody ever writes virtual on trivial accessors.
 virtual accessors/properties will likely eliminate many more libraries
 on the spot for being used in high frequency situations.
I don't think a "high frequency situation" would use classes designed naively. Again, the kind of persona you are discussing are very weird chaps.
 Again, refer to Steven's pattern. Methods will almost always be virtual
 in D (because the author didn't care), until someone flags the issue
 years later... and then can it realistically be changed? Is it too late?
 Conversely, if virtual needs to be added at a later time, there are no
 such nasty side effects. It is always safe.
Again: - changing a method final -> overridable is nonbreaking. YOU ARE RIGHT HERE. - changing a method overridable -> final will break PRECISELY code that was finding that design choice USEFUL. YOU SEEM TO BE MISSING THIS.
         And I argue the subjective opinion, that code can't possibly be
         correct
         if the author never considered how the API may be used outside his
         design premise, and can never test it.


     I think you are wrong in thinking traditional procedural testing
     methods should apply to OOP designs. I can see how that fails indeed.


 Can you elaborate?
 And can you convince me that an author of a class that can be
 transformed/abused in any way that he may have never even considered,
 can realistically reason about how to design his class well without
 being explicit about virtuals?
I can try. You don't understand at least this aspect of OOP (honest affirmation, not intended to offend). If class A chooses to inherit class B, it shouldn't do so to reuse B, but to be reused by code that manipulates Bs. In a nutshell: "inherit not to reuse, but to be reused". I hope this link works: http://goo.gl/ntRrt (If all A wants is to reuse B, it just uses composition.) You should agree as a simple matter that there's no reasonable way one can design a software library that would be transformed, abused, and misused. Although class designers should definitely design to make good use easy and bad use difficult, they routinely are unable to predict all different ways in which clients would use the class, so designing with flexibility in mind is the safest route (unless concerns for performance overrides that). Your concern with performance overrides that for flexibility, and that's entirely fine. What I disagree with is that you believe what's best for everybody.
 I've made the point before that the sorts of super-polymorphic classes
 that might have mostly-virtuals are foundational classes, written once
 and used many times.
I don't know what a super-polymorphic class is, and google fails to list it: http://goo.gl/i53hS
 These are not the classes that programmers sitting at their desk are
 banging out day after day. This are not the common case. Such a
 carefully designed and engineered base class can afford a moment to type
 'virtual:' at the top.
I won't believe this just because you said it (inventing terminology in the process), it doesn't rhyme with my experience, so do you have any factual evidence to back that up? Andrei
Jun 03 2013
next sibling parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 15:22, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/4/13 12:53 AM, Manu wrote:

 I don't buy the flexibility argument as a plus. I think that's a
 mistake, but I granted that's a value judgement.
Great.
That doesn't mean it's wrong, just that there are other opinions. But it's a breaking change to the API no matter which way you slice it,
 and I suspect this will be the prevalent pattern.
 So it basically commits to a future of endless breaking changes when
 someone wants to tighten up the performance of their library, and
 typically only after it has had time in the wild to identify the problem.
You're framing the matter all wrongly. Changing a method from virtual to final breaks the code of people who chose to override it - i.e. EXACTLY those folks who found it useful to TAP into the FLEXIBILITY of the design. Do you understand how you are wrong about this particular little thing?
Well first, there's a very high probability the number of people in that group is precisely zero, but since you can't know the size of your audience, library dev's will almost always act conservatively on that matter. In the alternate universe, those folks that really want to extend the class in unexpected ways may need to contact the author and request the change. Unlike the situation where I need to do that (where it will probably be rejected), the author will either give them advice about a better solution, or will probably be happy to help and make the change, since it's not a breaking change, and there's no risk of collateral damage. There's a nice side-effect that comes from the inconvenience too, which is that the author now has more information from his customers about how his library is being used, and can factor that into future thought/design. Surely you can see this point right? Going virtual is a one-way change. Situation: I have a closed source library I want to use. I test and find
 that it doesn't meet our requirements for some trivial matter like
 performance (super common, I assure you).
 The author is not responsive, possibly because it would be a potentially
 breaking change to all the other customers of the library, I've now
 wasted a month of production time in discussions in an already tight
 schedule, and I begin the process of re-inventing the wheel.
 I've spent 10 years repeating this pattern. It will still be present
 with final-by-default, but it will be MUCH WORSE with
 virtual-by-default. I don't want to step backwards on this front.
Situation: I have a closed source library I want to use. I test and find that it doesn't meet our requirements for some trivial matter like the behavior of a few methods (super common, I assure you). The author is not responsive, possibly because it would be a potentially breaking change to all the other customers of the library, I've now wasted a month of production time in discussions in an already tight schedule, and I begin the process of re-inventing the wheel. I've spent 10 years repeating this pattern. It will still be present with virtual-by-default, but it will be MUCH WORSE with final-by-default. I don't want to step backwards on this front. Destroyed?
What? I don't really know what you're saying here, other than mocking me and trivialising the issue. This is a very real and long-term problem. Even with C++ final-by-default, we've had to avoid libraries because C++
 developers can be virtual-tastic sticking it on everything.
Oh, so now the default doesn't matter. The amount of self-destruction is high in this post.
No, you're twisting my words and subverting my point. I'm saying that virtual-by-default will _make the problem much worse_. It's already a problem enough. Where once there might be one or 2 important methods that can't be used inside a loop, now there's a situation where we can't even do 'thing.length', or 'entity.get', which appear completely benign, but they're virtual accessors. This has now extended the problem into the realm of the most trivial of loops, and the most basic of interactions with the class in question. The point of my comment is to demonstrate that it's a REAL problem that does happen, and under the virtual-by-default standard, it will become much worse. D will magnify this issue immensely with virtual-by-default.

 It will also magnify the flexibility benefits.
And this (dubious) point alone is compelling enough to negate everything I've presented? Tell me honestly, when was the last time you were working with a C++ class, and you wanted to override a method that the author didn't mark virtual? Has that ever happened to you? It's never happened to me in 15 years. So is there a real loss of flexibility, or just a theoretical one? At least in
 C++, nobody ever writes virtual on trivial accessors.
 virtual accessors/properties will likely eliminate many more libraries
 on the spot for being used in high frequency situations.
I don't think a "high frequency situation" would use classes designed naively. Again, the kind of persona you are discussing are very weird chaps.
No it wouldn't, but everyone needs to make use of 3rd party code. And even the internal code is prone to forgetfulness and mistakes, as I've said countless times. Which cost time and money to find and fix. Again, refer to Steven's pattern. Methods will almost always be virtual
 in D (because the author didn't care), until someone flags the issue
 years later... and then can it realistically be changed? Is it too late?
 Conversely, if virtual needs to be added at a later time, there are no
 such nasty side effects. It is always safe.
Again: - changing a method final -> overridable is nonbreaking. YOU ARE RIGHT HERE. - changing a method overridable -> final will break PRECISELY code that was finding that design choice USEFUL. YOU SEEM TO BE MISSING THIS.
No it won't break, it wouldn't be there in the first place, because the function wasn't virtual. I realise I've eliminated a (potentially dangerous) application of a class, but the author is more than welcome to use 'virtual:' if it's important to them. I also think that saying people might want to override something is purely theoretical, and I've certainly never encountered a problem of this sort in C++. In my opinion, C++ users often tend to over-use virtual if anything, and I expect that practise would continue unchanged. And you've missed (or at least not addressed) why I actually think this is positive. Again to repeat myself. I think this sort of code is highly more likely to be open source, it's also highly more likely to contain templates (in which case the source is available anyway), and in lieu of those points, it's also of some benefit for the user that wants to bend this object in an unexpected direction to have some contact with the author. The author will surely have some opinion on the new usage pattern, and will now know the library is being used in this previously unexpected way, and can consider that user-base in the future. Again, both are still possible. But which should be the DEFAULT? Which is a more dangerous default? And I argue the subjective opinion, that code can't possibly be
         correct
         if the author never considered how the API may be used outside his
         design premise, and can never test it.


     I think you are wrong in thinking traditional procedural testing
     methods should apply to OOP designs. I can see how that fails indeed.


 Can you elaborate?
 And can you convince me that an author of a class that can be
 transformed/abused in any way that he may have never even considered,
 can realistically reason about how to design his class well without
 being explicit about virtuals?
I can try. You don't understand at least this aspect of OOP (honest affirmation, not intended to offend). If class A chooses to inherit class B, it shouldn't do so to reuse B, but to be reused by code that manipulates Bs. In a nutshell: "inherit not to reuse, but to be reused". I hope this link works: http://goo.gl/ntRrt
I understand the scripture, but I don't buy it outright. In practise, people derive to 'reuse' just as often (or even more often) than they do to be reused. API's are often defined to be used by deriving and implementing some little thing. Java is probably the most guilty of this pattern I've ever seen, you typically need to derive a class to do something trivial like provide a delegate. I'm not suggesting it should be that way, just that it's often not that way in practise. And regardless, I don't see how the default virtual-ness interferes with the reuse of A in any way. Why do these principles require that EVERYTHING be virtual. (If all A wants is to reuse B, it just uses composition.)
 You should agree as a simple matter that there's no reasonable way one can
 design a software library that would be transformed, abused, and misused.
 Although class designers should definitely design to make good use easy and
 bad use difficult, they routinely are unable to predict all different ways
 in which clients would use the class, so designing with flexibility in mind
 is the safest route (unless concerns for performance overrides that). Your
 concern with performance overrides that for flexibility, and that's
 entirely fine. What I disagree with is that you believe what's best for
 everybody.
D usually has quite an obsession with correctness, how can it be safe to encourage use of classes in ways that it was never designed or considered for? Outside of the simplest of classes, I can't imagine any designer can consider all possibilities, they will have had a very specific usage pattern in mind. At best, your 'creative' application won't have been tested. As new usage scenario's develop, it's useful for the author to know about it, and consider it in future. But this isn't a rule, only a default (in this case, paranoid safety first, a typical pattern for D). A class that wants to offer the flexibility you desire can easily use 'virtual:', or if the author is sufficiently confident that any part can be safely extended, they're perfectly welcome to make everything virtual. There's no loss of possibility, just that the default would offer some more confidence that your usage of a given API is correct; you'll get a compile error if you use beyond the author's intent. I've made the point before that the sorts of super-polymorphic classes
 that might have mostly-virtuals are foundational classes, written once
 and used many times.
I don't know what a super-polymorphic class is, and google fails to list it: http://goo.gl/i53hS These are not the classes that programmers sitting at their desk are
 banging out day after day. This are not the common case. Such a
 carefully designed and engineered base class can afford a moment to type
 'virtual:' at the top.
I won't believe this just because you said it (inventing terminology in the process), it doesn't rhyme with my experience, so do you have any factual evidence to back that up?
It's very frustrating working with proprietary code, I can't paste a class diagram or anything, but I'm sure you've seen a class diagram before. You understand that classes have a many:1 relationship with their base class? So logically, for every 1 day spent writing a base, there are 'many' days working on specialisations. So which is the common case?
Jun 04 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/4/13 4:36 AM, Manu wrote:
[snip]

I've read this, thanks for answering. Unfortunately I need to retire 
from this thread - there's only so many hours in the day, and it seems 
we got to the point where all sides shout the same malarkey over and 
over again past one another.

It would be great if this thread results in a language improvement - a 
means to negate a storage class label inside a class.


Andrei
Jun 04 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 12:47:46 UTC, Andrei Alexandrescu 
wrote:
 It would be great if this thread results in a language 
 improvement - a means to negate a storage class label inside a 
 class.
Yes please ! I didn't felt that need specifically in cases of classes of final, but in many other context I wish this was possible. I propose ~storageclass to mimick the this/~this patner
Jun 04 2013
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 22:47, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/4/13 4:36 AM, Manu wrote:
 [snip]

 I've read this, thanks for answering. Unfortunately I need to retire from
 this thread - there's only so many hours in the day, and it seems we got to
 the point where all sides shout the same malarkey over and over again past
 one another.

 It would be great if this thread results in a language improvement - a
 means to negate a storage class label inside a class.
I think that's required anyway, separately to this discussion.
Jun 04 2013
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 4 June 2013 at 13:06:32 UTC, Manu wrote:
 I think that's required anyway, separately to this discussion.
+1
Jun 04 2013
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 4 June 2013 at 05:22:39 UTC, Andrei Alexandrescu 
wrote:
 Situation: I have a closed source library I want to use. I test 
 and find that it doesn't meet our requirements for some trivial 
 matter like the behavior of a few methods (super common, I 
 assure you).
 The author is not responsive, possibly because it would be a 
 potentially breaking change to all the other customers of the 
 library, I've now wasted a month of production time in 
 discussions in an already tight schedule, and I begin the 
 process of re-inventing the wheel.
 I've spent 10 years repeating this pattern. It will still be 
 present with virtual-by-default, but it will be MUCH WORSE with 
 final-by-default. I don't want to step backwards on this front.

 Destroyed?
I don't buy this. Overriding a method from a class in a closed source library is only a sane thing to do if the docs explicitly say you can. If the docs explicitly say you can, then one can assume that the author will have marked it virtual. This virtual-by-default flexibility only exists when you're working with classes that you understand the internals of. Consider these situations, assuming lazy but not completely incompetent library authors: Hidden source, virtual by default: You can override most things, but you're playing with fire unless you have a written promise that it's safe to do so. Open source, virtual by default: Once you understand the internals of a class, you can safely override whatever you want. You are exposed to breakage due to implementation detail, but documentation represents a promise of sorts. Hidden source, final by default: You can only override what the author allows you to. This will have at least some connection with what is safe to override. Open source, final by default: Once you understand the internals of a class, you can fork the library and add virtual on the methods you need to override that the author did not consider.* Basically, final-by-default is safer and faster, virtual-by-default is more convenient when working with open source libraries. * you might consider this an unacceptable extra burden, especially considering distribution problems. However, I would counter that if you're going to override a method that isn't explicitly intended to be, you are exposing yourself to breakage due to implementation detail and therefore it would be distributing your own version anyway.
Jun 04 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 12:29:10 UTC, John Colvin wrote:
 On Tuesday, 4 June 2013 at 05:22:39 UTC, Andrei Alexandrescu 
 wrote:
 Situation: I have a closed source library I want to use. I 
 test and find that it doesn't meet our requirements for some 
 trivial matter like the behavior of a few methods (super 
 common, I assure you).
 The author is not responsive, possibly because it would be a 
 potentially breaking change to all the other customers of the 
 library, I've now wasted a month of production time in 
 discussions in an already tight schedule, and I begin the 
 process of re-inventing the wheel.
 I've spent 10 years repeating this pattern. It will still be 
 present with virtual-by-default, but it will be MUCH WORSE 
 with final-by-default. I don't want to step backwards on this 
 front.

 Destroyed?
I don't buy this. Overriding a method from a class in a closed source library is only a sane thing to do if the docs explicitly say you can.
For what it's worth, I did it a countless number of time in software that is in production right now.
 This virtual-by-default flexibility only exists when you're 
 working with classes that you understand the internals of.
No you understand its usage.
 Basically, final-by-default is safer and faster, 
 virtual-by-default is more convenient when working with open 
 source libraries.
Once again the fast claim fail to address or even consider other technique that can be used to finalize methods.
Jun 04 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 4 June 2013 at 12:51:35 UTC, deadalnix wrote:
 On Tuesday, 4 June 2013 at 12:29:10 UTC, John Colvin wrote:
 On Tuesday, 4 June 2013 at 05:22:39 UTC, Andrei Alexandrescu 
 wrote:
 Situation: I have a closed source library I want to use. I 
 test and find that it doesn't meet our requirements for some 
 trivial matter like the behavior of a few methods (super 
 common, I assure you).
 The author is not responsive, possibly because it would be a 
 potentially breaking change to all the other customers of the 
 library, I've now wasted a month of production time in 
 discussions in an already tight schedule, and I begin the 
 process of re-inventing the wheel.
 I've spent 10 years repeating this pattern. It will still be 
 present with virtual-by-default, but it will be MUCH WORSE 
 with final-by-default. I don't want to step backwards on this 
 front.

 Destroyed?
I don't buy this. Overriding a method from a class in a closed source library is only a sane thing to do if the docs explicitly say you can.
For what it's worth, I did it a countless number of time in software that is in production right now.
What happens when the library author adds some critical book-keeping to a method that you're overriding?
 This virtual-by-default flexibility only exists when you're 
 working with classes that you understand the internals of.
No you understand its usage.
See my point above. you need to be certain that the exact behaviour of the original function is not in some way critical to the correctness of the class in general.
 Basically, final-by-default is safer and faster, 
 virtual-by-default is more convenient when working with open 
 source libraries.
Once again the fast claim fail to address or even consider other technique that can be used to finalize methods.
I agree it would be nice to follow another route on this. Final vs virtual defaults is probably an endless debate, sidestepping it completely with a clever finalizing technique would be ideal.
Jun 04 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 13:26:26 UTC, John Colvin wrote:
 What happens when the library author adds some critical 
 book-keeping to a method that you're overriding?
It shouldn't do so on public method, as the problem is the exact same as override. It the method is private, then the problem goes away. Finally, if the method is protected, it doesn't make sense.
 I agree it would be nice to follow another route on this. Final 
 vs virtual defaults is probably an endless debate, sidestepping 
 it completely with a clever finalizing technique would be ideal.
The information missing for the compiler at link time right now is the overridability of a method in a shared object. This can be solved for a lot of code by enforcing stringer semantic for extern.
Jun 04 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 04:13:57 UTC, Manu wrote:
 And I argue the subjective opinion, that code can't possibly be 
 correct if
 the author never considered how the API may be used outside his 
 design
 premise, and can never test it.
This very sentence show that you miss the point of OOP and Liskov substitution principle. To make the argument cleared, let's consider a lib with a class A. The whole lib uses A and must now know about subclasses of A. Not even A itself. As a consequence, A don't need to be tested for all kind of future possible override. If I, as a programmer, create a class B that extends A, it is my responsibility to ensure that my class really behave as an A. As a matter of fact, the lib don't know anything about B, and that is the whole point of OOP.
Jun 03 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-06-04 02:19, Manu wrote:

 That's not quite the case though. Even if I could retrain internal staff
 to start doing that everywhere
I have an idea that could potentially help you here. Have a look at my new thread: "Idea to verify virtual/final methods". http://forum.dlang.org/thread/kok86c$126l$1 digitalmars.com -- /Jacob Carlborg
Jun 04 2013
prev sibling next sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Tuesday, 4 June 2013 at 00:19:39 UTC, Manu wrote:

 But this is a practical solution. The only counter-proposal 
 I've heard is
 Andrei's 'all methods use ufcs' idea, but I think that one 
 would be a much
 harder sell to the community. I'm certainly not convinced.
It would be hard to sell for at least one reason - protected non-virtual methods are quite common: module a; class A { } // should be accessible to derived classes // and is not part of A's public interface private void foo(A this_); ---- module b; import a; class B : A { virtual void bar() { foo(); // How? } }
Jun 04 2013
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 4 June 2013 at 00:19:39 UTC, Manu wrote:
 Is there a reason this change offends you enough to call me 
 names? Or can
 you at least tell how I'm being narrow-minded?
It's not at all the change itself that irritates me. I would never resort to making ad hominem arguments in a technical discussion, at least not deliberately so. In fact, from what I remember from the various discussions at DConf, I think we have pretty much the same opinion regarding how hidden costs are a bit too pervasive in present-day D, respectively how it encourages an inherently wasteful style of coding in more places than necessary. I also agree that in the current class design, virtual-by-default is dangerous. If we were to go back to the drawing board, though, I'd be interested in exploring alternative directions in the design space, away from the Java-style OOP model altogether. What I disagree with is just the style of the discussion, for reasons that Andrei already summarized in a much more eloquent way than I could. Trying to Be Right on the Internet is one thing, but does not necessarily improve the situation at all. And even if they eventually do, such discussions tend to be a much bigger waste of time and energy for everybody involved than necessary, which is also the reason why I mostly ignored this thread so far. David
Jun 06 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 6 June 2013 at 15:33:19 UTC, David Nadlinger wrote:
 In fact, from what I remember from the various discussions at 
 DConf, I think we have pretty much the same opinion regarding 
 how hidden costs are a bit too pervasive in present-day D, 
 respectively how it encourages an inherently wasteful style of 
 coding in more places than necessary. I also agree that in the 
 current class design, virtual-by-default is dangerous. If we 
 were to go back to the drawing board, though, I'd be interested 
 in exploring alternative directions in the design space, away 
 from the Java-style OOP model altogether.
scala's is pretty much the definition of awesome on that one. If it had to be redone, I'd push in that direction.
Jun 06 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
06-Jun-2013 21:47, deadalnix пишет:
 On Thursday, 6 June 2013 at 15:33:19 UTC, David Nadlinger wrote:
 In fact, from what I remember from the various discussions at DConf, I
 think we have pretty much the same opinion regarding how hidden costs
 are a bit too pervasive in present-day D, respectively how it
 encourages an inherently wasteful style of coding in more places than
 necessary. I also agree that in the current class design,
 virtual-by-default is dangerous. If we were to go back to the drawing
 board, though, I'd be interested in exploring alternative directions
 in the design space, away from the Java-style OOP model altogether.
+1 Never liked current OOP scheme.
 scala's is pretty much the definition of awesome on that one. If it had
 to be redone, I'd push in that direction.
And another plus one, though I've only glimpsed over scala by reading Martin's book "Programming in Scala" and trying simple "scripts". It felt very nice though. -- Dmitry Olshansky
Jun 06 2013
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 03 Jun 2013 12:25:11 -0400, Manu <turkeyman gmail.com> wrote:

 You won't break every single method, they already went through that
 recently when override was made a requirement.
 It will only break the base declarations, which are far less numerous.
Coming off the sidelines: 1. I think in the general case, virtual by default is fine. In code that is not performance-critical, it's not a big deal to have virtual functions, and it's usually more useful to have them virtual. I've experienced plenty of times with C++ where I had to go back and 'virtualize' a function. Any time you change that, you must recompile everything, it's not a simple change. It's painful either way. To me, this is simply a matter of preference. I understand that it's difficult to go from virtual to final, but in practice, breakage happens rarely, and will be loud with the new override requirements. 2. I think your background may bias your opinions :) We aren't all working on making lightning fast bare-metal game code. 3. It sucks to have to finalize all but N methods. In other words, we need a virtual *keyword* to go back to virtual-land. Then, one can put final: at the top of the class declaration, and virtualize a few methods. This shouldn't be allowed for final classes though. My one real experience on this was with dcollections. I had not declared anything final, and I realized I was paying a performance penalty for it. I then made all the classes final, and nobody complained. -Steve
Jun 03 2013
parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 12:50, Steven Schveighoffer <schveiguy yahoo.com> wrote:

 On Mon, 03 Jun 2013 12:25:11 -0400, Manu <turkeyman gmail.com> wrote:

  You won't break every single method, they already went through that
 recently when override was made a requirement.
 It will only break the base declarations, which are far less numerous.
Coming off the sidelines: 1. I think in the general case, virtual by default is fine. In code that is not performance-critical, it's not a big deal to have virtual functions, and it's usually more useful to have them virtual. I've experienced plenty of times with C++ where I had to go back and 'virtualize' a function. Any time you change that, you must recompile everything, it's not a simple change. It's painful either way. To me, this is simply a matter of preference. I understand that it's difficult to go from virtual to final, but in practice, breakage happens rarely, and will be loud with the new override requirements.
I agree that in the general case, it's 'fine', but I still don't see how it's a significant advantage. I'm not sure what the loss is, but I can see clear benefits to being explicit from an API point of view about what is safe to override, and implicitly, how the API is intended to be used. Can you see my point about general correctness? How can a class be correct if everything can be overridden, but it wasn't designed for it, and certainly never been tested?
 2. I think your background may bias your opinions :)  We aren't all
 working on making lightning fast bare-metal game code.
Of course it does. But what I'm trying to do is show the relative merits of one default vs the other. I may be biased, but I feel I've presented a fair few advantages to final-by-default, and I still don't know what the advantages to virtual-by-default are, other than people who don't care about the matter feel it's an inconvenience to type 'virtual:'. But that inconvenience is going to be forced upon one party either way, so the choice needs to be based on relative merits.
 3. It sucks to have to finalize all but N methods.  In other words, we
 need a virtual *keyword* to go back to virtual-land.  Then, one can put
 final: at the top of the class declaration, and virtualize a few methods.
  This shouldn't be allowed for final classes though.
The thing that irks me about that is that most classes aren't base classes, and most methods are trivial accessors and properties... why cater to the minority case? It also doesn't really address the problem where programmers just won't do that. Libraries suffer, I'm still inventing wheels 10 years from now, and I'm wasting time tracking down slip ups. What are the relative losses to the if it were geared the other way? My one real experience on this was with dcollections. I had not declared
 anything final, and I realized I was paying a performance penalty for it.
  I then made all the classes final, and nobody complained.
The userbase of a library will grow with time. Andrei wants a million D users, that's a lot more opportunities to break peoples code and gather complaints. Surely it's best to consider these sorts of changes sooner than later? And where is the most likely source of those 1 million new users to migrate from? Java?
Jun 03 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 03 Jun 2013 23:39:25 -0400, Manu <turkeyman gmail.com> wrote:

 On 4 June 2013 12:50, Steven Schveighoffer <schveiguy yahoo.com> wrote:

 On Mon, 03 Jun 2013 12:25:11 -0400, Manu <turkeyman gmail.com> wrote:

  You won't break every single method, they already went through that
 recently when override was made a requirement.
 It will only break the base declarations, which are far less numerous.
Coming off the sidelines: 1. I think in the general case, virtual by default is fine. In code that is not performance-critical, it's not a big deal to have virtual functions, and it's usually more useful to have them virtual. I've experienced plenty of times with C++ where I had to go back and 'virtualize' a function. Any time you change that, you must recompile everything, it's not a simple change. It's painful either way. To me, this is simply a matter of preference. I understand that it's difficult to go from virtual to final, but in practice, breakage happens rarely, and will be loud with the new override requirements.
I agree that in the general case, it's 'fine', but I still don't see how it's a significant advantage. I'm not sure what the loss is, but I can see clear benefits to being explicit from an API point of view about what is safe to override, and implicitly, how the API is intended to be used. Can you see my point about general correctness? How can a class be correct if everything can be overridden, but it wasn't designed for it, and certainly never been tested?
Since when is that on the base class author? Doctor, I overrode this class, and it doesn't work. Well, then don't override it :) Also there is the possibility that a class that isn't designed from the start to be overridden. But overriding one or two methods works, and has no adverse effects. Then it is a happy accident. And it even enables designs that take advantage of this default, like mock objects. I would point out that in Objective-C, ALL methods are virtual, even class methods and properties. It seems to work fine there. What I'm really trying to say is, when final is the default, and you really should have made some method virtual (but didn't), then you have to pay for it later when you update the base class. When virtual is the default, and you really wanted it to be final (but didn't do that), then you have to pay for it later when you update the base class. There is no way that is advantageous to *everyone*.
 2. I think your background may bias your opinions :)  We aren't all
 working on making lightning fast bare-metal game code.
Of course it does. But what I'm trying to do is show the relative merits of one default vs the other. I may be biased, but I feel I've presented a fair few advantages to final-by-default, and I still don't know what the advantages to virtual-by-default are, other than people who don't care about the matter feel it's an inconvenience to type 'virtual:'. But that inconvenience is going to be forced upon one party either way, so the choice needs to be based on relative merits.
It's advantageous to a particular style of coding. If you know everything is virtual by default, then you write code expecting that. Like mock objects. Or extending a class simply to change one method, even when you weren't expecting that to be part of the design originally. I look at making methods final specifically for optimization. It doesn't occur to me that the fact that it's overridable is a "leak" in the API, it's at your own peril if you want to extend a class that I didn't intend to be extendable. Like changing/upgrading engine parts in a car.
 3. It sucks to have to finalize all but N methods.  In other words, we
 need a virtual *keyword* to go back to virtual-land.  Then, one can put
 final: at the top of the class declaration, and virtualize a few  
 methods.
  This shouldn't be allowed for final classes though.
The thing that irks me about that is that most classes aren't base classes, and most methods are trivial accessors and properties... why cater to the minority case?
I think it is unfair to say most classes are not base classes. This would mean most classes are marked as final. I don't think they are. One of the main reasons to use classes in the first place is for extendability. Essentially, making virtual the default enables the *extender* to determine whether it's a good base class, when the original author doesn't care. I think classes fall into 3 categories: 1. Declared a base class (abstract) 2. Declared NOT a base class (final) 3. Don't care. I'd say most classes fall in category 3. For that, I think having virtual by default isn't a hindrance, it's simply giving the most flexibility to the user.
 It also doesn't really address the problem where programmers just won't  
 do
 that. Libraries suffer, I'm still inventing wheels 10 years from now, and
 I'm wasting time tracking down slip ups.
 What are the relative losses to the if it were geared the other way?
The losses are that if category 3 were simply always final, some other anti-Manu who wanted to extend everything has to contact all the original authors to get them to change their classes to virtual :) BTW, did you know you can extend a base class and simply make the extension final, and now all the methods on that derived class become non-virtual calls? Much easier to do than making the original base virtual (Note I haven't tested this to verify, but if not, it should be changed in the compiler).
 My one real experience on this was with dcollections.  I had not declared
 anything final, and I realized I was paying a performance penalty for  
 it.
  I then made all the classes final, and nobody complained.
The userbase of a library will grow with time. Andrei wants a million D users, that's a lot more opportunities to break peoples code and gather complaints. Surely it's best to consider these sorts of changes sooner than later?
I think it vastly depends on the intent of the code. If your classes simply don't lend themselves to extending, then making them final is a non-issue.
 And where is the most likely source of those 1 million new users to  
 migrate
 from? Java?
From all over the place, I would say. D seems to be an island of misfit programmers. -Steve
Jun 03 2013
next sibling parent reply "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Tue, 04 Jun 2013 06:16:45 +0200, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 I think it is unfair to say most classes are not base classes.  This  
 would mean most classes are marked as final.  I don't think they are.   
 One of the main reasons to use classes in the first place is for  
 extendability.
This is false. Consider this hierarchy: A->B->C, where x->y means 'x derives from y'. There is only one base class (A), and only one class that may be marked final (C). This will often be the case.
 BTW, did you know you can extend a base class and simply make the  
 extension final, and now all the methods on that derived class become  
 non-virtual calls?  Much easier to do than making the original base  
 virtual (Note I haven't tested this to verify, but if not, it should be  
 changed in the compiler).
This does however not help one iota when you have a reference to a base class. This will also often be the case. -- Simen
Jun 03 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/4/13 1:05 AM, Simen Kjaeraas wrote:
 On Tue, 04 Jun 2013 06:16:45 +0200, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:

 I think it is unfair to say most classes are not base classes. This
 would mean most classes are marked as final. I don't think they are.
 One of the main reasons to use classes in the first place is for
 extendability.
This is false. Consider this hierarchy: A->B->C, where x->y means 'x derives from y'. There is only one base class (A), and only one class that may be marked final (C). This will often be the case.
You two are in violent agreement. (B is also a base class, in addition to being a derived class.) Andrei
Jun 03 2013
parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Tue, 04 Jun 2013 07:23:47 +0200, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/4/13 1:05 AM, Simen Kjaeraas wrote:
 On Tue, 04 Jun 2013 06:16:45 +0200, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:

 I think it is unfair to say most classes are not base classes. This
 would mean most classes are marked as final. I don't think they are.
 One of the main reasons to use classes in the first place is for
 extendability.
This is false. Consider this hierarchy: A->B->C, where x->y means 'x derives from y'. There is only one base class (A), and only one class that may be marked final (C). This will often be the case.
You two are in violent agreement. (B is also a base class, in addition to being a derived class.)
Yes and no. I suspected after posting that this argument would appear. I believe a degradation of jargon has taken place - Manu originally spoke of foundation classes - classes with many overridable methods. A in my example is one of these, and I believe that's what Manu meant when he said 'base class' in the above discussion. -- Simen
Jun 03 2013
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 04 Jun 2013 01:05:28 -0400, Simen Kjaeraas  
<simen.kjaras gmail.com> wrote:

 On Tue, 04 Jun 2013 06:16:45 +0200, Steven Schveighoffer  
 <schveiguy yahoo.com> wrote:

 I think it is unfair to say most classes are not base classes.  This  
 would mean most classes are marked as final.  I don't think they are.   
 One of the main reasons to use classes in the first place is for  
 extendability.
This is false. Consider this hierarchy: A->B->C, where x->y means 'x derives from y'. There is only one base class (A), and only one class that may be marked final (C). This will often be the case.
I think you mean the other way around. x->y means 'y derives from x'. But I get your point. However, it's an invalid point. By this logic there is exactly one base class, Object. I think it's safe to say that way of thinking is not productive. Any class that is not final can be a base class. The classes it derives from are not relevant (including Object).
 BTW, did you know you can extend a base class and simply make the  
 extension final, and now all the methods on that derived class become  
 non-virtual calls?  Much easier to do than making the original base  
 virtual (Note I haven't tested this to verify, but if not, it should be  
 changed in the compiler).
This does however not help one iota when you have a reference to a base class. This will also often be the case.
I believe this is a red herring. If you are not in control of the creation of the object, the system may actually REQUIRE virtuality, since the base pointer might actually be to a derived type. -Steve
Jun 04 2013
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 14:16, Steven Schveighoffer <schveiguy yahoo.com> wrote:

 On Mon, 03 Jun 2013 23:39:25 -0400, Manu <turkeyman gmail.com> wrote:

  On 4 June 2013 12:50, Steven Schveighoffer <schveiguy yahoo.com> wrote:
  On Mon, 03 Jun 2013 12:25:11 -0400, Manu <turkeyman gmail.com> wrote:
  You won't break every single method, they already went through that

 recently when override was made a requirement.
 It will only break the base declarations, which are far less numerous.
Coming off the sidelines: 1. I think in the general case, virtual by default is fine. In code that is not performance-critical, it's not a big deal to have virtual functions, and it's usually more useful to have them virtual. I've experienced plenty of times with C++ where I had to go back and 'virtualize' a function. Any time you change that, you must recompile everything, it's not a simple change. It's painful either way. To me, this is simply a matter of preference. I understand that it's difficult to go from virtual to final, but in practice, breakage happens rarely, and will be loud with the new override requirements.
I agree that in the general case, it's 'fine', but I still don't see how it's a significant advantage. I'm not sure what the loss is, but I can see clear benefits to being explicit from an API point of view about what is safe to override, and implicitly, how the API is intended to be used. Can you see my point about general correctness? How can a class be correct if everything can be overridden, but it wasn't designed for it, and certainly never been tested?
Since when is that on the base class author? Doctor, I overrode this class, and it doesn't work. Well, then don't override it :)
Because it wastes your time (and money). And perhaps it only fails/causes problems in edge cases, or obscure side effects, or in internal code that you have no ability to inspect/debug. You have no reason to believe you're doing anything wrong; you're using the API in a perfectly valid way... it just happens that it is wrong (the author never considered it), and it doesn't work. Also there is the possibility that a class that isn't designed from the
 start to be overridden.  But overriding one or two methods works, and has
 no adverse effects.  Then it is a happy accident.  And it even enables
 designs that take advantage of this default, like mock objects.  I would
 point out that in Objective-C, ALL methods are virtual, even class methods
 and properties.  It seems to work fine there.
Even apple profess that Obj-C is primarily useful for UI code, and they use C for tonnes of other stuff. UI code is extremely low frequency by definition. I can't click my mouse very fast ;) What I'm really trying to say is, when final is the default, and you really
 should have made some method virtual (but didn't), then you have to pay for
 it later when you update the base class.
I recognise this, but I don't think that's necessarily a bad thing. It forces you a moment of consideration wrt making the change, and if it will affect anything else. If it feels like a significant change, you'll treat it as such (which it is). Even though you do need to make the change, it's not a breaking change, and you don't risk any side effects.
 When virtual is the default, and you really wanted it to be final (but
 didn't do that), then you have to pay for it later when you update the base
 class.  There is no way that is advantageous to *everyone*.
But unlike the first situation, this is a breaking change. If you are not the only user of your library, then this can't be done safely. 2. I think your background may bias your opinions :) We aren't all
 working on making lightning fast bare-metal game code.
Of course it does. But what I'm trying to do is show the relative merits of one default vs the other. I may be biased, but I feel I've presented a fair few advantages to final-by-default, and I still don't know what the advantages to virtual-by-default are, other than people who don't care about the matter feel it's an inconvenience to type 'virtual:'. But that inconvenience is going to be forced upon one party either way, so the choice needs to be based on relative merits.
It's advantageous to a particular style of coding. If you know everything is virtual by default, then you write code expecting that. Like mock objects. Or extending a class simply to change one method, even when you weren't expecting that to be part of the design originally.
If you write code like that, then write 'virtual:', it doesn't hurt anyone else. The converse is not true. I look at making methods final specifically for optimization. It doesn't
 occur to me that the fact that it's overridable is a "leak" in the API,
 it's at your own peril if you want to extend a class that I didn't intend
 to be extendable.  Like changing/upgrading engine parts in a car.
Precisely, this highlights one of the key issues. Optimising has now become a dangerous breaking process. 3. It sucks to have to finalize all but N methods. In other words, we
 need a virtual *keyword* to go back to virtual-land.  Then, one can put
 final: at the top of the class declaration, and virtualize a few methods.
  This shouldn't be allowed for final classes though.
The thing that irks me about that is that most classes aren't base classes, and most methods are trivial accessors and properties... why cater to the minority case?
I think it is unfair to say most classes are not base classes. This would mean most classes are marked as final. I don't think they are. One of the main reasons to use classes in the first place is for extendability.
People rarely use the final keyword on classes, even though they could 90% of the time. Class hierarchies only typically extend to a certain useful extent, but people usually leave the option to go further anyway. And the deeper the average hierarchy, the more leaf's there are - and the less drastic this change seems in contrast. Essentially, making virtual the default enables the *extender* to determine
 whether it's a good base class, when the original author doesn't care.

 I think classes fall into 3 categories:

 1. Declared a base class (abstract)
 2. Declared NOT a base class (final)
 3. Don't care.

 I'd say most classes fall in category 3.  For that, I think having virtual
 by default isn't a hindrance, it's simply giving the most flexibility to
 the user.
Precisely, we're back again at the only real argument for virtual-by-default: it'll slightly annoy some people to type 'virtual', but that goes both ways. I don't think this supports one position or the other. It also doesn't really address the problem where programmers just won't do
 that. Libraries suffer, I'm still inventing wheels 10 years from now, and
 I'm wasting time tracking down slip ups.
 What are the relative losses to the if it were geared the other way?
The losses are that if category 3 were simply always final, some other anti-Manu who wanted to extend everything has to contact all the original authors to get them to change their classes to virtual :)
Fine, they'll probably be receptive since it's not a breaking change. Can you guess how much traction I have when I ask an author of a popular library to remove some 'virtual' keywords in C++ code? "Oh we can't really do that, it could break any other users!", so then we rewrite the library. Who has been more inconvenienced in this scenario? Additionally, if it's the sort of library that's so polymorphic as you suggest, then what are the chances it also uses a lot of templates, and therefore you have the source code... I think the type of library you describe has a MUCH higher probability of being open-source, or that you have the source available. BTW, did you know you can extend a base class and simply make the extension
 final, and now all the methods on that derived class become non-virtual
 calls?  Much easier to do than making the original base virtual (Note I
 haven't tested this to verify, but if not, it should be changed in the
 compiler).
One presumes that the library that defines the base class deals with its own base pointers internally, and as such, the functions that I may have finalised in my code will still be virtual in the place that it counts. My one real experience on this was with dcollections. I had not declared
 anything final, and I realized I was paying a performance penalty for it.
  I then made all the classes final, and nobody complained.
The userbase of a library will grow with time. Andrei wants a million D users, that's a lot more opportunities to break peoples code and gather complaints. Surely it's best to consider these sorts of changes sooner than later?
I think it vastly depends on the intent of the code. If your classes simply don't lend themselves to extending, then making them final is a non-issue. And where is the most likely source of those 1 million new users to
 migrate
 from? Java?
From all over the place, I would say. D seems to be an island of misfit programmers. -Steve
Jun 03 2013
next sibling parent reply "Rob T" <alanb ucora.com> writes:
Manu, I'm wondering that perhaps you should not be using classes 
at all. You can still create a similar overridable scheme for 
struct methods, and although it may not be as convenient, it will 
work. However a big failure point with stucts is the lack of 
inheritance.

Structs would IMO be far more useful if they had inheritance. 
Inheritence can be fully removed from the rest of polymorphism, 
so there's no reason why structs which are not polymorphic cannot 
inherit.

Actually I'd like to know why structs cannot inherit? I hate it 
when I end up creating classes when I have no other reason to 
create a class other than for the ability to inherit.

--rt
Jun 03 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 05:41:16 UTC, Rob T wrote:
 Manu, I'm wondering that perhaps you should not be using 
 classes at all. You can still create a similar overridable 
 scheme for struct methods, and although it may not be as 
 convenient, it will work. However a big failure point with 
 stucts is the lack of inheritance.

 Structs would IMO be far more useful if they had inheritance. 
 Inheritence can be fully removed from the rest of polymorphism, 
 so there's no reason why structs which are not polymorphic 
 cannot inherit.

 Actually I'd like to know why structs cannot inherit? I hate it 
 when I end up creating classes when I have no other reason to 
 create a class other than for the ability to inherit.
struct are value type. You can't know the size of a polymorphic type. So you'll have trouble sooner than you imagine. The best part of that issue is that no problem occur when you copy only partially a struct, but then you corrupt memory when you call a virtual function that rely on thoses data. You can crash at this point, but more likely, you'll just corrupt memory, and the program will fail in some totally unrelated part of the code, depending on compiler switchs, in a non reproducible way. I assure you this is the kind of problem you don't want to have.
Jun 03 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/4/13 1:41 AM, Rob T wrote:
 Structs would IMO be far more useful if they had inheritance.
We do offer subtyping via alias this. Andrei
Jun 03 2013
parent reply "Rob T" <alanb ucora.com> writes:
On Tuesday, 4 June 2013 at 05:58:30 UTC, Andrei Alexandrescu 
wrote:
 On 6/4/13 1:41 AM, Rob T wrote:
 Structs would IMO be far more useful if they had inheritance.
We do offer subtyping via alias this. Andrei
Yeah, I saw that method described in another thread. The technique is not even remotely obvious, but the major problem is that it's very limited. After you do one alias this, you can't use alias this again for other things. Maybe that'll eventually change, I don't know. It seems like a hack to me, I'd rather see real inheritance. On Tuesday, 4 June 2013 at 05:58:58 UTC, Jonathan M Davis wrote:
 How would it even work for a struct to inherit without 
 polymorphism? The whole
 point of inheritance is to make it so that you can create types 
 that can be
 used in place of another,
.... The other significant reason for inheritance is to reuse pre-built sub-components. I rarely use polymorphism, but I make a lot of use out of inheritance, so what happens is that I end up creating classes when all I really need is structs. I cannot be the only person doing this either, and I suspect its very common.
 Use composition, and if you want to be able to call members of 
 the inner
 struct on the outer struct as if they were members of the outer 
 struct, then
 use alias this or opDispatch to forward them to the inner 
 struct.
For simulating inheritance, yes, you probably can make use out of inner structs, but how to make it all work seamlessly is not obvious and using opDispatch to make it stick together is time consuming and error prone. On Tuesday, 4 June 2013 at 05:56:49 UTC, deadalnix wrote: ...
 struct are value type. You can't know the size of a polymorphic 
 type. So you'll have trouble sooner than you imagine.
That's not an issue if you cut out the polymorphism nonsense from the feature set, which means that for structs the size is always knowable. I see no reason why structs cannot inherit and unfortunate that D forbids it. I'd like to hear what Manu says about it, because from what I am reading between the lines is that he probably does not need to be using classes but cannot use structs because the are too limited - that's my guess, but I really don't know. For me, I'd use structs much more often except that they cannot inherit. --rt
Jun 04 2013
parent Manu <turkeyman gmail.com> writes:
On 5 June 2013 02:21, Rob T <alanb ucora.com> wrote:

 On Tuesday, 4 June 2013 at 05:58:30 UTC, Andrei Alexandrescu wrote:

 On 6/4/13 1:41 AM, Rob T wrote:

 Structs would IMO be far more useful if they had inheritance.
We do offer subtyping via alias this. Andrei
Yeah, I saw that method described in another thread. The technique is not even remotely obvious, but the major problem is that it's very limited. After you do one alias this, you can't use alias this again for other things. Maybe that'll eventually change, I don't know. It seems like a hack to me, I'd rather see real inheritance. On Tuesday, 4 June 2013 at 05:58:58 UTC, Jonathan M Davis wrote:
 How would it even work for a struct to inherit without polymorphism? The
 whole
 point of inheritance is to make it so that you can create types that can
 be
 used in place of another,
.... The other significant reason for inheritance is to reuse pre-built sub-components. I rarely use polymorphism, but I make a lot of use out of inheritance, so what happens is that I end up creating classes when all I really need is structs. I cannot be the only person doing this either, and I suspect its very common. Use composition, and if you want to be able to call members of the inner
 struct on the outer struct as if they were members of the outer struct,
 then
 use alias this or opDispatch to forward them to the inner struct.
For simulating inheritance, yes, you probably can make use out of inner structs, but how to make it all work seamlessly is not obvious and using opDispatch to make it stick together is time consuming and error prone. On Tuesday, 4 June 2013 at 05:56:49 UTC, deadalnix wrote: ...
 struct are value type. You can't know the size of a polymorphic type. So
 you'll have trouble sooner than you imagine.
That's not an issue if you cut out the polymorphism nonsense from the feature set, which means that for structs the size is always knowable. I see no reason why structs cannot inherit and unfortunate that D forbids it. I'd like to hear what Manu says about it, because from what I am reading between the lines is that he probably does not need to be using classes but cannot use structs because the are too limited - that's my guess, but I really don't know. For me, I'd use structs much more often except that they cannot inherit.
I certainly have and do write shallow inheritance structures with no virtuals, it does occur from time to time, and I have missed struct inheritance in D, but alias this has met my needs so far. But I'd say the majority of classes are polymorphic. There's usually at least some sort of 'update()', or 'doWork()' function that needs to be virtual, but the vast majority of methods are trivial accessors throughout the hierarchy.
Jun 04 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, June 04, 2013 07:41:11 Rob T wrote:
 Manu, I'm wondering that perhaps you should not be using classes
 at all. You can still create a similar overridable scheme for
 struct methods, and although it may not be as convenient, it will
 work. However a big failure point with stucts is the lack of
 inheritance.
 
 Structs would IMO be far more useful if they had inheritance.
 Inheritence can be fully removed from the rest of polymorphism,
 so there's no reason why structs which are not polymorphic cannot
 inherit.
 
 Actually I'd like to know why structs cannot inherit? I hate it
 when I end up creating classes when I have no other reason to
 create a class other than for the ability to inherit.
How would it even work for a struct to inherit without polymorphism? The whole point of inheritance is to make it so that you can create types that can be used in place of another, and that won't work without references or pointers and polymorphism. Use composition, and if you want to be able to call members of the inner struct on the outer struct as if they were members of the outer struct, then use alias this or opDispatch to forward them to the inner struct. - Jonathan M Davis
Jun 03 2013
prev sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
On Tuesday, 4 June 2013 at 05:41:16 UTC, Rob T wrote:
 Structs would IMO be far more useful if they had inheritance. 
 Inheritence can be fully removed from the rest of polymorphism, 
 so there's no reason why structs which are not polymorphic 
 cannot inherit.
If no polymorphism is needed, there is no reason to use inheritance instead of template mixins.
Jun 04 2013
parent "Rob T" <alanb ucora.com> writes:
On Tuesday, 4 June 2013 at 07:33:04 UTC, Dicebot wrote:
 On Tuesday, 4 June 2013 at 05:41:16 UTC, Rob T wrote:
 Structs would IMO be far more useful if they had inheritance. 
 Inheritence can be fully removed from the rest of 
 polymorphism, so there's no reason why structs which are not 
 polymorphic cannot inherit.
If no polymorphism is needed, there is no reason to use inheritance instead of template mixins.
mixins make me shudder, however, if you can point out an example of this working for simulating struct inheritance, I'd be interested to have a look. Of course I strongly suspect that it will suffer from the same problems as the other suggested methods, not obvious and difficult to implement and maintain. --rt
Jun 04 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/4/13 1:16 AM, Manu wrote:
 But unlike the first situation, this is a breaking change. If you are
 not the only user of your library, then this can't be done safely.
Same fallacy all over again, for the third time in this thread. You keep on going about "breaking change" without recognizing that the now broken code was happily taking advantage of the very flexibility that you argue was useless and needed fixing. This is exactly the kind of argumentation that I disagree with, philosophically. Instead of carefully pondering the goods and the bads in a complex web of tradeoffs, it just clumsily gropes for the desired conclusion in ignorance of all that doesn't fit.
 If you write code like that, then write 'virtual:', it doesn't hurt
 anyone else. The converse is not true.
Fourth.
     I look at making methods final specifically for optimization.  It
     doesn't occur to me that the fact that it's overridable is a "leak"
     in the API, it's at your own peril if you want to extend a class
     that I didn't intend to be extendable.  Like changing/upgrading
     engine parts in a car.


 Precisely, this highlights one of the key issues. Optimising has now
 become a dangerous breaking process.
Fifth. [snip] Allow me to summarize my understanding of the most valuable parts your argument. * At the top level you believe ultimate efficiency should be default and OOP flexibility should be opt-in. * Classes routinely should make most methods final because it's hard to imagine why one would override all but a few. Since those are a minority, it's so much the better to make final the default. * Even the most performance-conscious people would not care to annotate classes and methods. It just doesn't happen. In contrast, people who want flexibility will annotate things for the simple reason they have to, otherwise overriding won't work. * You don't consider it a problem that one must go back to base classes and changing methods from final to overridable, whenever such a need arises. (It would be awesome if you had a similar list with the opposite arguments.) If the above is an accurate summary, I'd say it's a matter in which reasonable people might disagree. I take issue with each of the points above (not flat out disagree with each, more like amend and qualify etc). Unless fresh arguments, facts, or perspectives come about, I am personally not convinced, based on this thread so far, that we should operate a language change. Andrei
Jun 03 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 05:58:32 UTC, Andrei Alexandrescu 
wrote:
 Unless fresh arguments, facts, or perspectives come about, I am 
 personally not convinced, based on this thread so far, that we 
 should operate a language change.
export => finalization as LTO ?
Jun 03 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/4/13 2:04 AM, deadalnix wrote:
 On Tuesday, 4 June 2013 at 05:58:32 UTC, Andrei Alexandrescu wrote:
 Unless fresh arguments, facts, or perspectives come about, I am
 personally not convinced, based on this thread so far, that we should
 operate a language change.
export => finalization as LTO ?
I see some discussion about that at http://goo.gl/KIl8L, but am unclear on the exact idea. Is it some sort of class hierarchy analysis during link time? Andrei
Jun 03 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, June 04, 2013 02:10:40 Andrei Alexandrescu wrote:
 On 6/4/13 2:04 AM, deadalnix wrote:
 On Tuesday, 4 June 2013 at 05:58:32 UTC, Andrei Alexandrescu wrote:
 Unless fresh arguments, facts, or perspectives come about, I am
 personally not convinced, based on this thread so far, that we should
 operate a language change.
export => finalization as LTO ?
I see some discussion about that at http://goo.gl/KIl8L, but am unclear on the exact idea. Is it some sort of class hierarchy analysis during link time?
The idea is that any function which isn't overridden shouldn't be virtual. However, in the normal case, there's no way of knowing whether a class will be derived from or whether any particular function will be overidden. Code can be compiled separately such that the compiler has no clue what derived classes exist. However, in the case where you have a shared library and all symbols which will be used outside of the library must be exported, the linker could theoretically examine every function which is not exported and make it non- virtual if it's not being overridden, because no code outside of the shared library could possible override it. This will only help functions which aren't exported, and will only work on Windows, as no other platform currently requires that symbols be explicitly exported. It also requires that the linker do this, and as long as we're using the C linker, I don't know how it could. It would need to understand virtual functions (and possibly D virtual functions specifically) in order to make the optimization. - Jonathan M Davis
Jun 03 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 06:23:18 UTC, Jonathan M Davis wrote:
 The idea is that any function which isn't overridden shouldn't 
 be virtual.
 However, in the normal case, there's no way of knowing whether 
 a class will be
 derived from or whether any particular function will be 
 overidden. Code can be
 compiled separately such that the compiler has no clue what 
 derived classes
 exist.

 However, in the case where you have a shared library and all 
 symbols which
 will be used outside of the library must be exported, the 
 linker could
 theoretically examine every function which is not exported and 
 make it non-
 virtual if it's not being overridden, because no code outside 
 of the shared
 library could possible override it.
Yes.
 This will only help functions which aren't
 exported, and will only work on Windows, as no other platform 
 currently
 requires that symbols be explicitly exported.
It can't work on windows right now, as export's semantic is not strong enough (it do not apply to virtual methods). The proposal is to make export stringer (IE, makes it impossible to override class that aren't exported in a shared lib, on all systems. Today's linker support the functionnality.
 It also requires that the linker
 do this, and as long as we're using the C linker, I don't know 
 how it could.
linkers from GCC and LLVM toolchains allow for LTO. The concept is to dump the IR in the object file, so a compiler aware of it can run additional optimization at link time. I don't know how hard it is for GCC, but for LLVM, what needs to be done is for the frontend to dump the right metadata into the IR, so an optimizing pass can remove unneeded virtual calls. We can do it in a D specific way (using our own metadata and providing an optimization pas for LLVM) but most likely we won't even need to as the same feature is planned to be added to clang and we can most likely simply reuse clang's metadata.
Jun 03 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2013 11:49 PM, deadalnix wrote:
 We can do it in a D specific way (using our own metadata and providing an
 optimization pas for LLVM) but most likely we won't even need to as the same
 feature is planned to be added to clang and we can most likely simply reuse
 clang's metadata.
There is another way. D can be made aware that it is building an executable (after all, that is why it invokes the linker). If you shove all the source code into the compiler in one command, for an executable, functions that are not overridden can be made final.
Jun 04 2013
next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
On Tuesday, 4 June 2013 at 07:12:34 UTC, Walter Bright wrote:
 On 6/3/2013 11:49 PM, deadalnix wrote:
 We can do it in a D specific way (using our own metadata and 
 providing an
 optimization pas for LLVM) but most likely we won't even need 
 to as the same
 feature is planned to be added to clang and we can most likely 
 simply reuse
 clang's metadata.
There is another way. D can be made aware that it is building an executable (after all, that is why it invokes the linker). If you shove all the source code into the compiler in one command, for an executable, functions that are not overridden can be made final.
It has been discussed several times - no, they can't unless export is strict. One can create a shared library that inherits a class from main executable and get reference to it in main executable via reference to a known class (base).
Jun 04 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/4/2013 12:37 AM, Dicebot wrote:
 It has been discussed several times - no, they can't unless export is strict.
 One can create a shared library that inherits a class from main executable and
 get reference to it in main executable via reference to a known class (base).
Right, I had forgotten about that.
Jun 04 2013
prev sibling next sibling parent reply "Flamaros" <flamaros.xavier gmail.com> writes:
On Tuesday, 4 June 2013 at 07:12:34 UTC, Walter Bright wrote:
 On 6/3/2013 11:49 PM, deadalnix wrote:
 We can do it in a D specific way (using our own metadata and 
 providing an
 optimization pas for LLVM) but most likely we won't even need 
 to as the same
 feature is planned to be added to clang and we can most likely 
 simply reuse
 clang's metadata.
There is another way. D can be made aware that it is building an executable (after all, that is why it invokes the linker). If you shove all the source code into the compiler in one command, for an executable, functions that are not overridden can be made final.
I think is interesting, because all open source software can be build from sources and can also be done on some commercial products in certain conditions. And compiling the world with D is realistic, due to small compilation time. I also don't understand why compilers don't generate executable directly and use a linker, as they already know the binary format and do optimization. I case of DMD which take all source file in a raw, I don't see any issues. Do DMD do best inlining optimizations than the linker when it get all sources as parameter?
Jun 04 2013
parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 4 June 2013 at 07:39:19 UTC, Flamaros wrote:
 On Tuesday, 4 June 2013 at 07:12:34 UTC, Walter Bright wrote:
 On 6/3/2013 11:49 PM, deadalnix wrote:
 We can do it in a D specific way (using our own metadata and 
 providing an
 optimization pas for LLVM) but most likely we won't even need 
 to as the same
 feature is planned to be added to clang and we can most 
 likely simply reuse
 clang's metadata.
There is another way. D can be made aware that it is building an executable (after all, that is why it invokes the linker). If you shove all the source code into the compiler in one command, for an executable, functions that are not overridden can be made final.
I think is interesting, because all open source software can be build from sources and can also be done on some commercial products in certain conditions. And compiling the world with D is realistic, due to small compilation time. I also don't understand why compilers don't generate executable directly and use a linker, as they already know the binary format and do optimization. I case of DMD which take all source file in a raw, I don't see any issues. Do DMD do best inlining optimizations than the linker when it get all sources as parameter?
Because C language tooling still persists around us. In the Pascal family of languages, the linker is part of the compiler, like in Turbo Pascal, Delphi, Oberon, .... -- Paulo
Jun 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 June 2013 at 07:12:34 UTC, Walter Bright wrote:
 There is another way.

 D can be made aware that it is building an executable (after 
 all, that is why it invokes the linker). If you shove all the 
 source code into the compiler in one command, for an 
 executable, functions that are not overridden can be made final.
To do so, the compiler must know which function can be overriden in shared object (to not finalize them) and which can't. The whole thing can also be made a link time optimization, so separate compilation model isn't broken. It require strong (stronger that what we have now) enforcement of export.
Jun 04 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, June 04, 2013 00:12:30 Walter Bright wrote:
 On 6/3/2013 11:49 PM, deadalnix wrote:
 We can do it in a D specific way (using our own metadata and providing an
 optimization pas for LLVM) but most likely we won't even need to as the
 same feature is planned to be added to clang and we can most likely
 simply reuse clang's metadata.
There is another way. D can be made aware that it is building an executable (after all, that is why it invokes the linker). If you shove all the source code into the compiler in one command, for an executable, functions that are not overridden can be made final.
Shared libraries kill that - especially those that get loaded explicitly rather than linked to. They could have classes which derive from classes in the executable which are never derived from within the executable itself. - Jonathan M Davis
Jun 04 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-06-04 09:12, Walter Bright wrote:

 There is another way.

 D can be made aware that it is building an executable (after all, that
 is why it invokes the linker). If you shove all the source code into the
 compiler in one command, for an executable, functions that are not
 overridden can be made final.
Will that work with static libraries? -- /Jacob Carlborg
Jun 04 2013
prev sibling next sibling parent Sean Cavanaugh <WorksOnMyMachine gmail.com> writes:
On 6/4/2013 12:58 AM, Andrei Alexandrescu wrote:
 Unless fresh arguments, facts, or perspectives come about, I am
 personally not convinced, based on this thread so far, that we should
 operate a language change.
The best you could do without a language change is to establish some good conventions for class design: A) prefer writing UFCS functions B) make it the norm for classes to segment their functions a lot like the public/private setup of C++ classes: Have blocks for final, override, final override. Ideally there is a virtual keyword there like the others. At least in C++ land I can tell if a code library is useful to me by searching for how it uses virtuals, among other things like searching for catches and throws since I am stuck on platforms without exception handling and am working in a codebase that is designed without it. I also find the idea that the linker could help a bit iffy since once you start DLLEXPORT'ing your classes the dependency tree basically expands to your whole codebase and it quickly cascades into being unable to eliminate most functions and global variables from the final image. You end up with fat binaries and its hard to prune dead stuff, and it has to be done by hand by walking the map files for iffy-objects and tracking down their dependencies. This is especially true if nearly everything is virtual. I suppose in theory there is an alternative change which wouldn't require a new keyword, and that would be to make it so that methods are only virtual if they are defined in an interface, and forbidding them coming into existence at class scope. But this would increase the code maintenance probably a bit too much.
Jun 04 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2013 10:58 PM, Andrei Alexandrescu wrote:
 Unless fresh arguments, facts, or perspectives come about, I am personally not
 convinced, based on this thread so far, that we should operate a language
change.
One possibility is to introduce virtual as a storage class that overrides final. Hence, one could write a class like: class C { final: void foo(); void baz(); virtual int abc(); void def(); } This would not break any existing code, and Manu would just need to get into the habit of having "final:" as the first line in his classes.
Jun 04 2013
next sibling parent reply Sean Cavanaugh <WorksOnMyMachine gmail.com> writes:
On 6/4/2013 2:25 AM, Walter Bright wrote:
 One possibility is to introduce virtual as a storage class that
 overrides final. Hence, one could write a class like:

 class C {
    final:
      void foo();
      void baz();
      virtual int abc();
      void def();
 }

 This would not break any existing code, and Manu would just need to get
 into the habit of having "final:" as the first line in his classes.
The problem isn't going to be in your own code, it will be in using everyone elses.
Jun 04 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/4/2013 12:32 AM, Sean Cavanaugh wrote:
 The problem isn't going to be in your own code, it will be in using everyone
elses.
If you're forced to use someone else's code and are not allowed to change it in any way, then you're always going to have problems with badly written APIs. Even Manu mentioned that he's got problems with C++ libraries because of this, and C++ has non-virtual by default.
Jun 04 2013
next sibling parent Sean Cavanaugh <WorksOnMyMachine gmail.com> writes:
On 6/4/2013 2:46 AM, Walter Bright wrote:
 On 6/4/2013 12:32 AM, Sean Cavanaugh wrote:
 The problem isn't going to be in your own code, it will be in using
 everyone elses.
If you're forced to use someone else's code and are not allowed to change it in any way, then you're always going to have problems with badly written APIs. Even Manu mentioned that he's got problems with C++ libraries because of this, and C++ has non-virtual by default.
Changing third party libraries is a maintenance disaster, as they can rename files and make other changes that cause your local modifications to disappear into the ether after a merge. We have a good number of customizations to wxWidgets here, and I had to carefully port them all up to the current wx codebase because the old one wasn't safe to use in 64 bits on windows. Also, final-izing a third party library is going to be a pretty dangerous thing to do and likely introduce some serious bugs along the way.
Jun 04 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 4 June 2013 17:46, Walter Bright <newshound2 digitalmars.com> wrote:

 On 6/4/2013 12:32 AM, Sean Cavanaugh wrote:

 The problem isn't going to be in your own code, it will be in using
 everyone elses.
If you're forced to use someone else's code and are not allowed to change it in any way, then you're always going to have problems with badly written APIs. Even Manu mentioned that he's got problems with C++ libraries because of this, and C++ has non-virtual by default.
Indeed, I was just trying to illustrate that it is a real problem. Surely you can see how a language where accessors and properties are virtual by default will magnify the issue immensely right? Even the most trivial interactions with a class can't be used inside loops anymore.
Jun 04 2013
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, June 04, 2013 00:25:39 Walter Bright wrote:
 On 6/3/2013 10:58 PM, Andrei Alexandrescu wrote:
 Unless fresh arguments, facts, or perspectives come about, I am personally
 not convinced, based on this thread so far, that we should operate a
 language change.
One possibility is to introduce virtual as a storage class that overrides final. Hence, one could write a class like: class C { final: void foo(); void baz(); virtual int abc(); void def(); } This would not break any existing code, and Manu would just need to get into the habit of having "final:" as the first line in his classes.
That would be good regardless of whether virtual or non-virtual is the default. In general, the function attributes other than access level specifiers and safety attributes suffer from not being able to be undone once you use them with a colon or {}. - Jonathan M Davis
Jun 04 2013
next sibling parent reply "Zach the Mystic" <reachzach gggggmail.com> writes:
On Tuesday, 4 June 2013 at 07:39:04 UTC, Jonathan M Davis wrote:
 On Tuesday, June 04, 2013 00:25:39 Walter Bright wrote:
 On 6/3/2013 10:58 PM, Andrei Alexandrescu wrote:
 Unless fresh arguments, facts, or perspectives come about, I 
 am personally
 not convinced, based on this thread so far, that we should 
 operate a
 language change.
One possibility is to introduce virtual as a storage class that overrides final. Hence, one could write a class like: class C { final: void foo(); void baz(); virtual int abc(); void def(); } This would not break any existing code, and Manu would just need to get into the habit of having "final:" as the first line in his classes.
That would be good regardless of whether virtual or non-virtual is the default. In general, the function attributes other than access level specifiers and safety attributes suffer from not being able to be undone once you use them with a colon or {}. - Jonathan M Davis
Yeah, it's basically removing D's inherent bias against programmers concerned with performance as opposed to flexibility by allowing performance people such as Manu to structure their code however they want. The price is a keyword new to D but so common elsewhere it hardly seems noticeable as such.
Jun 04 2013
parent reply "Zach the Mystic" <reachzach gggggmail.com> writes:
On Tuesday, 4 June 2013 at 07:50:31 UTC, Zach the Mystic wrote:
 Yeah, it's basically removing D's inherent bias against 
 programmers concerned with performance as opposed to 
 flexibility by allowing performance people such as Manu to 
 structure their code however they want. The price is a keyword 
 new to D but so common elsewhere it hardly seems noticeable as 
 such.
Right?
Jun 04 2013
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, June 04, 2013 09:51:31 Zach the Mystic wrote:
 On Tuesday, 4 June 2013 at 07:50:31 UTC, Zach the Mystic wrote:
 Yeah, it's basically removing D's inherent bias against
 programmers concerned with performance as opposed to
 flexibility by allowing performance people such as Manu to
 structure their code however they want. The price is a keyword
 new to D but so common elsewhere it hardly seems noticeable as
 such.
Right?
Yes. That's essentially right. For some attributes, it's not currently possible to mark functions with them en masse while not having that attribute on some of the functions (this is particularly bad wtih the colon syntax, since it affects the rest of the file rather than just a specific scope like the braces do). Adding virtual as a keyword which would undo a final: or final {} on any functions that it's marked with would be useful. - Jonathan M Davis
Jun 04 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-06-04 09:38, Jonathan M Davis wrote:

 That would be good regardless of whether virtual or non-virtual is the
 default. In general, the function attributes other than access level specifiers
 and  safety attributes suffer from not being able to be undone once you use
 them with a colon or {}.
Something like !final would be a good idea. It can be generalized for all attributes. No new keywords need to be introduced. -- /Jacob Carlborg
Jun 04 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/4/13 3:25 AM, Walter Bright wrote:
 On 6/3/2013 10:58 PM, Andrei Alexandrescu wrote:
 Unless fresh arguments, facts, or perspectives come about, I am
 personally not
 convinced, based on this thread so far, that we should operate a
 language change.
One possibility is to introduce virtual as a storage class that overrides final. Hence, one could write a class like: class C { final: void foo(); void baz(); virtual int abc(); void def(); } This would not break any existing code, and Manu would just need to get into the habit of having "final:" as the first line in his classes.
This is generally good but I'd prefer a way to undo a storage class instead of introducing two keywords for each. Andrei
Jun 04 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-06-04 14:21, Andrei Alexandrescu wrote:

 This is generally good but I'd prefer a way to undo a storage class
 instead of introducing two keywords for each.
I like the idea of !final as someone suggested. That can be generalized for all attributes. -- /Jacob Carlborg
Jun 04 2013
prev sibling next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Tuesday, 4 June 2013 at 05:58:32 UTC, Andrei Alexandrescu 
wrote:
 On 6/4/13 1:16 AM, Manu wrote:
 But unlike the first situation, this is a breaking change. If 
 you are
 not the only user of your library, then this can't be done 
 safely.
Same fallacy all over again, for the third time in this thread. You keep on going about "breaking change" without recognizing that the now broken code was happily taking advantage of the very flexibility that you argue was useless and needed fixing.
I believe Manu's point is that the original flexibility was a mistake: the author of the library never intended for the method to be overridden; it was an accident of virtual-by-default. The fact that overriding methods on the class works for a client is just a coincidence, and could be dangerous. The problem occurs when, later, the author profiles and notices that the virtual methods are causing performance issues, or notices that the method should not have been virtual (perhaps he relies on the base behaviour's semantics). Marking it final now would be a breaking change for the client relying on the accidental virtual functions. If things were final by default, then you would have to opt-in to virtual, so it is unlikely that you will mark a method virtual by accident. There is no breakage changing a method from final to virtual. FWIW: I think making method final by default now would cause too much breakage, but I do agree with Manu that it would have been a better choice in the beginning.
Jun 04 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 4 June 2013 15:58, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/4/13 1:16 AM, Manu wrote:

 But unlike the first situation, this is a breaking change. If you are
 not the only user of your library, then this can't be done safely.
Same fallacy all over again, for the third time in this thread. You keep on going about "breaking change" without recognizing that the now broken code was happily taking advantage of the very flexibility that you argue was useless and needed fixing.
And the same fallacious response. The code you refer to wouldn't exist, because it wasn't possible to write, because what they have allegedly overridden isn't virtual. Nobody's code can break. They may have lost an opportunity to twist some code into an unexpected use case, but chances are, it wasn't their only possible solution, and possible that it might have even been dangerous. This is exactly the kind of argumentation that I disagree with,
 philosophically. Instead of carefully pondering the goods and the bads in a
 complex web of tradeoffs, it just clumsily gropes for the desired
 conclusion in ignorance of all that doesn't fit.
And this pisses me off, because you're implying that it's okay for you to disagree on some points in principle, but not for anyone else. I've clearly (and repeatedly) given my reasonings, and you haven't responded to many of them head-on. I don't agree the sacrifice is anywhere as significant as you suggest other than a breakage which which we can quantify, it's entirely theoretical for a start, whereas my points are all taken from a decade of experience that I don't want to see become worse in the future. I've also acknowledged that these are my opinions, but you can't say that I haven't put any thought into my position and I'm 'clumsily groping for desired conclusions' out of ignorance. That's just basically offensive. If I'm found to be wrong by the majority here, that will become evident soon enough. If you write code like that, then write 'virtual:', it doesn't hurt
 anyone else. The converse is not true.
Fourth. I look at making methods final specifically for optimization. It
     doesn't occur to me that the fact that it's overridable is a "leak"
     in the API, it's at your own peril if you want to extend a class
     that I didn't intend to be extendable.  Like changing/upgrading
     engine parts in a car.


 Precisely, this highlights one of the key issues. Optimising has now
 become a dangerous breaking process.
Fifth. [snip] Allow me to summarize my understanding of the most valuable parts your argument. * At the top level you believe ultimate efficiency should be default and OOP flexibility should be opt-in. * Classes routinely should make most methods final because it's hard to imagine why one would override all but a few. Since those are a minority, it's so much the better to make final the default. * Even the most performance-conscious people would not care to annotate classes and methods. It just doesn't happen. In contrast, people who want flexibility will annotate things for the simple reason they have to, otherwise overriding won't work. * You don't consider it a problem that one must go back to base classes and changing methods from final to overridable, whenever such a need arises. (It would be awesome if you had a similar list with the opposite arguments.) If the above is an accurate summary, I'd say it's a matter in which reasonable people might disagree. I take issue with each of the points above (not flat out disagree with each, more like amend and qualify etc). Unless fresh arguments, facts, or perspectives come about, I am personally not convinced, based on this thread so far, that we should operate a language change.
I'll summarise my arguments, though I've done this at least 3 times now. Sorry, I 'value' more of my points than you, so my summary is quite longer. These are all supporting reasons why I think it would be a good change, and naturally, some are of lower significance then than others. I'd like to think that most of them should be objectively rejected, or the counter arguments list grows in size a whole lot to justify the insults you offer: * At top level I believe D aspires to be a systems language, and performance should certainly be a key concern. - 'Flexibility' [at the expense of performance] should be opt-in. It comes at the expense of what I presume should be a core audience for a systems language. - x86 is the most tolerant architecture _by far_, and we're committed to a cost that isn't even known yet on the vast majority of computers in the world. * virtual is a one-way trip. It can't be undone without risking breaking code once released to the wild. How can that state be a sensible default? - Can not be un-done by the compiler/linker like it can in other (dynamic) languages. No sufficiently smart compiler can ever address this problem as an optimisation. * The result of said performance concern has a cost in time and money for at least one of D's core audiences (realtime systems programming). - I don't believe the converse case, final-by-default, would present any comparative loss for users that want to write 'virtual:' at the top of their class. - 'Opportunistic de-virtualisation' is a time consuming and tedious process, and tends to come up only during crunch times. * "Classes routinely should make most methods final because it's hard to imagine why one would override [the intended] few. Since those are a minority, it's so much the better to make final the default." - The majority of classes are leaf's, and there's no reason for leaf methods to be virtual by default. Likewise, most methods are trivial accessors (the most costly) which have no business being virtual either. - It's also self-documenting. It makes it clear to a customer how the API is to be used. * Libraries written in D should hope to be made available to the widest audience possible. - Library author's certainly don't consider (or care about) everyone's usage cases, but they often write useful code that many people want to make use of. This is the definition of a library. - They are almost certainly not going to annotate their classes with lots of 'final'. - Given hard experience, when asked to revoke virtual, even if authors agree in principle, they will refuse to do it given the risk of breakage for unknown customers. - Adding final as an optimisation is almost always done post-release, so it will almost always run the risk of breaking someones code somewhere. even when they know they should. Users from Java don't do it either, but mainly because they don't consider it important. - Note: 'the most performance conscious users' that you refer to are often not the ones writing the code. Programmers work in teams, sometimes those teams are large, and many programmers are inexperienced. * final-by-default promotes awareness of virtual-ness, and it's associated costs. - If it's hidden, it will soon be forgotten or dismissed as a trivial detail. It's not... at least, not in a systems language that attracts high-frequency programmers. * 'Flexibility' may actually be a fallacy anyway. I personally like the idea of requiring an explicit change to 'virtual' in the base when a new and untested usage pattern is to be exploited, it gives me confidence. - People are usually pretty permissive when marking functions virtual in C++, and people like to consider many possibilities. - When was the last time you wanted to override a function in C++, but the author didn't mark it virtual? Is there actually a reduction in flexibility in practise? Is this actually a frequent reality? - Overriding unintended functions may lead to dangerous behaviours never considered by the author in the first place. - How can I be confident in an API when I know the author couldn't have possibly tested all obscure possibilities available. And how can I know the extent of his consideration of usage scenarios when authoring the class? - At best, my obscure use case has never been tested. - 'virtual is self-documenting, succinctly communicating the authors design/intent. * Bonus: Improve interoperability with C++, which I will certainly appreciate, but this point manifested from the D-DFE guys at dconf. And I'll summarise my perception of the counter arguments argument: * It's a breaking change. * 'Flexibility'; someone somewhere might want to make use of a class in a creative way that wasn't intended or tested. They shouldn't be prohibited from this practise _by default_, in principle. - They would have to contact the author to request a method be made virtual in the unlikely event that source isn't available, and they want to use it in some obscure fashion that the author never considered. - Note: This point exists on both sides, but on this side, the author is likely to be accommodating to their requests. - Authors would have to write 'virtual:' if they want to offer this style of fully extensible class.
Jun 04 2013
next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
 - Can not be un-done by the compiler/linker like it can in other
 (dynamic) languages. No sufficiently smart compiler can ever 
 address this
 problem as an optimisation.
It can be done if you are fine marking every single class supposed to be used across the binary boundaries as "export". See deadalnix explanations in this thread for details.
Jun 04 2013
parent Manu <turkeyman gmail.com> writes:
On 4 June 2013 21:43, Dicebot <m.strashun gmail.com> wrote:

 - Can not be un-done by the compiler/linker like it can in other
 (dynamic) languages. No sufficiently smart compiler can ever address this
 problem as an optimisation.
It can be done if you are fine marking every single class supposed to be used across the binary boundaries as "export". See deadalnix explanations in this thread for details.
I don't see how it's an intuitive connection between 'export' and 'virtual-by-default'. And it doesn't address the accessor/property problem, which would often remain inlined even across a DLL. It also relies on non-standard/unexpected behaviour for .so's (which export everything right?).
Jun 04 2013
prev sibling next sibling parent reply Jerry <jlquinn optonline.net> writes:
+1
Jun 04 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, June 04, 2013 14:41:30 Jerry wrote:
 +1
Please always quote at least some of the post that you're replying to. Posts don't always thread properly on all clients, so it's not always obvious who someone is replying to if they don't quote anything. And some people don't use threading at all when they view posts. - Jonathan M Davis
Jun 04 2013
parent Jerry <jlquinn optonline.net> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> writes:

 On Tuesday, June 04, 2013 14:41:30 Jerry wrote:
 +1
Please always quote at least some of the post that you're replying to. Posts don't always thread properly on all clients, so it's not always obvious who someone is replying to if they don't quote anything. And some people don't use threading at all when they view posts.
Sorry about that. I was endorsing Manu's view of the virtual vs final class method by default argument. I also work with researchers and sloppy research code and have spend plenty of time fixing performance problems due to slow methods inside tight loops. In C++, it's more likely people not making simple calls inline, but I've also seen poor choice of virtual functions as well. Jerry
Jun 04 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Ok, Manu, you win, I'm pretty much convinced.
Jun 06 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 2:31 PM, Walter Bright wrote:
 Ok, Manu, you win, I'm pretty much convinced.
In my heart of hearts I somehow hope this will blow over and we'll get to some actually interesting stuff... Andrei
Jun 06 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 12:55 PM, Andrei Alexandrescu wrote:
 On 6/6/13 2:31 PM, Walter Bright wrote:
 Ok, Manu, you win, I'm pretty much convinced.
In my heart of hearts I somehow hope this will blow over and we'll get to some actually interesting stuff...
You could review my proposal on inferring immutability and uniqueness. Getting that working right will be kick ass!
Jun 06 2013
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 6/6/13, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 On 6/6/13 2:31 PM, Walter Bright wrote:
 Ok, Manu, you win, I'm pretty much convinced.
In my heart of hearts I somehow hope this will blow over and we'll get to some actually interesting stuff...
Which stuff? :)
Jun 06 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 8:01 PM, Andrej Mitrovic wrote:
 On 6/6/13, Andrei Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:
 On 6/6/13 2:31 PM, Walter Bright wrote:
 Ok, Manu, you win, I'm pretty much convinced.
In my heart of hearts I somehow hope this will blow over and we'll get to some actually interesting stuff...
Which stuff? :)
E.g. finalizing shared and threading. Andrei
Jun 06 2013
next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 7 June 2013 at 00:04:18 UTC, Andrei Alexandrescu wrote:
 On 6/6/13 8:01 PM, Andrej Mitrovic wrote:
 On 6/6/13, Andrei Alexandrescu<SeeWebsiteForEmail erdani.org>  
 wrote:
 In my heart of hearts I somehow hope this will blow over and 
 we'll get
 to some actually interesting stuff...
Which stuff? :)
E.g. finalizing shared and threading.
+1. By the way, was that a deliberate pun? If so, I bow to the master. ;) David
Jun 06 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/6/13 8:16 PM, David Nadlinger wrote:
 On Friday, 7 June 2013 at 00:04:18 UTC, Andrei Alexandrescu wrote:
 On 6/6/13 8:01 PM, Andrej Mitrovic wrote:
 On 6/6/13, Andrei Alexandrescu<SeeWebsiteForEmail erdani.org> wrote:
 In my heart of hearts I somehow hope this will blow over and we'll get
 to some actually interesting stuff...
Which stuff? :)
E.g. finalizing shared and threading.
+1. By the way, was that a deliberate pun? If so, I bow to the master. ;)
Wasn't, but I can't admit that now! Andrei
Jun 06 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 6/7/13, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 E.g. finalizing shared and threading.
Complex stuff.. But as far as I know people tend to avoid using shared (what does it to, except not work with Phobos most of the time?). And since we're moving into a direction of thread isolation, it seems to me like shared is something that should be removed from the language and replaced with a library type. Couldn't it be replaced with a library type? Disabling ++/-- is easy, we can write an opAssign, we could make opDispatch automatically return a Shared!(member), etc. I don't see why something that's both unsafe (sharing data) and slow should ever be directly supported by the language.
Jun 06 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 7 June 2013 10:04, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:

 On 6/6/13 8:01 PM, Andrej Mitrovic wrote:

 On 6/6/13, Andrei Alexandrescu<SeeWebsiteForEmai**l erdani.org<SeeWebsiteForEmail erdani.org>>
  wrote:

 On 6/6/13 2:31 PM, Walter Bright wrote:

  Ok, Manu, you win, I'm pretty much convinced.

 In my heart of hearts I somehow hope this will blow over and we'll get
 to some actually interesting stuff...
Which stuff? :)
E.g. finalizing shared and threading.
Looking forward to it! :)
Jun 06 2013
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 15:55:02 Andrei Alexandrescu wrote:
 On 6/6/13 2:31 PM, Walter Bright wrote:
 Ok, Manu, you win, I'm pretty much convinced.
In my heart of hearts I somehow hope this will blow over and we'll get to some actually interesting stuff...
Well, we could just officially decide that we're making member functions non- virtual by default and move on. It at least sounds like Walter is now sure that that's what we should do. - Jonathan M Davis
Jun 06 2013
prev sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Thursday, 6 June 2013 at 18:31:08 UTC, Walter Bright wrote:
 Ok, Manu, you win, I'm pretty much convinced.
So what does this mean? We got 'virtual' with the next release? :)
Jun 07 2013
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/04/2013 01:15 PM, Manu wrote:
 * virtual is a one-way trip. It can't be undone without risking breaking code
 once released to the wild. How can that state be a sensible default?
   - Can not be un-done by the compiler/linker like it can in other (dynamic)
 languages. No sufficiently smart compiler can ever address this problem as an
 optimisation.
Have to say that for me, this is a bit of a killer point. If a programmer mistakenly picks the default option instead of the desired qualifier, ideally you want the fix to be non-breaking. It's complicated by the fact that code might get broken _now_ while changing the default, but the question is whether the price is worth it for saving future pain.
Jun 04 2013
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 04 Jun 2013 07:32:31 -0400, Joseph Rushton Wakeling  
<joseph.wakeling webdrake.net> wrote:

 On 06/04/2013 01:15 PM, Manu wrote:
 * virtual is a one-way trip. It can't be undone without risking  
 breaking code
 once released to the wild. How can that state be a sensible default?
   - Can not be un-done by the compiler/linker like it can in other  
 (dynamic)
 languages. No sufficiently smart compiler can ever address this problem  
 as an
 optimisation.
Have to say that for me, this is a bit of a killer point. If a programmer mistakenly picks the default option instead of the desired qualifier, ideally you want the fix to be non-breaking.
Define non-breaking. If your code still compiles but all of a sudden becomes horrendously slow, is that a non-breaking change? -Steve
Jun 04 2013
prev sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On 2013-06-04, 13:15, Manu wrote:

 I'll summarise my arguments, though I've done this at least 3 times now.
 Sorry, I 'value' more of my points than you, so my summary is quite  
 longer.
 These are all supporting reasons why I think it would be a good change,  
 and
 naturally, some are of lower significance then than others.
 I'd like to think that most of them should be objectively rejected, or  
 the
 counter arguments list grows in size a whole lot to justify the insults  
 you
 offer:

 * At top level I believe D aspires to be a systems language, and
 performance should certainly be a key concern.
   - 'Flexibility' [at the expense of performance] should be opt-in. It
 comes at the expense of what I presume should be a core audience for a
 systems language.
   - x86 is the most tolerant architecture _by far_, and we're committed  
 to
 a cost that isn't even known yet on the vast majority of computers in the
 world.

 * virtual is a one-way trip. It can't be undone without risking breaking
 code once released to the wild. How can that state be a sensible default?
   - Can not be un-done by the compiler/linker like it can in other
 (dynamic) languages. No sufficiently smart compiler can ever address this
 problem as an optimisation.

 * The result of said performance concern has a cost in time and money for
 at least one of D's core audiences (realtime systems programming).
   - I don't believe the converse case, final-by-default, would present  
 any
 comparative loss for users that want to write 'virtual:' at the top of
 their class.
   - 'Opportunistic de-virtualisation' is a time consuming and tedious
 process, and tends to come up only during crunch times.

 * "Classes routinely should make most methods final because it's hard to
 imagine why one would override [the intended] few. Since those are a
 minority, it's so much the better to make final the default."
   - The majority of classes are leaf's, and there's no reason for leaf
 methods to be virtual by default. Likewise, most methods are trivial
 accessors (the most costly) which have no business being virtual either.
   - It's also self-documenting. It makes it clear to a customer how the  
 API
 is to be used.

 * Libraries written in D should hope to be made available to the widest
 audience possible.
   - Library author's certainly don't consider (or care about) everyone's
 usage cases, but they often write useful code that many people want to  
 make
 use of. This is the definition of a library.
   - They are almost certainly not going to annotate their classes with  
 lots
 of 'final'.
   - Given hard experience, when asked to revoke virtual, even if authors
 agree in principle, they will refuse to do it given the risk of breakage
 for unknown customers.
   - Adding final as an optimisation is almost always done post-release,  
 so
 it will almost always run the risk of breaking someones code somewhere.


 'final'
 even when they know they should. Users from Java don't do it either, but
 mainly because they don't consider it important.
   - Note: 'the most performance conscious users' that you refer to are
 often not the ones writing the code. Programmers work in teams, sometimes
 those teams are large, and many programmers are inexperienced.

 * final-by-default promotes awareness of virtual-ness, and it's  
 associated
 costs.
   - If it's hidden, it will soon be forgotten or dismissed as a trivial
 detail. It's not... at least, not in a systems language that attracts
 high-frequency programmers.

 * 'Flexibility' may actually be a fallacy anyway. I personally like the
 idea of requiring an explicit change to 'virtual' in the base when a new
 and untested usage pattern is to be exploited, it gives me confidence.
   - People are usually pretty permissive when marking functions virtual  
 in
 C++, and people like to consider many possibilities.
     - When was the last time you wanted to override a function in C++,  
 but
 the author didn't mark it virtual? Is there actually a reduction in
 flexibility in practise? Is this actually a frequent reality?
   - Overriding unintended functions may lead to dangerous behaviours  
 never
 considered by the author in the first place.
     - How can I be confident in an API when I know the author couldn't  
 have
 possibly tested all obscure possibilities available. And how can I know  
 the
 extent of his consideration of usage scenarios when authoring the class?
       - At best, my obscure use case has never been tested.
     - 'virtual is self-documenting, succinctly communicating the authors
 design/intent.

 * Bonus: Improve interoperability with C++, which I will certainly
 appreciate, but this point manifested from the D-DFE guys at dconf.


 And I'll summarise my perception of the counter arguments argument:

 * It's a breaking change.

 * 'Flexibility'; someone somewhere might want to make use of a class in a
 creative way that wasn't intended or tested. They shouldn't be prohibited
 from this practise _by default_, in principle.
   - They would have to contact the author to request a method be made
 virtual in the unlikely event that source isn't available, and they want  
 to
 use it in some obscure fashion that the author never considered.
     - Note: This point exists on both sides, but on this side, the author
 is likely to be accommodating to their requests.
   - Authors would have to write 'virtual:' if they want to offer this  
 style
 of fully extensible class.
For whatever it's worth, I have the same impression as you - virtual-by- default is one-way and unsafe, and its benefits seem questionable. I admit at first I thought it was a good idea - "great, the compiler'll figure it out for me!". But that's not the case, so I was forced to revise my opinion. -- Simen
Jun 04 2013
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 04 Jun 2013 01:16:22 -0400, Manu <turkeyman gmail.com> wrote:

 On 4 June 2013 14:16, Steven Schveighoffer <schveiguy yahoo.com> wrote:

 Since when is that on the base class author?  Doctor, I overrode this
 class, and it doesn't work.  Well, then don't override it :)
Because it wastes your time (and money). And perhaps it only fails/causes problems in edge cases, or obscure side effects, or in internal code that you have no ability to inspect/debug. You have no reason to believe you're doing anything wrong; you're using the API in a perfectly valid way... it just happens that it is wrong (the author never considered it), and it doesn't work.
Technically and narrow-mindedly, yes, it will not waste your time and money to try extending it -- you will know right up front that you can't use it via extension, and therefore cannot use the library if it doesn't fit exactly what you need. You will simply waste time and money re-implementing it. There is also a quite likely possibility that you have the source to the base class, in which case you can determine whether it's possible to extend. This view that you've taken is that if I can do something, then the library developer has expected that usage, simply by it being possible. This is a bad way to look at APIs. Documentation and intent are important to consider.
 Also there is the possibility that a class that isn't designed from the
 start to be overridden.  But overriding one or two methods works, and  
 has
 no adverse effects.  Then it is a happy accident.  And it even enables
 designs that take advantage of this default, like mock objects.  I would
 point out that in Objective-C, ALL methods are virtual, even class  
 methods
 and properties.  It seems to work fine there.
Even apple profess that Obj-C is primarily useful for UI code, and they use C for tonnes of other stuff.
First, I've never heard that statement or read it anywhere (you have a link?). Second, the idea that if you use Objective C objects for your API, then you must use method calls for EVERYTHING is ridiculous. Pretty much all the OS functionality is exposed via Objective-C objects. It doesn't mean the underlying implementation is pure objects, like wrapping ints in objects or something. I don't know of any language that would do that. The public API is all virtual, including networking, I/O, image processing, threading, etc. and it works quite well. C is a subset of Objective-C, so it's quite easy to switch back and forth.
 What I'm really trying to say is, when final is the default, and you  
 really
 should have made some method virtual (but didn't), then you have to pay  
 for
 it later when you update the base class.
I recognise this, but I don't think that's necessarily a bad thing. It forces you a moment of consideration wrt making the change, and if it will affect anything else. If it feels like a significant change, you'll treat it as such (which it is). Even though you do need to make the change, it's not a breaking change, and you don't risk any side effects.
I find this VERY ironic :) Library Author: After careful consideration, we have decided that we are going to make all our classes virtual, to allow more flexibility. Library user Manu: NOOOOO! That will make all my code horribly slow! Library Author: Don't worry! Your code will still compile and work! It's a non-breaking change with no risk of side effects.
 When virtual is the default, and you really wanted it to be final (but
 didn't do that), then you have to pay for it later when you update the  
 base
 class.  There is no way that is advantageous to *everyone*.
But unlike the first situation, this is a breaking change. If you are not the only user of your library, then this can't be done safely.
I think it breaks both ways, just in different ways.
 It's advantageous to a particular style of coding.  If you know  
 everything
 is virtual by default, then you write code expecting that.  Like mock
 objects.  Or extending a class simply to change one method, even when  
 you
 weren't expecting that to be part of the design originally.
If you write code like that, then write 'virtual:', it doesn't hurt anyone else. The converse is not true.
This really is simply a matter of preference. Your preference for performance over flexibility is biasing your judgment. You can just as easly write 'final'. The default is an arbitrary decision. When I first came across D, I was experiencing "D euphoria" and I wholeheartedly considered the decision to have virtual-by-default a very wise one. At this point, I'm indifferent. It could have been either way, and I think we would be fine. But to SWITCH mid-stream would be a horrible breaking change, and needs to have a very compelling reason.
 I think it is unfair to say most classes are not base classes.  This  
 would
 mean most classes are marked as final.  I don't think they are.  One of  
 the
 main reasons to use classes in the first place is for extendability.
People rarely use the final keyword on classes, even though they could 90% of the time.
Let me fix that for you: "People rarely use the final keyword on classes, even though I wish they would 90% of the time." A non-final class is, by definition, a base class. To say that a non-final class is not a base class because it 'could be' final is just denial :)
 The losses are that if category 3 were simply always final, some other
 anti-Manu who wanted to extend everything has to contact all the  
 original
 authors to get them to change their classes to virtual :)
Fine, they'll probably be receptive since it's not a breaking change. Can you guess how much traction I have when I ask an author of a popular library to remove some 'virtual' keywords in C++ code? "Oh we can't really do that, it could break any other users!", so then we rewrite the library.
This is a horrible argument. C++ IS final by default. They HAVE TO opt in by default. You have been spending all this time arguing we should go the C++ route only to tell me that your experience with C++ is that you can't get what you want there either?!!! Alternatively, we can say the two situations aren't the same. In the C++ situation, the author opted for virtuality. In the D case, the author may have simply not cared. In the not caring case, they may be much more open to adding final (I did). In the case where they specifically want virtuality, they aren't going to drop it whether it's the default or not.
 BTW, did you know you can extend a base class and simply make the  
 extension
 final, and now all the methods on that derived class become non-virtual
 calls?  Much easier to do than making the original base virtual (Note I
 haven't tested this to verify, but if not, it should be changed in the
 compiler).
One presumes that the library that defines the base class deals with its own base pointers internally, and as such, the functions that I may have finalised in my code will still be virtual in the place that it counts.
Methods take the base pointer, but will be inlinable on a final class, and any methods they call will be inlinable and final. Any closed source code is already compiled, and it's too bad you can't fix it. But that is simply a missed optimization for the library writer. It's no different than someone having a poorly implemented algorithm, or doing something stupid like unaligned simd loads :) -Steve
Jun 04 2013
prev sibling next sibling parent Robert <jfanatiker gmx.at> writes:
Another issue: It does not play well with DIP26 :-)
Jun 06 2013
prev sibling parent Robert <jfanatiker gmx.at> writes:
On Mon, 2013-06-03 at 17:05 +1000, Manu wrote:
 Interestingly, you didn't actually disagree with my point about the
 common case here, and I don't buy the Java doctrine.
Also Java is interpreted with hotspot JIT -> methods can be de-virtualized at runtime. So Java is on another playground than D.
Jun 06 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 2 June 2013 20:16, Jonathan M Davis <jmdavisProg gmx.com> wrote:

 On Sunday, June 02, 2013 11:53:26 Jacob Carlborg wrote:
 On 2013-06-01 23:08, Jonathan M Davis wrote:
 If you don't need polymorphism, then in general, you shouldn't use a
class
 (though sometimes it might make sense simply because it's an easy way
to
 get a reference type). Where it becomes more of a problem is when you
 need a few polymorphic functions and a lot of non-polymorphic functions
 (e.g. when a class has a few methods which get overridden and then a
lot
 of properties which it makes no sense to override). In that case, you
 have to use a class, and then you have to mark a lot of functions as
 final. This is what folks like Manu and Don really don't like,
 particularly when they're in environments where the extra cost of the
 virtual function calls actually matters.
If a reference type is needed but not a polymorphic type, then a final class can be used.
Yes. The main problem is when you have a class with a few methods which should be virtual and a lot that don't. You're forced to mark a large number of functions as final. That burden can be lessened by using final with a colon rather than marking them individually, but rather what seems to inevitably happen is that programmers forget to mark any of them as final (Manu can rant quite a bit about that, as he's had to deal with it at work, and it's cost him quite a bit of time, as he has to go through every function which wasn't marked as final and determine whether it's actuallly supposed to be virtual or not). Having non-virtual be the default makes functions efficient by default.
Aye. This, and maybe even a more important result is it makes it _explicit_ by default. Ie, it was intended by the author. You can tell just by looking how the interface is intended to be used, and the intent of the programmer who wrote it. No more guessing, or lengthy searches through all derived classes to find out if it actually is overridden. And what if your interface is public...? Making a function virtual is a one-way trip, it can never be revoked. Making it virtual-by-default eliminates the possibility of revoking that permission to override ever in the future. The converse isn't true, a non-virtual can safely become virtual at any later time if it becomes a requirement, or is requested by a customer. There's some correctness advantages too. A function that's not marked virtual was obviously not intended to be overridden by design; the author of the class may have never considered the possibility, and the class might not even work if someone unexpectedly overrides it somewhere. If virtual is requested/added at a later time, the author, when considering if it's a safe change, will likely take the time to validate that the new usage (which he obviously didn't consider when designing the class initially) is sound against the rest of the class... this is a good thing if you ask me.
Jun 02 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 06/01/2013 08:22 PM, Walter Bright wrote:> On 5/30/2013 7:56 PM, 
Andrei Alexandrescu wrote:
 On 5/30/13 9:26 PM, finalpatch wrote:
 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Manu's gonna love this one: make all methods final.
I have another suggestion. class Sphere and class Ray should be structs. Neither class uses polymorphism in any way, so there's no reason to make them classes with virtual functions.
I was about to write that too, but when the goal is to write a scenegraph polymorphism is what you want, i.e. Sphere is just one object and there is different lights. Ray already is a struct.
Jun 02 2013
prev sibling next sibling parent reply "FeepingCreature" <default_357-line yahoo.de> writes:
There's some issues involving the use of array literals - they
get allocated on the heap for no clear reason. Create a version
of your vector constructor that uses four floats, then call that
instead in your line 324.
May 30 2013
next sibling parent "FeepingCreature" <default_357-line yahoo.de> writes:
On Friday, 31 May 2013 at 03:26:16 UTC, FeepingCreature wrote:
 There's some issues involving the use of array literals - they
 get allocated on the heap for no clear reason. Create a version
 of your vector constructor that uses four floats, then call that
 instead in your line 324.
Addendum: in general, always, **ALWAYS PROFILE BEFORE OPTIMIZING**. I'm sure there's good profilers for OSX, or you can just use -g -pg and gprof. (perf is good under linux!)
May 30 2013
prev sibling parent reply "finalpatch" <fengli gmail.com> writes:
Hi FeepingCreature,

Thanks for the tip, getting rid of the array constructor helped a 
lot, Runtime is down from 800+ms to 583ms (with LDC, still cannot 
match C++ though). Maybe I should get rid of all arrays and use 
hardcoded x,y,z member variables instead, or use tuples.

On Friday, 31 May 2013 at 03:26:16 UTC, FeepingCreature wrote:
 There's some issues involving the use of array literals - they
 get allocated on the heap for no clear reason. Create a version
 of your vector constructor that uses four floats, then call that
 instead in your line 324.
May 30 2013
parent reply Juan Manuel Cabo <juanmanuel.cabo gmail.com> writes:
On 05/31/2013 12:45 AM, finalpatch wrote:
 Hi FeepingCreature,
 
 Thanks for the tip, getting rid of the array constructor helped a lot, Runtime
is down from 800+ms to 583ms (with LDC,
 still cannot match C++ though). Maybe I should get rid of all arrays and use
hardcoded x,y,z member variables instead,
 or use tuples.
 
 On Friday, 31 May 2013 at 03:26:16 UTC, FeepingCreature wrote:
 There's some issues involving the use of array literals - they
 get allocated on the heap for no clear reason. Create a version
 of your vector constructor that uses four floats, then call that
 instead in your line 324.
I just shaved 1.2 seconds trying with dmd by changing the dot function from: float dot(in Vec3 v1, in Vec3 v2) { auto t = v1.v * v2.v; auto p = t.ptr; return p[0] + p[1] + p[2]; } to: float dot(in Vec3 v1, in Vec3 v2) { auto one = v1.v.ptr; auto two = v2.v.ptr; return one[0] * two[0] + one[1] * two[1] + one[2] * two[2]; } Before: 2 secs, 895 ms, 891 μs, and 7 hnsecs After: 1 sec, 648 ms, 698 μs, and 1 hnsec For others who might want to try, I downloaded the necessary derelict files from: http://www.dsource.org/projects/derelict/browser/branches/Derelict2 (DerelictUtil and DerelictSDL directories). And compiled with: dmd -O -inline -noboundscheck -release raytracer.d \ derelict/sdl/sdl.d derelict/sdl/sdlfuncs.d \ derelict/sdl/sdltypes.d derelict/util/compat.d \ derelict/util/exception.d derelict/util/loader.d \ derelict/util/sharedlib.d -L-ldl I also ran it with the -profile switch. Here are the top functions in trace.log: ======== Timer Is 3579545 Ticks/Sec, Times are in Microsecs ======== Num Tree Func Per Calls Time Time Call 11834377 688307713 688307713 58 const(bool function(raytracer.Ray, float*)) raytracer.Sphere.intersect 1446294 2922493954 582795433 402 raytracer.Vec3 raytracer.trace(const(raytracer.Ray), raytracer.Scene, int) 1 1748464181 296122753 296122753 void raytracer.render(raytracer.Scene, derelict.sdl.sdltypes.SDL_Surface*) 933910 309760738 110563786 118 _D9raytracer5traceFxS9raytracer3RayS .... (lambda) 1 1829865336 78200113 78200113 _Dmain 933910 42084879 42084879 45 const(raytracer.Vec3 function(raytracer.Vec3)) raytracer.Sphere.normal 795095 13423716 13423716 16 const(raytracer.Vec3 function(const(raytracer.Vec3))) raytracer.Vec3.opBinary!("*").opBinary 933910 11122934 11122934 11 pure nothrow trusted float std.math.pow!(float, int).pow(float, int) 933910 313479603 3718864 3 _D9raytracer5traceFxS9raytracer3RayS9raytracer5SceneiZS ... (lambda) 1 3014385 2991659 2991659 void derelict.util.sharedlib.SharedLib.load(immutable(char)[][]) 1 152945 152945 152945 void derelict.util.loader.SharedLibLoader.unload() 1590190 89018 89018 0 const(raytracer.Vec3 function(const(float))) raytracer.Vec3.opBinary!("*").opBinary 1047016 70383 70383 0 const(float function()) raytracer.Sphere.transparency 186 66925 66925 359 void derelict.util.loader.SharedLibLoader.bindFunc(void**, immutable(char)[], bool)
May 30 2013
next sibling parent "finalpatch" <fengli gmail.com> writes:
Hi,

I think you are using the version(D_SIMD) path, which is my (not 
very successful) attempt at vectorizing the thing.

change the version(D_SIMD) line to version(none) and it will use 
the scalar path, which has exactly the same dot() function as 
yours.

On Friday, 31 May 2013 at 04:29:19 UTC, Juan Manuel Cabo wrote:
 I just shaved 1.2 seconds trying with dmd by changing the dot 
 function from:

     float dot(in Vec3 v1, in Vec3 v2)
     {
         auto t = v1.v * v2.v;
         auto p = t.ptr;
         return p[0] + p[1] + p[2];
     }

 to:

     float dot(in Vec3 v1, in Vec3 v2)
     {
         auto one = v1.v.ptr;
         auto two = v2.v.ptr;
         return one[0] * two[0]
             + one[1] * two[1]
             + one[2] * two[2];
     }

 Before:
 	2 secs, 895 ms, 891 μs, and 7 hnsecs
 After:
         1 sec, 648 ms, 698 μs, and 1 hnsec
May 30 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 05/31/2013 06:29 AM, Juan Manuel Cabo wrote:
 I just shaved 1.2 seconds trying with dmd by changing the dot function from:
Yep, using -profile on the original code shows that 33% is spend in Sphere.intersect. And the asm for the dot product is horrible.
Jun 02 2013
prev sibling next sibling parent reply "nazriel" <spam dzfl.pl> writes:
On Friday, 31 May 2013 at 01:26:13 UTC, finalpatch wrote:
 Recently I ported a simple ray tracer I wrote in C++11 to D. 
 Thanks to the similarity between D and C++ it was almost a line 
 by line translation, in other words, very very close. However, 
 the D verson runs much slower than the C++11 version. On 
 Windows, with MinGW GCC and GDC, the C++ version is twice as 
 fast as the D version. On OSX, I used Clang++ and LDC, and the 
 C++11 version was 4x faster than D verson.  Since the 
 comparison were between compilers that share the same codegen 
 backends I suppose that's a relatively fair comparison.  (flags 
 used for GDC: -O3 -fno-bounds-check -frelease,  flags used for 
 LDC: -O3 -release)

 I really like the features offered by D but it's the raw 
 performance that's worrying me. From what I read D should offer 
 similar performance when doing similar things but my own test 
 results is not consistent with this claim. I want to know 
 whether this slowness is inherent to the language or it's 
 something I was not doing right (very possible because I have 
 only a few days of experience with D).

 Below is the link to the D and C++ code, in case anyone is 
 interested to have a look.

 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Greetings. After few fast changes I manage to get such results: [raz d3 tmp]$ ./a.out rendering time 276 ms [raz d3 tmp]$ ./test 346 ms, 814 μs, and 5 hnsecs ./a.out being binary compiled with clang++ ./test.cxx -std=c++11 -lSDL -O3 ./test being binary compiled with ldmd2 -O3 -release -inline -noboundscheck ./test.d (Actually I used rdmd with --compiler=ldmd2 but I omitted it because it was rather long cmd line :p) Here is source code with changes I applied to D-code (I hope you don't mind repasting it): http://dpaste.dzfl.pl/84bb308d I am sure there is way more room for improvements and at minimum achieving C++ performance.
May 30 2013
next sibling parent reply Juan Manuel Cabo <juanmanuel.cabo gmail.com> writes:
On 05/31/2013 02:15 AM, nazriel wrote:
 On Friday, 31 May 2013 at 01:26:13 UTC, finalpatch wrote:
 Recently I ported a simple ray tracer I wrote in C++11 to D. Thanks to the
similarity between D and C++ it was almost
 a line by line translation, in other words, very very close. However, the D
verson runs much slower than the C++11
 version. On Windows, with MinGW GCC and GDC, the C++ version is twice as fast
as the D version. On OSX, I used Clang++
 and LDC, and the C++11 version was 4x faster than D verson.  Since the
comparison were between compilers that share
 the same codegen backends I suppose that's a relatively fair comparison. 
(flags used for GDC: -O3 -fno-bounds-check
 -frelease,  flags used for LDC: -O3 -release)

 I really like the features offered by D but it's the raw performance that's
worrying me. From what I read D should
 offer similar performance when doing similar things but my own test results is
not consistent with this claim. I want
 to know whether this slowness is inherent to the language or it's something I
was not doing right (very possible
 because I have only a few days of experience with D).

 Below is the link to the D and C++ code, in case anyone is interested to have
a look.

 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Greetings. After few fast changes I manage to get such results: [raz d3 tmp]$ ./a.out rendering time 276 ms [raz d3 tmp]$ ./test 346 ms, 814 μs, and 5 hnsecs ./a.out being binary compiled with clang++ ./test.cxx -std=c++11 -lSDL -O3 ./test being binary compiled with ldmd2 -O3 -release -inline -noboundscheck ./test.d (Actually I used rdmd with --compiler=ldmd2 but I omitted it because it was rather long cmd line :p) Here is source code with changes I applied to D-code (I hope you don't mind repasting it): http://dpaste.dzfl.pl/84bb308d I am sure there is way more room for improvements and at minimum achieving C++ performance.
You might also try changing: float[3] t = mixin("v[]"~op~"rhs.v[]"); return Vec3(t[0], t[1], t[2]); for: Vec3 t; t.v[0] = mixin("v[0] "~op~" rhs.v[0]"); t.v[1] = mixin("v[1] "~op~" rhs.v[1]"); t.v[2] = mixin("v[2] "~op~" rhs.v[2]"); return t; and so on, avoiding the float[3] and the v[] operations (which would loop, unless the compiler/optimizer unrolls them (didn't check)). I tested this change (removing v[] ops) in Vec3 and in normalize(), and it made your version slightly faster with DMD (didn't check with ldmd2). --jm
May 30 2013
next sibling parent "nazriel" <spam dzfl.pl> writes:
On Friday, 31 May 2013 at 05:35:58 UTC, Juan Manuel Cabo wrote:
 On 05/31/2013 02:15 AM, nazriel wrote:
 On Friday, 31 May 2013 at 01:26:13 UTC, finalpatch wrote:
 Recently I ported a simple ray tracer I wrote in C++11 to D. 
 Thanks to the similarity between D and C++ it was almost
 a line by line translation, in other words, very very close. 
 However, the D verson runs much slower than the C++11
 version. On Windows, with MinGW GCC and GDC, the C++ version 
 is twice as fast as the D version. On OSX, I used Clang++
 and LDC, and the C++11 version was 4x faster than D verson.  
 Since the comparison were between compilers that share
 the same codegen backends I suppose that's a relatively fair 
 comparison.  (flags used for GDC: -O3 -fno-bounds-check
 -frelease,  flags used for LDC: -O3 -release)

 I really like the features offered by D but it's the raw 
 performance that's worrying me. From what I read D should
 offer similar performance when doing similar things but my 
 own test results is not consistent with this claim. I want
 to know whether this slowness is inherent to the language or 
 it's something I was not doing right (very possible
 because I have only a few days of experience with D).

 Below is the link to the D and C++ code, in case anyone is 
 interested to have a look.

 https://dl.dropboxusercontent.com/u/974356/raytracer.d
 https://dl.dropboxusercontent.com/u/974356/raytracer.cpp
Greetings. After few fast changes I manage to get such results: [raz d3 tmp]$ ./a.out rendering time 276 ms [raz d3 tmp]$ ./test 346 ms, 814 μs, and 5 hnsecs ./a.out being binary compiled with clang++ ./test.cxx -std=c++11 -lSDL -O3 ./test being binary compiled with ldmd2 -O3 -release -inline -noboundscheck ./test.d (Actually I used rdmd with --compiler=ldmd2 but I omitted it because it was rather long cmd line :p) Here is source code with changes I applied to D-code (I hope you don't mind repasting it): http://dpaste.dzfl.pl/84bb308d I am sure there is way more room for improvements and at minimum achieving C++ performance.
You might also try changing: float[3] t = mixin("v[]"~op~"rhs.v[]"); return Vec3(t[0], t[1], t[2]); for: Vec3 t; t.v[0] = mixin("v[0] "~op~" rhs.v[0]"); t.v[1] = mixin("v[1] "~op~" rhs.v[1]"); t.v[2] = mixin("v[2] "~op~" rhs.v[2]"); return t; and so on, avoiding the float[3] and the v[] operations (which would loop, unless the compiler/optimizer unrolls them (didn't check)). I tested this change (removing v[] ops) in Vec3 and in normalize(), and it made your version slightly faster with DMD (didn't check with ldmd2). --jm
Right, I missed that. Thanks! Now it is: [raz d3 tmp]$ ./a.out rendering time 276 ms [raz d3 tmp]$ ./test 238 ms, 35 μs, and 7 hnsecs So D version starts to be faster than C++ one.
May 30 2013
prev sibling parent reply "finalpatch" <fengli gmail.com> writes:
You guys are awesome! I am happy to know that D can indeed offer 
comparable speed to C++.

But it also shows there is room for the compiler to improve as 
the C++ version also makes heavy use of loops (or STL algorithms) 
but they get inlined or unrolled automatically.

On Friday, 31 May 2013 at 05:35:58 UTC, Juan Manuel Cabo wrote:
 You might also try changing:

             float[3] t = mixin("v[]"~op~"rhs.v[]");
             return Vec3(t[0], t[1], t[2]);

 for:
             Vec3 t;
             t.v[0] = mixin("v[0] "~op~" rhs.v[0]");
             t.v[1] = mixin("v[1] "~op~" rhs.v[1]");
             t.v[2] = mixin("v[2] "~op~" rhs.v[2]");
             return t;

 and so on, avoiding the float[3] and the v[] operations (which 
 would
 loop, unless the compiler/optimizer unrolls them (didn't 
 check)).

 I tested this change (removing v[] ops) in Vec3 and in
 normalize(), and it made your version slightly faster
 with DMD (didn't check with ldmd2).

 --jm
May 30 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 31 May 2013 at 05:59:00 UTC, finalpatch wrote:
 You guys are awesome! I am happy to know that D can indeed 
 offer comparable speed to C++.

 But it also shows there is room for the compiler to improve as 
 the C++ version also makes heavy use of loops (or STL 
 algorithms) but they get inlined or unrolled automatically.
Have you tried to use GDC or LDC ? They use similar optimizers and code generator than GCC and clang, so should be able to do it as well.
May 30 2013
parent dennis luehring <dl.soluz gmx.net> writes:
Am 31.05.2013 08:11, schrieb deadalnix:
 On Friday, 31 May 2013 at 05:59:00 UTC, finalpatch wrote:
 You guys are awesome! I am happy to know that D can indeed
 offer comparable speed to C++.

 But it also shows there is room for the compiler to improve as
 the C++ version also makes heavy use of loops (or STL
 algorithms) but they get inlined or unrolled automatically.
Have you tried to use GDC or LDC ? They use similar optimizers and code generator than GCC and clang, so should be able to do it as well.
he only used GDC,LDC
May 30 2013
prev sibling parent "nazriel" <spam dzfl.pl> writes:
On Friday, 31 May 2013 at 05:59:00 UTC, finalpatch wrote:
 You guys are awesome! I am happy to know that D can indeed 
 offer comparable speed to C++.

 But it also shows there is room for the compiler to improve as 
 the C++ version also makes heavy use of loops (or STL 
 algorithms) but they get inlined or unrolled automatically.
Agree. I feel big hammer going towards my head from Walter/Andrei side but IMHO abandoning DMD in the first place would be the best idea. Focusing on LDC or GDC would bring way much more benefits than trying to make anything from DMD. Version compiled with LDC runs in 202 ms and 192 μs. DMD... 1 sec, 891 ms, 571 μs, and 1 hnsec
 On Friday, 31 May 2013 at 05:35:58 UTC, Juan Manuel Cabo wrote:
 You might also try changing:

            float[3] t = mixin("v[]"~op~"rhs.v[]");
            return Vec3(t[0], t[1], t[2]);

 for:
            Vec3 t;
            t.v[0] = mixin("v[0] "~op~" rhs.v[0]");
            t.v[1] = mixin("v[1] "~op~" rhs.v[1]");
            t.v[2] = mixin("v[2] "~op~" rhs.v[2]");
            return t;

 and so on, avoiding the float[3] and the v[] operations (which 
 would
 loop, unless the compiler/optimizer unrolls them (didn't 
 check)).

 I tested this change (removing v[] ops) in Vec3 and in
 normalize(), and it made your version slightly faster
 with DMD (didn't check with ldmd2).

 --jm
May 30 2013
prev sibling parent reply "finalpatch" <fengli gmail.com> writes:
Thanks Nazriel,

It is very cool you are able to narrow the gap to within 1.5x of 
c++ with a few simple changes.

I checked your version, there are 3 changes (correct me if i 
missed any):

* Change the (float) constructor from v= [x,x,x] to v[0] = x; 
v[1] = x; v[2] = x;
* Get rid of the (float[]) constructor and use 3 floats instead
* Change class methods to final

The first change alone shaved off 220ms off the runtime, the 2nd 
one cuts 130ms
and the 3rd one cuts 60ms.

Lesson learned: by very very careful about dynamic arrays.

On Friday, 31 May 2013 at 05:15:11 UTC, nazriel wrote:
 After few fast changes I manage to get such results:
 [raz d3 tmp]$ ./a.out
 rendering time 276 ms
 [raz d3 tmp]$ ./test
 346 ms, 814 μs, and 5 hnsecs


 ./a.out being binary compiled with clang++ ./test.cxx 
 -std=c++11 -lSDL -O3
 ./test being binary compiled with ldmd2 -O3 -release -inline 
 -noboundscheck ./test.d (Actually I used rdmd with 
 --compiler=ldmd2 but I omitted it because it was rather long 
 cmd line :p)


 Here is source code with changes I applied to D-code (I hope 
 you don't mind repasting it): http://dpaste.dzfl.pl/84bb308d

 I am sure there is way more room for improvements and at 
 minimum achieving C++ performance.
May 30 2013
next sibling parent "nazriel" <spam dzfl.pl> writes:
I managed to get it even faster.

[raz d3 tmp]$ ./a.out
rendering time 282 ms
[raz d3 tmp]$ ./test
202 ms, 481 μs, and 8 hnsecs

So D version is 1,4x faster than C++ version.
At least on my computer.

Same compilers flags etc

Final code:
http://dpaste.dzfl.pl/61626e88

I guess there is still more room for improvements.

On Friday, 31 May 2013 at 05:49:55 UTC, finalpatch wrote:
 Thanks Nazriel,

 It is very cool you are able to narrow the gap to within 1.5x 
 of c++ with a few simple changes.

 I checked your version, there are 3 changes (correct me if i 
 missed any):

 * Change the (float) constructor from v= [x,x,x] to v[0] = x; 
 v[1] = x; v[2] = x;
Correct
 * Get rid of the (float[]) constructor and use 3 floats instead
It was just for debbuging so compiler would yell at me if I use array literal
 * Change class methods to final
Correct
 The first change alone shaved off 220ms off the runtime, the 
 2nd one cuts 130ms
 and the 3rd one cuts 60ms.

 Lesson learned: by very very careful about dynamic arrays.
Yeah, it is currently a problem with array literals. They're always allocated on heap even if they shouldn't be. Final before methods is something that needs to be remembered
May 30 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 31 May 2013 15:49, finalpatch <fengli gmail.com> wrote:

 Thanks Nazriel,

 It is very cool you are able to narrow the gap to within 1.5x of c++ with
 a few simple changes.

 I checked your version, there are 3 changes (correct me if i missed any):

 * Change the (float) constructor from v= [x,x,x] to v[0] = x; v[1] = x;
 v[2] = x;
 * Get rid of the (float[]) constructor and use 3 floats instead
 * Change class methods to final

 The first change alone shaved off 220ms off the runtime, the 2nd one cuts
 130ms
 and the 3rd one cuts 60ms.

 Lesson learned: by very very careful about dynamic arrays.
Yeah, I've actually noticed this too on a few occasions. It would be nice if array operations would unroll for short arrays. Particularly so for static arrays!
May 30 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Manu:

 Yeah, I've actually noticed this too on a few occasions. It 
 would be nice
 if array operations would unroll for short arrays. Particularly 
 so for static arrays!
Thanks to Kenji the latest dmd 2.063 solves part of this problem: http://d.puremagic.com/issues/show_bug.cgi?id=2356 Maybe this improvement is not yet in LDC/GDC. But avoiding heap allocations for array literals is a change that needs to be discussed. Bye, bearophile
May 31 2013
next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Friday, 31 May 2013 at 15:02:00 UTC, bearophile wrote:
 Thanks to Kenji the latest dmd 2.063 solves part of this 
 problem:
 http://d.puremagic.com/issues/show_bug.cgi?id=2356

 Maybe this improvement is not yet in LDC/GDC.
Regarding static array initialization from literals, yes, that change isn't in 2.062. However, what pains me a bit is that LDC does even manage to remove the dynamic array allocation after the fact: --- __D9raytracer4Vec36__ctorMFxfZS9raytracer4Vec3: movss DWORD PTR [RSP - 12], XMM0 movss DWORD PTR [RSP - 8], XMM0 movss DWORD PTR [RSP - 4], XMM0 mov EAX, DWORD PTR [RSP - 4] mov DWORD PTR [RDI + 8], EAX mov RAX, QWORD PTR [RSP - 12] mov QWORD PTR [RDI], RAX mov RAX, RDI ret --- It just happens too late in the optimizer pipeline for the temporary memcpy to go away. Re-running the IR through the LLVM optimizer then produces this: --- __D9raytracer4Vec36__ctorMFxfZS9raytracer4Vec3: movss DWORD PTR [RDI], XMM0 movss DWORD PTR [RDI + 4], XMM0 movss DWORD PTR [RDI + 8], XMM0 mov RAX, RDI ret --- — David
Jun 01 2013
prev sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
 But avoiding heap allocations for array literals is a change 
 that needs to be discussed.
In the meantime I have written a small ER regarding escape analysis for dynamic arrays: http://d.puremagic.com/issues/show_bug.cgi?id=10242 Bye, bearophile
Jun 02 2013
prev sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Friday, 31 May 2013 at 06:41:19 UTC, Manu wrote:
 It would be nice
 if array operations would unroll for short arrays. Particularly 
 so for
 static arrays!
It definitely is something we need to implement. There is just no excuse not to, and it hampers the practicality of a nice language feature. I didn't profile the original program yet, but I bet it's one of the major reasons why the LDC version is so slow. I wonder where the best place to do this would be though. Doing it in the frontend would allow all three compilers to take advantage of the transformation, but it's really glue code stuff. IIRC, GDC already does something like that? David
Jun 01 2013
prev sibling parent reply "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 31 May 2013 at 05:49:55 UTC, finalpatch wrote:
 Thanks Nazriel,

 It is very cool you are able to narrow the gap to within 1.5x 
 of c++ with a few simple changes.

 I checked your version, there are 3 changes (correct me if i 
 missed any):

 * Change the (float) constructor from v= [x,x,x] to v[0] = x; 
 v[1] = x; v[2] = x;
 * Get rid of the (float[]) constructor and use 3 floats instead
I thought GDC or LDC have something like: float[$] v = [x, x, x]; which is converted to flot[3] v = [x, x, x]; Am I wrong? DMD need something like this too.
May 31 2013
parent "bearophile" <bearophileHUGS lycos.com> writes:
Namespace:

 I thought GDC or LDC have something like:
 float[$] v = [x, x, x];
 which is converted to
 flot[3] v = [x, x, x];

 Am I wrong?
 DMD need something like this too.
Right. Vote (currently only 6 votes): http://d.puremagic.com/issues/show_bug.cgi?id=481 Bye, bearophile
May 31 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 31 May 2013 11:26, finalpatch <fengli gmail.com> wrote:

 Recently I ported a simple ray tracer I wrote in C++11 to D. Thanks to the
 similarity between D and C++ it was almost a line by line translation, in
 other words, very very close. However, the D verson runs much slower than
 the C++11 version. On Windows, with MinGW GCC and GDC, the C++ version is
 twice as fast as the D version. On OSX, I used Clang++ and LDC, and the
 C++11 version was 4x faster than D verson.  Since the comparison were
 between compilers that share the same codegen backends I suppose that's a
 relatively fair comparison.  (flags used for GDC: -O3 -fno-bounds-check
 -frelease,  flags used for LDC: -O3 -release)

 I really like the features offered by D but it's the raw performance
 that's worrying me. From what I read D should offer similar performance
 when doing similar things but my own test results is not consistent with
 this claim. I want to know whether this slowness is inherent to the
 language or it's something I was not doing right (very possible because I
 have only a few days of experience with D).

 Below is the link to the D and C++ code, in case anyone is interested to
 have a look.

 https://dl.dropboxusercontent.**com/u/974356/raytracer.d<https://dl.dropboxusercontent.com/u/974356/raytracer.d>
 https://dl.dropboxusercontent.**com/u/974356/raytracer.cpp<https://dl.dropboxusercontent.com/u/974356/raytracer.cpp>
Can you paste the disassembly of the inner loop (trace()) for each G++/GDC, Or LDC/Clang++? That said, I can see almost innumerable red flags (on basically every line). The fact that it takes 200ms to render a frame in C++ (I would expect <10ms) suggests that your approach is amazingly slow to begin with, at which point I would start looking for much higher level problems. Once you have an implementation that's approaching optimal, then we can start making comparisons. Here are some thoughts at first glance: * The fact that you use STL makes me immediately concerned. Generic code for this sort of work will never run well. That said, STL has decades more time spent optimising, so it stands to reason that the C++ compiler will be able to do more to improve the STL code. * Your vector class both in C++/D are pretty nasty. Use 4d SIMD vectors. * So many integer divisions! * There are countless float <-> int casts. * Innumerable redundant loads/stores. * I would have raised the virtual-by-default travesty, but Andrei did it for me! ;) * intersect() should be __forceinline. * intersect() is full of if's (it's hard to predict if the optimiser can work across those if's. maybe it can...) What's taking the most time? The lighting loop is so template-tastic, I can't get a feel for how fast that loop would be. I believe the reason for the difference is not going to be so easily revealed. It's probably hidden largely in the fact that C++ has had a good decade of optimisation spent on STL over D. It's also possible that the C++ compiler hooks many of those STL functions as compiler intrinsics with internalised logic. Frankly, this is a textbook example of why STL is the spawn of satan. For some reason people are TAUGHT that it's reasonable to write code like this.
May 30 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Manu:

 Frankly, this is a textbook example of why STL is the spawn of 
 satan. For
 some reason people are TAUGHT that it's reasonable to write 
 code like this.
There are many kinds of D code, not everything is a high performance ray-tracer or 3D game. So I'm sure there are many many situations where using the C++ STL is more than enough. As most tools, you need to know where and when to use them. So it's not a Satan-spawn :-) Bye, bearophile
May 31 2013
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/31/2013 05:12 PM, bearophile wrote:
 There are many kinds of D code, not everything is a high performance ray-tracer
 or 3D game. So I'm sure there are many many situations where using the C++ STL
 is more than enough. As most tools, you need to know where and when to use
them.
 So it's not a Satan-spawn :-)
It's the Microsoft Office of programming solutions -- people get taught that this is the tool to use and don't get adequately taught about the alternatives or the rationale ... I well remember my first experience with STL as a new postgraduate, _very_ inexperienced at programming (what's changed? :-P ) and having learned basically a small part of C. I was writing some code where I needed at each time step to collect a list of indices of array entries whose value would change. The only way I could think of how to do this in C was to use malloc to create an array of length N, and to store the actual number of indices stored at a given time step as a size_t which got reset to zero at the start of each step. So, I could store as many indices as there were to list; the malloc was done at the start of the whole process, outside the loop of time steps. Then I talked with a (much more experienced) colleague about the design of my code and this thing I was doing with a set of data that started from empty every time and built up the list of indices, and he said, "Oh, try using the vector type in C++." So, I went, and discovered a bit about how it worked, and rewrote all my code using vectors in place of arrays and other nice C++ stuff, and I managed to come up with code that was much, much easier to read, and was ten zillion times slower. (I can't remember now, but I suspect that my use of vector was carrying out lots of unnecessary allocs and frees within the core loop.) I'd surely do much better these days (e.g. I'd make sure to define the vector outside the loop and to reserve the exact amount of memory required up front), but it was a nice lesson in how clever, generic-looking tools can be a big problem if you don't understand how to use them.
May 31 2013
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 1 June 2013 01:12, bearophile <bearophileHUGS lycos.com> wrote:

 Manu:


  Frankly, this is a textbook example of why STL is the spawn of satan. For
 some reason people are TAUGHT that it's reasonable to write code like
 this.
There are many kinds of D code, not everything is a high performance ray-tracer or 3D game. So I'm sure there are many many situations where using the C++ STL is more than enough. As most tools, you need to know where and when to use them. So it's not a Satan-spawn :-)
So why are we having this conversation at all then if faster isn't better in this instance?
May 31 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Manu:

 On 1 June 2013 01:12, bearophile <bearophileHUGS lycos.com> 
 wrote:

 Manu:


  Frankly, this is a textbook example of why STL is the spawn 
 of satan. For
 some reason people are TAUGHT that it's reasonable to write 
 code like
 this.
There are many kinds of D code, not everything is a high performance ray-tracer or 3D game. So I'm sure there are many many situations where using the C++ STL is more than enough. As most tools, you need to know where and when to use them. So it's not a Satan-spawn :-)
So why are we having this conversation at all then if faster isn't better in this instance?
Faster is better in this instance. What's wrong is your thinking that the STL as the spawn of Satan in general. Bye, bearophile
May 31 2013
parent reply Manu <turkeyman gmail.com> writes:
On 1 June 2013 09:15, bearophile <bearophileHUGS lycos.com> wrote:

 Manu:

  On 1 June 2013 01:12, bearophile <bearophileHUGS lycos.com> wrote:
  Manu:
  Frankly, this is a textbook example of why STL is the spawn of satan.
 For

 some reason people are TAUGHT that it's reasonable to write code like
 this.
There are many kinds of D code, not everything is a high performance ray-tracer or 3D game. So I'm sure there are many many situations where using the C++ STL is more than enough. As most tools, you need to know where and when to use them. So it's not a Satan-spawn :-)
So why are we having this conversation at all then if faster isn't better in this instance?
Faster is better in this instance. What's wrong is your thinking that the STL as the spawn of Satan in general.
Ah, but that's because it is ;) Rule of thumb: never use STL in tight loops. problem solved (well, mostly)...
May 31 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.06.2013 01:30, schrieb Manu:
 On 1 June 2013 09:15, bearophile <bearophileHUGS lycos.com
 <mailto:bearophileHUGS lycos.com>> wrote:

     Manu:

         On 1 June 2013 01:12, bearophile <bearophileHUGS lycos.com
         <mailto:bearophileHUGS lycos.com>> wrote:

             Manu:


               Frankly, this is a textbook example of why STL is the
             spawn of satan. For

                 some reason people are TAUGHT that it's reasonable to
                 write code like
                 this.


             There are many kinds of D code, not everything is a high
             performance
             ray-tracer or 3D game. So I'm sure there are many many
             situations where
             using the C++ STL is more than enough. As most tools, you
             need to know
             where and when to use them. So it's not a Satan-spawn :-)


         So why are we having this conversation at all then if faster
         isn't better in this instance?


     Faster is better in this instance.
     What's wrong is your thinking that the STL as the spawn of Satan in
     general.


 Ah, but that's because it is ;)
 Rule of thumb: never use STL in tight loops. problem solved (well,
 mostly)...
I have to agree here. Whenever you have a codebase that has to work on 9 platforms and 6 compilers the S in STL vanishes. Also the implementations are so varying in quality that you might get really good performance on one platform but really bad on another. It seems like everyone in the games industry avoids STL like the plague. Kind Regards Benjamin Thaut
Jun 01 2013
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 01.06.2013 16:24, schrieb Benjamin Thaut:
 Am 01.06.2013 01:30, schrieb Manu:
 On 1 June 2013 09:15, bearophile <bearophileHUGS lycos.com
 <mailto:bearophileHUGS lycos.com>> wrote:

     Manu:

         On 1 June 2013 01:12, bearophile <bearophileHUGS lycos.com
         <mailto:bearophileHUGS lycos.com>> wrote:

             Manu:


               Frankly, this is a textbook example of why STL is the
             spawn of satan. For

                 some reason people are TAUGHT that it's reasonable to
                 write code like
                 this.


             There are many kinds of D code, not everything is a high
             performance
             ray-tracer or 3D game. So I'm sure there are many many
             situations where
             using the C++ STL is more than enough. As most tools, you
             need to know
             where and when to use them. So it's not a Satan-spawn :-)


         So why are we having this conversation at all then if faster
         isn't better in this instance?


     Faster is better in this instance.
     What's wrong is your thinking that the STL as the spawn of Satan in
     general.


 Ah, but that's because it is ;)
 Rule of thumb: never use STL in tight loops. problem solved (well,
 mostly)...
I have to agree here. Whenever you have a codebase that has to work on 9 platforms and 6 compilers the S in STL vanishes. Also the implementations are so varying in quality that you might get really good performance on one platform but really bad on another. It seems like everyone in the games industry avoids STL like the plague. Kind Regards Benjamin Thaut
I used to have that experience even with C, when I started using it around 1994. C++ was even worse between CFront, ARM and ongoing standardization work. As for STL, I can assure that HPC guys are huge fans of STL and Boost. At least when I did my traineeship at CERN (2003-2004) that was the case. -- Paulo
Jun 01 2013
next sibling parent reply Manu <turkeyman gmail.com> writes:
On 2 June 2013 01:19, Paulo Pinto <pjmlp progtools.org> wrote:

 Am 01.06.2013 16:24, schrieb Benjamin Thaut:

  Am 01.06.2013 01:30, schrieb Manu:
 On 1 June 2013 09:15, bearophile <bearophileHUGS lycos.com
 <mailto:bearophileHUGS lycos.**com <bearophileHUGS lycos.com>>> wrote:

     Manu:

         On 1 June 2013 01:12, bearophile <bearophileHUGS lycos.com
         <mailto:bearophileHUGS lycos.**com <bearophileHUGS lycos.com>>>
 wrote:

             Manu:


               Frankly, this is a textbook example of why STL is the
             spawn of satan. For

                 some reason people are TAUGHT that it's reasonable to
                 write code like
                 this.


             There are many kinds of D code, not everything is a high
             performance
             ray-tracer or 3D game. So I'm sure there are many many
             situations where
             using the C++ STL is more than enough. As most tools, you
             need to know
             where and when to use them. So it's not a Satan-spawn :-)


         So why are we having this conversation at all then if faster
         isn't better in this instance?


     Faster is better in this instance.
     What's wrong is your thinking that the STL as the spawn of Satan in
     general.


 Ah, but that's because it is ;)
 Rule of thumb: never use STL in tight loops. problem solved (well,
 mostly)...
I have to agree here. Whenever you have a codebase that has to work on 9 platforms and 6 compilers the S in STL vanishes. Also the implementations are so varying in quality that you might get really good performance on one platform but really bad on another. It seems like everyone in the games industry avoids STL like the plague. Kind Regards Benjamin Thaut
I used to have that experience even with C, when I started using it around 1994. C++ was even worse between CFront, ARM and ongoing standardization work. As for STL, I can assure that HPC guys are huge fans of STL and Boost.
The funny thing about HPC guys though, at least in my experience (a bunch of researchers from Cambridge who I often give _basic_ optimisation tips), is they don't write/run 'high performance software', they're actually pretty terrible programmers and have a tendency to write really low performing software, but run it on super high performance computers, and then call the experience high performance computing... It bends my mind to see them demand an order of magnitude more computing power to run an algorithm that's hamstrung by poor choices of containers or algorithms that probably cost them an order of magnitude in performance ;) And then the Universities take their demands seriously and deliver them more hardware! O_O At least when I did my traineeship at CERN (2003-2004) that was the case.

I hope CERN has better software engineers than Cambridge University ;)
Most of these guys are mathematicians and physicists first, and programmers
second.
Jun 01 2013
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 02.06.2013 08:33, schrieb Manu:
 On 2 June 2013 01:19, Paulo Pinto <pjmlp progtools.org
 <mailto:pjmlp progtools.org>> wrote:
 [...]

     At least when I did my traineeship at CERN (2003-2004) that was the
     case.


 I hope CERN has better software engineers than Cambridge University ;)
 Most of these guys are mathematicians and physicists first, and
 programmers second.
I was on the Atlas group that does real time processing from the data coming out of the particle accelerator, as a way to filter out uninteresting data. And also offline multicore clustered data analysis of the saved data. They even rewrote the network stack to bypass the latency introduced by IP protocols. You are right on the guys backgrounds, but on our case we had monthly meetings about performance of the whole stack, to look for spots worth improving. -- Paulo
Jun 02 2013
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 2 June 2013 at 07:32:10 UTC, Manu wrote:
 On 2 June 2013 01:19, Paulo Pinto <pjmlp progtools.org> wrote:

 Am 01.06.2013 16:24, schrieb Benjamin Thaut:

  Am 01.06.2013 01:30, schrieb Manu:
 On 1 June 2013 09:15, bearophile <bearophileHUGS lycos.com
 <mailto:bearophileHUGS lycos.**com 
 <bearophileHUGS lycos.com>>> wrote:

     Manu:

         On 1 June 2013 01:12, bearophile 
 <bearophileHUGS lycos.com
         <mailto:bearophileHUGS lycos.**com 
 <bearophileHUGS lycos.com>>>
 wrote:

             Manu:


               Frankly, this is a textbook example of why STL 
 is the
             spawn of satan. For

                 some reason people are TAUGHT that it's 
 reasonable to
                 write code like
                 this.


             There are many kinds of D code, not everything 
 is a high
             performance
             ray-tracer or 3D game. So I'm sure there are 
 many many
             situations where
             using the C++ STL is more than enough. As most 
 tools, you
             need to know
             where and when to use them. So it's not a 
 Satan-spawn :-)


         So why are we having this conversation at all then 
 if faster
         isn't better in this instance?


     Faster is better in this instance.
     What's wrong is your thinking that the STL as the spawn 
 of Satan in
     general.


 Ah, but that's because it is ;)
 Rule of thumb: never use STL in tight loops. problem solved 
 (well,
 mostly)...
I have to agree here. Whenever you have a codebase that has to work on 9 platforms and 6 compilers the S in STL vanishes. Also the implementations are so varying in quality that you might get really good performance on one platform but really bad on another. It seems like everyone in the games industry avoids STL like the plague. Kind Regards Benjamin Thaut
I used to have that experience even with C, when I started using it around 1994. C++ was even worse between CFront, ARM and ongoing standardization work. As for STL, I can assure that HPC guys are huge fans of STL and Boost.
The funny thing about HPC guys though, at least in my experience (a bunch of researchers from Cambridge who I often give _basic_ optimisation tips), is they don't write/run 'high performance software', they're actually pretty terrible programmers and have a tendency to write really low performing software, but run it on super high performance computers, and then call the experience high performance computing... It bends my mind to see them demand an order of magnitude more computing power to run an algorithm that's hamstrung by poor choices of containers or algorithms that probably cost them an order of magnitude in performance ;) And then the Universities take their demands seriously and deliver them more hardware! O_O At least when I did my traineeship at CERN (2003-2004) that was the case.

 I hope CERN has better software engineers than Cambridge 
 University ;)
 Most of these guys are mathematicians and physicists first, and 
 programmers
 second.
In my experience, physicists are terrible programmers. I should know, I am one! As soon as you step outside the realm of simple, < 10kloc, pure procedural code, the supposed "HPC" guys don't generally have the first clue how to write something fast. CERN is responsible for the abomination that is ROOT, but to be fair to them there is a lot of good code from there too.
Jun 02 2013
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 02.06.2013 13:08, schrieb John Colvin:
 On Sunday, 2 June 2013 at 07:32:10 UTC, Manu wrote:
 On 2 June 2013 01:19, Paulo Pinto <pjmlp progtools.org> wrote:

 Am 01.06.2013 16:24, schrieb Benjamin Thaut:

  Am 01.06.2013 01:30, schrieb Manu:
 On 1 June 2013 09:15, bearophile <bearophileHUGS lycos.com
 <mailto:bearophileHUGS lycos.**com <bearophileHUGS lycos.com>>> wrote:

     Manu:

         On 1 June 2013 01:12, bearophile <bearophileHUGS lycos.com
         <mailto:bearophileHUGS lycos.**com
 <bearophileHUGS lycos.com>>>
 wrote:

             Manu:


               Frankly, this is a textbook example of why STL is the
             spawn of satan. For

                 some reason people are TAUGHT that it's reasonable to
                 write code like
                 this.


             There are many kinds of D code, not everything is a high
             performance
             ray-tracer or 3D game. So I'm sure there are many many
             situations where
             using the C++ STL is more than enough. As most tools, you
             need to know
             where and when to use them. So it's not a Satan-spawn :-)


         So why are we having this conversation at all then if faster
         isn't better in this instance?


     Faster is better in this instance.
     What's wrong is your thinking that the STL as the spawn of
 Satan in
     general.


 Ah, but that's because it is ;)
 Rule of thumb: never use STL in tight loops. problem solved (well,
 mostly)...
I have to agree here. Whenever you have a codebase that has to work on 9 platforms and 6 compilers the S in STL vanishes. Also the implementations are so varying in quality that you might get really good performance on one platform but really bad on another. It seems like everyone in the games industry avoids STL like the plague. Kind Regards Benjamin Thaut
I used to have that experience even with C, when I started using it around 1994. C++ was even worse between CFront, ARM and ongoing standardization work. As for STL, I can assure that HPC guys are huge fans of STL and Boost.
The funny thing about HPC guys though, at least in my experience (a bunch of researchers from Cambridge who I often give _basic_ optimisation tips), is they don't write/run 'high performance software', they're actually pretty terrible programmers and have a tendency to write really low performing software, but run it on super high performance computers, and then call the experience high performance computing... It bends my mind to see them demand an order of magnitude more computing power to run an algorithm that's hamstrung by poor choices of containers or algorithms that probably cost them an order of magnitude in performance ;) And then the Universities take their demands seriously and deliver them more hardware! O_O At least when I did my traineeship at CERN (2003-2004) that was the case.

 I hope CERN has better software engineers than Cambridge University ;)
 Most of these guys are mathematicians and physicists first, and
 programmers
 second.
In my experience, physicists are terrible programmers. I should know, I am one! As soon as you step outside the realm of simple, < 10kloc, pure procedural code, the supposed "HPC" guys don't generally have the first clue how to write something fast. CERN is responsible for the abomination that is ROOT, but to be fair to them there is a lot of good code from there too.
There was an office there that had the sentence "You can program Fortran in any language" on the door. :) -- Paulo
Jun 02 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/2/2013 5:29 AM, Paulo Pinto wrote:
 There was an office there that had the sentence "You can program Fortran in any
 language" on the door. :)
I think that joke is older than me!
Jun 02 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/02/2013 08:33 AM, Manu wrote:
 Most of these guys are mathematicians and physicists first, and programmers
second.
You've hit the nail on the head, but it's also a question of priorities. It's _essential_ that the maths or physics be understood and done right. It's essential that the programs correctly reflect that maths or physics. It's merely _desirable_ that the programs run as fast as possible, or be well designed from a maintenance point of view, or any of the other things that matter to trained software developers. (In my day job I have to continually force myself to _not_ refactor or optimize my code, even though I'd get a lot of pleasure out of doing so, because it's working adequately and my work priority is to get results out of it.) That in turn leads to a hiring situation where the preference is to have mathematicians or physicists who can program, rather than programmers who can learn the maths. It doesn't help that because of the way academic funding is made available, the pay scales mean that it's not really possible to attract top-level developers (unless they have some kind of keen moral desire to work on academic research); in addition, you usually have to hire them as PhD students or postdocs or so on (I've also seen masters' students roped in to this end), which obviously constrains the range of people that you can hire and the range of skills that will be available, and also the degree of commitment these people can put into long-term vision and maintenance of the codebase. There's also a training problem -- in my experience, most physics undergraduates are given a crash course in C++ in their first year and not much in the way of real computer science or development training. In my case as a maths undergraduate the first opportunity to learn programming was in the 3rd year of my degree course, and it was a crash course in a very narrow subset of C dedicated towards numerical programming. And if (like me) you then go on into research, you largely have to self-teach, which can lead to some very idiosyncratic approaches. I hope that this will change, because programming is now an absolutely essential part of just about every avenue of scientific research. But as it stands, it's a serious problem.
Jun 02 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 2 June 2013 21:46, Joseph Rushton Wakeling
<joseph.wakeling webdrake.net>wrote:

 On 06/02/2013 08:33 AM, Manu wrote:
 Most of these guys are mathematicians and physicists first, and
programmers second. You've hit the nail on the head, but it's also a question of priorities. It's _essential_ that the maths or physics be understood and done right.
Well this is another classic point actually. I've been asked by my friends at Cambridge to give their code a once-over for them on many occasions, and while I may not understand exactly what their code does, I can often spot boat-loads of simple functional errors. Like basic programming bugs; out-by-ones, pointer logic fails, clear lack of understanding of floating point, or logical structure that will clearly lead to incorrect/unexpected edge cases. And it blows my mind that they then run this code on their big sets of data, write some big analysis/conclusions, and present this statistical data in some journal somewhere, and are generally accepted as an authority and taken seriously! *brain asplode* I can tell you I usually offer more in the way of fixing basic logical errors than actually making it run faster ;) And they don't come to me with everything, just the occasional thing that they have a hunch should probably be faster than it is. I hope my experience there isn't too common, but I actually get the feeling it's more common that you'd like to think! This is a crowd I'd actually love to promote D to! But the tools they need aren't all there yet... It's essential that the programs correctly reflect that maths or physics.
  It's
 merely _desirable_ that the programs run as fast as possible, or be well
 designed from a maintenance point of view, or any of the other things that
 matter to trained software developers.  (In my day job I have to
 continually
 force myself to _not_ refactor or optimize my code, even though I'd get a
 lot of
 pleasure out of doing so, because it's working adequately and my work
 priority
 is to get results out of it.)

 That in turn leads to a hiring situation where the preference is to have
 mathematicians or physicists who can program, rather than programmers who
 can
 learn the maths.  It doesn't help that because of the way academic funding
 is
 made available, the pay scales mean that it's not really possible to
 attract
 top-level developers (unless they have some kind of keen moral desire to
 work on
 academic research); in addition, you usually have to hire them as PhD
 students
 or postdocs or so on (I've also seen masters' students roped in to this
 end),
 which obviously constrains the range of people that you can hire and the
 range
 of skills that will be available, and also the degree of commitment these
 people
 can put into long-term vision and maintenance of the codebase.

 There's also a training problem -- in my experience, most physics
 undergraduates
 are given a crash course in C++ in their first year and not much in the
 way of
 real computer science or development training.  In my case as a maths
 undergraduate the first opportunity to learn programming was in the 3rd
 year of
 my degree course, and it was a crash course in a very narrow subset of C
 dedicated towards numerical programming.  And if (like me) you then go on
 into
 research, you largely have to self-teach, which can lead to some very
 idiosyncratic approaches.
Yeah, this is an interesting point. These friends of mine all write C code, not even C++. Why is that? I guess it's promoted, because they're supposed to be into the whole 'HPC' thing, but C is really not a good language for doing maths! I see stuff like this: float ***cubicMatrix = (float***)malloc(sizeof(float**)depth); for(int z=0; z<width; z++) { cubicMatrix[z] = (float**)malloc(sizeof(float**)*height); for(int y=0; y<height; y++) { cubicMatrix[z][y] = (float*)malloc(sizeof(float*)*width); } } Seriously, float***. Each 1d row is an individual allocation! And then somewhere later on they want to iterate a column rather than a row, and get confused about the pointer arithmetic (well, maybe not precisely that, but you get the idea). I hope that this will change, because programming is now an absolutely
 essential
 part of just about every avenue of scientific research.  But as it stands,
 it's
 a serious problem.
Jun 02 2013
next sibling parent reply "Roy Obena" <roy.u gmail.com> writes:
On Sunday, 2 June 2013 at 14:34:43 UTC, Manu wrote:
 On 2 June 2013 21:46, Joseph Rushton Wakeling
 Well this is another classic point actually. I've been asked by 
 my friends
 at Cambridge to give their code a once-over for them on many 
 occasions, and
 while I may not understand exactly what their code does, I can 
 often spot
 boat-loads of simple functional errors. Like basic programming 
 bugs;
 out-by-ones, pointer logic fails, clear lack of understanding 
 of floating
 point, or logical structure that will clearly lead to 
 incorrect/unexpected
 edge cases.
 And it blows my mind that they then run this code on their big 
 sets of
 data, write some big analysis/conclusions, and present this 
 statistical
 data in some journal somewhere, and are generally accepted as 
 an authority
 and taken seriously!
You're making this up. I'm sure they do a lot of data-driven tests or simulations that make most errors detectable. They may not be savvy programmers, and their programs may not be error-free, but boat-loads of errors? C'mon.
Jun 02 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 2 June 2013 at 15:53:58 UTC, Roy Obena wrote:
 On Sunday, 2 June 2013 at 14:34:43 UTC, Manu wrote:
 On 2 June 2013 21:46, Joseph Rushton Wakeling
 Well this is another classic point actually. I've been asked 
 by my friends
 at Cambridge to give their code a once-over for them on many 
 occasions, and
 while I may not understand exactly what their code does, I can 
 often spot
 boat-loads of simple functional errors. Like basic programming 
 bugs;
 out-by-ones, pointer logic fails, clear lack of understanding 
 of floating
 point, or logical structure that will clearly lead to 
 incorrect/unexpected
 edge cases.
 And it blows my mind that they then run this code on their big 
 sets of
 data, write some big analysis/conclusions, and present this 
 statistical
 data in some journal somewhere, and are generally accepted as 
 an authority
 and taken seriously!
You're making this up. I'm sure they do a lot of data-driven tests or simulations that make most errors detectable. They may not be savvy programmers, and their programs may not be error-free, but boat-loads of errors? C'mon.
I really wish he was making it up. Sadly, he's not. A lot of HPC scientific code is, at best, horribly fragile.
Jun 02 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/2/13 12:16 PM, John Colvin wrote:
 A lot of HPC scientific code is, at best, horribly fragile.
Reminds me of that internal joke at www.llnl.gov. "If we make a mistake, millions of people will live." True story. Andrei
Jun 02 2013
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/02/2013 06:44 PM, Andrei Alexandrescu wrote:
 On 6/2/13 12:16 PM, John Colvin wrote:
 A lot of HPC scientific code is, at best, horribly fragile.
Reminds me of that internal joke at www.llnl.gov. "If we make a mistake, millions of people will live." True story.
A story I heard from a lecturer of mine -- one of the interesting little factors of the Cold War was how Soviet Russia managed to keep parity with NATO in terms of missile guidance and other computer-related military technologies. The US had far superior hardware, so it was both a concern and a mystery. What came out after the end of the Cold War was quite impressive -- Soviet scientists had realized very well that they couldn't compete on the hardware front and so had focused a very intense effort on really, really efficient algorithms that squeezed every drop of performance out of the hardware they had available, far greater performance than Western computer scientists had ever imagined possible.
Jun 02 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/02/2013 05:53 PM, Roy Obena wrote:
 You're making this up. I'm sure they do a lot of data-driven
 tests or simulations that make most errors detectable. They may
 not be savvy programmers, and their programs may not be
 error-free, but boat-loads of errors? C'mon.
I don't think he's making this up. I would not want to make any assumptions about any particular institutions, and I would like to believe that institutions with large-scale, long-term computational projects have better practices, but I think that most people in maths and physics research have very limited experience of good code design and testing practices (working with D, and its unit-testing framework and contract guarantees, has certainly been an eye-opener for me). Generally speaking it's true that in maths and physics there are often either theoretical calculations or empirical data points for you to compare your computational results to, so you can usually confirm that your program is doing what it's supposed to, but not always. There may not be boat-loads of errors in terms of output, but I bet there are boat-loads of loopholes that will result in insane mistakes if the input were to step outside the use-cases or parameter constraints the researcher has considered. I don't exclude my own code from that criticism, and the reason it's tolerated is because it's often the quickest way to get to working code; you know that you have to constrain parameters in such-and-such a way for it to work, and you know that you _will_ constrain yourself accordingly. But of course it's easy to shoot yourself in the foot when you start tweaking things. D's assert() and enforce() functions, and contract checks, are very useful here.
Jun 02 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 3 June 2013 01:53, Roy Obena <roy.u gmail.com> wrote:

 On Sunday, 2 June 2013 at 14:34:43 UTC, Manu wrote:

 On 2 June 2013 21:46, Joseph Rushton Wakeling
Well this is another classic point actually. I've been asked by my friends
 at Cambridge to give their code a once-over for them on many occasions,
 and
 while I may not understand exactly what their code does, I can often spot
 boat-loads of simple functional errors. Like basic programming bugs;
 out-by-ones, pointer logic fails, clear lack of understanding of floating
 point, or logical structure that will clearly lead to incorrect/unexpected
 edge cases.
 And it blows my mind that they then run this code on their big sets of
 data, write some big analysis/conclusions, and present this statistical
 data in some journal somewhere, and are generally accepted as an authority
 and taken seriously!
You're making this up. I'm sure they do a lot of data-driven tests or simulations that make most errors detectable. They may not be savvy programmers, and their programs may not be error-free, but boat-loads of errors? C'mon.
I'm really not. I mean, this won't all appear in the same function, but I've seen all these sorts of errors on more than one occasion. I suspect that in most cases it will just increase their perceived standard deviation, otherwise I'm sure they'd notice it's all wrong and look for their bugs. But it's sad if a study shows higher than true standard deviation because of code errors, or worse, if it does influence the averages slightly, but they feel the result is plausible within their expected tolerance. The scariest state is the idea that their code is *almost correct*. Clearly, they should be using D ;)
Jun 02 2013
prev sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Sunday, 2 June 2013 at 14:34:43 UTC, Manu wrote:
 Yeah, this is an interesting point. These friends of mine all 
 write C code,
 not even C++.
Maybe you should mention to them Julia. It's quite a good scientific language.
Jun 04 2013
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/02/2013 04:34 PM, Manu wrote:
 Well this is another classic point actually. I've been asked by my friends at
 Cambridge to give their code a once-over for them on many occasions, and while
I
 may not understand exactly what their code does, I can often spot boat-loads of
 simple functional errors. Like basic programming bugs; out-by-ones, pointer
 logic fails, clear lack of understanding of floating point, or logical
structure
 that will clearly lead to incorrect/unexpected edge cases.
 And it blows my mind that they then run this code on their big sets of data,
 write some big analysis/conclusions, and present this statistical data in some
 journal somewhere, and are generally accepted as an authority and taken
seriously!
 
 *brain asplode*
Yes, I can imagine. I've seen more than enough researcher-written code that made my own brain explode, and I don't consider myself in any way expert in program design. You have to hope that there were sufficient checks against empirical or theoretical results, that at least any error was minimized ... What bothers me more than the "trust" issues about the code is that very often, the code is never made available for review. It's astonishing how timid journals and funding organizations are about trying to resolve this.
 I can tell you I usually offer more in the way of fixing basic logical errors
 than actually making it run faster ;)
 And they don't come to me with everything, just the occasional thing that they
 have a hunch should probably be faster than it is.
I certainly went through a phase of extreme speed-related paranoia in my C programming past, and I think it's a common trait. Speed is the thing you worry about because it's the most observable problem. And of course C tends to bias you towards daft micro-optimization rather than basic things like getting the algorithms right.
 I hope my experience there isn't too common, but I actually get the feeling
it's
 more common that you'd like to think!
 This is a crowd I'd actually love to promote D to! But the tools they need
 aren't all there yet...
I think it's probably very common, exacerbated by the fact that most researchers (not just in maths and physics) are amateur, self-taught programmers with limited time to study the art of programming for itself, and limited opportunities for formal training, who are under great pressure to produce research results quickly and continuously. I do what I can to promote D to colleagues, and I've had one or two people come up to me spontaneously and ask about it (because they've seen my name on the mailing lists), so I think the interest is growing and it will get there. 90% of the worries I have with it concern 3rd-party libraries -- obviously C/C++ gives you access to a much wider range of stuff without having to write bindings (which to me at least, is a scary prospect, not so much the technical side as the hassle and time requirement of having to write and maintain them). The other 10% of the worries are about potential issues with the standard library that might result in incorrect results (actually, I'm pretty confident here really, but there are a number of known issues in std.random which I make sure to work around).
 Yeah, this is an interesting point. These friends of mine all write C code, not
 even C++.
 Why is that?
 I guess it's promoted, because they're supposed to be into the whole 'HPC'
 thing, but C is really not a good language for doing maths!
Well, I can't speak for proper HPC because it's not my field (I work in complexity science, which does involve a lot of intensive computation but has developed somewhat independently of traditional HPC fields). However, my guess would be that it's a mix of what people are first trained in, together with a measure of conservatism and "What's the lowest common denominator?". I also don't think I can stress enough how true it is that mathematicians, physicists and other researchers tend to be trained to program _in C_, or in C++, or in FORTRAN, rather than _how to program_. Speed concerns might be a factor, as C++ offers you rather more ways to shoot yourself in the foot speed-wise than C -- there might be some prejudice about C being the one to use for really heavy-duty computation, though I imagine that says more about the programmer's skill than the reality of the languages. In my own field the norm these days seems to be a hybrid of C/C++ and Python -- the former for the high-intensity stuff, the latter for convenience or to have a friendly surface from which to call the high-intensity routines, although libraries like NumPy seem to be challenging the C dominance for some intense calculations -- I'm seeing an increasing number of Python libraries being written and used. That said, I don't think language lock-in is unique to mathematicians and physicists. Many of the computer scientists I've worked with have been wedded to Java with a strength that is astonishing given that you'd expect them to be trained well enough to really appreciate the variety of choices available. (In my experience, mathematicians and physicists tend to be far more comfortable with the command line and in programming without an IDE. That may of course explain some of the errors, too:-)
 I see stuff like this:
 float ***cubicMatrix = (float***)malloc(sizeof(float**)depth);
 for(int z=0; z<width; z++)
 {
   cubicMatrix[z] = (float**)malloc(sizeof(float**)*height);
   for(int y=0; y<height; y++)
   {
     cubicMatrix[z][y] = (float*)malloc(sizeof(float*)*width);
   }
 }
 
 Seriously, float***. Each 1d row is an individual allocation!
 And then somewhere later on they want to iterate a column rather than a row,
and
 get confused about the pointer arithmetic (well, maybe not precisely that, but
 you get the idea).
Oh yes, I've written code like that. :-P I can only say that it's the way that I was shown how to create matrices in C. I can't remember the context; possibly I read it in a book, or possibly it was by browsing other code, possibly it was in lecture notes. That said, it's the _obvious_ way to create a matrix (if you think of a matrix as being an entity whose values are accessed by indices [x][y][z]), and if you're not trained in program design, the obvious way tends to be the thing you pick, and it seems to work, so ... I mean, I guess (I've never had call to do it, so never looked into the detail) the way to _really_ build an effective matrix design is to have a single array and wrap it with functions that translate x, y, z indices to appropriate array locations. But you wouldn't think of that as a novice programmer, and unless someone teaches you, the multi-level alloc solution probably seems to work well enough that you never think to question it. (For the avoidance of doubt: I'm at least experienced enough to have questioned it before this email exchange:-) That kind of "works well enough" probably explains 99% of the programming faults made by researchers, together with the fact that they very rarely encounter anyone with the experience to question their approach -- and of course, their own sense of the problems within their code can (as you've experienced) be very different from the problems an experienced developer will focus on. I have to say that it'd be very tempting to try and organize an annual event (and maybe also an online space) where researchers using computation are brought together with genuinely expert developers for lectures, brainstorming and collaboration, with the specific aim of getting away from these kinds of habitual errors.
Jun 02 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Joseph Rushton Wakeling:

 I have to say that it'd be very tempting to try and organize an 
 annual event
 (and maybe also an online space) where researchers using 
 computation are brought
 together with genuinely expert developers for lectures, 
 brainstorming and
 collaboration, with the specific aim of getting away from these 
 kinds of habitual errors.
It seems a nice idea. But I think it's better for this event to be also a chance for developers to learn something from researchers, things like the value of having a control, why people do scientific research, etc :-) Bye, bearophile
Jun 02 2013
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/02/2013 11:31 PM, bearophile wrote:
 It seems a nice idea. But I think it's better for this event to be also a
chance
 for developers to learn something from researchers, things like the value of
 having a control, why people do scientific research, etc :-)
Sure. :-)
Jun 02 2013
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/31/2013 08:34 AM, Manu wrote:
 What's taking the most time?
 The lighting loop is so template-tastic, I can't get a feel for how fast that
 loop would be.
Hah, I found this out the hard way recently -- have been doing some experimental reworking of code where some key inner functions were templatized, and it had a nasty effect on performance. I'm guessing it made it impossible for the compilers to inline these functions :-(
May 31 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/31/2013 12:58 PM, Joseph Rushton Wakeling wrote:
 On 05/31/2013 08:34 AM, Manu wrote:
 What's taking the most time?
 The lighting loop is so template-tastic, I can't get a feel for how fast that
 loop would be.
Hah, I found this out the hard way recently -- have been doing some experimental reworking of code where some key inner functions were templatized, and it had a nasty effect on performance. I'm guessing it made it impossible for the compilers to inline these functions :-(
That wouldn't make any sense though, since after template expansion there is no difference between the generated version and a particular handwritten version.
May 31 2013
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/31/2013 01:05 PM, Timon Gehr wrote:
 That wouldn't make any sense though, since after template expansion there is no
 difference between the generated version and a particular handwritten version.
That's what I'd assumed too, but there _is_ a speed difference. I'm open to suggestions as to why. Compare these profile results for the core inner function -- the original (even though there's a 'New' in there somewhere): % cumulative self self total time seconds seconds calls s/call s/call name 93.33 4.69 4.69 4084637 0.00 0.00 _D8infected5model115__T13NewSimulationS248infected5model8StateSISS238infected5model7SeedSISS388infected5model21NewUpdateMeanFieldSISTdZ13NewSimulation19__T11UpdateStateTdZ6updateMFKC8infected5model15__T8StateSISTdZ8StateSISKxC8infected5model14__T7SeedSISTdZ7SeedSISZd ... and the newer version: % cumulative self self total time seconds seconds calls s/call s/call name 92.73 5.23 5.23 4078287 0.00 0.00 _D8infected5model292__T10SimulationS358infected5model18UpdateMeanFieldSISTC8infected5model15__T8StateSISTdZ8StateSISTC8infected5model164__T7SeedSISTdTAyAS3std8typecons50__T5TupleTmVAyaa2_6964TdVAyaa9_696e666c75656e6365Z5TupleTyS3std8typecons50__T5TupleTmVAyaa2_6964TdVAyaa9_696e666c75656e6365Z5TupleZ7SeedSISTydZ10Simulation17__T11UpdateStateZ163__T6updateTdTAyAS3std8typecons50__T5TupleTmVAyaa2_6964TdVAyaa9_696e666c75656e6365Z5TupleTyS3std8typecons50__T5TupleTmVAyaa2_6964TdVAyaa9_696e666c75656e6365Z5TupleZ6updateMFNbNfKC8infected5model15__T8StateSISTdZ8StateSISKxC8infected5model164__T7SeedSISTdTAyAS3std8typecons50__T5TupleTmVAyaa2_6964TdVAyaa9_696e666c75656e6365Z5TupleTyS3std8typecons50__T5TupleTmVAyaa2_6964TdVAyaa9_696e666c75656e6365Z5TupleZ7SeedSISZd I'm not sure what, other than a change of template design, could be responsible here. The key bits of code follow -- the original version: mixin template UpdateState(T) { T update(ref StateSIS!T st, const ref SeedSIS!T sd) { T d = to!T(0); static T[] sick; sick.length = st.infected.length; sick[] = st.infected[]; foreach(i; 0..sick.length) { T noTransmission = to!T(1); foreach(link; sd.network[i]) noTransmission *= (to!T(1) - sick[link.id] * link.influence); T getSick = (to!T(1) - sick[i]) * (sd.susceptible[i] + (to!T(1) - sd.susceptible[i]) * (to!T(1) - noTransmission)); T staySick = sick[i] * (to!T(1) - sd.recover[i]); st.infected[i] = (to!T(1) - sd.immune[i]) * (getSick + staySick); assert(to!T(0) <= st.infected[i]); assert(st.infected[i] <= to!T(1)); d = max(abs(st.infected[i] - sick[i]), d); } return d; } } ... and for clarity, the StateSIS and SeedSIS classes: class StateSIS(T) { T[] infected; this(){} this(T[] inf) { infected = inf; } auto size() property pure const nothrow { return infected.length; } T infection() property pure const nothrow { return reduce!"a+b"(to!T(0), infected); } } class SeedSIS(T) { T[] immune; T[] susceptible; T[] recover; Link!T[][] network; this() {} this(T[] imm, T[] sus, T[] rec, Link!T[][] net) { immune = imm; susceptible = sus; recover = rec; network = net; } auto size() property pure const nothrow in { assert(immune.length == susceptible.length); assert(immune.length == recover.length); assert(immune.length == network.length); } body { return immune.length; } } ... and the "Link" template: template Link(T) { alias Tuple!(size_t, "id", T, "influence") Link; } ... and now for comparison the new versions: mixin template UpdateState() { T update(T, N : L[][], L)(ref StateSIS!T st, const ref SeedSIS!(T, N, L) sd) { T d = to!T(0); static T[] sick; sick.length = st.infected.length; sick[] = st.infected[]; foreach(i; 0..sick.length) { T noTransmission = to!T(1); foreach(link; sd.network[i]) noTransmission *= (to!T(1) - sick[link.id] * link.influence); T getSick = (to!T(1) - sick[i]) * (sd.susceptible[i] + (to!T(1) - sd.susceptible[i]) * (to!T(1) - noTransmission)); T staySick = sick[i] * (to!T(1) - sd.recover[i]); st.infected[i] = (to!T(1) - sd.immune[i]) * (getSick + staySick); assert(to!T(0) <= st.infected[i]); assert(st.infected[i] <= to!T(1)); d = max(abs(st.infected[i] - sick[i]), d); } return d; } } class StateSIS(T) { T[] infected; this() {} this(T[] inf) { infected = inf; } auto size() property pure const nothrow { return infected.length; } T infection() property pure const nothrow { return reduce!"a+b"(to!T(0), infected); } } auto stateSIS(T)(T[] inf) { return new StateSIS!T(inf); } class SeedSIS(T, Network : L[][], L) { T[] immune; T[] susceptible; T[] recover; Network network; this() {} this(T[] imm, T[] sus, T[] rec, Network net) { immune = imm; susceptible = sus; recover = rec; network = net; } auto size() property pure const nothrow in { assert(immune.length == susceptible.length); assert(immune.length == recover.length); assert(immune.length == network.length); } body { return immune.length; } } auto seedSIS(T, Network : L[][], L)(T[] imm, T[] sus, T[] rec, Network net) { return new SeedSIS!(T, Network, L)(imm, sus, rec, net); } ... note that the Network that is passed to SeedSIS is still always a Link!T[][].
May 31 2013
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On 31 May 2013 21:05, Timon Gehr <timon.gehr gmx.ch> wrote:

 On 05/31/2013 12:58 PM, Joseph Rushton Wakeling wrote:

 On 05/31/2013 08:34 AM, Manu wrote:

 What's taking the most time?
 The lighting loop is so template-tastic, I can't get a feel for how fast
 that
 loop would be.
Hah, I found this out the hard way recently -- have been doing some experimental reworking of code where some key inner functions were templatized, and it had a nasty effect on performance. I'm guessing it made it impossible for the compilers to inline these functions :-(
That wouldn't make any sense though, since after template expansion there is no difference between the generated version and a particular handwritten version.
Assuming that you would hand-write exactly the same code as the template expansion... Typically template expansion leads to countless temporary redundancies, which you expect the compiler to try and optimise away, but it's not always able to do so, especially if there is an if() nearby, or worse, a pointer dereference.
May 31 2013
parent reply "finalpatch" <fengli gmail.com> writes:
I actually have some experience with C++ template
meta-programming in HD video codecs. My experience is that it is
possible for generic code through TMP to match or even beat hand
written code. Modern C++ compilers are very good, able to
optimize away most of the temporary variables resulting very
compact object code, provides you can avoid branches and keep the
arguments const refs as much as possible. A real example is my
TMP generic codec beat the original hand optimized c/asm version
(both use sse intrinsics) by as much as 30% with only a fraction
of the line of code. Another example is the Eigen linear algebra
library, through template meta-programming it is able to match
the speed of Intel MKL.

D is very strong at TMP, it provides a lot more tools
specifically designed for TMP, that is vastly superior than C++
which relies on abusing the templates. This is actually the main
reason drawing me to D: TMP in a more pleasant way. IMO one thing
D needs to address is less surprises, eg. innocent looking code
like v[] = [x,x,x] shouldn't cause major performance hit. In c++
memory allocation is explicit, either operator new or malloc, or
indirectly through a method call, otherwise the language would
not do heap allocation for you.

On Friday, 31 May 2013 at 11:51:04 UTC, Manu wrote:
 Assuming that you would hand-write exactly the same code as the 
 template
 expansion...
 Typically template expansion leads to countless temporary 
 redundancies,
 which you expect the compiler to try and optimise away, but 
 it's not always
 able to do so, especially if there is an if() nearby, or worse, 
 a pointer
 dereference.
May 31 2013
next sibling parent Manu <turkeyman gmail.com> writes:
On 31 May 2013 23:07, finalpatch <fengli gmail.com> wrote:

 I actually have some experience with C++ template
 meta-programming in HD video codecs. My experience is that it is
 possible for generic code through TMP to match or even beat hand
 written code. Modern C++ compilers are very good, able to
 optimize away most of the temporary variables resulting very
 compact object code, provides you can avoid branches and keep the
 arguments const refs as much as possible. A real example is my
 TMP generic codec beat the original hand optimized c/asm version
 (both use sse intrinsics) by as much as 30% with only a fraction
 of the line of code. Another example is the Eigen linear algebra
 library, through template meta-programming it is able to match
 the speed of Intel MKL.
Just to clarify, I'm not trying to say templates are slow because they're tempaltes. There's no reason carefully crafted template code couldn't be identical to hand crafted code. What I am saying, is that it introduces the possibility for countless subtle details to get in the way. If you want maximum performance from templates, you often need to be really good at expanding the code in your mind, and visualising it all in expanded context, so you can then reason whether anything is likely to get in the way of the optimiser or not. A lot of people don't possess this skill, and for good reason, it's hard! It usually takes considerable time to optimise template code, and optimised template code may often only be optimal in the context you tested against. At some point, depending on the complexity of your code, it might just be easier/less time consuming to write the code directly. It's a fine line, but I've seen so much code that takes it WAAAAY too far. There's always the unpredictable element too. Imagine a large-ish template function, and one very small detail inside is customised of otherwise identical functions. Let's say 2 routines are generated for int and long; the cost of casting int -> long and calling the long function in both cases is insignificant, but using templates, your exe just got bigger, branches less predictable, icache got more noisy, and there's no way to profile for loss of performance introduced this way. In-fact, the profiler will typically erroneously lead you to believe your code is FASTER, but it results in code that may be slower at net. I'm attracted to D for the power of it's templates too, but that attraction is all about simplicity and readability. In D, you can do more with less. The goal is not to use more and more templates, but make the few templates I use, more readable and maintainable. D is very strong at TMP, it provides a lot more tools
 specifically designed for TMP, that is vastly superior than C++
 which relies on abusing the templates. This is actually the main
 reason drawing me to D: TMP in a more pleasant way. IMO one thing
 D needs to address is less surprises, eg. innocent looking code
 like v[] = [x,x,x] shouldn't cause major performance hit. In c++
 memory allocation is explicit, either operator new or malloc, or
 indirectly through a method call, otherwise the language would
 not do heap allocation for you.
Yeah well... I have a constant inner turmoil with this in D. I want to believe the GC is the future, but I'm still trying to convince myself of that (and I think the GC is losing the battle at the moment). Fortunately you can avoid the GC fairly effectively (if you forego large parts of phobos!). Buy things like the array initialisation are inexcusable. Array literals should NOT allocate, this desperately needs to be fixed. And scope/escape analysis, so local dynamic arrays can be lowered onto the stack in self-contained situations. That's the biggest source of difficult-to-control allocations in my experience. On Friday, 31 May 2013 at 11:51:04 UTC, Manu wrote:
 Assuming that you would hand-write exactly the same code as the template
 expansion...
 Typically template expansion leads to countless temporary redundancies,
 which you expect the compiler to try and optimise away, but it's not
 always
 able to do so, especially if there is an if() nearby, or worse, a pointer
 dereference.
May 31 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/31/13 9:07 AM, finalpatch wrote:
 D is very strong at TMP, it provides a lot more tools
 specifically designed for TMP, that is vastly superior than C++
 which relies on abusing the templates. This is actually the main
 reason drawing me to D: TMP in a more pleasant way. IMO one thing
 D needs to address is less surprises, eg. innocent looking code
 like v[] = [x,x,x] shouldn't cause major performance hit. In c++
 memory allocation is explicit, either operator new or malloc, or
 indirectly through a method call, otherwise the language would
 not do heap allocation for you.
It would be great if we addressed that in 2.064. I'm sure I've seen the report in bugzilla, but the closest I found were: http://d.puremagic.com/issues/show_bug.cgi?id=9335 http://d.puremagic.com/issues/show_bug.cgi?id=8449 Andrei
May 31 2013
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 31 May 2013 10:49:21 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/31/13 9:07 AM, finalpatch wrote:
 D is very strong at TMP, it provides a lot more tools
 specifically designed for TMP, that is vastly superior than C++
 which relies on abusing the templates. This is actually the main
 reason drawing me to D: TMP in a more pleasant way. IMO one thing
 D needs to address is less surprises, eg. innocent looking code
 like v[] = [x,x,x] shouldn't cause major performance hit. In c++
 memory allocation is explicit, either operator new or malloc, or
 indirectly through a method call, otherwise the language would
 not do heap allocation for you.
It would be great if we addressed that in 2.064. I'm sure I've seen the report in bugzilla, but the closest I found were: http://d.puremagic.com/issues/show_bug.cgi?id=9335 http://d.puremagic.com/issues/show_bug.cgi?id=8449
There was this: http://d.puremagic.com/issues/show_bug.cgi?id=2356 I know Don has suggested in the past that all array literals be immutable, like strings, and I agree with that. But it would be a huge breaking change. I agree with finalpatch that array literals allocating is not obvious or expected in many cases. I wonder if the compiler can prove that an array literal isn't referenced outside the function (or statement at least), it can allocate it on the stack instead of the heap? That would be a huge improvement, and good middle ground. -Steve
May 31 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 31 May 2013 20:58, Joseph Rushton Wakeling
<joseph.wakeling webdrake.net>wrote:

 On 05/31/2013 08:34 AM, Manu wrote:
 What's taking the most time?
 The lighting loop is so template-tastic, I can't get a feel for how fast
that
 loop would be.
Hah, I found this out the hard way recently -- have been doing some experimental reworking of code where some key inner functions were templatized, and it had a nasty effect on performance. I'm guessing it made it impossible for the compilers to inline these functions :-(
I find that using templates actually makes it more likely for the compiler to properly inline. But I think the totally generic expressions produce cases where the compiler is considering too many possibilities that inhibit many optimisations. It might also be that the optimisations get a lot more complex when the code fragments span across a complex call tree with optimisation dependencies on non-deterministic inlining. One of the most important jobs for the optimiser is code re-ordering. Generic code is often written in such a way that makes it hard/impossible for the optimiser to reorder the flattened code properly. Hand written code can have branches and memory accesses carefully placed at the appropriate locations. Generic code will usually package those sorts of operations behind little templates that often flatten out in a different order. The optimiser is rarely able to re-order code across if statements, or pointer accesses. __restrict is very important in generic code to allow the optimiser to reorder across any indirection, otherwise compilers typically have to be conservative and presume that something somewhere may have changed the destination of a pointer, and leave the order as the template expanded. Sadly, D doesn't even support __restrict, and nobody ever uses it in C++ anyway. I've always has better results with writing precisely what I intend the compiler to do, and using __forceinline where it needs a little extra encouragement.
May 31 2013
parent "Dicebot" <m.strashun gmail.com> writes:
On Friday, 31 May 2013 at 11:49:05 UTC, Manu wrote:
 I find that using templates actually makes it more likely for 
 the compiler
 to properly inline. But I think the totally generic expressions 
 produce
 cases where the compiler is considering too many possibilities 
 that inhibit
 many optimisations.
 It might also be that the optimisations get a lot more complex 
 when the
 code fragments span across a complex call tree with optimisation
 dependencies on non-deterministic inlining.

 One of the most important jobs for the optimiser is code 
 re-ordering.
 Generic code is often written in such a way that makes it 
 hard/impossible
 for the optimiser to reorder the flattened code properly.
 Hand written code can have branches and memory accesses 
 carefully placed at
 the appropriate locations.
 Generic code will usually package those sorts of operations 
 behind little
 templates that often flatten out in a different order.
 The optimiser is rarely able to re-order code across if 
 statements, or
 pointer accesses. __restrict is very important in generic code 
 to allow the
 optimiser to reorder across any indirection, otherwise 
 compilers typically
 have to be conservative and presume that something somewhere 
 may have
 changed the destination of a pointer, and leave the order as 
 the template
 expanded. Sadly, D doesn't even support __restrict, and nobody 
 ever uses it
 in C++ anyway.

 I've always has better results with writing precisely what I 
 intend the
 compiler to do, and using __forceinline where it needs a little 
 extra
 encouragement.
Thanks for valuable input. Have never had a pleasure to actually try templates in performance-critical code and this a good stuff to remember about. Have added to notes.
May 31 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/31/2013 01:48 PM, Manu wrote:
 I find that using templates actually makes it more likely for the compiler to
 properly inline. But I think the totally generic expressions produce cases
where
 the compiler is considering too many possibilities that inhibit many
optimisations.
 It might also be that the optimisations get a lot more complex when the code
 fragments span across a complex call tree with optimisation dependencies on
 non-deterministic inlining.
Thanks for the detailed advice. :-) There are two particular things I noted about my own code. One is that whereas in the original the template variables were very simple (just a floating-point type) in the new version they are more complex structures that are indeed more generic (the idea was to enable the code to handle both mutable and immutable forms of one particular data structure). The second is that the templatization gets moved from the mixin to the functions themselves. I guess that the mixin has the effect of copy-pasting _as if_ I was just writing precisely what I intended.
May 31 2013
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Saturday, June 01, 2013 09:04:50 Manu wrote:
 **applause**
 Someone other than me said it, out loud!
 This is a magnificent day! :)
Well, the discussions at dconf convinced me. Certainly, at this point, I think that the only semi-viable excuse for not making functions non-virtual by default is the code breakage that it would cause, and given how surprisingly minimal that is, I think that it's definitely worth it - especially when the kind of folks whose code Walter is most worried about breaking are the guys most interested in the change. - Jonathan M Davis
May 31 2013
next sibling parent reply Juan Manuel Cabo <juanmanuel.cabo gmail.com> writes:
On 05/31/2013 10:27 PM, Jonathan M Davis wrote:
 On Saturday, June 01, 2013 09:04:50 Manu wrote:
 **applause**
 Someone other than me said it, out loud!
 This is a magnificent day! :)
Well, the discussions at dconf convinced me. Certainly, at this point, I think that the only semi-viable excuse for not making functions non-virtual by default is the code breakage that it would cause, and given how surprisingly minimal that is, I think that it's definitely worth it - especially when the kind of folks whose code Walter is most worried about breaking are the guys most interested in the change. - Jonathan M Davis
Making everything final by default would IMO kind of break automated mock classes generation for unit testing, automatic proxy class generation for DB entities, and other OOP niceities. And it wasn't just marking methods final which got the D version of the raytracer in this thread faster than the C++ version in the end. (it was a combination of four or five things, which involved a bit of unrolling, avoiding array literals, and so on). --jm
May 31 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 1 June 2013 at 02:58:59 UTC, Juan Manuel Cabo wrote:
 Well, the discussions at dconf convinced me. Certainly, at 
 this point, I think that the only semi-viable excuse for not 
 making functions non-virtual by default is the code breakage 
 that it would cause, and given how surprisingly minimal that 
 is, I think that it's definitely worth it - especially when 
 the kind of folks whose code Walter is most worried about 
 breaking are the guys most interested in the change.
 
 - Jonathan M Davis
 
Making everything final by default would IMO kind of break automated mock classes generation for unit testing, automatic proxy class generation for DB entities, and other OOP niceities.
Yeah, everybody seems to ignore that. OOP is slow in general, due to excess of indirections, so if its loose its benefits . . .
May 31 2013
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 01 Jun 2013 06:49:47 +0200
schrieb "deadalnix" <deadalnix gmail.com>:

 On Saturday, 1 June 2013 at 02:58:59 UTC, Juan Manuel Cabo wrote:
 Making everything final by default would IMO kind of break
 automated mock classes generation for unit testing,
 automatic proxy class generation for DB entities, and
 other OOP niceities.
Yeah, everybody seems to ignore that. OOP is slow in general, due to excess of indirections, so if its loose its benefits . . .
Does it really change anything? Today people forget the final keyword and tomorrow they forget the virtual keyword? :D What you prefer here depends on what languages you are used to do (e.g. C++, Java), how much you use OOP for anything, your use of unittesting and mock objects, your need for speed etc. In any case D is often compared to C++ for memory management, speed, executable size etc. And D is trying to appeal to too many different fields of software development to be perfect for any of them. So maybe this should just be solved in a democratic vote after hearing the pros and cons. :) -- Marco
Jun 01 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 1 June 2013 at 21:14:43 UTC, Marco Leise wrote:
 Does it really change anything? Today people forget the
 final keyword and tomorrow they forget the virtual
 keyword? :D

 What you prefer here depends on what languages you are used to
 do (e.g. C++, Java), how much you use OOP for anything, your
 use of unittesting and mock objects, your need for speed etc.
I actually use that quite a lot. Additionally compiler technology exists now to finalize methods that aren't overriden. It is also possible to generate code that replace the virtual dispatch by an simple if if only 2 overrides exists, or even provide a fast path for the most common override, only conditionally doing the virtual dispatch. In other terms, as long as the compiler know about all the overrides, it can do better than the programmer manually annotating final/virtual.
Jun 02 2013
prev sibling parent reply Juan Manuel Cabo <juanmanuel.cabo gmail.com> writes:
On 05/31/2013 10:27 PM, Jonathan M Davis wrote:
 On Saturday, June 01, 2013 09:04:50 Manu wrote:
 **applause**
 Someone other than me said it, out loud!
 This is a magnificent day! :)
Well, the discussions at dconf convinced me. Certainly, at this point, I think that the only semi-viable excuse for not making functions non-virtual by default is the code breakage that it would cause, and given how surprisingly minimal that is, I think that it's definitely worth it - especially when the kind of folks whose code Walter is most worried about breaking are the guys most interested in the change. - Jonathan M Davis
Making everything final by default would IMO kind of break automated mock classes generation for unit testing, automatic proxy class generation for DB entities, and other OOP niceities. And it wasn't just marking methods final which got the D version of the raytracer in this thread faster than the C++ version in the end. (it was a combination of four or five things, which involved a bit of unrolling, avoiding array literals, and so on). --jm
May 31 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, May 31, 2013 23:59:45 Juan Manuel Cabo wrote:
 Making everything final by default would IMO kind of break
 automated mock classes generation for unit testing,
 automatic proxy class generation for DB entities, and
 other OOP niceities.
Then just mark them all as virtual if you don't want to have to worry about it.
 And it wasn't just marking methods final which got the
 D version of the raytracer in this thread faster than
 the C++ version in the end. (it was a combination of
 four or five things, which involved a bit of unrolling,
 avoiding array literals, and so on).
Far more than just a raytracer cares about whether functions are virtual, and no, making member functions non-virtual by default will not suddenly speed up code everywhere, but there _is_ a cost to having member functions virtual by default, and there's almost no benefit to it IMHO. And it seems to be folks who use D in real programs in real companies who most want functions to be non- virtual by default. Both Don and Manu are completely on board with making member functions non-virtual by default and hate the fact that they're not. - Jonathan M Davis
May 31 2013
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 01.06.2013 05:08, schrieb Jonathan M Davis:
 On Friday, May 31, 2013 23:59:45 Juan Manuel Cabo wrote:
 Making everything final by default would IMO kind of break
 automated mock classes generation for unit testing,
 automatic proxy class generation for DB entities, and
 other OOP niceities.
Then just mark them all as virtual if you don't want to have to worry about it.
 And it wasn't just marking methods final which got the
 D version of the raytracer in this thread faster than
 the C++ version in the end. (it was a combination of
 four or five things, which involved a bit of unrolling,
 avoiding array literals, and so on).
Far more than just a raytracer cares about whether functions are virtual, and no, making member functions non-virtual by default will not suddenly speed up code everywhere, but there _is_ a cost to having member functions virtual by default, and there's almost no benefit to it IMHO. And it seems to be folks who use D in real programs in real companies who most want functions to be non- virtual by default. Both Don and Manu are completely on board with making member functions non-virtual by default and hate the fact that they're not. - Jonathan M Davis
In OO languages where methods are virtual by default, writing mock classes is usually a matter of extending and overriding said classes. In languages with final by default, one is forced to use Assembly or bytecode rewriting tools to be able to override those classes behaviour. In .NET case the Moles/Pex frameworks from Microsoft research were so loved by developers that Microsoft made it part of .NET 4.5 under the name Fake framework. The same for the C++ with frameworks like Isolator++. Of course with virtual by default it is also possible to make a method or class as final, thus causing the same issues with mocking as final by default does. Just my .02€ input to the discussion. -- Paulo
May 31 2013
parent reply "finalpatch" <fengli gmail.com> writes:
I actually don't feel final by default is a big deal at all. Of 
all the factors that caused the poor performance that was 
discussed in the original post, final is the least significant 
one, only caused 5% to %7 of a speed penalty in a heavily 
recursive and looping program. Because of this I think its effect 
in less demanding scenario is negligible. Those who really needs 
the extra speed can simply add final to their methods in the hot 
path.

On Saturday, 1 June 2013 at 06:35:38 UTC, Paulo Pinto wrote:
 In OO languages where methods are virtual by default, writing 
 mock classes is usually a matter of extending and overriding 
 said classes.

 In languages with final by default, one is forced to use 
 Assembly or bytecode rewriting tools to be able to override 
 those classes behaviour.

 In .NET  case the Moles/Pex frameworks from Microsoft research 
 were so loved by developers that Microsoft made it part of .NET 
 4.5 under the name  Fake framework.

 The same for the C++ with frameworks like Isolator++.

 Of course with virtual by default it is also possible to make a 
 method
 or class as final, thus causing the same issues with mocking as 
 final by
 default does.

 Just my .02€ input to the discussion.

 --
 Paulo
May 31 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
finalpatch:

 I actually don't feel final by default is a big deal at all. Of 
 all the factors that caused the poor performance that was 
 discussed in the original post, final is the least significant 
 one, only caused 5% to %7 of a speed penalty in a heavily 
 recursive and looping program. Because of this I think its 
 effect in less demanding scenario is negligible. Those who 
 really needs the extra speed can simply add final to their 
 methods in the hot path.
So profiling matters not just for people that write D code, but also for compiler implementers that should focus on the things that most impact the performance. Is the escape analysis for dynamic arrays already in Bugzilla as enhancement request? How much work does it need to be implemented? And how much gain is it going to give to D programs performance? Bye, bearophile
Jun 01 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/1/13 2:54 AM, finalpatch wrote:
 I actually don't feel final by default is a big deal at all. Of all the
 factors that caused the poor performance that was discussed in the
 original post, final is the least significant one, only caused 5% to %7
 of a speed penalty in a heavily recursive and looping program. Because
 of this I think its effect in less demanding scenario is negligible.
 Those who really needs the extra speed can simply add final to their
 methods in the hot path.
Would be great to collect a summary of the changes and their benefits. Thanks finalpatch and everyone else for this work. Andrei
Jun 01 2013
parent reply "finalpatch" <fengli gmail.com> writes:
Hi Andrei,

I have summarized the results with LDC on Mac OSX in a previous 
posts:

orignal: 760ms
* change v = [x,x,x] to v[0]=x... in this(float) constructor - 
540ms (220ms improvment)
* remove this(float[]) constructor and replace it with 
this(float,float,float) - 410ms (130ms improvment)
* "final" to all class methods - 350 (60ms improvment)
* hand unroll overloaded vector arithmetic operators, 260ms (90ms 
improvment)

However I retested on a windows 7 machine with GDC compiler and 
the results were very different.

orignal: 545ms
* the first 2 optimizations which helped the most on OSX with LDC 
has almost zero effect
* hand unroll overloaded vector arithmetic operators - 280ms 
(265ms improvment)
* "final" to all class methods - 200ms (80ms improvment)


On Saturday, 1 June 2013 at 13:36:44 UTC, Andrei Alexandrescu 
wrote:
 Would be great to collect a summary of the changes and their 
 benefits.

 Thanks finalpatch and everyone else for this work.


 Andrei
Jun 01 2013
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/01/2013 04:58 PM, finalpatch wrote:
 However I retested on a windows 7 machine with GDC compiler and the results
were
 very different.
 
 orignal: 545ms
 * the first 2 optimizations which helped the most on OSX with LDC has almost
 zero effect
 * hand unroll overloaded vector arithmetic operators - 280ms (265ms improvment)
 * "final" to all class methods - 200ms (80ms improvment)
What flags were you using with each compiler?
Jun 02 2013
parent reply finalpatch <fengli gmail.com> writes:
Hi Joseph,

The flags I used 
OSX LDC: -O3 -release
WIN GDC: -O3 -fno-bounds-check -frelease

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

 On 06/01/2013 04:58 PM, finalpatch wrote:
 However I retested on a windows 7 machine with GDC compiler and the results
were
 very different.
 
 orignal: 545ms
 * the first 2 optimizations which helped the most on OSX with LDC has almost
 zero effect
 * hand unroll overloaded vector arithmetic operators - 280ms (265ms improvment)
 * "final" to all class methods - 200ms (80ms improvment)
What flags were you using with each compiler?
-- finalpatch
Jun 02 2013
next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/02/2013 11:32 AM, finalpatch wrote:
 The flags I used 
 OSX LDC: -O3 -release
 WIN GDC: -O3 -fno-bounds-check -frelease
Does adding -inline make a difference to initial performance (i.e. before your manual interventions)? I guess it's already covered by -O3 in both cases, but a while back I did notice some differences in "default" LDC and GDC performance that seemed to relate to inlining.
Jun 02 2013
parent reply finalpatch <fengli gmail.com> writes:
Joseph,

IIRC -inline is a DMD specific switch. Adding this to gdc command line
produces this:

gdc.exe: error: unrecognized command line option '-inline'

Besides, the improvements mainly come from unrolling short loops not
from inlining.

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

 On 06/02/2013 11:32 AM, finalpatch wrote:
 The flags I used 
 OSX LDC: -O3 -release
 WIN GDC: -O3 -fno-bounds-check -frelease
Does adding -inline make a difference to initial performance (i.e. before your manual interventions)? I guess it's already covered by -O3 in both cases, but a while back I did notice some differences in "default" LDC and GDC performance that seemed to relate to inlining.
-- finalpatch
Jun 02 2013
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/02/2013 02:26 PM, finalpatch wrote:
 IIRC -inline is a DMD specific switch. Adding this to gdc command line
 produces this:
 
 gdc.exe: error: unrecognized command line option '-inline'
GDC and LDC both have their own equivalent flags -- for GDC it's -finline-functions (which as Iain says is already covered by -O3), for LDC it's -enable-inlining. It probably won't make any difference for LDC either with -O3 enabled, but might be worth checking.
Jun 02 2013
parent "David Nadlinger" <see klickverbot.at> writes:
On Sunday, 2 June 2013 at 15:59:38 UTC, Joseph Rushton Wakeling 
wrote:
 GDC and LDC both have their own equivalent flags -- for GDC it's
 -finline-functions (which as Iain says is already covered by 
 -O3), for LDC it's
 -enable-inlining.

 It probably won't make any difference for LDC either with -O3 
 enabled, but might
 be worth checking.
It doesn't affect behavior on -O2 and above. David
Jun 02 2013
prev sibling parent reply Iain Buclaw <ibuclaw ubuntu.com> writes:
On 2 June 2013 12:05, Joseph Rushton Wakeling
<joseph.wakeling webdrake.net> wrote:
 On 06/02/2013 11:32 AM, finalpatch wrote:
 The flags I used
 OSX LDC: -O3 -release
 WIN GDC: -O3 -fno-bounds-check -frelease
Does adding -inline make a difference to initial performance (i.e. before your manual interventions)? I guess it's already covered by -O3 in both cases, but a while back I did notice some differences in "default" LDC and GDC performance that seemed to relate to inlining.
-O3 is covered in the GDC case (turns on -finline-functions). -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0';
Jun 02 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/2/2013 4:10 AM, Iain Buclaw wrote:
 -O3 is covered in the GDC case (turns on -finline-functions).
The reason -inline is a separate switch is for debugging purposes (to set breakpoints on the function that would have been inlined), and also to generate profile statistics.
Jun 02 2013
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, June 05, 2013 22:17:49 H. S. Teoh wrote:
 On Wed, Jun 05, 2013 at 09:14:08PM -0400, Jonathan M Davis wrote:
 On Wednesday, June 05, 2013 17:49:17 Walter Bright wrote:
 I think we accomplish this in a simpler way:
 
 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means a method overrides a base virtual function with a
 final
 function.
 3. 'override virtual' means override with a non-final function.
 4. none means final and non-overriding.
I would have expected something more like 1. 'virtual' means a method is an "introducing" one. 2. 'override' means override with a non-final function. 3. 'final override' means a method overrides a base virtual function with a final function. 4. 'final' by itself both mean final and non-overriding.
 As for no specification, I thought the whole point was to have it
 default to final? So 'final' should be optional in this case.
you. It should say 4. 'final' by itself and none both mean final and non-overriding. So, the _only_ things that would change would be 1. virtual would now be required on the "introducing" function to make it virtual. 2. Functions without virtual or override would now be implicitly final and non- virtual. - Jonathan M Davis
Jun 05 2013
parent reply "Michal Minich" <michal.minich gmail.com> writes:
On Thursday, 6 June 2013 at 05:52:28 UTC, Jonathan M Davis wrote:

 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means override with a non-final function.
 3. 'final override' means a method overrides a base virtual 
 function with a final function.
 4. 'final' by itself and none both mean final and 
 non-overriding.
I like the 'final override', it is more natural. What about case when you want to introduce new final method of 'new' for this) class Base { final void foo () } class Derived : Base { new void foo () } what would be in place of 'new' in D?
Jun 06 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 10:42:26 Michal Minich wrote:
 On Thursday, 6 June 2013 at 05:52:28 UTC, Jonathan M Davis wrote:
 1. 'virtual' means a method is an "introducing" one.
 2. 'override' means override with a non-final function.
 3. 'final override' means a method overrides a base virtual
 function with a final function.
 4. 'final' by itself and none both mean final and
 non-overriding.
I like the 'final override', it is more natural. What about case when you want to introduce new final method of 'new' for this) class Base { final void foo () } class Derived : Base { new void foo () } what would be in place of 'new' in D?
We could probably use new to mean the same thing, but I confess that even allowing this seems incredibly bad to me. You end up with a base class function which isn't being overidden looking like it's being overriden in a derived class. And even if it's obvious when look at Derived's declaration thanks to the new, anything which is derived from Derived would just be marked with override. So, it would be incredibly easy to think that calling Base.foo would call Derived.foo or the foo function of the class derived from Derived. So, certainly my initial reaction is to say that because Base.foo was marked with final, it shouldn't be possible for any of its derived classes to have a function with the same signature. - Jonathan M Davis
Jun 06 2013
parent reply "Michal Minich" <michal.minich gmail.com> writes:
 On Thursday, June 06, 2013 10:42:26 Michal Minich wrote:
 On Thursday, 6 June 2013 at 05:52:28 UTC, Jonathan M Davis
 What about case when you want to introduce new final method of

 'new' for this)
 
 class Base { final void foo () }
 class Derived : Base { new void foo () }
 
 what would be in place of 'new' in D?
We could probably use new to mean the same thing, but I confess that even allowing this seems incredibly bad to me. You end up with a base class function which isn't being overidden looking like it's being overriden in a derived class. And even if it's obvious when look at Derived's declaration thanks to the new, anything which is derived from Derived would just be marked with override. So, it would be incredibly easy to think that calling Base.foo would call Derived.foo or the foo function of the class derived from Derived. So, certainly my initial reaction is to say that because Base.foo was marked with final, it shouldn't be possible for any of its derived classes to have a function with the same signature.
That can cause problem for author of base class - if he add any final method, he can break derived classes he may not know of. Example - if you update your external lib you are using in your project, and it happens that new version of some base has 'search' final function added, and you happen to have fn with same name in in your derived, you must now rename all your usages of 'search' function... (i would be preferable if you could just add new on in your derived class 'search'). more frequently used as in D. Also this issue will be more frequent in D with final as default, because to this time virtual method were more common - and they don't have this issue as they have 'override' behavior. 'new' would be orthogonal to 'override' for final functions. virutals can be overridden finals can be newed I think it is very clearly explained in pasted specification in comment http://forum.dlang.org/post/op.wx8biyx7eav7ka stevens-macbook-pro.local
Jun 06 2013
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jun 06, 2013 at 11:16:01AM +0200, Michal Minich wrote:
On Thursday, June 06, 2013 10:42:26 Michal Minich wrote:
On Thursday, 6 June 2013 at 05:52:28 UTC, Jonathan M Davis
What about case when you want to introduce new final method of

'new' for this)

class Base { final void foo () }
class Derived : Base { new void foo () }

what would be in place of 'new' in D?
We could probably use new to mean the same thing, but I confess that even allowing this seems incredibly bad to me. You end up with a base class function which isn't being overidden looking like it's being overriden in a derived class. And even if it's obvious when look at Derived's declaration thanks to the new, anything which is derived from Derived would just be marked with override. So, it would be incredibly easy to think that calling Base.foo would call Derived.foo or the foo function of the class derived from Derived. So, certainly my initial reaction is to say that because Base.foo was marked with final, it shouldn't be possible for any of its derived classes to have a function with the same signature.
That can cause problem for author of base class - if he add any final method, he can break derived classes he may not know of. Example - if you update your external lib you are using in your project, and it happens that new version of some base has 'search' final function added, and you happen to have fn with same name in in your derived, you must now rename all your usages of 'search' function... (i would be preferable if you could just add new on in your derived class 'search').
[...] Wait, why would this be a problem? D already requires that you specify 'override' if you're overriding a base class method. Since final methods cannot be overridden, if the derived class declares a method of the same signature as a base class method, it should be obvious that it's *not* overriding anything. There's no need to use 'new'. T -- Nearly all men can stand adversity, but if you want to test a man's character, give him power. -- Abraham Lincoln
Jun 06 2013
prev sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/06/2013 04:17 PM, H. S. Teoh wrote:
 Wait, why would this be a problem? D already requires that you specify
 'override' if you're overriding a base class method. Since final methods
 cannot be overridden, if the derived class declares a method of the same
 signature as a base class method, it should be obvious that it's *not*
 overriding anything. There's no need to use 'new'.
The rationale is given in Steven's earlier post: http://forum.dlang.org/post/op.wx8biyx7eav7ka stevens-macbook-pro.local
Jun 06 2013