www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - visibility vs. accessibility of protected symbols

reply "Martin Nowak" <dawg dawgfoto.de> writes:
I made a pull request to fix private/package access checks.
https://github.com/D-Programming-Language/dmd/pull/669

Now I'm wondering if that goes into the right direction.

The shallow distinction of visibility vs. accessibility breaks the module  
system because
one can't safely add a private symbol without possibly affecting every  
dependent module.
Thus we're back at using underscore names to protect from that.

 From implementing access checks I found the right amount of checking is to  
do them
when resolving identifiers. Thus it's almost equal to visibility based  
protection.

We already hide symbols that are privately imported by an imported module.

martin
Feb 12 2012
next sibling parent reply David Nadlinger <see klickverbot.at> writes:
On 2/12/12 7:28 PM, Martin Nowak wrote:

 The shallow distinction of visibility vs. accessibility breaks the
 module system because
 one can't safely add a private symbol without possibly affecting every
 dependent module.
 Thus we're back at using underscore names to protect from that.


Yes, and this is exactly why I argued to disregard invisible symbols 
during overload resolution in the past. Walter seems to be firmly 
convinced that the current solution is the right thing to do, but I 
can't recall what his reasons were.

David
Feb 12 2012
next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Sun, 12 Feb 2012 19:32:28 +0100, David Nadlinger <see klickverbot.at>  
wrote:


 On 2/12/12 7:28 PM, Martin Nowak wrote:

 The shallow distinction of visibility vs. accessibility breaks the
 module system because
 one can't safely add a private symbol without possibly affecting every
 dependent module.
 Thus we're back at using underscore names to protect from that.


 Yes, and this is exactly why I argued to disregard invisible symbols  
 during overload resolution in the past. Walter seems to be firmly  
 convinced that the current solution is the right thing to do, but I  
 can't recall what his reasons were.

 David


Overloading is only the trickier part of it, but it affects all other  
symbols as well.
As far as overloading goes C++ does it one way and Java choses the other.
IMHO mixing protection levels for overloads could be disallowed.
Feb 12 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Agreed with David. I find these things a pointless waste of time:

module a;
private {
    struct foo {
    }
}

module b;
void foo() { }

import a;
import b;

void main() {
    foo();
}

main.d(7): Error: a.foo at a.d(3) conflicts with b.foo at b.d(2)
main.d(7): Error: structliteral has no effect in expression (foo())

One of those cases exists in Phobos somewhere, where a private type is
declared with the same name as an unrelated public function in some
other module. I can't recall which module it is though.
Feb 12 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Sunday, February 12, 2012 19:32:28 David Nadlinger wrote:

 On 2/12/12 7:28 PM, Martin Nowak wrote:

 The shallow distinction of visibility vs. accessibility breaks the
 module system because
 one can't safely add a private symbol without possibly affecting every
 dependent module.
 Thus we're back at using underscore names to protect from that.


 
 Yes, and this is exactly why I argued to disregard invisible symbols
 during overload resolution in the past. Walter seems to be firmly
 convinced that the current solution is the right thing to do, but I
 can't recall what his reasons were.


I'm not 100% sure that it's a good idea, depending on what the exact side 
effects are, but in principle, I'd love it if private symbols were invisible to 
other modules.

Now, this does _not_ fly with private being overridable, as TDPL claims it 
should be. But I've already argued that that was a mistake due to the 
performance issues that it causes. I'm not sure what Walter's current plans 
with that are though.

http://d.puremagic.com/issues/show_bug.cgi?id=4542

- Jonathan M Davis
Feb 13 2012
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Mon, 13 Feb 2012 19:07:20 +0100, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:


 On Sunday, February 12, 2012 19:32:28 David Nadlinger wrote:

 On 2/12/12 7:28 PM, Martin Nowak wrote:

 The shallow distinction of visibility vs. accessibility breaks the
 module system because
 one can't safely add a private symbol without possibly affecting every
 dependent module.
 Thus we're back at using underscore names to protect from that.


 Yes, and this is exactly why I argued to disregard invisible symbols
 during overload resolution in the past. Walter seems to be firmly
 convinced that the current solution is the right thing to do, but I
 can't recall what his reasons were.


 I'm not 100% sure that it's a good idea, depending on what the exact side
 effects are, but in principle, I'd love it if private symbols were  
 invisible to
 other modules.

 Now, this does _not_ fly with private being overridable, as TDPL claims  
 it
 should be. But I've already argued that that was a mistake due to the
 performance issues that it causes. I'm not sure what Walter's current  
 plans
 with that are though.


Can you elaborate on what issues you see with NVI. After all it's only the  
final public
method that needs to call the virtual private methods.
Feb 13 2012
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, February 14, 2012 00:15:40 Martin Nowak wrote:

 Can you elaborate on what issues you see with NVI. After all it's only the
 final public
 method that needs to call the virtual private methods.


As I explain in that bug report, NVI can be acheived with protected rather 
than private. So, we can do NVI now with private being non-virtual. However, 
if we make private virtual, then the compiler can no longer inline private 
functions in classes unless they're marked as final. That's a performance hit 
to pretty much every class ever written.

Yes, the conscientious programmer will mark their classes' private functions 
final unless they intend to use them for NVI, but the _default_ is then 
inefficient for little to no gain. All you gain is the ability to do NVI with 
private instead of protected, and that is not even vaguely worth the 
performance hit to almost every D class ever IMHO.

- Jonathan M Davis
Feb 13 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 02/14/2012 12:26 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 00:15:40 Martin Nowak wrote:

 Can you elaborate on what issues you see with NVI. After all it's only the
 final public
 method that needs to call the virtual private methods.


 As I explain in that bug report, NVI can be acheived with protected rather
 than private. So, we can do NVI now with private being non-virtual. However,
 if we make private virtual, then the compiler can no longer inline private
 functions in classes unless they're marked as final. That's a performance hit
 to pretty much every class ever written.

 Yes, the conscientious programmer will mark their classes' private functions
 final unless they intend to use them for NVI, but the _default_ is then
 inefficient for little to no gain. All you gain is the ability to do NVI with
 private instead of protected, and that is not even vaguely worth the
 performance hit to almost every D class ever IMHO.

 - Jonathan M Davis


Virtual private does not allow for cross-module NVI, so what would be 
the point anyway?
Feb 13 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, February 14, 2012 00:36:31 Timon Gehr wrote:

 On 02/14/2012 12:26 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 00:15:40 Martin Nowak wrote:

 Can you elaborate on what issues you see with NVI. After all it's only
 the final public
 method that needs to call the virtual private methods.


 
 As I explain in that bug report, NVI can be acheived with protected
 rather than private. So, we can do NVI now with private being
 non-virtual. However, if we make private virtual, then the compiler can
 no longer inline private functions in classes unless they're marked as
 final. That's a performance hit to pretty much every class ever
 written.
 
 Yes, the conscientious programmer will mark their classes' private
 functions final unless they intend to use them for NVI, but the
 _default_ is then inefficient for little to no gain. All you gain is
 the ability to do NVI with private instead of protected, and that is
 not even vaguely worth the performance hit to almost every D class ever
 IMHO.
 
 - Jonathan M Davis


 
 Virtual private does not allow for cross-module NVI, so what would be
 the point anyway?


Why wouldn't it? The whole point of NVI is that you override it but don't call 
it - the base class does that. So, the derived class could override the 
private function but couldn't call it. It _is_ a bit weird though, all the 
same.

Regardless, I'm against making private virtual because of the needless 
performance hit.

- Jonathan M Davis
Feb 13 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 02/14/2012 01:28 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 00:36:31 Timon Gehr wrote:

 On 02/14/2012 12:26 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 00:15:40 Martin Nowak wrote:

 Can you elaborate on what issues you see with NVI. After all it's only
 the final public
 method that needs to call the virtual private methods.


 As I explain in that bug report, NVI can be acheived with protected
 rather than private. So, we can do NVI now with private being
 non-virtual. However, if we make private virtual, then the compiler can
 no longer inline private functions in classes unless they're marked as
 final. That's a performance hit to pretty much every class ever
 written.

 Yes, the conscientious programmer will mark their classes' private
 functions final unless they intend to use them for NVI, but the
 _default_ is then inefficient for little to no gain. All you gain is
 the ability to do NVI with private instead of protected, and that is
 not even vaguely worth the performance hit to almost every D class ever
 IMHO.

 - Jonathan M Davis


 Virtual private does not allow for cross-module NVI, so what would be
 the point anyway?


 Why wouldn't it? The whole point of NVI is that you override it but don't call
 it - the base class does that. So, the derived class could override the
 private function but couldn't call it. It _is_ a bit weird though, all the
 same.


It would break information hiding. A private method declared in a 
different module is not supposed to be visible (hopefully will get 
fixed), let alone overrideable.


 Regardless, I'm against making private virtual because of the needless
 performance hit.


There are no performance implications because the compiler has the whole 
module ready to analyse. Therefore it can de-virtualise any calls to 
private members that are never overridden. A trivial addition to the 
class layout could furthermore allow the compiler to remove the vtable 
entry for such a member function entirely (essentially, virtual private 
members would get negative vtable offsets). I actually think that should 
be implemented in order to get a potential TDPL error off the table.
Feb 13 2012
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, February 14, 2012 01:55:51 Timon Gehr wrote:

 On 02/14/2012 01:28 AM, Jonathan M Davis wrote:
 It would break information hiding. A private method declared in a
 different module is not supposed to be visible (hopefully will get
 fixed), let alone overrideable.


You misunderstand then. Private symbols are most definitely visible. Access 
modifiers do _not_ affect visibility, only accessibility. Private symbols are 
visible _by design_. That's how it works in C++, and D inherited that. The 
whole point of this thread is to discuss whether access modifiers should affect 
visibility, but right now they _do_, and it's _on purpose_ that they do. It's 
not a bug.

As it stands, if I declare a private function which conflicts with a public 
function, the compiler will complain. For example, If I were to add a private 
abs function to std.file which took an int, then any module which imported both 
std.math and std.file would have to use std.math.abs instead of just abs, 
because there would be a conflict. The fact that std.file's abs was private 
wouldn't mean anything to overload resolution.

In fact, just recently, a private abs function was added to core.time for what 
it needed, and it caused problems. It had to be changed to _abs to not screw 
up std.math.abs.

The whole point of this thread is to discuss whether access modifiers should be 
changed to affect visibility rather than access.


 Regardless, I'm against making private virtual because of the needless
 performance hit.


 
 There are no performance implications because the compiler has the whole
 module ready to analyse. Therefore it can de-virtualise any calls to
 private members that are never overridden. A trivial addition to the
 class layout could furthermore allow the compiler to remove the vtable
 entry for such a member function entirely (essentially, virtual private
 members would get negative vtable offsets). I actually think that should
 be implemented in order to get a potential TDPL error off the table.


No. It can't. Because a derived class in an entirely different module could 
override the private function. private does _not_ affect visibility. The whole 
idea of overriding private for NVI is to allow derived classes to override the 
private function's implementation but not call it. If private becomes virtual, 
then the compiler loses the ability to inline non-final private functions 
unless it can guarantee that it's dealing with that exact type, which it 
usually can't do in the type's own member functions.

- Jonathan M Davis
Feb 13 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 02/14/2012 02:16 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 01:55:51 Timon Gehr wrote:

 On 02/14/2012 01:28 AM, Jonathan M Davis wrote:
 It would break information hiding. A private method declared in a
 different module is not supposed to be visible (hopefully will get
 fixed), let alone overrideable.


 You misunderstand then. Private symbols are most definitely visible. Access
 modifiers do _not_ affect visibility, only accessibility. Private symbols are
 visible _by design_. That's how it works in C++, and D inherited that. The
 whole point of this thread is to discuss whether access modifiers should affect
 visibility, but right now they _do_, and it's _on purpose_ that they do. It's
 not a bug.


It is a bug on purpose.



 As it stands, if I declare a private function which conflicts with a public
 function, the compiler will complain. For example, If I were to add a private
 abs function to std.file which took an int, then any module which imported both
 std.math and std.file would have to use std.math.abs instead of just abs,
 because there would be a conflict. The fact that std.file's abs was private
 wouldn't mean anything to overload resolution.


Borken.


 In fact, just recently, a private abs function was added to core.time for what
 it needed, and it caused problems. It had to be changed to _abs to not screw
 up std.math.abs.


Exactly what I am saying. This is laughable (!) and should vanish from 
Ds otherwise very sane approach at modularity.


 The whole point of this thread is to discuss whether access modifiers should be
 changed to affect visibility rather than access.


Most definitely yes. The only hindrance are implementation issues.


 Regardless, I'm against making private virtual because of the needless
 performance hit.


 There are no performance implications because the compiler has the whole
 module ready to analyse. Therefore it can de-virtualise any calls to
 private members that are never overridden. A trivial addition to the
 class layout could furthermore allow the compiler to remove the vtable
 entry for such a member function entirely (essentially, virtual private
 members would get negative vtable offsets). I actually think that should
 be implemented in order to get a potential TDPL error off the table.


 No. It can't. Because a derived class in an entirely different module could
 override the private function. private does _not_ affect visibility.


Well, not being able override what cannot be accesses is a quite basic 
requirement of security. Private members cannot be overriden in a 
different module.


 The whole idea of overriding private for NVI is to allow derived classes to
override the
 private function's implementation but not call it.


This is confusing method implementation with overriding. I just read the 
relevant sections of TDPL:

"D fully supports NVI by providing _specific guarantees_ when 
*interfaces* use access specifiers."

It is an explicit special case because private wouldn't have much of a 
meaning for interfaces otherwise. This means private interface members 
would not be hidden, they are special _already_.



 If private becomes virtual, then the compiler loses the ability to inline
non-final private functions
 unlessit can guarantee that it's dealing with that exact type, which it 


usually can't do in the type's own member functions.


The optimisation works for all private class members. The issue is the 
virtual table layout for separate compilation with .di files, therefore 
it is not a good idea to make private methods virtual. But that is not 
required to satisfy TDPL anyway:

As far as I can see TDPL only requires private *interface* members to be 
implementable. That is not a problem at all, interface vtbls use 
thunking anyway. Therefore the private members could just be plugged 
there and be called virtually through base interface calls and 
non-virtually otherwise.



 Feb 13 2012



  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:



 Well, not being able override what cannot be accesses is a quite basic
 requirement of security. Private members cannot be overriden in a
 different module.


Have you ever read up on NVI? That's how it's supposed to work. The whole idea 
is to have derived classes override functions without being able to use them. 
It makes it so that the public function is non-virtual and contains whatever 
stuff you want to guarantee is called before or after the private function, 
which is then virtual.

http://www.gotw.ca/publications/mill18.htm


 The whole idea of overriding private for NVI is to allow derived classes
 to override the private function's implementation but not call it.


 
 This is confusing method implementation with overriding. I just read the
 relevant sections of TDPL:
 
 "D fully supports NVI by providing _specific guarantees_ when
 *interfaces* use access specifiers."
 
 It is an explicit special case because private wouldn't have much of a
 meaning for interfaces otherwise. This means private interface members
 would not be hidden, they are special _already_.


The fact that TDPL talks about interfaces in specific could be a good argument 
for _not_ making private virtual for classes. But regardless, as it stands, 
_nothing_ which is private is hidden. So, there's nothing special going on 
with interfaces and private, except that you can't override them to actually 
do NVI, so they're kind of pointless.


 If private becomes virtual, then the compiler loses the ability to
 inline non-final private functions unlessit can guarantee that it's
 dealing with that exact type, which it


 usually can't do in the type's own member functions.
 
 The optimisation works for all private class members. The issue is the
 virtual table layout for separate compilation with .di files, therefore
 it is not a good idea to make private methods virtual. But that is not
 required to satisfy TDPL anyway:
 
 As far as I can see TDPL only requires private *interface* members to be
 implementable. That is not a problem at all, interface vtbls use
 thunking anyway. Therefore the private members could just be plugged
 there and be called virtually through base interface calls and
 non-virtually otherwise.


The optimization only works as long as the compiler can guarantee that no 
derived classes override the private functions, since it doesn't necessarily 
know about all of the derived classes. If D allowed NVI with classes the way 
that C++ does by making private virtual, then derived classes would _have_ to 
be able to override private functions in base classes which are in separate 
modules; they might even be in separate libraries. And then the compiler could 
not guarantee that no derived classes overrode the private functions, and the 
optimization could not be made. Hence why I'm arguing against making private 
functions virtual.

Personally, I'd very much like to see access level affect visibility, but 
Walter must be convinced.

- Jonathan M Davis



 Feb 13 2012



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 02/14/2012 03:05 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:



 Well, not being able override what cannot be accesses is a quite basic
 requirement of security. Private members cannot be overriden in a
 different module.


 Have you ever read up on NVI? That's how it's supposed to work. The whole idea
 is to have derived classes override functions without being able to use them.
 It makes it so that the public function is non-virtual and contains whatever
 stuff you want to guarantee is called before or after the private function,
 which is then virtual.

 http://www.gotw.ca/publications/mill18.htm


That article uses C++. It is not really a fair comparison, because in D 
there is no way to explicitly declare a method "private virtual", and 
D's private is different from C++'s private, and C++ has a different 
sub-typing and inheritance model. I think there are different design 
trade-offs at play.

But I can make better sense of your point of view now: When I was 
talking about virtual I meant just a function that has an entry in a 
vtbl and is invoked virtually, while you probably had C++'s semantics in 
mind.



 The whole idea of overriding private for NVI is to allow derived classes
 to override the private function's implementation but not call it.


 This is confusing method implementation with overriding. I just read the
 relevant sections of TDPL:

 "D fully supports NVI by providing _specific guarantees_ when
 *interfaces* use access specifiers."

 It is an explicit special case because private wouldn't have much of a
 meaning for interfaces otherwise. This means private interface members
 would not be hidden, they are special _already_.


 The fact that TDPL talks about interfaces in specific could be a good argument
 for _not_ making private virtual for classes. But regardless, as it stands,
 _nothing_ which is private is hidden. So, there's nothing special going on
 with interfaces and private, except that you can't override them to actually
 do NVI, so they're kind of pointless.


The fact that they cannot be implemented in a class that implements the 
interface seems to be a [TDPL] bug.



 If private becomes virtual, then the compiler loses the ability to
 inline non-final private functions unlessit can guarantee that it's
 dealing with that exact type, which it


 usually can't do in the type's own member functions.

 The optimisation works for all private class members. The issue is the
 virtual table layout for separate compilation with .di files, therefore
 it is not a good idea to make private methods virtual. But that is not
 required to satisfy TDPL anyway:

 As far as I can see TDPL only requires private *interface* members to be
 implementable. That is not a problem at all, interface vtbls use
 thunking anyway. Therefore the private members could just be plugged
 there and be called virtually through base interface calls and
 non-virtually otherwise.


 The optimization only works as long as the compiler can guarantee that no
 derived classes override the private functions, since it doesn't necessarily
 know about all of the derived classes.


The optimization works under the precondition that derived classes from 
a different module cannot override private member functions.


 If D allowed NVI with classes the way that C++ does by making private virtual,
then derived classes would _have_ to
 be able to override private functions in base classes which are in separate
 modules; they might even be in separate libraries. And then the compiler could
 not guarantee that no derived classes overrode the private functions, and the
 optimization could not be made. Hence why I'm arguing against making private
 functions virtual.


Here you probably mean virtual as in C++. I agree.


 Personally, I'd very much like to see access level affect visibility, but
 Walter must be convinced.


I think there were some issues with overload sets, but I think overloads 
with differing accessibility should just be disallowed.



 Feb 13 2012



  parent reply Jonathan M Davis <jmdavisProg gmx.com>  writes:


On Tuesday, February 14, 2012 03:45:54 Timon Gehr wrote:

 On 02/14/2012 03:05 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
 
 Well, not being able override what cannot be accesses is a quite basic
 requirement of security. Private members cannot be overriden in a
 different module.


 
 Have you ever read up on NVI? That's how it's supposed to work. The whole
 idea is to have derived classes override functions without being able to
 use them. It makes it so that the public function is non-virtual and
 contains whatever stuff you want to guarantee is called before or after
 the private function, which is then virtual.
 
 http://www.gotw.ca/publications/mill18.htm


 
 That article uses C++. It is not really a fair comparison, because in D
 there is no way to explicitly declare a method "private virtual", and
 D's private is different from C++'s private, and C++ has a different
 sub-typing and inheritance model. I think there are different design
 trade-offs at play.
 
 But I can make better sense of your point of view now: When I was
 talking about virtual I meant just a function that has an entry in a
 vtbl and is invoked virtually, while you probably had C++'s semantics in
 mind.


I'm not aware of _any_ difference between virtual functions in C++ and D except 
for the fact that all public and protected member functions are always virtual 
in D (unless the compiler is able to determine that it can make them non-
virtual), and currently private and protected functions are never virtual. 
What differences are you referring to?

- Jonathan M Davis



 Feb 13 2012



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 02/14/2012 03:54 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 03:45:54 Timon Gehr wrote:

 On 02/14/2012 03:05 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:

 Well, not being able override what cannot be accesses is a quite basic
 requirement of security. Private members cannot be overriden in a
 different module.


 Have you ever read up on NVI? That's how it's supposed to work. The whole
 idea is to have derived classes override functions without being able to
 use them. It makes it so that the public function is non-virtual and
 contains whatever stuff you want to guarantee is called before or after
 the private function, which is then virtual.

 http://www.gotw.ca/publications/mill18.htm


 That article uses C++. It is not really a fair comparison, because in D
 there is no way to explicitly declare a method "private virtual", and
 D's private is different from C++'s private, and C++ has a different
 sub-typing and inheritance model. I think there are different design
 trade-offs at play.

 But I can make better sense of your point of view now: When I was
 talking about virtual I meant just a function that has an entry in a
 vtbl and is invoked virtually, while you probably had C++'s semantics in
 mind.


 I'm not aware of _any_ difference between virtual functions in C++ and D except
 for the fact that all public and protected member functions are always virtual
 in D (unless the compiler is able to determine that it can make them non-
 virtual), and currently private and protected functions are never virtual.
 What differences are you referring to?

 - Jonathan M Davis


The main difference is that C++ has an explicit 'virtual' modifier that 
naturally grants override-access to private symbols.



 Feb 13 2012



  parent reply Jonathan M Davis <jmdavisProg gmx.com>  writes:


On Tuesday, February 14, 2012 04:39:18 Timon Gehr wrote:

 On 02/14/2012 03:54 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 03:45:54 Timon Gehr wrote:

 On 02/14/2012 03:05 AM, Jonathan M Davis wrote:

 On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
 
 Well, not being able override what cannot be accesses is a quite basic
 requirement of security. Private members cannot be overriden in a
 different module.


 
 Have you ever read up on NVI? That's how it's supposed to work. The
 whole
 idea is to have derived classes override functions without being able to
 use them. It makes it so that the public function is non-virtual and
 contains whatever stuff you want to guarantee is called before or after
 the private function, which is then virtual.
 
 http://www.gotw.ca/publications/mill18.htm


 
 That article uses C++. It is not really a fair comparison, because in D
 there is no way to explicitly declare a method "private virtual", and
 D's private is different from C++'s private, and C++ has a different
 sub-typing and inheritance model. I think there are different design
 trade-offs at play.
 
 But I can make better sense of your point of view now: When I was
 talking about virtual I meant just a function that has an entry in a
 vtbl and is invoked virtually, while you probably had C++'s semantics in
 mind.


 
 I'm not aware of _any_ difference between virtual functions in C++ and D
 except for the fact that all public and protected member functions are
 always virtual in D (unless the compiler is able to determine that it can
 make them non- virtual), and currently private and protected functions
 are never virtual. What differences are you referring to?
 
 - Jonathan M Davis


 
 The main difference is that C++ has an explicit 'virtual' modifier that
 naturally grants override-access to private symbols.


In other words, the only difference is what I said - that public and protected 
in D are automatically virtual, and private and protected are automatically 
non-virtual. So, assuming that private were made virtual, it would be the same 
as a private virtual function in C++ as far as virtuality goes. The only 
differences would be that D effectively makes all of the functions in the rest 
of the module friends and has no other way of declaring friends.

Regardless, I think that we agree that it would be better if private were non-
virtual and access level also affected visibility so that private functions 
were not visible outside of the module that they're declared in.

- Jonathan M Davis



 Feb 13 2012



  parent reply "Daniel Murphy" <yebblies nospamgmail.com>  writes:


 private and protected are automatically non-virtual


You've said that twice now, I think you mean private and package. 



 Feb 14 2012



  parent  Jonathan M Davis <jmdavisProg gmx.com>  writes:


On Tuesday, February 14, 2012 21:13:46 Daniel Murphy wrote:

 private and protected are automatically non-virtual


 
 You've said that twice now, I think you mean private and package.


Yes. You're right. The excess of p's is confusing me, I guess.

public and protected are automatically virtual.

private and package are automatically non-virtual.

- Jonathan M Davis



 Feb 14 2012



prev sibling  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Mon, 13 Feb 2012 21:05:35 -0500, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:


 On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:



 Well, not being able override what cannot be accesses is a quite basic
 requirement of security. Private members cannot be overriden in a
 different module.


 Have you ever read up on NVI? That's how it's supposed to work. The  
 whole idea
 is to have derived classes override functions without being able to use  
 them.
 It makes it so that the public function is non-virtual and contains  
 whatever
 stuff you want to guarantee is called before or after the private  
 function,
 which is then virtual.

 http://www.gotw.ca/publications/mill18.htm


This does nothing to protect the implementation.  You are deluding  
yourself if you think you can stop a derived class from calling it's own  
implementation!  In this case, protected is good enough, and I like how  
private is always non-virtual.  As Timon says, it's good for security and  
module encapsulation.

-Steve



 Feb 17 2012



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 2/17/12 8:03 AM, Steven Schveighoffer wrote:

 On Mon, 13 Feb 2012 21:05:35 -0500, Jonathan M Davis
 <jmdavisProg gmx.com> wrote:


 On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:



 Well, not being able override what cannot be accesses is a quite basic
 requirement of security. Private members cannot be overriden in a
 different module.


 Have you ever read up on NVI? That's how it's supposed to work. The
 whole idea
 is to have derived classes override functions without being able to
 use them.
 It makes it so that the public function is non-virtual and contains
 whatever
 stuff you want to guarantee is called before or after the private
 function,
 which is then virtual.

 http://www.gotw.ca/publications/mill18.htm


 This does nothing to protect the implementation. You are deluding
 yourself if you think you can stop a derived class from calling it's own
 implementation! In this case, protected is good enough, and I like how
 private is always non-virtual. As Timon says, it's good for security and
 module encapsulation.

 -Steve


I agree. Walter and I chalked that up to a bug in TDPL that will be 
superseded by a different implementation.

Andrei



 Feb 17 2012



  parent  "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Friday, February 17, 2012 10:18:32 Andrei Alexandrescu wrote:

 I agree. Walter and I chalked that up to a bug in TDPL that will be
 superseded by a different implementation.


Good to hear!

- Jonathan M Davis



 Feb 17 2012



prev sibling  parent reply "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Friday, February 17, 2012 09:03:32 Steven Schveighoffer wrote:

 On Mon, 13 Feb 2012 21:05:35 -0500, Jonathan M Davis <jmdavisProg gmx.com>
 
 wrote:

 On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
 
 Well, not being able override what cannot be accesses is a quite basic
 requirement of security. Private members cannot be overriden in a
 different module.


 
 Have you ever read up on NVI? That's how it's supposed to work. The
 whole idea
 is to have derived classes override functions without being able to use
 them.
 It makes it so that the public function is non-virtual and contains
 whatever
 stuff you want to guarantee is called before or after the private
 function,
 which is then virtual.
 
 http://www.gotw.ca/publications/mill18.htm


 
 This does nothing to protect the implementation. You are deluding
 yourself if you think you can stop a derived class from calling it's own
 implementation! In this case, protected is good enough, and I like how
 private is always non-virtual. As Timon says, it's good for security and
 module encapsulation.


I know. That's part of why I'm arguing that protected is enough and that 
private shouldn't be virtual. But pretty much the whole point of doing NVI 
with private is the theory that you want to guarantee that the derived classes 
can't call the function. The fact that you can't actually guarantee that just 
makes the arguments for making private virtual for NVI that much weaker.

But Timon's arguments make it sound like he doesn't understand what NVI is, 
which was the point of my post.

- Jonathan M Davis



 Feb 17 2012



  parent reply "Timon Gehr" <timon.gehr gmx.ch>  writes:


On Friday, 17 February 2012 at 18:35:38 UTC, Jonathan M Davis 
wrote:

 On Friday, February 17, 2012 09:03:32 Steven Schveighoffer 
 wrote:

 On Mon, 13 Feb 2012 21:05:35 -0500, Jonathan M Davis 
 <jmdavisProg gmx.com>
 
 wrote:

 On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:

 On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
 
 Well, not being able override what cannot be accesses is a 
 quite basic
 requirement of security. Private members cannot be 
 overriden in a
 different module.


 
 Have you ever read up on NVI? That's how it's supposed to 
 work. The
 whole idea
 is to have derived classes override functions without being 
 able to use
 them.
 It makes it so that the public function is non-virtual and 
 contains
 whatever
 stuff you want to guarantee is called before or after the 
 private
 function,
 which is then virtual.
 
 http://www.gotw.ca/publications/mill18.htm


 
 This does nothing to protect the implementation. You are 
 deluding
 yourself if you think you can stop a derived class from 
 calling it's own
 implementation! In this case, protected is good enough, and I 
 like how
 private is always non-virtual. As Timon says, it's good for 
 security and
 module encapsulation.


 I know. That's part of why I'm arguing that protected is enough 
 and that private shouldn't be virtual. But pretty much the 
 whole point of doing NVI with private is the theory that you 
 want to guarantee that the derived classes can't call the 
 function. The fact that you can't actually guarantee that just 
 makes the arguments for making private virtual for NVI that 
 much weaker.

 But Timon's arguments make it sound like he doesn't understand 
 what NVI is, which was the point of my post.

 - Jonathan M Davis


I understand what NVI is. It is just that private methods 
overridable from a different module don't have many merits, 
regardless of how NVI is apparently sometimes implemented in C++. 
The fact that C++ has overridable private methods is quite likely 
just a side-effect of wanting to give some kind of semantics to a 
method declared as both private and virtual. I could argue that 
it sounds like you don't understand what NVI is because you 
assume C++'s private virtual semantics are required for the 
implementation of NVI. ;)

As far as information hiding design goes, we certainly don't want 
to borrow heavily from C++. Its system is unsound.



 Feb 17 2012



  parent  "Jonathan M Davis" <jmdavisProg gmx.com>  writes:


On Friday, February 17, 2012 20:02:53 Timon Gehr wrote:

 I understand what NVI is. It is just that private methods
 overridable from a different module don't have many merits,
 regardless of how NVI is apparently sometimes implemented in C++.
 The fact that C++ has overridable private methods is quite likely
 just a side-effect of wanting to give some kind of semantics to a
 method declared as both private and virtual. I could argue that
 it sounds like you don't understand what NVI is because you
 assume C++'s private virtual semantics are required for the
 implementation of NVI. ;)


It's required if you implement NVI the way that's typical in C++ and the way 
that is requested in the NVI-related bug report.

You can get close enough to the same effect using protected, so it's arguably 
not necessary (which is one of the reasons that I'm arguing against making 
private virtual), but if we're talking about NVI in the traditional sense 
(which is what the bug report is looking for and TDPL indicates that you can 
do in its discussion on NVI), it _is_ required.

Fortunately, it looks like Andrei and Walter have agreed that private shouldn't
be made virtual, regardless what TDPL says - though what that means for NVI
and interfaces, I don't know. Protected may be enough in that case as well.

- Jonathan M Davis



 Feb 17 2012



prev sibling  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 02/14/2012 01:55 AM, Timon Gehr wrote:

 There are no performance implications because the compiler has the whole
 module ready to analyse. Therefore it can de-virtualise any calls to
 private members that are never overridden. A trivial addition to the
 class layout could furthermore allow the compiler to remove the vtable
 entry for such a member function entirely (essentially, virtual private
 members would get negative vtable offsets). I actually think that should
 be implemented in order to get a potential TDPL error off the table.


Nevermind, separate compilation could get tricky.



 Feb 13 2012



prev sibling  parent  "Martin Nowak" <dawg dawgfoto.de>  writes:


On Tue, 14 Feb 2012 00:26:14 +0100, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:


 On Tuesday, February 14, 2012 00:15:40 Martin Nowak wrote:

 Can you elaborate on what issues you see with NVI. After all it's only  
 the
 final public
 method that needs to call the virtual private methods.


 As I explain in that bug report, NVI can be acheived with protected  
 rather
 than private. So, we can do NVI now with private being non-virtual.  
 However,
 if we make private virtual, then the compiler can no longer inline  
 private
 functions in classes unless they're marked as final. That's a  
 performance hit
 to pretty much every class ever written.

 Yes, the conscientious programmer will mark their classes' private  
 functions
 final unless they intend to use them for NVI, but the _default_ is then
 inefficient for little to no gain. All you gain is the ability to do NVI  
 with
 private instead of protected, and that is not even vaguely worth the
 performance hit to almost every D class ever IMHO.

 - Jonathan M Davis


OK, that was known and I agree with you. I even find the concept of  
overriding private
functions flawed. I thought you had something in mind with overload  
resolution.



 Feb 13 2012