www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Updated DIP22 - Private symbol visibility

reply Martin Nowak <code dawg.eu> writes:
I updated DIP22 - Private symbol visibility with my findings from 
implementing the 'Hide module members' pull request.

https://github.com/D-Programming-Language/dmd/pull/739

I don't have much time to cope with the topic but any feedback is welcome.
Dec 20 2013
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
I don't have anything really substantial to say except that this 
looks good to me and i look forward to it being 
implemented/pulled.
Dec 20 2013
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/20/2013 09:48 PM, Martin Nowak wrote:

 I updated DIP22 - Private symbol visibility with my findings from
 implementing the 'Hide module members' pull request.

 https://github.com/D-Programming-Language/dmd/pull/739

 I don't have much time to cope with the topic but any feedback is welcome.


Looks mostly good to me, but the following snippets that both seem 
reasonable unfortunately contradict each other.

"private is an encapsulation tool. If it is not intended to be used by 
"outsiders", it should not interfere with them at all."

"* The least protected symbol determines the visibility for an overload 
set.
     After overload resolution an access check will be performed.
     Thereby overload resolution is independent of look-up origin."

Adding a private symbol to an overload set can break 3rd party code with 
these rules. Since private symbols are usually excluded from di files I 
don't think this design is viable. I think it would be better to 
disallow overload sets with mixed protection.
Dec 21 2013
parent reply "Martin Nowak" <code dawg.eu> writes:
On Saturday, 21 December 2013 at 17:41:32 UTC, Timon Gehr wrote:

 Adding a private symbol to an overload set can break 3rd party 
 code with these rules.


Yes, this is understood, but it seems like a good tradeoff when 
compared to the alternative where overload resolution depends on 
look-up origin. The latter could easily break generic code and 
simple refactorings could introduce subtle bugs.


 Since private symbols are usually excluded from di files I 
 don't think this design is viable.


This, I don't have a good answer for, except that we have to put 
the private overload in the .di file as well.


 I think it would be better to disallow overload sets with mixed 
 protection.


Tried that, doesn't work. The most common counterexample is 
mixing private/public constructors.
Dec 21 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 21 December 2013 at 19:29:28 UTC, Martin Nowak wrote:

 Yes, this is understood, but it seems like a good tradeoff when 
 compared to the alternative where overload resolution depends 
 on look-up origin. The latter could easily break generic code 
 and simple refactorings could introduce subtle bugs.


Still don't understand how intended and expected behavior can be 
called "bug".

(I don't like updated DIP at all and will be against anything 
that puts private symbols into overload sets)
Dec 21 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 12/21/2013 08:54 PM, Dicebot wrote:

 On Saturday, 21 December 2013 at 19:29:28 UTC, Martin Nowak wrote:
 Still don't understand how intended and expected behavior can be called
 "bug".


If I make a refactoring and move a function from module A to B, but the 
function calls an overload set in A it should not have a different 
overload resolution.
Similar case, if I pass an overload set via alias to a traits template 
in module B, the template should follow the same overload resolution.
It's very important that we preserve these properties.


 (I don't like updated DIP at all and will be against anything that puts
 private symbols into overload sets)


I don't like them either, but they have some use-cases.
Dec 21 2013
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/21/2013 10:14 PM, Martin Nowak wrote:

 On 12/21/2013 08:54 PM, Dicebot wrote:

 On Saturday, 21 December 2013 at 19:29:28 UTC, Martin Nowak wrote:
 Still don't understand how intended and expected behavior can be called
 "bug".


 If I make a refactoring and move a function from module A to B, but the
 function calls an overload set in A it should not have a different
 overload resolution.
 Similar case, if I pass an overload set via alias to a traits template
 in module B, the template should follow the same overload resolution.


Well, you pass an alias. Are you saying that for alias template 
arguments, the scope of the template declaration applies for 
accessibility checking? I.e. we cannot pass private symbols as alias 
arguments? I think this is an unreasonably high price to pay.

Note that if we actually allow this, having any kind of differing 
behaviour when accessing an overload set from different modules is a 
consistency issue, since the same template with an alias parameter might 
be instantiated from both modules with the same overload set.


 It's very important that we preserve these properties.


 (I don't like updated DIP at all and will be against anything that puts
 private symbols into overload sets)


 I don't like them either, but they have some use-cases.


Basically just the constructor thing though?
Note that constructors cannot be passed by alias due to a syntax 
restriction... :o)
Dec 21 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 12/21/2013 10:28 PM, Timon Gehr wrote:

 Well, you pass an alias. Are you saying that for alias template
 arguments, the scope of the template declaration applies for
 accessibility checking? I.e. we cannot pass private symbols as alias
 arguments? I think this is an unreasonably high price to pay.


You can pass them and basically it behaves like public aliases, in the 
sense that the one who instantiates a template passes a public alias to 
a possibly private symbol.

It's a question of transitivity what the template can do with the symbol.

Currently it's not possible to call private functions or private methods 
of variables that are passed via alias parameter.


 Basically just the constructor thing though?
 Note that constructors cannot be passed by alias due to a syntax
 restriction... :o)
Dec 21 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/21/2013 11:03 PM, Martin Nowak wrote:

 On 12/21/2013 10:28 PM, Timon Gehr wrote:

 Well, you pass an alias. Are you saying that for alias template
 arguments, the scope of the template declaration applies for
 accessibility checking? I.e. we cannot pass private symbols as alias
 arguments? I think this is an unreasonably high price to pay.


 You can pass them and basically it behaves like public aliases, in the
 sense that the one who instantiates a template passes a public alias to
 a possibly private symbol.
 ...


I see. Then how are you going to handle the following:

module a;

private auto foo(int x){ }
auto foo(double x){ }

auto call(alias a){ a(0); }

void x(){
     call!foo(); // ok
}

module b;
import a;

void y(){
     call!foo(); // _same_ symbol as in module 'a'.
}



 It's a question of transitivity what the template can do with the symbol.

 Currently it's not possible to call private functions or private methods
 of variables that are passed via alias parameter.
 ...


That's probably fine. (It is certainly nothing to touch while fixing the 
private symbol clash issue.)
Dec 21 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 12/21/2013 11:17 PM, Timon Gehr wrote:

 On 12/21/2013 11:03 PM, Martin Nowak wrote:

 On 12/21/2013 10:28 PM, Timon Gehr wrote:

 Well, you pass an alias. Are you saying that for alias template
 arguments, the scope of the template declaration applies for
 accessibility checking? I.e. we cannot pass private symbols as alias
 arguments? I think this is an unreasonably high price to pay.


 You can pass them and basically it behaves like public aliases, in the
 sense that the one who instantiates a template passes a public alias to
 a possibly private symbol.
 ...


 I see. Then how are you going to handle the following:

 module a;

 private auto foo(int x){ }
 auto foo(double x){ }

 auto call(alias a){ a(0); }

 void x(){
      call!foo(); // ok
 }


This will work.


 module b;
 import a;

 void y(){
      call!foo(); // _same_ symbol as in module 'a'.
 }


This would say, Error: overload auto a.foo(int) is not accessible from 
module b.
Dec 21 2013
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/21/2013 11:56 PM, Martin Nowak wrote:

 On 12/21/2013 11:17 PM, Timon Gehr wrote:

 ...

 I see. Then how are you going to handle the following:

 module a;

 private auto foo(int x){ }
 auto foo(double x){ }

 auto call(alias a){ a(0); }

 void x(){
      call!foo(); // ok
 }


 This will work.


 module b;
 import a;

 void y(){
      call!foo(); // _same_ symbol as in module 'a'.
 }


 This would say, Error: overload auto a.foo(int) is not accessible from
 module b.


How is this going to work? Are you arguing for creating multiple 
instances of 'call'?
Dec 21 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 12/22/2013 12:05 AM, Timon Gehr wrote:


 How is this going to work?


Ah sorry, I misread your example.
The call template function is in module a, so it has access too.


 Are you arguing for creating multiple instances of 'call'?


So this is indeed not necessary, the function is called within the 
template, so only the template scope determines whether a function is 
accessible.



 Dec 22 2013



  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 12/22/2013 08:48 PM, Martin Nowak wrote:

 On 12/22/2013 12:05 AM, Timon Gehr wrote:


 How is this going to work?


 Ah sorry, I misread your example.
 The call template function is in module a, so it has access too.

  > Are you arguing for creating multiple instances of 'call'?

 So this is indeed not necessary, the function is called within the
 template, so only the template scope determines whether a function is
 accessible.


Therefore none of the following two calls will work?

module a;
import b;

auto call(alias a){ a(0); }

void x(){
     call!foo();
}

module b;
import a;

private auto foo(int x){ }
auto foo(double x){ }

void y(){
     call!foo();
}



 Dec 22 2013



prev sibling  parent reply "Dicebot" <public dicebot.lv>  writes:


On Saturday, 21 December 2013 at 21:14:43 UTC, Martin Nowak wrote:

 If I make a refactoring and move a function from module A to B, 
 but the function calls an overload set in A it should not have 
 a different overload resolution.


If MUST have different overload resolution if B does not have 
access to those A function. Anything else is damn broken module 
system. Why would anyone expect to move functions between modules 
and have accessibility rules magically persistent?


 Similar case, if I pass an overload set via alias to a traits 
 template in module B, the template should follow the same 
 overload resolution.
 It's very important that we preserve these properties.


It is a major design issue with templates using declaration scope 
for access resolution instead of instantation scope. I think it 
was a mistake and is unrelated to general visibility rules.



 Dec 21 2013



 next sibling parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 12/21/2013 10:36 PM, Dicebot wrote:


 It is a major design issue with templates using declaration scope for
 access resolution instead of instantation scope. I think it was a
 mistake and is unrelated to general visibility rules.


I think there is quite some overlap.



 Dec 21 2013



prev sibling  parent reply Martin Nowak <code dawg.eu>  writes:


On 12/21/2013 10:36 PM, Dicebot wrote:

 If MUST have different overload resolution if B does not have access to
 those A function. Anything else is damn broken module system. Why would
 anyone expect to move functions between modules and have accessibility
 rules magically persistent?


It's important for example to define a forwarding template in a 
different module.

void call(alias funcs, Args...)(auto ref Args args)
{
     funcs(args); // overload resolution happens here
}


 It is a major design issue with templates using declaration scope for
 access resolution instead of instantation scope. I think it was a
 mistake and is unrelated to general visibility rules.


What? It's a major design win that we don't have to rerun semantic for 
every identical instantiation, in the same sense that we don't have to 
reinclude header files.
Or are you only talking about access checks?
I don't think that handling access resolution separately from look-up 
resolution is desirable.



 Dec 21 2013



  parent  "Dicebot" <public dicebot.lv>  writes:


On Saturday, 21 December 2013 at 22:14:51 UTC, Martin Nowak wrote:

 What? It's a major design win that we don't have to rerun 
 semantic for every identical instantiation, in the same sense 
 that we don't have to reinclude header files.


I know that and completely agree with that.


 Or are you only talking about access checks?


Yes.


 I don't think that handling access resolution separately from 
 look-up resolution is desirable.


That was my attitude initially when I have learned about this 
decision but it is root to certain class of problems in generic 
code and I have slowly moved to position that RAI for this 
exception is high enough.



 Dec 21 2013



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 12/21/2013 08:29 PM, Martin Nowak wrote:

 On Saturday, 21 December 2013 at 17:41:32 UTC, Timon Gehr wrote:

 Adding a private symbol to an overload set can break 3rd party code
 with these rules.


 Yes, this is understood, but it seems like a good tradeoff when compared
 to the alternative where overload resolution depends on look-up origin.
 The latter could easily break generic code and simple refactorings could
 introduce subtle bugs.
 ...


I think both options are similarly undesirable.


 Since private symbols are usually excluded from di files I don't think
 this design is viable.


 This, I don't have a good answer for, except that we have to put the
 private overload in the .di file as well.


 I think it would be better to disallow overload sets with mixed
 protection.


 Tried that, doesn't work. The most common counterexample is mixing
 private/public constructors.


You have a point there. :o)



 Dec 21 2013



  parent reply Martin Nowak <code dawg.eu>  writes:


On 12/21/2013 09:18 PM, Timon Gehr wrote:

 On 12/21/2013 08:29 PM, Martin Nowak wrote:

 On Saturday, 21 December 2013 at 17:41:32 UTC, Timon Gehr wrote:

 Adding a private symbol to an overload set can break 3rd party code
 with these rules.


 Yes, this is understood, but it seems like a good tradeoff when compared
 to the alternative where overload resolution depends on look-up origin.
 The latter could easily break generic code and simple refactorings could
 introduce subtle bugs.
 ...


 I think both options are similarly undesirable.


Mixed protection in overload sets are pretty rare.
So we just need to find an acceptable solution to deal
with them without breaking valid use-cases.
I can hardly come up with any use-case but the constructor example though.



 Dec 21 2013



  parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Sat, Dec 21, 2013 at 10:09:44PM +0100, Martin Nowak wrote:

 On 12/21/2013 09:18 PM, Timon Gehr wrote:

On 12/21/2013 08:29 PM, Martin Nowak wrote:

On Saturday, 21 December 2013 at 17:41:32 UTC, Timon Gehr wrote:

Adding a private symbol to an overload set can break 3rd party code
with these rules.


Yes, this is understood, but it seems like a good tradeoff when
compared to the alternative where overload resolution depends on
look-up origin.  The latter could easily break generic code and
simple refactorings could introduce subtle bugs.
...


I think both options are similarly undesirable.


 
 Mixed protection in overload sets are pretty rare.
 So we just need to find an acceptable solution to deal
 with them without breaking valid use-cases.
 I can hardly come up with any use-case but the constructor example
 though.


I was pretty upset when I encountered the following situation:

	---std/regex.d---
	...
	private struct Stack { ... }
	...

	---mymodule.d---
	...
	/* public */ struct Stack { ... }
	...

	---main.d---
	import std.regex;
	import mymodule;

	Stack s; // compile error: 'Stack' is ambiguous

I find it unacceptable that introducing a *private* symbol to std.regex
will break existing code (std.regex.Stack wasn't there in earlier
versions of Phobos). Since std.regex.Stack is private, why would it even
be in *any* overload set when compiling main.d??! This is leaky
abstraction at its best: changing the implementation of a module without
changing its API can randomly break user code due to newly-introduced
private symbols clashing with user-defined symbols.


T

-- 
Those who've learned LaTeX swear by it. Those who are learning LaTeX
swear at it. -- Pete Bleackley



 Dec 21 2013



  parent  Martin Nowak <code dawg.eu>  writes:


On 12/21/2013 10:22 PM, H. S. Teoh wrote:

 I was pretty upset when I encountered the following situation:

 	---std/regex.d---
 	...
 	private struct Stack { ... }
 	...

 	---mymodule.d---
 	...
 	/* public */ struct Stack { ... }
 	...

 	---main.d---
 	import std.regex;
 	import mymodule;

 	Stack s; // compile error: 'Stack' is ambiguous

 I find it unacceptable that introducing a *private* symbol to std.regex
 will break existing code (std.regex.Stack wasn't there in earlier
 versions of Phobos). Since std.regex.Stack is private, why would it even
 be in *any* overload set when compiling main.d??! This is leaky
 abstraction at its best: changing the implementation of a module without
 changing its API can randomly break user code due to newly-introduced
 private symbols clashing with user-defined symbols.


Yes, this is the big issue that we try to address with DIP22.
Happened to me too when I added a private `abs` helper to core.time
which then caused conflicts somewhere else in phobos.



 Dec 21 2013