www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Usability of "allMembers and derivedMembers traits now only return

reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
v2.071.2-b3 is bringing a change for this bug:

   https://issues.dlang.org/show_bug.cgi?id=15907

I don't agree with the current solution:

   http://dlang.org/changelog/2.071.2.html#traits-members-visibility

Modules should be able to use library templates without needing to mix 
them in first.

Do you think the solution in the change log usable? I don't think so 
because silently skipping my private members is an unexpected behavior 
that will cause bugs.

Further, do I understand the example right? Am I supposed to mixin the 
same template twice for two different types? The following code which 
adds another struct does not compile:

import std.stdio;
import std.traits;

enum UDA;
struct S
{
      UDA int visible;
      UDA private int invisible;
}

// only returns symbols visible from std.traits
static assert(getSymbolsByUDA!(S, UDA).length == 1);
// mixin the template instantiation, using a name to avoid namespace 
pollution
mixin getSymbolsByUDA!(S, UDA) symbols;
// as the template is instantiated in the current scope, it can see 
private members
static assert(symbols.getSymbolsByUDA.length == 2);

// --- The following is added by Ali: ---

struct S2 {
      UDA int s2;
}

mixin getSymbolsByUDA!(S2, UDA) symbolsS2;    // COMPILATION ERROR:
// Error: mixin deneme.getSymbolsByUDA!(S2, UDA) TList isn't a template

static assert(symbolsS2.getSymbolsByUDA.length == 1);

void main() {
     foreach (i; 0 .. symbols.getSymbolsByUDA.length) {
         // ...
     }
}

I can't wrap my head around the fact that a library template called by 
my module cannot see my private members.

Ali
Aug 30 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 v2.071.2-b3 is bringing a change for this bug:

   https://issues.dlang.org/show_bug.cgi?id=15907

 I don't agree with the current solution:

   
 http://dlang.org/changelog/2.071.2.html#traits-members-visibility

 Modules should be able to use library templates without needing 
 to mix them in first.

 Do you think the solution in the change log usable? I don't 
 think so because silently skipping my private members is an 
 unexpected behavior that will cause bugs.

 Further, do I understand the example right? Am I supposed to 
 mixin the same template twice for two different types? The 
 following code which adds another struct does not compile:

 import std.stdio;
 import std.traits;

 enum UDA;
 struct S
 {
      UDA int visible;
      UDA private int invisible;
 }

 // only returns symbols visible from std.traits
 static assert(getSymbolsByUDA!(S, UDA).length == 1);
 // mixin the template instantiation, using a name to avoid 
 namespace pollution
 mixin getSymbolsByUDA!(S, UDA) symbols;
 // as the template is instantiated in the current scope, it can 
 see private members
 static assert(symbols.getSymbolsByUDA.length == 2);

 // --- The following is added by Ali: ---

 struct S2 {
      UDA int s2;
 }

 mixin getSymbolsByUDA!(S2, UDA) symbolsS2;    // COMPILATION 
 ERROR:
 // Error: mixin deneme.getSymbolsByUDA!(S2, UDA) TList isn't a 
 template

 static assert(symbolsS2.getSymbolsByUDA.length == 1);

 void main() {
     foreach (i; 0 .. symbols.getSymbolsByUDA.length) {
         // ...
     }
 }

 I can't wrap my head around the fact that a library template 
 called by my module cannot see my private members.

 Ali
Yes I agree, a change of the specifications in a dot release seems a bit extreme, especially since there was a deeper problem **before** the broken imports were fixed. This problem was discovered a while back when the library traits related to UDAs were added to phobos. The little story began here: https://issues.dlang.org/show_bug.cgi?id=15335. Then the proposal to give super powers to certain traits verbs: https://issues.dlang.org/show_bug.cgi?id=15371.
Aug 30 2016
parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 30 August 2016 at 22:46:58 UTC, Basile B. wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 v2.071.2-b3 is bringing a change for this bug:
Yes I agree, a change of the specifications in a dot release seems a bit extreme, especially since there was a deeper problem **before** the broken imports were fixed. This problem was discovered a while back when the library traits related to UDAs were added to phobos. The little story began here: https://issues.dlang.org/show_bug.cgi?id=15335. Then the proposal to give super powers to certain traits verbs: https://issues.dlang.org/show_bug.cgi?id=15371.
To be clear, the logic I see for traits "getMember", "allMember", "getOverloads", "derivedMembers" (etc, all the traits that might be today limited by the protection attribute) is: allow them to see everything, then use "getProtection" if you wanna be conform with the protection attributes. Maybe it's worth a DIP ? Casual and informal discussions have **failed**. I see now the answer that mentions ".tupleof". I see no valid logic that would allow ".tupleof" to see everything and not the traits.
Aug 31 2016
parent reply Ethan Watson <gooberman gmail.com> writes:
On Wednesday, 31 August 2016 at 08:06:05 UTC, Basile B. wrote:
 allow them to see everything, then use "getProtection" if you 
 wanna be conform with the protection attributes.
That's how it used to work, but getProtection would fail if the symbol wasn't public. Which led to me using a workaround to something of this effect: enum PrivacyLevel : string { Public = "public", Private = "private", Protected = "protected", Export = "export", Package = "package", Inaccessible = "inaccessible" }; //---------------------------------------------------------------------------- template PrivacyOf( alias symbol ) { static if( __traits( compiles, __traits( getProtection, symbol ) ) ) { enum PrivacyOf = cast(PrivacyLevel) __traits( getProtection, symbol ); } else { enum PrivacyOf = PrivacyLevel.Inaccessible; } } //---------------------------------------------------------------------------- Still not an ideal solution - because if I'm trying to serialise and deserialise everything in between module reloads I still need to do the .tupleof method to get all data members; and if I want to define privacy levels for functions I'm automatically binding from C++ I need to muddy those waters with UDAs etc.
Aug 31 2016
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 31 August 2016 at 08:33:28 UTC, Ethan Watson wrote:
 That's how it used to work, but getProtection would fail if the 
 symbol wasn't public. Which led to me using a workaround to 
 something of this effect:
Yeah, I kinda regret the design of getProtection (which is basically 100% my fault, I implemented it myself and pushed it through without thinking about private - I was only interested in public vs export for my personal use case...), but if getMember worked on private things too, getProtection would never have to return inaccessible and it would be pretty again.
Aug 31 2016
prev sibling next sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8/31/16, Ali Çehreli via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 mixin getSymbolsByUDA!(S, UDA) symbols;
This is such a bizarre workaround to be listed in the changelog since mixing in non-mixin templates is not an official feature (am I wrong?). getSymbolsByUDA is a template, not a mixin template.
Aug 30 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-08-31 01:01, Andrej Mitrovic via Digitalmars-d wrote:

 This is such a bizarre workaround to be listed in the changelog since
 mixing in non-mixin templates is not an official feature (am I
 wrong?).
Yes. Originally one could not use the "mixin" keyword in front of a template. There was no difference between mixin templates and non-mixin templates, on the declaration site. Later the language was changed to allow to put "mixin" in front of a template. That restricted all mixin templates to only be used as mixin templates. It's still possible to mixin non-mixin templates, most likely to avoid breaking existing code. -- /Jacob Carlborg
Aug 30 2016
prev sibling next sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8/31/16, Ali Çehreli via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 v2.071.2-b3 is bringing a change for this bug:

    https://issues.dlang.org/show_bug.cgi?id=15907
PSA: If all one cares about are UDAs for fields and not functions then .tupleof is still a viable workaround. Working example: ----- import getSymbols; enum UDA; struct S { UDA float visible; float dont_care; private UDA int invisible; } void main() { static assert(getSymbolsByUDA!(S, UDA).length == 2); static assert(getSymbolsByUDA!(S, UDA).stringof == "tuple(visible, invisible)"); } ----- ----- module getSymbols; import std.meta : AliasSeq; import std.traits : staticIndexOf; template getSymbolsByUDA ( T, alias uda ) { alias getSymbolsByUDA = getSymbolsByUDAImpl!(T, uda); } template getSymbolsByUDAImpl ( T, alias uda, size_t idx = 0 ) { static if (idx + 1 < T.tupleof.length) { static if (hasUDA!(T.tupleof[idx], uda)) alias Field = AliasSeq!(T.tupleof[idx]); else alias Field = AliasSeq!(); alias getSymbolsByUDAImpl = AliasSeq!(Field, getSymbolsByUDAImpl!(T, uda, idx + 1)); } else { static if (hasUDA!(T.tupleof[idx], uda)) alias getSymbolsByUDAImpl = AliasSeq!(T.tupleof[idx]); else alias getSymbolsByUDAImpl = AliasSeq!(); } } template hasUDA ( alias field, alias uda ) { enum hasUDA = staticIndexOf!(uda, __traits(getAttributes, field)) != -1; } ----- This is the reason msgpack-d still works and wasn't broken by the change.
Aug 30 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-08-31 02:16, Andrej Mitrovic via Digitalmars-d wrote:

 PSA: If all one cares about are UDAs for fields and not functions then
 .tupleof is still a viable workaround. Working example:

 -----
 import getSymbols;

 enum UDA;

 struct S
 {
      UDA float visible;

     float dont_care;

     private  UDA int invisible;
 }

 void main()
 {
     static assert(getSymbolsByUDA!(S, UDA).length == 2);
     static assert(getSymbolsByUDA!(S, UDA).stringof == "tuple(visible,
 invisible)");
 }
 -----

 -----
 module getSymbols;

 import std.meta : AliasSeq;
 import std.traits : staticIndexOf;

 template getSymbolsByUDA ( T, alias uda )
 {
     alias getSymbolsByUDA = getSymbolsByUDAImpl!(T, uda);
 }

 template getSymbolsByUDAImpl ( T, alias uda, size_t idx = 0 )
 {
     static if (idx + 1 < T.tupleof.length)
     {
         static if (hasUDA!(T.tupleof[idx], uda))
             alias Field = AliasSeq!(T.tupleof[idx]);
         else
             alias Field = AliasSeq!();

         alias getSymbolsByUDAImpl = AliasSeq!(Field,
             getSymbolsByUDAImpl!(T, uda, idx + 1));
     }
     else
     {
         static if (hasUDA!(T.tupleof[idx], uda))
             alias getSymbolsByUDAImpl = AliasSeq!(T.tupleof[idx]);
         else
             alias getSymbolsByUDAImpl = AliasSeq!();
     }
 }

 template hasUDA ( alias field, alias uda )
 {
     enum hasUDA = staticIndexOf!(uda, __traits(getAttributes, field)) != -1;
 }
 -----

 This is the reason msgpack-d still works and wasn't broken by the change.
Ah, nice workaround. Last time I tried to use __traits(getAttributes) with a "tupleof expression" it didn't work. -- /Jacob Carlborg
Aug 30 2016
prev sibling next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
yeah, the whole feature smells for me. the sole need of mixin 
hack indicates that something is very wrong here. i never ever 
needed that for normal D code. and now suddenly i have to 
remember that some thing is a template, that it needs mixing it 
to work properly, etc.

all this mess should be resolved in compiler by assigning 
template *two* visibility scopes: one is template's "normal" 
scope (so it can see symbols from it's originating module), and 
second is it's "instantiation" scope. after all, this is exactly 
what programmer is exepecting. current solution is not a solution 
at all, it's a hacky workaround promoted to "official technique".
Aug 30 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 31 August 2016 at 05:33:50 UTC, ketmar wrote:
 all this mess should be resolved in compiler by assigning 
 template *two* visibility scopes: one is template's "normal" 
 scope (so it can see symbols from it's originating module), and 
 second is it's "instantiation" scope. after all, this is 
 exactly what programmer is exepecting. current solution is not 
 a solution at all, it's a hacky workaround promoted to 
 "official technique".
This would require a new instance for every template instantiation. The instantiation scope does not leak into the template by design, it allows us to cache instantiations and greatly speed up compilation.
Sep 03 2016
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Saturday, 3 September 2016 at 14:48:54 UTC, Martin Nowak wrote:
 On Wednesday, 31 August 2016 at 05:33:50 UTC, ketmar wrote:
 all this mess should be resolved in compiler by assigning 
 template *two* visibility scopes: one is template's "normal" 
 scope (so it can see symbols from it's originating module), 
 and second is it's "instantiation" scope. after all, this is 
 exactly what programmer is exepecting. current solution is not 
 a solution at all, it's a hacky workaround promoted to 
 "official technique".
This would require a new instance for every template instantiation. The instantiation scope does not leak into the template by design, it allows us to cache instantiations and greatly speed up compilation.
just a wrapper class, which will hold the actual instantiation and a scope. most of the code should "pass thru" the wrapper (i thing that `alias this` can be used for that, but have to check it), yet `__traits` can use additional info. sure, that will require some work, but it's not that impossible, and should not ruin caching. yeah, we will waste additional 16/32 bytes per template instance this way. not that much, and it will solve the problem in most natural way.
Sep 03 2016
parent Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 15:48:24 UTC, ketmar wrote:
 just a wrapper class, which will hold the actual instantiation 
 and a scope. most of the code should "pass thru" the wrapper (i 
 thing that `alias this` can be used for that, but have to check 
 it), yet `__traits` can use additional info. sure, that will 
 require some work, but it's not that impossible, and should not 
 ruin caching. yeah, we will waste additional 16/32 bytes per 
 template instance this way. not that much, and it will solve 
 the problem in most natural way.
Well, if you're making a difference based on the instantiation scope inside the template, then you're are leaking it into the template, do require a new instance for every instantiation, are ruining caching, do require unique mangling and redundant codegen. We have mixin templates exactly for the purpose of instantiating them in the origin scope.
Sep 03 2016
prev sibling next sibling parent reply Ethan Watson <gooberman gmail.com> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 I don't agree with the current solution:
I'm somewhat surprised myself that "allMembers doesn't return all members" needs highlighting. Why not have a new trait "allVisibleMembers" and just fix the privacy issues?
Aug 31 2016
parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 31 August 2016 at 08:36:37 UTC, Ethan Watson wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 I don't agree with the current solution:
I'm somewhat surprised myself that "allMembers doesn't return all members" needs highlighting. Why not have a new trait "allVisibleMembers" and just fix the privacy issues?
nice idea, but this doesn't change the fact that the traits that access the results of the "omniscient" allMember must be tweaked to access everything.
Aug 31 2016
parent reply Ethan Watson <gooberman gmail.com> writes:
On Wednesday, 31 August 2016 at 09:25:52 UTC, Basile B. wrote:
 nice idea, but this doesn't change the fact that the traits 
 that access the results of the "omniscient" allMember must be 
 tweaked to access everything.
I'm okay with this. My PrivacyLevel workaround does exactly this in fact. But I would like to be able to read (and write) all members of a class without needing to mixin a template and without having to resort to .tupleof. Use case here is the extensive struct use we have, if we want them to match C++ exactly then that's where mixins become potentially hairy.
Aug 31 2016
parent Ethan Watson <gooberman gmail.com> writes:
On Wednesday, 31 August 2016 at 09:30:43 UTC, Ethan Watson wrote:
 I'm okay with this. My PrivacyLevel workaround does exactly 
 this in fact.
I keep forgetting that I'm all open sourced now and can just link directly to the full example. https://github.com/Remedy-Entertainment/binderoo/blob/master/binderoo_client/d/src/binderoo/objectprivacy.d
Aug 31 2016
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Ugh, it really should just give everything and have getMember 
bypass it. That won't even break any code!
Aug 31 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:
 Ugh, it really should just give everything and have getMember 
 bypass it. That won't even break any code!
you're right. "allMembers" means "all" after all. Another reason why the idea of "allVisibleMembers" is good. Puristes will be able to use this traits without messing with "getProtection".
Aug 31 2016
parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 31 August 2016 at 13:29:52 UTC, Basile B. wrote:
 On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe 
 wrote:
 Ugh, it really should just give everything and have getMember 
 bypass it. That won't even break any code!
you're right. "allMembers" means "all" after all. Another reason why the idea of "allVisibleMembers" is good. Puristes will be able to use this traits without messing with "getProtection".
https://github.com/dlang/DIPs/pull/39 Co-authors welcome.
Sep 01 2016
parent Martin Nowak <code dawg.eu> writes:
On Thursday, 1 September 2016 at 19:30:41 UTC, Basile B. wrote:
 https://github.com/dlang/DIPs/pull/39

 Co-authors welcome.
Slow down a bit until we've finally decided out how to resolve the problems.
Sep 03 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:
 Ugh, it really should just give everything and have getMember 
 bypass it. That won't even break any code!
It will, e.g. having getMember bypass protection means vibe.d would now serialize private members https://github.com/rejectedsoftware/vibe.d/blob/c1180791de61d0f8c9bfb584c2551a5b64627e32/source/vibe/internal/meta/traits.d#L146. Until now getMember was just a DotIdExp (i.e. `T.indent`/`var.ident`), so it was used to test whether sth. can be accessed.
Sep 03 2016
next sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/3/16, Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:
 Ugh, it really should just give everything and have getMember
 bypass it. That won't even break any code!
It will, e.g. having getMember bypass protection means vibe.d would now serialize private members
Then just add a check in vibe.d itself to avoid serializing private members. You can still call getProtection on the symbol and skip serializing it. Alternatively you can use UDAs so you can mark which fields should or shouldn't be serializible. For example https://github.com/msgpack/msgpack-d/blob/6046808c2e678e27cb2e2d9314241c361a6fd0ae/src/msgpack/attribute.d#L21 The bottom line is with restricting access to private symbols you have no choice on the matter, while allowing access lets you specialize whether to ignore the symbols or not.
Sep 03 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/3/16 6:39 PM, Andrej Mitrovic via Digitalmars-d wrote:
 On 9/3/16, Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:
 Ugh, it really should just give everything and have getMember
 bypass it. That won't even break any code!
It will, e.g. having getMember bypass protection means vibe.d would now serialize private members
Then just add a check in vibe.d itself to avoid serializing private members. You can still call getProtection on the symbol and skip serializing it. Alternatively you can use UDAs so you can mark which fields should or shouldn't be serializible. For example https://github.com/msgpack/msgpack-d/blob/6046808c2e678e27cb2e2d9314241c361a6fd0ae/src/msgpack/attribute.d#L21 The bottom line is with restricting access to private symbols you have no choice on the matter, while allowing access lets you specialize whether to ignore the symbols or not.
I didn't follow this closely, but clearly we need some means to access private members for introspection purposes. I hope we don't paint ourselves out of that corner. -- Andrei
Sep 03 2016
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Saturday, September 03, 2016 19:30:54 Andrei Alexandrescu via Digitalmars-d 
wrote:
 On 9/3/16 6:39 PM, Andrej Mitrovic via Digitalmars-d wrote:
 On 9/3/16, Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> 
wrote:
 On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:
 Ugh, it really should just give everything and have getMember
 bypass it. That won't even break any code!
It will, e.g. having getMember bypass protection means vibe.d would now serialize private members
Then just add a check in vibe.d itself to avoid serializing private members. You can still call getProtection on the symbol and skip serializing it. Alternatively you can use UDAs so you can mark which fields should or shouldn't be serializible. For example https://github.com/msgpack/msgpack-d/blob/6046808c2e678e27cb2e2d9314241c36 1a6fd0ae/src/msgpack/attribute.d#L21 The bottom line is with restricting access to private symbols you have no choice on the matter, while allowing access lets you specialize whether to ignore the symbols or not.
I didn't follow this closely, but clearly we need some means to access private members for introspection purposes. I hope we don't paint ourselves out of that corner. -- Andrei
Well, that certainly seems to be exactly what we're doing with the changes that Ali is complaining about. Previously __traits(allMembers, T) listed _all_ members, whereas now it wil only list the ones that are accessible. The same for __traits(derivedMembers, T). So, they'll still give you the private members if you use them inside the module in question but not elsewhere. I would have thought that the way to handle this would be to have them list all members regardless of access level and then to have code check for the access level if it cares. I would have thought that doing stuff like looking at private symbols would be perfectly okay with no errors or deprecation warnings and that it would just be _accessing_ the symbol that would be disallowed and would require you to check for in code that used introspection. But clearly, there must be something that I misunderstand here, because that's not at all where this seems to be going. I do get the feeling that too much of what we do with features like this is ad-hoc without really designing things up front, and then we keep having to tweak stuff as we go along in ways that aren't always particularly nice. And this particular change seems to just be a reaction to problems that we ran into with changing how imports work. And maybe it's the right choice, but it sure doesn't seem like it. - Jonathan M Davis
Sep 03 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/3/16 7:57 PM, Jonathan M Davis via Digitalmars-d wrote:
 Previously __traits(allMembers, T) listed _all_ members, whereas now it wil
 only list the ones that are accessible. The same for
 __traits(derivedMembers, T). So, they'll still give you the private members
 if you use them inside the module in question but not elsewhere.
That... doesn't sound good. I wonder how such important changes slip by Walter and myself unnoticed. My thinking is that the plebes should be able to access things via the object.member syntax by obeying the usual visibility rules. But __traits(allMembers, T) should be the reflection backdoor that gives the savvy users total access, at the obvious cost of an awkward syntax. The fact that __traits(allMembers, T) compiles in all cases and has DIFFERENT semantics depending on whether T is in the same vs. a different module is the deadly sign of poor language design. Martin, any chance we can undo this change to the language? Andrei
Sep 03 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Saturday, 3 September 2016 at 20:40:57 UTC, Andrei 
Alexandrescu wrote:
 Martin, any chance we can undo this change to the language?
The problem is deeper. If he undoes the change then what will happen again is that the results of allMembers won't be usable by the other traits, e.g getMember, because them are still limited by the protection attributes.
Sep 03 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/3/16 10:50 PM, Basile B. wrote:
 On Saturday, 3 September 2016 at 20:40:57 UTC, Andrei Alexandrescu wrote:
 Martin, any chance we can undo this change to the language?
The problem is deeper. If he undoes the change then what will happen again is that the results of allMembers won't be usable by the other traits, e.g getMember, because them are still limited by the protection attributes.
Where are the past discussions on this matter? -- Andrei
Sep 03 2016
parent Basile B. <b2.temp gmx.com> writes:
On Saturday, 3 September 2016 at 20:54:19 UTC, Andrei 
Alexandrescu wrote:
 On 9/3/16 10:50 PM, Basile B. wrote:
 On Saturday, 3 September 2016 at 20:40:57 UTC, Andrei 
 Alexandrescu wrote:
 Martin, any chance we can undo this change to the language?
The problem is deeper. If he undoes the change then what will happen again is that the results of allMembers won't be usable by the other traits, e.g getMember, because them are still limited by the protection attributes.
Where are the past discussions on this matter? -- Andrei
The change in 2.071.2-beta3 to allMembers is designed to fix https://issues.dlang.org/show_bug.cgi?id=15907 which happens because of a __traits(getMember,...) on the results of a __traits(allMembers,...).
Sep 03 2016
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-03 22:40, Andrei Alexandrescu wrote:

 That... doesn't sound good. I wonder how such important changes slip by
 Walter and myself unnoticed.
Here's the PR that introduced the change: https://github.com/dlang/dmd/pull/6078 -- /Jacob Carlborg
Sep 03 2016
next sibling parent reply Ethan Watson <gooberman gmail.com> writes:
On Saturday, 3 September 2016 at 21:54:24 UTC, Jacob Carlborg 
wrote:
 Here's the PR that introduced the change: 
 https://github.com/dlang/dmd/pull/6078
I'm certainly not going to upgrade to the next DMD if this change is retained. allMembers not returning all members makes introspection entirely useless when it comes to Binderoo. The wrong conclusions were made from that bug to begin with it seems. allMembers should return all members. getProtection should report on the protection of a symbol *regardless* of whether getMember will succeed or not (this is currently why I have my PrivacyOf workaround - to stop the compiler crashing when using a template instead of a mixin template to do introspection). getMember itself, well, I'd honestly prefer if there was a way to get to members without having to correlate with .tupleof as it will simplify Binderoo code. The .tupleof method doesn't help me when it comes to introspecting private/protected functions for C++ binding.
Sep 03 2016
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/03/2016 03:13 PM, Ethan Watson wrote:
 On Saturday, 3 September 2016 at 21:54:24 UTC, Jacob Carlborg wrote:
 allMembers not returning all members makes introspection
 entirely useless when it comes to Binderoo.
Same problem with Weka's code base...
 getMember itself, well, I'd honestly prefer if there was a way to get to
 members without having to correlate with .tupleof as it will simplify
 Binderoo code.
Again, same problem at Weka... Protection attributes are for protecting programmers from implementation changes. If I reach for allMembers or getMember, it's an explicit way of saying "I don't care about the consequences." Ali
Sep 07 2016
next sibling parent Chris Wright <dhasenan gmail.com> writes:
On Wed, 07 Sep 2016 12:21:40 -0700, Ali Çehreli wrote:
 Protection attributes are for protecting programmers from implementation
 changes. If I reach for allMembers or getMember, it's an explicit way of
 saying "I don't care about the consequences."
If it's just about implementation changes, allMembers / getMember is a way of inspecting types to get what currently exists. It will continue to be correct in the face of changing fields.
Sep 07 2016
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/7/16 9:21 PM, Ali Çehreli wrote:
 On 09/03/2016 03:13 PM, Ethan Watson wrote:
 On Saturday, 3 September 2016 at 21:54:24 UTC, Jacob Carlborg wrote:
 allMembers not returning all members makes introspection
 entirely useless when it comes to Binderoo.
Same problem with Weka's code base...
 getMember itself, well, I'd honestly prefer if there was a way to get to
 members without having to correlate with .tupleof as it will simplify
 Binderoo code.
Again, same problem at Weka... Protection attributes are for protecting programmers from implementation changes. If I reach for allMembers or getMember, it's an explicit way of saying "I don't care about the consequences."
Martin has a PR to improve on this, could you all please take a look? I can't really do much until Sunday, I'm on a WiFi connection slower than a herd of turtles, and which only works if I hold my laptop over my head. So I'll click "Send" on this then push the roof with my laptop. -- Andrei
Sep 08 2016
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/3/16 11:54 PM, Jacob Carlborg wrote:
 On 2016-09-03 22:40, Andrei Alexandrescu wrote:

 That... doesn't sound good. I wonder how such important changes slip by
 Walter and myself unnoticed.
Here's the PR that introduced the change: https://github.com/dlang/dmd/pull/6078
Thanks. -- Andrei
Sep 03 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/3/2016 2:54 PM, Jacob Carlborg wrote:
 On 2016-09-03 22:40, Andrei Alexandrescu wrote:

 That... doesn't sound good. I wonder how such important changes slip by
 Walter and myself unnoticed.
Here's the PR that introduced the change: https://github.com/dlang/dmd/pull/6078
Looks like I'm to blame as I pulled it.
Sep 04 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 20:40:57 UTC, Andrei 
Alexandrescu wrote:
 On 9/3/16 7:57 PM, Jonathan M Davis via Digitalmars-d wrote:
 Previously __traits(allMembers, T) listed _all_ members, 
 whereas now it wil
 only list the ones that are accessible. The same for
 __traits(derivedMembers, T). So, they'll still give you the 
 private members
 if you use them inside the module in question but not 
 elsewhere.
That... doesn't sound good. I wonder how such important changes slip by Walter and myself unnoticed.
It didn't slip, but I wish Walter had at least stated his opinion on the PR before merging.
 My thinking is that the plebes should be able to access things 
 via the object.member syntax by obeying the usual visibility 
 rules. But __traits(allMembers, T) should be the reflection 
 backdoor that gives the savvy users total access, at the 
 obvious cost of an awkward syntax.
As explained several times here and in the announce thread, private members have never been accessible, other than introspecting attributes, and making them accessible comes with a performance cost and a fairly big language change. So the real question is, why do we need introspection without access, and can we handle that few cases with mixin templates. If we really need introspection of private members than we might need to go back to the drawing board and modify the visibility concept introduced with DIP22.
Sep 03 2016
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 4 September 2016 at 03:14:18 UTC, Martin Nowak wrote:
 It didn't slip, but I wish Walter had at least stated his 
 opinion on the PR before merging.

 My thinking is that the plebes should be able to access things 
 via the object.member syntax by obeying the usual visibility 
 rules. But __traits(allMembers, T) should be the reflection 
 backdoor that gives the savvy users total access, at the 
 obvious cost of an awkward syntax.
As explained several times here and in the announce thread, private members have never been accessible, other than introspecting attributes, and making them accessible comes with a performance cost and a fairly big language change. So the real question is, why do we need introspection without access, and can we handle that few cases with mixin templates. If we really need introspection of private members than we might need to go back to the drawing board and modify the visibility concept introduced with DIP22.
While I do understand, that there could be a potential performance when private members could be changed around because they are not visible form outside. I fail to see how we would take advantage of that without breaking our object-model.
Sep 03 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 4 September 2016 at 03:21:05 UTC, Stefan Koch wrote:
 While I do understand, that there could be a potential 
 performance when  private members could be changed around 
 because they are not visible form outside.

 I fail to see how we would take advantage of that without 
 breaking our object-model.
It's mostly about linkage of private methods.
Sep 03 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 04 Sep 2016 03:29:59 +0000
schrieb Martin Nowak <code dawg.eu>:

 On Sunday, 4 September 2016 at 03:21:05 UTC, Stefan Koch wrote:
 While I do understand, that there could be a potential 
 performance when  private members could be changed around 
 because they are not visible form outside.

 I fail to see how we would take advantage of that without 
 breaking our object-model.  
It's mostly about linkage of private methods.
Allowing access to private fields / functions also means that those members are part of the public API/ABI of a library. So changing private members can break dependent libraries...
Sep 04 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-09-04 09:39, Johannes Pfau wrote:

 Allowing access to private fields / functions also means that
 those members are part of the public API/ABI of a library. So changing
 private members can break dependent libraries...
They've always been accessible using .tupleof and taking the address of a member, before the recent lookup rule changes. -- /Jacob Carlborg
Sep 04 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 5:14 AM, Martin Nowak wrote:
 If we really need introspection of private members than we might need to
 go back to the drawing board and modify the visibility concept
 introduced with DIP22.
Thanks for answering. Yes, we really need introspection of private members. One way or another we need to support that. -- Andrei
Sep 04 2016
parent reply David Nadlinger <code klickverbot.at> writes:
On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu 
wrote:
 Thanks for answering. Yes, we really need introspection of 
 private members. One way or another we need to support that.
Do we, though? It's easy to state a general claim like this, but I find it hard to actually justify. — David
Sep 04 2016
next sibling parent David Nadlinger <code klickverbot.at> writes:
On Sunday, 4 September 2016 at 12:37:47 UTC, David Nadlinger 
wrote:
 Do we, though? It's easy to state a general claim like this, 
 but I find it hard to actually justify.
(And to pre-empt the collective freak-out, note that this statement is coming from somebody who has done quite a bit of pioneering serialisation/meta-programming work. — David)
Sep 04 2016
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 2:37 PM, David Nadlinger wrote:
 On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu wrote:
 Thanks for answering. Yes, we really need introspection of private
 members. One way or another we need to support that.
Do we, though? It's easy to state a general claim like this, but I find it hard to actually justify.
Static introspection is by far the most powerful feature of D. The last thing to do with it is hamstring it with nonsense like private :o). Quoting elsethread:
 There are a few cases I can think of:

 * Serialization and deserialization, shallow and deep. These would need access
to the object layout, and possibly field names too (for cross-checking and
versioning purposes).

 * Binary saving, loading, and fixup (a subset of
serialization/deserialization) - that thing when you copy raw memory to a file
and then load it raw and fix pointers up.

 * Memory management and garbage collection: access to field types allows
efficient generic tracing.

 * Database interfacing, automated binding, object-relational mapping, etc. It
stands to reason that field names would be needed and fields would be private,
yet the database loader does have access to them.

 Of course the more important applications are those I can't yet imagine. The
way I see it introspection must have full power and unfettered access. So
definitely it must be able to pass through regular protection.
Andrei
Sep 04 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 4 Sep 2016 14:54:33 +0200
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 9/4/16 2:37 PM, David Nadlinger wrote:
 On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu
 wrote:  
 Thanks for answering. Yes, we really need introspection of private
 members. One way or another we need to support that.  
Do we, though? It's easy to state a general claim like this, but I find it hard to actually justify.
Static introspection is by far the most powerful feature of D. The last thing to do with it is hamstring it with nonsense like private :o). Quoting elsethread:
 There are a few cases I can think of:

 * Serialization and deserialization, shallow and deep. These would
 need access to the object layout, and possibly field names too (for
 cross-checking and versioning purposes).

 * Binary saving, loading, and fixup (a subset of
 serialization/deserialization) - that thing when you copy raw
 memory to a file and then load it raw and fix pointers up.

 * Memory management and garbage collection: access to field types
 allows efficient generic tracing.

 * Database interfacing, automated binding, object-relational
 mapping, etc. It stands to reason that field names would be needed
 and fields would be private, yet the database loader does have
 access to them.

 Of course the more important applications are those I can't yet
 imagine. The way I see it introspection must have full power and
 unfettered access. So definitely it must be able to pass through
 regular protection.  
Andrei
All these examples need access to private data fields only. Most performance optimizations are only possible for private functions anyway. So is there a use case to access/call private functions?
Sep 05 2016
parent Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Monday, 5 September 2016 at 07:48:11 UTC, Johannes Pfau wrote:
 Am Sun, 4 Sep 2016 14:54:33 +0200
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 9/4/16 2:37 PM, David Nadlinger wrote:
 [...]
Static introspection is by far the most powerful feature of D. The last thing to do with it is hamstring it with nonsense like private :o). Quoting elsethread:
 [...]
Andrei
All these examples need access to private data fields only. Most performance optimizations are only possible for private functions anyway. So is there a use case to access/call private functions?
One use case would be accessing UDA on private fields, to inspect which fields need to be injected with a dependency, in aggregate type, just like spring from java does with autowire annotation. You mark a private field with autowired, and spring calls associated setter for the field. Does tupleof allow access to associated UDAs of contained fields, as well field names?
Sep 05 2016
prev sibling next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/04/2016 05:37 AM, David Nadlinger wrote:
 On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu wrote:
 Thanks for answering. Yes, we really need introspection of private
 members. One way or another we need to support that.
Do we, though? It's easy to state a general claim like this, but I find it hard to actually justify. — David
Let me try to reword what I've already said elsewhere in this thread. As the user of a library I shouldn't need to know whether a template of that library uses __traits(allMembers) or not. Unfortunately, if __traits(allMembers) depends on access rights, I am forced to mixin *every* template in my code because I don't and should not know the template's implementation. If I don't know their implementation, I have to prepare myself for the worst and mixin *every* template. That's insanity. Even if I know that they don't use __traits(allMembers) today, they may change their implementation in the future. So, I really have to mixin every template today. Further, I have to be on my toes and watch every feature of every library in case they changed e.g. a function to a function template in a new release. Oh yes, I have to mixin that new template as well! (Again, because I can't be sure whether they use __traits(allMembers) or not.) This new behavior and its mixin workaround is so insane to me that I can't even find ways to spell it out clearly. This behavior kills IFTI altogether because I don't know who uses __traits(allMembers). I can't rely on IFTI. I have to mixin every template instantiation myself. Crazy. Ali
Sep 06 2016
prev sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/04/2016 05:37 AM, David Nadlinger wrote:
 On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu wrote:
 Thanks for answering. Yes, we really need introspection of private
 members. One way or another we need to support that.
Do we, though? It's easy to state a general claim like this, but I find it hard to actually justify. — David
Let me flip the question: What harm can there be when I pass my type to a template and that template accesses the private members? Ali
Sep 06 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 17:57:08 UTC, Jonathan M Davis 
wrote:
 I do get the feeling that too much of what we do with features 
 like this is ad-hoc without really designing things up front, 
 and then we keep having to tweak stuff as we go along in ways 
 that aren't always particularly nice.
That's not true, there was DIP22, and it explicitly mentions how traits should work. The problem that was new and very hard to see, is that allowing traits to bypass visibility checks means we either can't replace access checks, must allow access of private members (w/ the cost tradeoffs mentioned in the announce thread), or come up with a different solution. While it's fairly straightforward to skip visibility checks during traits lookups, I couldn't come up with a useful use-case while writing tests for that implementation, and after some more thought came up with the allMembers change.
Sep 03 2016
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-09-03 17:50, Martin Nowak wrote:

 It will, e.g. having getMember bypass protection means vibe.d would now
 serialize private members
 https://github.com/rejectedsoftware/vibe.d/blob/c1180791de61d0f8c9bfb584c2551a5b64627e32/source/vibe/internal/meta/traits.d#L146.

 Until now getMember was just a DotIdExp (i.e. `T.indent`/`var.ident`),
 so it was used to test whether sth. can be accessed.
Without looking at the how the rest of the code works, it already checks if a member is public or not [1]. Or are you saying that "isPublicMember" would return a different value? [1] https://github.com/rejectedsoftware/vibe.d/blob/c1180791de61d0f8c9bfb584c2551a5b64627e32/source/vibe/internal/meta/traits.d#L159 -- /Jacob Carlborg
Sep 03 2016
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, August 30, 2016 15:24:12 Ali Çehreli via Digitalmars-d wrote:
 v2.071.2-b3 is bringing a change for this bug:

    https://issues.dlang.org/show_bug.cgi?id=15907

 I don't agree with the current solution:

    http://dlang.org/changelog/2.071.2.html#traits-members-visibility

 Modules should be able to use library templates without needing to mix
 them in first.
Agreed. Having to mix stuff in for introspection is downright ugly. It makes far more sense to provide everything and then check the attributes as discussed elsewhere in this thread. IMHO, having allMembers give different results depending on access level is just plain broken. I confess that I see no problem being able to examine symbols that are not accessible due to the access level. They just shouldn't be usable or be involved in overload sets. Looking at their declarations and attributes and whatnot should be fine. - Jonathan M Davis
Aug 31 2016
prev sibling next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 08/30/2016 03:24 PM, Ali Çehreli wrote:
 v2.071.2-b3 is bringing a change for this bug:

   https://issues.dlang.org/show_bug.cgi?id=15907

 I don't agree with the current solution:

   http://dlang.org/changelog/2.071.2.html#traits-members-visibility

 Modules should be able to use library templates without needing to mix
 them in first.

 Do you think the solution in the change log usable? I don't think so
 because silently skipping my private members is an unexpected behavior
 that will cause bugs.
Here is a regression caused by the above change: interface I { void foo(); package: void foo(int); } string how(alias Base, alias func)() { return ""; } import std.typecons; class C : AutoImplement!(I, how) { } void main() { auto c = new C(); c.foo(); c.foo(42); // COMPILATION ERROR: // deneme.o: In function `_Dmain': // deneme.d:(.text._Dmain+0x39): undefined reference to `_D6deneme1I3fooMFiZv' // collect2: error: ld returned 1 exit status } WAT? It's not reasonable for the user to somehow suspect that AutoImplement may be using a D feature (__traits(allMembers) in this case) which happens to not see foo(int). (Note that AutoImplement may see foo(int) today but not tomorrow, if it's moved out of this package.) The recommended solution of mixing in every template instance is not a viable solution because that would effectively remove IFTI from D. What a huge loss that would be. We can't afford that. So, to be safe, every D code out there that happens to pass a struct to a piece of library function would have to 1) Know that that library function happens to be a template and 2) mixin the particular instantiation of that template. That would be the only sane thing to do. Further, what happens if a function changes its implementation and becomes a template that happens to call __traits(allMembers)? How can the library notify its users to tell them to mixin an explicit instantiation? Ali
Aug 31 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 31 August 2016 at 21:12:54 UTC, Ali Çehreli wrote:
 On 08/30/2016 03:24 PM, Ali Çehreli wrote:
 v2.071.2-b3 is bringing a change for this bug:

   https://issues.dlang.org/show_bug.cgi?id=15907

 I don't agree with the current solution:

 
http://dlang.org/changelog/2.071.2.html#traits-members-visibility
 Modules should be able to use library templates without
needing to mix
 them in first.

 Do you think the solution in the change log usable? I don't
think so
 because silently skipping my private members is an unexpected
behavior
 that will cause bugs.
Here is a regression caused by the above change: interface I { void foo(); package: void foo(int); } string how(alias Base, alias func)() { return ""; } import std.typecons; class C : AutoImplement!(I, how) { }
What this derives a class in std.typecons that implements the interface, and yes such a derived class can't override/implement package protected methods. AutoImplement would also work better as mixin in order to instantiate the passed in how template in the original instantiation scope.
 void main() {
     auto c = new C();
     c.foo();
     c.foo(42);    // COMPILATION ERROR:
 // deneme.o: In function `_Dmain':
 // deneme.d:(.text._Dmain+0x39): undefined reference to 
 `_D6deneme1I3fooMFiZv'
The missing interface method implementation should've give a compile error, please file a bug.
 The recommended solution of mixing in every template instance 
 is not a viable solution because that would effectively remove 
 IFTI from D. What a huge loss that would be. We can't afford 
 that.
Exaggeration won't help us to find good solutions. Remember that private fields never were accessible, so only some edge cases will be affected. The need for mixin templates will remain rare.
Sep 03 2016
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/03/2016 09:37 AM, Martin Nowak wrote:
 On Wednesday, 31 August 2016 at 21:12:54 UTC, Ali Çehreli wrote:
 The recommended solution of mixing in every template instance is not a
 viable solution because that would effectively remove IFTI from D.
 What a huge loss that would be. We can't afford that.
Exaggeration won't help us to find good solutions. Remember that private fields never were accessible, so only some edge cases will be affected. The need for mixin templates will remain rare.
I don't see how the user of a template library can decide whether to mixin or not without knowing the current implementation if of the library. For that reason, I must mixin. For example, the following program is broken because I don't know whether writeln uses allMembers or not: // BROKEN CODE: MyStruct s; writeln(s); Do you see the problem? I can't leave that code that way. I have to figure out mixin writeln's instantiation. As you see, mixins will not be rare. Every template use must be mixed in. Ali
Sep 07 2016
parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 7 September 2016 at 19:29:05 UTC, Ali Çehreli wrote:
 On 09/03/2016 09:37 AM, Martin Nowak wrote:
 On Wednesday, 31 August 2016 at 21:12:54 UTC, Ali Çehreli
wrote:
 The recommended solution of mixing in every template
instance is not a
 viable solution because that would effectively remove IFTI
from D.
 What a huge loss that would be. We can't afford that.
Exaggeration won't help us to find good solutions. Remember
that private
 fields never were accessible, so only some edge cases will be
affected.
 The need for mixin templates will remain rare.
I don't see how the user of a template library can decide whether to mixin or not without knowing the current implementation if of the library. For that reason, I must mixin. For example, the following program is broken because I don't know whether writeln uses allMembers or not: // BROKEN CODE: MyStruct s; writeln(s); Do you see the problem? I can't leave that code that way. I have to figure out mixin writeln's instantiation. As you see, mixins will not be rare. Every template use must be mixed in.
Well, I can only repeat what I stated several times before, private members were never accessible by name (via getMember or .name). So either writeln uses .tupleof (unnamed) or ignores private members. This is the pre 2.071 state. method | visible | accessible -------------------------------- .name | y | n getMember | y | n .tupleof | y | y This was the 2.071.1 state (for which the allMembers fix was a valid solution). method | visible | accessible -------------------------------- .name | n | n getMember | n | n .tupleof | y | y This is what we're now doing in 2.071.2 (see https://github.com/dlang/dmd/pull/6111), method | visible | accessible -------------------------------- .name | n | n getMember | y | n .tupleof | y | y and what we plan to do with 2.072 or 2.073. method | visible | accessible -------------------------------- .name | n | n getMember | y | y .tupleof | y | y In fact access checks will get removed soon.
Sep 12 2016
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 I can't wrap my head around the fact that a library template 
 called by my module cannot see my private members.
Well, it was never possible to access them either, didn't seem to cause much confusion. Also getSymbolsByUDA is built on a hack around that access check https://github.com/dlang/phobos/blob/cb09746cb11bcbe7b730f05d29792e6252669175/std/traits.d#L6979. Note that you can pass private symbols to templates, so `hasUDA!(S.invisible, UDA)` works fine. I don't see any other examples where we'll have much issues with that change other than w/ this weird getSymbolsByUDA API, which aliases to !(S.field, S.func, S.Nested). Most templates operating on wholes types already didn't have access to private members.
Sep 03 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 I don't agree with the current solution:
Well let's come up with a better solution then. Let's start by finding some proper use-cases that require introspection on private members w/o having access to them.
Sep 03 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 I don't agree with the current solution:
Well let's come up with a better solution then. Let's start by finding some proper use-cases that require introspection on private members w/o having access to them.
In my user library, the serialization is based on the Set and Get UDAs. Typically there would be a setter (a public method) and no getter (i.e the data is read directly either from a protected or private variable). The introspection is used to create a property descriptor. But because of the protection attributes I have to mix the introspection features (== a template) in each aggregate that declares properties and in each descendant class that declare new properties. Class Foo { mixin features f; // so that the private stuff are visible private: Get int _field; public Set void field(int value){} this() {f.analyze!Foo;} } class Bar: Foo { mixin features f; // so that the private stuff are visible private: Get int _otherfield; public Set void otherfield(int value){} this() {f.analyze!Bar;} } while I could do, with the omniscient traits: Class Foo { private: Get int _field; public Set void field(int value){} this(this T)() { features.analyze!T; } } Class Bar: Foo { private: Get int _otherfield; public Set void otherfield(int value){} } no need to re-mix the features for each derived class...
Sep 03 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 17:23:53 UTC, Basile B. wrote:
 On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak 
 wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 I don't agree with the current solution:
Well let's come up with a better solution then. Let's start by finding some proper use-cases that require introspection on private members w/o having access to them.
In my user library, the serialization is based on the Set and Get UDAs. Typically there would be a setter (a public method) and no getter (i.e the data is read directly either from a protected or private variable).
A public setter for private members is weird, but well. As a library template can't read private fields, you don't need the Get attribute.
 The introspection is used to create a property descriptor.
What does that mean? You're creating a property in the serialised data? Defining de-/serialize methods in the class/struct, e.g. with a mixin template would be the cleaner and more obvious approach IMO.
Sep 03 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Sunday, 4 September 2016 at 03:45:50 UTC, Martin Nowak wrote:
 On Saturday, 3 September 2016 at 17:23:53 UTC, Basile B. wrote:
 On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak 
 wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 I don't agree with the current solution:
Well let's come up with a better solution then. Let's start by finding some proper use-cases that require introspection on private members w/o having access to them.
In my user library, the serialization is based on the Set and Get UDAs. Typically there would be a setter (a public method) and no getter (i.e the data is read directly either from a protected or private variable).
A public setter for private members is weird, but well.
No in Object Pascal that's a common to have property Foo: string read fFoo write setFoo; By the way there's an error from my part, in my lib it's SetGet which gives access to private/protected field "directly".
 As a library template can't read private fields, you don't need 
 the  Get attribute.

 The introspection is used to create a property descriptor.
What does that mean? You're creating a property in the serialised data?
The introspection creates a special structure for each property annotated with Set, Get, or SetGet. This is a kind of interface for serialization/binding/object inspector (a bit like "published" in Object Pascal). https://gist.github.com/BBasile/39fb66f7a0189660182cc637ab8d698b/archive/3e0f7441cfba89b5959b78ee 63b9538d137a55a.zip (you can "dub a.d")
 Defining de-/serialize methods in the class/struct, e.g. with a 
 mixin template would be the cleaner and more obvious approach 
 IMO.
I see your strategy...each time someone will find an argument to make the traits omniscients you'll say, "no because it's cleaner to do like that" ?
Sep 04 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 4 September 2016 at 07:24:42 UTC, Basile B. wrote:
 The introspection creates a special structure for each property 
 annotated with  Set,  Get, or  SetGet. This is a kind of 
 interface for serialization/binding/object inspector (a bit 
 like "published" in Object Pascal).
But you can't access private fields outside of the current module, so you need to mixin sth. to set/get the value anyhow.
 Defining de-/serialize methods in the class/struct, e.g. with 
 a mixin template would be the cleaner and more obvious 
 approach IMO.
I see your strategy...each time someone will find an argument to make the traits omniscients you'll say, "no because it's cleaner to do like that" ?
No, I'm just trying to understand the requirement for introspection without access.
Sep 04 2016
parent Basile B. <b2.temp gmx.com> writes:
On Sunday, 4 September 2016 at 08:42:43 UTC, Martin Nowak wrote:
 On Sunday, 4 September 2016 at 07:24:42 UTC, Basile B. wrote:
 The introspection creates a special structure for each 
 property annotated with  Set,  Get, or  SetGet. This is a kind 
 of interface for serialization/binding/object inspector (a bit 
 like "published" in Object Pascal).
But you can't access private fields outside of the current module, so you need to mixin sth. to set/get the value anyhow.
But that's the point ! We are many willing to see this restriction removed for the traits because they are special., i.e they should allow to implement, using meta programming, features that less powerful languages do in intern. That's also why I propose to add a 3rd optional bool parameter to the allMembers trait: enum RespectProtection = false; enum IgnoreProtection = true; __traits(allMembers, T, RespectProtection) __traits(allMembers, T, IgnoreProtection) The 3rd optional parameter, when not specified, would be set a default value of false (equivalent to __traits(allMembers, T, RespectProtection), to prevent breakage. Because allMembers would act as a filter for getMember, getOverload...the protection check in getMember, getOverload... would be removed. This will be obviously more complex than that but that's the idea.
Sep 04 2016
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-09-04 05:45, Martin Nowak wrote:

 A public setter for private members is weird, but well.
The setter can do validations, manipulation and other kinds of calculations based on the given value. -- /Jacob Carlborg
Sep 04 2016
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 5:45 AM, Martin Nowak wrote:
 A public setter for private members is weird
How do you mean that? It's an absolute classic. Please elaborate, thanks. -- Andrei
Sep 04 2016
prev sibling next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: Usability of "allMembers and derivedMembers traits now only
 return visible symbols"
References: <nq512d$9po$1 digitalmars.com>
 <ghusqqkmtakmehzyxpsl forum.dlang.org>
In-Reply-To: <ghusqqkmtakmehzyxpsl forum.dlang.org>

--qarTW6Qv25f05gfq2sSivoM3CvOUUC4e2
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 09/03/2016 07:52 PM, Martin Nowak wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali =C3=87ehreli wrote:
 I don't agree with the current solution:
=20 Well let's come up with a better solution then. Let's start by finding some proper use-cases that require introspection=
 on private members w/o having access to them.
Obviously serialization libraries come to mind, especially important for binary serialization because private members still affect the layout of the struct. --qarTW6Qv25f05gfq2sSivoM3CvOUUC4e2--
Sep 03 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-03 20:56, Dicebot wrote:

 Obviously serialization libraries come to mind, especially important for
 binary serialization because private members still affect the layout of
 the struct.
That can usually be solved using .tupleof[i], but I agree. -- /Jacob Carlborg
Sep 03 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 19:31:12 UTC, Jacob Carlborg 
wrote:
 That can usually be solved using .tupleof[i], but I agree.
Well, .tupleof gives you a typed representation of the memory layout, to me it's something different than qualified access of fields, just a safer mean to access memory. And sure you can always memcpy data. Also introspection works on .tupleof, so it's not affected by the allMembers change and it's awkward to mix them b/c names and layout don't have a simple 1-to-1 relation.
Sep 03 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 5:56 AM, Martin Nowak wrote:
 On Saturday, 3 September 2016 at 19:31:12 UTC, Jacob Carlborg wrote:
 That can usually be solved using .tupleof[i], but I agree.
Well, .tupleof gives you a typed representation of the memory layout, to me it's something different than qualified access of fields, just a safer mean to access memory. And sure you can always memcpy data. Also introspection works on .tupleof, so it's not affected by the allMembers change and it's awkward to mix them b/c names and layout don't have a simple 1-to-1 relation.
Yah, .tupleof is great to have. I think that should be enough for most introspection needs. Only the field names are missing, can those be accessed somehow? -- Andrei
Sep 04 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-04 14:36, Andrei Alexandrescu wrote:

 Yah, .tupleof is great to have. I think that should be enough for most
 introspection needs. Only the field names are missing, can those be
 accessed somehow? -- Andrei
Yes: module foo; struct Foo { private int a; } module main; import foo; static assert(__traits(identifier, Foo.tupleof[0]) == "a"); -- /Jacob Carlborg
Sep 05 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/5/16 10:38 AM, Jacob Carlborg wrote:
 On 2016-09-04 14:36, Andrei Alexandrescu wrote:

 Yah, .tupleof is great to have. I think that should be enough for most
 introspection needs. Only the field names are missing, can those be
 accessed somehow? -- Andrei
Yes: module foo; struct Foo { private int a; } module main; import foo; static assert(__traits(identifier, Foo.tupleof[0]) == "a");
Thank you. This is good news. -- Andrei
Sep 05 2016
prev sibling parent reply Johan Engelen <j j.nl> writes:
On Sunday, 4 September 2016 at 12:36:43 UTC, Andrei Alexandrescu 
wrote:
 Yah, .tupleof is great to have. I think that should be enough 
 for most introspection needs. Only the field names are missing, 
 can those be accessed somehow? -- Andrei
Note that the spec tells us: "The .tupleof property returns an ExpressionTuple of all the fields in the class, __excluding__ the hidden fields and __the fields in the base class__." (emphasis mine) So then you have to do __traits(parent, ) recursively...?
Sep 07 2016
parent Basile B. <b2.temp gmx.com> writes:
On Wednesday, 7 September 2016 at 15:08:17 UTC, Johan Engelen 
wrote:
 Note that the spec tells us:
 "The .tupleof property returns an ExpressionTuple of all the 
 fields in the class, __excluding__ the hidden fields and __the 
 fields in the base class__." (emphasis mine)
 So then you have to do __traits(parent, ) recursively...?
or use std.traits.BaseClassesTuple.
Sep 07 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 18:56:33 UTC, Dicebot wrote:
 Obviously serialization libraries come to mind, especially 
 important for binary serialization because private members 
 still affect the layout of the struct.
Let me repeat that another time, private members have never been accessible by traits. That's the main reasoning this change was based upon.
Sep 03 2016
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require 
 introspection on private members w/o having access to them.
Still haven't heard really good use cases for the above, but the trade-offs for also allowing access to private members aren't that bad, thus it seems like the better path forward. See http://forum.dlang.org/post/ymkehalxcigswvltlfjj forum.dlang.org
Sep 04 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 11:09 AM, Martin Nowak wrote:
 On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require
 introspection on private members w/o having access to them.
Still haven't heard really good use cases for the above, but the trade-offs for also allowing access to private members aren't that bad, thus it seems like the better path forward. See http://forum.dlang.org/post/ymkehalxcigswvltlfjj forum.dlang.org
There are a few cases I can think of: * Serialization and deserialization, shallow and deep. These would need access to the object layout, and possibly field names too (for cross-checking and versioning purposes). * Binary saving, loading, and fixup (a subset of serialization/deserialization) - that thing when you copy raw memory to a file and then load it raw and fix pointers up. * Memory management and garbage collection: access to field types allows efficient generic tracing. * Database interfacing, automated binding, object-relational mapping, etc. It stands to reason that field names would be needed and fields would be private, yet the database loader does have access to them. Of course the more important applications are those I can't yet imagine. The way I see it introspection must have full power and unfettered access. So definitely it must be able to pass through regular protection. Andrei
Sep 04 2016
next sibling parent Martin Nowak <code dawg.eu> writes:
On Sunday, 4 September 2016 at 12:51:05 UTC, Andrei Alexandrescu 
wrote:
 See 
 http://forum.dlang.org/post/ymkehalxcigswvltlfjj forum.dlang.org
There are a few cases I can think of:
The classical ones, but again the question was for introspection without access (b/c named access through getMember was never allowed, only .tupleof). As the trade-offs for allowing access to private members don't seem that bad, we'll now try to go in the other direction, skipping visibility checks now and removing the access checks in a later release.
Sep 04 2016
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-09-04 14:51, Andrei Alexandrescu wrote:

 Of course the more important applications are those I can't yet imagine.
 The way I see it introspection must have full power and unfettered
 access. So definitely it must be able to pass through regular protection.
* Linking up GUI controls to instance variables in the code, like Xcode/Interface Builder -- /Jacob Carlborg
Sep 05 2016
prev sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/04/2016 05:51 AM, Andrei Alexandrescu wrote:

* Tracing how and when a member mutates.

Ali
Sep 06 2016
prev sibling parent earthfront <earthfront mailinator.com> writes:
On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak wrote:
 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 I don't agree with the current solution:
Well let's come up with a better solution then. Let's start by finding some proper use-cases that require introspection on private members w/o having access to them.
Use case that I think is pretty durn important: using Allocator.make and forwarding a call to the private constructor on a class. This does not work now: --- class A { int b; private this(int a){b=a;} } auto ptr = make!A(Mallocator.instance, 42); // Compile Error! --- Allocators are second class in this respect compared to GC. Mixins aren't a good solution to this.
Sep 14 2016