www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Fix #2529: explicit protection package #3651

reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/19/2014 7:01 AM, Dicebot wrote:

 Walter, now that release is out can you please state your opinion about
 https://github.com/D-Programming-Language/dmd/pull/3651 ? It is blocking Phobos
 module split and decoupling.


I keep thinking there's gotta be a way to do this without language changes.
Aug 19 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:

 On 8/19/2014 7:01 AM, Dicebot wrote:

 Walter, now that release is out can you please state your


 opinion about

 https://github.com/D-Programming-Language/dmd/pull/3651 ? It


 is blocking Phobos

 module split and decoupling.


 I keep thinking there's gotta be a way to do this without 
 language changes.


Any specific ideas? I can't imagine any clean solution - and 
proposed language extensions fits naturally into existing system 
without introducing any new concepts. It is also somewhat 
frequently asked about in NG.
Aug 19 2014
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote:

 On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:

 On 8/19/2014 7:01 AM, Dicebot wrote:

 Walter, now that release is out can you please state your


 opinion about

 https://github.com/D-Programming-Language/dmd/pull/3651 ? It


 is blocking Phobos

 module split and decoupling.


 I keep thinking there's gotta be a way to do this without 
 language changes.


 Any specific ideas? I can't imagine any clean solution - and 
 proposed language extensions fits naturally into existing 
 system without introducing any new concepts. It is also 
 somewhat frequently asked about in NG.


Yeah, I don't see how this could be done without a language 
change. Currently, modules in sub-packages are treated no 
differently from modules in completely different packages, so 
anything you did to give access to a module in a super-package to 
one in a sub-package would give access to any module.

- Jonathan M Davis
Aug 19 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote:

 On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:

 On 8/19/2014 7:01 AM, Dicebot wrote:

 Walter, now that release is out can you please state your


 opinion about

 https://github.com/D-Programming-Language/dmd/pull/3651 ? It


 is blocking Phobos

 module split and decoupling.


 I keep thinking there's gotta be a way to do this without 
 language changes.


 Any specific ideas? I can't imagine any clean solution - and 
 proposed language extensions fits naturally into existing 
 system without introducing any new concepts. It is also 
 somewhat frequently asked about in NG.


To put it differently - this issue is a very real blocker for 
some of planned Phobos changes (pretty much anything that is 
going to use package.d) and it has been hanging around for a long 
time. If you are going to propose alternative solution I'd be 
very glad to see it sooner than later as everyone else who has 
commented on that topic seems to be satisfied with the proposal.
Aug 19 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote:

 Any specific ideas? I can't imagine any clean solution - and 
 proposed language extensions fits naturally into existing 
 system without introducing any new concepts. It is also 
 somewhat frequently asked about in NG.


---
module std.internal.mod1;
package(std) void foo() {}

module std.mod2;
import std.internal.mod2;
void bar() { foo(); }
---

Isn't it equivalent to
---
module std.internal_mod1;
package void foo() {}

module std.mod2;
import std.internal_mod1;
void bar() { foo(); }
---
Aug 20 2014
parent reply ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Wed, 20 Aug 2014 07:25:43 +0000
Kagamin via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

but it requires to dump all 'subpackages' into one flat directory and
with ugly names.

proposed extension will not break any existing code, yet will allow much
nicer hierarchy.
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 07:47:36 UTC, ketmar via 
Digitalmars-d-announce wrote:

 but it requires to dump all 'subpackages' into one flat 
 directory and
 with ugly names.


Huh? If std.datetime and std.regex are subpackages of `std` 
package, it usually means, they should reside in `std` directory, 
no? And how explicit package protection can change that?
Aug 20 2014
next sibling parent reply ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Wed, 20 Aug 2014 09:26:36 +0000
Kagamin via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:


 Huh? If std.datetime and std.regex are subpackages of `std`=20
 package, it usually means, they should reside in `std` directory,=20
 no? And how explicit package protection can change that?


imagine mybiglib, mybiglib.internal, mybiglib.other, and each of them
not single module, but package itself. so we have module
mybiglib.other.thingy which wants to use functions from
module mybiglib.internal.wisdom, which is part of mybiglib.internal
package.
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 09:40:37 UTC, ketmar via 
Digitalmars-d-announce wrote:

 imagine mybiglib, mybiglib.internal, mybiglib.other, and each 
 of them
 not single module, but package itself. so we have module
 mybiglib.other.thingy which wants to use functions from
 module mybiglib.internal.wisdom, which is part of 
 mybiglib.internal
 package.


Do we need a hierarchy of internals, is the problem this big? Why 
mybiglib.wisdom is not good?
Aug 20 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:33:53 UTC, Kagamin wrote:

 On Wednesday, 20 August 2014 at 09:40:37 UTC, ketmar via 
 Digitalmars-d-announce wrote:

 imagine mybiglib, mybiglib.internal, mybiglib.other, and each 
 of them
 not single module, but package itself. so we have module
 mybiglib.other.thingy which wants to use functions from
 module mybiglib.internal.wisdom, which is part of 
 mybiglib.internal
 package.


 Do we need a hierarchy of internals, is the problem this big? 
 Why mybiglib.wisdom is not good?


As I see on the realistic example of datetime, which is BIG, we 
only need to split it into a flat set of files without an overly 
deep package hierarchy.
Aug 20 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote:

 As I see on the realistic example of datetime, which is BIG, we 
 only need to split it into a flat set of files without an 
 overly deep package hierarchy.


We _may_ split it into flat set files (solving only part of the 
problem) but it is desirable to have a deeper package hierarchy. 
Package hierarchy is not just an encapsulation tool, it is also a 
great way to simplify navigation and finding needed modules - 
something that Phobos is current terrible at exactly because of 
flat hierarchies.
Aug 20 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:40:34 UTC, Dicebot wrote:

 On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote:

 As I see on the realistic example of datetime, which is BIG, 
 we only need to split it into a flat set of files without an 
 overly deep package hierarchy.


 We _may_ split it into flat set files (solving only part of the 
 problem) but it is desirable to have a deeper package 
 hierarchy. Package hierarchy is not just an encapsulation tool, 
 it is also a great way to simplify navigation and finding 
 needed modules - something that Phobos is current terrible at 
 exactly because of flat hierarchies.


What is difficult to find? With flat structure you have all files 
right before your eyes. If you need std.datetime.systime module, 
you open std/datetime/systime.d file - that's the reason of 
needlessly restricting code structure with modules as if one size 
fits all.
Aug 23 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 23 August 2014 at 09:00:30 UTC, Kagamin wrote:

 What is difficult to find? With flat structure you have all 
 files right before your eyes. If you need std.datetime.systime 
 module, you open std/datetime/systime.d file - that's the 
 reason of needlessly restricting code structure with modules as 
 if one size fits all.


It is the same reasoning as with deep filesystem hierarchies and, 
well, any data hierarchies - once the element (module / file) 
count becomes bigger than ~dozen you only really notice things 
you know to look for. Contrary to that deeply nested categorized 
hierarchies are easy to casually to search through if you don't 
know exact module name - just iteratively pick whatever package 
fits the theme until you find what you want.


totally possible to find relevant modules without even looking in 
docs, just using auto-complete through suggested package names 
for import. It was really positive experience for a newbie I was. 
At the same time a lot of people have no idea how many cool 
things Phobos actually has.
Aug 23 2014
next sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Sun, 24 Aug 2014 02:39:39 +0000
Dicebot via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:


 It is the same reasoning as with deep filesystem hierarchies and,=20
 well, any data hierarchies


i already told that, but without any effects.
Aug 23 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Sunday, 24 August 2014 at 02:22:41 UTC, Dicebot wrote:

 Well difference is that "internal" substring in the fully 
 qualified name that is much more likely to tell user he is 
 better to not touch it. However, original Kagamin proposal of 
 embedding it into module names themselves does address that if 
 you are ok with resulting uglyness.


Both ways it's a convention, and I don't see, why such convention 
should exist in the first place, member protection has nothing to 
do with module name, which reflects module's functionality.

On Sunday, 24 August 2014 at 02:34:01 UTC, Dicebot wrote:

 It can be a philosophical matter, but in my experience 
 grouping by functionality is genuine, and access is an 
 afterthought, so grouping by access doesn't work in the long 
 term, as code naturally (spontaneously) groups by 
 functionality.


 That does contradict your statement that any stuff used in 
 parent packages must go to up the hierarchy which is exactly 
 grouping by access :)


Access there means protection, usage reflects functionality.


 I can probably give a more practical example I have just 
 recently encountered when re-designing one of our internal 
 library packages. It was a "serialization" package and 
 sub-packages defined different (de)serialization models, some 
 of those defining special wrapper types for resulting data that 
 enforce certain semantics specific to that serialization model 
 via the type system. It is not surprising that most internal 
 details of such wrappers were kept package protected as 
 exposing those to user could have easily violated all 
 assumptions (but was needed to implement (de)serialization 
 efficiently).

 It has become more complicated when "meta-serializers" have 
 been added to parent package - templated stuff that took any of 
 sub-package implementations and added some functionality (like 
 versioning support) on top. Being generic thing it reside in 
 higher level "serialization" package but to be implemented 
 efficiently it needs access to those internal wrapper fields. 
 Moving wrapper modules to parent package is not an option here 
 because those are closely related to specific serialization 
 model. Exposing stuff as public is not an option because anyone 
 not deeply familiar with package implementation can easily 
 break all type system guarantees that way. Moving 
 "meta-serializers" to sub-packages is quite a code duplication.

 Right now I keep that stuff public and say in docs "please 
 don't use this" which is hardly a good solution.


1. Making wrapper protected will preclude writing new serializer.
2. Using wrapper methods can be meaningless without serializer.
3. Serializer may just not expose wrapper, then user will have no 
way to access it.
4. .net has quite a lot of things like 
http://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.configuredtaskawaitabl
%28v=vs.110%29.aspx 
and nothing explodes even though .net programmers are believed to 
be really stupid and evil. It's a virtue of Stackoverflow Driven 
Development.

On Sunday, 24 August 2014 at 02:39:40 UTC, Dicebot wrote:

 On Saturday, 23 August 2014 at 09:00:30 UTC, Kagamin wrote:

 What is difficult to find? With flat structure you have all 
 files right before your eyes. If you need std.datetime.systime 
 module, you open std/datetime/systime.d file - that's the 
 reason of needlessly restricting code structure with modules 
 as if one size fits all.


 It is the same reasoning as with deep filesystem hierarchies 
 and, well, any data hierarchies - once the element (module / 
 file) count becomes bigger than ~dozen you only really notice 
 things you know to look for. Contrary to that deeply nested 
 categorized hierarchies are easy to casually to search through 
 if you don't know exact module name - just iteratively pick 
 whatever package fits the theme until you find what you want.


I'm afraid, hierarchies make things harder to find. If you don't 
know what is where, flat organization will present you with 
everything readily available right before your eyes. With deep 
hierarchy you're left with abstract or poorly chosen categories 
at every hierarchy level, so effectively you have to walk the 
entire hierarchy, which is much more tedious than scroll a flat 
list of modules viewing ten modules per scroll. Badly named list 
of modules (like what we have now in phobos) scales well up to 
100, well-named list is much more manageable: if you need xml, 
you already know it's near the end of the list - it's easy to 
find even among 1000 files - you don't ever need to scroll entire 
list. If it's not there, where do you go? There's no obvious 
answer. So even shallow hierarchy is more troublesome than a flat 
list of 1000 modules. I don't believe hierarchy will magically 
solve navigation problems just because it has folders.



 totally possible to find relevant modules without even looking 
 in docs, just using auto-complete through suggested package 
 names for import. It was really positive experience for a 
 newbie I was. At the same time a lot of people have no idea how 
 many cool things Phobos actually has.


And System.Xml is not System.Text.Xml, System.Net is not 
System.IO.Net - where is deep hierarchy? If you want System.Xml, 
you type `s.x` and it doesn't matter, how many namespaces are 
there, you're presented with System.Xml. If it's not there, where 
to find it, in System.Text, System.Formats, System.Parsers, 
System.Lang, System.Markup, System.Sgml?
Aug 24 2014
parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 24 August 2014 at 14:15:25 UTC, Kagamin wrote:

 On Sunday, 24 August 2014 at 02:22:41 UTC, Dicebot wrote:

 Well difference is that "internal" substring in the fully 
 qualified name that is much more likely to tell user he is 
 better to not touch it. However, original Kagamin proposal of 
 embedding it into module names themselves does address that if 
 you are ok with resulting uglyness.


 Both ways it's a convention, and I don't see, why such 
 convention should exist in the first place, member protection 
 has nothing to do with module name, which reflects module's 
 functionality.


Because we have no other means of enforcing this protection - 
this is why my PR event exists.


 On Sunday, 24 August 2014 at 02:34:01 UTC, Dicebot wrote:

 It can be a philosophical matter, but in my experience 
 grouping by functionality is genuine, and access is an 
 afterthought, so grouping by access doesn't work in the long 
 term, as code naturally (spontaneously) groups by 
 functionality.


 That does contradict your statement that any stuff used in 
 parent packages must go to up the hierarchy which is exactly 
 grouping by access :)


 Access there means protection, usage reflects functionality.


And how that makes your statement less contradictive?


 I can probably give a more practical example <...>
 code duplication.


 1. Making wrapper protected will preclude writing new 
 serializer.
 2. Using wrapper methods can be meaningless without serializer.
 3. Serializer may just not expose wrapper, then user will have 
 no way to access it.
 4. .net has quite a lot of things like 
 http://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.configuredtaskawaitabl
%28v=vs.110%29.aspx 
 and nothing explodes even though .net programmers are believed 
 to be really stupid and evil. It's a virtue of Stackoverflow 
 Driven Development.


I am afraid I don't really understand what your point here is - 
that my design is wrong and must be banned or that package 
concept is flawed and must be reworked? Neither is really an 
acceptable argument.

There is a simple and strict requirement - no one must be able to 
extend serializers but few developers that routinely maintain 
that package (and all sub-packages). Never in user code. This 
idea is not a subject to reconsideration - if language says such 
design is illegal I am going to call that language bullshit.


 I'm afraid, hierarchies make things harder to find. If you 
 don't know what is where, flat organization will present you 
 with everything readily available right before your eyes. With 
 deep hierarchy you're left with abstract or poorly chosen 
 categories at every hierarchy level, so effectively you have to 
 walk the entire hierarchy, which is much more tedious than 
 scroll a flat list of modules viewing ten modules per scroll. 
 Badly named list of modules (like what we have now in phobos) 
 scales well up to 100, well-named list is much more manageable: 
 if you need xml, you already know it's near the end of the list 
 - it's easy to find even among 1000 files - you don't ever need 
 to scroll entire list. If it's not there, where do you go? 
 There's no obvious answer. So even shallow hierarchy is more 
 troublesome than a flat list of 1000 modules. I don't believe 
 hierarchy will magically solve navigation problems just because 
 it has folders.


You are speaking about _finding_ things. I am speaking about 
_discovering_ things. See the difference? There is no way you can 
quickly tell "aha, so this data formats are supported out of the 
box" when looking at plain flat list of 1000 files. With solid 
hierarchy it becomes trivial.


 And System.Xml is not System.Text.Xml, System.Net is not 
 System.IO.Net - where is deep hierarchy? If you want 
 System.Xml, you type `s.x` and it doesn't matter, how many 
 namespaces are there, you're presented with System.Xml. If it's 
 not there, where to find it, in System.Text, System.Formats, 
 System.Parsers, System.Lang, System.Markup, System.Sgml?


You pick few that make most sense to you and ignore the rest. If 
routinely needed module is not exposed via obvious path it is an 
issue worth filing in the bug tracker. More rarely used / obscure 
modules can have more surprising paths. I don't see a problem 
here. Again, searching for a known package is never a problem 
both ways - you can just, well, run the search query. 
Discoverability of uknowns is the key.
Aug 25 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:40:34 UTC, Dicebot wrote:

 On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote:

 As I see on the realistic example of datetime, which is BIG, 
 we only need to split it into a flat set of files without an 
 overly deep package hierarchy.


 We _may_ split it into flat set files (solving only part of the 
 problem) but it is desirable to have a deeper package 
 hierarchy. Package hierarchy is not just an encapsulation tool, 
 it is also a great way to simplify navigation and finding 
 needed modules - something that Phobos is current terrible at 
 exactly because of flat hierarchies.


And with an excessively deep folder structure it will be much 
more tedious to walk the whole thing. It's deep nesting, which 
will make things hard to find.
Aug 23 2014
prev sibling next sibling parent reply Rory McGuire via Digitalmars-d-announce writes:
On Wed, Aug 20, 2014 at 4:33 PM, Kagamin via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 Do we need a hierarchy of internals, is the problem this big? Why
 mybiglib.wisdom is not good?


It gets really unwieldy when you want to put a whole bunch of things into
one project and give control of different packages to different teams. Yes
you could use multiple libraries in a lot of cases but if you're wanting it
in a single setup / package (like phobos), especially one that is internal
to another package you need sub-packages.

Examples that would use this are:
- kitchen sink libraries like phobos
- frameworks such as vibe.d
- libraries with lots of internal code  such as Ogre3d if it were made in D

Well that seems to be good reasons to me.
Aug 20 2014
parent "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:44:21 UTC, Rory McGuire via 
Digitalmars-d-announce wrote:

 On Wed, Aug 20, 2014 at 4:33 PM, Kagamin via 
 Digitalmars-d-announce <
 digitalmars-d-announce puremagic.com> wrote:


 Do we need a hierarchy of internals, is the problem this big? 
 Why
 mybiglib.wisdom is not good?


 It gets really unwieldy when you want to put a whole bunch of 
 things into
 one project and give control of different packages to different 
 teams. Yes
 you could use multiple libraries in a lot of cases but if 
 you're wanting it
 in a single setup / package (like phobos), especially one that 
 is internal
 to another package you need sub-packages.


Yes, public interfaces of packages can form a hierarchy (which 
still shouldn't be more complex than necessary), but internals?


 Examples that would use this are:
 - kitchen sink libraries like phobos


I proposed an example of datetime module, that I don't see a 
problem with internals there. Can you point to an exact line in 
source, which suffers from this?
Aug 23 2014
prev sibling parent reply ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Wed, 20 Aug 2014 14:33:52 +0000
Kagamin via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:


 Do we need a hierarchy of internals, is the problem this big? Why=20
 mybiglib.wisdom is not good?


ah, why we need such things as subdirectories at all? CP/M was fine
without concept of subdirectories!
Aug 20 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/20/14, 7:49 AM, ketmar via Digitalmars-d-announce wrote:

 On Wed, 20 Aug 2014 14:33:52 +0000
 Kagamin via Digitalmars-d-announce
 <digitalmars-d-announce puremagic.com> wrote:


 Do we need a hierarchy of internals, is the problem this big? Why
 mybiglib.wisdom is not good?


 ah, why we need such things as subdirectories at all? CP/M was fine
 without concept of subdirectories!


No need to demean the question. It is valid. -- Andrei
Aug 20 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 17:19:58 UTC, Andrei Alexandrescu 
wrote:

 On 8/20/14, 7:49 AM, ketmar via Digitalmars-d-announce wrote:

 On Wed, 20 Aug 2014 14:33:52 +0000
 Kagamin via Digitalmars-d-announce
 <digitalmars-d-announce puremagic.com> wrote:


 Do we need a hierarchy of internals, is the problem this big? 
 Why
 mybiglib.wisdom is not good?


 ah, why we need such things as subdirectories at all? CP/M was 
 fine
 without concept of subdirectories!


 No need to demean the question. It is valid. -- Andrei


Originally flat Phobos hierarchy was considered "good enough". 
Now we can see that such approach doesn't scale well - features 
are often missed because of non-intuitive module placement, 
compile times suffer because of many cross-module dependencies 
(flat hierarchy encourages big modules).

Is there any reason to think that same logic applied to 
sub-packages will scale any better as Phobos size grows? I doubt 
so. In fact I feel it is already beyond the size where it is 
convenient and only reason why even smaller deeply nested modules 
are not an option is exactly because it is too hard to keep both 
existing protection attribute relation and provide easy to 
navigate hierarchy at the same time.

The fact that some of people who have actually tried to use 
package.d support this language change is pretty good anecdotal 
evidence that there is a problem to be solved.
Aug 20 2014
prev sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Wed, 20 Aug 2014 10:19:59 -0700
Andrei Alexandrescu via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:


 No need to demean the question. It is valid. -- Andrei


sorry, i don't mean to insult anyone, just trying to make people see
analogies. hierarchies are everywhere, it's convient way to store alot
of things. modules itself is a great example of hierarchies -- we can
dump all source code in one file instead of using modules.

i.e. file system hierarchy is handy what there are many files. module
hierarchy is handy when there are many modules in library. the ability
to have fine-grained visibility control in module hierarchy is handy.

this was written numerous times in this thread, yet he continues to ask
the same question again and again as if he just don't bother to read
any answers. that's why i asked about subdirs.
Aug 20 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 09:26:37 UTC, Kagamin wrote:

 On Wednesday, 20 August 2014 at 07:47:36 UTC, ketmar via 
 Digitalmars-d-announce wrote:

 but it requires to dump all 'subpackages' into one flat 
 directory and
 with ugly names.


 Huh? If std.datetime and std.regex are subpackages of `std` 
 package, it usually means, they should reside in `std` 
 directory, no? And how explicit package protection can change 
 that?


Your proposal makes such module layout illegal:

std.package1.module1
std.package1.module2
std.package1.subpackage.module1
std.package1.subpackage.module2

(can't have symbols in subpackage that are also available to 
package1 but not whole std / public, must have 
`subpackage_module1` instead, `package_1_subpackage_module1` if 
it must be available for whole std)

It severely limits expressive power of package system for code 
organization and encourages flat hierarchies with minimal 
nesting. It also looks ugly - anyone seeing module name with 
package name mangled in it will feel that there is something 
wrong with the language if it needs to resort to such hacks.
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 13:05:54 UTC, Dicebot wrote:

 Your proposal makes such module layout illegal:

 std.package1.module1
 std.package1.module2
 std.package1.subpackage.module1
 std.package1.subpackage.module2

 (can't have symbols in subpackage that are also available to 
 package1 but not whole std / public, must have 
 `subpackage_module1` instead, `package_1_subpackage_module1` if 
 it must be available for whole std)


If some utility is internal to std.package1 an used in the entire 
std.package1, shouldn't it sit in std.package1? Why push it to 
subpackage?

std.package1.module1
std.package1.module2
std.package1.internal <- package1's internals go here
std.package1.subpackage.module1
std.package1.subpackage.module2
Aug 20 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 14:25:59 UTC, Kagamin wrote:

 If some utility is internal to std.package1 an used in the 
 entire std.package1, shouldn't it sit in std.package1? Why push 
 it to subpackage?

 std.package1.module1
 std.package1.module2
 std.package1.internal <- package1's internals go here
 std.package1.subpackage.module1
 std.package1.subpackage.module2


It may semantically belong to subpackage but still needs to be 
available to package1, something not uncommon in templated code 
(subpackage is generic implementation, package1 is specialization 
that still needs access to non-public functions).

Also you seem to imply that "internal" is always small tightly 
coupled module which is hardly true for any higher level 
functionality. So in practice it will be more like this:

std.package1.module1
std.package1.module2
std.package1.internal1
std.package1.internal2
...
std.package1.internal20

Not really encouraging..

What you propose is effectively prohibiting to use packages to 
organize your code and requiring to design your module hierarchy 
based on desired protection relations, not other way around. I 
think it is conceptually wrong approach and unnecessarily 
restrictive compared to overall D design principles (no single 
"true" way of doing things)
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 14:35:31 UTC, Dicebot wrote:

 It may semantically belong to subpackage but still needs to be 
 available to package1, something not uncommon in templated code 
 (subpackage is generic implementation, package1 is 
 specialization that still needs access to non-public functions).


If generic implementation is designed to be customized, it 
probably means, it should be widely available for customization, 
like xml document and base64 encoder. Also it's probably a 
logical error if base type is less visible than the derived type.


 What you propose is effectively prohibiting to use packages to 
 organize your code and requiring to design your module 
 hierarchy based on desired protection relations, not other way 
 around. I think it is conceptually wrong approach and 
 unnecessarily restrictive compared to overall D design 
 principles (no single "true" way of doing things)


It's ok for packages to exchange public interface, but internals? 
If a subpackage has internals, they are for its usage, and 
consumable functionality should be provided through public 
interface.
Aug 20 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 20 August 2014 at 15:13:06 UTC, Kagamin wrote:

 On Wednesday, 20 August 2014 at 14:35:31 UTC, Dicebot wrote:

 It may semantically belong to subpackage but still needs to be 
 available to package1, something not uncommon in templated 
 code (subpackage is generic implementation, package1 is 
 specialization that still needs access to non-public 
 functions).


 If generic implementation is designed to be customized, it 
 probably means, it should be widely available for 
 customization, like xml document and base64 encoder. Also it's 
 probably a logical error if base type is less visible than the 
 derived type.


This is just your design preference and hardly a good "one size 
fits them all" decision. I also don't speak about inheritance but 
about composition - customization may be implemented via unsafe 
field exposure and unsuitable for general public usage but useful 
for simplifying internal implementation maintenance. We should 
not force users into single good package structure based on 
certain design beliefs.


 What you propose is effectively prohibiting to use packages to 
 organize your code and requiring to design your module 
 hierarchy based on desired protection relations, not other way 
 around. I think it is conceptually wrong approach and 
 unnecessarily restrictive compared to overall D design 
 principles (no single "true" way of doing things)


 It's ok for packages to exchange public interface, but 
 internals? If a subpackage has internals, they are for its 
 usage, and consumable functionality should be provided through 
 public interface.


Same here. As library size grows you completely lose the 
distinction between "public" and "internal", certain parts of the 
library may become hidden from other parts and public but 
available for others. It is pretty much a necessity to keep up 
with maintenance when there is a large team working on it 
simultaneously. Right now people mostly rely on convention, I 
have seem quite some comments like "should have been private but 
needs to be accessed from module X, don't ever touch it" 
(public-but-undocumented functions of Phobos sometimes fall into 
the same category).

Telling people that they design applications in a wrong way and 
need to change their habits is a good approach to alienate them 
against the language.
Aug 20 2014
prev sibling parent reply "Chris Nicholson-Sauls" <ibisbasenji gmail.com> writes:
module foo.bar.one;
module foo.bar.internals; // package-protected utilities
module foo.bar.subpkg.two;
module foo.bar.subpkg.internals; // package-protected utilities


Current situation: module 'one' cannot access the 'bar.subpkg' 
utilities, and module 'two' cannot access the 'bar' utilities.  
This is needlessly limiting, forcing design choices that should 
not be dictated by the ability/inability to separate public and 
private API.  It also precludes many valid and good uses of 
nested package.d modules.

I really don't see any equally strong counter-argument.  But 
then, I've wanted this exact fix for literally years now.

It also is not limited to internal utility modules.  It can be 
useful for systems that select at compile time from one of a 
number of system-specific implementations of a given 
interface/api.  It can be useful for granting privileged access 
to certain api's and/or resources to specific module(s).  An 
example being: grant access to unsafe but versitile data 
manipulators solely to the subpackage containing the loaders.

Currently, there is no absolutely enforceable way of doing such 
things; so one ends up falling back on convention... and 
conventions, sadly, never hold in the real world for long.
Aug 20 2014
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 August 2014 at 20:39:06 UTC, Chris 
Nicholson-Sauls wrote:

 module foo.bar.one;
 module foo.bar.internals; // package-protected utilities
 module foo.bar.subpkg.two;
 module foo.bar.subpkg.internals; // package-protected utilities


 and module 'two' cannot access the 'bar' utilities.


Really? Looks like a bug. You'd better check if it's intended to 
work that way.


 This is needlessly limiting, forcing design choices that should 
 not be dictated by the ability/inability to separate public and 
 private API.  It also precludes many valid and good uses of 
 nested package.d modules.


If some thing is used in entire foo.bar package, then it's 
foo.bar's internal, not foo.bar.subpkg's internal. I think, it's 
natural when the wider the thing is used, the higher in hierarchy 
it sits.


 I really don't see any equally strong counter-argument.  But 
 then, I've wanted this exact fix for literally years now.


I don't really see, what it blocks.


 It also is not limited to internal utility modules.  It can be 
 useful for systems that select at compile time from one of a 
 number of system-specific implementations of a given 
 interface/api.  It can be useful for granting privileged access 
 to certain api's and/or resources to specific module(s).  An 
 example being: grant access to unsafe but versitile data 
 manipulators solely to the subpackage containing the loaders.


If those manipulators are used solely in loaders subpackage, they 
should belong to that package. Though I don't really see a 
problem here: there was an implementation of xml document, which 
allowed unsafe (but fast) specialization, which was believed to 
be useful for public usage. It fits well into D philosophy that 
safe things should be available by default and unsafe/fast things 
should be possible.
Aug 23 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-23 11:12, Kagamin wrote:


 If some thing is used in entire foo.bar package, then it's foo.bar's
 internal, not foo.bar.subpkg's internal. I think, it's natural when the
 wider the thing is used, the higher in hierarchy it sits.


A symbol declared "package" can only be accessed within its own package. 
It cannot be accessed from sub packages or super packages.

-- 
/Jacob Carlborg
Aug 23 2014
parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 23 August 2014 at 09:24:04 UTC, Jacob Carlborg wrote:

 A symbol declared "package" can only be accessed within its own 
 package. It cannot be accessed from sub packages or super 
 packages.


I didn't know about that. Should it really work that way?
Aug 23 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-23 11:35, Kagamin wrote:


 I didn't know about that. Should it really work that way?


That's what we're trying to fix with this change.

-- 
/Jacob Carlborg
Aug 23 2014
parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 23 August 2014 at 11:04:35 UTC, Jacob Carlborg wrote:

 That's what we're trying to fix with this change.


The proposal is to make internals visible to super packages, I 
only wonder whether internals visible to some package should be 
hidden from its subpackages.
Aug 23 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 23 August 2014 at 11:12:34 UTC, Kagamin wrote:

 On Saturday, 23 August 2014 at 11:04:35 UTC, Jacob Carlborg 
 wrote:

 That's what we're trying to fix with this change.


 The proposal is to make internals visible to super packages, I 
 only wonder whether internals visible to some package should be 
 hidden from its subpackages.


It addresses both - explicitly supplied package identifiers imply 
access to all subpackages. Current implementation is very 
striaghtforward and naive in that regard.
Aug 23 2014
prev sibling next sibling parent reply "Dmitry Olshansky" <dmitry.olsh gmail.com> writes:
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:

 On 8/19/2014 7:01 AM, Dicebot wrote:

 Walter, now that release is out can you please state your


 opinion about

 https://github.com/D-Programming-Language/dmd/pull/3651 ? It


 is blocking Phobos

 module split and decoupling.


 I keep thinking there's gotta be a way to do this without 
 language changes.


For what it's worth I don't care how things turn out w.r.t. 
syntax, language extension or convention but the problem must be 
solved.


The way I see the problem:

There must be nice and clean way to have internal stuff that is 
invisible for user. Preferably this stuff must not be monolithic 
and have form of package.

Example structure:
mod/package.d
mod/internal/bar.d
mod/...

I actually face it in Phobos right now, my blocked pull:
https://github.com/D-Programming-Language/phobos/pull/2412

Options:

1. Use current package attribute.
Disadvantages:
a) Must retain internals in the same package, user may still 
import them no problem at all. At least it does not do much harm 
as nothing is accessible.
b) Can't use std.mod.internal.bar 'package' stuff in std.mod 
(upper one). That makes it next to unusable. Basically it means 
we have to push internals up, instead of pushing them down.

2. Use separate public internal package.
Main disadvantage:
a) Now user may import and actually USE internal stuff.
b) Not only that but also internal methods of equally internal 
structs and classes leak out. In other words: a struct defined in 
internal module may never be exposed without some kind of 
wrapping. Ugly and unnecessary complication.


I'd rather have a language feature, but whatever you guys invent 
as 3rd option that doesn't suck will have my vote.


---
Dmitry Olshansky
Aug 22 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
All I want is that whatever decision Walter makes to happen 
sooner than in few years from now.
Aug 22 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Dicebot:


 All I want is that whatever decision Walter makes to happen 
 sooner than in few years from now.


There are other pending patches, like the support for the nice 
[$] syntax by Kenji.

"I keep thinking there's gotta be a way to do this without 
language changes." or "I keep thinking there must be a better way 
than [$]." are the kind of standard answers Walter gives when he 
doesn't have time to think about some design (because he has time 
he usually needs only hours or very few days to implement 
features, like when he made UDAs or C++ namespeces).

So I guess Walter is currently busy with something (like 
finishing the 2.066 release and its regressions, or developing 
Warp for Facebook, or something else), so you have to wait some 
time for him to come back to the language development side, to 
receive a good answer and a good solution.

Bye,
bearophile
Aug 22 2014
next sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Fri, 22 Aug 2014 20:48:22 +0000
bearophile via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

or (smart plan!) just apply the necessary PRs, build dmd from sources
and start using the features. if authors of popular libraries will do
this, there will be no choice and maintainers will be forced to apply
this patches. this is called 'driven by the community'. ;-)

ok-ok-ok, you got me, this was a silly joke! or maybe not?..
Aug 22 2014
prev sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Sat, 23 Aug 2014 04:50:36 +0300
ketmar via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

to be honest, i myself is a happy user of Kenji's $ patch and
package(...) patch. yes, it costs some efforts to keep 'em up-to-date,
but hey, i have no life anyway.
Aug 22 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Friday, 22 August 2014 at 15:06:14 UTC, Dmitry Olshansky wrote:

 Example structure:
 mod/package.d
 mod/internal/bar.d
 mod/...

 I actually face it in Phobos right now, my blocked pull:
 https://github.com/D-Programming-Language/phobos/pull/2412


I think, it should have structure

std.regex.package;
std.regex.backtracking;
std.regex.ir;

What is the reason to have regex stuff in a remote directory? 
That would make things hard to find and navigate.
Aug 23 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
23-Aug-2014 12:57, Kagamin пишет:

 On Friday, 22 August 2014 at 15:06:14 UTC, Dmitry Olshansky wrote:

 Example structure:
 mod/package.d
 mod/internal/bar.d
 mod/...

 I actually face it in Phobos right now, my blocked pull:
 https://github.com/D-Programming-Language/phobos/pull/2412


 I think, it should have structure

 std.regex.package;
 std.regex.backtracking;
 std.regex.ir;

 What is the reason to have regex stuff in a remote directory? That would
 make things hard to find and navigate.


Yes, that was my initial pull. The stuff inside was marked as `package`, 
as only package.d currently has the same public API.

Then Dicebot suggested that since all modules are for the moment 
internal they have to be moved somewhere until public API is fleshed out 
(as it may change module name etc.).

2 options that both do NOT currently work:
std.regex.internal
std.internal.regex

-- 
Dmitry Olshansky
Aug 23 2014
parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 23 August 2014 at 09:14:13 UTC, Dmitry Olshansky 
wrote:

 Yes, that was my initial pull. The stuff inside was marked as 
 `package`, as only package.d currently has the same public API.

 Then Dicebot suggested that since all modules are for the 
 moment internal they have to be moved somewhere until public 
 API is fleshed out (as it may change module name etc.).

 2 options that both do NOT currently work:
 std.regex.internal
 std.internal.regex


If the user relies that a symbol is found in 
std.internal.regex.backtracking, moving it will break the code 
too. So what's the difference?
Aug 23 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
23-Aug-2014 13:33, Kagamin пишет:

 On Saturday, 23 August 2014 at 09:14:13 UTC, Dmitry Olshansky wrote:

 Yes, that was my initial pull. The stuff inside was marked as
 `package`, as only package.d currently has the same public API.

 Then Dicebot suggested that since all modules are for the moment
 internal they have to be moved somewhere until public API is fleshed
 out (as it may change module name etc.).

 2 options that both do NOT currently work:
 std.regex.internal
 std.internal.regex


 If the user relies that a symbol is found in
 std.internal.regex.backtracking, moving it will break the code too. So
 what's the difference?


I don't see the difference. In any case user should see public aliases 
and never know where they lead to. Dicebot - any objections?

However since my pull hang there for months, I decide to appease 
puristic requests and move things to internals, only to find out that it 
doesn't work.

For what its worth I'd _prefer_ std.regex.internal as in my opinion 
internals got to be well contained. But as discussed, `package` doesn't 
work this way.


-- 
Dmitry Olshansky
Aug 23 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky 
wrote:

 For what its worth I'd _prefer_ std.regex.internal as in my 
 opinion internals got to be well contained. But as discussed, 
 `package` doesn't work this way.


It can be a philosophical matter, but in my experience grouping 
by functionality is genuine, and access is an afterthought, so 
grouping by access doesn't work in the long term, as code 
naturally (spontaneously) groups by functionality. The reason is 
when you maintain code, you traverse it according to 
functionality, because you work with implementation, and 
protection is interface, so protection is ignored. If you add a 
public functionality to internal module (ranges often end up like 
that), you can either just qualify it with public access, as it's 
done in normal modules, or refactor and move it to public module, 
so you end up with logically grouped functionality separated 
between two or more modules, which try to enforce grouping by 
access, while in normal modules you have logically grouped 
functionality in the same module even if they have different 
access. Also you can end up with public functionality in internal 
module, if the member proved to be stable, robust and use case 
for it arises.
Aug 23 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 23 August 2014 at 11:08:31 UTC, Kagamin wrote:

 On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky 
 wrote:

 For what its worth I'd _prefer_ std.regex.internal as in my 
 opinion internals got to be well contained. But as discussed, 
 `package` doesn't work this way.


 It can be a philosophical matter, but in my experience grouping 
 by functionality is genuine, and access is an afterthought, so 
 grouping by access doesn't work in the long term, as code 
 naturally (spontaneously) groups by functionality.


That does contradict your statement that any stuff used in parent 
packages must go to up the hierarchy which is exactly grouping by 
access :)

I can probably give a more practical example I have just recently 
encountered when re-designing one of our internal library 
packages. It was a "serialization" package and sub-packages 
defined different (de)serialization models, some of those 
defining special wrapper types for resulting data that enforce 
certain semantics specific to that serialization model via the 
type system. It is not surprising that most internal details of 
such wrappers were kept package protected as exposing those to 
user could have easily violated all assumptions (but was needed 
to implement (de)serialization efficiently).

It has become more complicated when "meta-serializers" have been 
added to parent package - templated stuff that took any of 
sub-package implementations and added some functionality (like 
versioning support) on top. Being generic thing it reside in 
higher level "serialization" package but to be implemented 
efficiently it needs access to those internal wrapper fields. 
Moving wrapper modules to parent package is not an option here 
because those are closely related to specific serialization 
model. Exposing stuff as public is not an option because anyone 
not deeply familiar with package implementation can easily break 
all type system guarantees that way. Moving "meta-serializers" to 
sub-packages is quite a code duplication.

Right now I keep that stuff public and say in docs "please don't 
use this" which is hardly a good solution.
Aug 23 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky 
wrote:

 If the user relies that a symbol is found in
 std.internal.regex.backtracking, moving it will break the code 
 too. So
 what's the difference?


 I don't see the difference. In any case user should see public 
 aliases and never know where they lead to. Dicebot - any 
 objections?


Well difference is that "internal" substring in the fully 
qualified name that is much more likely to tell user he is better 
to not touch it. However, original Kagamin proposal of embedding 
it into module names themselves does address that if you are ok 
with resulting uglyness.
Aug 23 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:

 On 8/19/2014 7:01 AM, Dicebot wrote:

 Walter, now that release is out can you please state your


 opinion about

 https://github.com/D-Programming-Language/dmd/pull/3651 ? It


 is blocking Phobos

 module split and decoupling.


 I keep thinking there's gotta be a way to do this without 
 language changes.


In the meanwhile I have updated
https://github.com/D-Programming-Language/dmd/pull/3651 to
address latest Walter comments and be based on recent master
Aug 25 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
..and it has just been merged! ^_^

Thanks Walter!
Aug 26 2014
next sibling parent ketmar via Digitalmars-d-announce <digitalmars-d-announce puremagic.com> writes:
On Tue, 26 Aug 2014 16:46:19 +0000
Dicebot via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:


 ..and it has just been merged! ^_^


WOW! that's great!


 Thanks Walter!


second that!
Aug 26 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/26/14, 9:46 AM, Dicebot wrote:

 ...and it has just been merged! ^_^

 Thanks Walter!


Congratulations for a job well done. -- Andrei
Aug 26 2014
parent "Andrew Godfrey" <X y.com> writes:
On 8/26/14, 9:46 AM, Dicebot wrote:

 ...and it has just been merged! ^_^

 Thanks Walter!


Nice! This felt like a glaring hole,
and you've filled it!
Aug 28 2014