www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Time to destroy Walter: breaking modules into packages

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://github.com/D-Programming-Language/dmd/pull/2139

Andrei
Jun 05 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:


 https://github.com/D-Programming-Language/dmd/pull/2139


Probably you already know the following things because they see, 
similar solutions.

Python uses "__init__.py" instead of "package.d", it can also be 
empty:
http://docs.python.org/2/tutorial/modules.html

__init__.py can also execute initialization code for the package, 
like static this() in D.

Inside __init__.py you can also define a "__all__" variable where 
you list all the modules imported when "from package import *" is 
used.


If __all__ is not defined, the statement from sound.effects 
import * does not import all submodules from the package 
sound.effects into the current namespace; it only ensures that 
the package sound.effects has been imported (possibly running 
any initialization code in __init__.py) and then imports 
whatever names are defined in the package. This includes any 
names defined (and submodules explicitly loaded) by __init__.py. 
It also includes any submodules of the package that were 
explicitly loaded by previous import statements.<


Bye,
bearophile
Jun 05 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 3:31 PM, Andrei Alexandrescu wrote:

 https://github.com/D-Programming-Language/dmd/pull/2139



Well, I am destroyed. I need to do some more engineering. The problem is that 
packages and modules have become conflated with this change.
Jun 05 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2013 5:13 PM, Walter Bright wrote:

 On 6/5/2013 3:31 PM, Andrei Alexandrescu wrote:

 https://github.com/D-Programming-Language/dmd/pull/2139



 Well, I am destroyed. I need to do some more engineering. The problem is that
 packages and modules have become conflated with this change.


Fixed and reopened.
Jun 05 2013
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 05 Jun 2013 18:31:22 -0400
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 https://github.com/D-Programming-Language/dmd/pull/2139
 
 Andrei


To those who don't want to see a < 1000 lines module: Please
consider that as soon as we'd merge multiple compression
algorithms into one file for the sake of making the module
"large enough", it will turn back on us when we decide to
implement any larger number of those:
http://en.wikipedia.org/wiki/Category:Lossless_compression_algorithms

I see one down side though: Some algorithms are alterations of
others and could reuse code. For the sake of speaking module
names I'd still split the compression algorithms into multiple
modules and probably let one import the other.

-- 
Marco
Jun 05 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 04:10:26 Marco Leise wrote:

 Am Wed, 05 Jun 2013 18:31:22 -0400
 
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 https://github.com/D-Programming-Language/dmd/pull/2139
 
 Andrei


 
 To those who don't want to see a < 1000 lines module: Please
 consider that as soon as we'd merge multiple compression
 algorithms into one file for the sake of making the module
 "large enough", it will turn back on us when we decide to
 implement any larger number of those:
 http://en.wikipedia.org/wiki/Category:Lossless_compression_algorithms
 
 I see one down side though: Some algorithms are alterations of
 others and could reuse code. For the sake of speaking module
 names I'd still split the compression algorithms into multiple
 modules and probably let one import the other.


If each of the compression algorithms is in its own module which is in the 
same package as other compression modules, then they can use package access to 
share stuff that would be specific to them but not belong in Phobos in general. 
But I believe that package level access only works on the same level, so you 
couldn't have separate modules for compressing and decompressing as was being 
suggested. It would need to be more like

std.compress.zlib;
std.compress.lzw;

At that point, it would be trivial to have a common module of some kind with 
shared functionality which has package access level.

- Jonathan M Davis
Jun 05 2013
next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Thursday, 6 June 2013 at 02:36:12 UTC, Jonathan M Davis wrote:


 But I believe that package level access only works on the same 
 level, so you couldn't have separate modules for compressing 
 and decompressing as was being suggested. It would need to be 
 more like

 std.compress.zlib;
 std.compress.lzw;

 At that point, it would be trivial to have a common module of 
 some kind with
 shared functionality which has package access level.

 - Jonathan M Davis


+1

That seems to be the right level of granularity.
Jun 05 2013
prev sibling next sibling parent reply "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 6 June 2013 at 02:36:12 UTC, Jonathan M Davis wrote:


 But I believe that package level access only works on the same 
 level, so you
 couldn't have separate modules for compressing and 
 decompressing as was being
 suggested.


'package' should be fixed so that 'package' declarations are 
accessible within nested packages.
Jun 05 2013
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 08:09:38 Max Samukha wrote:

 On Thursday, 6 June 2013 at 02:36:12 UTC, Jonathan M Davis wrote:

 But I believe that package level access only works on the same
 level, so you
 couldn't have separate modules for compressing and
 decompressing as was being
 suggested.


 
 'package' should be fixed so that 'package' declarations are
 accessible within nested packages.


Well, it _is_ debatable as to whether that's desirable or not. With the 
current behavior, you can have a package which shares stuff within itself but 
not with its sub-packages, but there's no way to share with the sub-packages 
without making the symbols public; whereas if sub-packages have access to 
their parent packages' package functions, then packages _can_ share with their 
sub-packages, but they can't restrict anything to just the package. Both ways 
have their pros and cons. I don't know which is ultimately better.

- Jonathan M Davis
Jun 05 2013
parent reply "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 6 June 2013 at 06:26:17 UTC, Jonathan M Davis wrote:

 On Thursday, June 06, 2013 08:09:38 Max Samukha wrote:

 On Thursday, 6 June 2013 at 02:36:12 UTC, Jonathan M Davis 
 wrote:

 But I believe that package level access only works on the 
 same
 level, so you
 couldn't have separate modules for compressing and
 decompressing as was being
 suggested.


 
 'package' should be fixed so that 'package' declarations are
 accessible within nested packages.


 Well, it _is_ debatable as to whether that's desirable or not. 
 With the
 current behavior, you can have a package which shares stuff 
 within itself but
 not with its sub-packages, but there's no way to share with the 
 sub-packages
 without making the symbols public; whereas if sub-packages have 
 access to
 their parent packages' package functions, then packages _can_ 
 share with their
 sub-packages, but they can't restrict anything to just the 
 package. Both ways
 have their pros and cons. I don't know which is ultimately 
 better.


There is not much to debate. Get your hands dirty with a real 
project having more than two levels of packages and you'll see 
that the first is ultimately better.

While limiting 'package' to one level may be sometimes desirable, 
forcing package-private members to be world-public is highly 
undesirable. As a package author I can tolerate loose access 
policies within a package tree that *I* control but I definitely 
don't want to expose to the user of my package what he shouldn't 
have access to.
Jun 06 2013
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 6/6/13, Max Samukha <maxsamukha gmail.com> wrote:

 There is not much to debate. Get your hands dirty with a real
 project having more than two levels of packages and you'll see
 that the first is ultimately better.


I've argued the same thing before. And as a cream on top I'd like
allowing package on virtual methods. That way I can have internal
virtual methods which can be extended in other modules or modules in
subpackages (if the two features are supported). This allows me to
specialize behavior in subclasses, but at the same time disallows the
user from calling such methods (a final protected override comes
close, except it will still be callable in user-code).

Currently the way I work around this is to prepend an underscore to a
public virtual and not document the method. It's very much a
convention thing.

I could instead use some kind of template mixin.. but those are so
full of bugs that I eagerly avoid them.
Jun 06 2013
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 13:47:29 Andrej Mitrovic wrote:

 I've argued the same thing before. And as a cream on top I'd like
 allowing package on virtual methods. That way I can have internal
 virtual methods which can be extended in other modules or modules in
 subpackages (if the two features are supported). This allows me to
 specialize behavior in subclasses, but at the same time disallows the
 user from calling such methods (a final protected override comes
 close, except it will still be callable in user-code).


Well, since Walter seems to have been convinced in the "Slow performance 
compared to C++, ideas?" thread to make it so that non-virtual is the default, 
the presumably it could be changed so that package could be virtual when it's 
marked as virtual.

- Jonathan M Davis
Jun 06 2013
prev sibling parent reply "TommiT" <tommitissari hotmail.com> writes:
On Thursday, 6 June 2013 at 06:09:41 UTC, Max Samukha wrote:

 'package' should be fixed so that 'package' declarations are 
 accessible within nested packages.


I don't think the 'package' access specifier should give 
privileges to *all* nested packages. Rather, I think we just need 
to invent a new word, and then specifying what 'package' access 
specifier does becomes easy. My suggestion for this new word is 
"module-pack"...

Module-pack:
"A folder which has a file called package.d inside it. Can be 
imported as if it was a module. etc."

...then, the definition of 'package' access modifier would be:

"The label package can be specified at class level, outside all 
classes (module-level), or inside a struct. In all contexts, 
package introduces protection that allows access to the symbol to 
all modules and module-packs which are either within the same 
directory as the current module or within the same directory as 
the module-pack which holds the current module within. 
Non-module-pack subdirectories and the parent directory of the 
current module’s directory have no special privileges."

Basically we'd pretend that module-packs are like modules.
Jun 18 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, June 19, 2013 06:49:49 TommiT wrote:

 On Thursday, 6 June 2013 at 06:09:41 UTC, Max Samukha wrote:

 'package' should be fixed so that 'package' declarations are
 accessible within nested packages.


=20
 I don't think the 'package' access specifier should give
 privileges to *all* nested packages. Rather, I think we just need
 to invent a new word, and then specifying what 'package' access
 specifier does becomes easy. My suggestion for this new word is
 "module-pack"...
=20
 Module-pack:
 "A folder which has a file called package.d inside it. Can be
 imported as if it was a module. etc."
=20
 ...then, the definition of 'package' access modifier would be:
=20
 "The label package can be specified at class level, outside all
 classes (module-level), or inside a struct. In all contexts,
 package introduces protection that allows access to the symbol to
 all modules and module-packs which are either within the same
 directory as the current module or within the same directory as
 the module-pack which holds the current module within.
 Non-module-pack subdirectories and the parent directory of the
 current module=E2=80=99s directory have no special privileges."
=20
 Basically we'd pretend that module-packs are like modules.


I _really_ don't like the idea of having the presence of package.d affe=
ct the=20
package modifier, and I don't think that anything which controls how ma=
ny=20
levels deep the package modifier gives the package access to is worth t=
he extra=20
complication. If it's worth letting nested packages have access to thei=
r=20
parent packages package-level stuff, then let's just do that for everyt=
hing.=20
Trying to make it configurable really doesn't buy us much and adds yet =
more=20
complication to an already complicated language.

- Jonathan M Davis
Jun 18 2013
parent reply "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 05:35:17 UTC, Jonathan M Davis 
wrote:

 I _really_ don't like the idea of having the presence of 
 package.d affect the package modifier, and I don't think
 that anything which controls how many levels deep the
 package modifier gives the package access to is worth the extra 
 complication.


I don't like the extra complication either, but I think it's the 
lesser of two evils. Worse is a situation where you would want to 
break multiple modules, but you can't do it easily because of the 
way you've structured your code up to that point.

On Wednesday, 19 June 2013 at 05:35:17 UTC, Jonathan M Davis 
wrote:

 If it's worth letting nested packages have access to their 
 parent packages package-level stuff, then
 let's just do that for everything.


But this is not enough. Let's say you have modules A.d and B.d 
under the same package. Now, if A.d uses something that is marked 
'package' inside B.d, you can't break both A.d and B.d into these 
new module-package thingies without re-structuring your code.
Jun 18 2013
next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 06:25:02 UTC, TommiT wrote:

 Now, if A.d uses something that is marked 'package' inside B.d, 
 you can't break both A.d and B.d into these new module-package 
 thingies without re-structuring your code.


And it may not be even possible to re-structure the code, like 
when the thing marked 'package' is a member function for example.
Jun 18 2013
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, June 19, 2013 08:25:00 TommiT wrote:

 If it's worth letting nested packages have access to their
 parent packages package-level stuff, then
 let's just do that for everything.


 
 But this is not enough. Let's say you have modules A.d and B.d
 under the same package. Now, if A.d uses something that is marked
 'package' inside B.d, you can't break both A.d and B.d into these
 new module-package thingies without re-structuring your code.


Then just put whatever is supposed to be shared between them in the package 
that they were both in originally. package stuff is not part of the public API, 
so when you're refactoring package code, you're refactoring code that you own 
entirely, so it shouldn't be an issue if you have to rearrange it a bit, 
especially since it's all localized to a single package by definition. It's 
only when something is part of a public API that refactoring becomes a 
problem.

- Jonathan M Davis
Jun 18 2013
parent reply "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 06:47:05 UTC, Jonathan M Davis 
wrote:

 Then just put whatever is supposed to be shared between them in 
 the package
 that they were both in originally. package stuff is not part of 
 the public API,
 so when you're refactoring package code, you're refactoring 
 code that you own
 entirely, so it shouldn't be an issue if you have to rearrange 
 it a bit,
 especially since it's all localized to a single package by 
 definition. It's
 only when something is part of a public API that refactoring 
 becomes a
 problem.

 - Jonathan M Davis


I can't put a member function into a different file if it's 
supposed to have access to private data of its enclosing 
class/struct.
Jun 18 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, June 19, 2013 08:51:38 TommiT wrote:

 On Wednesday, 19 June 2013 at 06:47:05 UTC, Jonathan M Davis
 
 wrote:

 Then just put whatever is supposed to be shared between them in
 the package
 that they were both in originally. package stuff is not part of
 the public API,
 so when you're refactoring package code, you're refactoring
 code that you own
 entirely, so it shouldn't be an issue if you have to rearrange
 it a bit,
 especially since it's all localized to a single package by
 definition. It's
 only when something is part of a public API that refactoring
 becomes a
 problem.
 
 - Jonathan M Davis


 
 I can't put a member function into a different file if it's
 supposed to have access to private data of its enclosing
 class/struct.


Then you can put the class in a different file. But even if you can't, I don't 
think that it's worth it to complicate the package attribute any further. Even 
if we lose something in the process, we really don't lose much, and since 
currently, package only applies to modules directly in the same package and 
not nested packages, you'd still be gaining over what we currently have.

- Jonathan M Davis
Jun 19 2013
next sibling parent reply "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis 
wrote:

 On Wednesday, June 19, 2013 08:51:38 TommiT wrote:

 I can't put a member function into a different file if it's
 supposed to have access to private data of its enclosing
 class/struct.


 Then you can put the class in a different file. But even if
 you can't, I don't think that it's worth it to complicate the
 package attribute any further.


Let's now assume that we change the language specification so 
that 'package' access specifier allows access also to modules in 
nested packages. I'll show how difficult it can be to refactor 
your code in order to break it up into a package:

The file structure:
-------------------
/my_lib/mod_x.d
/my_lib/mod_y.d


File: /my_lib/mod_x.d
----------------------
module my_lib.mod_x;

import my_lib.mod_y;

void fun()
{
     my_lib.mod_y.S1 s1;
     s1.detail();
}


File: /my_lib/mod_y.d
----------------------
module my_lib.mod_y;

struct S1
{
     public void api()
     {
         S2 s2;
         s2.detail();
     }

     package void detail() { }
}

struct S2
{
     public void api() { }

     package void detail() { }
}


Now, I want to break up both mod_x.d and mod_y.d into packages. 
I'll start by doing it naively and end up with this:

The file structure:
-------------------
/my_lib/mod_x/submod.d
/my_lib/mod_x/package.d
/my_lib/mod_y/submod_s1.d
/my_lib/mod_y/submod_s2.d
/my_lib/mod_y/package.d


File: /my_lib/mod_x/submod.d
-----------------------------
module my_lib.mod_x.submod;

import my_lib.mod_y;

void fun()
{
     my_lib.mod_y.S1 s1;
     s1.detail();
}


File: /my_lib/mod_x/package.d
------------------------------
module my_lib.mod_x;

public import my_lib.mod_x.submod;


File: /my_lib/mod_y/submod_s1.d
--------------------------------
module my_lib.mod_y.submod_s1;

import my_lib.mod_y.submod_s2;

struct S1
{
     public void api()
     {
         S2 s2;
         s2.detail();
     }

     package void detail() { }
}


File: /my_lib/mod_y/submod_s2.d
--------------------------------
module my_lib.mod_y.submod_s2;

struct S2
{
     public void api() { }

     package void detail() { }
}


File: /my_lib/mod_y/package.d
------------------------------
module my_lib.mod_y;

public import my_lib.mod_y.submod_s1;
public import my_lib.mod_y.submod_s2;


At this point I notice this doesn't work because 
my_lib.mod_x.submod.fun tries to call my_lib.mod_y.S1.detail 
which is marked package and now in different package. In order to 
fix this without changing the public API, I'll create a new file 
called shared_s1.d, move the struct S1 there, and public import 
shared_s1.d back into the S1's original submod_s1.d file:

The file structure:
-------------------
/my_lib/mod_x/submod.d
/my_lib/mod_x/package.d
/my_lib/mod_y/submod_s1.d
/my_lib/mod_y/submod_s2.d
/my_lib/mod_y/package.d
/my_lib/shared_s1.d


File: /my_lib/shared_s1.d
--------------------------
module my_lib.shared_s1;

import my_lib.mod_y.submod_s2;

struct S1
{
     public void api()
     {
         S2 s2;
         s2.detail();
     }

     package void detail() { }
}


File: /my_lib/mod_y/submod_s1.d
--------------------------------
module my_lib.mod_y.submod_s1;

public import my_lib.shared_s1;


(other files stay the same)

Now my_lib.mod_x.submod.fun has access rights and is able to call 
my_lib.mod_y.S1.detail because it's now an alias to 
my_lib.shared_s1.S1.detail. But then we notice that it doesn't 
work, because by moving the definition of S1 one folder up, we 
made it so that my_lib.shared_s1.S1.api isn't able to call 
my_lib.mod_y.submod_s2.S2.detail. So, we must do the same trick 
for the struct S2 now. Notice how viral this effect is. We end up 
with:

The file structure:
-------------------
/my_lib/mod_x/submod.d
/my_lib/mod_x/package.d
/my_lib/mod_y/submod_s1.d
/my_lib/mod_y/submod_s2.d
/my_lib/mod_y/package.d
/my_lib/shared_s1.d
/my_lib/shared_s2.d


File: /my_lib/shared_s2.d
--------------------------
module my_lib.shared_s2;

struct S2
{
     public void api() { }

     package void detail() { }
}


File: /my_lib/mod_y/submod_s2.d
--------------------------------
module my_lib.mod_y.submod_s2;

public import my_lib.shared_s2;



And now it all works. But what did we learn from all this:

Your suggestion makes breaking modules into packages easier to 
understand but harder to do.

My suggestion makes breaking modules into packages harder to 
understand but easier to do.


On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis 
wrote:

 Even if we lose something in the process, we really don't lose
 much, and since currently, package only applies to modules
 directly in the same package and not nested packages, you'd
 still be gaining over what we currently have.


I don't want 'package' symbols to give away access to nested 
packages.
Jun 19 2013
parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 09:55:32 UTC, TommiT wrote:

 Your suggestion makes breaking modules into packages easier to 
 understand but harder to do.


...and not to mention, just plain ugly. I mean look at how those 
files that are part of a package end up being in different 
folders and you have those dummy (shared) files which the end 
user is not supposed to import.
Jun 19 2013
prev sibling parent reply "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis 
wrote:

 [..] But even if you can't, I don't think that it's worth it to
 complicate the package attribute any further. [..]


I'd like to quote the TDPL foreword by Walter:
"To the best of my knowledge, D offers an unprecedentedly adroit 
integration of several powerful programming paradigms: 
imperative, object-oriented, functional, and meta.
At first blush, it would appear that such a language could not be 
simple. And indeed, D is not a simple language. But I’d argue 
that is the wrong way to view a language. A more useful view is, 
what do programming solutions in that language look like? Are D 
programs complicated and obtuse, or simple and elegant?"
   -Walter Bright

Thus, we shouldn't strive so much to make the language simple (by 
having the simplest possible definition for the 'package' 
keyword), but rather, we should try to make programs, which use 
modules broken into packages, look simple, elegant and easy to 
understand.
Jun 19 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, June 19, 2013 13:13:54 TommiT wrote:

 On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis
 
 wrote:

 [..] But even if you can't, I don't think that it's worth it to
 complicate the package attribute any further. [..]


 
 I'd like to quote the TDPL foreword by Walter:
 "To the best of my knowledge, D offers an unprecedentedly adroit
 integration of several powerful programming paradigms:
 imperative, object-oriented, functional, and meta.
 At first blush, it would appear that such a language could not be
 simple. And indeed, D is not a simple language. But I’d argue
 that is the wrong way to view a language. A more useful view is,
 what do programming solutions in that language look like? Are D
 programs complicated and obtuse, or simple and elegant?"
 -Walter Bright
 
 Thus, we shouldn't strive so much to make the language simple (by
 having the simplest possible definition for the 'package'
 keyword), but rather, we should try to make programs, which use
 modules broken into packages, look simple, elegant and easy to
 understand.


Sure, but there's a limit to how much complexity is reasonable, and every 
feature has to pull its own weight. And while I agree that there are cases 
where refactoring packages may cause complications with the package modifier 
specfically giving nested packages access to their parent packages' package 
stuff but not do anything like what you're suggesting, I don't agree that it's 
a big enough deal to merit complicating the package access level any further. 
If it's a problem, then maybe the code just plain needs to be redesigned 
rather than simply shuffled around. If we were really looking for full power 
with rearranging stuff like that, we'd have friend functions and classes, and D 
specifically elected _not_ to go for that level of complication. And overall, 
it's worked great for us thus far.

- Jonathan M Davis
Jun 19 2013
parent reply "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 20:05:56 UTC, Jonathan M Davis
wrote:

 On Wednesday, June 19, 2013 13:13:54 TommiT wrote:

 On Wednesday, 19 June 2013 at 07:12:30 UTC, Jonathan M Davis
 
 wrote:

 [..] But even if you can't, I don't think that it's worth it 
 to
 complicate the package attribute any further. [..]


 
 I'd like to quote the TDPL foreword by Walter:
 "To the best of my knowledge, D offers an unprecedentedly 
 adroit
 integration of several powerful programming paradigms:
 imperative, object-oriented, functional, and meta.
 At first blush, it would appear that such a language could not 
 be
 simple. And indeed, D is not a simple language. But I’d argue
 that is the wrong way to view a language. A more useful view 
 is,
 what do programming solutions in that language look like? Are D
 programs complicated and obtuse, or simple and elegant?"
 -Walter Bright
 
 Thus, we shouldn't strive so much to make the language simple 
 (by
 having the simplest possible definition for the 'package'
 keyword), but rather, we should try to make programs, which use
 modules broken into packages, look simple, elegant and easy to
 understand.


 Sure, but there's a limit to how much complexity is reasonable, 
 and every
 feature has to pull its own weight. And while I agree that 
 there are cases
 where refactoring packages may cause complications with the 
 package modifier
 specfically giving nested packages access to their parent 
 packages' package
 stuff but not do anything like what you're suggesting, I don't 
 agree that it's
 a big enough deal to merit complicating the package access 
 level any further.
 If it's a problem, then maybe the code just plain needs to be 
 redesigned
 rather than simply shuffled around. If we were really looking 
 for full power
 with rearranging stuff like that, we'd have friend functions 
 and classes, and D
 specifically elected _not_ to go for that level of 
 complication. And overall,
 it's worked great for us thus far.

 - Jonathan M Davis


No, we are not hoping to rearrange stuff like that. We are 
_forced_
to rearrange stuff like that if we go with your suggestion. With
my suggestion, what I presented as the first attempt at breaking
modules into packages would just work. (in case you actually read 
my post)

And no, we don't want friend function, I never said that.

Also, I'm just curious why do you keep saying "we don't want to
complicate the package access specifier any further"? Because
isn't the current specification of the package access specifier
the simplest possible that it could ever be? "Everything under
the same folder has access to symbols labeled package". It takes
just 11 words to define it. You're talking about it like it's
already somehow complicated.
Jun 19 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 20, 2013 00:21:45 TommiT wrote:

 Also, I'm just curious why do you keep saying "we don't want to
 complicate the package access specifier any further"? Because
 isn't the current specification of the package access specifier
 the simplest possible that it could ever be? "Everything under
 the same folder has access to symbols labeled package". It takes
 just 11 words to define it. You're talking about it like it's
 already somehow complicated.


No, it's not complicated, but the lanugage as a whole is complicated, and any 
new feature that's added to it increases its complexity. As such, it needs to 
pull its weight, and I really don't believe that that's the case here. I just 
don't think that complicating the package access modifier any further is worth 
the gain. There is some gain, but I think that it's ultimately quite small, 
and I'd much prefer that access modifiers stay simple. Obviously, you're 
entitled to think that the extra complexity is worth it, but I don't agree.

- Jonathan M Davis
Jun 19 2013
parent reply "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 22:40:47 UTC, Jonathan M Davis 
wrote:

 On Thursday, June 20, 2013 00:21:45 TommiT wrote:

 Also, I'm just curious why do you keep saying "we don't want to
 complicate the package access specifier any further"? Because
 isn't the current specification of the package access specifier
 the simplest possible that it could ever be? "Everything under
 the same folder has access to symbols labeled package". It 
 takes
 just 11 words to define it. You're talking about it like it's
 already somehow complicated.


 No, it's not complicated, but the lanugage as a whole is 
 complicated, and any
 new feature that's added to it increases its complexity. As 
 such, it needs to
 pull its weight, and I really don't believe that that's the 
 case here. I just
 don't think that complicating the package access modifier any 
 further is worth
 the gain. There is some gain, but I think that it's ultimately 
 quite small,
 and I'd much prefer that access modifiers stay simple. 
 Obviously, you're
 entitled to think that the extra complexity is worth it, but I 
 don't agree.

 - Jonathan M Davis


You're willing to add extra complexity and inconvenience to 
programming in D just in order to keep the language specification 
simple. I don't think it's a good trade-off. Learning the complex 
details of language is a one-time cost that all programmers must 
pay when they start with the language. Whereas complexity and 
inconvenience in actually programming with the language is a 
running cost and may be a source of bugs as well. The running 
cost should clearly over-weight the one-time cost here.
Jun 19 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 20, 2013 05:39:30 TommiT wrote:

 On Wednesday, 19 June 2013 at 22:40:47 UTC, Jonathan M Davis
 
 wrote:

 On Thursday, June 20, 2013 00:21:45 TommiT wrote:

 Also, I'm just curious why do you keep saying "we don't want to
 complicate the package access specifier any further"? Because
 isn't the current specification of the package access specifier
 the simplest possible that it could ever be? "Everything under
 the same folder has access to symbols labeled package". It
 takes
 just 11 words to define it. You're talking about it like it's
 already somehow complicated.


 
 No, it's not complicated, but the lanugage as a whole is
 complicated, and any
 new feature that's added to it increases its complexity. As
 such, it needs to
 pull its weight, and I really don't believe that that's the
 case here. I just
 don't think that complicating the package access modifier any
 further is worth
 the gain. There is some gain, but I think that it's ultimately
 quite small,
 and I'd much prefer that access modifiers stay simple.
 Obviously, you're
 entitled to think that the extra complexity is worth it, but I
 don't agree.
 
 - Jonathan M Davis


 
 You're willing to add extra complexity and inconvenience to
 programming in D just in order to keep the language specification
 simple. I don't think it's a good trade-off. Learning the complex
 details of language is a one-time cost that all programmers must
 pay when they start with the language. Whereas complexity and
 inconvenience in actually programming with the language is a
 running cost and may be a source of bugs as well. The running
 cost should clearly over-weight the one-time cost here.


I don't think that this adds much complication to programs, because I don't 
think that the sorts of problems that you're worried about when refactoring 
code are likely to be common, and they can be gotten around by reworking how 
your code is designed. Worst case, you just make the functionality public but 
undocumented. And while that may not be ideal, I really don't think that it 
will happen enough to be worth the complication to the language required to 
avoid it.

And consider the fact that even with your suggested changes, there are going 
to be ways that you can't refactor your code and still keep it using the same 
API and/or the same access level. We can pretty much always find more ways to 
refactor code that won't work with the access specifiers unless you introduce 
something like friend which allows you to explicitly specify who can access 
who rather than using general groups like public and private. And you have to 
draw the line somewhere.

I think that making it so that the package access specifier lets nested 
packages access it in addition to the exact package is a good balance between 
simplicity and power.

However, if you want to try and convince Walter, feel free. But from what he's 
said in the past, I suspect that he leans more towards getting rid of package 
entirely than thinking that it's worth making it more complicated in the 
manner that you're looking for.

- Jonathan M Davis
Jun 19 2013
next sibling parent reply "TommiT" <tommitissari hotmail.com> writes:
This is how I see the logic of this new feature you're adding:

When you have a folder named "mod" which contains a file named 
package.d, the mod folder becomes conceptually a file with same 
name and the extension "d". I.e. the folder conceptually becomes 
a module, which you can then import to other modules. All the 
code contained within all the files contained within the mod 
folder becomes conceptually the content of this imaginary mod.d 
file. Therefore it is only logical that all the code within this 
imaginary mod.d file would have access to 'package'-labeled 
symbols that are in files which are located under the same folder 
under which this imaginary mod.d file is.
Jun 19 2013
parent reply "TommiT" <tommitissari hotmail.com> writes:
On Thursday, 20 June 2013 at 05:24:12 UTC, TommiT wrote:

 This is how I see the logic of this new feature you're adding:

 When you have a folder named "mod" which contains a file named 
 package.d, the mod folder becomes conceptually a file with same 
 name and the extension "d". I.e. the folder conceptually 
 becomes a module, which you can then import to other modules. 
 All the code contained within all the files contained within 
 the mod folder becomes conceptually the content of this 
 imaginary mod.d file. Therefore it is only logical that all the 
 code within this imaginary mod.d file would have access to 
 'package'-labeled symbols that are in files which are located 
 under the same folder under which this imaginary mod.d file is.


...and following that thread, I would also argue, that 
'private'-labeled symbols in files that are inside this mod 
folder should be accessible to all the other files that are 
inside this mod folder, because conceptually all those files are 
part of the same (imaginary) module (mod.d).
Jun 19 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 20, 2013 08:45:21 TommiT wrote:

 On Thursday, 20 June 2013 at 05:24:12 UTC, TommiT wrote:

 This is how I see the logic of this new feature you're adding:
 
 When you have a folder named "mod" which contains a file named
 package.d, the mod folder becomes conceptually a file with same
 name and the extension "d". I.e. the folder conceptually
 becomes a module, which you can then import to other modules.
 All the code contained within all the files contained within
 the mod folder becomes conceptually the content of this
 imaginary mod.d file. Therefore it is only logical that all the
 code within this imaginary mod.d file would have access to
 'package'-labeled symbols that are in files which are located
 under the same folder under which this imaginary mod.d file is.


 
 ...and following that thread, I would also argue, that
 'private'-labeled symbols in files that are inside this mod
 folder should be accessible to all the other files that are
 inside this mod folder, because conceptually all those files are
 part of the same (imaginary) module (mod.d).


The _only_ reason that we added package.d was to allow us to split up modules 
in place without breaking existing code. And it works as well as it does, 
because the _only_ new feature that it provides is making it so that when you 
import a package, you import package.d. All other aspects about it were 
already part of the language, and we weren't looking to add any new 
functionality whatsoever beyond being able to break up modules like 
std.algorithm or std.datetime. Trying to treat a package as if it were a 
module because it has a package.d file is contorting things considerably. That 
was never the intention of package.d at all. It's to help us evolve code 
without breaking public APIs. package is not part of the public API. private 
is not part of the public API. As such, they have _nothing_ to do with the 
purpose of package.d. And it was never intended that packages be treated as 
modules due to the presence of package.d save for the purpose of avoiding code 
breakage. IMHO, you are trying to take this _way_ too far.

- Jonathan M Davis
Jun 20 2013
parent "TommiT" <tommitissari hotmail.com> writes:
On Thursday, 20 June 2013 at 07:59:57 UTC, Jonathan M Davis wrote:

 The _only_ reason that we added package.d was to allow us to 
 split up modules
 in place without breaking existing code. And it works as well 
 as it does,
 because the _only_ new feature that it provides is making it so 
 that when you
 import a package, you import package.d. All other aspects about 
 it were
 already part of the language, and we weren't looking to add any 
 new
 functionality whatsoever beyond being able to break up modules 
 like
 std.algorithm or std.datetime. Trying to treat a package as if 
 it were a
 module because it has a package.d file is contorting things 
 considerably. That
 was never the intention of package.d at all. It's to help us 
 evolve code
 without breaking public APIs. package is not part of the public 
 API. private
 is not part of the public API. As such, they have _nothing_ to 
 do with the
 purpose of package.d. And it was never intended that packages 
 be treated as
 modules due to the presence of package.d save for the purpose 
 of avoiding code
 breakage. IMHO, you are trying to take this _way_ too far.

 - Jonathan M Davis


Ok. Then this feature is not really about being able to easily 
break an existing module into a package, but rather about 
providing a different way to organize your code from the get-go. 
You _can_ break existing modules into packages, but you may need 
to go through some hoops to do so, and that's fine, because 
that's not the primary purpose of this feature anyway.

I'd say let's keep the package access specifier as it is now.
Jun 20 2013
prev sibling parent "TommiT" <tommitissari hotmail.com> writes:
Also, if we loosen the encapsulation provided by the package 
access specifier so that it allows access also to modules in 
nested packages, then why is this keyword even named "package". 
Newcomers to the language would be asking us why is this keyword 
names "package", when it provides access beyond the package where 
the module is in. Loosening the encapsulation provided by 
'package' is not logical nor called for.
Jun 19 2013
prev sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Wednesday, 19 June 2013 at 06:25:02 UTC, TommiT wrote:

 I don't like the extra complication either, ...


I should have said, I don't like the added complexity. But I 
think this is one of those things that becomes totally obvious 
once you use it for a while and get used to thinking about these 
packaged-module thingies as just simply modules.
Jun 18 2013
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 05 Jun 2013 22:35:59 -0400
schrieb "Jonathan M Davis" <jmdavisProg gmx.com>:


 std.compress.zlib;
 std.compress.lzw;
 
 At that point, it would be trivial to have a common module of some kind with 
 shared functionality which has package access level.
 
 - Jonathan M Davis


+1

If package is ambiguous, maybe it needs to be split in two or
take a parameter package(std.algorithm) or something like that
to serve both use cases.

-- 
Marco
Jun 05 2013
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 08:31:57 Marco Leise wrote:

 If package is ambiguous, maybe it needs to be split in two or
 take a parameter package(std.algorithm) or something like that
 to serve both use cases.


While that might be nice, I'm inclined to think that it would be better to 
avoid the extra complication and just pick the behavior that seems the most 
useful. In neither case is the loss all that large, particularly since package 
is likely to be used fairly rarely. It's quite useful to be sure, but it has a 
narrow enough set of typical use cases that complicating it probably isn't 
worth it.

- Jonathan M Davis
Jun 06 2013
prev sibling next sibling parent reply "nazriel" <spam dzfl.pl> writes:
On Wednesday, 5 June 2013 at 22:31:21 UTC, Andrei Alexandrescu 
wrote:

 https://github.com/D-Programming-Language/dmd/pull/2139

 Andrei


I had a spin[1] with it long before Adam Wilson made it so 
popular topic. Feature was frequently mentioned on IRC as 
desired. Unfortunately I faced few holes in both Martin's and 
Andrei's DIPs and gave up on implementing it.

Few issues I remember were:

---

module std.datetime.core;
class Foo {}

module std.datetime.calendar;
class Foo {}

module std.datetime;

public import std.datetime.core;
public import std.datetime.calendar;

module usage;

import std.datetime;

Foo x; // is it std.datetime.core or std.datetime.calender?

// ok workaround..
std.datetime.Foo x; // same issue as above

std.datetime.core.Foo x; // works atleast

---

My point is, are directions Jonathan wants to go are right?
I think std.datetime.Foo shouldn't be available.

I remember also having some issues with ambiguous symbols. I had 
more corner cases in IRC Logs, but can't find it now.

Also I see we are going with Andrei's DIP route.
It is worth noting that Martin's one may have some additional 
functionality.

Like glueing modules. Ie.

---

module std.net;

public import std.net.http;
public import std.net.uri;

shared(Uri) uri = new shared(Uri);

---

In Martin's proposal uri symbol is reachable via
std.net.__init__.uri in case of ambiguous. In Andrei's it is 
unreachable due to package being keyword.

I am glad Walter is tackling it, because it is really useful 
feature, but please take a chill pill and rethink all corner 
cases.

[1] https://github.com/nazriel/dmd/tree/dip16++
Jun 06 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 16:53:37 nazriel wrote:

 Few issues I remember were:
 
 module std.datetime.core;
 class Foo {}
 
 module std.datetime.calendar;
 class Foo {}
 
 module std.datetime;
 
 public import std.datetime.core;
 public import std.datetime.calendar;
 
 module usage;
 
 import std.datetime;
 
 Foo x; // is it std.datetime.core or std.datetime.calender?
 
 // ok workaround..
 std.datetime.Foo x; // same issue as above
 
 std.datetime.core.Foo x; // works atleast


If you have that problem, then you don't publicly import the entire module. 
It's up to the package designer to decide which portions of the package get 
publicly imported. But since std.datetime.Foo would be ambiguous, I don't 
think that it really matters anyway. In that case, you're simply forced to 
refer to it by its actual module rather than by the package. The only time 
that something like this is likely to be a concern is when you add a 
conflicting function to another module in the package, because then 
std.datime.Foo was working and then ceased to work. That's arguably cause for 
being careful about what you choose to publicly import in package.d, but I 
don't think that it invalidates the feature design at all.


 Also I see we are going with Andrei's DIP route.


How so? What Walter has done is almost identical to DIP 37. I believe that the 
only difference is that std.datetime.package would have module std.datetime; at 
the top, whereas DIP 37 currently says that it would have module 
std.datetime.package;


 I am glad Walter is tackling it, because it is really useful
 feature, but please take a chill pill and rethink all corner
 cases.


We've already discussed this at length. It's possible that we missed 
something, but this proposal is not something that we just jumped into without 
thinking it through first. In fact, it actually took quite a bit to talk Walter 
into the necessity of it in the first place.

- Jonathan M Davis
Jun 06 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2013 11:09 AM, Jonathan M Davis wrote:

 We've already discussed this at length. It's possible that we missed
 something, but this proposal is not something that we just jumped into without
 thinking it through first. In fact, it actually took quite a bit to talk Walter
 into the necessity of it in the first place.


The only reason I'm for it is because we need to break up std.algorithm (for
the 
canonical example) without breaking existing code.

I'm sure the problem with:

simple module => kitchen sink monstrosity => break up module

is a recurring issue that most everyone has sooner or later. We need to support
it.
Jun 06 2013
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, June 06, 2013 11:16:03 Walter Bright wrote:

 On 6/6/2013 11:09 AM, Jonathan M Davis wrote:

 We've already discussed this at length. It's possible that we missed
 something, but this proposal is not something that we just jumped into
 without thinking it through first. In fact, it actually took quite a bit
 to talk Walter into the necessity of it in the first place.


 
 The only reason I'm for it is because we need to break up std.algorithm (for
 the canonical example) without breaking existing code.
 
 I'm sure the problem with:
 
 simple module => kitchen sink monstrosity => break up module
 
 is a recurring issue that most everyone has sooner or later. We need to
 support it.-


Yes. If it weren't for the need to break up modules in-place without breaking 
code, then I'd argue for simply using the foo/all.d idiom where all.d publicly 
imports all of the modules in foo. Then you just import foo, and you get 
everything. IMHO, that solves the need for importing packages well enough to 
avoid needing a language change for it. People have been using it for that 
already. But it _doesn't_ allow you to break up modules in place without 
breaking code, which is why I think that this DIP is worth implementing.

So, I think that we're in pretty strong agreement about it at this point.


features which promote being able to evolve code can be very valuable, and I 
think that we have some good additions in that camp over languages like C++ or 
Java (alias this and deprecated in particular come to mind) - maybe not as 
many as would be nice, but we do have some solid features for code evolution. 
And I think that this will be a good addition to that.

- Jonathan M Davis
Jun 06 2013
prev sibling parent reply "nazriel" <spam dzfl.pl> writes:
On Thursday, 6 June 2013 at 18:10:11 UTC, Jonathan M Davis wrote:

 If you have that problem, then you don't publicly import the 
 entire module.
 It's up to the package designer to decide which portions of the 
 package get
 publicly imported. But since std.datetime.Foo would be 
 ambiguous, I don't
 think that it really matters anyway. In that case, you're 
 simply forced to
 refer to it by its actual module rather than by the package. 
 The only time
 that something like this is likely to be a concern is when you 
 add a
 conflicting function to another module in the package, because 
 then
 std.datime.Foo was working and then ceased to work. That's 
 arguably cause for
 being careful about what you choose to publicly import in 
 package.d, but I
 don't think that it invalidates the feature design at all.


Ok, after carefully rereading thread now I understand your point. 
Preventing breakage when std.datetime is splited into sub 
packages - ok.

But still it looks weird to me that allowing access to 
std.datetime.time.Clock via std.datetime.Clock, even though Clock 
is defined in std.datetime.time and std.datetime.package is only 
publicly importing it, is ok. I guess whatever suits you :))


 Also I see we are going with Andrei's DIP route.


 How so? What Walter has done is almost identical to DIP 37. I 
 believe that the
 only difference is that std.datetime.package would have module 
 std.datetime; at
 the top, whereas DIP 37 currently says that it would have module
 std.datetime.package;


Ach pardon, I am lost in all those DIPs. I recalled that Andrei's 
proposed package.d file, those I referred to your DIP37 (which 
seems to be summary of Andrei's DIP?) as Andrei's


 I am glad Walter is tackling it, because it is really useful
 feature, but please take a chill pill and rethink all corner
 cases.


 We've already discussed this at length. It's possible that we 
 missed
 something, but this proposal is not something that we just 
 jumped into without
 thinking it through first. In fact, it actually took quite a 
 bit to talk Walter
 into the necessity of it in the first place.


Sorry, it is just the impression that this feature was requested 
very long time ago, Martin's and Andrei's DIPs were staging and 
recently talk by Adam Wilson made it interesting to the D "crew".

If it was carefully discussed and I somehow missed those 
discussion or I am not allowed to see them, then I am sorry and 
please ignore this and my previous post in this topic.


 - Jonathan M Davis


Best regards,
Damian Ziemba
Jun 07 2013
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 6/7/13, nazriel <spam dzfl.pl> wrote:

 But still it looks weird to me that allowing access to
 std.datetime.time.Clock via std.datetime.Clock, even though Clock
 is defined in std.datetime.time and std.datetime.package is only
 publicly importing it, is ok.


This could come as a benefit. For example, a user might not know
exactly which modules a package module is composed of, so he'll
instinctively try to type "std.datetime.time" and it will work.
Jun 07 2013
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, June 07, 2013 10:55:36 nazriel wrote:

 If it was carefully discussed and I somehow missed those
 discussion or I am not allowed to see them, then I am sorry and
 please ignore this and my previous post in this topic.


Actually, Daniel Murphy, Martin Nowak, and I discussed it with Walter and 
Andrei at dconf the night of Adam's talk, so it wasn't on the newsgroup. 
There's nothing secret about it though. It's just that it was discussed in 
person, so there is no record. I believe that some further discussion has 
occured in the newsgroup in the two pull requests for it as well as the DIP 
announcement, but we went into a fair bit of detail in the discussion at dconf 
of what exactly the pros and cons would be as well as the side effects.

What's really nice about DIP 37 is that it takes advantage of a number of 
existing features rather than requiring much new. It's essentially just taking 
what currently works with a module like all.d (publicly importing the 
everything in the package in the one module) and making it work in a manner 
that we can avoid code breakage. And naming the file package.d is a particular 
stroke of genius (provided by someone in the original discussion on Martion 
Nowak's DIP on the issue IIRC), because then the file can't even conflict with 
any existing code. So, while the implementation does have some definite nuances 
to it (like making sure that using the explict path to something like 
std.datetime.DosFileTime still works), it's ultimately fairly simple and 
straightforward. And given the need to break up modules in place, we decided 
that it was worth implementing.

- Jonathan M Davis
Jun 07 2013
prev sibling parent reply "Sad panda" <sunspyre gmail.com> writes:
Can we use some other word for counterargument than destroy 
please. :(

(resume discussion)
Jun 06 2013
parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Thu, 06 Jun 2013 22:59:41 +0200, Sad panda <sunspyre gmail.com> wrote:


 Can we use some other word for counterargument than destroy please. :(

 (resume discussion)


But we like destroy.

-- 
Simen
Jun 07 2013