www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - phobos dependencies

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1) 
for each module in phobos. Those include druntime dependencies.

With https://github.com/D-Programming-Language/phobos/pull/1768 
dependencies would be generated automatically, which should be a good 
step toward reining in.

The zsh command I ran was  wc -w generated/**/*.deps | sort -nr.


Andrei
Dec 18 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
19-Dec-2013 01:06, Andrei Alexandrescu пишет:

 http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1)
 for each module in phobos. Those include druntime dependencies.


So the bill of using pretty much anything in Phobos is pulling in 87 
modules. Pretty much what I feared it is.

-- 
Dmitry Olshansky
Dec 18 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/18/13 1:16 PM, Dmitry Olshansky wrote:

 19-Dec-2013 01:06, Andrei Alexandrescu пишет:

 http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1)
 for each module in phobos. Those include druntime dependencies.


 So the bill of using pretty much anything in Phobos is pulling in 87
 modules. Pretty much what I feared it is.


There are several directions we can take this.

1. Improve the compiler to handle imports lazily, i.e. an unused import 
is never opened. That's unlikely to help a lot of uses because most 
unqualified name lookups require all imports to be loaded (even after 
the name if resolved, the compiler must still look for ambiguities).

2. Push imports from top level into the entities (functions, classes 
etc) that use them.

3. Apply classic dependency management (break larger modules in smaller 
ones, accept some code duplication etc).

I favor (2).


Andrei
Dec 18 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
19-Dec-2013 01:40, Andrei Alexandrescu пишет:

 On 12/18/13 1:16 PM, Dmitry Olshansky wrote:

 19-Dec-2013 01:06, Andrei Alexandrescu пишет:

 http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1)
 for each module in phobos. Those include druntime dependencies.


 So the bill of using pretty much anything in Phobos is pulling in 87
 modules. Pretty much what I feared it is.


 There are several directions we can take this.

 1. Improve the compiler to handle imports lazily, i.e. an unused import
 is never opened. That's unlikely to help a lot of uses because most
 unqualified name lookups require all imports to be loaded (even after
 the name if resolved, the compiler must still look for ambiguities).

 2. Push imports from top level into the entities (functions, classes
 etc) that use them.

 3. Apply classic dependency management (break larger modules in smaller
 ones, accept some code duplication etc).

 I favor (2).


I'd add something that might help make (2) possible in case of templates.

4. Split modules at least into a package of with at least 2 parts:
- Meta (traits) - constraints, type definitions etc.
- API - functions, globals and other "meat" of the module.

One prime example is std.range - isInputRange, isForwardRange etc. need 
not to be "bought together" with all of range adapters and std.algorithm 
dependency.

-- 
Dmitry Olshansky
Dec 18 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Dec 19, 2013 at 01:47:26AM +0400, Dmitry Olshansky wrote:

 19-Dec-2013 01:40, Andrei Alexandrescu пишет:

On 12/18/13 1:16 PM, Dmitry Olshansky wrote:

19-Dec-2013 01:06, Andrei Alexandrescu пишет:

http://chopapp.com/#fvepfd8 shows the number of dependencies (plus
1) for each module in phobos. Those include druntime dependencies.


So the bill of using pretty much anything in Phobos is pulling in 87
modules. Pretty much what I feared it is.


There are several directions we can take this.

1. Improve the compiler to handle imports lazily, i.e. an unused
import is never opened. That's unlikely to help a lot of uses because
most unqualified name lookups require all imports to be loaded (even
after the name if resolved, the compiler must still look for
ambiguities).

2. Push imports from top level into the entities (functions, classes
etc) that use them.

3. Apply classic dependency management (break larger modules in
smaller ones, accept some code duplication etc).

I favor (2).




I vote for (2) as well.

And for the first part of (3): I think std.algorithm is due for a
breakup. There is no reason why using find, for example, should pull in
sorting algorithms, and why using sorting algorithms should pull in
cartesianProduct.



 I'd add something that might help make (2) possible in case of
 templates.
 
 4. Split modules at least into a package of with at least 2 parts:
 - Meta (traits) - constraints, type definitions etc.
 - API - functions, globals and other "meat" of the module.
 
 One prime example is std.range - isInputRange, isForwardRange etc.
 need not to be "bought together" with all of range adapters and
 std.algorithm dependency.


[...]

I've been saying this over and over: now we have package.d, let's make
use of it. Jonathan has been working on splitting up std.datetime
(though that has yet to materialize); I think std.algorithm and
std.range are due, too.


T

-- 
"Hi." "'Lo."
Dec 18 2013
parent Shammah Chancellor <anonymous coward.com> writes:
On 2013-12-18 22:19:14 +0000, H. S. Teoh said:


 I've been saying this over and over: now we have package.d, let's make
 use of it. Jonathan has been working on splitting up std.datetime
 (though that has yet to materialize); I think std.algorithm and
 std.range are due, too.


+1 on this.   When I have some time I was going to make a pull request 
on some of the modules.

-Shammah
Dec 19 2013
prev sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 18 December 2013 at 21:48:14 UTC, Dmitry Olshansky 
wrote:

 4. Split modules at least into a package of with at least 2 
 parts:
 - Meta (traits) - constraints, type definitions etc.
 - API - functions, globals and other "meat" of the module.


Yes, yes, yes. This could also help with encapsulation, since 
private works across module boundaries. The type listing only has 
minimal functions for its interface, then other functions are 
used, with UFCS, from the meat module.
Dec 19 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/18/2013 1:40 PM, Andrei Alexandrescu wrote:

 1. Improve the compiler to handle imports lazily, i.e. an unused import is
never
 opened. That's unlikely to help a lot of uses because most unqualified name
 lookups require all imports to be loaded (even after the name if resolved, the
 compiler must still look for ambiguities).

 2. Push imports from top level into the entities (functions, classes etc) that
 use them.

 3. Apply classic dependency management (break larger modules in smaller ones,
 accept some code duplication etc).

 I favor (2).


(1) is impractical for the reasons you mentioned.

I favor (2), and also (4):

4. Break kitchen sink modules like std.algorithm into one module per algorithm. 
This should not result in code duplication.
Dec 18 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/18/13 3:08 PM, Walter Bright wrote:

 On 12/18/2013 1:40 PM, Andrei Alexandrescu wrote:

 1. Improve the compiler to handle imports lazily, i.e. an unused
 import is never
 opened. That's unlikely to help a lot of uses because most unqualified
 name
 lookups require all imports to be loaded (even after the name if
 resolved, the
 compiler must still look for ambiguities).

 2. Push imports from top level into the entities (functions, classes
 etc) that
 use them.

 3. Apply classic dependency management (break larger modules in
 smaller ones,
 accept some code duplication etc).

 I favor (2).


 (1) is impractical for the reasons you mentioned.

 I favor (2), and also (4):

 4. Break kitchen sink modules like std.algorithm into one module per
 algorithm. This should not result in code duplication.


That's (3), which enumerates a number of possible effects that may or 
may not overlap/

Andrei
Dec 18 2013
prev sibling next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Wednesday, 18 December 2013 at 23:08:21 UTC, Walter Bright 
wrote:

 I favor (2), and also (4):

 4. Break kitchen sink modules like std.algorithm into one 
 module per algorithm. This should not result in code 
 duplication.


That seems a little over the top. While I guess it's not 
necessarily a bad thing to increase modularity, users would have 
to pull in a large number of imports to do anything nontrivial. 
For example, when you want to use sort, find, splitter and swap 
all together, you either have the choice of doing:

import std.algorithm;

And pull in everything, or doing:

import std.algorithm.sort, std.algorithm.find, 
std.algorithm.splitter, std.algorithm.swap;

Or is there something I'm missing here?

Of course, another large boon would be to correct the 
implementation of:

import std.algorithm: sort, find, splitter, swap;

So that it actually works in a sane way as opposed to pulling in 
everything.
Dec 18 2013
next sibling parent reply Justin Whear <justin economicmodeling.com> writes:
On Thu, 19 Dec 2013 00:51:04 +0100, Meta wrote:


 On Wednesday, 18 December 2013 at 23:08:21 UTC, Walter Bright wrote:

 I favor (2), and also (4):

 4. Break kitchen sink modules like std.algorithm into one module per
 algorithm. This should not result in code duplication.


 
 That seems a little over the top. While I guess it's not necessarily a
 bad thing to increase modularity, users would have to pull in a large
 number of imports to do anything nontrivial. For example, when you want
 to use sort, find, splitter and swap all together, you either have the
 choice of doing:
 
 import std.algorithm;
 
 And pull in everything, or doing:
 
 import std.algorithm.sort, std.algorithm.find, std.algorithm.splitter,
 std.algorithm.swap;
 
 Or is there something I'm missing here?
 
 Of course, another large boon would be to correct the implementation of:
 
 import std.algorithm: sort, find, splitter, swap;
 
 So that it actually works in a sane way as opposed to pulling in
 everything.


Presumably there would still be a std.algorithm package module;  the 
questions is: does
  import std.algorithm : joiner;
still work if std.algorithm is actually a package module and joiner is in 
a std.algorithm.joiner module?
Dec 18 2013
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 19 December 2013 at 00:15:12 UTC, Justin Whear wrote:

 Presumably there would still be a std.algorithm package module;
  the
 questions is: does
   import std.algorithm : joiner;
 still work if std.algorithm is actually a package module and 
 joiner is in
 a std.algorithm.joiner module?


Intended to work by matching DIP.
Dec 18 2013
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 18 December 2013 at 23:51:06 UTC, Meta wrote:

 Of course, another large boon would be to correct the 
 implementation of:

 import std.algorithm: sort, find, splitter, swap;

 So that it actually works in a sane way as opposed to pulling 
 in everything.


I don't see how it is even theoretically possible. You still need 
to lex/parse whole file to locate needed symbols (and semantic 
phases are already lazy in most cases afaik)
Dec 18 2013
parent reply "Meta" <jared771 gmail.com> writes:
On Thursday, 19 December 2013 at 00:23:47 UTC, Dicebot wrote:

 On Wednesday, 18 December 2013 at 23:51:06 UTC, Meta wrote:

 Of course, another large boon would be to correct the 
 implementation of:

 import std.algorithm: sort, find, splitter, swap;

 So that it actually works in a sane way as opposed to pulling 
 in everything.


 I don't see how it is even theoretically possible. You still 
 need to lex/parse whole file to locate needed symbols (and 
 semantic phases are already lazy in most cases afaik)


Maybe it's not possible, I'm not well versed on how the compiler 
works. But I think that one module per algorithm may be too 
granular. Hasn't someone suggested splitting it up by category, 
e.g., sorting, mutation, searching, etc.?
Dec 18 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-19 08:26, Meta wrote:


 Maybe it's not possible, I'm not well versed on how the compiler works.
 But I think that one module per algorithm may be too granular. Hasn't
 someone suggested splitting it up by category, e.g., sorting, mutation,
 searching, etc.?


Sounds more reasonable.

-- 
/Jacob Carlborg
Dec 19 2013
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 19/12/13 13:55, Jacob Carlborg wrote:

 Sounds more reasonable.


Is there any reason why one couldn't have subpackages of packages, and so allow 
imports to be as fine-grained as the user wants?

   std.algorithm -> std.algorithm.sorting -> std.algorithm.sorting.quickSort ...
Dec 19 2013
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 19 December 2013 at 13:20:30 UTC, Joseph Rushton 
Wakeling wrote:

 Is there any reason why one couldn't have subpackages of 
 packages, and so allow imports to be as fine-grained as the 
 user wants?


On the other hand, going too far might backfire, as more modules 
might lead to additional bloat with moduleinfo in the exe and 
more compile time cuz of more file loading/seeking.

I don't know if this will matter or not, but something we should 
test to know for sure.
Dec 19 2013
prev sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 19 December 2013 at 07:26:54 UTC, Meta wrote:

 Maybe it's not possible, I'm not well versed on how the 
 compiler works. But I think that one module per algorithm may 
 be too granular. Hasn't someone suggested splitting it up by 
 category, e.g., sorting, mutation, searching, etc.?


Previous discussions about this have come up with just using the 
categories already present in std.algorithm's documentation so 
it'd be:

std.algorithm.searching
std.algorithm.iteration
std.algorithm.sorting
std.algorithm.set
std.algorithm.mutation

Module tenses and exact naming could change, of course, but just 
those categories would break it up pretty evenly while keeping 
similar things close together.

Jonathan could probably give some guidance here because as far as 
I know he's the only person that has any experience attempting to 
break up a large phobos module into a package (std.datetime). I'm 
not sure what he has left or what blockers remain for finishing 
that up.
Dec 19 2013
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/19/13, Meta <jared771 gmail.com> wrote:

 For example, when you want to use sort, find, splitter and swap
 all together, you either have the choice of doing:

 import std.algorithm;

 And pull in everything, or doing:

 import std.algorithm.sort, std.algorithm.find,
 std.algorithm.splitter, std.algorithm.swap;

 Or is there something I'm missing here?


Hopefully we would add a feature that implements selective module imports, e.g.:

// either a symbol in the algorithm.d/package.d module or
// another module in the algorithm package *if* std.algorithm is a package.
import std.algorithm : sort;

I think there's a bugzilla enhancement for this.
Dec 19 2013
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 19/12/13 00:08, Walter Bright wrote:

 4. Break kitchen sink modules like std.algorithm into one module per algorithm.
 This should not result in code duplication.


That might be desirable for other purposes anyway (std.algorithm is big!), but 
how does this differ from the existing possibility of doing, e.g.,

     import std.algorithm : sort;
Dec 19 2013
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 19 Dec 2013 10:56:19 +0100
schrieb Joseph Rushton Wakeling <joseph.wakeling webdrake.net>:


 On 19/12/13 00:08, Walter Bright wrote:

 4. Break kitchen sink modules like std.algorithm into one module per algorithm.
 This should not result in code duplication.


 
 That might be desirable for other purposes anyway (std.algorithm is big!), but 
 how does this differ from the existing possibility of doing, e.g.,
 
      import std.algorithm : sort;


Selective imports are even slower than normal imports. You
still have to load all of std.algorithm to find and
disambiguate the symbol. Only when you split up the file
physically it will reduce the compiler's workload.

-- 
Marco
Dec 19 2013
prev sibling next sibling parent =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On 2013-12-19 10:56, Joseph Rushton Wakeling wrote:

 On 19/12/13 00:08, Walter Bright wrote:

 4. Break kitchen sink modules like std.algorithm into one module per
 algorithm.
 This should not result in code duplication.


 That might be desirable for other purposes anyway (std.algorithm is
 big!), but how does this differ from the existing possibility of doing,
 e.g.,

      import std.algorithm : sort;


Your line of code imports all of std.algorithm, as well as all modules 
imported by std.algorithm, etc. This includes functions used by topN, 
largestPartialIntersectionWeighted, findSplit, and so on.

A modularized std.algorithm.sort would only import what sort needs (and 
what sort's needs need, and so on, but likely a much small graph).

"But why can't we make the compiler smrter, so import std.algorithm : 
sort; only imports what's needed?" you might ask. I believe it's already 
been covered in this thread: overloads, for one (knowledge of all 
overloads is required). Second, if an imported type is used, how should 
the compiler guess which module contains the type? There may be more 
reasons, too.

--
   Simen
Dec 19 2013
prev sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 19/12/13 12:33, Simen Kjærås wrote:

 Your line of code imports all of std.algorithm, as well as all modules imported
 by std.algorithm, etc. This includes functions used by topN,
 largestPartialIntersectionWeighted, findSplit, and so on.


Ah, OK.  Thanks for the clarification.  A shame, but understandable.
Dec 19 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei 
Alexandrescu wrote:

 1. Improve the compiler to handle imports lazily, i.e. an 
 unused import is never opened. That's unlikely to help a lot of 
 uses because most unqualified name lookups require all imports 
 to be loaded (even after the name if resolved, the compiler 
 must still look for ambiguities).

 2. Push imports from top level into the entities (functions, 
 classes etc) that use them.

 3. Apply classic dependency management (break larger modules in 
 smaller ones, accept some code duplication etc).

 I favor (2).


 Andrei


2 + 3 (breaking modules into packages part) should do the tricky 
as far as I can see. Lot of current dependency bloat comes from 
importing whole std.range / std.algorithm to use one or two 
utility functions in few places.

On topic of (2) I propose to add it to Phobos style guide and 
start enforcing during PR review.
Dec 18 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-18 22:40, Andrei Alexandrescu wrote:


 There are several directions we can take this.

 1. Improve the compiler to handle imports lazily, i.e. an unused import
 is never opened. That's unlikely to help a lot of uses because most
 unqualified name lookups require all imports to be loaded (even after
 the name if resolved, the compiler must still look for ambiguities).

 2. Push imports from top level into the entities (functions, classes
 etc) that use them.

 3. Apply classic dependency management (break larger modules in smaller
 ones, accept some code duplication etc).

 I favor (2).


I would guess that not a single of these solutions alone would solve the 
problem. Most likely all will be needed.

-- 
/Jacob Carlborg
Dec 18 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 18/12/13 22:40, Andrei Alexandrescu wrote:

 2. Push imports from top level into the entities (functions, classes etc) that
 use them.


This is probably going to be very productive.  Example: a while back we had a 
discussion about how std.stdio wound up pulling in std.complex.  John Colvin 
worked out that the module dependency goes something like this:

      std.stdio -> std.algorithm -> std.random -> std.numeric -> std.complex

... so I looked a bit deeper and found that there are at least 2 points where 
that dependency chain could be broken:

    * std.algorithm relies on std.random I think only one actual function,
namely
      topN (where it calls std.random.uniform).  All other uses of std.random
are
      inside unittests.
         (Incidentally it's a bit bad that topN makes use of randomness but
          does not AFAICS allow you to specify a RNG.)

    * std.numeric relies on std.complex for quite a few functions and objects,
      but by no means all, and I doubt that std.random's dependency on
      std.numeric makes use of any of those.  The problem here is that those
      functions need awareness of std.complex.Complex at the module level, e.g.
      because they have Complex return-types.  So, it might be productive to
      separate out std.numeric based on std.complex dependency.

I can provide a patch for std.random very easily (I already have it
un-committed 
on my system right now as a result of re-running the above analysis).

I also think it should be policy for Phobos that if an import is required only 
for unittests, it must be imported in those individual unittests, not for the 
module as a whole (and not in a version(unittest) block, which would disguise 
the problem and cause failures between unittest release builds vs. release
builds).
Dec 19 2013
prev sibling next sibling parent "Idan Arye" <GenericNPC gmail.com> writes:
On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei 
Alexandrescu wrote:

 On 12/18/13 1:16 PM, Dmitry Olshansky wrote:

 19-Dec-2013 01:06, Andrei Alexandrescu пишет:

 http://chopapp.com/#fvepfd8 shows the number of dependencies 
 (plus 1)
 for each module in phobos. Those include druntime 
 dependencies.


 So the bill of using pretty much anything in Phobos is pulling 
 in 87
 modules. Pretty much what I feared it is.


 There are several directions we can take this.

 1. Improve the compiler to handle imports lazily, i.e. an 
 unused import is never opened. That's unlikely to help a lot of 
 uses because most unqualified name lookups require all imports 
 to be loaded (even after the name if resolved, the compiler 
 must still look for ambiguities).

 2. Push imports from top level into the entities (functions, 
 classes etc) that use them.

 3. Apply classic dependency management (break larger modules in 
 smaller ones, accept some code duplication etc).

 I favor (2).


 Andrei


I suggested 2 a half a year ago - 
http://forum.dlang.org/thread/rrjaopnvapwhbatlsyvo forum.dlang.org

One of the 
replys(http://forum.dlang.org/thread/rrjaopnvapwhbatlsyvo forum.dlang.org#post-heaeasepzokrwdsvytb
:40forum.dlang.org) 
mentioned an 
issue(https://d.puremagic.com/issues/show_bug.cgi?id=7016) that 
blocks this solution.
Dec 19 2013
prev sibling parent reply "Meta" <jared771 gmail.com> writes:
On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei 
Alexandrescu wrote:

 2. Push imports from top level into the entities (functions, 
 classes etc) that use them.


2 comes with its own problems, though. With imports at the top, 
you can easily tell what the module imports at a glance. Putting 
imports into the deepest possible scopes where they are used will 
make it a huge chore to hunt all the imports down. You can of 
course grep for "import" (or ctrl+f for more plebeian editors), 
but you still don't have that singular list in one place that you 
can easily look at to tell what the module imports.

Also, as was mentioned before, some imports (or lack of) will go 
unnoticed until the particular template or template path that it 
resides in gets instantiated. I don't relish the idea of thinking 
that my code is all clear, then getting some missing dependency 
error down the line after refactoring. This will make code more 
brittle.
Dec 20 2013
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/20/13, Meta <jared771 gmail.com> wrote:

 2 comes with its own problems, though. With imports at the top,
 you can easily tell what the module imports at a glance. Putting
 imports into the deepest possible scopes where they are used will
 make it a huge chore to hunt all the imports down.


Why is that important though? You only need to worry about these
scoped imports if you actually use the code that has the scoped
imports, rather than preemptively worry about imports at the top.


 I don't relish the idea of thinking
 that my code is all clear, then getting some missing dependency
 error down the line after refactoring. This will make code more
 brittle.


If you don't unittest your templates (which would catch these types of
errors), then you have much more to worry about.
Dec 20 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/13 1:06 AM, Meta wrote:

 On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei Alexandrescu wrote:

 2. Push imports from top level into the entities (functions, classes
 etc) that use them.


 2 comes with its own problems, though. With imports at the top, you can
 easily tell what the module imports at a glance. Putting imports into
 the deepest possible scopes where they are used will make it a huge
 chore to hunt all the imports down. You can of course grep for "import"
 (or ctrl+f for more plebeian editors), but you still don't have that
 singular list in one place that you can easily look at to tell what the
 module imports.


I think that's a price we need to pay the same way cars replaced 
carriages in spite of them having a lot of disadvantages at the time. 
It's simply a mindset to do away with.

Transitive imports can be figured with:

dmd -c -o- -v module | grep '^import '

I felt the need for non-transitive imports, and I'm sure a tool will 
show up before long.

Local imports are the future. Better get used to them.


 Also, as was mentioned before, some imports (or lack of) will go
 unnoticed until the particular template or template path that it resides
 in gets instantiated. I don't relish the idea of thinking that my code
 is all clear, then getting some missing dependency error down the line
 after refactoring. This will make code more brittle.


Unittests/coverage to the rescue. Tooling tooling tooling.


Andrei
Dec 20 2013
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/20/13, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:

 Local imports are the future. Better get used to them.


To be fair, it's modularization that is the future. And that implies
modules that don't stand tall at 8K, 16K, 32K lines+. If the modules
have less code and the code is limited in scope then you automatically
need less imports.
Dec 20 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-20 10:37, Andrei Alexandrescu wrote:


 I think that's a price we need to pay the same way cars replaced
 carriages in spite of them having a lot of disadvantages at the time.
 It's simply a mindset to do away with.

 Transitive imports can be figured with:

 dmd -c -o- -v module | grep '^import '

 I felt the need for non-transitive imports, and I'm sure a tool will
 show up before long.


I agree. It would be nice if an IDE could show a list of all imports, 
including locals, of a module.

-- 
/Jacob Carlborg
Dec 20 2013
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 20/12/13 10:06, Meta wrote:

 2 comes with its own problems, though. With imports at the top, you can easily
 tell what the module imports at a glance. Putting imports into the deepest
 possible scopes where they are used will make it a huge chore to hunt all the
 imports down. You can of course grep for "import" (or ctrl+f for more plebeian
 editors), but you still don't have that singular list in one place that you can
 easily look at to tell what the module imports.


Conversely, one problem of imports at the top is that it's easy for unnecessary 
imports to hang around just because you don't have a direct association between 
the import and what it's actually used for.

I think the real issue we face is that it's not possible to really gain the 
advantages of deeply-nested-as-possible imports without _also_ breaking up the 
modules; we can't have Andrei's strategy (2) without some of strategy (3). 
Example: std.algorithm.topN, the only part of std.algorithm (apart from 
unittests) that requires std.random:

     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range)(Range r, size_t nth)
         if (isRandomAccessRange!(Range) && hasLength!Range)
     {
         static assert(ss == SwapStrategy.unstable,
                 "Stable topN not yet implemented");
         while (r.length > nth)
         {
             auto pivot = uniform(0, r.length);
             // ... etc. ...
         }
     }

Now, at first glance it's easy to just insert an extra line before "auto pivot
= 
...":

     import std.random : uniform;

... but it quickly becomes non-trivial if you want to do what really ought to
be 
an option here, and allow a non-default RNG to be passed to the function:

     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
         if (isRandomAccessRange!(Range) && hasLength!Range
             && isUniformRNG!RandomGen)  // <--- needs std.random.isUniformRNG
     {
         static assert(ss == SwapStrategy.unstable,
                 "Stable topN not yet implemented");
         while (r.length > nth)
         {
             auto pivot = uniform(0, r.length, rng);
             // ... etc. ...
         }
     }

     // New function to support old 2-parameter version using default RNG
     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range)(Range r, size_t nth)
         if (isRandomAccessRange!(Range) && hasLength!Range)
     {
         topN(r, ss, rndGen);    <---- needs std.random.rndGen;
     }

The second of the two needed imports is easy and can be nested inside the 
function, but the first -- the isUniformRNG template -- requires the import in 
the module's own scope.

So, to really allow std.algorithm to do away with its dependency on std.random, 
you need to break isUniformRNG and similar templates away from the rest of the 
std.random functionality.

(Yes, I know, you could break topN out from the rest of std.random, but
remember 
that topN itself is only dependent on std.random for the case where you're
using 
the unstable swap strategy.)

I'd hypothesize that probably it would be productive to start modularizing 
Phobos modules by separating out all the various helper templates like 
isUniformRNG, after which it should be much easier to avoid needing top-level 
module imports.
Dec 20 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/13 5:45 AM, Joseph Rushton Wakeling wrote:

 On 20/12/13 10:06, Meta wrote:
      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && isUniformRNG!RandomGen)  // <--- needs
 std.random.isUniformRNG
      {
          static assert(ss == SwapStrategy.unstable,
                  "Stable topN not yet implemented");
          while (r.length > nth)
          {
              auto pivot = uniform(0, r.length, rng);
              // ... etc. ...
          }
      }


I had this idea fot a while, and Walter is favorable of it as well - 
extend "import" for one-shot use. With that feature the example would 
become:

     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
         if (isRandomAccessRange!(Range) && hasLength!Range
             && import.std.random.isUniformRNG!RandomGen)
     { ... }

In this case "import" would syntactically be placed at the beginning of 
a qualified name, meaning "import this module lazily and look up the 
symbol in it".

This would simplify quite a lot of two-liners into one-liners in other 
places, too.


Andrei
Dec 20 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 09:27:55AM -0800, Andrei Alexandrescu wrote:

 On 12/20/13 5:45 AM, Joseph Rushton Wakeling wrote:

On 20/12/13 10:06, Meta wrote:
     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
         if (isRandomAccessRange!(Range) && hasLength!Range
             && isUniformRNG!RandomGen)  // <--- needs
std.random.isUniformRNG
     {
         static assert(ss == SwapStrategy.unstable,
                 "Stable topN not yet implemented");
         while (r.length > nth)
         {
             auto pivot = uniform(0, r.length, rng);
             // ... etc. ...
         }
     }


 
 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example
 would become:
 
     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
         if (isRandomAccessRange!(Range) && hasLength!Range
             && import.std.random.isUniformRNG!RandomGen)
     { ... }
 
 In this case "import" would syntactically be placed at the beginning
 of a qualified name, meaning "import this module lazily and look up
 the symbol in it".


[...]

Hmm. Why do we need to incorporate the 'import' keyword in the first
place? What about extending symbol lookup, so that if a fully-qualified
symbol x.y.z can't be found in the current symbol tables, and x/y exists
in the current import path, then implicitly try to import x.y and lookup
z in that module. Then you could just write:

	void f(T)(T t) if (std.range.isInputRange!T) ...

and the compiler will automatically import std.range within that scope.
Obviously, it's a bad idea to import std.range into module scope, since
it would pollute the module namespace, but it seems a good idea to do a
one-shot import automatically, since the qualified symbol itself already
says which module the symbol is supposed to be defined in. There's no
need to add another "import." prefix to it, IMO.


T

-- 
Today's society is one of specialization: as you grow, you learn more and more
about less and less. Eventually, you know everything about nothing.
Dec 20 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/13 9:38 AM, H. S. Teoh wrote:

 Hmm. Why do we need to incorporate the 'import' keyword in the first
 place?


To make the intent explicit. But that's an interesting idea. I'd 
probably require a leading '.' just to make it unambiguous that lookup 
must start from top level.


 What about extending symbol lookup, so that if a fully-qualified
 symbol x.y.z can't be found in the current symbol tables, and x/y exists
 in the current import path, then implicitly try to import x.y and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...


Due to a bug that's actually the case today to some extent :o).


Andrei
Dec 20 2013
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 09:57:46AM -0800, Andrei Alexandrescu wrote:

 On 12/20/13 9:38 AM, H. S. Teoh wrote:

Hmm. Why do we need to incorporate the 'import' keyword in the first
place?


 
 To make the intent explicit. But that's an interesting idea. I'd
 probably require a leading '.' just to make it unambiguous that
 lookup must start from top level.
 

What about extending symbol lookup, so that if a fully-qualified
symbol x.y.z can't be found in the current symbol tables, and x/y
exists in the current import path, then implicitly try to import x.y
and lookup z in that module. Then you could just write:

	void f(T)(T t) if (std.range.isInputRange!T) ...


 
 Due to a bug that's actually the case today to some extent :o).


[...]

"That's not a bug, that's an unintentional feature!" :-P


T

-- 
By understanding a machine-oriented language, the programmer will tend
to use a much more efficient method; it is much closer to reality. -- D.
Knuth
Dec 20 2013
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/20/2013 9:57 AM, Andrei Alexandrescu wrote:

 On 12/20/13 9:38 AM, H. S. Teoh wrote:

 Hmm. Why do we need to incorporate the 'import' keyword in the first
 place?


 To make the intent explicit. But that's an interesting idea. I'd probably
 require a leading '.' just to make it unambiguous that lookup must start from
 top level.


Yes, it's interesting, but I think it's a bit too clever :-)

I think importing should be explicit, not implicit.
Dec 20 2013
prev sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 20 December 2013 at 17:57:46 UTC, Andrei Alexandrescu 
wrote:

 On 12/20/13 9:38 AM, H. S. Teoh wrote:

 What about extending symbol lookup, so that if a 
 fully-qualified
 symbol x.y.z can't be found in the current symbol tables, and 
 x/y exists
 in the current import path, then implicitly try to import x.y 
 and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...


 Due to a bug that's actually the case today to some extent :o).


 Andrei


In regards to template restraints, it *would* be kind of nice to 
be able to trigger import only when overload resolution begins.

For example, just because a function requires 
"std.range.isInputRange" to validate its inputs/overloads, 
doesn't mean it *has* to be globally imported into the entire 
module.

In this sense, it would be nice to be able to define an "import" 
block for such a function:
//----
module wow;

void bar(T)(T t)
if (isInputRange!T)
import
{
     std.range;
}
body
{
     ....
}

void foo()
{
     ....
}

//----

With such an approach, "std.range" only gets imported if a call 
is "attempted" to bar. if no such call is even attempted, then 
range isn't imported at all.

In this example, "foo" simply does not know about range. It is 
scoped to bar.

As a client, *if* I make no calls to bar, then I will not trigger 
a dependency to range in wow at all.
Dec 20 2013
prev sibling next sibling parent reply captaindet <2krnk gmx.net> writes:
On 2013-12-20 11:38, H. S. Teoh wrote:

 On Fri, Dec 20, 2013 at 09:27:55AM -0800, Andrei Alexandrescu wrote:

 On 12/20/13 5:45 AM, Joseph Rushton Wakeling wrote:

 On 20/12/13 10:06, Meta wrote:
      void topN(alias less = "a<  b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range)&&  hasLength!Range
              &&  isUniformRNG!RandomGen)  //<--- needs
 std.random.isUniformRNG
      {
          static assert(ss == SwapStrategy.unstable,
                  "Stable topN not yet implemented");
          while (r.length>  nth)
          {
              auto pivot = uniform(0, r.length, rng);
              // ... etc. ...
          }
      }


 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example
 would become:

      void topN(alias less = "a<  b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range)&&  hasLength!Range
              &&  import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning
 of a qualified name, meaning "import this module lazily and look up
 the symbol in it".


 [...]

 Hmm. Why do we need to incorporate the 'import' keyword in the first
 place? What about extending symbol lookup, so that if a fully-qualified
 symbol x.y.z can't be found in the current symbol tables, and x/y exists
 in the current import path, then implicitly try to import x.y and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...

 and the compiler will automatically import std.range within that scope.
 Obviously, it's a bad idea to import std.range into module scope, since
 it would pollute the module namespace, but it seems a good idea to do a
 one-shot import automatically, since the qualified symbol itself already
 says which module the symbol is supposed to be defined in. There's no
 need to add another "import." prefix to it, IMO.


 T


one could make it a bit more explicit by requiring

	import std;

at module level, or even more explicit

	auto import std;

to enable lazy import of everything that is hierarchically below. could then be
used for user libraries as well:

	import mybreadandbutter;		// has several sub folders / packages

/det
Dec 20 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 01:17:29PM -0600, captaindet wrote:

 On 2013-12-20 11:38, H. S. Teoh wrote:


[...]

Hmm. Why do we need to incorporate the 'import' keyword in the first
place? What about extending symbol lookup, so that if a
fully-qualified symbol x.y.z can't be found in the current symbol
tables, and x/y exists in the current import path, then implicitly
try to import x.y and lookup z in that module. Then you could just
write:

	void f(T)(T t) if (std.range.isInputRange!T) ...

and the compiler will automatically import std.range within that
scope.  Obviously, it's a bad idea to import std.range into module
scope, since it would pollute the module namespace, but it seems a
good idea to do a one-shot import automatically, since the qualified
symbol itself already says which module the symbol is supposed to be
defined in. There's no need to add another "import." prefix to it,
IMO.


T


 
 one could make it a bit more explicit by requiring
 
 	import std;
 
 at module level, or even more explicit
 
 	auto import std;
 
 to enable lazy import of everything that is hierarchically below.
 could then be used for user libraries as well:


[...]

I like this idea. It could also be denoted as 'lazy import std', that
is, if something references 'std.xyz.abc', then import the module
std.xyz and lookup the symbol 'abc'. Then you could just use
std.range.isInputRange in your signature constraints, and the import
will only happen if you actually instantiate that template.

A more restricted variant, which may be more palatable to Walter, is to
require explicit modules, i.e., you can't just say 'lazy import std',
but you have to say 'lazy import std.range':

	lazy import std.range;

	void func(T)(T t)
		if (std.range.isInputRange!T)
	{
		...
	}

So the compiler at least knows that 'std.range' refers to some as-yet
unimported module, then when you actually reference a symbol under
std.range, the compiler will go and import the module to find its
definition.


T

-- 
IBM = I'll Buy Microsoft!
Dec 20 2013
prev sibling parent reply "Patrick Down" <patrick.down gmail.com> writes:
On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh wrote:

 in the current import path, then implicitly try to import x.y 
 and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...

 and the compiler will automatically import std.range within 
 that scope.


How about:

scope import std.range;
// lazy import std.range; ?

void f(T)(T t) if (std.range.isInputRange!T) ...
Dec 20 2013
parent reply "Meta" <jared771 gmail.com> writes:
On Friday, 20 December 2013 at 19:34:10 UTC, Patrick Down wrote:

 On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh wrote:

 in the current import path, then implicitly try to import x.y 
 and lookup
 z in that module. Then you could just write:

 	void f(T)(T t) if (std.range.isInputRange!T) ...

 and the compiler will automatically import std.range within 
 that scope.


 How about:

 scope import std.range;
 // lazy import std.range; ?

 void f(T)(T t) if (std.range.isInputRange!T) ...


I think the best keyword to use in this situation would be 
stati... Oh, dammit, not again.
Dec 20 2013
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-12-20 19:36:28 +0000, "Meta" <jared771 gmail.com> said:


 On Friday, 20 December 2013 at 19:34:10 UTC, Patrick Down wrote:

 On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh wrote:

 in the current import path, then implicitly try to import x.y and lookup
 z in that module. Then you could just write:
 
 	void f(T)(T t) if (std.range.isInputRange!T) ...
 
 and the compiler will automatically import std.range within that scope.


 
 How about:
 
 scope import std.range;
 // lazy import std.range; ?
 
 void f(T)(T t) if (std.range.isInputRange!T) ...


 
 I think the best keyword to use in this situation would be stati... Oh, 
 dammit, not again.


Actually, "static import" already exists. And semantically it's pretty 
much the same thing as the above: you have to use the symbol's full 
name. But currently the compiler will import eagerly. I doubt there'd 
be any breakage if "static" changed to mean "lazily imported".

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca
Dec 20 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 02:40:22PM -0500, Michel Fortin wrote:

 On 2013-12-20 19:36:28 +0000, "Meta" <jared771 gmail.com> said:
 

On Friday, 20 December 2013 at 19:34:10 UTC, Patrick Down wrote:

On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh wrote:

in the current import path, then implicitly try to import x.y and
lookup z in that module. Then you could just write:

	void f(T)(T t) if (std.range.isInputRange!T) ...

and the compiler will automatically import std.range within that
scope.


How about:

scope import std.range;
// lazy import std.range; ?

void f(T)(T t) if (std.range.isInputRange!T) ...


I think the best keyword to use in this situation would be
stati... Oh, dammit, not again.


 
 Actually, "static import" already exists. And semantically it's
 pretty much the same thing as the above: you have to use the
 symbol's full name. But currently the compiler will import eagerly.
 I doubt there'd be any breakage if "static" changed to mean "lazily
 imported".


[...]

Hmm. In that case, looks like we already have the solution. :)


T

-- 
"Real programmers can write assembly code in any language. :-)" -- Larry Wall
Dec 20 2013
parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 20 December 2013 at 19:51:03 UTC, H. S. Teoh wrote:

 On Fri, Dec 20, 2013 at 02:40:22PM -0500, Michel Fortin wrote:

 On 2013-12-20 19:36:28 +0000, "Meta" <jared771 gmail.com> said:
 

On Friday, 20 December 2013 at 19:34:10 UTC, Patrick Down 
wrote:

On Friday, 20 December 2013 at 17:40:08 UTC, H. S. Teoh 
wrote:

in the current import path, then implicitly try to import 
x.y and
lookup z in that module. Then you could just write:

	void f(T)(T t) if (std.range.isInputRange!T) ...

and the compiler will automatically import std.range within 
that
scope.


How about:

scope import std.range;
// lazy import std.range; ?

void f(T)(T t) if (std.range.isInputRange!T) ...


I think the best keyword to use in this situation would be
stati... Oh, dammit, not again.


 
 Actually, "static import" already exists. And semantically it's
 pretty much the same thing as the above: you have to use the
 symbol's full name. But currently the compiler will import 
 eagerly.
 I doubt there'd be any breakage if "static" changed to mean 
 "lazily
 imported".


 [...]

 Hmm. In that case, looks like we already have the solution. :)


 T


Yes, that's actually quite smart. I like it.
+1.
Dec 20 2013
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On 12/20/2013 08:40 PM, Michel Fortin wrote:

 Actually, "static import" already exists. And semantically it's pretty
 much the same thing as the above: you have to use the symbol's full
 name. But currently the compiler will import eagerly. I doubt there'd be
 any breakage if "static" changed to mean "lazily imported".


Ah, same idea :).
Dec 20 2013
prev sibling parent "Meta" <jared771 gmail.com> writes:
On Friday, 20 December 2013 at 19:40:22 UTC, Michel Fortin wrote:

 Actually, "static import" already exists. And semantically it's 
 pretty much the same thing as the above: you have to use the 
 symbol's full name. But currently the compiler will import 
 eagerly. I doubt there'd be any breakage if "static" changed to 
 mean "lazily imported".


Just a joke =)
Dec 20 2013
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-12-20 17:27:55 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 I had this idea fot a while, and Walter is favorable of it as well - 
 extend "import" for one-shot use. With that feature the example would 
 become:
 
      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }
 
 In this case "import" would syntactically be placed at the beginning of 
 a qualified name, meaning "import this module lazily and look up the 
 symbol in it".
 
 This would simplify quite a lot of two-liners into one-liners in other 
 places, too.


How do you solve the problem that in D you can't tell the module name 
from a fully qualified names? For instance:

	import.std.something.somethingelse.anotherthing = 1;

Is the module std? std.something? std.something.somethingelse? It could 
be any of these answers.

	module std.something;
	struct somethingelse { static int anotherthing = 0; }

or

	module std.something.somethingelse;
	int anotherthing = 0;


-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca
Dec 20 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 02:20:12PM -0500, Michel Fortin wrote:

 On 2013-12-20 17:27:55 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:
 

I had this idea fot a while, and Walter is favorable of it as well
- extend "import" for one-shot use. With that feature the example
would become:

     void topN(alias less = "a < b",
             SwapStrategy ss = SwapStrategy.unstable,
             Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
         if (isRandomAccessRange!(Range) && hasLength!Range
             && import.std.random.isUniformRNG!RandomGen)
     { ... }

In this case "import" would syntactically be placed at the
beginning of a qualified name, meaning "import this module lazily
and look up the symbol in it".

This would simplify quite a lot of two-liners into one-liners in
other places, too.


 
 How do you solve the problem that in D you can't tell the module
 name from a fully qualified names? For instance:
 
 	import.std.something.somethingelse.anotherthing = 1;
 
 Is the module std? std.something? std.something.somethingelse? It
 could be any of these answers.


[...]

Using captaindet's idea of lazy imports, we could solve this problem:

	// Tell compiler that 'std.range' is a module, but don't import
	// it just yet.
	lazy import std.range;

	void func() {
		// Since the compiler already knows that 'std.range' is
		// a module, it knows to now import std.range and look
		// up 'InputRange.front' in it.
		auto x = std.range.InputRange.front;
	}


T

-- 
EMACS = Extremely Massive And Cumbersome System
Dec 20 2013
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 12/20/2013 06:27 PM, Andrei Alexandrescu wrote:

 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example would
 become:

      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning of
 a qualified name, meaning "import this module lazily and look up the
 symbol in it".

 This would simplify quite a lot of two-liners into one-liners in other
 places, too.


The fact that template constraints use the module scope is indeed a root 
cause for a lot of module dependencies, so we should tackle this problem 
specifically.

Couldn't static imports be made lazy without breaking any code?
The above example would read.

static import std.range.

void foo(R)(R range) if (std.range.isForwardRange!R)
{
}
Dec 20 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/13 12:43 PM, Martin Nowak wrote:

 Couldn't static imports be made lazy without breaking any code?


One simple idea is to make all imports lazy - no change in semantics.

Andrei
Dec 20 2013
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 20 December 2013 at 21:36:15 UTC, Andrei Alexandrescu 
wrote:

 On 12/20/13 12:43 PM, Martin Nowak wrote:

 Couldn't static imports be made lazy without breaking any code?


 One simple idea is to make all imports lazy - no change in 
 semantics.

 Andrei


If the import is lazy, how do you know which import to load when 
you see an unknown symbol? lazy static import works, because they 
have to be fully qualified.
Dec 20 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 10:51:13PM +0100, monarch_dodra wrote:

 On Friday, 20 December 2013 at 21:36:15 UTC, Andrei Alexandrescu
 wrote:

On 12/20/13 12:43 PM, Martin Nowak wrote:

Couldn't static imports be made lazy without breaking any code?


One simple idea is to make all imports lazy - no change in
semantics.

Andrei


 
 If the import is lazy, how do you know which import to load when you
 see an unknown symbol? lazy static import works, because they have
 to be fully qualified.


Yeah, much as I like the idea of all imports being lazy by default, I
see a lot of potential problems in the implementation. If you have 10
lazy imports and you reference an unknown symbol, will the compiler now
go and import all 10 modules just so it can search them for the unknown
symbol?  Sounds like recipe for disaster.


T

-- 
It only takes one twig to burn down a forest.
Dec 20 2013
prev sibling parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2013-12-20 21:36:15 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 On 12/20/13 12:43 PM, Martin Nowak wrote:

 Couldn't static imports be made lazy without breaking any code?


 
 One simple idea is to make all imports lazy - no change in semantics.


Sure, why not. Except that upon reaching the first non-fully-qualified 
symbol you'd have to load them all because they could all contain a 
symbol with that name. So in practice it'd change nothing, except for 
static imports.

Beside that, loading modules eagerly is more parallelizable.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca
Dec 20 2013
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
21-Dec-2013 00:43, Martin Nowak пишет:

 On 12/20/2013 06:27 PM, Andrei Alexandrescu wrote:

 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example would
 become:

      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning of
 a qualified name, meaning "import this module lazily and look up the
 symbol in it".

 This would simplify quite a lot of two-liners into one-liners in other
 places, too.


 The fact that template constraints use the module scope is indeed a root
 cause for a lot of module dependencies, so we should tackle this problem
 specifically.

 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


That has the disadvantage of importing the whole std.range.
I seriously doubt that we'd get anything better then:


 import std.range.traits;

 void foo(R)(R range) if (isForwardRange!R)
 {


        import std.range;
...

 }


Stays practical w.r.t. cutting down dependencies and no need to uglify 
constraints. They are not that readable already.

-- 
Dmitry Olshansky
Dec 21 2013
next sibling parent "Chris Cain" <clcain uncg.edu> writes:
On Saturday, 21 December 2013 at 08:08:42 UTC, Dmitry Olshansky 
wrote:

 That has the disadvantage of importing the whole std.range.
 I seriously doubt that we'd get anything better then:


 import std.range.traits;

 void foo(R)(R range) if (isForwardRange!R)
 {


        import std.range;
 ...

 }


 Stays practical w.r.t. cutting down dependencies and no need to 
 uglify constraints. They are not that readable already.


I agree, to an extent. That's definitely the ultimate solution 
that should be taken. Smaller modules are better in general.

That said, supporting a lazy static import feature might not be a 
bad idea. Then a hybrid approach could be taken, which would help 
things in the short-run. Maybe doing something like this:

     static import std.range;
     alias isForwardRange = std.range.isForwardRange;

     void foo(R)(R range) if (isForwardRange!R)
     {
         ...
     }

That would be readable now and would support easy changes to the 
better packaged approach later. I'd think this would be something 
possible to do in 2.066 whereas splitting up everything will 
likely take several versions.
Dec 21 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 21 December 2013 at 08:08:42 UTC, Dmitry Olshansky 
wrote:

 That has the disadvantage of importing the whole std.range.
 I seriously doubt that we'd get anything better then:


 import std.range.traits;

 void foo(R)(R range) if (isForwardRange!R)
 {


        import std.range;
 ...

 }


 Stays practical w.r.t. cutting down dependencies and no need to 
 uglify constraints. They are not that readable already.


I think it was mostly agreed here that we should do both for best 
result.
Dec 21 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/21/13 12:08 AM, Dmitry Olshansky wrote:

 21-Dec-2013 00:43, Martin Nowak пишет:

 On 12/20/2013 06:27 PM, Andrei Alexandrescu wrote:

 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example would
 become:

      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning of
 a qualified name, meaning "import this module lazily and look up the
 symbol in it".

 This would simplify quite a lot of two-liners into one-liners in other
 places, too.


 The fact that template constraints use the module scope is indeed a root
 cause for a lot of module dependencies, so we should tackle this problem
 specifically.

 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


 That has the disadvantage of importing the whole std.range.


Yah but only if the symbol foo is actually used.

Andrei
Dec 21 2013
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
21-Dec-2013 21:10, Andrei Alexandrescu пишет:

 On 12/21/13 12:08 AM, Dmitry Olshansky wrote:

 21-Dec-2013 00:43, Martin Nowak пишет:





 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


 That has the disadvantage of importing the whole std.range.


 Yah but only if the symbol foo is actually used.


That assuming static import becomes lazy (if/when).

In such a case I'd be against the idom still if only because of extra 
verbosity in constraints - it's a place where we'd want to have less of it.

Second point is that even if import becomes lazy it's still 
analyze-the-whole-module at the first reference required.

No escaping the fact that both constraints and bodies of templates 
function need to use fine grained imports (more . The more can be 
shifted inside of the body, and the more specific it gets the better 
dependency management we have. The latter implies the well-known benefit 
of having less stuff to analyze, compile and link.

Thus I conclude that introducing lazy loading of symbols
accomplishes too little for the amount of changes it entails.

-- 
Dmitry Olshansky
Dec 21 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/21/13 1:06 PM, Dmitry Olshansky wrote:

 21-Dec-2013 21:10, Andrei Alexandrescu пишет:

 On 12/21/13 12:08 AM, Dmitry Olshansky wrote:

 21-Dec-2013 00:43, Martin Nowak пишет:





 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


 That has the disadvantage of importing the whole std.range.


 Yah but only if the symbol foo is actually used.


 That assuming static import becomes lazy (if/when).

 In such a case I'd be against the idom still if only because of extra
 verbosity in constraints - it's a place where we'd want to have less of it.


That's why I'm saying: make all imports lazy!!!!


Andrei
Dec 21 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 21 December 2013 at 21:16:23 UTC, Andrei 
Alexandrescu wrote:

 That's why I'm saying: make all imports lazy!!!!


How? It has been already mentioned in this thread that this does 
not seem possible, at least withing existing language.
Dec 21 2013
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/21/13, Dicebot <public dicebot.lv> wrote:

 On Saturday, 21 December 2013 at 21:16:23 UTC, Andrei
 Alexandrescu wrote:

 That's why I'm saying: make all imports lazy!!!!


 How? It has been already mentioned in this thread that this does
 not seem possible, at least withing existing language.


The first step is to avoid reading the imports at all *unless* there's
symbols missing. And then try to match selective imports first if
there are any missing symbols. To demonstrate, here's the first
test-case:

-----
module test;

import std.stdio : writeln;
import std.algorithm;

void main()
{
}
-----

The compiler *does not* need to import any other modules (other than
object.d, of course), because it doesn't find any missing symbols
referenced from "test.d".

Test-case 2:
-----
import std.stdio : writeln;
import std.algorithm;

void main()
{
    writeln("");
}
-----

The compiler matches the missing symbol with the selective import
"writeln", so it knows it only has to load std.stdio, *but not*
std.algorithm.

Test-case 3:
-----
import std.stdio : writeln;
import std.algorithm : map;
import std.range;  // might be here (compiler doesn't know)
import std.container;  // or here

void main()
{
    "foo".front;
}
-----

The compiler tries to find a selective import "front", but it doesn't
find it. What's important here is that it still does not have to load
std.stdio or std.algorithm, as we're explicitly loading only a set of
symbols which were never referenced from the test.d module.

So the next step here is for the compiler to try and load each module
in sequence (probably via the declaration order, first std.container),
and if there's a match the compiler would stop loading other modules
(no need to load std.container if std.range has "front").

-----

I could think of more optimizations, for example if we had a way of
exporting a list of module-level symbols into some kind of
intermediary format (say JSON), the compiler could look up this list
rather than to have to eagerly load every module in search of a
symbol. For example:

Test-case 4:
-----
import std.stdio;
import std.algorithm;
import std.range;
import std.container;

void main()
{
    "foo".front;
}
-----

If we had a "symbols.json", it might list things like: "front": {
std.range, std.stdio }, so the compiler would know only to look for
this symbol in these modules (and /if/ they are actually imported in
the test.d module).
Dec 21 2013
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 21 Dec 2013 22:43:58 +0100
schrieb Andrej Mitrovic <andrej.mitrovich gmail.com>:


 I could think of more optimizations, for example if we had a way of
 exporting a list of module-level symbols into some kind of
 intermediary format (say JSON), the compiler could look up this list
 rather than to have to eagerly load every module in search of a
 symbol.


We have .di files already, which could cut on the amount of
parsing required.

Other than that your idea for partially lazy imports looks like
a good first optimization to me. Selective imports always
override symbols of the same name in other imported modules,
right?

-- 
Marco
Dec 22 2013
prev sibling next sibling parent Robert Schadek <realburner gmx.de> writes:
On 12/21/2013 10:43 PM, Andrej Mitrovic wrote:

 The compiler matches the missing symbol with the selective import
 "writeln", so it knows it only has to load std.stdio, *but not*
 std.algorithm.

 Test-case 3:
 -----
 import std.stdio : writeln;
 import std.algorithm : map;
 import std.range;  // might be here (compiler doesn't know)
 import std.container;  // or here

 void main()
 {
     "foo".front;
 }
 -----

 The compiler tries to find a selective import "front", but it doesn't
 find it. What's important here is that it still does not have to load
 std.stdio or std.algorithm, as we're explicitly loading only a set of
 symbols which were never referenced from the test.d module.

 So the next step here is for the compiler to try and load each module
 in sequence (probably via the declaration order, first std.container),
 and if there's a match the compiler would stop loading other modules
 (no need to load std.container if std.range has "front").


There is a flaw in that design. What if std.range and std.container both
have a matching .front? If you stop after the first you'll not find that
conflict. If you search both you'll not have any speed gain.
Jan 07 2014
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 1/7/14, Robert Schadek <realburner gmx.de> wrote:

 There is a flaw in that design. What if std.range and std.container both
 have a matching .front? If you stop after the first you'll not find that
 conflict. If you search both you'll not have any speed gain.


Ah damn, you're right. It was too good to be true.
Jan 07 2014
parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 7 January 2014 at 12:02:32 UTC, Andrej Mitrovic wrote:

 On 1/7/14, Robert Schadek <realburner gmx.de> wrote:

 There is a flaw in that design. What if std.range and 
 std.container both
 have a matching .front? If you stop after the first you'll not 
 find that
 conflict. If you search both you'll not have any speed gain.


 Ah damn, you're right. It was too good to be true.


Still pretty good with static / selective imports, just no magic
;)
Jan 07 2014
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
22-Dec-2013 01:16, Andrei Alexandrescu пишет:

 On 12/21/13 1:06 PM, Dmitry Olshansky wrote:

 21-Dec-2013 21:10, Andrei Alexandrescu пишет:

 On 12/21/13 12:08 AM, Dmitry Olshansky wrote:

 21-Dec-2013 00:43, Martin Nowak пишет:





 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


 That has the disadvantage of importing the whole std.range.


 Yah but only if the symbol foo is actually used.


 That assuming static import becomes lazy (if/when).

 In such a case I'd be against the idom still if only because of extra
 verbosity in constraints - it's a place where we'd want to have less
 of it.


 That's why I'm saying: make all imports lazy!!!!


Unless language defines a way to tell apart and split off a group of 
declarations inside of a module as independent block laziness doesn't 
help any. The whole reason is to avoid analyzing the whole module and 
pulling in its globals. If lazy import can pull only pieces (per symbol 
dependencies) of module that are actually required - cool, but it's 
seems very distant possibility.

As it stands the only thing lazy buys us is "pay as you touch" contrary 
to "pay as you name the intent to touch". The problem is that the 
payment is for the whole stock of the said "shop". I see second problem 
(granularity of imports) as far more critical then the first (condition 
under which the pieces are imported). The second problem seems solvable 
within the current implementation, the first seems like it would need 
arbitrary amount of time to fix and gains are marginal.



-- 
Dmitry Olshansky
Dec 21 2013
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Saturday, 21 December 2013 at 21:38:59 UTC, Dmitry Olshansky 
wrote:

 As it stands the only thing lazy buys us is "pay as you touch" 
 contrary to "pay as you name the intent to touch". The problem 
 is that the payment is for the whole stock of the said "shop". 
 I see second problem (granularity of imports) as far more 
 critical then the first (condition under which the pieces are 
 imported). The second problem seems solvable within the current 
 implementation, the first seems like it would need arbitrary 
 amount of time to fix and gains are marginal.


We should also keep in mind that as we split up modules and split 
apart dependencies, it also means that *as* we import a specific 
package, we are increasing our "use/import" ratio, further 
diminishing the issue of "import things we don't need." (who 
would import "std.foo.bar.baz", if they weren't planning to use 
baz?).

Arguably, we'd get "quadratic" effectiveness ;)
Dec 21 2013
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
22-Dec-2013 02:16, monarch_dodra пишет:

 On Saturday, 21 December 2013 at 21:38:59 UTC, Dmitry Olshansky wrote:

 As it stands the only thing lazy buys us is "pay as you touch"
 contrary to "pay as you name the intent to touch". The problem is that
 the payment is for the whole stock of the said "shop". I see second
 problem (granularity of imports) as far more critical then the first
 (condition under which the pieces are imported). The second problem
 seems solvable within the current implementation, the first seems like
 it would need arbitrary amount of time to fix and gains are marginal.


 We should also keep in mind that as we split up modules and split apart
 dependencies, it also means that *as* we import a specific package, we
 are increasing our "use/import" ratio, further diminishing the issue of
 "import things we don't need." (who would import "std.foo.bar.baz", if
 they weren't planning to use baz?).


The main gain is to the library code be it Phobos itself or any 3rd 
party code. Application code is import-happy by definition and only few 
folks at this level would care to import things in such a fine garined 
manner.


 Arguably, we'd get "quadratic" effectiveness ;)


In mind the constant factor is increasing (as in more files, more 
syscalls) but as the total LOCs per package version of a module stays 
the same I fail to see any substantial efficiency loss.

-- 
Dmitry Olshansky
Dec 21 2013
prev sibling parent "Paolo Invernizzi" <paolo.invernizzi gmail.com> writes:
On Saturday, 21 December 2013 at 22:16:13 UTC, monarch_dodra 
wrote:

 We should also keep in mind that as we split up modules and 
 split apart dependencies, it also means that *as* we import a 
 specific package, we are increasing our "use/import" ratio, 
 further diminishing the issue of "import things we don't need." 
 (who would import "std.foo.bar.baz", if they weren't planning 
 to use baz?).

 Arguably, we'd get "quadratic" effectiveness ;)


I agree: I would be happy to have a warning for a module import 
when any of its symbols is used.
That would lead to cleanup in our code...

/Paolo
Dec 22 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/21/13 1:38 PM, Dmitry Olshansky wrote:

 As it stands the only thing lazy buys us is "pay as you touch" contrary
 to "pay as you name the intent to touch". The problem is that the
 payment is for the whole stock of the said "shop". I see second problem
 (granularity of imports) as far more critical then the first (condition
 under which the pieces are imported). The second problem seems solvable
 within the current implementation, the first seems like it would need
 arbitrary amount of time to fix and gains are marginal.


The way I see it is we can improve the compilation speed for everyone 
without having them change the code in any way, as opposed to toiling 
over phobos to the benefit of nothing else but phobos. Which is the 
better win?

Andrei
Dec 21 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
22-Dec-2013 08:23, Andrei Alexandrescu пишет:

 On 12/21/13 1:38 PM, Dmitry Olshansky wrote:

 As it stands the only thing lazy buys us is "pay as you touch" contrary
 to "pay as you name the intent to touch". The problem is that the
 payment is for the whole stock of the said "shop". I see second problem
 (granularity of imports) as far more critical then the first (condition
 under which the pieces are imported). The second problem seems solvable
 within the current implementation, the first seems like it would need
 arbitrary amount of time to fix and gains are marginal.


 The way I see it is we can improve the compilation speed for everyone
 without having them change the code in any way, as opposed to toiling
 over phobos to the benefit of nothing else but phobos.


Because of poor organization of Phobos we have to. Anyhow work on std 
library is a major benefit to everyone who uses it (including 
compilation speed). And nobody have to change code to get the benefits 
e.g. the proverbial size of hello world is going to be smaller (you 
surely recall that recent problem).

Unlike Phobos many projects already have sane package structure but the 
moment they use this tiny primitive (e.g. writeln) in Phobos compile 
times drop.


 Which is the
 better win?


I'd take both :o)

I'm not against lazy imports per see. The problem is that the "instant" 
benefit is going to be smaller then you seem to imply. I'd like to be 
proven otherwise but there is no experimental compiler to see yet.

If Phobos stays pretty much as is lazy imports is just a new 
(potentially slow and bogus) feature. And we know how new stuff works - 
see -allinst switch. On the other hand restructuring Phobos is a bonus 
that users don't have to suffer for.

-- 
Dmitry Olshansky
Dec 22 2013
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On 12/20/2013 09:43 PM, Martin Nowak wrote:

 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


Furthermore selective import can always be made lazily without changing 
code.

import std.algorithm : min;

Only if `min` or `std.algorithm` are used the compiler needs to open 
std.algorithm.
Jan 05 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/5/14 5:22 PM, Martin Nowak wrote:

 On 12/20/2013 09:43 PM, Martin Nowak wrote:

 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


 Furthermore selective import can always be made lazily without changing
 code.

 import std.algorithm : min;

 Only if `min` or `std.algorithm` are used the compiler needs to open
 std.algorithm.


I thought more about this and I think that's wrong. Making all imports 
lazy would be a net large win.

The missing link is transitivity: if one imports only min from 
std.algorithm, std.algorithm itself indeed needs to be opened and 
parsed, but ONLY transitive imports of std.algorithm that min itself 
needs would be opened.

We can do lazy importing with no change to the language semantics/


Andrei
Jan 05 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/5/14 5:22 PM, Martin Nowak wrote:

 On 12/20/2013 09:43 PM, Martin Nowak wrote:

 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


 Furthermore selective import can always be made lazily without changing
 code.

 import std.algorithm : min;

 Only if `min` or `std.algorithm` are used the compiler needs to open
 std.algorithm.


I thought more about this and I think that's wrong. Making all imports 
lazy would be a net large win.

The missing link is transitivity: if one imports only min from 
std.algorithm, std.algorithm itself indeed needs to be opened and 
parsed, but ONLY transitive imports of std.algorithm that min itself 
needs would be opened.

We can do lazy importing with no change to the language semantics.


Andrei
Jan 05 2014
parent reply Kenji Hara <k.hara.pg gmail.com> writes:
2014/1/6 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>


 On 1/5/14 5:22 PM, Martin Nowak wrote:


 On 12/20/2013 09:43 PM, Martin Nowak wrote:


 Couldn't static imports be made lazy without breaking any code?
 The above example would read.

 static import std.range.

 void foo(R)(R range) if (std.range.isForwardRange!R)
 {
 }


 Furthermore selective import can always be made lazily without changing
 code.

 import std.algorithm : min;

 Only if `min` or `std.algorithm` are used the compiler needs to open
 std.algorithm.


 I thought more about this and I think that's wrong. Making all imports
 lazy would be a net large win.


Honestly, "lazy import" (== defer loading module file and running semantic
analysis for symbol search) would improve compilation speed for selective
imports and static imports, but it would have no merit for basic imports.

So precisely, "all imports lazy would be a net large win." is not correct.

Kenji Hara
Jan 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/5/14 8:44 PM, Kenji Hara wrote:

 Honestly, "lazy import" (== defer loading module file and running
 semantic analysis for symbol search) would improve compilation speed for
 selective imports and static imports, but it would have no merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." is not correct.


Consider:

import std.stdio;
void main() { writeln("Hello, world!"); }

Currently std.stdio and all modules transitively imported by it would be 
opened.

With lazy imports, std.stdio gets opened and then writeln() gets 
semantically analyzed. Only modules required by writeln() itself will 
actually be opened. Big difference.


Andrei
Jan 05 2014
next sibling parent Kenji Hara <k.hara.pg gmail.com> writes:
2014/1/6 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>


 On 1/5/14 8:44 PM, Kenji Hara wrote:


 Honestly, "lazy import" (== defer loading module file and running
 semantic analysis for symbol search) would improve compilation speed for
 selective imports and static imports, but it would have no merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." is not correct.


 Consider:

 import std.stdio;
 void main() { writeln("Hello, world!"); }

 Currently std.stdio and all modules transitively imported by it would be
 opened.

 With lazy imports, std.stdio gets opened and then writeln() gets
 semantically analyzed. Only modules required by writeln() itself will
 actually be opened. Big difference.



Thanks for explanation. Indeed it would be big difference.

Kenji Hara
Jan 05 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 05, 2014 at 09:22:09PM -0800, Andrei Alexandrescu wrote:

 On 1/5/14 8:44 PM, Kenji Hara wrote:

Honestly, "lazy import" (== defer loading module file and running
semantic analysis for symbol search) would improve compilation speed
for selective imports and static imports, but it would have no merit
for basic imports.

So precisely, "all imports lazy would be a net large win." is not
correct.


 
 Consider:
 
 import std.stdio;
 void main() { writeln("Hello, world!"); }
 
 Currently std.stdio and all modules transitively imported by it
 would be opened.
 
 With lazy imports, std.stdio gets opened and then writeln() gets
 semantically analyzed. Only modules required by writeln() itself
 will actually be opened. Big difference.


[...]

It would be even better if only those modules required by the specific
instantiation of writeln() being used would be loaded. So if you only
instantiate writeln() with string arguments, then std.format doesn't
even need to be pulled in.

My guess is that this alone would reduce std.stdio template bloat by
about 10-20% or so, possibly more, because most uses of writeln()
actually only need a small part of the entire code that implements it.


T

-- 
Never step over a puddle, always step around it. Chances are that whatever made
it is still dripping.
Jan 05 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/5/2014 9:22 PM, Andrei Alexandrescu wrote:

 On 1/5/14 8:44 PM, Kenji Hara wrote:

 Honestly, "lazy import" (== defer loading module file and running
 semantic analysis for symbol search) would improve compilation speed for
 selective imports and static imports, but it would have no merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." is not correct.


 Consider:

 import std.stdio;
 void main() { writeln("Hello, world!"); }

 Currently std.stdio and all modules transitively imported by it would be
opened.

 With lazy imports, std.stdio gets opened and then writeln() gets semantically
 analyzed. Only modules required by writeln() itself will actually be opened.
Big
 difference.


import bar;
import foo;

Importing foo can never be done lazily if there are unqualified references to 
symbols in bar.
Jan 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/5/14 10:31 PM, Walter Bright wrote:

 On 1/5/2014 9:22 PM, Andrei Alexandrescu wrote:

 On 1/5/14 8:44 PM, Kenji Hara wrote:

 Honestly, "lazy import" (== defer loading module file and running
 semantic analysis for symbol search) would improve compilation speed for
 selective imports and static imports, but it would have no merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." is not
 correct.


 Consider:

 import std.stdio;
 void main() { writeln("Hello, world!"); }

 Currently std.stdio and all modules transitively imported by it would
 be opened.

 With lazy imports, std.stdio gets opened and then writeln() gets
 semantically
 analyzed. Only modules required by writeln() itself will actually be
 opened. Big
 difference.


 import bar;
 import foo;

 Importing foo can never be done lazily if there are unqualified
 references to symbols in bar.


Yah, but modules transitively imported in foo and bar need not be loaded 
eagerly. That's where the win comes from. Took me a while to figure.

Andrei
Jan 05 2014
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 6 January 2014 at 07:02:03 UTC, Andrei Alexandrescu 
wrote:

 On 1/5/14 10:31 PM, Walter Bright wrote:

 On 1/5/2014 9:22 PM, Andrei Alexandrescu wrote:

 On 1/5/14 8:44 PM, Kenji Hara wrote:

 Honestly, "lazy import" (== defer loading module file and 
 running
 semantic analysis for symbol search) would improve 
 compilation speed for
 selective imports and static imports, but it would have no 
 merit for
 basic imports.

 So precisely, "all imports lazy would be a net large win." 
 is not
 correct.


 Consider:

 import std.stdio;
 void main() { writeln("Hello, world!"); }

 Currently std.stdio and all modules transitively imported by 
 it would
 be opened.

 With lazy imports, std.stdio gets opened and then writeln() 
 gets
 semantically
 analyzed. Only modules required by writeln() itself will 
 actually be
 opened. Big
 difference.


 import bar;
 import foo;

 Importing foo can never be done lazily if there are unqualified
 references to symbols in bar.


 Yah, but modules transitively imported in foo and bar need not 
 be loaded eagerly. That's where the win comes from. Took me a 
 while to figure.

 Andrei


Right, but this fails quite quickly. Consider this:

//----
module Foo;
import tons_of_imports_here;

void foo()
{
     unqualified_call();
}
//----
import Foo;

void main()
{
     foo();
}
//----

The *instant* the implementation makes an unqualified call, you 
*have* to import *all* the imports of the module, to figure out 
what to call.

Given that you *probably* imported "foo" with the plan to *use* 
one of its functions, you'll encounter an unqualified call sooner 
rather than later, and any "win" will promptly be lost.
Jan 07 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/7/14 3:25 AM, monarch_dodra wrote:

 Given that you *probably* imported "foo" with the plan to *use* one of
 its functions, you'll encounter an unqualified call sooner rather than
 later, and any "win" will promptly be lost.


It's not a one- or two-levels win, it's a transitive win. An unqualified 
call in one implementation would trigger only one level of import.

That said I agree it's suboptimal, but it's a net improvement in the 
compiler that requires zero changes to source code. According to Walter 
it would also get rid of some forward declarations issues.


Andrei
Jan 07 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 01/07/2014 05:20 PM, Andrei Alexandrescu wrote:

 On 1/7/14 3:25 AM, monarch_dodra wrote:

 Given that you *probably* imported "foo" with the plan to *use* one of
 its functions, you'll encounter an unqualified call sooner rather than
 later, and any "win" will promptly be lost.


 It's not a one- or two-levels win, it's a transitive win. An unqualified
 call in one implementation would trigger only one level of import.

 That said I agree it's suboptimal, but it's a net improvement in the
 compiler that requires zero changes to source code. According to Walter
 it would also get rid of some forward declarations issues.


 Andrei


Without introducing new ones?

The problem with getting rid of some forward declarations issues in the 
past has been that some new ones were often introduced due to the 
analysis order changing in ad-hoc ways.

What is the status of getting rid of _all_ forward declarations issues 
except those in a precisely specified set of uninterpretable usages?

E.g. I have no precise image of how arbitrary forward referencing of 
enum members is actually supposed to work in a consistent way without 
base type annotation.
Jan 07 2014
prev sibling parent "Martin Nowak" <code dawg.eu> writes:
On Monday, 6 January 2014 at 07:02:03 UTC, Andrei Alexandrescu 
wrote:

 Yah, but modules transitively imported in foo and bar need not 
 be loaded eagerly. That's where the win comes from. Took me a 
 while to figure.

 Andrei


Yes, it would help in certain cases (writeln doesn't have 
template constraints), but we should still use more static and 
scoped selective imports to reduce coupling.
Otherwise we can't improve on a quite common case.

import foo_dep, bar_dep;

void foo(T)(T) if(isSomeFoo!T) {}
void bar(T)(T) if(isSomeBar!T) {}

When using foo the compiler will have to import bar's 
dependencies and vice versa. Refactoring the imports gets rid of 
this problem.

import foo_dep : isSomeFoo;
static import bar_dep;

void foo(T)(T) if(isSomeFoo!T) { import foo_dep; /*..*/ }
void bar(T)(T) if(bar_dep.isSomeBar!T) { import bar_dep : 
someBar; /*..*/ }
Aug 04 2014
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 12/20/2013 06:27 PM, Andrei Alexandrescu wrote:

 I had this idea fot a while, and Walter is favorable of it as well -
 extend "import" for one-shot use. With that feature the example would
 become:

      void topN(alias less = "a < b",
              SwapStrategy ss = SwapStrategy.unstable,
              Range, RandomGen)(Range r, size_t nth, ref RandomGen rng)
          if (isRandomAccessRange!(Range) && hasLength!Range
              && import.std.random.isUniformRNG!RandomGen)
      { ... }

 In this case "import" would syntactically be placed at the beginning of
 a qualified name, meaning "import this module lazily and look up the
 symbol in it".

 This would simplify quite a lot of two-liners into one-liners in other
 places, too.


 Andrei


This is problematic for dependency generation à la rdmd.
The compiler won't know the import until the lazy symbol
is used, thereby making it impossible to build all dependencies
into a library using rdmd.
Dec 20 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 20, 2013 at 10:06:01AM +0100, Meta wrote:

 On Wednesday, 18 December 2013 at 21:40:08 UTC, Andrei Alexandrescu
 wrote:

2. Push imports from top level into the entities (functions,
classes etc) that use them.


 
 2 comes with its own problems, though. With imports at the top, you
 can easily tell what the module imports at a glance. Putting imports
 into the deepest possible scopes where they are used will make it a
 huge chore to hunt all the imports down. You can of course grep for
 "import" (or ctrl+f for more plebeian editors), but you still don't
 have that singular list in one place that you can easily look at to
 tell what the module imports.


But we're only doing this for Phobos. You can still use global imports
in your own modules. I think for Phobos this makes sense, because it's
the standard *library*, which means all of its pieces will always be
available (being, y'know, standard), and you really only care about what
that particular piece of functionality you're using imports. When you
import std.algorithm.find, you don't really care about what
std.algorithm.cartesianProduct imports, do you?



 Also, as was mentioned before, some imports (or lack of) will go
 unnoticed until the particular template or template path that it
 resides in gets instantiated. I don't relish the idea of thinking
 that my code is all clear, then getting some missing dependency
 error down the line after refactoring. This will make code more
 brittle.


In your own code you can still use imports at the top of the file. We
never said we're going to get rid of that.


T

-- 
IBM = I Blame Microsoft
Dec 20 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-18 22:06, Andrei Alexandrescu wrote:

 http://chopapp.com/#fvepfd8 shows the number of dependencies (plus 1)
 for each module in phobos. Those include druntime dependencies.


What is this page, it takes 15 seconds to load the content.

-- 
/Jacob Carlborg
Dec 19 2013
parent reply "Craig Dillabaugh" <cdillaba cg.scs.carleton.ca> writes:
On Thursday, 19 December 2013 at 13:00:04 UTC, Jacob Carlborg
wrote:

 On 2013-12-18 22:06, Andrei Alexandrescu wrote:

 http://chopapp.com/#fvepfd8 shows the number of dependencies 
 (plus 1)
 for each module in phobos. Those include druntime dependencies.


 What is this page, it takes 15 seconds to load the content.


I noticed the same thing.  When I loaded the page I heard my disk
crank up and the computer working like crazy.  I was expecting
some magnificent graphical display, and then I got a list of 100
or so lines of plain text.

Maybe it uses the Javascript hypenator or something.  After all
it was Andrei that posted it :o)
Dec 19 2013
parent reply Martin Nowak <code dawg.eu> writes:
On 12/19/2013 04:13 PM, Craig Dillabaugh wrote:

 Maybe it uses the Javascript hypenator or something.  After all
 it was Andrei that posted it :o)


Hyphenate with D. http://code.dlang.org/packages/hyphenate
If someone would wire it up with http://code.dlang.org/packages/gumbo-d 
it could run on HTML files.
Dec 19 2013
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/20/13, Martin Nowak <code dawg.eu> wrote:

 On 12/19/2013 04:13 PM, Craig Dillabaugh wrote:

 Maybe it uses the Javascript hypenator or something.  After all
 it was Andrei that posted it :o)


 Hyphenate with D. http://code.dlang.org/packages/hyphenate


P.S. That homepage link leads to a dead link. The repo link works though.
Dec 20 2013