www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Speeding up importing Phobos files

reply Walter Bright <newshound2 digitalmars.com> writes:
Andrei and I were talking on the phone today, trading ideas about speeding up 
importation of Phobos files. Any particular D file tends to import much of 
Phobos, and much of Phobos imports the rest of it. We've both noticed that file 
size doesn't seem to matter much for importation speed, but file lookups remain 
slow.

So looking up fewer files would make it faster.

Here's the idea: Place all Phobos source files into a single zip file, call it 
phobos.zip (duh). Then,

     dmd myfile.d phobos.zip

and the compiler will look in phobos.zip to resolve, say, std/stdio.d. If 
phobos.zip is opened as a memory mapped file, whenever std/stdio.d is read, the 
file will be "faulted" into memory rather than doing a file lookup / read.
We're 
speculating that this should be significantly faster, besides being very 
convenient for the user to treat Phobos as a single file rather than a
blizzard. 
(phobos.lib could also be in the same file!)

It doesn't have to be just phobos, this can be a general facility. People can 
distribute their D libraries as a zip file that never needs unzipping.

We already have https://dlang.org/phobos/std_zip.html to do the dirty work. We 
can experiment to see if compressed zips are faster than uncompressed ones.

This can be a fun challenge! Anyone up for it?

P.S. dmd's ability to directly manipulate object library files, rather than 
going through lib or ar, has been a nice success.
Jan 19 2019
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas 
 about speeding up importation of Phobos files. Any particular D 
 file tends to import much of Phobos, and much of Phobos imports 
 the rest of it. We've both noticed that file size doesn't seem 
 to matter much for importation speed, but file lookups remain 
 slow.

 [...]


If we are going there we might as well use a proper database as 
the compiler cache format.
similar to to pre-compiled headers.

I'd be interested to see in how this bears out.
I am going to devote some time this weekend to it.

But using sqlite rather than zip.
Jan 19 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 19, 2019 at 08:59:37AM +0000, Stefan Koch via Digitalmars-d wrote:

 On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas about
 speeding up importation of Phobos files. Any particular D file tends
 to import much of Phobos, and much of Phobos imports the rest of it.
 We've both noticed that file size doesn't seem to matter much for
 importation speed, but file lookups remain slow.
 
 [...]


 
 If we are going there we might as well use a proper database as the
 compiler cache format.
 similar to to pre-compiled headers.


[...]

I'd like to see us go in this direction.  It could lead to other new
things, like the compiler inferring attributes for all functions (not
just template / auto functions) and storing the inferred attributes in
the precompiled cache.  It could even store additional derived
information not representable in the source that could be used for
program-wide optimization, etc..


T

-- 
Let's call it an accidental feature. -- Larry Wall
Jan 19 2019
prev sibling next sibling parent reply Temtaime <temtaime gmail.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas 
 about speeding up importation of Phobos files. Any particular D 
 file tends to import much of Phobos, and much of Phobos imports 
 the rest of it. We've both noticed that file size doesn't seem 
 to matter much for importation speed, but file lookups remain 
 slow.

 So looking up fewer files would make it faster.

 Here's the idea: Place all Phobos source files into a single 
 zip file, call it phobos.zip (duh). Then,

     dmd myfile.d phobos.zip

 and the compiler will look in phobos.zip to resolve, say, 
 std/stdio.d. If phobos.zip is opened as a memory mapped file, 
 whenever std/stdio.d is read, the file will be "faulted" into 
 memory rather than doing a file lookup / read. We're 
 speculating that this should be significantly faster, besides 
 being very convenient for the user to treat Phobos as a single 
 file rather than a blizzard. (phobos.lib could also be in the 
 same file!)

 It doesn't have to be just phobos, this can be a general 
 facility. People can distribute their D libraries as a zip file 
 that never needs unzipping.

 We already have https://dlang.org/phobos/std_zip.html to do the 
 dirty work. We can experiment to see if compressed zips are 
 faster than uncompressed ones.

 This can be a fun challenge! Anyone up for it?

 P.S. dmd's ability to directly manipulate object library files, 
 rather than going through lib or ar, has been a nice success.


C'mon, everyone has a SSD, OS tends to cache previously opened 
files. What's the problem ?
Better speedup compilation speed.
Jan 19 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/19/2019 1:00 AM, Temtaime wrote:

 C'mon, everyone has a SSD, OS tends to cache previously opened files. What's
the 
 problem ?
 Better speedup compilation speed.


You'd think that'd be true, but it isn't. File reads are fast, but file lookups 
are slow. Searching for a file along a path is particularly slow.
Jan 19 2019
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Saturday, 19 January 2019 at 09:08:00 UTC, Walter Bright wrote:

 On 1/19/2019 1:00 AM, Temtaime wrote:

 C'mon, everyone has a SSD, OS tends to cache previously opened 
 files. What's the problem ?
 Better speedup compilation speed.


 You'd think that'd be true, but it isn't. File reads are fast, 
 but file lookups are slow. Searching for a file along a path is 
 particularly slow.


If you've benchmarked this, could you please post your benchmark 
source so people can reproduce it? Probably be good to gather 
data from more than one PC. Maybe make a minisurvey for the 
results.
Jan 19 2019
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/19/2019 1:12 AM, FeepingCreature wrote:

 If you've benchmarked this, could you please post your benchmark source so 
 people can reproduce it?


I benchmarked it while developing Warp (the C preprocessor replacement I did
for 
Facebook). I was able to speed up searches for .h files substantially by 
remembering previous lookups in a hash table. The speedup persisted across 
Windows and Linux.

https://github.com/facebookarchive/warp



 Probably be good to gather data from more than one PC. 
 Maybe make a minisurvey for the results.


Sounds like a good idea. Please take charge of this!
Jan 19 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/19/19 4:12 AM, FeepingCreature wrote:

 On Saturday, 19 January 2019 at 09:08:00 UTC, Walter Bright wrote:

 On 1/19/2019 1:00 AM, Temtaime wrote:

 C'mon, everyone has a SSD, OS tends to cache previously opened files. 
 What's the problem ?
 Better speedup compilation speed.


 You'd think that'd be true, but it isn't. File reads are fast, but 
 file lookups are slow. Searching for a file along a path is 
 particularly slow.


 
 If you've benchmarked this, could you please post your benchmark source 
 so people can reproduce it? Probably be good to gather data from more 
 than one PC. Maybe make a minisurvey for the results.


I've done a bunch of measurements while I was working on 
https://github.com/dlang/DIPs/blob/master/DIPs/DIP1005.md, on a modern 
machine with SSD and Linux (which aggressively caches file contents). I 
don't think I still have the code, but it shouldn't be difficult to sit 
down and produce some. The overall conclusion of those experiments was 
that if you want to improve compilation speed, you need to minimize the 
number of files opened; once opened, whether it was 1 KB or 100 KB made 
virtually no difference.

One thing I didn't measure was whether opening the file was most 
overhead, or closing also had a large share.
Jan 19 2019
parent Pjotr Prins <pjotr.public12 thebird.nl> writes:
On Saturday, 19 January 2019 at 16:30:39 UTC, Andrei Alexandrescu 
wrote:

 On 1/19/19 4:12 AM, FeepingCreature wrote:

 On Saturday, 19 January 2019 at 09:08:00 UTC, Walter Bright 
 wrote:

 On 1/19/2019 1:00 AM, Temtaime wrote:

 C'mon, everyone has a SSD, OS tends to cache previously 
 opened files. What's the problem ?
 Better speedup compilation speed.


 You'd think that'd be true, but it isn't. File reads are 
 fast, but file lookups are slow. Searching for a file along a 
 path is particularly slow.


 
 If you've benchmarked this, could you please post your 
 benchmark source so people can reproduce it? Probably be good 
 to gather data from more than one PC. Maybe make a minisurvey 
 for the results.


 I've done a bunch of measurements while I was working on 
 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1005.md, on a 
 modern machine with SSD and Linux (which aggressively caches 
 file contents). I don't think I still have the code, but it 
 shouldn't be difficult to sit down and produce some. The 
 overall conclusion of those experiments was that if you want to 
 improve compilation speed, you need to minimize the number of 
 files opened; once opened, whether it was 1 KB or 100 KB made 
 virtually no difference.

 One thing I didn't measure was whether opening the file was 
 most overhead, or closing also had a large share.


I deal with large compressed files. For large data lz4 would 
probably be a better choice over zip these days. And, even with 
cached lookups of dir entries, I think one file that is 
sequentially read will always be an improvement. Note also that 
compressed files may even be faster than uncompressed ones with 
some system configurations (relatively slow disk IO, many 
processors).

Another thing to look at is indexed compressed files. For example 
http://www.htslib.org/doc/tabix.html. Using those we may 
partition phobos into sensible sub-sections. Particularly section 
out those submodules people hardly ever use.
Jan 20 2019
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 We already have https://dlang.org/phobos/std_zip.html to do the 
 dirty work. We can experiment to see if compressed zips are 
 faster than uncompressed ones.


BTW firefox uses fast compression option indicated by general 
purpose flags 2, std.zip uses default compression option.
Jan 19 2019
prev sibling next sibling parent reply Boris-Barboris <ismailsiege gmail.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas 
 about speeding up importation of Phobos files. Any particular D 
 file tends to import much of Phobos, and much of Phobos imports 
 the rest of it. We've both noticed that file size doesn't seem 
 to matter much for importation speed, but file lookups remain 
 slow.

 So looking up fewer files would make it faster.


Sounds rather strange that on modern operating systems, that 
cache files themselves and the metadata even more surely (VFS 
directory cache as an example), file lookup is a problem.

Calls to something like glob(3) could render the whole phobos 
directory tree to your memory in milliseconds.
Jan 19 2019
parent sarn <sarn theartofmachinery.com> writes:
On Saturday, 19 January 2019 at 15:25:37 UTC, Boris-Barboris 
wrote:

 On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright 
 wrote:

 Andrei and I were talking on the phone today, trading ideas 
 about speeding up importation of Phobos files. Any particular 
 D file tends to import much of Phobos, and much of Phobos 
 imports the rest of it. We've both noticed that file size 
 doesn't seem to matter much for importation speed, but file 
 lookups remain slow.

 So looking up fewer files would make it faster.


 Sounds rather strange that on modern operating systems, that 
 cache files themselves and the metadata even more surely (VFS 
 directory cache as an example), file lookup is a problem.

 Calls to something like glob(3) could render the whole phobos 
 directory tree to your memory in milliseconds.


It's a known problem that Windows can't cache file metadata as 
aggressively as Posix systems because of differences in 
filesystem semantics.  (See 
https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829)

Walter did say he saw benchmarked improvements on Linux for Warp, 
though.
Jan 19 2019
prev sibling next sibling parent reply Neia Neutuladh <neia ikeran.org> writes:
On Sat, 19 Jan 2019 00:45:27 -0800, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas about
 speeding up importation of Phobos files. Any particular D file tends to
 import much of Phobos, and much of Phobos imports the rest of it. We've
 both noticed that file size doesn't seem to matter much for importation
 speed, but file lookups remain slow.
 
 So looking up fewer files would make it faster.


I compiled one file with one extra -I directive as a test. The file had 
two imports, one for url.d (which depends on std.conv and std.string) and 
one for std.stdio.

For each transitive import encountered, the compiler:
* looked at each import path (total of four: ., druntime, phobos, and 
urld's source dir)
* looked at each possible extension (.d, .di, and none)
* built a matching filename
* checked if it existed

Maybe it should do a shallow directory listing for the current directory 
and every -I path at startup, using that information to prune the list of 
checks it needs to do. It could also do that recursively when importing 
from a subpackage. Then it would have called 'opendir' about four times 
and readdir() about 70 times; it wouldn't ever have had to call exists(). 
It would allocate a lot less memory. And that's a change that will help 
people who don't zip up their source code every time they want to compile 
it.

It could even do this in another thread while parsing the files passed on 
the command line. That would require a bit of caution, of course.
Jan 19 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 19, 2019 at 06:05:13PM +0000, Neia Neutuladh via Digitalmars-d
wrote:
[...]

 I compiled one file with one extra -I directive as a test. The file
 had two imports, one for url.d (which depends on std.conv and
 std.string) and one for std.stdio.
 
 For each transitive import encountered, the compiler:
 * looked at each import path (total of four: ., druntime, phobos, and
 urld's source dir)
 * looked at each possible extension (.d, .di, and none)
 * built a matching filename
 * checked if it existed
 
 Maybe it should do a shallow directory listing for the current
 directory and every -I path at startup, using that information to
 prune the list of checks it needs to do. It could also do that
 recursively when importing from a subpackage. Then it would have
 called 'opendir' about four times and readdir() about 70 times; it
 wouldn't ever have had to call exists().  It would allocate a lot less
 memory. And that's a change that will help people who don't zip up
 their source code every time they want to compile it.


[...]

Excellent finding! I *knew* something was off when looking up a file is
more expensive than reading it.  I'm thinking a quick fix could be to
just cache the intermediate results of each lookup, and reuse those
instead of issuing another call to opendir() each time.  I surmise that
after this change, this issue may no longer even be a problem anymore.


T

-- 
It is of the new things that men tire --- of fashions and proposals and
improvements and change. It is the old things that startle and intoxicate. It
is the old things that are young. -- G.K. Chesterton
Jan 19 2019
parent reply Neia Neutuladh <neia ikeran.org> writes:
On Sat, 19 Jan 2019 11:56:29 -0800, H. S. Teoh wrote:

 Excellent finding! I *knew* something was off when looking up a file is
 more expensive than reading it.  I'm thinking a quick fix could be to
 just cache the intermediate results of each lookup, and reuse those
 instead of issuing another call to opendir() each time.  I surmise that
 after this change, this issue may no longer even be a problem anymore.


It doesn't even call opendir(). It assembles each potential path and calls 
exists(). Which might be better for only having a small number of imports, 
but that's not the common case.

I've a partial fix for Posix, and I'll see about getting dev tools running 
in WINE to get a Windows version. (Which isn't exactly the same, but if I 
find a difference in FindFirstFile / FindNextFile between Windows and 
WINE, I'll be surprised.)

I'm not sure what it should do when the same module is found in multiple 
locations, though -- the current code seems to take the first match. I'm 
also not sure whether it should be lazy or not.

Also symlinks and case-insensitive filesystems are annoying.

https://github.com/dhasenan/dmd/tree/fasterimport
Jan 19 2019
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 19 January 2019 at 20:32:07 UTC, Neia Neutuladh 
wrote:

 Also symlinks and case-insensitive filesystems are annoying.

 https://github.com/dhasenan/dmd/tree/fasterimport


Considering it's only ever used in one place it's not a canidate 
for root.
Maybe you could just put it in src.
Jan 19 2019
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/19/19 3:32 PM, Neia Neutuladh wrote:

 On Sat, 19 Jan 2019 11:56:29 -0800, H. S. Teoh wrote:

 Excellent finding! I *knew* something was off when looking up a file is
 more expensive than reading it.  I'm thinking a quick fix could be to
 just cache the intermediate results of each lookup, and reuse those
 instead of issuing another call to opendir() each time.  I surmise that
 after this change, this issue may no longer even be a problem anymore.


 
 It doesn't even call opendir(). It assembles each potential path and calls
 exists(). Which might be better for only having a small number of imports,
 but that's not the common case.
 
 I've a partial fix for Posix, and I'll see about getting dev tools running
 in WINE to get a Windows version. (Which isn't exactly the same, but if I
 find a difference in FindFirstFile / FindNextFile between Windows and
 WINE, I'll be surprised.)
 
 I'm not sure what it should do when the same module is found in multiple
 locations, though -- the current code seems to take the first match. I'm
 also not sure whether it should be lazy or not.
 
 Also symlinks and case-insensitive filesystems are annoying.
 
 https://github.com/dhasenan/dmd/tree/fasterimport


This is great, looking forward to seeing this improvement merged. (There 
are packaging- and distribution-related advantages to archives 
independent of this.)
Jan 20 2019
parent reply Doc Andrew <x x.com> writes:
On Sunday, 20 January 2019 at 15:10:58 UTC, Andrei Alexandrescu 
wrote:

 On 1/19/19 3:32 PM, Neia Neutuladh wrote:

 [...]


 This is great, looking forward to seeing this improvement 
 merged. (There are packaging- and distribution-related 
 advantages to archives independent of this.)


Andrei,
    Are you envisioning something like the JAR format Java uses? 
That would be pretty convenient for D library installation and 
imports...

-Doc
Jan 20 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/20/2019 9:53 AM, Doc Andrew wrote:

     Are you envisioning something like the JAR format Java uses?


jar/zip/arc/tar/lib/ar/cab/lzh/whatever

They're all the same under the hood - a bunch of files concatenated together 
with a table of contents.
Jan 20 2019
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 1/20/19 9:31 PM, Walter Bright wrote:

 On 1/20/2019 9:53 AM, Doc Andrew wrote:

     Are you envisioning something like the JAR format Java uses?


 
 jar/zip/arc/tar/lib/ar/cab/lzh/whatever
 
 They're all the same under the hood - a bunch of files concatenated 
 together with a table of contents.


I notice a trend here. You eventually end up at the Java/JAR or 
.NET/Assembly model where everything need to compile against a library 
is included in a single file. I have often wondered how hard it would be 
to teach D how to automatically include the contents of a DI file into a 
library file and read the contents of the library at compile time.

Having studied dependencies/packaging in D quite a bit; a readily 
apparent observation is that D is much more difficult to work with than 
Java/.NET/JavaScript in regards to packaging. Unified 
interface/implementation would go a LONG way to simplifying the problem.

I even built a ZIP based packing model for D.

-- 
Adam Wilson
IRC: EllipticBit
import quiet.dlang.dev;
Jan 21 2019
parent reply Jacob Carlborg <doob me.com> writes:
On 2019-01-21 11:21, Adam Wilson wrote:


 I notice a trend here. You eventually end up at the Java/JAR or 
 .NET/Assembly model where everything need to compile against a library 
 is included in a single file. I have often wondered how hard it would be 
 to teach D how to automatically include the contents of a DI file into a 
 library file and read the contents of the library at compile time.
 
 Having studied dependencies/packaging in D quite a bit; a readily 
 apparent observation is that D is much more difficult to work with than 
 Java/.NET/JavaScript in regards to packaging. Unified 
 interface/implementation would go a LONG way to simplifying the problem.
 
 I even built a ZIP based packing model for D.


For distributing libraries you use Dub. For distributing applications to 
end users you distribute the executable.

-- 
/Jacob Carlborg
Jan 21 2019
parent Adam Wilson <flyboynw gmail.com> writes:
On 1/21/19 3:50 AM, Jacob Carlborg wrote:

 On 2019-01-21 11:21, Adam Wilson wrote:
 

 I notice a trend here. You eventually end up at the Java/JAR or 
 .NET/Assembly model where everything need to compile against a library 
 is included in a single file. I have often wondered how hard it would 
 be to teach D how to automatically include the contents of a DI file 
 into a library file and read the contents of the library at compile time.

 Having studied dependencies/packaging in D quite a bit; a readily 
 apparent observation is that D is much more difficult to work with 
 than Java/.NET/JavaScript in regards to packaging. Unified 
 interface/implementation would go a LONG way to simplifying the problem.

 I even built a ZIP based packing model for D.


 
 For distributing libraries you use Dub. For distributing applications to 
 end users you distribute the executable.
 


DUB does nothing to solve the file lookup problem so I am curious, how 
does it apply to this conversation? I was talking about how the files 
are packaged for distribution, not the actual distribution of the 
package itself.

And don't even get me started on DUB...

-- 
Adam Wilson
IRC: EllipticBit
import quiet.dlang.dev;
Jan 21 2019
prev sibling next sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 19 January 2019 at 20:32:07 UTC, Neia Neutuladh 
wrote:

 https://github.com/dhasenan/dmd/tree/fasterimport


Any benchmarks?
Jan 20 2019
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/19/19 3:32 PM, Neia Neutuladh wrote:

 On Sat, 19 Jan 2019 11:56:29 -0800, H. S. Teoh wrote:

 Excellent finding! I *knew* something was off when looking up a file is
 more expensive than reading it.  I'm thinking a quick fix could be to
 just cache the intermediate results of each lookup, and reuse those
 instead of issuing another call to opendir() each time.  I surmise that
 after this change, this issue may no longer even be a problem anymore.


 
 It doesn't even call opendir(). It assembles each potential path and calls
 exists(). Which might be better for only having a small number of imports,
 but that's not the common case.
 
 I've a partial fix for Posix, and I'll see about getting dev tools running
 in WINE to get a Windows version. (Which isn't exactly the same, but if I
 find a difference in FindFirstFile / FindNextFile between Windows and
 WINE, I'll be surprised.)
 
 I'm not sure what it should do when the same module is found in multiple
 locations, though -- the current code seems to take the first match. I'm
 also not sure whether it should be lazy or not.
 
 Also symlinks and case-insensitive filesystems are annoying.
 
 https://github.com/dhasenan/dmd/tree/fasterimport
 


I wonder if packages could be used to eliminate possibilities.

For example, std is generally ONLY going to be under a phobos import 
path. That can eliminate any other import directives from even being 
tried (if they don't have a std directory, there's no point in looking 
for an std.algorithm directory in there).

Maybe you already implemented this, I'm not sure.

I still find it difficult to believe that calling exists x4 is a huge 
culprit. But certainly, caching a directory structure is going to be 
more efficient than reading it every time.

-Steve
Jan 21 2019
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 21 January 2019 at 19:01:57 UTC, Steven Schveighoffer 
wrote:

 I still find it difficult to believe that calling exists x4 is 
 a huge culprit. But certainly, caching a directory structure is 
 going to be more efficient than reading it every time.


For large directories, opendir+readdir, especially with stat, is 
much slower than open/access. Most filesystems already use a hash 
table or equivalent, so looking up a known file name is faster 
because it's a hash table lookup.

This whole endeavor generally seems like poorly reimplementing 
what the OS should already be doing.
Jan 21 2019
next sibling parent reply Neia Neutuladh <neia ikeran.org> writes:
On Mon, 21 Jan 2019 19:10:11 +0000, Vladimir Panteleev wrote:

 On Monday, 21 January 2019 at 19:01:57 UTC, Steven Schveighoffer wrote:

 I still find it difficult to believe that calling exists x4 is a huge
 culprit. But certainly, caching a directory structure is going to be
 more efficient than reading it every time.


 
 For large directories, opendir+readdir, especially with stat, is much
 slower than open/access.


We can avoid stat() except with symbolic links.

Opendir + readdir for my example would be about 500 system calls, so it 
breaks even with `import std.stdio;` assuming the cost per call is 
identical and we're reading eagerly. Testing shows that this is the case.

With a C preprocessor, though, you're dealing with /usr/share with 
thousands of header files.


 This whole endeavor generally seems like poorly reimplementing what the
 OS should already be doing.


The OS doesn't have a "find a file with one of this handful of names among 
these directories" call.
Jan 21 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/21/19 2:35 PM, Neia Neutuladh wrote:

 On Mon, 21 Jan 2019 19:10:11 +0000, Vladimir Panteleev wrote:

 On Monday, 21 January 2019 at 19:01:57 UTC, Steven Schveighoffer wrote:

 I still find it difficult to believe that calling exists x4 is a huge
 culprit. But certainly, caching a directory structure is going to be
 more efficient than reading it every time.


 For large directories, opendir+readdir, especially with stat, is much
 slower than open/access.


 
 We can avoid stat() except with symbolic links.
 
 Opendir + readdir for my example would be about 500 system calls, so it
 breaks even with `import std.stdio;` assuming the cost per call is
 identical and we're reading eagerly. Testing shows that this is the case.


Another simple test:

import std.experimental.all;
void main(){}

Use "time -c test.d". On my SSD laptop that takes 0.55 seconds. Without 
the import, it takes 0.02 seconds. In an ideal world there should be no 
difference. Those 0.53 seconds are the upper bound of the gains to be 
made by first-order improvements to import mechanics. (IMHO: low impact 
yet not negligible.)
Jun 08 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/8/19 3:12 AM, Andrei Alexandrescu wrote:

 On 1/21/19 2:35 PM, Neia Neutuladh wrote:

 On Mon, 21 Jan 2019 19:10:11 +0000, Vladimir Panteleev wrote:

 On Monday, 21 January 2019 at 19:01:57 UTC, Steven Schveighoffer wrote:

 I still find it difficult to believe that calling exists x4 is a huge
 culprit. But certainly, caching a directory structure is going to be
 more efficient than reading it every time.


 For large directories, opendir+readdir, especially with stat, is much
 slower than open/access.


 We can avoid stat() except with symbolic links.

 Opendir + readdir for my example would be about 500 system calls, so it
 breaks even with `import std.stdio;` assuming the cost per call is
 identical and we're reading eagerly. Testing shows that this is the case.


 
 Another simple test:
 
 import std.experimental.all;
 void main(){}
 
 Use "time -c test.d". On my SSD laptop that takes 0.55 seconds. Without 
 the import, it takes 0.02 seconds. In an ideal world there should be no 
 difference. Those 0.53 seconds are the upper bound of the gains to be 
 made by first-order improvements to import mechanics. (IMHO: low impact 
 yet not negligible.)



Might it be due to something like this?

https://issues.dlang.org/show_bug.cgi?id=19874

-Steve
Jun 10 2019
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jan 21, 2019 at 07:10:11PM +0000, Vladimir Panteleev via Digitalmars-d
wrote:

 On Monday, 21 January 2019 at 19:01:57 UTC, Steven Schveighoffer wrote:

 I still find it difficult to believe that calling exists x4 is a
 huge culprit. But certainly, caching a directory structure is going
 to be more efficient than reading it every time.


 
 For large directories, opendir+readdir, especially with stat, is much
 slower than open/access. Most filesystems already use a hash table or
 equivalent, so looking up a known file name is faster because it's a
 hash table lookup.
 
 This whole endeavor generally seems like poorly reimplementing what
 the OS should already be doing.


I can't help wondering why we're making so much noise about a few
milliseconds on opening/reading import files, when there's the elephant
in the room of the 3-5 *seconds* of compile-time added by the mere act
of using a single instance of std.regex.Regex.

Shouldn't we be doing something about that first??


T

-- 
Verbing weirds language. -- Calvin (& Hobbes)
Jan 21 2019
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 21 January 2019 at 19:42:57 UTC, H. S. Teoh wrote:

 I can't help wondering why we're making so much noise about a 
 few milliseconds on opening/reading import files, when there's 
 the elephant in the room of the 3-5 *seconds* of compile-time 
 added by the mere act of using a single instance of 
 std.regex.Regex.

 Shouldn't we be doing something about that first??


 T


I am on it :P

I cannot do it any faster than I am currently doing it though.

1st Class Functions to replace recursive templates are still 
pursued by me as well.
Jan 21 2019
prev sibling next sibling parent Thomas Mader <thomas.mader gmail.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 We've both noticed that file size doesn't seem to matter much 
 for importation speed, but file lookups remain slow.


Maybe there are still some tricks to apply to make the lookup 
faster?
Ripgrep [1] is to my knowledge the fastest grepping tool out 
there currently and the speed is mostly about grepping probably 
but to be fast they need to get to the files fast too. So it 
might be helpful to look at the code.

[1] https://github.com/BurntSushi/ripgrep
Jan 20 2019
prev sibling next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 This can be a fun challenge! Anyone up for it?


This might get you some speed for the first compilation (fewer 
directory entry lookups), but I would expect that follow-up 
compilations to have a negligible overhead, compared to the CPU 
time needed to actually process the source code.
Jan 20 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/20/2019 10:21 PM, Vladimir Panteleev wrote:

 This might get you some speed for the first compilation (fewer directory entry 
 lookups), but I would expect that follow-up compilations to have a negligible 
 overhead, compared to the CPU time needed to actually process the source code.



In my benchmarks with Warp, the slowdown persisted with multiple sequential
runs.
Jan 21 2019
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 21 January 2019 at 08:11:22 UTC, Walter Bright wrote:

 On 1/20/2019 10:21 PM, Vladimir Panteleev wrote:

 This might get you some speed for the first compilation (fewer 
 directory entry lookups), but I would expect that follow-up 
 compilations to have a negligible overhead, compared to the 
 CPU time needed to actually process the source code.


 In my benchmarks with Warp, the slowdown persisted with 
 multiple sequential runs.


Would be nice if the benchmarks were reproducible.

Some things to consider:

- Would the same result apply to Phobos files?

   Some things that could be different between the two that would 
affect the viability of this approach:

   - Total number of files
   - The size of the files
   - The relative processing time needed to actually parse the 
files

     (Presumably, a C preprocessor would be faster than a full D 
compiler.)

- Could the same result be achieved through other means, 
especially those without introducing disruptive changes to 
tooling? For example, re-enabling the parallel / preemptive 
loading of source files in DMD.

Using JAR-like files would be disruptive to many tools. Consider:

- What should the file/path error message location look like when 
printing error messages with locations in Phobos?
- How should tools such as DCD, which need to scan source code, 
adapt to these changes?
- How should tools and editors with a "go to definition" function 
work when the definition is inside an archive, especially in 
editors which do not support visiting files inside archives?

Even if you have obvious answers to these questions, they still 
need to be implemented, so the speed gain from such a change 
would need to be significant in order to justify the disruption.
Jan 21 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/21/2019 1:02 AM, Vladimir Panteleev wrote:

 Even if you have obvious answers to these questions, they still need to be 
 implemented, so the speed gain from such a change would need to be significant 
 in order to justify the disruption.


The only way to get definitive answers is to try it. Fortunately, it isn't that 
difficult.
Jan 21 2019
prev sibling next sibling parent reply Neia Neutuladh <neia ikeran.org> writes:
On Sat, 19 Jan 2019 00:45:27 -0800, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas about
 speeding up importation of Phobos files. Any particular D file tends to
 import much of Phobos, and much of Phobos imports the rest of it. We've
 both noticed that file size doesn't seem to matter much for importation
 speed, but file lookups remain slow.


I should have started out by testing this.

I replaced the file lookup with, essentially:

    if (name.startsWith("std"))
        filename = "/phobos/" ~ name.replace(".", "/") ~ ".d";
    else
        filename = "/druntime/" ~ name.replace(".", "/") ~ ".d";

Plus a hard-coded set of package.d references.

Before, compiling my test file took about 0.67 to 0.70 seconds. After, it 
took about 0.67 to 0.70 seconds.

There is no point in optimizing filesystem access for importing phobos at 
this time.
Jan 21 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/21/19 2:46 PM, Neia Neutuladh wrote:

 On Sat, 19 Jan 2019 00:45:27 -0800, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas about
 speeding up importation of Phobos files. Any particular D file tends to
 import much of Phobos, and much of Phobos imports the rest of it. We've
 both noticed that file size doesn't seem to matter much for importation
 speed, but file lookups remain slow.


 
 I should have started out by testing this.
 
 I replaced the file lookup with, essentially:
 
      if (name.startsWith("std"))
          filename = "/phobos/" ~ name.replace(".", "/") ~ ".d";
      else
          filename = "/druntime/" ~ name.replace(".", "/") ~ ".d";
 
 Plus a hard-coded set of package.d references.
 
 Before, compiling my test file took about 0.67 to 0.70 seconds. After, it
 took about 0.67 to 0.70 seconds.
 
 There is no point in optimizing filesystem access for importing phobos at
 this time.


Word. (Unless the libs are installed over a networked mount. Not sure 
how much we need to worry about that.)
Jun 08 2019
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 This can be a fun challenge! Anyone up for it?


I wrote about this idea in my blog today:

http://dpldocs.info/this-week-in-d/Blog.Posted_2019_01_21.html#my-thoughts-on-forum-discussions


In short, it may be a fun challenge, and may be useful to some 
library distributors, but I don't think it is actually worth it.
Jan 21 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/21/19 4:13 PM, Adam D. Ruppe wrote:

 On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 This can be a fun challenge! Anyone up for it?


 
 I wrote about this idea in my blog today:
 
 http://dpldocs.info/this-week-in-d/Blog.Posted_2019_01_21.html#my-thoughts-
n-forum-discussions 
 
 
 
 In short, it may be a fun challenge, and may be useful to some library 
 distributors, but I don't think it is actually worth it.


Lot of good thoughts there, most of which I agree with. Thanks for sharing.

One note -- I don't think modules like std.datetime were split up for 
the sake of the compiler parsing speed, I thought they were split up to 
a) avoid the insane ddoc generation that came from it, and b) reduce 
dependencies on symbols that you didn't care about. Not to mention that 
github would refuse to load std.datetime for any PRs :)

But it does help to consider the cost of finding the file and the cost 
of using the file separately, and see how they compare.

-Steve
Jan 21 2019
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jan 21, 2019 at 04:38:21PM -0500, Steven Schveighoffer via
Digitalmars-d wrote:
[...]

 One note -- I don't think modules like std.datetime were split up for
 the sake of the compiler parsing speed, I thought they were split up
 to a) avoid the insane ddoc generation that came from it, and b)
 reduce dependencies on symbols that you didn't care about. Not to
 mention that github would refuse to load std.datetime for any PRs :)


And also, I originally split up std.algorithm (at Andrei's protest)
because it was so ridiculously huge that I couldn't get unittests to run
on my PC without dying with out-of-memory errors.



 But it does help to consider the cost of finding the file and the cost
 of using the file separately, and see how they compare.


[...]

I still think a lot of this effort is misdirected -- we're trying to
hunt small fish while there's a shark in the pond.  Instead of trying to
optimize file open / read times, what we *should* be doing is to reduce
the number of recursive templates heavy-weight Phobos modules like
std.regex are using, or improving the template expansion strategies
(e.g., the various PRs that have been checked in to replace O(n)
recursive template expansions with O(log n), or replace O(n^2) with
O(n), etc.).  Or, for that matter, optimizing how the compiler processes
templates so that it performs better.

Optimizing file open / file read in the face of these much heavier
components in the compiler sounds to me like straining out the gnat
while swallowing the camel.


T

-- 
Жил-был король когда-то, при нём блоха жила.
Jan 21 2019
next sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Monday, 21 January 2019 at 21:52:01 UTC, H. S. Teoh wrote:

 On Mon, Jan 21, 2019 at 04:38:21PM -0500, Steven Schveighoffer 
 via Digitalmars-d wrote: [...]

 [...]


 And also, I originally split up std.algorithm (at Andrei's 
 protest) because it was so ridiculously huge that I couldn't 
 get unittests to run on my PC without dying with out-of-memory 
 errors.

 [...]


Does dmd ever do dynamic programming when it does recursive 
templates?
-Alex
Jan 21 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/21/19 4:52 PM, H. S. Teoh wrote:

 split up std.algorithm (at Andrei's protest)


Shouldn't have been splitted.
Jun 08 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jun 08, 2019 at 09:02:46AM +0200, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 1/21/19 4:52 PM, H. S. Teoh wrote:

 split up std.algorithm (at Andrei's protest)


 
 Shouldn't have been splitted.


It should have been. The old std.algorithm was a monster of 10,000 LOC
that caused the compiler to exhaust my RAM and thrash on swap before
dying horribly, when building unittests. It was an embarrassment.

The old std.datetime had the same problem and I'm very glad Jonathan
eventually also split it up into more sensible chunks.


T

-- 
Береги платье снову, а здоровье смолоду.
Jun 08 2019
next sibling parent Guillaume Piolat <contact+spam spam.org> writes:
On Saturday, 8 June 2019 at 09:23:54 UTC, H. S. Teoh wrote:

 On Sat, Jun 08, 2019 at 09:02:46AM +0200, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 On 1/21/19 4:52 PM, H. S. Teoh wrote:

 split up std.algorithm (at Andrei's protest)


 
 Shouldn't have been splitted.


 It should have been. The old std.algorithm was a monster of 
 10,000 LOC that caused the compiler to exhaust my RAM and 
 thrash on swap before dying horribly, when building unittests. 
 It was an embarrassment.

 The old std.datetime had the same problem and I'm very glad 
 Jonathan eventually also split it up into more sensible chunks.


 T


+1

It's just about not having too much code in the first place.
Jun 08 2019
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/19 5:23 AM, H. S. Teoh wrote:

 The old std.datetime had the same problem


That should indeed have been broken as it was.
Jun 08 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/19 5:23 AM, H. S. Teoh wrote:

 It should have been. The old std.algorithm was a monster of 10,000 LOC
 that caused the compiler to exhaust my RAM and thrash on swap before
 dying horribly, when building unittests. It was an embarrassment.


The appropriate response would have been (and still is) to fix the 
compiler. A more compact working set will also accelerate execution due 
to better locality.
Jun 08 2019
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 8 June 2019 at 14:48:24 UTC, Andrei Alexandrescu 
wrote:

A more compact working set will also accelerate
 execution due to better locality.


OTOH, a more sparse working set will accelerate development since 
it lessens cognitive load.
Jun 08 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 8 June 2019 at 14:53:20 UTC, Nicholas Wilson wrote:

 OTOH, a more sparse working set will accelerate development 
 since it lessens cognitive load.


I personally find it is a LOT easier to work with one big file 
than multiple small files.

But that said, with something like an IDE, it shouldn't matter 
either way since the computer should be able to work with both 
equally well.
Jun 08 2019
next sibling parent matheus <matheus gmail.com> writes:
On Saturday, 8 June 2019 at 14:58:17 UTC, Adam D. Ruppe wrote:

 I personally find it is a LOT easier to work with one big file 
 than multiple small files.


I feel the same.


 But that said, with something like an IDE, it shouldn't matter 
 either way since the computer should be able to work with both 
 equally well.


Well a friend of mine would discord on this over his hog electron 
based IDE. :)

Depending on the IDE people will complain about the size of the 
file, but I'm not one of them, because I'm not IDE type of guy.

Matheus.
Jun 08 2019
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 8 June 2019 at 14:58:17 UTC, Adam D. Ruppe wrote:

 On Saturday, 8 June 2019 at 14:53:20 UTC, Nicholas Wilson wrote:

 OTOH, a more sparse working set will accelerate development 
 since it lessens cognitive load.


 I personally find it is a LOT easier to work with one big file 
 than multiple small files.

 But that said, with something like an IDE, it shouldn't matter 
 either way since the computer should be able to work with both 
 equally well.



I never understand complaints about where files are located and 
what goes where. That's for the computer to know and for me to 
pretty much never care about.

Likewise, I don't understand why anyone would want a tree view of 
their filesystem in their editor. When I know the file name and 
care, I use fuzzy matching to open it instead of clicking on 
directories.

Do you know when I care about code organisation? When it impacts 
compile times negatively and/or affect coupling.
Jun 08 2019
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 6/8/19 10:48 AM, Andrei Alexandrescu wrote:

 On 6/8/19 5:23 AM, H. S. Teoh wrote:

 It should have been. The old std.algorithm was a monster of 10,000 LOC
 that caused the compiler to exhaust my RAM and thrash on swap before
 dying horribly, when building unittests. It was an embarrassment.


 
 The appropriate response would have been (and still is) to fix the 
 compiler. A more compact working set will also accelerate execution due 
 to better locality.


Sheesh, this is *exactly* the sort of "Perfection is the enemy of the 
good" foot-shooting that's been keeping D years behind where it could be.

In D1 days, things got fixed and improved quickly. But now, problems 
that have a working solution available remain unfixed for years just 
because "there's a better way to do it, but it takes far more work, 
isn't going to ready anytime soon, there or may or may not even be 
anyone active being working on it."

A fix in master is worth two in the bush.
Jun 08 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/19 6:19 PM, Nick Sabalausky (Abscissa) wrote:

 On 6/8/19 10:48 AM, Andrei Alexandrescu wrote:

 On 6/8/19 5:23 AM, H. S. Teoh wrote:

 It should have been. The old std.algorithm was a monster of 10,000 LOC
 that caused the compiler to exhaust my RAM and thrash on swap before
 dying horribly, when building unittests. It was an embarrassment.


 The appropriate response would have been (and still is) to fix the 
 compiler. A more compact working set will also accelerate execution 
 due to better locality.


 
 Sheesh, this is *exactly* the sort of "Perfection is the enemy of the 
 good" foot-shooting that's been keeping D years behind where it could be.


I did allow the breaking up of std.algorithm.
Jun 09 2019
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 6/9/19 3:56 AM, Andrei Alexandrescu wrote:

 On 6/8/19 6:19 PM, Nick Sabalausky (Abscissa) wrote:

 On 6/8/19 10:48 AM, Andrei Alexandrescu wrote:

 On 6/8/19 5:23 AM, H. S. Teoh wrote:

 It should have been. The old std.algorithm was a monster of 10,000 LOC
 that caused the compiler to exhaust my RAM and thrash on swap before
 dying horribly, when building unittests. It was an embarrassment.


 The appropriate response would have been (and still is) to fix the 
 compiler. A more compact working set will also accelerate execution 
 due to better locality.


 Sheesh, this is *exactly* the sort of "Perfection is the enemy of the 
 good" foot-shooting that's been keeping D years behind where it could be.


 
 I did allow the breaking up of std.algorithm.


Yes, and that's definitely good - after all, it gave us a stopgap fix 
for the immediate problem, while the proper solution is(was?) still 
in-the-works. Besides, much of the time, once a "proper solution" does 
become available, the old stopgap can then be rolled back/deprecated if 
necessary. (Not sure whether or not rolling back the change would be 
appropriate in std.algorithm's case, but ATM, I'm not too terribly 
concerned with that either way.)

To clarify, in my previous post, I wasn't really talking *specifically* 
about the breaking up of std.algorithm (after all, like you said, that 
DID go through). I was just speaking in general about the overall 
strategy you were promoting: Preferring to forego stopgap measures in 
the interims before correct solutions are available. (Unless I 
misunderstood?)

(Of course, in the cases where a correct solution is just as 
quick-n-simple as any stopgap, well, then yes, certainly the correct 
solution should just be done instead. Again, not saying this was or 
wasn't the case with the breaking up of std.algorithm, I'm just speaking 
in general terms.)
Jun 09 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/19 2:52 PM, Nick Sabalausky (Abscissa) wrote:

 I was just speaking in general about the overall strategy you were 
 promoting: Preferring to forego stopgap measures in the interims before 
 correct solutions are available. (Unless I misunderstood?)


I can tell at least what I tried to do - use good judgment for each such 
decision. More often than not I propose workarounds that the community 
turns its nose at - see e.g. the lazy import. To this day I think that 
would have been a great thing to do. But no, we need to "wait" for the 
full lazy import that will never materialize.
Jun 09 2019
parent reply Amex <Amex gmail.com> writes:
On Sunday, 9 June 2019 at 19:51:14 UTC, Andrei Alexandrescu wrote:

 On 6/9/19 2:52 PM, Nick Sabalausky (Abscissa) wrote:

 I was just speaking in general about the overall strategy you 
 were promoting: Preferring to forego stopgap measures in the 
 interims before correct solutions are available. (Unless I 
 misunderstood?)


 I can tell at least what I tried to do - use good judgment for 
 each such decision. More often than not I propose workarounds 
 that the community turns its nose at - see e.g. the lazy 
 import. To this day I think that would have been a great thing 
 to do. But no, we need to "wait" for the full lazy import that 
 will never materialize.



Has anyone merged all of phobos in to one large file, removed all 
the modules(or modify the compile to handle multiple modules per 
file and internally break them up) and see where the bottle neck 
truly is? Is it specific template use or just all templates? Is 
there a specific template design pattern that is often used that 
kills D?

And alternatively, break phobos up in to many more files, one per 
template... and see the performance of it.

Maybe there is specific performance blocks in the compiler and 
those could be rewritten such as parallel compilation or 
rewriting that part of the compiler to be faster or whatever..

What I'm seeing is that it seems no one really knows the true 
culprit. Is it the file layout or templates? and each issue then 
branches...

After all, maybe it is a combination and all the areas could be 
optimized better. Even a 1% increase is 1% and if it stacks with 
another 1% then one has 2%. A journey starts with the first 1%.
Jun 10 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/10/19 4:54 AM, Amex wrote:

 On Sunday, 9 June 2019 at 19:51:14 UTC, Andrei Alexandrescu wrote:

 On 6/9/19 2:52 PM, Nick Sabalausky (Abscissa) wrote:

 I was just speaking in general about the overall strategy you were 
 promoting: Preferring to forego stopgap measures in the interims 
 before correct solutions are available. (Unless I misunderstood?)


 I can tell at least what I tried to do - use good judgment for each 
 such decision. More often than not I propose workarounds that the 
 community turns its nose at - see e.g. the lazy import. To this day I 
 think that would have been a great thing to do. But no, we need to 
 "wait" for the full lazy import that will never materialize.


 
 
 Has anyone merged all of phobos in to one large file, removed all the 
 modules(or modify the compile to handle multiple modules per file and 
 internally break them up) and see where the bottle neck truly is? Is it 
 specific template use or just all templates? Is there a specific 
 template design pattern that is often used that kills D?
 
 And alternatively, break phobos up in to many more files, one per 
 template... and see the performance of it.
 
 Maybe there is specific performance blocks in the compiler and those 
 could be rewritten such as parallel compilation or rewriting that part 
 of the compiler to be faster or whatever..
 
 What I'm seeing is that it seems no one really knows the true culprit. 
 Is it the file layout or templates? and each issue then branches...
 
 After all, maybe it is a combination and all the areas could be 
 optimized better. Even a 1% increase is 1% and if it stacks with another 
 1% then one has 2%. A journey starts with the first 1%.


Not if that 1% costs 99% of your budget.
Jun 10 2019
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 21 January 2019 at 21:38:21 UTC, Steven Schveighoffer 
wrote:

 One note -- I don't think modules like std.datetime were split 
 up for the sake of the compiler parsing speed


Yeah, I think std.datetime was about the size of unittest runs, 
but std.range, as I recall at least, was specifically to separate 
stuff to get quicker builds by avoiding the majority of the 
import for common cases via std.range.primitives which doesn't 
need to bring in as much code. The local import pattern also 
helps with this goal - lazy imports to only get what you need 
when you need it, so the compiler doesn't have to do as much work.

Maybe I am wrong about that, but still, two files that import 
each other aren't actually two modules. Phobos HAS been making a 
LOT of progress toward untangling that import mess and addressing 
specific compile time problems. A few years ago, any phobos 
import would cost you like a half second. It is down to a quarter 
second for the hello world example. Which is IMO still quite 
poor, but a lot better than it was.

But is this caused by finding the files?

$ cd dmd2/src/phobos
$ time find .
real    0m0.003s

I find that very hard to believe.

And let us remember, old D1 phobos wasn't this slow:

$ cat hi.d
import std.stdio;

void main() {
         writefln("Hello!");
}

$ time dmd-1.0 hi.d

real    0m0.042s
user    0m0.032s
sys     0m0.005s

$ time dmd hi.d

real    0m0.434s
user    0m0.383s
sys     0m0.044s



Using the old D compilers reminds me what quick compiles REALLY 
are. Sigh.
Jan 21 2019
parent Adam D. Ruppe <destructionator gmail.com> writes:
BTW

dmd2/src/phobos$ time cat `find . | grep -E '\.d$'` > catted.d

real    0m0.015s
user    0m0.006s
sys     0m0.009s

$ wc catted.d
   319707  1173911 10889167 catted.d


If it were the filesystem at fault, shouldn't that I/O heavy 
operation take a significant portion of the dmd runtime?

Yes, I know the kernel is caching these things and deferring 
writes and so on. But it does that for dmd too! Blaming the 
filesystem doesn't pass the prima facie test, at least on Linux. 
Maybe Windows is different, I will try that tomorrow, but I 
remain exceedingly skeptical.
Jan 21 2019
prev sibling next sibling parent reply Arun Chandrasekaran <aruncxy gmail.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas 
 about speeding up importation of Phobos files. Any particular D 
 file tends to import much of Phobos, and much of Phobos imports 
 the rest of it. We've both noticed that file size doesn't seem 
 to matter much for importation speed, but file lookups remain 
 slow.

 So looking up fewer files would make it faster.

 If phobos.zip is opened as a memory mapped file, whenever 
 std/stdio.d is read, the file will be "faulted" into memory 
 rather than doing a file lookup / read. We're speculating that 
 this should be significantly faster,


Speaking from Linux, the kernel already caches the file (after 
the first read) unless `echo 3 > /proc/sys/vm/drop_caches` is 
triggered. I've tested with the entire phobos cached and the 
compilation is still slow. IO is not the bottleneck here. The 
compilation needs to be speeded up.

If you still think the file read is the culprint, why does 
recompilation take the same amount of time as the first 
compilation (albeit kernel file cache)?


 being very convenient for the user to treat Phobos as a single 
 file rather than a blizzard. (phobos.lib could also be in the 
 same file!)


We already have std.experimental.all for convenience.
Jan 21 2019
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, January 21, 2019 5:46:32 PM MST Arun Chandrasekaran via 
Digitalmars-d wrote:

 being very convenient for the user to treat Phobos as a single
 file rather than a blizzard. (phobos.lib could also be in the
 same file!)


 We already have std.experimental.all for convenience.


If I understand correctly, that's an orthogonal issue. What Walter is
proposing wouldn't change how any code imported anything. Rather, it would
just change how the compiler reads the files. So, anyone wanting to import
all of Phobos at once would still need something like std.experimental.all,
but regardless of how much you were importing from Phobos, dmd would read in
all of Phobos at once, because it would be a single zip file. It would then
only actually compile what it needed to for the imports in your program, but
it would have read the entire zip file into memory so that it would only
have to open one file instead of searching for and opening each file
individually.

- Jonathan M Davis
Jan 21 2019
prev sibling parent Neia Neutuladh <neia ikeran.org> writes:
On Tue, 22 Jan 2019 00:46:32 +0000, Arun Chandrasekaran wrote:

 If you still think the file read is the culprint, why does recompilation
 take the same amount of time as the first compilation (albeit kernel
 file cache)?


And another quick way to test this is to use import() and a hard-coded 
switch statement instead of IO. Just get rid of all disk access and see 
how fast you can compile code. I'm betting you'll save 10% at most.
Jan 21 2019
prev sibling next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas 
 about speeding up importation of Phobos files. Any particular D 
 file tends to import much of Phobos, and much of Phobos imports 
 the rest of it. We've both noticed that file size doesn't seem 
 to matter much for importation speed, but file lookups remain 
 slow.


The topic of import speed when an implementation is spread over 
multiple files came up again on a PR I'm working on, so I wanted 
to share an idea I had.

Why not remove the arbitrary limitation that a module or package 
must be tied to a file or directory respectively?  That is within 
one file you could have something like this:

---
package thePackage
{
     module thePackage.module1
     {
         ...
     }

     module thePackage.module2
     {
         ...
     }
}

package thePackage2
{
     ....
}
---

Then when one distributes their library, they could concatenate 
all the files into one, potentially reducing the overhead 
required to querying the filesystem.  After loading the file, the 
entire library would essentially be cached in memory.

It would also help with a number of other helpful patterns.  For 
example, I currently use the following pattern for creating 
register maps of memory-mapped IO:

---
final abstract class MyRegisterBank
{
     final abstract class MyRegister
     {
        //static properties for bit-fields
     }
}
// See 
https://github.com/JinShil/stm32f42_discovery_demo/blob/master/s
urce/stm32f42/spi.d for the real thing
---

I could avoid doing silly things like that and use modules in a 
single file instead, which would be a more appropriate model for 
that use case.  I currently don't use modules simply because I'd 
end up with 100's of file to manage for a single MCU.

Another trivial use case would be the ability to test multiple 
packages and modules on run.dlang.io without the need for any 
additional infrastructure.

I'm sure the creativity of this community would find other 
interesting uses cases.

It seems like an arbitrary limitation to have the package/module 
system hard-coded to the underlying storage technology, and 
removing that limitation may help with the import speed while 
also enabling a few interesting use cases.

Mike
Jun 07 2019
next sibling parent reply KnightMare <black80 bk.ru> writes:
zip-archive allows you to unpack the file in its original form.
unpacking allows to see source code.

ur version - join sources to one file - is more complicated
// file pack/a.d
module pack.one;
//...

// file pack/b.d
module pack.one;
//...

// ur package
package pack
{
     module one {
         // both files here?
     }

      file( "pack/b.d" ); // ???
     module one {
         // or separate?
     }
     // how to restore individual files? need some special comment 
or attrs
}
Jun 07 2019
next sibling parent KnightMare <black80 bk.ru> writes:
On Friday, 7 June 2019 at 12:50:27 UTC, KnightMare wrote:

 zip-archive allows you to unpack the file in its original form.
 unpacking allows to see source code.


u can unzip w/o any special tools - OSes can work with zip usually
so, yes, 1stage - using ZIP as one file is ok.

2stage: pack sources to 1 file as AST
pro:
- no need unpack stage from mapped files - all string spans can 
be stored as string-table-indicies allows to pack 2x-3x times
- no need parsing/verifying stage - u already have checheked AST.
contra:
need special tools (or commands to compiler) to unpack source 
files
Jun 07 2019
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 6/7/19 8:50 AM, KnightMare wrote:

 zip-archive allows you to unpack the file in its original form.
 unpacking allows to see source code.
 


That's an interesting approach to the issue: Just allow a package to be 
either a directory tree OR a tar/tarball archive of a directory tree. 
Then, the language itself wouldn't need any provisions at all to support 
multiple packages in one file, and the compiler could still read an 
entire multi-module package by accessing only one file.
Jun 07 2019
parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Friday, 7 June 2019 at 19:56:27 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 6/7/19 8:50 AM, KnightMare wrote:

 zip-archive allows you to unpack the file in its original form.
 unpacking allows to see source code.
 


 That's an interesting approach to the issue: Just allow a 
 package to be either a directory tree OR a tar/tarball archive 
 of a directory tree. Then, the language itself wouldn't need 
 any provisions at all to support multiple packages in one file, 
 and the compiler could still read an entire multi-module 
 package by accessing only one file.


Isn't it what Java does? A jar file is nothing more than a zip 
file.
Jun 07 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 07, 2019 at 09:47:34AM +0000, Mike Franklin via Digitalmars-d wrote:
[...]

 It would also help with a number of other helpful patterns.  For
 example, I currently use the following pattern for creating register
 maps of memory-mapped IO:
 
 ---
 final abstract class MyRegisterBank
 {
     final abstract class MyRegister
     {
        //static properties for bit-fields
     }
 }
 // See https://github.com/JinShil/stm32f42_discovery_demo/blob/master/source/stm32f42/spi.d
 for the real thing
 ---


Why final abstract class? If all you have are static properties, you
could use structs instead.

Though of course, it still doesn't really address your fundamental
concerns here.


[...]

 It seems like an arbitrary limitation to have the package/module
 system hard-coded to the underlying storage technology, and removing
 that limitation may help with the import speed while also enabling a
 few interesting use cases.


[...]

The flip side of this coin is that having one file per module helps with
code organization in typical use cases. As opposed to the chaotic
situation in C/C++ where you can have one .h file with arbitrary numbers
of .c/.cc files that implement the function prototypes, with no obvious
mapping between them, or multiple .h files with a single amorphous messy
.c/.cc file that implements everything. Or where there are arbitrary
numbers of modules nested inside any number of .c/.cc files with no
clear separation between them, and where filenames bear no relation with
contents, making code maintenance challenging at best, outright
nightmarish at worst.


T

-- 
Real men don't take backups. They put their source on a public FTP-server and
let the world mirror it. -- Linus Torvalds
Jun 07 2019
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 7 June 2019 at 14:45:34 UTC, H. S. Teoh wrote:


 The flip side of this coin is that having one file per module 
 helps with code organization in typical use cases. As opposed 
 to the chaotic situation in C/C++ where you can have one .h 
 file with arbitrary numbers of .c/.cc files that implement the 
 function prototypes, with no obvious mapping between them, or 
 multiple .h files with a single amorphous messy .c/.cc file 
 that implements everything. Or where there are arbitrary 
 numbers of modules nested inside any number of .c/.cc files 
 with no clear separation between them, and where filenames bear 
 no relation with contents, making code maintenance challenging 
 at best, outright nightmarish at worst.


What I'm proposing is that a library's organization can be one 
file per module while it is being developed, but once it is 
published for consumption by others all packages/modules are 
concatenated into one file so it's faster for users to import.  
Or you could even concatenate some packages/modules, but not 
others depending on how users typically interact with the library.

There may be use cases (like mine with the memory-mapped IO 
registers) where users may want to actually develop with more 
than one module per file.  Or when one wants to share 
multi-package/module cut-and-pastable code examples. But those 
are special cases.

What's nice is that it changes nothing for users today.  It just 
removes an arbitrary limitation to help improve import speed 
while enabling some more flexibility for those who may it.

Mike
Jun 07 2019
next sibling parent reply Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Friday, 7 June 2019 at 15:34:22 UTC, Mike Franklin wrote:

 What I'm proposing is that a library's organization can be one 
 file per module while it is being developed, but once it is 
 published for consumption by others all packages/modules are 
 concatenated into one file so it's faster for users to import.  
 Or you could even concatenate some packages/modules, but not 
 others depending on how users typically interact with the 
 library.

 There may be use cases (like mine with the memory-mapped IO 
 registers) where users may want to actually develop with more 
 than one module per file.  Or when one wants to share 
 multi-package/module cut-and-pastable code examples. But those 
 are special cases.

 What's nice is that it changes nothing for users today.  It 
 just removes an arbitrary limitation to help improve import 
 speed while enabling some more flexibility for those who may it.

 Mike


How would compilation even work with multiple modules per file? 
Wouldn't the compiler  have to parse all .d files in the whole 
search path then? That would be the opposite of faster compile 
times.
Jun 07 2019
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 7 June 2019 at 15:46:03 UTC, Gregor Mückl wrote:

 How would compilation even work with multiple modules per file? 
 Wouldn't the compiler  have to parse all .d files in the whole 
 search path then? That would be the opposite of faster compile 
 times.


It doesn't have to search because you pass the modules to the 
compiler, just like we do now in the general case.

The search path and filename conventions are - today - just 
conventions, there's no requirement that the filename and module 
name match, which means you don't actually know what module a 
file is until it is parsed.

(which is quick and simple btw)
Jun 07 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 07, 2019 at 03:34:22PM +0000, Mike Franklin via Digitalmars-d wrote:
[...]

 What I'm proposing is that a library's organization can be one file
 per module while it is being developed, but once it is published for
 consumption by others all packages/modules are concatenated into one
 file so it's faster for users to import.  Or you could even
 concatenate some packages/modules, but not others depending on how
 users typically interact with the library.


[...]

What *would* be really nice is if dmd could read .zip archives. Then all
you need to use a library is to download the .zip into your source tree
and run `dmd -i`.


T

-- 
I'm still trying to find a pun for "punishment"...
Jun 07 2019
parent KnightMare <black80 bk.ru> writes:
 What *would* be really nice is if dmd could read .zip archives. 
 Then all you need to use a library is to download the .zip into 
 your source tree and run `dmd -i`.


or to use LZ4 for no-dependencies from zlib and smaller code:
decompressing is 50 LOC, and probably 200LOC for processing 
"directory entries" inside file. its faster by 10x than zip - 
decompressing working w/o hash-tables as LZW do.
contra: u cannot just unzip files
Jun 07 2019
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Friday, 7 June 2019 at 15:34:22 UTC, Mike Franklin wrote:

 On Friday, 7 June 2019 at 14:45:34 UTC, H. S. Teoh wrote:


 The flip side of this coin is that having one file per module 
 helps with code organization in typical use cases. As opposed 
 to the chaotic situation in C/C++ where you can have one .h 
 file with arbitrary numbers of .c/.cc files that implement the 
 function prototypes, with no obvious mapping between them, or 
 multiple .h files with a single amorphous messy .c/.cc file 
 that implements everything. Or where there are arbitrary 
 numbers of modules nested inside any number of .c/.cc files 
 with no clear separation between them, and where filenames 
 bear no relation with contents, making code maintenance 
 challenging at best, outright nightmarish at worst.


 What I'm proposing is that a library's organization can be one 
 file per module while it is being developed, but once it is 
 published for consumption by others all packages/modules are 
 concatenated into one file so it's faster for users to import.  
 Or you could even concatenate some packages/modules, but not 
 others depending on how users typically interact with the 
 library.

 There may be use cases (like mine with the memory-mapped IO 
 registers) where users may want to actually develop with more 
 than one module per file.  Or when one wants to share 
 multi-package/module cut-and-pastable code examples. But those 
 are special cases.

 What's nice is that it changes nothing for users today.  It 
 just removes an arbitrary limitation to help improve import 
 speed while enabling some more flexibility for those who may it.

 Mike


Reading files is really cheap, evaluating templates and running 
CTFE isn't. That's why importing Phobos modules is slow - not 
because of the files it imports, but because of all the CTFE 
these imports trigger. This means we can do e.g.

- improve CTFE performance
- cache templates over multiple compilations (there's a DMD PR 
for this)
- make imports lazy
- reduce all the big CTFE bottlenecks in Phobos (e.g. std.uni)
- convert more Phobos code into templates with local imports to 
reduce the baseline import overhead

Ordered from hardest to easiest.
Jun 07 2019
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 7 June 2019 at 16:38:56 UTC, Seb wrote:


 Reading files is really cheap, evaluating templates and running 
 CTFE isn't. That's why importing Phobos modules is slow - not 
 because of the files it imports, but because of all the CTFE 
 these imports trigger.


Yes that make much more sense to me.  But, if that's the case, 
what's all the concern from Walter and Andrei expressed in this 
thread and in the conversations linked below?

https://forum.dlang.org/post/q7dpmg$29oq$1 digitalmars.com
https://github.com/dlang/druntime/pull/2634#issuecomment-499494019
https://github.com/dlang/druntime/pull/2222#issuecomment-398390889

Mike
Jun 07 2019
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Saturday, 8 June 2019 at 00:00:17 UTC, Mike Franklin wrote:

 On Friday, 7 June 2019 at 16:38:56 UTC, Seb wrote:


 Reading files is really cheap, evaluating templates and 
 running CTFE isn't. That's why importing Phobos modules is 
 slow - not because of the files it imports, but because of all 
 the CTFE these imports trigger.


 Yes that makes much more sense to me.  But, if that's the case, 
 what's all the concern from Walter and Andrei expressed in this 
 thread and in the conversations linked below?

 https://forum.dlang.org/post/q7dpmg$29oq$1 digitalmars.com
 https://github.com/dlang/druntime/pull/2634#issuecomment-499494019
 https://github.com/dlang/druntime/pull/2222#issuecomment-398390889


Another recent comment from Walter:  
https://github.com/dlang/dmd/pull/9814#issuecomment-493773769

It seems there is disagreement about what the actual issue is.

Mike
Jun 08 2019
prev sibling parent Seb <seb wilzba.ch> writes:
On Saturday, 8 June 2019 at 00:00:17 UTC, Mike Franklin wrote:

 On Friday, 7 June 2019 at 16:38:56 UTC, Seb wrote:


 Reading files is really cheap, evaluating templates and 
 running CTFE isn't. That's why importing Phobos modules is 
 slow - not because of the files it imports, but because of all 
 the CTFE these imports trigger.


 Yes that make much more sense to me.  But, if that's the case, 
 what's all the concern from Walter and Andrei expressed in this 
 thread and in the conversations linked below?

 https://forum.dlang.org/post/q7dpmg$29oq$1 digitalmars.com
 https://github.com/dlang/druntime/pull/2634#issuecomment-499494019
 https://github.com/dlang/druntime/pull/2222#issuecomment-398390889

 Mike


If they wanted to make DMD faster, compiling with LDC would make 
it truly faster (more than halfs the runtime!!).

As mentioned above, the real reason for the import overhead are 
the ton of templates and CTFE evaluations and these either need 
to be cached, made faster, made lazier or reduced if any 
significant performance gains are anticipated. Tweaking the file 
tree won't help.
Jun 13 2019
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 6/7/19 5:47 AM, Mike Franklin wrote:

 
 Why not remove the arbitrary limitation that a module or package must be 
 tied to a file or directory respectively?


We don't really have that limitation. The compiler gets the 
package/module name from the `module ...;` statement (if any) at the 
beginning of a *.d file. It's only when the `module` statement is absent 
that the package/module name is inferred from the filepath. (There 
*might* also be such a requirement when importing *but not compiling* a 
given module, but I'm not sure on that.)

Beyond that, any other requirement for packages/modules to match the 
filesystem is purely a convention relied upon by certain buildsystems, 
like rdmd (and now, `dmd -i`), and otherwise has nothing to do with the 
compiler.

TBH, I've always kinda wanted to just do away with the ability to have 
modules/packages that DON'T match the filesystem. I never saw any 
sensible use-cases for supporting such weird mismatches that couldn't 
already be accomplished via -I (not to be confused with the new -i) or 
version/static if.

HOWEVER, that said, you do bring up an interesting point I'd never 
thought of: If concating a bunch of modules/packages into one file would 
improve compile time, than that would certainly be a feature worth 
considering.
Jun 07 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 07, 2019 at 03:48:53PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
[...]

 TBH, I've always kinda wanted to just do away with the ability to have
 modules/packages that DON'T match the filesystem. I never saw any
 sensible use-cases for supporting such weird mismatches that couldn't
 already be accomplished via -I (not to be confused with the new -i) or
 version/static if.
 
 HOWEVER, that said, you do bring up an interesting point I'd never
 thought of: If concating a bunch of modules/packages into one file
 would improve compile time, than that would certainly be a feature
 worth considering.


I honestly doubt it would improve compilation time that much. Reading
files from the filesystem is pretty fast, compared to the rest of the
stuff the compiler has to do afterwards.

If anything, it would *slow down* the process if you really only needed
to import one module but it was concatenated with a whole bunch of
others in a single file, thereby requiring the compiler to parse the
whole thing just to find it. There's also the problem that if multiple
-I's were given, and more than 1 of those paths contain concatenated
modules, then the compiler would potentially have to parse *everything*
in *every import path* just to be sure it will find the one import you
asked for.

I don't know, it just seems like there are too many disadvantages to
justify doing this.


T

-- 
Philosophy: how to make a career out of daydreaming.
Jun 07 2019
parent reply KnightMare <black80 bk.ru> writes:
 I honestly doubt it would improve compilation time that much. 
 Reading files from the filesystem is pretty fast, compared to 
 the rest of the stuff the compiler has to do afterwards.


need to search for existing files in dir-tree with some templates 
"datetime.d*" for each folder in -I.

I tried compile (in Windows) one small program from today issue 
https://issues.dlang.org/show_bug.cgi?id=19947 under Procmon.exe 
(tool from SysInternals that allow to see what doing some process 
with Net,FS,Registry,Process&Threads). Result for FileSystem only 
is:
3497 requests to FS for create/open,query,close,read/write (libs 
and dll counted too)
768! requests just "not found" (dlls and libs counted too)

2nd try with option "-c" - compile only (no linking):
2693 requests to FS
727 reqs are "not found"

so for clear benchmark need to compare compilation some middle 
program (without any dub packages coz they add mor path for 
searching maybe N! factorial) with SSD and with RAM-disk, imo we 
can win 1-2secs for compilation
Jun 07 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 07, 2019 at 09:09:48PM +0000, KnightMare via Digitalmars-d wrote:
[...]

 I tried compile (in Windows) one small program from today issue
 https://issues.dlang.org/show_bug.cgi?id=19947 under Procmon.exe (tool from
 SysInternals that allow to see what doing some process with
 Net,FS,Registry,Process&Threads). Result for FileSystem only is:
 3497 requests to FS for create/open,query,close,read/write (libs and dll
 counted too)
 768! requests just "not found" (dlls and libs counted too)


[...]

This is a known issue that has been discussed before.  The proposed
solution was to cache the contents of each directory in the import path
(probably lazily, so that we don't incur up-front costs) so that the
compiler can subsequently find a module pathname with just a single hash
lookup.

I don't know if anyone set about implementing this, though.


T

-- 
Error: Keyboard not attached. Press F1 to continue. -- Yoon Ha Lee, CONLANG
Jun 07 2019
prev sibling parent reply Amex <Amex gmail.com> writes:
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:

 Andrei and I were talking on the phone today, trading ideas 
 about speeding up importation of Phobos files. Any particular D 
 file tends to import much of Phobos, and much of Phobos imports 
 the rest of it. We've both noticed that file size doesn't seem 
 to matter much for importation speed, but file lookups remain 
 slow.

 So looking up fewer files would make it faster.

 Here's the idea: Place all Phobos source files into a single 
 zip file, call it phobos.zip (duh). Then,

     dmd myfile.d phobos.zip

 and the compiler will look in phobos.zip to resolve, say, 
 std/stdio.d. If phobos.zip is opened as a memory mapped file, 
 whenever std/stdio.d is read, the file will be "faulted" into 
 memory rather than doing a file lookup / read. We're 
 speculating that this should be significantly faster, besides 
 being very convenient for the user to treat Phobos as a single 
 file rather than a blizzard. (phobos.lib could also be in the 
 same file!)

 It doesn't have to be just phobos, this can be a general 
 facility. People can distribute their D libraries as a zip file 
 that never needs unzipping.

 We already have https://dlang.org/phobos/std_zip.html to do the 
 dirty work. We can experiment to see if compressed zips are 
 faster than uncompressed ones.

 This can be a fun challenge! Anyone up for it?

 P.S. dmd's ability to directly manipulate object library files, 
 rather than going through lib or ar, has been a nice success.


Why not compile phobos to an object file? Basically store the AST 
directly in to a file and just load it. Phobos never changes so 
why recompile it over and over and over and over?

This should be done with all files... sorta like rdmd, so to 
speak.

It might take some work to figure out how to get it all to work 
but maybe the time has come to stop using ancient design patterns 
and move on to a higher level?

After all, the issue is mainly templates since they cannot be 
compiled to a library... but if they could, then they wouldn't be 
an issue. .til -> .lib -> .exe

.til is a higher level library that includes objectified 
templates from D code, it is basically an extension of a lib file 
and eventually the lib is compiled in to the exe.
Jun 07 2019
parent reply KnightMare <black80 bk.ru> writes:
On Saturday, 8 June 2019 at 06:29:16 UTC, Amex wrote:

 Why not compile phobos to an object file? Basically store the 
 AST directly in to a file and just load it. Phobos never 
 changes so why recompile it over and over and over and over?


+1 for AST.
and even more: AST-builder make as RT-module. bonuses:
- storing packages as AST-asssembly: no parsing/building-AST for 
packages, only for user code - that increase compile speed.
- can be stored more compacted than source coz words (var names, 
keywords) are repeated many times through source. and many source 
files can be stored in one "assembly/AST-package" with one 
string-literal-table, with one typeinfo-table.
- DSL metaprogramming moves to a higher level: steps "parsing DSL 
- generate D code - parse D code - generate AST" will become 
"parsing DSL - generate AST" that increase compiling time and 
helps to appear many DSL for JSON/XML/DB-schemas, and will be 
used for UI description (same as QML) and many more.
- LDC(dunno with DMD/GCC) can generate code dynamically at 
runtime already (probably it stores LLVM-IR now) then can 
generate code from AST in runtime (not only in compile time for 
metaprogramming): same bonuses/possibilities as Expression Trees 
and Scripting for .NET. Yes, Dlang can use 3rd parties script 
engines but they are not native - interop is not free, many 
wrappers/dispatchers to and from, two different GC, failed with 
threads and TLS, so its for a toy now not more. With native 
scripting absent interop at all, execution speed as compiled C++ 
(LLVM generate fast code), one GC for all objects, one thread 
pools with same TLS..

I see only one minus: dont use AST for DRT and as module itself 
for programmers.
In any case AST-builder already exists in compiler, just bring it 
outside to public space and allow store packages as AST too.
Jun 08 2019
next sibling parent KnightMare <black80 bk.ru> writes:
 - can be stored more compacted than source coz words (var 
 names, keywords) are repeated many times through source. and


no need store any unittests and doc/comments in AST-packages. 
docs will be generated to html. unittests of packages need only 
to builders of those packages. if u needed u take sources of 
packages, build it and run unittests for it
Jun 08 2019
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 8 June 2019 at 10:35:58 UTC, KnightMare wrote:

 +1 for AST.


This doesn't make a significant difference. Parsing D source into 
an AST is quick and easy. The problem is applying that ast to 
user types.
Jun 08 2019
parent reply KnightMare <black80 bk.ru> writes:
On Saturday, 8 June 2019 at 11:58:36 UTC, Adam D. Ruppe wrote:

 +1 for AST.


 This doesn't make a significant difference. Parsing D source 
 into an AST is quick and easy. The problem is applying that ast 
 to user types.


one more idea. allow to store in AST-packages user/any metadata.
LDC can store there optimized LLVM-IR code for AST-tree, so 
compilation time will be: compile user code only, link and write 
result to FS
Jun 08 2019
parent reply KnightMare <black80 bk.ru> writes:
On Saturday, 8 June 2019 at 12:26:56 UTC, KnightMare wrote:

 LDC can store there optimized LLVM-IR code for AST-tree, so 
 compilation time will be: compile user code only, link and 
 write result to FS


template code will be taken from AST still.
(we just cant compile template for all types in universe to 
IR-code. also maybe IR can store some generics. dunno)
Jun 08 2019
parent KnightMare <black80 bk.ru> writes:
On Saturday, 8 June 2019 at 12:32:54 UTC, KnightMare wrote:

 On Saturday, 8 June 2019 at 12:26:56 UTC, KnightMare wrote:


I watched a video now from conference 
https://youtu.be/OsOfTVm2ExY?t=2107
where Walter said that deserialization some AST can be same speed 
as parsing source. should to think
Jun 08 2019